On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
You don't have a buffer overflow problem if you read it in a memory-safe way i.e. read it in chunks and realloc when you reach the size of your allocation.
What you will have is a potential denial-of-service attack - although this one isn't particularly great because there's zero amplification (they might as well just send garbage into your firewall)
In regular unicode, a grapheme can be made up of an arbitrary number of codepoints (and thus an arbitrary number of bytes), which does cause issues at times.
I don’t think there’s harm is speccing out the arbitrary encoding and then having a different spec that references that spec but puts hard limits on it. Many rfcs are like that.
Yes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later.
UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).
Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.
Correct me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character.
This would be because you can stack modifications as long as you like.
The first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.
Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.
I agree C has a lot of problems. The nice thing about C is that it’s a rigorous standard (C90, C99, C23, etc.) with multiple actively maintained implementations (GCC, LLVM/clang, TCC, etc.) so standards compliant C code should (in theory) compile and run for the foreseeable future.
Python, Rust, Perl, etc. are all languages with a single actively maintained implementation and haven’t had the rigorous standardization C and POSIX utilities (Awk, /bin/sh, sed, etc.) have had.
I'm failing to see why an embedded environment would have any need for a new encoding format, which is kind of my point: the types of things that are going to be written in C are not the ones going to be adopting completely new backwards-incompatible standards anyhow. If we refuse to try something based on how it would interact of the ecosystem that would likely never consider adopting something like it in the first place, we're literally fixing our computing to constraints from half a century ago and counting. Nobody who is going to write C would stop because of something like a new Unicode scheme with much larger encoding widths, so why should that be an argument against it happening?
A reasonable person would assume you’re talking about UTF-8000. It’s not completely new: RFC2279, the original UTF-8 proposal, worked exactly like UTF-8000 for codepoints 31 bits or smaller in size. It’s not backwards-incompatible: UTF-8000 is exactly like UTF-8 for 1, 2, and 3-byte long codepoints, and like UTF-8 codepoints for 4-byte long codepoints with a value of 0x10_ffff or smaller (so all UTF-8 codepoints encoded with the first byte being 0b1111_00xx or starting with the bytes 0b1111_0100 0b1000_xxxx).
It’s a backwards compatible way of encoding numbers in UTF-8 larger than 0x10_ffff or (0x7fff_ffff with the original RFC2279 proposal).
I agree that C isn’t the best language to start a new programming project in. There are things I don’t like about Rust, mainly that there’s only one implementation of it, but if I were to start a new project needing the speed of a system programming language, it makes a lot of sense.
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
I think that wouldn't change much. They would just make use of more grapheme clusters.
For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.
It's a joke based on the construction of emojis? Female plus fire engine obviously denotes a female fire engine but has apparently been repurposed as a female firefighter instead
I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI.
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
The number of glyphs available by adding additional bytes drops exponentially because each subsequent byte has one less bit available.
So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.
I don't quite get it. 5-byte utf-8 encoding gets extra 5 bits compared to 4 byte, and 6-byte gets extra 10 bits. If you were thinking about bits in leading byte, then yes, you are losing one bit for every extra trailing byte, but you also get 6 bits from it. So adding a byte gives you extra 5 bits.
It seems when designing Unicode most energy went into emoji. And there was nothing left for fancy things like fixed-length string buffers. The only explaination why UTF8 Buffers aren't compatible with UTF16 Buffers... is a really strong emoji...
Important to recognize that characters have individuality, that's why there can only be a limited number of them. Unicode is enumerating a finite set of things, not encoding an infinite set. Aenything without this property - any generic form of encoding - is not characters, it's something else like images. If it's not in any alphabet it shouldn't be in unicode, you should use an escape tag for image data instead. (Emojis probably shouldn't, but they do behave like an alphabet)
There cannot be 4 trillion characters because humans would need to know all of them and humans cannot know that many things.
256-bit addresses would be long enough proper cryptographic addressing. (128 is barely not enough due to how they're allocated)
512 would also for cryptographic addresses within cryptographic subnets (or 3 128-bit layers plus some extra bits), and 1024 would allow for up to 7 layers. It's not a completely silly idea.
Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it, which isn't adequate in cryptography. Individual addresses can also be created with 112 bits of entropy, which is barely adequate. That's why I say 128 bits total isn't enough for cryptography plus subnetting.
Is there a significant reason to want those to use the same address space as the normal internet?
Especially with the layering, I would think you could put this in a routing header?
> Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it, which isn't adequate in cryptography. Individual addresses can also be created with 112 bits of entropy, which is barely adequate.
If you make it cost a dollar to compute, and you account for that getting 1000x cheaper during the time it needs to stay secure, that's around 70 trillion dollars to attack? I would call that "adequate". And ten cents / 7 trillion too.
(This does assume an algorithm that runs reasonably fast on normal hardware, so that "2^56 times the cost" mostly holds up in the face of dedicated cracking chips.)
Since the Unicode Consortium isn't going to actually assign those code points, this is functionally just a scheme for encoding variable-length integers designed as an extension of UTF-8 more or less arbitrarily.
There's a long history of designs for these (https://en.wikipedia.org/wiki/Variable-length_integer) that the author might be interested in. I used to think about these things myself, including the "zigzag encoding" for signed values (not a difficult idea; this "marvelous bijective mapping" is the standard one used in math class to demonstrate that the integers are countable, and the nice implementation properties are a consequence of the choice to "zig" from 0 to -1 first combined with how two's-complement works).
C0 and C1 can do so though according to the author.
> because all 4 of 2-byte UTF-8's mandatory content bits lie in the first-and-final start byte, we can explicitly rule out 11000000 (0xC0) and 11000001 (0xC1) as permanently invalid bytes. They will never ever appear anywhere in a valid UTF-8000 code unit!
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
> "UTF-8" support was often just single byte, so it was little more than ASCII
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
Your opinion on the BOM isn't wrong, but it's also not germaine to whether or not you're actually following the spec. The spec is the spec. If you don't like it you can get the spec changed. You don't get to ignore the spec and then claim support. That's not how standards work. "I don't like it," isn't a good explanation.
Otherwise I'd be inclined to fix the spelling error in the HTTP referrer.
> BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing
How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).
Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.
> How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM?
Simple: switch to UTF-8 as the only encoding standard for sharing text data, keep UTF-32 as 'internal' runtime format for random access to codepoints, and get rid of all other legacy encodings (UCS-2, UTF-16, Extended ASCII with code pages, and all the other region specific encodings that popped up in the 70s and 80s because UTF-8 wasn't invented yet.
This general switch to UTF-8 should have happend in the mid-to-late 1990s (e.g. together with the web becoming popular), and Microsoft alone is to blame for dragging this shit along for the next three decades. If all Microsoft tools would only save text data as UTF-8 starting by the end of the last century, but still support reading all sorts of encodings for a decade or so, the transition would have been finished by 2010. Alas, that never happened.
And tbh, the file size argument for alphabets that don't fit into 7-bit ASCII doesn't really make sense anymore today where images and videos make up the vast majority of data volume.
CJK characters is the one big weak point of UTF-8. It forces them to be 3 bytes large instead of the local codepage which is 2 bytes per character, or UTF-16 which is also 2 bytes per character.
IMHO a 30 percent size increase for sharing text data is really not a problem when most data sharing (by volume) happens via audio, video and image data which is infinitely bigger. We obviously have both the bandwidth and storage volume for those 30% increased size for text data, it wouldn't even show up as a tiny blip among all the video, audio and image files.
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
If it's just a general Transformation Format and Unicode will never assign those code points, it'd be just TF-8. I can think of a second way to expand that abbreviation.
Why encode the length at all in the extension? Either the next byte is a continuation byte or not. Are there real world use cases where the length encoding is used in UTF-8 where individual bytes don’t have to be read anyway (for example, codepoint counting, but in practice what good is that
without knowing if the code points are combining)? Even in the case where you just want a byte count, an invalid UTF-8 stream won’t respect the first count marker. The arbitrarily sized count is no better than an arbitrarily sized code point - you’d still have to handle all the same resource and sizing issues for safety. And then what, are you going to allocate a buffer that can fit a length that needs more than 64-bits to represent?
It's a brilliant hack. It means that when you're dealing with valid UTF-8 strings, various kinds of string relationships enjoy a homomorphism between byte strings and Unicode strings. Specifically, where S and T are Unicode strings and E is the UTF-8 encoding operation:
• E(S concatenated with T) = E(S) concatenated with E(T)†
• S starts with T iff E(S) starts with E(T)
• S ends with T iff E(S) ends with E(T)
• S contains T iff E(T) contains E(T)
This means that, as long as you know your encoded strings don't contain invalid UTF-8, you can do a great deal of your string processing on the byte-encoded form, which is enormously faster than decoding the strings before doing the string processing, permits efficient radix-256 tries, and is much smaller in many common cases.
It also bounds the work you have to do if you're processing a string starting from the end, while certain other encodings require looking back in the string arbitrarily far to figure out how to interpret the bytes you're looking at. This is particularly important for Boyer–Moore string search, but in many cases it's also an express trip to getting your code featured on https://www.tumblr.com/accidentallyquadratic.
In short, yes.
______
†Surprisingly, even this most basic homomorphism is not true of many other character encodings, which may need extra bytes to be inserted in between.
Having the length of encoded in each codepoint means that you know if a stream ends at codepoint boundary or if it has a truncated final codepoint. You could probably do away with the length and maintain this property if you had distinct forms for start, continuer-mid and continuer-final bytes.
Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.
1. You can partition an input file at any offsets, parallelize, and adjust partition boundaries to a valid offset independently.
Without the property, parallelization is hard.
This is how mapreduce has been used to process large text files, except at line boundaries.
Now, for this that might not be a useful enough property, given that we already do similar things for newlines, and UTF-8 guarantees ASCII is always recognizable and hence newlines are always recognizable.
2. It might have been more useful in the era of dial-up where we still had occasional corrupted bytes in the transmission.
3. It helps regain sanity if e.g. a background process outputs bytes that get interleaved at the tty. For example, cat a large text file, the write boundaries won't always align at UTF-8 boundaries, then have a background process output get interleaved in an unfortunate way. If it self-synchronizes, it'll knock itself back into sync after a small amount of garbage.
I don’t see how UTF-8000 doesn’t have it. The first byte of any code point is either 0xxxxxxx or 10xxxxxxx, which is distinct from all non-first bytes which are 11xxxxxx. Thus any UTF-8000 sub-bytestring must necessarily have the start aligned at a code point boundary, at which point all the subsequent bytes are interpreted as codepoints in the same way.
> Does anyone have a good example of when it‘s useful?
It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
> Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.
Isn't a limit by definition no longer dependent on n?
U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.
2-3 bytes are not much space for anything. Sure, you could use multiple successive ones of these code points and define your own "continuation" encoding in these ranges, but that doesn't seem right to me somehow
That combination is how it already works. You can build combined "characters" (grapheme clusters) from multiple code points, e.g. you could have a "base emoji" followed by a "modifier" emoji, and AFAIK that's how emojis with different skin colors work (one code point for the base emoji (e.g. 'thumbs up'), and a number of skin color modification code points which can be applied to all emojis that involve skin color.
I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course allowing a script to be written in a single codepoint is the much more elegant solution.
It also neatly solves the problem of how to write Unicode strings inside scripts inside Unicode strings and also scripts inside Unicode strings inside scripts inside Unicode strings.
Another one: encode instructions on how to draw the glyph into the text itself. The string becomes both the text and the font. Why not make it turing complete and as powerful/complex as TTF.
Imagine someone using the same fully customized emoji multiple times in the same text. Seems like a waste of space. Maybe better to encode just a UUID, and send the image codebook separately.
IIRC UTF-8 proposal used to go to 6 bytes as they already felt that not having part of the code point in the first byte was an issue somehow.
I think at least 7 should be allowed. 42 bits could contain all sorts of information.
"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.
I was about to immediately suggest UCS-X, and then pleased that the author offers a fair comparison to UCS-X and many other alternatives. Great. Too bad UCS-X is possibly more memorable name.
On the opposite end of the spectrum from this idea, I've been thinking for a long time now that much of what's weird/redundant about Unicode comes from its self-synchronization requirement. And that you could create a very compact and flexible "re-embedding" of Unicode if you dropped this requirement.
Self-synchronization is the idea that if you take an arbitrary Unicode-encoding-encoded text stream, and then do any combination of 1. flipping bits in it at random, 2. injecting random extra whole bytes into the stream, and 3. dropping random whole bytes from the stream, then each such change will corrupt at most one Unicode code-unit in the stream, and from that, corrupt at most one semantic grapheme-cluster encoded by the stream. No such change will corrupt the stream itself so as to leave the stream in an invalid/indeterminate state that a Unicode parser can't know how to recover from. You'll have a one-character-wide "hole", and then the stream will resume. Unless that hole occurred at a character that's critical to the stream's meaning on an application-semantics level, the document will still be valid/useful (especially for archival/forensic-recovery purposes); just like a printed document is still valid/useful even if you drip a bit of ink on it.
Unicode encodings are self-synchronizing at the byte-pattern level. But, much less often discussed, Unicode itself is also designed to be self-synchronizing in how it encodes interactions between code-units. (In other words, Unicode is designed to never have modal or stateful semantics, beyond the boundary of a single grapheme cluster.) And this really constrains how certain Unicode features can be, and historically have been, designed.
If you want a run of Unicode code-units to all be "tagged" or "colored" with some property, then, due to the self-synchronization requirement (i.e. due to the assumption that any single one of those bytes could be corrupted or blown away, including whatever metadata-encoding bytes your scheme wants to use), you have to either:
- define a "pre-colored" alphabet, and express your tagged content in that alphabet. (Think of e.g. the Unicode flag codepoints [https://en.wikipedia.org/wiki/Regional_indicator_symbol], which are essentially a special namespaced copy of the roman upper-case alphabet intended to be used only to spell out two-letter contiguous pairs that are [or at some point were] valid ISO country codes; where, when used in this way, the resulting 'colored'-letter-pair sequence has 'flag semantics', i.e. is meant to be rendered as a flag and machine-legible as a flag)
- or individually tag each and every one of those codepoints with its own tag/color codepoint (as in Unicode variation selectors)
- or interject binary-infix "operator" codepoints as glue between each codepoint in a codepoint sequence, so that those "operator" codepoints each affix together their immediate sibling codepoints, essentially constructing an abstract-Unicode-semantics list ADT "cons by cons", so that said list then may then be assigned its own semantics, e.g. being treated as a single grapheme-cluster with its own rendering (as in e.g. ZERO WIDTH JOINER used in its role in constructing complex emoji)
These encodings are all very high-overhead; but these are the kinds of trade-offs you have to make for self-synchronization to work.
And these trade-offs are sensible to make... if you're Ken Thompson in 1992, having to consider e.g. plaintexts being transmitted over raw RS232, or filesystems that just blast bytes to a spinning-rust disk without so much as a checksum, and then read them back "blindly" years later with that disk potentially highly-degraded.
But what if you live in the modern world, and you only care about holding and manipulating known-length strings in memory and/or embedded into code-signed (and thereby hashed) binaries; checksummed-block filesystems over rarely-corrupting NVMe; and transmission of data mostly over encrypted-stream protocols, where even the rare packet-level corruption that still TCP-checksums correctly, doesn't decrypt successfully, and therefore causes TLS-level retransmission?
Well, then you could define a much-more-concise reformulation of Unicode, that uses all the "forbidden" semantics-encoding techniques that Unicode itself avoids due to the self-synchronization constraint.
Where by "reformulation", I mean: a standard that keeps parity with Unicode in terms of what it can encode; and which at all times maintains a clearly-defined lossless bijective transformation between it and Unicode, evolving in lockstep with Unicode; but where Unicode and this formulation have their own distinct universes of codepoints, that compose using different rules, into the same ultimate sets of reachable grapheme-clusters with the same text-segmentation/collation/etc semantics.
I'm honestly kind of surprised that there isn't already a project somewhere to define an alternative Unicode formulation that looks like this.
It'd not only be potentially a highly-efficient representation for many in-memory string operations (that text shaping libraries would also love); it'd also likely perform impressively (compared to regular UTF-8) as a canonical representation for documents used to train+prompt LLMs. It'd give "more meaning per token", via all the repeated-per-codel overhead becoming once-per-sequence overhead; and it'd also allow many layers of meaning that are currently encoded via in-band protocols (ANSI escape codes, Markdown, HTML/XML, etc) that the LLM needs to learn additional recognition logic for, to instead be parsed out "during" initial text-stream recognition.
(And it's taking me real willpower not to go into depth on all the features such a formulation could have, and all the benefits it could provide. I should probably stop here before I nerd-snipe myself!)
"your first 2 extensions (5 and 6 bytes) were clearly envisioned.
the standard (up to 4 bytes) was created to cover the size of
unicode. i thought any more description would be a waste of
paper. i think your extension from 7 to 8 bytes is a little hoaky.
i requires reading the whole string rather than "knowing" the
number of follow on bytes. so, i think the only thing new is the
7 byte version.
i appreciate the mail, but i really dont think it is useful. it is
like replacing ipv6 with ipv50."
It’s a fairly simple encoding which is UTF-8 compatible and allows codepoints to be arbitrarily large. It’s a real encoding proposal with a simple rigorous specification. I don’t have a use case for needing arbitrarily large codepoints, but for people who do, this makes more sense than hacks such as having surrogates in Unicode’s hi-bit private use area.
I have entertained ideas like this every since becoming familiar with the original RFC2279 UTF-8 well over two decades ago, before UTF-8 was subsequently restricted to 21 bits.
Another idea that would work is to start an encoding with 0b10xxxxxx have subsequent 0b10xxxxxx bytes continue the encoding, and end it with 11xxxxxx. 0b1000_0000 (i.e. 128) is not allowed for the first byte of this sequence, and we add 127 to the resulting number (so each code point has only one representation) This way, the ♥ symbol (that’s 0x2665 or 9829 in decimal) would be represented as follows:
This allows 18 bits to be encoded in three bytes, and, like UTF-8000, allows arbitrarily long sequences.
In the real world, the last time I needed to have a custom encoding, I went the other way and converted Unicode in to a 7-bit ASCII compatible encoding, where most ASCII control characters were converted into glyphs, as follows:
The reason for this encoding is that it allows me to write stuff in languages I am actually fluent in (English and Spanish), and adds a handful of useful non-ASCII punctuation (smart quotes, etc.).
“.” represents a control character here; only “null” (0x00), “line feed” (0x0a), “form feed” (0x0c), “tab” (0x09), “carriage return” (0x0d), and “escape” (0x1b) are control characters; the rest are printable glyphs. I used this character set for my blog, since my blog is processed using HTMLDOC (which I learned the hard way is buggy with full Unicode) and my own Lua script.
With Lua, Unicode regexes are not readily supported because its regular expression engine assumes a codepoint is only one byte long when forming character classes such as %u for upper case letters. My Lua code has character classes for the non-ASCII glyphs such as %t for all letters in this custom encoding.
I have made this encoding 7 bits because that gives me the option to use the eight bit for future expansion.
The real reason for so few glyphs is because, in addition to having things work nicely with code which assumes 8-bit codepoints, with the modern web, I need to send to the viewer the font a page will be rendered with (it would had been nice if Microsoft had open sourced Verdana, Georgia, and the rest of the core fonts, and had those fonts became a part of the browser standards, so we could have cross-platform font stacks, but oh well) [1], so I aggressively subset the font to minimize the page load time.
[1] I still wish for the day when Apple, Google, Microsoft, and Mozilla come together, decide to include “Noto” and “Noto Serif” with their browsers, so “font-family: Noto (Serif)” always renders the exact same font without having to download a font over the network when loading a web page.
Actually, my proposed encoding only needs to add 64 to the code point (or subtract 64 when encoding a code point), not 128. That in mind, ♥ is encoded as follows:
C99 can do up to over 10 bytes long (uint64_t) without a bignum library since 10 bytes gives us 60 bits. C23 gives us a bignum library so the sky’s the limit.
Again, I only need 7 bits to represent every character the font for my blog has.
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.
Allowing more than one lead byte is a complication of the existing standard. As many others have pointed out, we have plenty of coding space without that (e.g. by allowing 5- and 6-byte UTF-8 again), so the case for the extra complexity is currently not compelling.
Perhaps we can use a generalized Unicode-like encoding space for LLM text tokens. A text tokenization scheme uses up a few hundred thousand entries, with say one thousand new entries added annually. These can be called amojis, meaning AI mojis.
On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
You don't have a buffer overflow problem if you read it in a memory-safe way i.e. read it in chunks and realloc when you reach the size of your allocation.
What you will have is a potential denial-of-service attack - although this one isn't particularly great because there's zero amplification (they might as well just send garbage into your firewall)
DOS in what way? Can you clarify? Thx.
In regular unicode, a grapheme can be made up of an arbitrary number of codepoints (and thus an arbitrary number of bytes), which does cause issues at times.
I don’t think there’s harm is speccing out the arbitrary encoding and then having a different spec that references that spec but puts hard limits on it. Many rfcs are like that.
OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
The bonus is synchonizing at arbitrary point in stream and that ASCII is UTF-8
Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
Yes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later.
UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).
Even better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width.
Truly the worst of all worlds.
Correct me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character. This would be because you can stack modifications as long as you like.
What you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.
The first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.
Limit the codepoint to the number of atoms in the universe (less than 32 bytes).
> this just screams buffer overflow problems
Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.
I agree C has a lot of problems. The nice thing about C is that it’s a rigorous standard (C90, C99, C23, etc.) with multiple actively maintained implementations (GCC, LLVM/clang, TCC, etc.) so standards compliant C code should (in theory) compile and run for the foreseeable future.
Python, Rust, Perl, etc. are all languages with a single actively maintained implementation and haven’t had the rigorous standardization C and POSIX utilities (Awk, /bin/sh, sed, etc.) have had.
It’s not about the language. It’s about the runtime environment. Not everything is fully developed UI running on beefy CPUs with gigs of RAM.
Sometimes the environment forces a language choice.
I'm failing to see why an embedded environment would have any need for a new encoding format, which is kind of my point: the types of things that are going to be written in C are not the ones going to be adopting completely new backwards-incompatible standards anyhow. If we refuse to try something based on how it would interact of the ecosystem that would likely never consider adopting something like it in the first place, we're literally fixing our computing to constraints from half a century ago and counting. Nobody who is going to write C would stop because of something like a new Unicode scheme with much larger encoding widths, so why should that be an argument against it happening?
“completely new backwards-incompatible standards”
A reasonable person would assume you’re talking about UTF-8000. It’s not completely new: RFC2279, the original UTF-8 proposal, worked exactly like UTF-8000 for codepoints 31 bits or smaller in size. It’s not backwards-incompatible: UTF-8000 is exactly like UTF-8 for 1, 2, and 3-byte long codepoints, and like UTF-8 codepoints for 4-byte long codepoints with a value of 0x10_ffff or smaller (so all UTF-8 codepoints encoded with the first byte being 0b1111_00xx or starting with the bytes 0b1111_0100 0b1000_xxxx).
It’s a backwards compatible way of encoding numbers in UTF-8 larger than 0x10_ffff or (0x7fff_ffff with the original RFC2279 proposal).
I agree that C isn’t the best language to start a new programming project in. There are things I don’t like about Rust, mainly that there’s only one implementation of it, but if I were to start a new project needing the speed of a system programming language, it makes a lot of sense.
> UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
> […] why limit yourself to 24-bit numbers?
For compatibility with UTF-16:
* https://datatracker.ietf.org/doc/html/rfc3629#section-12* https://en.wikipedia.org/wiki/UTF-16
The original spec had 31 bits (the UTF-32/UCS-4 range):
* https://datatracker.ietf.org/doc/html/rfc2279
* https://en.wikipedia.org/wiki/UTF-32
> ...24-bit numbers?
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
I think that wouldn't change much. They would just make use of more grapheme clusters.
For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.
I wonder how they're going to encode a female fire engine in the future.
No worries, that would use female sign, not woman.
I don't understand why you are saying this. That just seems a bit inappropriate
It's a joke based on the construction of emojis? Female plus fire engine obviously denotes a female fire engine but has apparently been repurposed as a female firefighter instead
I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI.
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
We still have about 85% of codepoint space unused. Hopefully, by the time it becomes a problem, UTF-16 will be long dead
i hope so too, but UTF-16 being used by languages such as java and javascript makes me fear it might be here to stay.... i hope im wrong
The number of glyphs available by adding additional bytes drops exponentially because each subsequent byte has one less bit available.
So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.
I don't quite get it. 5-byte utf-8 encoding gets extra 5 bits compared to 4 byte, and 6-byte gets extra 10 bits. If you were thinking about bits in leading byte, then yes, you are losing one bit for every extra trailing byte, but you also get 6 bits from it. So adding a byte gives you extra 5 bits.
Yeah, you’re right. I might have attempted to do mental arithmetic before coffee…
Utf-16 is famously used by Windows for everything important as well.
Thankfully, this is changing. Win32's 'A' ANSI/ASCII APIs now support UTF-8 if your app declares such a wish. https://stackoverflow.com/a/69181417/1350209
And UTF-8 isn't even fully compatible with windows UTF-16 - UTF8 can't encode a lot of truncated windows UTF-16 filenames.. You need WTF-8 for that.
https://artoria2e5.github.io/XB18030/
It seems when designing Unicode most energy went into emoji. And there was nothing left for fancy things like fixed-length string buffers. The only explaination why UTF8 Buffers aren't compatible with UTF16 Buffers... is a really strong emoji...
But by then, the 4-byte limit of UTF-8 will itself have ossified. Even today, reverting back to the 6-byte limit is nigh impossible.
By then we will have quaternary quantum computers and FTL circuits where the information appears request it before you
> No special cases introduced. All properties preserved.
I don’t actually know if this is LLM-generated, but phrasing like this is weirdly triggering to me now
Yeah that kind of line is what I see all the time in my chats. Even worse worse is when they put it in code comments.
Just limit it to 8 bytes at which point you always do 'know the number of follow on bytes' from the first byte.
Nobody needs more than 4.47 trillion characters. (famous last words)
Important to recognize that characters have individuality, that's why there can only be a limited number of them. Unicode is enumerating a finite set of things, not encoding an infinite set. Aenything without this property - any generic form of encoding - is not characters, it's something else like images. If it's not in any alphabet it shouldn't be in unicode, you should use an escape tag for image data instead. (Emojis probably shouldn't, but they do behave like an alphabet)
There cannot be 4 trillion characters because humans would need to know all of them and humans cannot know that many things.
> Ken Thompson: "...i really dont think it is useful. it is like replacing ipv6 with ipv50"
256-bit addresses would be long enough proper cryptographic addressing. (128 is barely not enough due to how they're allocated)
512 would also for cryptographic addresses within cryptographic subnets (or 3 128-bit layers plus some extra bits), and 1024 would allow for up to 7 layers. It's not a completely silly idea.
Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it, which isn't adequate in cryptography. Individual addresses can also be created with 112 bits of entropy, which is barely adequate. That's why I say 128 bits total isn't enough for cryptography plus subnetting.
Is there a significant reason to want those to use the same address space as the normal internet?
Especially with the layering, I would think you could put this in a routing header?
> Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it, which isn't adequate in cryptography. Individual addresses can also be created with 112 bits of entropy, which is barely adequate.
If you make it cost a dollar to compute, and you account for that getting 1000x cheaper during the time it needs to stay secure, that's around 70 trillion dollars to attack? I would call that "adequate". And ten cents / 7 trillion too.
(This does assume an algorithm that runs reasonably fast on normal hardware, so that "2^56 times the cost" mostly holds up in the face of dedicated cracking chips.)
The other option is to start making more software AF-agnostic and not assume that an address is always an ipv4 or ipv6 address.
At some point it just collapses into a sort of Huffman coding of every possible 4096 bit embedding vector.
Since the Unicode Consortium isn't going to actually assign those code points, this is functionally just a scheme for encoding variable-length integers designed as an extension of UTF-8 more or less arbitrarily.
There's a long history of designs for these (https://en.wikipedia.org/wiki/Variable-length_integer) that the author might be interested in. I used to think about these things myself, including the "zigzag encoding" for signed values (not a difficult idea; this "marvelous bijective mapping" is the standard one used in math class to demonstrate that the integers are countable, and the nice implementation properties are a consequence of the choice to "zig" from 0 to -1 first combined with how two's-complement works).
FF bytes are an easy way to identify an invalid UTF-8 file. This idea doesn't have that property.
C0 and C1 can do so though according to the author.
> because all 4 of 2-byte UTF-8's mandatory content bits lie in the first-and-final start byte, we can explicitly rule out 11000000 (0xC0) and 11000001 (0xC1) as permanently invalid bytes. They will never ever appear anywhere in a valid UTF-8000 code unit!
True, but not all non-UTF8 bytestrings contain 0xFF bytes, so it’s not very useful in practice.
Yes, I agree.
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
> "UTF-8" support was often just single byte, so it was little more than ASCII
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
Your opinion on the BOM isn't wrong, but it's also not germaine to whether or not you're actually following the spec. The spec is the spec. If you don't like it you can get the spec changed. You don't get to ignore the spec and then claim support. That's not how standards work. "I don't like it," isn't a good explanation.
Otherwise I'd be inclined to fix the spelling error in the HTTP referrer.
> BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing
How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).
Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.
> How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM?
Simple: switch to UTF-8 as the only encoding standard for sharing text data, keep UTF-32 as 'internal' runtime format for random access to codepoints, and get rid of all other legacy encodings (UCS-2, UTF-16, Extended ASCII with code pages, and all the other region specific encodings that popped up in the 70s and 80s because UTF-8 wasn't invented yet.
This general switch to UTF-8 should have happend in the mid-to-late 1990s (e.g. together with the web becoming popular), and Microsoft alone is to blame for dragging this shit along for the next three decades. If all Microsoft tools would only save text data as UTF-8 starting by the end of the last century, but still support reading all sorts of encodings for a decade or so, the transition would have been finished by 2010. Alas, that never happened.
And tbh, the file size argument for alphabets that don't fit into 7-bit ASCII doesn't really make sense anymore today where images and videos make up the vast majority of data volume.
CJK characters is the one big weak point of UTF-8. It forces them to be 3 bytes large instead of the local codepage which is 2 bytes per character, or UTF-16 which is also 2 bytes per character.
IMHO a 30 percent size increase for sharing text data is really not a problem when most data sharing (by volume) happens via audio, video and image data which is infinitely bigger. We obviously have both the bandwidth and storage volume for those 30% increased size for text data, it wouldn't even show up as a tiny blip among all the video, audio and image files.
As are FE, FD, FC, FB, FA, F9, F8, F7, F6 and F5.
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
On the naming issue with a "UTF-8" prefix, I suggest "UTF+8" which puns earlier in the string for a double effect. Fun paper!
If it's just a general Transformation Format and Unicode will never assign those code points, it'd be just TF-8. I can think of a second way to expand that abbreviation.
Or UTF-∞
Why encode the length at all in the extension? Either the next byte is a continuation byte or not. Are there real world use cases where the length encoding is used in UTF-8 where individual bytes don’t have to be read anyway (for example, codepoint counting, but in practice what good is that without knowing if the code points are combining)? Even in the case where you just want a byte count, an invalid UTF-8 stream won’t respect the first count marker. The arbitrarily sized count is no better than an arbitrarily sized code point - you’d still have to handle all the same resource and sizing issues for safety. And then what, are you going to allocate a buffer that can fit a length that needs more than 64-bits to represent?
It's a brilliant hack. It means that when you're dealing with valid UTF-8 strings, various kinds of string relationships enjoy a homomorphism between byte strings and Unicode strings. Specifically, where S and T are Unicode strings and E is the UTF-8 encoding operation:
• E(S concatenated with T) = E(S) concatenated with E(T)†
• S starts with T iff E(S) starts with E(T)
• S ends with T iff E(S) ends with E(T)
• S contains T iff E(T) contains E(T)
This means that, as long as you know your encoded strings don't contain invalid UTF-8, you can do a great deal of your string processing on the byte-encoded form, which is enormously faster than decoding the strings before doing the string processing, permits efficient radix-256 tries, and is much smaller in many common cases.
It also bounds the work you have to do if you're processing a string starting from the end, while certain other encodings require looking back in the string arbitrarily far to figure out how to interpret the bytes you're looking at. This is particularly important for Boyer–Moore string search, but in many cases it's also an express trip to getting your code featured on https://www.tumblr.com/accidentallyquadratic.
In short, yes.
______
†Surprisingly, even this most basic homomorphism is not true of many other character encodings, which may need extra bytes to be inserted in between.
Having the length of encoded in each codepoint means that you know if a stream ends at codepoint boundary or if it has a truncated final codepoint. You could probably do away with the length and maintain this property if you had distinct forms for start, continuer-mid and continuer-final bytes.
Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
Rely on it? Not that I can remember.
However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.
I can think of 3 things:
1. You can partition an input file at any offsets, parallelize, and adjust partition boundaries to a valid offset independently.
Without the property, parallelization is hard.
This is how mapreduce has been used to process large text files, except at line boundaries.
Now, for this that might not be a useful enough property, given that we already do similar things for newlines, and UTF-8 guarantees ASCII is always recognizable and hence newlines are always recognizable.
2. It might have been more useful in the era of dial-up where we still had occasional corrupted bytes in the transmission.
3. It helps regain sanity if e.g. a background process outputs bytes that get interleaved at the tty. For example, cat a large text file, the write boundaries won't always align at UTF-8 boundaries, then have a background process output get interleaved in an unfortunate way. If it self-synchronizes, it'll knock itself back into sync after a small amount of garbage.
I don’t see how UTF-8000 doesn’t have it. The first byte of any code point is either 0xxxxxxx or 10xxxxxxx, which is distinct from all non-first bytes which are 11xxxxxx. Thus any UTF-8000 sub-bytestring must necessarily have the start aligned at a code point boundary, at which point all the subsequent bytes are interpreted as codepoints in the same way.
Right, UTF-8000 does have this property. Too late to edit my comment now, thanks for noticing that.
> Does anyone have a good example of when it‘s useful?
It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).
Only if its normalized unicode. Most strings are not, and utf-8 does not guarantee normalization.
Only if they've already both undergone normalization to NFC or NFD.
Some ideas of what to do with this space:
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
> Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.
Isn't a limit by definition no longer dependent on n?
U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.
2-3 bytes are not much space for anything. Sure, you could use multiple successive ones of these code points and define your own "continuation" encoding in these ranges, but that doesn't seem right to me somehow
That combination is how it already works. You can build combined "characters" (grapheme clusters) from multiple code points, e.g. you could have a "base emoji" followed by a "modifier" emoji, and AFAIK that's how emojis with different skin colors work (one code point for the base emoji (e.g. 'thumbs up'), and a number of skin color modification code points which can be applied to all emojis that involve skin color.
The point is probably to encode the emoji (image) in the codepoint itself.
> some unicode programming script (running Doom)
I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course allowing a script to be written in a single codepoint is the much more elegant solution.
It also neatly solves the problem of how to write Unicode strings inside scripts inside Unicode strings and also scripts inside Unicode strings inside scripts inside Unicode strings.
(*) in RFC 04-01-2027
[1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)
in RFC 04-01-2027
I, for one, welcome our new time-traveling overlords.
Another one: encode instructions on how to draw the glyph into the text itself. The string becomes both the text and the font. Why not make it turing complete and as powerful/complex as TTF.
Imagine someone using the same fully customized emoji multiple times in the same text. Seems like a waste of space. Maybe better to encode just a UUID, and send the image codebook separately.
> - some unicode programming script (running Doom)
> - ?
Bad Apple, oh, wait: https://github.com/hsfzxjy/Bad-Apple-Font
One more idea:
- Unicode-as-a-character, meaning: a whole text displayed as if it were one character, made for infinite zooming
IIRC UTF-8 proposal used to go to 6 bytes as they already felt that not having part of the code point in the first byte was an issue somehow.
I think at least 7 should be allowed. 42 bits could contain all sorts of information.
"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.
Alright, so now we can fit an entire LLM model into one character.
I was about to immediately suggest UCS-X, and then pleased that the author offers a fair comparison to UCS-X and many other alternatives. Great. Too bad UCS-X is possibly more memorable name.
On the opposite end of the spectrum from this idea, I've been thinking for a long time now that much of what's weird/redundant about Unicode comes from its self-synchronization requirement. And that you could create a very compact and flexible "re-embedding" of Unicode if you dropped this requirement.
Self-synchronization is the idea that if you take an arbitrary Unicode-encoding-encoded text stream, and then do any combination of 1. flipping bits in it at random, 2. injecting random extra whole bytes into the stream, and 3. dropping random whole bytes from the stream, then each such change will corrupt at most one Unicode code-unit in the stream, and from that, corrupt at most one semantic grapheme-cluster encoded by the stream. No such change will corrupt the stream itself so as to leave the stream in an invalid/indeterminate state that a Unicode parser can't know how to recover from. You'll have a one-character-wide "hole", and then the stream will resume. Unless that hole occurred at a character that's critical to the stream's meaning on an application-semantics level, the document will still be valid/useful (especially for archival/forensic-recovery purposes); just like a printed document is still valid/useful even if you drip a bit of ink on it.
Unicode encodings are self-synchronizing at the byte-pattern level. But, much less often discussed, Unicode itself is also designed to be self-synchronizing in how it encodes interactions between code-units. (In other words, Unicode is designed to never have modal or stateful semantics, beyond the boundary of a single grapheme cluster.) And this really constrains how certain Unicode features can be, and historically have been, designed.
If you want a run of Unicode code-units to all be "tagged" or "colored" with some property, then, due to the self-synchronization requirement (i.e. due to the assumption that any single one of those bytes could be corrupted or blown away, including whatever metadata-encoding bytes your scheme wants to use), you have to either:
- define a "pre-colored" alphabet, and express your tagged content in that alphabet. (Think of e.g. the Unicode flag codepoints [https://en.wikipedia.org/wiki/Regional_indicator_symbol], which are essentially a special namespaced copy of the roman upper-case alphabet intended to be used only to spell out two-letter contiguous pairs that are [or at some point were] valid ISO country codes; where, when used in this way, the resulting 'colored'-letter-pair sequence has 'flag semantics', i.e. is meant to be rendered as a flag and machine-legible as a flag)
- or individually tag each and every one of those codepoints with its own tag/color codepoint (as in Unicode variation selectors)
- or interject binary-infix "operator" codepoints as glue between each codepoint in a codepoint sequence, so that those "operator" codepoints each affix together their immediate sibling codepoints, essentially constructing an abstract-Unicode-semantics list ADT "cons by cons", so that said list then may then be assigned its own semantics, e.g. being treated as a single grapheme-cluster with its own rendering (as in e.g. ZERO WIDTH JOINER used in its role in constructing complex emoji)
These encodings are all very high-overhead; but these are the kinds of trade-offs you have to make for self-synchronization to work.
And these trade-offs are sensible to make... if you're Ken Thompson in 1992, having to consider e.g. plaintexts being transmitted over raw RS232, or filesystems that just blast bytes to a spinning-rust disk without so much as a checksum, and then read them back "blindly" years later with that disk potentially highly-degraded.
But what if you live in the modern world, and you only care about holding and manipulating known-length strings in memory and/or embedded into code-signed (and thereby hashed) binaries; checksummed-block filesystems over rarely-corrupting NVMe; and transmission of data mostly over encrypted-stream protocols, where even the rare packet-level corruption that still TCP-checksums correctly, doesn't decrypt successfully, and therefore causes TLS-level retransmission?
Well, then you could define a much-more-concise reformulation of Unicode, that uses all the "forbidden" semantics-encoding techniques that Unicode itself avoids due to the self-synchronization constraint.
Where by "reformulation", I mean: a standard that keeps parity with Unicode in terms of what it can encode; and which at all times maintains a clearly-defined lossless bijective transformation between it and Unicode, evolving in lockstep with Unicode; but where Unicode and this formulation have their own distinct universes of codepoints, that compose using different rules, into the same ultimate sets of reachable grapheme-clusters with the same text-segmentation/collation/etc semantics.
I'm honestly kind of surprised that there isn't already a project somewhere to define an alternative Unicode formulation that looks like this.
It'd not only be potentially a highly-efficient representation for many in-memory string operations (that text shaping libraries would also love); it'd also likely perform impressively (compared to regular UTF-8) as a canonical representation for documents used to train+prompt LLMs. It'd give "more meaning per token", via all the repeated-per-codel overhead becoming once-per-sequence overhead; and it'd also allow many layers of meaning that are currently encoded via in-band protocols (ANSI escape codes, Markdown, HTML/XML, etc) that the LLM needs to learn additional recognition logic for, to instead be parsed out "during" initial text-stream recognition.
(And it's taking me real willpower not to go into depth on all the features such a formulation could have, and all the benefits it could provide. I should probably stop here before I nerd-snipe myself!)
The letter from Ken Thompson is spot on:
"your first 2 extensions (5 and 6 bytes) were clearly envisioned. the standard (up to 4 bytes) was created to cover the size of unicode. i thought any more description would be a waste of paper. i think your extension from 7 to 8 bytes is a little hoaky. i requires reading the whole string rather than "knowing" the number of follow on bytes. so, i think the only thing new is the 7 byte version.
i appreciate the mail, but i really dont think it is useful. it is like replacing ipv6 with ipv50."
Logo and name need more work but other than that, ship it.
I love it.
Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.
Phew, and I was worried that we'd be running out of UNICODE space for new emojis ;)
This is like TIME CUBE (https://en.wikipedia.org/wiki/Time_Cube) but for encoding nerds.
edit: replaced false prophet site with wiki link for the original one and true timecube.com...lost to the sands of ...time.
That’s a pretty harsh comparison.
It’s a fairly simple encoding which is UTF-8 compatible and allows codepoints to be arbitrarily large. It’s a real encoding proposal with a simple rigorous specification. I don’t have a use case for needing arbitrarily large codepoints, but for people who do, this makes more sense than hacks such as having surrogates in Unicode’s hi-bit private use area.
timecube.com is lost, zombo.com has been redesigned. If this is not proof of a dead internet, I don’t know what is.
Finally, an ASCII-safe way to encode all the vectors of Neuralese.
More like WTF-8.
https://xkcd.com/1953/
No, WTF-8 is its own encoding: https://wtf-8.codeberg.page/
I have entertained ideas like this every since becoming familiar with the original RFC2279 UTF-8 well over two decades ago, before UTF-8 was subsequently restricted to 21 bits.
Another idea that would work is to start an encoding with 0b10xxxxxx have subsequent 0b10xxxxxx bytes continue the encoding, and end it with 11xxxxxx. 0b1000_0000 (i.e. 128) is not allowed for the first byte of this sequence, and we add 127 to the resulting number (so each code point has only one representation) This way, the ♥ symbol (that’s 0x2665 or 9829 in decimal) would be represented as follows:
♥ → 0b0010_0110_0110_0101 → 0b0010_0101_1110_0110 (subtract 127) → 0b10_00_0010 0b10_0101_11 0b11_10_0110
This allows 18 bits to be encoded in three bytes, and, like UTF-8000, allows arbitrarily long sequences.
In the real world, the last time I needed to have a custom encoding, I went the other way and converted Unicode in to a 7-bit ASCII compatible encoding, where most ASCII control characters were converted into glyphs, as follows:
The reason for this encoding is that it allows me to write stuff in languages I am actually fluent in (English and Spanish), and adds a handful of useful non-ASCII punctuation (smart quotes, etc.).“.” represents a control character here; only “null” (0x00), “line feed” (0x0a), “form feed” (0x0c), “tab” (0x09), “carriage return” (0x0d), and “escape” (0x1b) are control characters; the rest are printable glyphs. I used this character set for my blog, since my blog is processed using HTMLDOC (which I learned the hard way is buggy with full Unicode) and my own Lua script.
With Lua, Unicode regexes are not readily supported because its regular expression engine assumes a codepoint is only one byte long when forming character classes such as %u for upper case letters. My Lua code has character classes for the non-ASCII glyphs such as %t for all letters in this custom encoding.
I have made this encoding 7 bits because that gives me the option to use the eight bit for future expansion.
The real reason for so few glyphs is because, in addition to having things work nicely with code which assumes 8-bit codepoints, with the modern web, I need to send to the viewer the font a page will be rendered with (it would had been nice if Microsoft had open sourced Verdana, Georgia, and the rest of the core fonts, and had those fonts became a part of the browser standards, so we could have cross-platform font stacks, but oh well) [1], so I aggressively subset the font to minimize the page load time.
[1] I still wish for the day when Apple, Google, Microsoft, and Mozilla come together, decide to include “Noto” and “Noto Serif” with their browsers, so “font-family: Noto (Serif)” always renders the exact same font without having to download a font over the network when loading a web page.
Actually, my proposed encoding only needs to add 64 to the code point (or subtract 64 when encoding a code point), not 128. That in mind, ♥ is encoded as follows:
♥ → 0b0010_0110_0110_0101 → 0b0010_0110_0010_0101 (subtract 64) → 0b10_00_0010 0b10_0110_00 0b11_10_0101
Code point 128 is encoded as follows:
128 → 0b1000_0000 → 0b0100_0000 (subtract 64) → 0b10_0000_01 0b11_00_0000
C99 can do up to over 10 bytes long (uint64_t) without a bignum library since 10 bytes gives us 60 bits. C23 gives us a bignum library so the sky’s the limit.
Again, I only need 7 bits to represent every character the font for my blog has.
It would be nice if the first paragraph gave a motivation for this. Why did you start this?
I mean, I can derive from the text that this can encode arbitrarily large integers. So what? What's the point? I also understand that this is for fun.
> This is a fun standalone project / proposal
Yes, sure. But what was the incentive? The goal? The reason to start the project (and not another one)? What was interesting enough to start this?
No project is ever safe from complicators.
This is why we need the KISS enforcers.
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.
Allowing more than one lead byte is a complication of the existing standard. As many others have pointed out, we have plenty of coding space without that (e.g. by allowing 5- and 6-byte UTF-8 again), so the case for the extra complexity is currently not compelling.
It starts like this, and ends up like x86.
Perhaps we can use a generalized Unicode-like encoding space for LLM text tokens. A text tokenization scheme uses up a few hundred thousand entries, with say one thousand new entries added annually. These can be called amojis, meaning AI mojis.