“For performance reasons Unicode strings are stored in UTF-16 internally, but you will likely stick with UTF-8 and let Python take care of the translation.”

http://nathany.com/developer/vanpyz-from-future/

I’m no guru on codepoints and encodings and all, but that sounds like pretty standard fair… doesn’t Windows use UTF-16 for the same reason?

As other commenters have more or less pointed out, there are two encoding steps in modern character representation systems: the mapping from linguistic symbols to integers (Unicode “code-points”), and the mapping from integers to bitstrings. Let’s call these “logical encoding” and “physical encoding” respectively.

In our increasingly Unicode-standardized world, logical encoding is rarely a question in most new software projects, while physical encoding frequently is. So I don’t blame your commenters for being confused, especially when your two examples included one related only to logical encoding (Han unification) and one related only to physical encoding (space efficiency of Asian scripts).

The difference between Ruby and Python seems to be that, while both support various physical encodings for I/O, Python 3′s string objects are always represented in memory as sequences of Unicode code-points, while Ruby 1.9 can use any supported physical encoding as the in-memory representation. Theoretically this would allow Ruby to support encoding schemes that are not Unicode-compatible, as suggested above. I’m not sure any have been implemented in practice, though. In my Ruby 1.9 installation, all 81 installed encodings are compatible (using Encoding.compatible?).

Regardless of the file encoding, “\uXXXX” in a Ruby 1.9 string literal refers to a Unicode code-point. So the Unicode logical encoding scheme does still have a special place in Ruby’s heart.

My favorite character is a ẽ. That is a small e with a tilde over it. Notice that it takes 3 characters in a row to render it.

0xE1 0xBA 0xBD

I really don’t care much for Unicode, but it isn’t going away so I guess I will just have to embrace it.

Unicode: “Encoded Character. An association (or mapping) between an abstract character and a code point.”

XML: “The mechanism for encoding character code points into bit patterns may vary from entity to entity. All XML processors MUST accept the UTF-8 and UTF-16 encodings of Unicode”

I’m not personally aware of how the language drifted apart between the Unicode standard and the others. Seems odd.

To determine whether Ruby _really_ supports non-Unicode “encodings” (according to the Unicode definition) better than Python does, I’d need to know what Ruby’s definition of “character” is. i.e. how are characters named, how is equivalence defined, and what properties are available on character objects?

In Python, Java, Javascript, C#, etc. two character objects are equivalent if they have the same code point. Given the code point, you can infer the character’s name, classification and other properties based on the Unicode database

http://unicode.org/ucd/

Can somebody please explain Ruby’s definition of “character”?

“Python is only capable of representing Unicode characters internally, which means that round-trips from encodings that are not injective onto Unicode will be lossy.”

A) Name one such encoding.

Hell, I’ll spot you one: Emoji.

Ah, but what’s this?

Never mind then. For all practical purposes, every major coding in the world today is either an official part of Unicode or using a section of the private use characters in Unicode by convention with proposals being bandied about for eventual inclusion in Unicode. The only exceptions are things like Mojikyo which have no real support anywhere anyway. (So far as I can tell, Ruby doesn’t support it any meaningful sense either.) And if you really must work with Mojikyo, well there’s always bytestrings.

B) The problem for the Japanese is not roundtripping (“The list of Japanese characters in Unicode was ripped straight from JIS, so JIS can be converted into Unicode without many problems.”). The problem is that they didn’t want Chinese characters to map onto Japanese characters. So, there’s no problem with round-tripping. The problem is going from ShiftJIS to Unicode to some Chinese encoding. Or going from ShiftJIS to some Chinese encoding for that matter. They didn’t want it to be possible, but it is. So, we live with it.

“The Unicode character encoding treats alphabetic characters, ideographic characters, and symbols equivalently, which means they can be used in any mixture and with equal facility.”

also:

“The Unicode Standard specifies a numeric value (code point) and a name for each of its characters. In this respect, it is similar to other character encoding standards from ASCII onward.”

According to you, the above statements are nonsense, because the they concern the character set, not the byte-encoding. And yet this is straight out of page 1 of the standard. You’re going to have to admit that there’s some latitude in terminology here, unless this list of people who “don’t know the difference between an encoding and a character set” includes the Unicode Standard 5.0, page 1.

I’m quite aware of the difference between a character set and a character encoding, when this specific distinction is drawn. Python is only capable of representing Unicode characters internally, which means that round-trips from encodings that are not injective onto Unicode will be lossy.

Your last entry showed that you either don’t know what’s different about Python 3.0 (which is forgivable for a Ruby person) or that you don’t know what the difference between an encoding and a character set is (which is unforgivable for anyone doing web programming). Based on this “defense,” I’m leaning toward the former.