UTF-16

Print Print
Reading time 19:9
UTF-16
Unifont Full Map.png
The first 216 Unicode code points. The stripe of solid gray near the bottom are the surrogate halves used by UTF-16 (the white region below the stripe is the Private Use Area)
Language(s)International
StandardUnicode Standard
ClassificationUnicode Transformation Format, variable-width encoding
ExtendsUCS-2
Transforms / EncodesISO 10646 (Unicode)

UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid character code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as code points are encoded with one or two 16-bit code units. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding, now known as UCS-2 (for 2-byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed.[1]

UTF-16 is used internally by systems such as Microsoft Windows, the Java programming language and JavaScript/ECMAScript. It is also often used for plain text and for word-processing data files on Microsoft Windows. It is rarely used for files on Unix-like systems. As of May 2019, Microsoft reversed its course of only emphasizing UTF-16 for Unicode; for Windows applications recommends and supports UTF-8, e.g. for Universal Windows Platform (UWP) apps.[2]

UTF-16 is the only web-encoding incompatible with ASCII,[3] and never gained popularity on the web, where it is used by under 0.002% (little over 1 thousand of 1 percent) of web pages.[4]UTF-8, by comparison, is used by 97% of all web pages.[5] The Web Hypertext Application Technology Working Group (WHATWG) considers UTF-8 "the mandatory encoding for all [text]" and that for security reasons browser applications should not use UTF-16.[6]

History

In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" (UCS) that would replace earlier language-specific encodings with one coordinated system. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (216) values, which would require 2 bytes (16 bits) per character.

Two groups worked on this in parallel, ISO/IEC JTC 1/SC 2 and the Unicode Consortium, the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2-byte encoding was originally called "Unicode", but is now called "UCS-2".[7]

When it became increasingly clear that 216 characters would not suffice,[1]IEEE introduced a larger 31-bit space and an encoding (UCS-4) that would require 4 bytes per character. This was resisted by the Unicode Consortium, both because 4 bytes per character wasted a lot of memory and disk space, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise and introduced with version 2.0 of the Unicode standard in July 1996.[8] It is fully specified in RFC 2781, published in 2000 by the IETF.[9][10]

In the UTF-16 encoding, code points less than 216 are encoded with a single 16-bit code unit equal to the numerical value of the code point, as in the older UCS-2. The newer code points greater than or equal to 216 are encoded by a compound value using two 16-bit code units. These two 16-bit code units are chosen from the UTF-16 surrogate range 0xD800–0xDFFF which had not previously been assigned to characters. Values in this range are not used as characters, and UTF-16 provides no legal way to code them as individual code points. A UTF-16 stream, therefore, consists of single 16-bit code points outside the surrogate range for code points in the Basic Multilingual Plane (BMP), and pairs of 16-bit values within the surrogate range for code points above the BMP.

UTF-16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard. "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard."[11] There are no plans as of 2021 to extend UTF-16 to support a larger number of code points or the code points replaced by surrogates, as this would violate the Unicode Stability Policy with respect to general category or surrogate code points.[12] (any scheme that remains a self-synchronizing code would require allocating at least one BMP code point to start a sequence. Changing the purpose of a code point is disallowed).

Description

Each Unicode code point is encoded either as one or two 16-bit code units. How these 16-bit codes are stored as bytes then depends on the 'endianness' of the text file or communication protocol.

A "character" may need from as few as two bytes to fourteen[13] or even more bytes to be recorded. For instance an emoji flag character takes 8 bytes, since it is "constructed from a pair of Unicode scalar values"[14] (and those vales are outside the BMP and require 4 bytes each).

U+0000 to U+D7FF and U+E000 to U+FFFF

U+D800 to U+DFFF have a special purpose, see below.

Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. These code points in the Basic Multilingual Plane (BMP) are the only code points that can be represented in UCS-2.[citation needed] As of Unicode 9.0, some modern non-Latin Asian, Middle-Eastern, and African scripts fall outside this range, as do most emoji characters.

Code points from U+010000 to U+10FFFF

Code points from the other planes (called Supplementary Planes) are encoded as two 16-bit code units called a surrogate pair, by the following scheme:

UTF-16 decoder
Low
High
DC00 DC01    ...    DFFF
D800 010000 010001 ... 0103FF
D801 010400 010401 ... 0107FF
  ⋮ ⋮ ⋮ ⋱ ⋮
DBFF 10FC00 10FC01 ... 10FFFF
  • 0x10000 is subtracted from the code point (U), leaving a 20-bit number (U') in the hex number range 0x00000–0xFFFFF. Note for these purposes, U is defined to be no greater than 0x10FFFF.
  • The high ten bits (in the range 0x000–0x3FF) are added to 0xD800 to give the first 16-bit code unit or high surrogate (W1), which will be in the range 0xD800–0xDBFF.
  • The low ten bits (also in the range 0x000–0x3FF) are added to 0xDC00 to give the second 16-bit code unit or low surrogate (W2), which will be in the range 0xDC00–0xDFFF.

Illustrated visually, the distribution of U' between W1 and W2 looks like:[15]

U' = yyyyyyyyyyxxxxxxxxxx  // U - 0x10000
W1 = 110110yyyyyyyyyy      // 0xD800 + yyyyyyyyyy
W2 = 110111xxxxxxxxxx      // 0xDC00 + xxxxxxxxxx

The high surrogate and low surrogate are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8.[16]

Since the ranges for the high surrogates (0xD800–0xDBFF), low surrogates (0xDC00–0xDFFF), and valid BMP characters (0x0000–0xD7FF, 0xE000–0xFFFF) are disjoint, it is not possible for a surrogate to match a BMP character, or for two adjacent code units to look like a legal surrogate pair. This simplifies searches a great deal. It also means that UTF-16 is self-synchronizing on 16-bit words: whether a code unit starts a character can be determined without examining earlier code units (i.e. the type of code unit can be determined by the ranges of values in which it falls). UTF-8 shares these advantages, but many earlier multi-byte encoding schemes (such as Shift JIS and other Asian multi-byte encodings) did not allow unambiguous searching and could only be synchronized by re-parsing from the start of the string (UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte).

Because the most commonly used characters are all in the BMP, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. CVE-2008-2938, CVE-2012-2135).

The Supplementary Planes contain emoji, historic scripts, less used symbols, less used Chinese ideographs, etc. Since the encoding of Supplementary Planes contains 20 significant bits (10 of 16 bits in each of the high and low surrogates), 220 code points can be encoded, divided into 16 planes of 216 code points each. Including the separately-handled Basic Multilingual Plane, there are a total of 17 planes.

U+D800 to U+DFFF

The Unicode standard permanently reserves these code point values for UTF-16 encoding of the high and low surrogates, and they will never be assigned a character, so there should be no reason to encode them. The official Unicode standard says that no UTF forms, including UTF-16, can encode these code points.[citation needed]

However, UCS-2, UTF-8, and UTF-32 can encode these code points in trivial and obvious ways, and a large amount of software does so,[citation needed] even though the standard states that such arrangements should be treated as encoding errors.[citation needed]

It is possible to unambiguously encode an unpaired surrogate (a high surrogate code point not followed by a low one, or a low one not preceded by a high one) in the format of UTF-16 by using a code unit equal to the code point. The result is not valid UTF-16, but the majority of UTF-16 encoder and decoder implementations do this then when translating between encodings.[citation needed] Windows allows unpaired surrogates in filenames and other places,[citation needed] which generally means they have to be supported by software in spite of their exclusion from the Unicode standard.

Examples

To encode U+10437 (𐐷) to UTF-16:

  • Subtract 0x10000 from the code point, leaving 0x0437.
  • For the high surrogate, shift right by 10 (divide by 0x400), then add 0xD800, resulting in 0x0001 + 0xD800 = 0xD801.
  • For the low surrogate, take the low 10 bits (remainder of dividing by 0x400), then add 0xDC00, resulting in 0x0037 + 0xDC00 = 0xDC37.

To decode U+10437 (𐐷) from UTF-16:

  • Take the high surrogate (0xD801) and subtract 0xD800, then multiply by 0x400, resulting in 0x0001 × 0x400 = 0x0400.
  • Take the low surrogate (0xDC37) and subtract 0xDC00, resulting in 0x37.
  • Add these two results together (0x0437), and finally add 0x10000 to get the final decoded UTF-32 code point, 0x10437.

The following table summarizes this conversion, as well as others. The colors indicate how bits from the code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black.

Character Binary code point Binary UTF-16 UTF-16 hex
code units 		UTF-16BE
hex bytes 		UTF-16LE
hex bytes
$ U+0024 0000 0000 0010 0100 0000 0000 0010 0100 0024 00 24 24 00
€ U+20AC 0010 0000 1010 1100 0010 0000 1010 1100 20AC 20 AC AC 20
𐐷 U+10437 0001 0000 0100 0011 0111 1101 1000 0000 0001 1101 1100 0011 0111 D801 DC37 D8 01 DC 37 01 D8 37 DC
𤭢 U+24B62 0010 0100 1011 0110 0010 1101 1000 0101 0010 1101 1111 0110 0010 D852 DF62 D8 52 DF 62 52 D8 62 DF

Byte-order encoding schemes

UTF-16 and UCS-2 produce a sequence of 16-bit code units. Since most communication and storage protocols are defined for bytes, and each unit thus takes two 8-bit bytes, the order of the bytes may depend on the endianness (byte order) of the computer architecture.

To assist in recognizing the byte order of code units, UTF-16 allows a Byte Order Mark (BOM), a code point with the value U+FEFF, to precede the first actual coded value.[nb 1] (U+FEFF is the invisible zero-width non-breaking space/ZWNBSP character.)[nb 2] If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the non-character value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values.

If the BOM is missing, RFC 2781 recommends[nb 3] that big-endian encoding be assumed. In practice, due to Windows using little-endian order by default, many applications assume little-endian encoding. It is also reliable to detect endianness by looking for null bytes, on the assumption that characters less than U+0100 are very common. If more even bytes (starting at 0) are null, then it is big-endian.

The standard also allows the byte order to be stated explicitly by specifying UTF-16BE or UTF-16LE as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically not supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Most applications ignore a BOM in all cases despite this rule.

For Internet protocols, IANA has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as the names for these encodings (the names are case insensitive). The aliases UTF_16 or UTF16 may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols.

Similar designations, UCS-2BE and UCS-2LE, are used to show versions of UCS-2.

Usage

UTF-16 is used for text in the OS API of all currently supported versions of Microsoft Windows (and including at least all since Windows CE/2000/XP/2003/Vista/7[17]) including Windows 10. In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages.[18][19] Older Windows NT systems (prior to Windows 2000) only support UCS-2.[20] Files and network data tend to be a mix of UTF-16, UTF-8, and legacy byte encodings.

While there's been some UTF-8 support for even Windows XP,[21] it was improved (in particular the ability to name a file using UTF-8) in Windows 10 insider build 17035 and the April 2018 update, and as of May 2019 Microsoft recommends software use it instead of UTF-16.[2]

The IBM i operating system designates CCSID (code page) 13488 for UCS-2 encoding and CCSID 1200 for UTF-16 encoding, though the system treats them both as UTF-16.[22]

UTF-16 is used by the Qualcomm BREW operating systems; the .NET environments; and the Qt cross-platform graphical widget toolkit.

Symbian OS used in Nokia S60 handsets and Sony Ericsson UIQ handsets uses UCS-2. iPhone handsets use UTF-16 for Short Message Service instead of UCS-2 described in the 3GPP TS 23.038 (GSM) and IS-637 (CDMA) standards.[23]

The Joliet file system, used in CD-ROM media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name).

The Python language environment officially only uses UCS-2 internally since version 2.0, but the UTF-8 decoder to "Unicode" produces correct UTF-16. Since Python 2.2, "wide" builds of Unicode are supported which use UTF-32 instead;[24] these are primarily used on Linux. Python 3.3 no longer ever uses UTF-16, instead an encoding that gives the most compact representation for the given string is chosen from ASCII/Latin-1, UCS-2, and UTF-32.[25]

Java originally used UCS-2, and added UTF-16 supplementary character support in J2SE 5.0.

JavaScript may use UCS-2 or UTF-16.[26] As of ES2015, string methods and regular expression flags have been added to the language that permit handling strings from an encoding-agnostic perspective.

In many languages, quoted strings need a new syntax for quoting non-BMP characters, as the C-style "\uXXXX" syntax explicitly limits itself to 4 hex digits. The following examples illustrate the syntax for the non-BMP character "𝄞" (U+1D11E, MUSICAL SYMBOL G CLEF). The most common (used by C++, C#, D, and several other languages) is to use an upper-case 'U' with 8 hex digits such as "\U0001D11E".[27] In Java 7 regular expressions, ICU, and Perl, the syntax "\x{1D11E}" must be used; similarly, in ECMAScript 2015 (JavaScript), the escape format is "\u{1D11E}". In many other cases (such as Java outside of regular expressions),[28] the only way to get non-BMP characters is to enter the surrogate halves individually, for example: "\uD834\uDD1E" for U+1D11E.

String implementations based on UTF-16 typically define lengths of the string and allow indexing in terms of these 16-bit code units, not in terms of code points. Neither code points nor code units correspond to anything an end user might recognize as a “character”; the things users identify as characters may in general consist of a base code point and a sequence of combining characters (or might be a sequence of code points of some other kind, for example Hangul conjoining jamos) – Unicode refers to this construct as a grapheme cluster[29] – and as such, applications dealing with Unicode strings, whatever the encoding, must cope with the fact that this limits their ability to arbitrarily split and combine strings.

UCS-2 is also supported by the PHP language[30] and MySQL.[7]

Swift, version 5, Apple's preferred application language, switched from UTF-16 to UTF-8 as the preferred encoding.[31]

See also

  • Comparison of Unicode encodings
  • Plane (Unicode)
  • UTF-8
  • UTF-32

Notes

  1. ^ UTF-8 encoding produces byte values strictly less than 0xFE, so either byte in the BOM sequence also identifies the encoding as UTF-16 (assuming that UTF-32 is not expected).
  2. ^ Use of U+FEFF as the character ZWNBSP instead of as a BOM has been deprecated in favor of U+2060 (WORD JOINER); see Byte Order Mark (BOM) FAQ at unicode.org. But if an application interprets an initial BOM as a character, the ZWNBSP character is invisible, so the impact is minimal.
  3. ^ RFC 2781 section 4.3 says that if there is no BOM, "the text SHOULD be interpreted as being big-endian." According to section 1.2, the meaning of the term "SHOULD" is governed by RFC 2119. In that document, section 3 says "... there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course".

References

  1. ^ a b "What is UTF-16?". The Unicode Consortium. Unicode, Inc. Retrieved 29 March 2018.
  2. ^ a b "Use the Windows UTF-8 code page - UWP applications". docs.microsoft.com. Retrieved 2020-06-06. As of Windows Version 1903 (May 2019 Update), you can use the ActiveCodePage property in the appxmanifest for packaged apps, or the fusion manifest for unpackaged apps, to force a process to use UTF-8 as the process code page. [..] CP_ACP equates to CP_UTF8 only if running on Windows Version 1903 (May 2019 Update) or above and the ActiveCodePage property described above is set to UTF-8. Otherwise, it honors the legacy system code page. We recommend using CP_UTF8 explicitly.
  3. ^ "HTML Living Standard". w3.org. 2020-06-10. Retrieved 2020-06-15. UTF-16 encodings are the only encodings that this specification needs to treat as not being ASCII-compatible encodings.
  4. ^ "Usage Statistics of UTF-16 for Websites, June 2021". w3techs.com. Retrieved 2021-06-17.
  5. ^ "Usage Statistics of UTF-8 for Websites, February 2021". w3techs.com. Retrieved 2021-02-25.
  6. ^ "Encoding Standard". encoding.spec.whatwg.org. Retrieved 2018-10-22. The UTF-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set. Therefore for new protocols and formats, as well as existing formats deployed in new contexts, this specification requires (and defines) the UTF-8 encoding. [..] The problems outlined here go away when exclusively using UTF-8, which is one of the many reasons that UTF-8 is now the mandatory encoding for all text things on the Web.
  7. ^ a b "MySQL :: MySQL 5.7 Reference Manual :: 10.1.9.4 The ucs2 Character Set (UCS-2 Unicode Encoding)". dev.mysql.com.
  8. ^ "Questions about encoding forms". Retrieved 2010-11-12.
  9. ^ ISO/IEC 10646:2014 "Information technology – Universal Coded Character Set (UCS)" sections 9 and 10.
  10. ^ The Unicode Standard version 7.0 (2014) section 2.5.
  11. ^ "The Unicode® Standard Version 10.0 – Core Specification. Appendix C Relationship to ISO/IEC 10646" (PDF). Unicode Consortium. section C.2 page 913 (pdf page 10)
  12. ^ "Unicode Character Encoding Stability Policies". unicode.org.
  13. ^ "It's not wrong that "🤦🏼‍♂️".length == 7". hsivonen.fi. Retrieved 2021-03-15.
  14. ^ "Apple Developer Documentation". developer.apple.com. Retrieved 2021-03-15.
  15. ^ Yergeau, Francois; Hoffman, Paul. "UTF-16, an encoding of ISO 10646". tools.ietf.org. Retrieved 2019-06-18.
  16. ^ Allen, Julie D.; Anderson, Deborah; Becker, Joe; Cook, Richard, eds. (2014). "3.8 Surrogates" (PDF). The Unicode Standard, Version 7.0—Core Specification. Mountain View: The Unicode Consortium. p. 118. Retrieved 3 November 2014.
  17. ^ Unicode (Windows). Retrieved 2011-03-08 "These functions use UTF-16 (wide character) encoding (…) used for native Unicode encoding on Windows operating systems."
  18. ^ "Unicode". microsoft.com. Retrieved 2009-07-20.
  19. ^ "Surrogates and Supplementary Characters". microsoft.com. Retrieved 2009-07-20.
  20. ^ "Description of storing UTF-8 data in SQL Server". microsoft.com. 7 December 2005. Retrieved 2008-02-01.
  21. ^ "[Updated] Patch for cmd.exe for windows xp for cp 65001 - Page 2 - DosTips.com". www.dostips.com. Retrieved 2021-06-17.
  22. ^ "UCS-2 and its relationship to Unicode (UTF-16)". IBM. Retrieved 2019-04-26.
  23. ^ Selph, Chad (2012-11-08). "Adventures in Unicode SMS". Twilio. Archived from the original on 2012-11-09. Retrieved 2015-08-28.
  24. ^ "PEP 261 – Support for "wide" Unicode characters". Python.org. Retrieved 2015-05-29.
  25. ^ "PEP 0393 – Flexible String Representation". Python.org. Retrieved 2015-05-29.
  26. ^ "JavaScript's internal character encoding: UCS-2 or UTF-16? · Mathias Bynens".
  27. ^ "ECMA-334: 9.4.1 Unicode escape sequences". en.csharp-online.net. Archived from the original on 2013-05-01.
  28. ^ Lexical Structure: Unicode Escapes in "The Java Language Specification, Third Edition". Sun Microsystems, Inc. 2005. Retrieved 2019-10-11.
  29. ^ "Glossary of Unicode Terms". Retrieved 2016-06-21.
  30. ^ "PHP: Supported Character Encodings - Manual". php.net.
  31. ^ "UTF-8 String". Swift.org. 2019-03-20. Retrieved 2020-08-20.

By: Wikipedia.org
Edited: 2021-06-18 14:08:53
Source: Wikipedia.org

© 2024 CodeDocs.org
 Privacy Policy • Contact Us • Terms of Use