Computers & Writing Systems
Implementing Writing Systems: An introduction
Understanding Unicode™ - I
This article continues at: Understanding Unicode™: A general introduction to the Unicode Standard (Sections 6-15).
Unicode is a hot topic these days among computer users that work with multilingual text. They know it is important, and they hear it will solve problems, especially for dealing with text involving multiple scripts. They may not know where to go to learn about it, though. Or they may have read a few things about it and perhaps have seen some code charts, but they are at a point at which they need to gain a firmer understanding so that they can start to develop implementations or create content. This introduction is intended to give such people the basic grounding that they need.
As a very succinct introduction to the subject, Unicode is an industry standard character set encoding developed and maintained by The Unicode® Consortium. The Unicode character set is able to support over one million characters, and is being developed with an aim to have a single character set that supports all characters from all scripts, as well as many symbols, that are in common use around the world today or in the past. Currently, the Standard supports over 94,000 characters representing a large number of scripts. Unicode also defines three encoding forms, each of which is able to represent the entire character set. The three encoding forms are based on 8-, 16- and 32-bit code units, providing a flexibility that makes Unicode suitable for implementation in a wide variety of environments. The benefits of a single, universal character set and the practical considerations for implementation that have gone into the design of Unicode have made it a success, and it is well on the way to becoming a dominant and ubiquitous standard.
This section will give an introduction to the Unicode Standard and the most important concepts that implementers and users need to be familiar with. No previous in-depth knowledge of Unicode is assumed, though I expect that the reader will probably have at least seen some code charts at some point. I will be assuming familiarity with basic concepts of character set encoding, though, as described in “Character set encoding basics”. In particular, I will assume familiarity with the distinction between a coded character set and a character encoding form, and with the importance of character semantics for software processes, all of which I describe in that section.
I will go into some detail on several topics, and some sections are rather technical, which should be expected given the subject matter. I have put some of the more detailed and technical discussion into appendices. They are not essential reading for the beginner, but may be useful reading as you progress in your understanding and your use of the Standard.
This introduction is not intended as a substitute for the Standard itself. The only way to get authoritative information about Unicode is to read the source documents that comprise it. The Standard consists of a rather large book, however, together with a number of other electronic documents and data files. The volume of information can be somewhat daunting for a beginner. Moreover, the Standard was not written for instructional purposes. It is technical documentation. Neither the language nor the organisation was created with didactic purposes in mind. To learn about the Standard, you want something that will guide you through the important topics and point you to other sources that can take you into more depth at appropriate points. Hopefully this introduction achieves that.
On the other hand, there are topics that this introduction does not cover in any depth, and it does not provide an authoritative statement of the details of the Standard. For example, the Unicode bi-directional algorithm is introduced in Section 8, but it is not explained here in any detail. Similarly, the conformance requirements of the Standard are described in Section 12, but this document does not provide the precise wording of each conformance requirement and of the definitions for the terms and constructs that they refer to. In both cases, that level of detail is inappropriate for an introduction.
The terms canonical equivalence, canonical decomposition, compatibility equivalence and compatibility decomposition recur frequently in many parts of this section. These are important topics that affect many aspects of the Unicode Standard. Different aspects of these issues are covered in various sections. The nature and source of canonical and compatibility equivalence is discussed in Section 6 and in “Mapping codepoints to Unicode encoding forms”, and also in Section 8. The manner and location in which decomposition mappings are specified is covered in Section 7.5. The way in which these mappings are applied in relation to Unicode normalization forms is discussed in Section 10. Finally, some of the implications are considered in Section 11. Initially, it may seem unfortunate to have the coverage of these issues spread across several sections. In fact, these issues are so involved that they take up the bulk of these sections.
Before continuing, let me briefly explain some of the notational devices that are used:
1 A brief history of Unicode
In order to understand Unicode, it is helpful to know a little about the history of its development. Early on in the history of computing, character set encoding standards based on 8-bit technologies became the norm. As different vendors developed their systems and began to adapt them for use in markets that used several different languages and scripts, a large number of different encoding standards resulted.
This situation led to considerable difficulty for developers and for users working with multilingual data. Products were often tied to a single encoding, which did not allow users to work with multilingual data or with data coming from incompatible systems. Developers were also required to support multiple versions of their products to serve different markets, making development and deployment for multiple markets a difficult process. In order to support data created using others’ products, developers had to support a variety of different standards for a single language. In order to work with multilingual data, they needed to support several standards simultaneously since no one standard supported more than a handful of languages. In turn, it was impossible to support multilingual data in plain text. Developing software that had anything to do with multilingual text had become incredibly difficult.
By the early 1980s, the software industry was starting to recognise the need for a solution to the problems involved with using multiple character encoding standards. Some particularly innovative work was begun at Xerox. The Xerox Star workstation used a multi-byte encoding that allowed it to support a single character set with potentially millions of characters. Using this system, they implemented a word-processing system that had support for several scripts, including Roman, Cyrillic, Greek, Arabic, Hebrew, Chinese, Korean and the Japanese kana syllabaries.1 The work at Xerox was a direct inspiration for Unicode.
The Unicode project began in 1988, with representatives from several companies collaborating to develop a single character set encoding standard that could support all of the world’s scripts. This led to the formation of the Unicode Consortium in January of 1991, and the publication of Version 1.0 of the Unicode Standard in October of the same year (The Unicode Consortium 1991).
There were four key original design goals for Unicode:
How well these goals have been achieved as the Standard has developed is probably a matter of opinion. There is no question, however, that some compromises were necessary along the way. To fully understand Unicode and the compromises that were made, it is also important to understand another, related standard: ISO/IEC 10646.2
In 1984, a joint ISO/IEC working group was formed to begin work on an international character set standard that would support all of the world’s writing systems. This became known as the Universal Character Set (UCS). By 1989, drafts of the new standard were starting to get circulated.
At this point, people became aware that there were two efforts underway to achieve similar ends, those ends being a comprehensive standard that everyone could use. Of course, the last thing anybody wanted was to have two such standards. As a result, in 1991 the ISO/IEC working group and the Unicode Consortium began to discuss a merger of the two standards. There were some obstacles to overcome in negotiating a merger, however. Firstly, at this point the ISO/IEC standard was at an advanced draft stage and the Unicode 1.0 had already been published, and there were incompatibilities in the two sets. Secondly, Unicode was being designed using a uniform, 16-bit encoding form, which allowed for up to 65,536 characters, but the ISO/IEC standard used a 31-bit codespace that allowed for over 2 billion characters.
The complete details of the merger were worked out over many years, but the most important issues were resolved early on. The first step was that the repertoires in Unicode and the draft 10646 standard were aligned, and an agreement was reached that the two character sets should remain aligned. This required some changes in Unicode, including several additions, a few deletions, and the reassignment of a significant number of characters to different codepoints.3 These changes were reflected in TUS 1.1.4
As of Unicode 1.1, the potential repertoire supportable by Unicode was seen to be a proper subset of that for ISO/IEC 10646. In other words, any of the 65,536 characters that might eventually be defined in Unicode would also be in ISO/IEC 10646, but it was possible that characters could be defined in ISO/IEC 10646 in the vast portion of the codespace above 64K that was unavailable to Unicode.
At some point along the way, people involved in Unicode began to recognise that 65,536 codepoints was not going to be enough to cover all of the Chinese ideographs. A solution was needed, but it would require giving up the original goal of having a uniform, 16-bit encoding form. Obviously, the solution to this problem would have an impact on the discrepancy between the two standards in terms of the potential size of their character repertoires.
The solution came in the form of a revised 16-bit encoding that both standards adopted, known as UTF-16. This encoding form would allow for support of over a million characters. ISO/IEC 10646 still had a potential capacity for far more, but it was conceded that a million characters were more than enough. Eventually, the ISO/IEC standard was formally limited to that number of assignable characters by permanently reserving the rest of the codespace.5
There was also an implementation issue that arose that would affect how well either standard was accepted within industry: many 8-bit processes were in place that would not be able to handle 16-bit data properly, and it was not going to be practical to change them all. Thus, there was a practical need for a second encoding form based on 8-bit code units. An 8-bit encoding form was developed, known as UTF-8, which was able to support the full potential range of characters, and which was adopted by both standards.6
The replacement of the original single, uniform 16-bit encoding in Unicode with UTF-16 and UTF-8 was formalised in TUS 2.0 of Unicode (The Unicode Consortium 1996). The language of that version still treated the 16-bit representation as primary, but a fundamental change had occurred. The relationship between the Unicode character set and the alternate encoding forms was clarified with the approval in 1999 of Unicode Technical Report #17 (see Whistler and Davis 2000 for the current version).
Since it was first published, ISO/IEC 10646 has supported a 32-bit encoding form, known as UCS-4. Eventually, a 32-bit encoding form was introduced for Unicode, known as UTF-32. This was formally adopted in the Standard in TUS 3.1 (Davis et al 2001).7
Thus, the key results of the merger have been that Unicode is now kept synchronised with ISO/IEC 10646, and it now supports three encoding forms based on 8-, 16- and 32-bit code units.
The Unicode Standard has continued to be developed up to the present, and work is still continuing with an aim to make the Standard more complete, covering more of the world’s writing systems, to correct errors in details, and to make it better meet the needs of implementers. Unicode Version 3.0 was published in 2000 (The Unicode Consortium 2000), introducing over 10,000 new characters. The most current version at this time, Version 3.1, was published this year (Davis et al 2001). This version added another 44,946 new characters, bringing the total number of characters to 94,140 encoded characters.
The most important points to be learned from this history lesson relate to the relationship with ISO/IEC 10646, the size of the Unicode codespace, and the fact that there are three encoding forms for Unicode. I will describe the codespace and the encoding forms in greater detail in Sections 3 and 4. First, though, I will take a brief look at who the Unicode Consortium is and how the Standard gets developed and maintained.
2 The Unicode Consortium and the maintenance of the Unicode Standard
In this section, I will describe the Unicode Consortium and the way in which the Unicode Standard is maintained. This will include looking at the versioning system used for the Standard, and also at Unicode Technical Reports, a set of satellite documents that have an integral relationship to the Standard.
2.1 The Unicode Consortium
The Unicode Consortium is a not-for-profit organisation that exists to develop and promote the Unicode Standard. Anyone can be a member of the consortium, though there are different types of memberships, and not everyone gets the privilege of voting on decisions regarding the Standard. That privilege is given only to those in the category of Full Member. There are two requirements for Full Membership: this category is available only for organisational members, not to individuals; and there is an annual fee of US$12,000. At the time of writing, there are currently 21 Full Members.
General information about the Unicode Consortium is available at http://www.unicode.org/unicode/consortium/consort.html. The current list of members is available at http://www.unicode.org/unicode/consortium/memblogo.html. (A text-only list is available at http://www.unicode.org/unicode/consortium/memblist.html.)
The work of developing the Standard is done by the Unicode Technical Committee (UTC). Every Full Member organization is eligible to have a voting position on the UTC, though they are not required to participate.
There are three other categories of membership: Individual Member, Specialist Member, and Associate Member. Each of these has increasing levels of privileges. The Associate and Specialist Member categories offer the privilege of being able to participate in the regular work of the UTC through an email discussion list—the “unicore” list. All members are eligible to attending meetings.
The UTC maintains a close working relationship with the corresponding body within ISO/IEC that develops and maintains ISO/IEC 10646. Any time one body considers adding new characters to the common character set, those proposals need to be evaluated by both bodies. Before any new character assignments can officially be made, approval of both bodies is required. This is how the two standards are kept in synchronization.
2.2 Versions of the Unicode Standard
A three-level versioning system is used for the Unicode Standard. Major versions (e.g. from 2.1.9 to 3.0) are used for significant additions to the Standard, and are published as a book. Minor versions (e.g. from 3.0.1 to 3.1) are used for the addition of new characters or for significant normative changes that may affect implementations, and are published as Technical Reports on the Unicode Web site (see below). The key distinction between major and minor versions is one of degree: a major version will include a number of significant additions to the Standard—enough to warrant the production of a new book. At the third level, an update (e.g. from 3.0 to 3.0.1) is used for any other important changes that can affect implementations. (Note that updates never include addition of new characters.) These are reflected primarily in the form of revised data files; there has not always been prose documentation to accompany an update of the Standard. Minor corrections can be made at any time and are published as errata on the Web site (see http://www.unicode.org/unicode/uni2errata/UnicodeErrata.html).
The Unicode Standard is embodied in the form of three types of information:
Thus, the current version of Unicode, TUS 3.1, consists of the published book for TUS 3.0, plus the UAX that describes the minor version for TUS 3.1, UAX #27, together with the current versions of the other annexes and data files.
For more information on the various versions of the Unicode Standard, see http://www.unicode.org/unicode/standard/versions/. A complete description of all of the items that constitute part of any version of Unicode is available at http://www.unicode.org/unicode/standard/versions/enumeratedversions.html.
2.3 Unicode Technical Reports, Unicode Standard Annexes, and Unicode Technical Standards
Unicode Technical Reports are satellite documents that complement the Standard in a variety of ways. As mentioned above, some embody portions of the Standard itself. Others discuss various implementation issues that relate to Unicode.
To date, there have been a total of 23 UTRs that have been published. Seven of these, all of them written prior to TUS 3.0, have been superseded. This has happened for one of two reasons:
Thus, there are 16 UTRs that are still current.
Of the UTRs that are current at this time, seven are presently designated as Unicode Standard Annexes and represent normative parts of TUS 3.1. These include the following:
Some UTRs are designated to be Unicode Technical Standards (UTSs). These are considered subsidiary standards that complement, but are independent from it. It is not necessary for software to conform to these other standards in order to be considered conformant to the Unicode Standard. Currently, there are two UTSs.
Other UTRs discuss various implementation issues or provide supplementary information regarding certain aspects of the Standard:
A complete list of current UTRs can be found online at http://www.unicode.org/unicode/reports/index.html. This also lists documents that are being considered as possible UTRs (“proposed draft UTRs”), as well as UTRs that are in the process of being drafted. Presently, there are no proposed draft or draft UTRs.
3 Codepoints, the Unicode codespace and script blocks
We have covered some important background on Unicode. Now we need to begin exploring the technical aspects of the Standard in more depth. In this section, I will describe the Unicode coded character set in greater detail. I will give particular focus to how the codespace and character set are organised, and will also introduce the code charts and other resources that provide information about the characters themselves.
3.1 Codepoints and the Unicode codespace
The Unicode coded character set is coded in terms of integer values, which are referred to in Unicode as Unicode scalar values (USVs). By convention, Unicode codepoints are represented in hexadecimal notation with a minimum of four digits and preceded with “U+”; so, for example, “U+0345”, “U+10345” and “U+20345”. Also by convention, any leading zeroes above four digits are suppressed; thus we would write “U+0456 CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I” but not “U+03456 !!unknown USV!!”.
Every character in Unicode can be uniquely identified by its codepoint, or also by its name. Unicode character names use only ASCII characters and by convention are written entirely in upper case. Characters are often referred to using both the codepoint and the name; e.g. U+0061 LATIN SMALL LETTER A. In discussions where the actual characters are unimportant or are assumed to be recognisable using only the codepoints, people will often save space and use only the codepoints. Also, in informal contexts where it is clear that Unicode codepoints are involved, people will often suppress the string “U+”. I will continue to write “U+” in this document for clarity, however.
The Unicode codespace ranges from U+0000 to U+10FFFF. Borrowing terminology from ISO/IEC 10646, the codespace is described in terms of 17 planes of 64K codepoints each. Thus, Plane 0 includes codepoints U+0000..U+FFFF, Plane 1 includes codepoints U+10000..U+1FFFF, etc.In the original design of Unicode, all characters were to have codepoints in the range U+0000..U+FFFF. In keeping with this, Plane 0 is set apart as the portion of the codespace in which all of the most commonly used characters are encoded, and is designated the Basic Multilingual Plane (BMP). The remainder of the codespace, Planes 1 to 1610, are referred to collectively as the Supplementary Planes.9 Up to and including TUS 3.0.1, characters were assigned only in the BMP
. In TUS 3.1, characters were assigned in the Supplementary Planes for the first time, in Planes 1, 2 and 14.
There are gaps in the Unicode codespace: codepoints that are permanently unassigned and reserved as non-characters. These include the last two codepoints in each plane, U+nFFFE and U+nFFFF (where n ranges from 0 to 1016). These have always been reserved, and characters will never be assigned at these codepoints. One implication of this is that these codepoints are available to software developers to use for proprietary purposes in internal processing. Note, however, that care must be taken not to transmit these codepoints externally.
Unassigned codepoints can be reserved in a similar manner at any time if there is a reason for doing so. This was recently done in TUS 3.1, reserving 32 codepoints from U+FDD0..U+FDEF as non-characters. This was done specifically in order to make additional codes available to programmers to use for internal processing purposes. Again, these should never appear in data.
There is another special range of 2,048 codepoints that are reserved, creating an effective gap in the codespace. These occupy the range U+D800..U+DFFF and are reserved due to the mechanism used in the UTF-16 encoding form (described in Section 4.1). In UTF-16, codepoints in the BMP are represented as code units having the same integer value. The code units in the range 0xD800–0xDFFF, serve a special purpose, however. These code units, known as surrogate code units (or simply surrogates), are used in representing codepoints from Planes 1 to 16. As a result, it is not possible to represent the corresponding codepoints in UTF-16. Hence, these codepoints are reserved.
The overall range U+0000..U+10FFFF includes 1,114,112 codepoints. Subtracting 66 for the non-character positions and 2,048 for the range reserved for surrogates, we find that the Unicode codespace includes 1,111,998 assignable codepoints.
3.2 Script blocks and the organisation of the Unicode character set
As has been mentioned, the Basic Multilingual Plane is intended for those characters that are most commonly used. This implies that the BMP is primarily for scripts that are currently in use, and that other planes are primarily for scripts that are not in current use. This is true to a certain extent.
As development of Unicode began, characters were first taken from existing industry standards. For the most part, those included characters used in writing modern languages, but also included a number of commonly used symbols. As these characters were assigned, they were added to the BMP. Assignments to the BMP were done in an organised manner, with some allowances for possible future additions.
The overall organisation of the BMP is illustrated in Figure 1.
There are a couple of things to be noted straight away. Firstly, note the range of unused codepoints. This is the range U+D800..U+DFFF that is reserved to allow for the surrogates mechanism in UTF-16, as mentioned above (and described in more detail in Section 4).
Secondly, notice the range of codepoints designated “Private Use”. This is a block of codepoints called the Private Use Area (PUA). These codepoints are permanently unassigned, and are available for custom use by users or vendors. This occupies the range U+E000..U+F8FF, giving a total of 6,400 private-use codepoints in the BMP. In addition, the last two planes, Plane 15 and Plane 16, are reserved for private use, giving an additional 131,068 codepoints. Thus, there are a total of 137,468 private-use codepoints that are available for private definition. These codepoints will never be given any fixed meaning in Unicode. Any meaning is purely by individual agreement between a sender and a receiver or within a given group of users.
Before commenting further on the characters in the BMP, let me briefly outline the organisation of the supplementary planes. As just mentioned, Planes 15 and 16 are set aside for private use. Prior to TUS 3.1, no character assignments had been made in the supplementary planes. In TUS 3.1, however, a number of characters were assigned to Planes 1, 2 and 14. Plane 1 is being used primarily for scripts that are no longer in use or for large sets of symbols used in particular fields, such as music and mathematics. Plane 2 is set aside for additional Han Chinese characters. Plane 14 is designated for special-purpose characters; for example, characters that are required for use only in certain communications protocols. No characters or specific purposes have yet been assigned to Planes 3 to 13.
In any of the planes, characters are assigned in named ranges referred to as blocks. Each block occupies a contiguous range of codepoints, and generally contains characters that are somehow related. Typically, a block contains characters for a given script. For example, the Thaana block occupies the range U+0780..U+07BF and contains all of the characters of the Thaana script.
In Section 2, I mentioned that the Standard includes a set of data files. One of these, Blocks.txt, lists all of the assigned blocks in Unicode, giving the name and range for each. The current version may be found at http://www.unicode.org/Public/UNIDATA/Blocks.txt.
While a given language may be written using a script for which there is a named block in Unicode, that block may not contain all of the characters needed for writing that language. Some of the characters for that language’s writing system may be found in other blocks. For example, the Cyrillic block (U+0400..U+04FF) does not contain any punctuation characters. The writing system for a language such as Russian will require punctuation characters in the Basic Latin block (U+0020..U+007F)10 as well as the General Punctuation block (U+2000..U+206F).
Also, the characters for some scripts are distributed between two or more blocks. For example, the Basic Latin block (U+0020..U+007F) and the Latin 1 Supplement block (U+00A0..U+00FF) were assigned as separate blocks because of the relationship each has to source legacy character sets. There are a number of other blocks also containing Latin characters. Thus, if you are working with a writing system based on Latin script, you may need to become familiar with all of these various blocks. Fortunately, only a limited number of scripts are broken up among multiple blocks in this manner. There is also a data file, Scripts.txt, which identifies exactly which Unicode codepoints are associated with each script. The format and contents of this file are described in UTR #24 (Davis 2001e). You are best off simply familiarising yourself with the character blocks in the Unicode character set, but if you need some help, these files are available.
Tables 1 and 2 summarise the scripts covered in the general scripts and East Asian scripts regions of the BMP. (Note that there are a large number of additional Han ideographs in Plane 2.) In addition, there are a number of blocks containing various dingbats and symbols, such as arrows, box-drawing characters, mathematical operators and Braille patterns. Apart from the Braille patterns, most symbols were taken from various source legacy standards.
Table 1. Scripts in the general scripts region of the BMP (in alphabetical order)
Table 2. Scripts in the Asian scripts region of the BMP (in alphabetical order)
There are currently six blocks of assigned characters in Plane 1: Old Italic, Gothic, Deseret, Byzantine Musical Symbols, Musical Symbols (for Western musical notation), and Mathematical Alphanumeric Symbols. These new blocks are described in UTR #27 (Davis et al 2001).
I will not take up space describing each of the scripts and blocks here. They are all described in the Standard. The Standard contains individual chapters (Chapters 6–13) that describe groups of related scripts. For example, Chapter 9 discusses scripts of South and Southeast Asia. If you need to learn how to implement support for a given script using Unicode, then the relevant chapter in the Standard for that script is essential reading.
3.3 Getting acquainted with Unicode characters and the code charts
In addition to the chapters in the Standard that describe different scripts, the Standard also contains a complete set of code charts, organised by block. The best way to learn about the characters in the Unicode Standard is to read the Standard and browse through its charts.
The code charts are included in the printed editions of the Standard and are also available online at http://www.unicode.org/charts/. You can also download the free UniBook™ Character Browser program, which is a very handy chart viewer. In fact, this program was originally created in order to produce the charts used in the production of the Standard. It is available at http://www.unicode.org/unibook/index.html.
The code charts include tables of characters organised in columns of 16 rows. The column headings show all but the last hexadecimal digit of the USVs; thus, the column labelled “21D” shows glyphs for characters U+21D0..U+21DF. Within the charts, combining marks are shown with a dotted circle that represents a base character with which the mark would combine (as explained at the start of this paper). Also, unassigned codepoints are indicated by table cells that are shaded in grey or that have a diagonal lined pattern fill.
The regular code charts are organised in the numerical order of Unicode scalar values. There are also various other charts available online that are organised in different orders. In particular, there are a set of charts available at http://www.unicode.org/unicode/reports/tr24/charts/ that show all characters from all blocks for a given script, sorted in a default collating order. This can provide a useful way to find characters that you are looking for.11 Note that these other charts do not necessarily use the same presentation as the regular code charts, such as using tables with 16 rows. Also, you will probably find it helpful to use both these charts that are organised by scripts as well as the regular charts that are organised by blocks. Because the text describing characters and scripts and the code charts in the Standard itself are organised around blocks, it is important that you not only become familiar with the individual characters used in the writing systems that you work with but also with the blocks in which they are located.
One important note about using the online charts: some are Adobe Acrobat PDF documents, which contain all of the glyphs that are used in the charts. Others are HTML pages, however, and rely on you having appropriate fonts installed in your system and on having your web browser set up to use those fonts. The Unibook program also makes use of fonts installed on your system. If you use Microsoft Office 2000 or individual applications from that suite, you may already have the Arial Unicode MS font installed on your system. This font includes glyphs for all of the characters that were assigned in TUS 2.1. Also, James Kass has made available a pair of fonts named Code2000 (shareware) and Code2001 (freeware) that include glyphs for a large portion of TUS 3.1. These are available from his web site: http://web.archive.org/web/http://code2000.net/. These fonts may be useful to you in viewing some of the online charts or in using the UniBook program.12
Each of the regular code charts, both in the printed book and online, is accompanied by one or more pages of supporting information that is known as the names list. (The UniBook program gives options for viewing the charts with or without the names list.) The names list includes certain useful information regarding each character. This includes some of the normative character properties, specifically the character name and the canonical or compatibility decompositions.13 In addition, it includes a representative glyph for each character as well as some additional notes that provide some explanation as to what this character is and how it is intended to be used.
If you take a quick glance at the names list, you will quickly note that certain special symbols are used. The organisation and presentation of the names list is fully explained in the introduction to Chapter 14 (the code charts) in TUS 3.0. As a convenience, I will briefly describe the meaning of some of the symbols. To illustrate, let us consider a sample entry:
This example is useful as it contains each of the different types of basic element that may be found in an names list entry.
The first line of an entry always shows the Unicode scalar value (without the prefix “U+”), the representative glyph, and the character name.
If a character is also known by other names, these are given next on lines beginning with the equal sign “=”. If there are multiple aliases, they will appear one per line. These are generally given in lower case. In some cases, they will appear in all caps; that indicates that the alternate name was the name used in TUS 1.0, prior to the merger with ISO/IEC 10646.
Lines beginning with bullets “” are informative notes. Generally, these are added to clarify the identity of the character. For example, the note shown above helps to prevent confusion with an over-striking (combining) macron. Informative notes may also be added to point out special properties or behaviour, such as a case mapping that might otherwise not have been expected. Notes are also often added to indicate particular languages for which the given character is used.
Lines that begin with arrows “U+00AF is similar in appearance to U+02C9, but the two are distinct. Another use of cross-references is to point to other characters that are in some linguistic relationship with the given character, such as the other member of a case pair (see, for instance, U+0272 LATIN SMALL LETTER N WITH LEFT HOOK), or a transliteration (see, for instance, U+045A CYRILLIC SMALL LETTER NJE).” are cross-references. The most common purpose of cross-references is to highlight distinct characters with which the given character might easily be confused. For example,
Lines that begin with an equivalence sign “14” or an approximate equality sign “ ” are used to indicate canonical and compatibility decompositions respectively.
At first, you might not always remember the meaning of these symbols in the names list. If you own a copy of the book, you may want to put a tab at the beginning of Chapter 14 for quick reference.
The complete names list is available online as an ASCII-encoded plain text file (without representative glyphs for characters) at http://www.unicode.org/Public/UNIDATA/NamesList.txt. The format used for this file is documented at http://www.unicode.org/Public/UNIDATA/NamesList.html.
As you look through the code charts and the names list, bear in mind that this is not all of the information that the Standard provides about characters. It is just a limited amount that is generally adequate to establish the identity of each character. That is the main purpose they are intended for. If you need to know more about the intended use and behaviour of a particular character, you should read the section that describes the particular block containing that character (within Chapters 6–13), and also check the semantic character properties for that character in the relevant parts of the Standard.15
There are additional things you need to know in order to work with the characters in Unicode, particularly if you are trying to determine how the writing-system of a lesser-known language should be represented in Unicode. Before we look further at the details of the Unicode character set, however, we will explore the various encoding forms and encoding schemes used in Unicode.
4 Unicode encoding forms and encoding schemes
In Section 1, I briefly introduced the three encoding forms that are part of Unicode: UTF-8, UTF-16 and UTF-32. In this section, I will describe each of these in greater detail. (Explicit specifications are provided in “Mapping codepoints to Unicode encoding forms”.) We will also look at two encoding forms used for ISO/IEC 10646, and consider the relative merits of these various alternatives. Finally, I will describe the various encoding schemes defined as part of Unicode, and also the mechanism provided for resolving byte-order issues.
Because of the early history of Unicode and the original design goal to have a uniform 16-bit encoding, many people today think of Unicode as a 16-bit-only encoding. This is so even though Unicode now supports three different encoding forms, none of which is, in general, given preference over the others. UTF-16 might be considered to have a special importance, though, precisely because it is the encoding form that matches popular impressions regarding Unicode.
The original design goal of representing all characters using exactly 16 bits had two benefits. First it made processing efficient since every character was exactly the same size, and there were never any special states or escape sequences. Secondly, it made the mapping between codepoints in the coded character set and code units in the encoding form trivial: each character would be encoded using the 16-bit integer that is equal to its Unicode scalar value. Although it is no longer possible to maintain this fully in the general case, there would still be some benefit it this could be maintained in common cases. UTF-16 does this.
As mentioned earlier, the characters that are most commonly used, on average, are encoded in the Basic Multilingual Plane. Thus, for many texts it is never necessary to refer to characters above U+FFFF. If a 16-bit encoding form were used in which characters in the range U+0000..U+FFFF were encoded as 16-bit integers that matched their scalar values, this would work for such texts, but fail if any supplementary-plane characters occurred. If, however, some of the codepoints in that range were permanently reserved, perhaps they could somehow be used in some scheme to encode characters in the supplementary planes. This is precisely the purpose of the surrogate code units in the range 0xD800–0xDFFF.
The surrogate range covers 2,048 code values. UTF-16 divides these into two halves: 0xD800–0xDBFF are called high surrogates; 0xDC00–0xDFFF are called low surrogates. With 1,024 code values in each of these two sub-ranges, there are 1,024 x 1,024 = 1,048,576 possible combinations. This matches exactly the number of codepoints in the supplementary planes. Thus, in UTF-16, a pair of high and low surrogate code values, known as a surrogate pair, is used in this way to encode characters in the supplementary planes. Characters in the BMP are directly encoded in terms of their own 16-bit values.
So, UTF-16 is a 16-bit encoding form that encodes characters either as a single 16-bit code unit or as a pair of 16-bit code units, as follows:
Table 3. Number of UTF-16 code units per codepoint
It should be pointed out that a surrogate pair must consist of a high surrogate followed by a low surrogate. If an unpaired high or low surrogate is encountered in data, it is considered ill-formed, and must not be interpreted as a character.
The calculation for converting from the code values for a surrogate pair to the Unicode scalar value of the character being represented is described in Section 2 of “Mapping codepoints to Unicode encoding forms”.
One of the purposes of Unicode was to make things simpler than the existing situation with legacy encodings such as the multi-byte encodings used for Far East languages. On learning that UTF-16 uses either one or two 16-bit code units, many people ask how this is any different from what was done before. There is a very significant difference in this regard between UTF-16 and legacy encodings. In the older encodings, the meaning of a code unit could be ambiguous. For example, a byte 0x73 by itself might represent the character U+0073, but it might also be the second byte in a two-byte sequence 0xA4 0x73 representing the Traditional Chinese character ‘mountain’. In order to determine what the correct interpretation of this byte should be, it is necessary to backtrack in the data stream, possibly all the way to the beginning. In contrast, the interpretation of code units in UTF-16 is never ambiguous: when a process inspects a code unit, it is immediately clear whether the code unit is a high or low surrogate. In the worst case, if the code unit is a low surrogate, the process will need to back up one code unit to get a complete surrogate pair before it can interpret the data.
The UTF-8 encoding form was developed to work with existing software implementations that were designed for processing 8-bit text data. In particular, it had to work with file systems in which certain byte values had special significance. (For example, 0x2A, which is “*” in ASCII, is typically used to indicate a wildcard character). It also had to work in communication systems that assumed bytes in the range 0x00 to 0x7F (especially the control characters) were defined in conformance to certain existing standards derived from ASCII. In other words, it was necessary for Unicode characters that are also in ASCII to be encoded exactly as they would be in ASCII using code units 0x00 to 0x7F, and that those code units should never be used in the representation of any other characters.
UTF-8 uses byte sequences of one to four bytes to represent the entire Unicode codespace. The number of bytes required depends upon the range in which a codepoint lies.
The details of the mapping between codepoints and the code units that represent them is described in Section 3 of “Mapping codepoints to Unicode encoding forms”. An examination of that mapping (see Table 3) reveals certain interesting properties of UTF-8 code units and sequences. Firstly, sequence-initial bytes and the non-initial bytes come from different ranges of possible values. Thus, you can immediately determine whether a UTF-8 code unit is an initial byte in a sequence or is a following byte. Secondly, the first byte in a UTF-8 sequence provides a clear indication, based on its range, as to how long the sequence is.
These two characteristics combine to make processing of UTF-8 sequences very efficient. As with UTF-16, this encoding form is far more efficient than the various legacy multi-byte encodings. The meaning of a code unit is always clear: you always know if it is a sequence-initial byte or a following byte, and you never have to backup more than three bytes in the data in order to interpret a character.
Another interesting by-product of the way UTF-8 is specified is that ASCII-encoded data automatically also conforms to UTF-8.
It should be noted that the mapping from codepoints to 8-bit code units used for UTF-8 could be misapplied so as to give more than one possible representation for a given character. The UTF-8 specification clearly limits which representations are legal and valid, however, allowing only the shortest representation. This matter is described in detail in Section 3 of “Mapping codepoints to Unicode encoding forms”.
The UTF-32 encoding form is very simple to explain: every codepoint is encoded using a 32-bit integer equal to the scalar value of the codepoint. This is described further in Section 1 of “Mapping codepoints to Unicode encoding forms”.
4.4 ISO/IEC 10646 encoding forms: UCS-4 and UCS-2
It is also useful to know about two additional encoding forms that are allowed in ISO/IEC 10646. UCS-4 is a 32-bit encoding form that supports the entire 31-bit codespace of ISO/IEC 10646. It is effectively equivalent to UTF-32, except with respect to the codespace: by definition UCS-4 can represent codepoints in the range U+0000..U+7FFFFFFF (the entire ISO/IEC 10646 codespace), whereas UTF-32 can represent only codepoints in the range U+0000..U+10FFFF (the entire Unicode codespace).16
UCS-2 is a 16-bit encoding form that can be used to encode only the Basic Multilingual Plane. It is essentially equivalent to UTF-16 but without surrogate pairs, and is comparable to what was available in TUS 1.1. References to UCS-2 are much less frequently encountered than was true in the past. You may still come across the term, though, so it is helpful to know. Also, it can be useful in describing the level of support for Unicode that certain software products may provide.
4.5 Which encoding is the right choice?
With three different encoding forms available, someone creating content is faced with the choice of which encoding they should use for the data they create. Likewise, software developers need to consider this question both for what they use as the internal memory representation of data and what they use when storing data on a disk or transmitting it over a wire. The answer depends on a variety of factors, including the nature of the data, the nature of the processing, and the contexts in which it will be used.
One of the original concerns people had regarding Unicode was that a 16-bit encoding form would automatically double file sizes in relation to an 8-bit encoding form.17 Unicode’s three encoding forms do differ in terms of their space efficiency, though the actual impact depends upon the range of characters being used and on the proportions of characters from different ranges within the codespace. Consider the following:
Table 4. Bytes required to represent a character in each encoding form
Clearly, UTF-32 is less efficient, unless a large proportion of characters in the data come from the supplementary planes, which is usually not likely. (For supplementary-plane characters, all three encoding forms are equal, requiring four bytes.) For characters in the Basic Latin block of Unicode (equivalent to the ASCII character set), i.e. U+0000..U+007F, UTF-8 is clearly the most efficient. On the other hand, for characters in the BMP used for Far East languages (all are in the range U+2E80..U+FFEF), UTF-8 is less efficient than UTF-16.
Another factor particularly for software developers to consider is efficiency in processing. UTF-32 has an advantage in that every character is exactly the same size, and there is never a need to test the value of a code unit to determine whether or not it is part of a sequence. Of course, this has to be weighed against considerations of the overall size of data, for which UTF-32 is generally quite inefficient. Also, while UTF-32 may allow for more efficient processing than UTF-16 or UTF-8, it should be noted that none of the three encoding forms is particularly inefficient with respect to processing. Certainly, it is true that all of them are much more efficient than are the various legacy multibyte encodings.
For general use with data that includes a variety of characters mostly from the BMP, UTF-16 is a good choice for software developers. BMP characters are all encoded as 16-bits, and testing for surrogates can be done very quickly. In terms of storage, it provides a good balance for multilingual data that may include characters from a variety of scripts in the BMP, and is no less efficient than other encoding forms for supplementary-plane characters. For these reasons, many applications that support Unicode use UTF-16 as the primary encoding form.
There are certain situations in which one of the other encoding forms may be preferred, however. In situations in which a software process needs to handle a single character (for example, to pass a character generated by a keyboard driver to an application), it is simplest to handle a single UTF-32 code unit. On the other hand, in situations in which software has to cooperate with existing implementations that were designed for 8-bit data only, then UTF-8 may be a necessity. UTF-8 has been most heavily used in the context of the Internet for this reason.
On first consideration, it may appear that having three encoding forms would be less desirable. In fact, having three encoding forms based on 8-, 16- and 32-bit code units has provided considerable flexibility for developers and has made it possible to begin making a transition to Unicode while mai
Note: If you want to add a response to this article, you need to enable cookies in your browser, and then restart your browser.
Note: the opinions expressed in submitted contributions below do not necessarily reflect the opinions of our website.
© 2003-2020 SIL International, all rights reserved, unless otherwise noted elsewhere on this page.