Thursday, November 24, 2005

How many Unicode characters are there ?

Otto Stolz asked on the Unicode List how many Unicode characters there were, classified as control characters, format characters, graphic characters, private use chacters, noncharacters, surrogate code points, etc. Now I love Unicode facts, figures and trivia, so I can't resist trying to answer this question, even if I only end up confusing everybody.

The "Unicode Version History" utility of BabelMap provides precisely the information requested by Otto for all versions of Unicode from 1.0.0 up to the current version (4.1 when I first wrote this post, but now updated to 5.1). This information is tabulated below :

BabelMap : Unicode Version History
Version1.0.01.0.11.12.02.13.03.13.24.04.15.05.1
DateOctober
1991
June
1992
June
1993
July
1996
May
1998
September
1999
March
2001
March
2002
April
2003
March
2005
July
2006
April
2008
Scripts242524252538414552596475
Blocks57596367678695107122142151168
Total
Code
Points
65,53665,53665,5361,114,1121,114,1121,114,1121,114,1121,114,1121,114,1121,114,1121,114,1121,114,112
Assigned
Code
Points
12,79534,50540,635178,500178,502188,809233,787234,803236,029237,302238,671240,295
Unassigned
Code
Points
52,74131,03124,901935,612935,610925,303880,325879,309878,083876,810875,441873,817
Encoded
Characters
7,16128,35934,23338,95038,95249,25994,20595,22196,44797,72099,089100,713
Private
Use
Characters
5,6326,1446,400137,468137,468137,468137,468137,468137,468137,468137,468137,468
Non
characters
222343434666666666666
Surrogate
Code
Points
0002,0482,0482,0482,0482,0482,0482,0482,0482,048
Graphic
Characters
7,08528,28334,15138,86738,86949,16894,00995,02396,24397,51598,884100,507
Format
Characters
222181826131133139140140141
Control
Characters
747480656565656565656565

Note that for historic versions of Unicode the statistics are based on the General Category of the characters at the time of encoding, and do not take into account any subsequent changes in General Category. Thus the fact that 4.0 has 139 format characters and 4.1 has 140 format characters is not due to a new format character having been added in 4.1, but rather due to the General Category of U+200B ZERO WIDTH SPACE having been changed from Zs to Cf in Unicode 4.0.1. Note that the statistics for 1.0.0 and 1.0.1 are based upon Ken Whistler's reconstructed Unicode Character Data.

To help understand what we're talking about, here are some definitions of some of the terms used in the table (see Section 2.4 of the Unicode Standard for further information).

  • Control characters are those characters with a General Category of Gc. These are invisible characters that perform a certain function that is defined by a protocol or standard other than Unicode (they are inherited from pre-existing 8-bit standards). They include familiar characters such as Tab, Carriage Return and Line Feed that are essential to writing Unicode (U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR never took off as Unicode replacements for Carriage Return and/or Line Feed), as well as many characters that you should never see in plain text (the full range of control characters is 0000..001F and 007F..009F).
  • Format characters are those characters with a General Category of Cf, Zl or Zp. These are invisible characters defined by Unicode for a particular function. These include things like U+200D ZERO WIDTH JOINER, U+202D LEFT-TO-RIGHT OVERRIDE, interlinear annotation characters (FFF9..FFFB) and the infamous set of Tag characters (E0001 and E0020..E007F). They work behind the scenes to do useful things like bidirectional control and character shaping.
  • Graphic characters are those characters with a General Category other than Cc, Cn, Co, Cs, Cf, Zl and Zp, that is to say ordinary visible characters.
  • Noncharacters are code points that are permanently reserved, and are guaranteed never to be assigned as characters. They are the thirty-four code points ending in xxFE and xxFF, as well as the thirty-two code points in the range FDD0..FDEF.
  • Encoded characters = Graphic characters + Format characters + Control characters.
  • Assigned code points = Encoded characters + Private Use characters + Noncharacters + Surrogate code points.
  • Total code points = Assigned code points + unassigned code points.

2 comments:

Suz said...

What about Phoenician? Is it in or out?

Suzanne

Andrew said...

In. Please see todays blog "What's new in Unicode 5.0".

Archive

Followers

About Me