Skip to content
Search

Unicode

lastmod 2026-07-15created 2015-05-0739 edits

Unicode is an industry standard designed so that every character in the world can be represented and handled consistently on computers, and it is maintained by the Unicode Consortium. The standard includes the ISO 10646 character set, character encodings, a character information database, and algorithms for handling characters.

The goal of Unicode is to replace all existing character encoding methods with Unicode. Existing encodings were limited in size and scope, and were incompatible with each other in multilingual environments. As Unicode succeeded in unifying various character sets, it came to be widely used for the internationalization and localization of computer software, and it is supported by relatively recent technologies such as XML, Java, and modern operating systems.

To represent Korean pronunciation, Unicode uses ISO/TR 11941, a variant of the Yale romanization.

What is unicode 

Unicode provides a unique number for every character,
no matter what the platform,
no matter what the program,
no matter what the language.

Unicode assigns an index to every character. Nothing more, nothing less. This index is called a code point, but here we will just call it an index.

  • The character 'A' has the index 0x0041.
  • The character 'a' has the index 0x0061.
  • The character '가' (Hangul syllable "ga") has the index 0xac00.

To see more characters and indexes, refer to http://www.unicode.org/charts/. There are two kinds of methods for representing these predefined indexes: UTF and UCS.

Unicode is a code chart in which numbers and characters, that is, keys and values, are mapped 1:1. UTF-8 and EUC-KR are encoding schemes.

Terms

The terms you encounter most when reading Unicode-related documents are words like UCS2, UCS4, UTF8, UTF16, and UTF32.

Basic plane, BMP (Basic Mulitilingual Plane)
Refers to the first 65,536 codes of Unicode.
Plane (a group of 256x256, i.e. 65,536 codes)
Unicode currently can use 17 planes. All of them are in group 00.
Plane group, Group (a group made of 256 planes)
All 17 planes of Unicode are in Group 00. Unicode is defined only up to 17 planes. In contrast, the ISO standard (UCS-4) can define up to 128 plane groups.
  • 1 Plane = 65,536 code points
  • 1 Group = 256 planes = 256x65,536 = 16,777,216 code points
  • UCS-4 = 128 groups = 128x16,777,216 = 2,147,483,648 code points
Encoding (the way a character set is represented)
Unicode names a code system or character set, and to represent it you need an encoding such as UTF-8, UTF-16, or UTF-32.
UCS-2: Universal Character Set 2(octets)
More precisely, Universal Multipe-Octet Coded Character Set 2. An ISO/IEC 10646 term that defines the 65,536 codes of the BMP, represented in 2 bytes. Only one plane, the BMP, falls under it. UCS-2 is not an encoding method but the character code itself. It would be fine to think of it as an encoding, though. The term octet used here is an ISO term with the same meaning as the byte used on the Unicode side
UCS-4: Universal Character Set 4(octets)
An ISO/IEC 10646 term, represented in 4 bytes. It defines 128 plane groups in total, i.e. 128*256 planes = 32,768 planes. This corresponds to about 231 = 2,147,483,648 codes. UCS-4 is not an encoding method but the character code itself.
UTF-8: UCS Transformation Format, 8-bit form
One of the encoding schemes of the Unicode standard. The standard can only represent characters in the 17 planes, but technically it can represent characters across the entire UCS-4 range. Depending on the character, it uses 1 to 4 (or 6) bytes.
UTF-16: UCS Transformation Format, 16-bit form
Unicode 3.0 describes the 16 not as 16 bits but as the 16 planes of group 00. Since the 32 in UTF-32 refers to 32 bits, it is better to understand it as 16 bits for consistency. It is similar to UCS-2 in that it uses 16 bits, but it is an encoding that can represent codes in the 16 supplementary planes using the surrogate area (Surrogates). Two surrogate units can represent the 16 supplementary planes. UCS-2 can define only 65536 codes, but UTF-16 can represent about one million more characters.
UTF-32: UCS Transformation Format, 32-bit form
Represents each character in 32 bits, i.e. 4 bytes. In this respect it is the same as UCS-4, but since it defines only 17 planes, you can consider it a subset of UCS-4. Think of it as the same as UCS-4, except that only the range 0x00000000 ~ 0x0010FFFF is treated as character codes.

Convert encoding

UTF-8, UTF-16, UTF-32, UCS-2, and UCS-4 are all ways of representing Unicode character indexes, so converting between them naturally works well. (However, UCS-2 has limitations)

Is Unicode in C++ actually UTF-16? 

I wonder if all of the functions marked with W suffix for Unicode, actually accept UTF-16 text only, or are they capable to handle other UTFs?

All Windows API functions that accept LP[C]WSTR expect UTF-16 encoding. 

And what about wchar_t type itself, is it made to specifically handle UTF-16 characters and strings?

Formally speaking, no. The size of wchar_t is implementation-defined, and in fact most Linux compilers (such as GCC) define it to be 32-bit large, designed to hold text in UTF-32 encoding. With MSVC compiler, wchar_t is 16-bit large, with the intention that it be used to represent Unicode strings in UTF-16 encoding. 

Also, what's the difference between UTF-8 and ANSI?

http://www.joelonsoftware.com/articles/Unicode.html 

Can a character of char type or string of char type characters handle UTF-8?

Yes. A UTF-8 string is a sequence of 8-bit elements (hence '8' in its name), and is thus readily representable in an array of char.

Plane (Unicode block)

A Unicode plane is a logical division of the whole of Unicode. There are 17 planes in total, numbered 0 (the Basic Multilingual Plane) through 16, and each plane consists of 65536 (216) codes.

Unicode has blocks, which are sets of contiguous code points. Block names are unique, and blocks do not overlap with other blocks. The size of a block is a multiple of 16, and a block starts at a code point that is a multiple of 16 (U+nnn0). A block can include code points that are unassigned.

Unicode 8.0 has 262 blocks. Of these, 160 are in plane 0, the Basic Multilingual Plane (BMP); 93 are in plane 1, the Supplementary Multilingual Plane (SMP); 5 are in plane 2, the Supplementary Ideographic Plane (SIP); 2 are in plane 14, the Supplementary Special-purpose Plane (SSP); and planes 15 and 16 exist in their entirety as 'Supplementary Private Use Area-A' and 'Supplementary Private Use Area-B', respectively.

Unicode list

Basic Multilingual Plane
BMP

Supplementary Multilingual Plane
SMP

Supplementary Ideographic Plane
SIP

Tertiary Ideographic Plane
TIP

Supplementary Special-purpose Plane
SSP

Private Use Area
PUA

0000~0FFF
1000~1FFF
2000~2FFF
3000~3FFF
4000~4FFF
5000~5FFF
6000~6FFF
7000~7FFF

8000~8FFF
9000~9FFF
A000~AFFF
B000~BFFF
C000~CFFF
D000~DFFF
E000~EFFF
F000~FFFF

10000~10FFF
11000~11FFF
12000~12FFF
13000~13FFF
14000~14FFF

16000~16FFF
17000~17FFF

18000~18FFF


1B000~1BFFF

1D000~1DFFF
1E000~1EFFF
1F000~1FFFF

20000~20FFF
21000~21FFF
22000~22FFF
23000~23FFF
24000~24FFF
25000~25FFF
26000~26FFF
27000~27FFF

28000~28FFF
29000~29FFF
2A000~2AFFF
2B000~2BFFF
2C000~2CFFF


2F000~2FFFF

No characters

E0000~E0FFF

F0000~FFFFF
100000~10FFFF

Basic Multilingual Plane

The Basic Multilingual Plane (BMP) is the first (number 0) plane of Unicode, occupying the range from U+0000 to U+FFFF. The BMP contains almost all modern characters and special characters, most of which are Hangul and CJK unified ideographs.

Private Use Area Planes

Abbreviated as PUA, the two planes 15 and 16 (planes F and 10 in hexadecimal) each contain a "private use area".

These contain the blocks PUA-A (Supplementary Private Use Area-A) and PUA-B (Supplementary Private Use Area-B).

The private use areas can be used by parties outside ISO and Unicode (private character encodings).

Plane

Block range

Block name

15 PUA-A

U+F0000..U+FFFFF

Supplementary Private Use Area-A

16 PUA-B

U+100000..U+10FFFF

Supplementary Private Use Area-B

For reference, Material Design Icons (MDI) uses codes in the PUA-A area.

List of blocks

Musical Symbols

See Standard Music Font Layout (SMuFL).

Hangul in Unicode

See also

Favorite site

Tutorials

Online tools

  • Shapecatcher - find similar Unicode characters
    • A website that finds similar Unicode characters when you draw on the screen
    • Currently 11817 characters in the DB
    • Korean/Japanese/Chinese characters are not currently supported (support planned)
    • Seems pretty much abandoned since 2012...

References

  1. Korean_and_Unicode.zip ↩

  2. Glossary_of_Unicode_Terms_-_unicode_v9.0.pdf ↩

  3. D2_naver_helloworld_-_Korean_encoding_01.pdf ↩

  4. D2_naver_helloworld_-_Korean_encoding_02.pdf ↩

  5. Bsidesoft.com_-_unicode_01.pdf ↩

  6. Bsidesoft.com_-_unicode_02.pdf ↩

  7. Bsidesoft.com_-_unicode_03.pdf ↩

author Zer0 sitemap