Skip to content
Search

International Components for Unicode

lastmod 2018-08-17created 2016-03-2913 edits

A cross-platform open-source C/C++ and Java library for handling Unicode in computer software.

Some of its features are supported more lightly and quickly by other libraries (for example, Unicode code conversion by iconv, etc.), but because it aims to support the latest Unicode standard almost exactly 100%, it holds a nearly unrivaled position in internationalization (Internationalization, Globalization), which makes software work with the characters and language systems of many countries.

It is almost always included in games made in the West that properly support Korean. (A representative example is Sid Meier's Civilization V)

Packages

Package

Contents

icu-uc

Common (uc) and Data (dt/data) libraries

icu-i18n

Internationalization (in/i18n) library

icu-le

Layout Engine

icu-lx

Paragraph Layout

icu-io

Ustdio/iostream library (icuio)

String handling (Char & String)

There are many functions for handling string length, index, position, etc. Functions in the standard c library such as strlen and strcat handle only ASCII codes, but ICU provides corresponding functions for Unicode. It also supports character/string properties (whether something is a letter, a digit, a symbol, etc.), string iterators, and even regular expressions.

icu::UnicodeString

UTF-8 Encoding/Decoding:

  • fromUTF8(const StringPiece &utf8)
  • toUTF8String(StringClass &result)
  • toUTF8(ByteSink &sink)

And extract() is not prefered now.

Code conversion (Conversion)

It can convert between almost all codes used on computers, including ASCII, EBCDIC, the various codes defined by ISO, and UTF-8, UTF-16 (LE, BE), UTF-32, etc.

Example

Locale (Locale)

A base class that does not do anything by itself but is closer to a concept that serves as the foundation for ICU's other features. It defines the concepts of language (Language), script (Script) and country (Country) separately, and on that basis makes the processing rules for strings, etc. clear.

For example:

  • Many countries use the same alphabet (Script) and the same language (Language), but there may be slight differences by country.
  • The Cia-Cia, an ethnic minority in Indonesia, have the Cia-Cia language (Launguage) but no script to write it, so they adopted Hangul as their official script.
  • Even for the same Hangul, the character order in the Republic of Korea differs from the character order in North Korea's Munhwaŏ (Cultured Language).

So when the same script or language needs different processing depending on the country where it is used, or conversely when one country has several official languages or official scripts, you need to define a Locale and process strings, etc. based on it to handle them differently.

Date and time (Date/Time)

Functions for handling all kinds of dates and times. Besides Unicode counterparts to the date-time functions in the standard c library, etc., it provides more diverse and powerful features.

  • Supports almost all calendars used around the world, including the Gregorian calendar (solar calendar) and the lunar calendar as well as the Persian, Indian and Ethiopian calendars, and date conversion between them.
  • Conversion between each country's local time <-> UTC

Formatting (Formatting)

  • Display of each country's currency symbols
  • Converts all kinds of dates and times to strings according to the locale - e.g. if you ask for 2015/5/15 in Korean style it returns 2015년 5월 15일 (2015 nyeon 5 wol 15 il), and in American style it returns 15, May, 2015

Transform (Transform)

  • Case conversion for each language, Titlecase (first letter of a word uppercase, the rest lowercase)
  • Bidi algorithm. Scripts such as Arabic and Hebrew are written from right to left. Of course, strings stored inside the computer are stored from left to right just like other scripts (more precisely, from lower to higher memory addresses).

Order (Collation)

Handles the order (when sorting), search, etc. for various languages and scripts. Even for the same Hangul, the order in South Korean standard language (ㄱ ㄲ ㄴ ㄷ ㄸ ㄹ ㅁ ㅂ ㅃ ㅅ ㅆ ㅇ ㅈ ㅉ ㅊ ㅋ ㅌ ㅍ ㅎ) differs from the order in North Korea's Munhwaŏ (ㄱ ㄴ ㄷ ㄹ ㅁ ㅂ ㅅ ㅈ ㅊ ㅋ ㅌ ㅍ ㅎ ㄲ ㄸ ㅃ ㅆ ㅉ ㅇ), and in German the order in dictionaries differs from the order in phone books.

Boundary analysis (Boundary Analysis)

Analyzes where the boundaries of characters/strings/sentences are, according to the country/language/script.

  • Character boundaries - The character Ä can be represented in Unicode as one code point or as two code points.
  • Word boundaries - English is simple because there is a rule that all words are separated by spaces, but even in Korean written in Hangul, it is ambiguous where one word ends. In languages such as Arabic and Thai there are no spaces at all, so all the characters are joined together!
  • Line break boundaries - When a sentence must be displayed on screen but is too long and needs a line break. It seems you could just cut by word, but this differs from word boundaries. Naturally, it all differs by script/language/country!

IO

ustdio, ustream

Layout Engine

ICU Data

ICU makes use of a wide variety of data tables to provide many of its services. Examples include converter mapping tables, collation rules, transliteration rules, break iterator rules and dictionaries, and other locale data. Additional data can be provided by users, either as customizations of ICU's data or as new data altogether.

How to compile static icudata

Download the ICU data library

http://apps.icu-project.org/datacustom/

Place the downloaded data library at the following path in the source code. 

./data/in/icudt57l.dat

Run configuration & make: 

$ ./configure && make -j8

Check that the icupkg.inc file was generated properly. 

./data/icupkg.inc

WARNING

You must not use the assembly flag used for genccode in the icupkg.inc file above. Remove the GENCCODE_ASSEMBLY_TYPE variable.

Manually create

If you want to generate the C files manually, proceed as follows. 

## Extract items from the package:
$ ./bin/icupkg -d ./data/out/build/icudt57l --list -x \* ./data/in/icudt57l.dat -o ./data/out/tmp/icudata.lst

## Produce packaged ICU data from the given list(s) of files:
## if you want to change the temp directory: -T ./data/out/tmp -> -T ./output_temp
$ ./bin/pkgdata -O ./data/icupkg.inc -q -c -s ./data/out/build/icudt57l -d ./lib -e icudt57 -T ./data/out/tmp \
                  -p icudt57l -m static -r 57.1 -L icudata ./data/out/tmp/icudata.lst

Check that the C files were generated in the ./data/out/tmp directory.

Referring to the icupkg.inc file, compile all of the source code above and then compile statically.

For reference, below are the COMPILE and LIBFLAGS variable parts of that file generated on MacOSX. 

COMPILE=clang -DU_ATTRIBUTE_DEPRECATED=   -DU_HAVE_ATOMIC=1  -fPIC -std=c99 -Wall -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings  -Qunused-arguments -Wno-parentheses-equality -fno-common -c
LIBFLAGS=-I../common -I../common  -dynamic

Documentation

GettingStartedWithICU_a005_iuc27.ppt.zip
http://www.icu-project.org/docs/papers/GettingStartedWithICU_a005_iuc27.ppt

See also

Favorite site

sitemap