Skip to content
Search

C++:Encoding

lastmod 2022-12-08created 2016-07-0312 edits

Encodings in C Strings.

Category

The items you need to know are as follows.

MSVC char encoding

ust for additional information:

'ANSI' refers to windows-125x, used for win32 applications while 'OEM' refers to the code page used by console/MS-DOS applications. Current active code-pages can be retrieved with functions GetOEMCP() and GetACP().

In order to output something correctly to the console, you should:

  • ensure the current OEM code page supports the characters you want to output (if necessary, use SetConsoleOutputCP() to set it properly)
  • convert the string from current ANSI code (win32) to the console OEM code page

Here are some utilities for doing so: 

// Convert a UTF-16 string (16-bit) to an OEM string (8-bit) 
#define UNICODEtoOEM(str)   WCHARtoCHAR(str, CP_OEMCP)

// Convert an OEM string (8-bit) to a UTF-16 string (16-bit) 
#define OEMtoUNICODE(str)   CHARtoWCHAR(str, CP_OEMCP)

// Convert an ANSI string (8-bit) to a UTF-16 string (16-bit) 
#define ANSItoUNICODE(str)  CHARtoWCHAR(str, CP_ACP)

// Convert a UTF-16 string (16-bit) to an ANSI string (8-bit)
#define UNICODEtoANSI(str)  WCHARtoCHAR(str, CP_ACP)


/* Convert a single/multi-byte string to a UTF-16 string (16-bit).
 We take advantage of the MultiByteToWideChar function that allows to specify the charset of the input string.
*/
LPWSTR CHARtoWCHAR(LPSTR str, UINT codePage) {
    size_t len = strlen(str) + 1;
    int size_needed = MultiByteToWideChar(codePage, 0, str, len, NULL, 0);
    LPWSTR wstr = (LPWSTR) LocalAlloc(LPTR, sizeof(WCHAR) * size_needed);
    MultiByteToWideChar(codePage, 0, str, len, wstr, size_needed);
    return wstr;
}


/* Convert a UTF-16 string (16-bit) to a single/multi-byte string.
 We take advantage of the WideCharToMultiByte function that allows to specify the charset of the output string.
*/
LPSTR WCHARtoCHAR(LPWSTR wstr, UINT codePage) {
    size_t len = wcslen(wstr) + 1;
    int size_needed = WideCharToMultiByte(codePage, 0, wstr, len, NULL, 0, NULL, NULL);
    LPSTR str = (LPSTR) LocalAlloc(LPTR, sizeof(CHAR) * size_needed );
    WideCharToMultiByte(codePage, 0, wstr, len, str, size_needed, NULL, NULL);
    return str;
}

Example

When printing a Unicode string in a Windows environment that uses the MS949 CodePage (assuming console output has been set to UTF-8 using SetConsoleOutputCP), the following two cases print correctly.

  • "\xea\xb0\x80"
  • u8"가"

Below is the code actually tested: 

//MS949 CODEPAGE

#include <iostream>
#include <locale>

using namespace std;

int main()
{
    SetConsoleOutputCP(CP_UTF8);
    //GetConsoleOutputCP();

    // TEST CODE:
    //std::cout.imbue(std::locale("ko_KR.UTF-8"));
    //std::cout.imbue(std::locale("kor"));
    //std::locale::global(std::locale("ko_KR.UTF-8"));
    //_setmode(_fileno(stdout), _O_U8TEXT);

    //std::cout << u8"\xea\xb0\x80" << "////";
    //std::cout << "\xea\xb0\x80" << "////";
    //std::cout << u8"가" << "////";
    //std::cout << "가" << "////";

    printf(u8"\xea\xb0\x80////"); // ???
    printf("\xea\xb0\x80////"); // UTF8 (0xEA 0xB0 0x80)
    printf(u8"가////"); // UTF8 (0xEA 0xB0 0x80)
    printf("가////"); // MS949 (0xB0 0xA1)

    return 0;
}

For reference, cout seems to have an issue related to locale or imbue. It does not print correctly in MSVC.

See also

Favorite site

References

  1. Working_with_Encodings_in_C_Strings.pdf ↩

  2. Unicode_in_C_and_Cpp_-_Cprogramming.pdf ↩

  3. Stackoverflow_-_wstring_vs_string.pdf ↩

  4. UTF-8_and_Unicode_FAQ.pdf ↩

sitemap