Comments on: Transcoding adventures with C

By: Andrew

Andrew — Thu, 07 Jun 2007 19:49:27 +0000

Counting Multi-Byte Characters using C99 WCS
======================================
Multi-Byte character sets like UTF-8 have a variable number of bytes per character. The ISO C99 has a special form of the MBSTOWCS(3) call to count the characters.

To count the characters (not bytes), compute the wchar_t fixed-width character-length. Set the result wide-char buffer to NULL and its width to ZERO. Then, mbstowcs() will just count the number of characters without really building the wchar_t* string.

char *str = “Some UTF-8 String”;
/* Count the characters in a multi-byte character set: */
size_t chars = mbstowcs(NULL, str, 0);

By: josh

josh — Thu, 07 Jun 2007 16:08:33 +0000

Good catch Andrew! My OS X box returns zero as well. I guess I wasn’t paying enough attention when I ran the test program initially.

By: Andrew

Andrew — Thu, 07 Jun 2007 15:10:00 +0000

ICONV(3) Return Value
===================
The manpage for ICONV(3) reports its return value as “the number of characters converted in a non-reversible way”. This is very different from the total number of characters in the converted buffer! A NON-ZERO return value really indicates a kind of error. Later, this string can’t be converted back to the original charset.

On my Fedora Linux box, the sample program reports a return value of ZERO in the “chars” variable. This means that the LATIN-1 character (octal 370) is reversibly converted to UTF-8.

size_t chars = iconv(…);