There are many standards for encoding Asian characters. In Japan, for example, five encodings are in broad use: JIS, SJIS, EUC, Unicode, and UTF-8.

Usually an application uses one encoding for all strings to be stored inside a database. The encoding chosen is most often the one used in the operating system of the ObjectStore client.

However, if the application has heterogeneous clients using a variety of encodings, conversion from one encoding to another is necessary at some point. The clients could be traditional ObjectStore client processes or thin-client browsers that emit data in different encodings.

The Class Library: os_str_conv

The library provides a facility to detect the encoding of a given string. This is useful for applications in which a client might send strings in an unknown format, a common problem for Internet applications.

The most common application of this class is conversion between EUC and SJIS to provide sharing of data from UNIX <-> Windows applications. JIS is commonly used for email. Applications normally store data in a homogeneous format inside a database, and incoming strings are converted as required before they are persistently allocated. Outgoing strings can also be converted to the client's native encoding. For Web applications this outgoing conversion is usually not necessary since internationally aware browsers (Netscape 2.0 and above, for example) can automatically detect and convert various incoming formats themselves.

The class library currently consists of a single class, os_str_conv, instantiated once for each conversion path required:

os_str_conv(encode_type dst, encode_type src=automatic); 

enum encode_type {    /* string encode type ------------------- */ 
UNKNOWN=0,              /* convert or automatic detect fail            */ 
AUTOMATIC,              /* detect automatically                               */ 
AUTOMATIC_ALLOW_KANA,
                                              /* detect automatically, allow half-width-kana */ 
ASCII,                                                               /* ASCII                                       */ 
SJIS,                                         /* Shift-JIS                                  */ 
EUC,                                          /* EUC                                          */ 
UNICODE,                                            /* Unicode (can't automatic detect)   */ 
JIS,                                         /* JIS                                                   */ 
UTF8                                         /* UTF-8 (can't automatic detect)     */ 
/* add new encode type here ! */ 
}; 

os_str_conv *sjis_to_euc = new os_str_conv(os_str_conv::EUC, 
os_str_conv::SJIS); 

char *euc_dest = new char[sjis_to_euc->get_converted_size(
      sjis_src)]; 
sjis_to_euc->convert(euc_dest, sjis_src); 

Automatic Detection of a Source String Encoding

os_str_conv *to_euc = new os_str_conv(
      os_str_conv::AUTOMATIC); len =
             to_euc->get_converted_size(unknown_src);
      if (len) 
      { 
      char *euc_dest = 
            new char[to_euc->get_converted_size(unknown_src)]; 
      to_euc->convert(euc_dest, sjis_src);
       } 
      else 
      { 
// couldn't convert -- application needs to handle this! 
      } 

If it fails inside get_converted_size, get_converted_size returns 0 to indicate the failure. Be careful not to allocate strings based on its return value without checking for failure!

Unfortunately, no automatic detection algorithm can correctly distinguish EUC from SJIS in all cases because of overlap in their assignment ranges. Clever algorithms exploit patterns typical of real text. This implementation is reasonably straightforward. The most difficult problem (distinguishing between SJIS half-width kana and EUC) is avoided by asking the user to choose between the two possible interpretations. In nearly all cases, os_str_conv::AUTOMATIC is the appropriate setting.

In practice, the problems of ambiguity are not likely to affect applications, since usually incoming text is all in the single encoding defined by the operating system used when generating it. Autodetect can be used at the beginning of a session only, and it can be reasonable to assume that it will not change.

As mentioned earlier, there are areas in the EUC and SJIS encodings that overlap, and so a given string might be valid in either encoding. This makes autodetection ambiguous.

There are two ambiguous cases:

• The half-width kana of SJIS. It is possible for a string consisting entirely of bytes in this range to be either SJIS or EUC. This is the most troublesome case.
• An obscure range of SJIS and EUC that overlaps. The characters represented by this range are rarely used, so it is highly unlikely that a string would consist entirely of such characters.

1. The algorithm examines each character of the string in sequence until the encoding is determined. Therefore, a string beginning with an unambiguous substring followed by an ambiguous substring is detected according to the first substring.
   
   
2. Strings consisting entirely of the second ambiguous type are handled as unknown. As mentioned, this case is very unlikely.
   
   
3. All SJIS half-width kana are single-byte encodings. Therefore, a string consisting entirely of an odd number of bytes in the SJIS half-width kana range is considered SJIS.
   
   
4. A string beginning with an even number of bytes in the SJIS half-width kana range is ambiguous until the following characters are examined according to normal detection rules.
   
   
5. A string beginning with an odd number of bytes in the SJIS half-width kana range requires special examination of the last character. If this is an EUC first-byte code, and it is followed by a valid EUC second-byte code, then the string is EUC. However, if the following code is not a valid EUC second-byte (it might be ordinary ASCII), then the final character is interpreted as SJIS half-width kana and the string is interpreted as SJIS.
   
   
6. A string consisting entirely of an even number of bytes in the SJIS half-width kana range is ambiguous. It is quite possible for such a string to appear in real applications. The os_str_conv::automatic setting causes the autodetector to interpret this case as SJIS. However, if os_str_conv::automatic_allow_kana, this case is interpreted as unknown. Ojbect Design believes that the SJIS interpretation is correct for most cases.
   
   

Unlike EUC and SJIS, JIS is a modal encoding that uses <Esc> to enter and exit from multibyte mode. Detecting JIS strings is accomplished by searching for these <Esc> characters.

How to Instantiate the Converter

Guidelines for Extensions to os_str_conv

Additional encodings can be appended to the existing enumeration. Note that os_str_conv depends on the ordering of the existing encodings, so if you extend os_str_conv, additional encodings must appear after the ones already provided.

What Are the Different Modes and Their Meanings?

• EUC and Unicode are a superset of SJIS and so roundtrip EUC/Unicode<->SJIS is not possible for all EUC/Unicode Japanese characters.
• There are a handful of cases of pairs of SJIS characters that map to a single character in Unicode.

• Third, SJIS contains some special characters that are printable on Windows. Although mappings are defined for EUC, attempts to view them on X-windows, at least, fail because the fonts in use do not provide glyphs for those codes. There are no encodings for these characters in Unicode.
• Lastly, JIS defines multiple ways to express a character (the base semantic unit), so a conversion from JIS to another encoding and back to JIS is not guaranteed to return an identical binary string. However, the meaning of the string (in the sense of the way it would appear if printed on a screen) is the same.

The deviations in mapping tend to be quite small. For example, here is a table that shows the incompatibility of the Unicode Consortium standard and the maps that Microsoft uses in Windows NT:

SJIS Code	Unicode Consortium Mapping	Microsoft Mapping
\ 5C	00A5 YEN SIGN	005C REVERSE SOLIDUS(*)
~ 7E	203E OVERLINE	007E TILDE
^[$B!@(B 81,5F	005C Reverse solidus	FF3C FULLWIDTH REVERSE SOLIDUS
^[$B!A(B 81,60	301C WAVE DASH	FF5E FULLWIDTH TILDE
^[$B!B(B 81,61	2016 DOUBLE VERTICAL LINE	2225 PARALLEL TO
^[$B!](B 81,7C	2212 MINUS SIGN	FF0D FULLWIDTH HYPHEN-MINUS
^[$B!q(B 81,91	00A2 CENT SIGN	FFE0 FULLWIDTH CENT SIGN
^[$B!r(B 81,92	00A3 POUND SIGN	FFE1 FULLWIDTH POUND SIGN
^[$B"L^(B 81,CA	00AC NOT SIGN	FFE2 FULLWIDTH NOT SIGN

Instructions on Overriding Particular Mappings

class os_str_conv { 
public:  
...
            struct mapping {
                  os_unsigned_int32 dest;      /* destination code */
                  os_unsigned_int32 src;         /* source code      */
             };
            int change_mapping(mapping table[],size_t table_sz);
      ...
      };

The override mapping information applies to whatever explicit mapping has been established for the given os_str_conv instance. Mappings of os_str_conv instances cannot be overridden by instances using autodetect. Attempts to do so return -1 from change_mapping() to indicate this error condition.

The change_mapping() method takes the following two parameters:

• os_str_conv::mapping_table[]

Internally, change_mapping() makes a sorted copy of mapping_table[]. The sorting provides quick lookup at run time. The internal copy is freed when the os_str_conv destructor is eventually called.

Note that the mapping pairs are unsigned 32-bit quantities. The LSB is on the right, so, for example, the single-byte character 0x5C is represented as 0x0000005C, and the two-byte code 0x81,0x54 is 0x0000815F.

• size_t table_sz

Example

os_str_conv::mapping mapping[] = {
      {0x0000005C,0x0000005C},
      {0x0000007E,0x0000007E},
      {0x0000815F,0x0000FF3C},
      {0x00008160,0x0000FF5E},
      {0x00008161,0x00002225},
      {0x0000817C,0x0000FF0D},
      {0x00008191,0x0000FFE0},
      {0x00008192,0x0000FFE1},
      {0x000081CA,0x0000FFE2},
};
void func(char* input,char* output) {
            ...
                                                            os_str_conv sjis_uni(os_str_conv::SJIS,os_str_conv::UNICODE);
      sjis_uni.change_mapping(mapping,sizeof(
            mapping)/sizeof(mapping[0]));
      sjis_uni.convert(output,input);
            ... 
}

Byte Order

encode_type convert(char* dest, const char* src); 
encode_type convert(os_unsigned_int16* dest, const char* src); 
encode_type convert(char* dest, const os_unsigned_int16* 

Overhead

• Some memory is consumed by the sorted override map.
• Some time is consumed when sorting the map at change_mapping time.
• Some time is consumed when converting characters because of the additional lookup.

Autodetect only detects SJIS, JIS, and EUC. Do not feed the autodetector Unicode or UTF-8 strings.

The EUC<->Unicode converter only works for characters in the SJIS set. While this might sound perverse, it is reasonable for actual applications, since characters outside the SJIS set are extremely rare.

Users should be aware that the 0 to 127 range of single-byte SJIS characters is not ASCII, even though the characters look like ASCII. This range is known as JIS-Roman. Specifically, the characters {'\', '~' , '|'} have different meanings. The practical significance is that the map of characters [0 to 127] from ASCII->Unicode->SJIS is not an identity.

Performance Notes

JIS conversion requires simple parsing for <Esc> characters. Once stripped of <Esc> characters, you can convert the multibyte sequences to EUC by setting the highest bit.

Updated: 03/31/98 17:04:12