User Guide : Encoding Reference : Determining Which Connector to Use : Flat File Connectors
Share this page             
Flat File Connectors
Unicode or ASCII
When working with text files with special encoding characters, use a Unicode connector instead of an ASCII connector. Even if you do not have Unicode data, you have the option of setting the encoding property in the Unicode connectors. The encoding property options include non-Unicode and Unicode formats.
Character Encoding
Setting the correct character encoding is important. If you are not sure which encoding to use, select UTF-8. It is the most common Unicode scheme and handles most issues. If needed, you can also use ASCII encoding.
Unicode Source Data
When connecting a Unicode source data file with the Unicode connector, the encoding scheme should automatically change from OEM to the proper character set after the source file is specified. For instance, your file may specify UCS-2 when used as a source file. However, if the file does not having leading byte characters as described below, you may have to set the encoding scheme (to UCS-2 or UTF-16, for example).
Unicode Target Data
When outputting to a Unicode target data file with the Unicode connector, the encoding scheme defaults to OEM. You should change the encoding to reflect the correct Unicode character set such as UTF-8 or UTF-16. Note that choosing UCS-2 in the Map window provides the same results as choosing UTF-16. For the purposes of data exchange, UTF-16 and UCS-2 are the same. Both are 16-bit and have the same code unit representation.
Troubleshooting the Replacement Character Issue
Choosing OEM can result in an output file containing replacement characters, such as "?" or "." in place of characters that were not transformed to the OEM code page.
To fix the issue, change the encoding to the correct Unicode character set of your target file.
If some characters still fail to display correctly, do the following:
Check that the font and script are set correctly.
Consider using Windows Notepad or a hex viewer to view the data directly. This helps you to distinguish between transformation and display problems.
XML Connector
The XML connector encoding is determined by the encoding property in the XML file.