Main   Products   Offshore Outsourcing   Customers   Partners   ContactUs  
JDBC Databases
  HXTT Access v7.1
  HXTT Cobol v5.0
  HXTT DBF v7.1
  HXTT Excel v6.1
  HXTT Json v1.0
  HXTT Paradox v7.1
  HXTT PDF v2.0
  HXTT Text(CSV) v7.1
  Buy Now
  HXTT Word v1.1
  HXTT XML v4.0
Offshore Outsourcing
Free Resources
  Firewall Tunneling
  Search Indexing Robot
  Conditional Compilation
  Password Recovery for MS Access
  Password Recovery for Corel Paradox
  Checksum Tool for MD5
  Character Set Converter
  Pyramid - Poker of ZYH
Heng Xing Tian Tai Lab of Xi'an City (abbr, HXTT)

UTF BOM marker
2006-09-17 02:51:06

UTF files sometimes contain a marker at the begining of the file called a BOM marker. It's maximum 4 bytes long and simply identifies the type of UTF file. Apparently the HXTT CSV driver doesn't take that BOM into account when opening the file leading to problems when performing queries. Could you please add support for these types of UTF files? I have added a piece of code on how to do the identification and skipping for your convenience. The code simply identifies the BOM marker and skips the BOM if one is found, it then returns an InputStreamReader that is positioned just after the BOM.

Thanks and best regards,

* Read-ahead four bytes and check for BOM marks. Extra bytes are unread back
* to the stream, only BOM bytes are skipped.
* @param oInputStream the original input stream.
* @param sEncoding the given encoding.
* @return InputStreamReader an input stream reader after scanning for BOM and special UTF encoding schemes.
* The encoding passed to the input srteam reader is adjusted if necessary.
public static InputStreamReader checkBOMAndAdjustEncoding(InputStream oInputStream, String sEncoding) throws IOException {
final int BOM_SIZE = 4;
PushbackInputStream oPushbackStream = null;
byte aBOM[];
int nBytesRead;
int nBytesToUnread;

oPushbackStream = new PushbackInputStream(oInputStream, BOM_SIZE);
aBOM = new byte[BOM_SIZE];
nBytesRead =, 0, aBOM.length);
// Check for BOM
if ((aBOM[0] == (byte) 0xEF) && (aBOM[1] == (byte) 0xBB) && (aBOM[2] == (byte) 0xBF)) {
sEncoding = "UTF-8";
nBytesToUnread = nBytesRead - 3;
else if ((aBOM[0] == (byte) 0xFE) && (aBOM[1] == (byte) 0xFF)) {
sEncoding = "UTF-16BE";
nBytesToUnread = nBytesRead - 2;
else if ((aBOM[0] == (byte) 0xFF) && (aBOM[1] == (byte) 0xFE)) {
sEncoding = "UTF-16LE";
nBytesToUnread = nBytesRead - 2;
else if ((aBOM[0] == (byte) 0x00) && (aBOM[1] == (byte) 0x00) &&
(aBOM[2] == (byte) 0xFE) && (aBOM[3] == (byte) 0xFF)) {
sEncoding = "UTF-32BE";
nBytesToUnread = nBytesRead - 4;
else if ((aBOM[0] == (byte) 0xFF) && (aBOM[1] == (byte)0xFE) &&
(aBOM[2] == (byte) 0x00) && (aBOM[3] == (byte) 0x00)) {
sEncoding = "UTF-32LE";
nBytesToUnread = nBytesRead - 4;
else {
// Unicode BOM mark not found, unead all bytes.
nBytesToUnread = nBytesRead;
if (nBytesToUnread > 0) oPushbackStream.unread(aBOM, (nBytesRead - nBytesToUnread), nBytesToUnread);
else if (nBytesToUnread < -1) oPushbackStream.unread(aBOM, 0, 0);
// Use calculated encoding.
return new InputStreamReader(oPushbackStream, sEncoding);
Re:UTF BOM marker
HXTT Support
2006-09-17 03:02:51
Please unload a data file sample:)
ftp site:
ftp user:
ftp password: (empty)
login mode: normal (not anonymous)
ftp port:21
upload directory: incoming
After upload, you can't see that upload file, but it has been upload.

then notify us through .

BTW, what's your data file type(text, csv, psv, or tsv)?
Re:Re:UTF BOM marker
2006-09-17 03:09:42
File was uploaded. The file inn UTF-8 format with BOM signature, the contents is tab delimited without quotes around the strings.

Re:Re:Re:UTF BOM marker
HXTT Support
2006-09-17 03:18:28
Thanks. If you have UTF-16BE sample or other sample, you can upload it too:)
We will complement that support soon:)
Re:Re:Re:Re:UTF BOM marker
2006-09-17 03:24:27
Thanks again for the quick response, like I said in the mail I sent to, you can download a free version of emeditor ( to play around with these signatures. I think I covered all of the possibilities in my code snippet.

Best regards,
Re:Re:Re:Re:Re:UTF BOM marker
HXTT Support
2006-09-17 04:16:15
Just a hint, there's a bug in your function for UTF-16LE and UTF-32LE:
else if ((aBOM[0] == (byte) 0xFF) && (aBOM[1] == (byte) 0xFE)) {
sEncoding = "UTF-16LE";
nBytesToUnread = nBytesRead - 2;
else if ((aBOM[0] == (byte) 0xFF) && (aBOM[1] == (byte)0xFE) &&
(aBOM[2] == (byte) 0x00) && (aBOM[3] == (byte) 0x00)) {//Won't be reached forever
sEncoding = "UTF-32LE";
nBytesToUnread = nBytesRead - 4;

It should be:
else if ((aBOM[0] == (byte) 0xFF) && (aBOM[1] == (byte)0xFE) &&
(aBOM[2] == (byte) 0x00) && (aBOM[3] == (byte) 0x00)) {//Now it can be reached forever
sEncoding = "UTF-32LE";
nBytesToUnread = nBytesRead - 4;
else if ((aBOM[0] == (byte) 0xFF) && (aBOM[1] == (byte) 0xFE)) {
sEncoding = "UTF-16LE";
nBytesToUnread = nBytesRead - 2;
Re:Re:Re:Re:Re:Re:UTF BOM marker
HXTT Support
2006-09-17 06:36:04
v3.0.019 supports to detect automatically and utilize byte-order marker (BOM) in UTF file for CSV, TSV, and PSV.
You can use:

select * from UTF8_BOM_SAMPLE.txt
select * from UTF8_BOM_SAMPLE

Please download the latest JDBC30 package. The JDBC20, and JDBC12 packages will be availabe in 2 hours.
Re:Re:Re:Re:Re:Re:Re:UTF BOM marker
HXTT Support
2006-09-17 06:38:11
The preferable url is:
Re:Re:Re:Re:Re:Re:Re:Re:UTF BOM marker
2006-09-17 15:00:03
Thanks for the hint, this was indeed a bug:)

Always nice to help each other out...


Search Key   Search by Last 50 Questions


Copyright © 2003-2019 Heng Xing Tian Tai Lab of Xi'an City. | All Rights Reserved. | Privacy | Legal | Sitemap