Main   Products   Offshore Outsourcing   Customers   Partners   ContactUs  
JDBC Databases
  HXTT Access v7.0
  HXTT Cobol v4.0
  HXTT DBF v7.0
  HXTT Excel v6.0
 
  Buy Now
  Support
  Download
  Document
  FAQ
  HXTT Paradox v7.0
  HXTT PDF v1.1
  HXTT Text(CSV) v7.0
  HXTT Word v1.0
  HXTT XML v3.0
Offshore Outsourcing
Free Resources
  Firewall Tunneling
  Search Indexing Robot
  Conditional Compilation
  Password Recovery for MS Access
  Password Recovery for Corel Paradox
  Checksum Tool for MD5
  Character Set Converter
  Pyramid - Poker of ZYH
   
   
   
Heng Xing Tian Tai Lab of Xi'an City (abbr, HXTT)

HXTT Excel
OutOfMemory error
Haiyuan Wang
2011-11-10 11:42:27.0
Hi,

We are currently evaluating the new version of Excel driver. When we tried to load a worksheet that contains more than 1 million of rows of data in Excel 2007 format, it will cause the server throw OutOfMemory error. We haven't seen this issue with Excel 2003 because the maximum number of rows allowed is 65536.

Is there any driver property that can be used to stream the result and not loaded into memory at once?

Thanks,
Haiyuan Wang
Re:OutOfMemory error
HXTT Support
2011-11-10 17:47:57.0
Excel 2007 is using XML format, so that it need more memory. You can email us your sample, and maybe we can try to find a solution.
Re:Re:OutOfMemory error
Haiyuan Wang
2011-11-14 08:36:37.0
Hi,

My sample xlsx file is having 7 columns, some columns are of type String, some columns are of type Number, you can manually create the sample worksheet just to have 1M rows of data in the worksheet. I don't know how to send you such big file over the email.

Thanks!
Re:Re:Re:OutOfMemory error
HXTT Support
2012-11-01 07:26:57.0
Please download the latest package.
  • 2012-11-01 v4.1.087 optimizes load random access speed for XLSX sheet with 1,048,576 rows

  • 2012-11-01 v4.1.087 uses smaller memory to load 1,048,576 rows of XLSX sheet to avoid OutOfMemory Exception.

  • 2012-11-01 v4.1.087 provides thread pool for XLSX parse and load.
  • Re:Re:Re:Re:OutOfMemory error
    Haiyuan Wang
    2012-11-01 09:46:35.0
    I downloaded the latest package and tested with our 1M rows of Excel data. During testing, noticed the following issues:
    (1) Got several NPEs during loading of the data
    Exception in thread "PooledThread-2" java.lang.NullPointerException
    at com.hxtt.sql.excel.ca.a(Unknown Source)
    at com.hxtt.sql.excel.c1.a(Unknown Source)
    at com.hxtt.sql.excel.c1.b(Unknown Source)
    at com.hxtt.sql.excel.c1.run(Unknown Source)
    at com.hxtt.global.c.run(Unknown Source)
    2012-11-01 13:28:41,514 Connecting with connect string: jdbc:excel:///my1Mdata.xlsx
    java.lang.NullPointerException
    at com.hxtt.b.bi.a(Unknown Source)
    at com.hxtt.b.as.j(Unknown Source)
    at com.hxtt.b.as.run(Unknown Source)
    at com.hxtt.global.c.run(Unknown Source)
    Exception in thread "PooledThread-1" java.lang.NullPointerException
    at com.hxtt.sql.excel.ca.a(Unknown Source)
    at com.hxtt.sql.excel.c1.a(Unknown Source)
    at com.hxtt.sql.excel.c1.b(Unknown Source)
    at com.hxtt.sql.excel.c1.run(Unknown Source)
    at com.hxtt.global.c.run(Unknown Source)
    (2) Sometimes not able to read data because of the following Error:
    Caused by: java.sql.SQLException: Failed to go Rowid 3841 of table 1Mdata:There're only row 4074 but you wish to get row 4075.
    at com.hxtt.global.SQLState.SQLException(Unknown Source)
    at com.hxtt.sql.excel.b0.a(Unknown Source)
    at com.hxtt.sql.du.a(Unknown Source)
    at com.hxtt.sql.cl.a(Unknown Source)
    at com.hxtt.sql.cl.a(Unknown Source)
    at com.hxtt.sql.af.x(Unknown Source)
    at com.hxtt.sql.d2.a(Unknown Source)
    at com.hxtt.sql.d2.a(Unknown Source)
    at com.hxtt.sql.dg.new(Unknown Source)
    at com.hxtt.sql.dg.next(Unknown Source)
    Re:Re:Re:Re:Re:OutOfMemory error
    HXTT Support
    2012-11-01 22:00:38.0
    Please redownload it.
    Re:Re:Re:Re:Re:Re:OutOfMemory error
    HXTT Support
    2012-11-02 04:56:25.0
    v4.1.088 provides independent thread for uncompress big XML file in XLSX. For smalll document, HXTT Excel engine will unzip first that Sheetn.xml from XLSX file, check secondly XML syntax, load thirdly all cell information, and allow SQL random access all rows at last. For big sheet, all operations (unzip file, XML parse, Excel sheet load, sql access) are concurrent now, so that it should be the fastest engine with a smaller memory requirement. Default 32000 rows are cached in memory, and you can ajust the cached row number through maxCacheSize connection property. It will be available after 2 hours.
    Re:Re:Re:Re:Re:Re:Re:OutOfMemory error
    Haiyuan Wang
    2012-11-07 12:22:44.0
    Has the package been tested in your side? Still got the same NPE and same error when tested with the latest package downloaded from your website.
    Re:Re:Re:Re:Re:Re:Re:Re:OutOfMemory error
    HXTT Support
    2012-11-07 16:45:34.0
    /* 4.1.088 is released on November 02, 2012 */ which has NPE still.
    /* 4.1.089 is released on November 03, 2012 */ which should disappear NPE.
    Re:Re:Re:Re:Re:Re:Re:Re:OutOfMemory error
    Haiyuan Wang
    2012-11-08 06:29:39.0
    Yes, I downloaded the most recent version and still got the NPE. This is from the MANIFEST.MF file in your package:

    Manifest-Version: 1.0
    Created-By: HXTT Version Robot
    Main-Class: com.hxtt.sql.admin.Admin

    Name: com/hxtt/sql/excel/
    Implementation-Vendor: Hongxin Technology & Trade Ltd.
    Implementation-Title: com.hxtt.sql.excel
    Implementation-Version: 4.1.089 on November 03, 2012
    Specification-Vendor: Hongxin Technology & Trade Ltd.
    Specification-Title: HXTT Excel JDBC 4.0 Package
    Implementation-URL: http://www.hxtt.com/excel.html
    Specification-Version: 4.1.089 on November 03, 2012
    Comment: JDBC 4.0 Package for MS Excel

    Name: com/hxtt/sql/admin/
    Implementation-Vendor: Hongxin Technology & Trade Ltd.
    Implementation-Title: com.hxtt.sql.admin
    Implementation-Version: 0.5 on November 03, 2012
    Specification-Vendor: Hongxin Technology & Trade Ltd.
    Specification-Title: HXTT Database Admin
    Implementation-URL: http://www.hxtt.com/excel/dbadmin.html
    Comment: HXTT Database Admin
    Specification-Version: 0.5 on November 03, 2012
    Re:Re:Re:Re:Re:Re:Re:Re:OutOfMemory error
    HXTT Support
    2012-11-09 07:13:08.0
    Please download and try the 4.1.090 now. AES 256 encrypt/decrypt function has been provided now.
    Re:Re:Re:Re:Re:Re:Re:Re:OutOfMemory error
    HXTT Support
    2012-11-11 07:15:20.0
    Please download the latest v4.1.091, which should have solved the NPE issue.

    Search Key   Search by Last 50 Questions




    Google
     

    Email: webmaster@hxtt.com
    Copyright © 2003-2019 Heng Xing Tian Tai Lab of Xi'an City. | All Rights Reserved. | Privacy | Legal | Sitemap