java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-05 12:13:51 |
I get an OutOfMemoryError when running a utility that reads data from an Access database. I'm using HXTT Access JDBC 3.0 Package Implementation-Version: 1.2.008 on June 01, 2006, and Java 1.5_06, with 128MB of heap memory, and Hibernate 3.1.2 as my ORM layer.
My application selects one set of records from Access, then loops through each record and reads in a set of related records. I get through about 800 of the second set of selects before the error.
I ran a profiler to see what was taking up so much memory, and it mostly comes from the following classes and fields:
com.hxtt.global.n
com.hxtt.global.n.do
com.hxtt.global.n[]
com.hxtt.global.q
com.hxtt.sql.access.z
com.hxtt.sql.access.cl
com.hxtt.b.i
All of these references eventually start at com.hxtt.global.ae
Any help on this would be appreciated.
Thanks!
|
Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-05 19:53:09 |
|
Hibernate need many memory. HXTT Access uses a few memory cache too. You can try java -Xmx256M to see whether your issue has disappeared.
|
Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-06 07:01:55 |
When I ran a memory profiler, the hibernate classes were only taking 3% of the total heap. The com.hxtt.* classes were taking 86% of the heap.
As I ran the test with a profiler, the amount of heap memory steadily increases, up until the point of of the OutOfMemoryError. When I try to increase the memory, that allows me to process some more records, but still causes the OutOfMemoryError. The error just occurs later in the process.
Is there any way that I could reset or clear the cache that HXTT Access uses? Is there a way to disable the cache, so that I don't run out of memory?
|
Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-06 07:11:53 |
>When I try to increase the memory, that allows me to process some more
> records, but still causes the OutOfMemoryError. The error just occurs later
> in the process.
According to your description, there should be a bug which resulted that issue. We will try to recur your issue. What's your sql? Whether that issue happened only when query on a specific table?
|
Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-06 07:34:01 |
|
I tested with DbVisualizer Free 4.3.4 to open a connection, and browsed many tables in more than 30 mdb files, the used memory increased from 42MB to 134MB. After I closed that connection, it decreaed steadily into 37MB, so that it seems all allocated cache has been correctly recycled. Maybe hibernate is holding some open ResultSet for cache purpose so that HXTT Access has to hold those values too. Whether your table has some MEMO(clob) or OLE(blob) column? What's the record count of your table? I will dig more to recur your issue in hibernate.
|
Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-06 07:46:59 |
BTW, what's your SessionFactory Configuration? At least, please let us know the values of hibernate.connection.pool_size and hibernate.jdbc.fetch_size .
|
Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-06 20:13:30 |
|
BTW, what's your jdbc url?
|
Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-07 06:13:47 |
These are my hibernate settings:
false
com.hxtt.sql.access.AccessDriver
jdbc:Access:///c:/tpc/ipartner/ipartner-support/test/PChef.mdb
20
org.hibernate.dialect.DB2Dialect
true
20
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-07 06:14:33 |
These are my hibernate settings:
<hibernate-configuration>
<session-factory>
<property name="hibernate.bytecode.use_reflection_optimizer">false</property>
<property name="hibernate.connection.driver_class">com.hxtt.sql.access.AccessDriver</property>
<property name="hibernate.connection.url">jdbc:Access:///c:/tpc/ipartner/ipartner-support/test/PChef.mdb</property>
<property name="hibernate.connection.username"></property>
<property name="hibernate.connection.pool_size">20</property>
<property name="hibernate.dialect">org.hibernate.dialect.DB2Dialect</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.jdbc.fetch_size">20</property>
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-07 06:21:26 |
|
What's the file size of PChef.mdb? Whether are you using some similar urls like jdbc:access:///c:/tpc/ipartner/ipartner-support/test/PChef.mdb in hibernate too?
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-07 06:23:02 |
The PChef.mdb is 33MB. All of my url's to access are of that format.
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-07 06:33:41 |
>org.hibernate.dialect.DB2Dialect
Youshould use com.hxtt.support.hibernate.HxttAccessDialect
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-07 06:34:56 |
|
You can find HxttAccessDialect at here
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-07 06:56:36 |
I installed the HxttAccessDialect, and ran my code with that. I got the same OutOfMemoryError as before, after the same number of records.
Can you explain to me a little more how the Driver's caching works, and if there is a way to turn that off?
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-07 07:46:16 |
|
maxCacheSize indicates the max memory utilization for per table on automatic temporary index or matched result cache. You can use 16~16384 kilo bytes. Default: 1024. you can use maxCacheSiz=16 as connection property. But you shouldn't use that solution because we have adjusted the default access page cache size from 1200(6MB to 200(1MB) for every mdb file, and the constant value pool size from 10000 to 1000. Please download the latest package, and try again, to see whether HXTT Access consumes smaller memory.
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-07 08:58:14 |
|
BTW, delayedClose indicates the delayed seconds for close transaction. That option is used to avoid frequent close/open table operations for following sqls. You can use 0~120 seconds. Default: 5. If you use delayedClose=1, HXTT Access will quicken to release those holding idle tables too.
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
Milo Todorovich |
2006-06-15 14:05:29 |
I've downloaded the update, and I've added the maxCacheSize=16 as a connection property. The memory has improved a little, but I still get an OutOfMemoryError.
Now, it just happens later in the process.
Running my program through a memory profiler shows that over 85% of the heap is occupied by objects from the com.hxtt.* packages.
Is there anyway to completely disable any caching?
Thanks!
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-18 02:47:57 |
I tested it in NetBeans 5.0 with hibernate3.1.2. That NetBeans Profiler hasn't showed such a big 85% of the heap, and those occupied memory will be released soon too. Anyway, we have adjusted the cache level from database level to table level, so that it will use a smaller size, and release those unused data in a shorted time. Please download the latest v1.2.019 package, and try again.
BTW, if possible, I wish to see your test case so that we can used simulated data to recur your issue in the older package. You can email it to webmaster@hxtt.com or webmaster@hxtt.net .
|
Re:Re:Re:Re:Re:Re:Re:Re:java.lang.OutOfMemoryError when trying to read data from an access database. |
HXTT Support |
2006-06-18 08:23:16 |
>>My application selects one set of records from Access, then loops through each record and reads in a set of related records. I get through about 800 of the second set of selects before the error.
It seems that your application uses the same session to access the data before OutOfMemoryError exception.
Can you send us your Hibernate config file and the O/R map file (*.nbm) file to us?
|
Main
webmaster@hxtt.com