I'm looking into performance improvements with my application. I'm noticing that when my queries are running, the HXTT driver is constantly writing to a temp file HXTT[pid].TMP. Is there a way to have HXTT put this in RAM, or can you tell my how I can avoid all the disk writes?
Thanks,
Mark
|
>Is there a way to have HXTT put this in RAM, or can you tell my how I can
> avoid all the disk writes?
HXTT Text (CSV) will use temporary file for distinct, group by, order by on million level table, and will use temporary file for parsing million level table, to avoid the possible OutoutMemory if load all into RAM. Those temporary files will be recycle automatically. You can set maxCacheSize to a big value, for instance, 16384 , to reduce some possible temporary files.
If you think some sqls is slow, please let us know jdbc url, those sqls, and the row count in those tables, then we will do simulated speed test, then we will try to do some optimizations:)
|
Hi,
I bumped the maxCacheSize to 16384, and it starts writing to the HXTT.tmp file at around 600,000 rows (my dataset is 3 Million rows). It looks like the driver is now less frequently writing to the tmp file (so its faster).. before, it seemed there was a write before every read. The machine that will run this app has at least 2 Gig.. Your suggestion is helping, but is there anyway we can make the cache larger than 16K to make it even faster?
Thanks,
Mark
|
Now the max maxCacheSize value has been adjusted to 65536.
>it starts writing to the HXTT.tmp file at around 600,000 rows (my dataset is 3 Million rows).
Enlarged the ratio. Now If you use 16384 still, it will build temporary file at 4 million rows. If you use 65536, it will build temporary file at 16 million rows.
Please download the latest package.
|
Main
webmaster@hxtt.com