HXTT DBF
Problems with the update of the index files
Marcelo Lopes
2009-08-28 05:50:03.0
Hello,

i'm having some problems with the update of the index files.
I use the .IDX index and it's taking a long time to update all the index after the .DBF be updated.

For example:
The file conh.dbf was modified at 21h 31min 48sec and this DBF has 8 index files. All of them were modified at 21h 32min 08sec.

The same occurred with the file cont.dbf modified at 21h 31min 57sec and his index ID_CONT3.IDX was modified at 21h 32min 08sec.

This is the options we use to get the connection:
Properties prop = new Properties();
prop.put("lockType", "CLIPPER");
prop.put("charSet", "ASCII");
prop.put("delayedClose", "0");
prop.put("maxCacheSize", "16");
prop.put("deletesAreVisible", "true");
Class.forName("com.hxtt.sql.dbf.DBFDriver");
conn = DriverManager.getConnection("jdbc:dbf:/" + getPathOperac(), prop);
conn.setAutoCommit(false);

This is causing some problems because the legacy Clipper system is been using at the same time.
Can you analyse this with certain urgency please?

Thanks in advance
Re:Problems with the update of the index files
HXTT Support
2009-08-28 05:59:26.0
>The same occurred with the file cont.dbf modified at 21h 31min 57sec and
> his index ID_CONT3.IDX was modified at 21h 32min 08sec.
Because it need to lock/load that index file first, Clipper and HXTT DBF need to wait the lock release each other.
Re:Re:Problems with the update of the index files
Marcelo Lopes
2009-08-28 09:57:05.0
I've some questions:
1) On the case of the conh.dbf file I do an INSERT, so I don't need to lock anything, right? Don't should Hxtt take care of this and avoid any concurrency problem? If not, what I need to do?
2) Our logic is the following:
a) My HXTT system and Clipper's system get a number 'X' and search in CONH.DBF file to see if this number exists there, before insert it.
b) If the number is not in CONH.DBF it insert a new record into CONH.DBF. If it finds the number there, then it get another number 'X + 1' and try again.

What I think is happening is that my HXTT system inserts into CONH.DBF (and commit) and before all indexes get updated, the Clipper's process search for the number 'X' at CONH.DBF and (has the index is not updated yet) he can't find it. So, at the end, both processes inserts the same number 'X' into CONH.DBF...
Can it be really happening? What do you think that is the problem?

Thanks in advance
Re:Re:Re:Problems with the update of the index files
Marcelo Lopes
2009-08-31 12:37:56.0
Hello,

do you have any feedback on this problem?

Thanks
Re:Re:Re:Re:Problems with the update of the index files
HXTT Support
2009-08-31 17:30:35.0
Sorry for missed that thread.

>1) On the case of the conh.dbf file I do an INSERT, so I don't need to lock
> anything, right? Don't should Hxtt take care of this and avoid any concurrency
> problem? If not, what I need to do?
Yeah. lock/unlock automatically.

>2) Our logic is the following:
>a) My HXTT system and Clipper's system get a number 'X' and search in CONH.DBF
> file to see if this number exists there, before insert it.
>b) If the number is not in CONH.DBF it insert a new record into CONH.DBF. If it
> finds the number there, then it get another number 'X + 1' and try again.
Clipper supports auto increment field, mabye you can use it.

>What I think is happening is that my HXTT system inserts into CONH.DBF (and
>commit) and before all indexes get updated, the Clipper's process search for the number 'X' at CONH.DBF and (has the index is not updated yet) he can't find
>it. So, at the end, both processes inserts the same number 'X' into CONH.DBF...
>Can it be really happening? What do you think that is the problem?
If it's controlled by your code, maybe it wil happen except you lock/unlock that table. If you build unique index limitation on that column, it shouldn't happen or throw error when insert.



Re:Re:Re:Re:Re:Problems with the update of the index files
Marcelo Lopes
2009-09-01 04:59:40.0
ok, but related to the item 2 what can i do to HXTT update the indexes more faster (we made a test and it took 20 seconds)? Is there any way?

We inserted one record in CONH.DBF and the indexes were updated 20 seconds after... It's a long time.

So I'm not shure if an unique index will fix the problem, because the refresh of the index by HXTT takes a long time.
Re:Re:Re:Re:Re:Re:Problems with the update of the index files
HXTT Support
2009-09-02 08:42:59.0
>We inserted one record in CONH.DBF and the indexes were updated 20 seconds
>after... It's a long time.
Copy your CONH.DBF database to another path, then tested it without lockType=Clipper. It should be quickest. Then use lockType=Clipper, it should be qucker still.
HXTT DBF has to wait a lock chance when Clipper is using the same database, so that it is the slowwest.

BTW, what's the file size of your index files?
Re:Re:Re:Re:Re:Re:Re:Problems with the update of the index files
Marcelo Lopes
2009-09-02 11:31:17.0
This are the index files:

ID_CONH1.IDX 1.704.448 bytes
ID_CONH2.IDX 2.058.752 bytes
ID_CONH3.IDX 2.150.400 bytes
ID_CONH4.IDX 1.704.960 bytes
ID_CONH5.IDX 1.979.904 bytes
ID_CONH6.IDX 2.366.976 bytes
ID_CONH7.IDX 5.888.512 bytes
ID_CONH8.IDX 2.447.360 bytes

8 File(s) 20.301.312 bytes
Re:Re:Re:Re:Re:Re:Re:Re:Problems with the update of the index files
Marcelo Lopes
2009-09-03 14:59:04.0
Hello,

Do you have any feedback about this?

Thanks
Re:Re:Re:Re:Re:Re:Re:Re:Problems with the update of the index files
HXTT Support
2009-09-19 07:13:20.0
>Sorry for m8 File(s) 20.301.312 bytes issed that issue.
Maybe it's unbalance tree. You can try "reindex all on ID_CONH" to balance that index file.

>>Copy your CONH.DBF database to another path, then tested it without
>>lockType=Clipper. It should be quickest. Then use lockType=Clipper, it should be
>>qucker still.
>>HXTT DBF has to wait a lock chance when Clipper is using the same database, so
>> that it is the slowwest.
You can do that test to make sure whether lock wait wasted some seconds too.


Google