8 pieces endgame database

Discussion about development of draughts in the time of computer and Internet.
Post Reply
MichelG
Posts: 244
Joined: Sun Dec 28, 2003 20:24
Contact:

Re: 8 pieces endgame database

Post by MichelG »

BertTuyt wrote:Michel,

maybe this question was already covered or answered.
Ed uses multiple instances of the DB-gen program to generate all DB's.
In your case did you use a parallel version of your Generator , or did you had a similar approach as Ed?
If you used a parallel DB-Gen, how does it scale with the number of cores.
As the core number will explode the next years, this would be good news for all of us .

Bert
I think the explanation of dragon's algorithm is somewhere on these pages... Dragon uses a multithreaded approach, unlike Ed's. I think the main advantage is the sharing of ram among the threads.

For the future, more cores would be welcome, but dragon currently does not scale well above 16 threads; at that point the disk I/O will become dominant and dragon would need updated algorithms to better deal with the IO.

Apart from the IO, scaling is pretty much linear though on 6 cores.

Michel
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

MichelG wrote:Both 4x4 and 5x3 have finally been build :-)

Last week the memory chips in my computer broke down, but i took out the faulty chips and let it continue with half the memory.

The entire computation took a bit more than 3 months on 6 cores. About 2.5 months was spend on computing, and about 3 weeks spend on delays caused by adjustments to the program to deal with memory failures and lack of disk space.

The counts are identical to Ed's numbers, thus confirming his work. Congratulations :-).

Dragon is now working on computing 6x2. Am i right to assume you did not compute these databases, Ed?

Michel
Hi Michel,

For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
I just begin computing the 6x2db. Did you finish building this last one?
Gérard
Ed Gilbert
Posts: 860
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: 8 pieces endgame database

Post by Ed Gilbert »

For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
Gerard, congrats on finishing all 4x4 and 5x3 with confirming counts.

Did you change from only storing 2 of the 3 WLD values to something else? My size for storing all 3 values of 4x4 and 5x3 is 365GB. I thought in the past that storing less info resulted in a greater reduction in size.

-- Ed
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

Ed Gilbert wrote:
For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
Gerard, congrats on finishing all 4x4 and 5x3 with confirming counts.

Did you change from only storing 2 of the 3 WLD values to something else? My size for storing all 3 values of 4x4 and 5x3 is 365GB. I thought in the past that storing less info resulted in a greater reduction in size.

-- Ed
Hi Ed,

Each Damy db has two entries in order to answer the two following questions :
1) Is the position a winning position ?
2) Is the position a losing position ?
As you see, should you have to prove that a position is a draw, then you need two db access

I chose this approach for two main reasons :
1) Because Damy search algorithm is mtd-f based, depending of the test value I always can chose the good question for the db and I never need two access disk
2) Each time I read a block on the disk, because the basic information is on single bit, I can store in memory more positions.
Gérard
Ed Gilbert
Posts: 860
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: 8 pieces endgame database

Post by Ed Gilbert »

Hi Gerard,

Yes, I understand and I think your idea fits nicely with mtd search. My point was that for smaller dbs I thought you got about a 25% reduction in size (compared to mine) by not storing draws, but for the 4x4 and 5x3 subsets it looks like the difference is much smaller, ~5%.

-- Ed
MichelG
Posts: 244
Joined: Sun Dec 28, 2003 20:24
Contact:

Re: 8 pieces endgame database

Post by MichelG »

TAILLE wrote: For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
I just begin computing the 6x2db. Did you finish building this last one?
Congratulations on that :-)

I build 6 pawns against 2 kings, but for now, i stopped there as i think i won't include the others in the list of actively used database. Besides that my computer still has broken ram chips and i need the CPU power to do some other stuff.

Michel
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

Ed Gilbert wrote:Hi Gerard,

Yes, I understand and I think your idea fits nicely with mtd search. My point was that for smaller dbs I thought you got about a 25% reduction in size (compared to mine) by not storing draws, but for the 4x4 and 5x3 subsets it looks like the difference is much smaller, ~5%.

-- Ed
The main explanation is certainly the fact that I count twice the db 0044, 1133, 2222, 3333, 4444. By removing this doublon I can gain 20,2Gb => total size = 326Gb.
In addition I give up some optimisations I made because it was not a good compromise between the size of the db and the number of access to the db. It will be clear with an example:
Let's take the 2132db with the following counters win=1 949 052 171, draw=59 850 673 173, loss=84 285 824 514
With my previous implementation and in order to optimize the size of the db I would here have build one db for the winning positions and one db for a draw positions => 1 949 052 171 + 59 850 673 173 positions stored in the db
The drawback is the following : if I reach a position in this db with an mtd-f test value = -100 I need two db access if the position is a losing position.
To avoid this I decided to not optimise the disk size in order to assure in all cases only one access disk. To achieve this the two db I really built in that case are the following :
1) the winning positions = 1 949 052 171 stored
2) the non-losing positions = 1 949 052 171 + 59 850 673 173 stored
With these two db you can see that I store twice the 1 949 052 171 winning positions but I am sure to have only one access disk to satisfy the need of the mtd-f algorithm

I am now quite happy with the result because
1)I can really reduce the size of the db to 326db
2)I always assure only one access disk to satisfy the mtd-f algorithm need
3)I certainly can quite double the number of positions stored in memory by storing only the relevant information
Gérard
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

MichelG wrote:
TAILLE wrote: For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
I just begin computing the 6x2db. Did you finish building this last one?
Congratulations on that :-)

I build 6 pawns against 2 kings, but for now, i stopped there as i think i won't include the others in the list of actively used database. Besides that my computer still has broken ram chips and i need the CPU power to do some other stuff.

Michel
Hi Michel,

Oops, for the very first time I found a discrepancy between our counters. It concerns the 1502 and 0215 db :
you found :
1502 20.266.071.917 829.455.221 0 0215 6.647.587 1.168.805.815 12.370.373.122
and I found :
1502 20.220.883.259 874.643.879 0 0215 6.647.587 1.195.248.328 12.343.930.609

For 3 days I have tried to find an explanation and of course I regenerate these db and the previous one in order to eliminate any hardware bug. No idea for the moment.

Is it possible for you to verify your counters ?
Gérard
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

TAILLE wrote:
MichelG wrote:
TAILLE wrote: For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
I just begin computing the 6x2db. Did you finish building this last one?
Congratulations on that :-)

I build 6 pawns against 2 kings, but for now, i stopped there as i think i won't include the others in the list of actively used database. Besides that my computer still has broken ram chips and i need the CPU power to do some other stuff.

Michel
Hi Michel,

Oops, for the very first time I found a discrepancy between our counters. It concerns the 1502 and 0215 db :
you found :
1502 20.266.071.917 829.455.221 0 0215 6.647.587 1.168.805.815 12.370.373.122
and I found :
1502 20.220.883.259 874.643.879 0 0215 6.647.587 1.195.248.328 12.343.930.609

For 3 days I have tried to find an explanation and of course I regenerate these db and the previous one in order to eliminate any hardware bug. No idea for the moment.

Is it possible for you to verify your counters ?
Sorry Michel for my last post; I finally fixed the bug (initialisation default for the specific 6x2db)
Gérard
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

Hi Michel,

All my counters are stricly identical to yours including now those concerning 1502 and 0215 db. 2402 and 0224db are now under generation.
Knowing that you generated the 6x2db till 6men x 2kings, if you are interesting by comparing some more counters you can simply update your site and I will do the corresponding comparaisons
Gérard
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

MichelG wrote:
TAILLE wrote: For Damy 5x3 db is now built; for the time being :
4x4 db : 100%
5x3 db : 100%
and all counters are strictly identical to yours and Ed's.
It took Damy 4 months for that computation (far more than for Dragon!) and the total disk size for these two db is 346Gb
I just begin computing the 6x2db. Did you finish building this last one?
Congratulations on that :-)

I build 6 pawns against 2 kings, but for now, i stopped there as i think i won't include the others in the list of actively used database. Besides that my computer still has broken ram chips and i need the CPU power to do some other stuff.

Michel
Hi Michel
Now I have also reached the 6002 and 0260 db with the following counters (non capture positions):
6002:
+: 1.207.376.766
=: 2.952.784.704
-: 138.362.310

0260:
+: 34.816.786.364
=: 843.383.259
-: 132.602.146

Do you have the same figures?
Gérard
MichelG
Posts: 244
Joined: Sun Dec 28, 2003 20:24
Contact:

Re: 8 pieces endgame database

Post by MichelG »

I am afraid that the numbers don't match. These are my numbers:

Code: Select all

0602    3.295.118.294      100.877.984                0   0206          347.582      181.471.448    2.402.181.023 
1502   20.266.071.917      829.455.221                0   0215        6.647.587    1.168.805.815   12.370.373.122 
2402   51.684.866.510    3.084.234.365                5   0224       43.434.474    3.154.738.399   26.288.659.968 
3302   68.951.466.851    7.047.909.521              377   0233      131.180.730    4.896.968.321   29.083.616.608 
4202   48.227.431.356   11.165.341.953            7.862   0242      198.737.339    5.564.162.649   16.352.151.924 
5102   14.321.284.670   10.439.942.305          274.408   0251      146.971.295    4.266.615.975    3.203.388.571 
6002    1.201.070.446    2.959.001.787      138.451.547   0260      112.764.503      844.243.886      131.674.829 
Michel
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

MichelG wrote:I am afraid that the numbers don't match. These are my numbers:

Code: Select all

0602    3.295.118.294      100.877.984                0   0206          347.582      181.471.448    2.402.181.023 
1502   20.266.071.917      829.455.221                0   0215        6.647.587    1.168.805.815   12.370.373.122 
2402   51.684.866.510    3.084.234.365                5   0224       43.434.474    3.154.738.399   26.288.659.968 
3302   68.951.466.851    7.047.909.521              377   0233      131.180.730    4.896.968.321   29.083.616.608 
4202   48.227.431.356   11.165.341.953            7.862   0242      198.737.339    5.564.162.649   16.352.151.924 
5102   14.321.284.670   10.439.942.305          274.408   0251      146.971.295    4.266.615.975    3.203.388.571 
6002    1.201.070.446    2.959.001.787      138.451.547   0260      112.764.503      844.243.886      131.674.829 
Michel
Thank you for having give us these figures which allow to gain a lot of time. All the counters are identical to mine except for the last db (6002 and 0260). As you know, in order to gain time I run my verification test only in doubts. Here I ran it on the 6002 db and I discovered an inconsistency (=> the bug is in my side!). The 5102 and 0251 seem correct according to the counters but, to be sure, I have just begin to run my verification test on it. For the moment everything is OK.
Gérard
TAILLE
Posts: 968
Joined: Thu Apr 26, 2007 18:51
Location: FRANCE

Re: 8 pieces endgame database

Post by TAILLE »

TAILLE wrote:
MichelG wrote:I am afraid that the numbers don't match. These are my numbers:

Code: Select all

0602    3.295.118.294      100.877.984                0   0206          347.582      181.471.448    2.402.181.023 
1502   20.266.071.917      829.455.221                0   0215        6.647.587    1.168.805.815   12.370.373.122 
2402   51.684.866.510    3.084.234.365                5   0224       43.434.474    3.154.738.399   26.288.659.968 
3302   68.951.466.851    7.047.909.521              377   0233      131.180.730    4.896.968.321   29.083.616.608 
4202   48.227.431.356   11.165.341.953            7.862   0242      198.737.339    5.564.162.649   16.352.151.924 
5102   14.321.284.670   10.439.942.305          274.408   0251      146.971.295    4.266.615.975    3.203.388.571 
6002    1.201.070.446    2.959.001.787      138.451.547   0260      112.764.503      844.243.886      131.674.829 
Michel
Thank you for having give us these figures which allow to gain a lot of time. All the counters are identical to mine except for the last db (6002 and 0260). As you know, in order to gain time I run my verification test only in doubts. Here I ran it on the 6002 db and I discovered an inconsistency (=> the bug is in my side!). The 5102 and 0251 seem correct according to the counters but, to be sure, I have just begin to run my verification test on it. For the moment everything is OK.
My bug is now fixed. The problem occured when the criterias for handling the mutithread environment was based on 6 men which is the case for the 6002 db. Initiallly I build the algorithm by ignoring the 6x1 and 6x2 db and I forgot a small detail when I try and adapt my algorithm for these db.

6011 and 1160 db are now under generation
Gérard
BertTuyt
Posts: 1592
Joined: Wed Sep 01, 2004 19:42

Re: 8 pieces endgame database

Post by BertTuyt »

I'm planning to buy next year a dual-processor system with a Intel Sandy Bridge or Ivy Bridge (most likely with 8-cores each). Maybe the Ivy Bridge is not available in 2012 (basically this is the tick from tick-tock, or otherwise stated the die-shrink to 22nm).
I also assume at that time as these processors have also more memory-channels, that such a motherboard could containing > 48 GByte Memory.
Assuming a clock speed around 3 - 3.4 Ghz, would the 8P DB generation (also backed up by a 500G SD) something which could be done in less then 1 month, or do you think this is still unrealistic?

Bert
Post Reply