Time needed to generate the 2man - 2man databases (and all subsequent other databases) = 107.3 seconds.
Measured on a Q6600 2.4 Ghz (Win64).
Will do also a measurement for 3man - 2man (also based on no databases at all, so starting from scratch).
Hope i can get some clues if this is slow or extremely slow (don't want to wait for 200 days, to find out that a trivial speed-up is possible).
Will also run a profile do get a better insight whats going on under the surface.
Bert
Damage does not have (yet) the 7 Piece Endgame Databases
Re: Damage does not have (yet) the 7 Piece Endgame Databases
If it could help you Damy needs 90" to generate the 4 pieces endgame database (2 pieces against 2 pieces + 3 pieces against 1 pieces) when using only one core of an intel core 2 duo processeur, 32bits and 2,0GHz. It seems that you have to begin by some work on the algorithm which is indeed a very intersting work!BertTuyt wrote:Time needed to generate the 2man - 2man databases (and all subsequent other databases) = 107.3 seconds.
Measured on a Q6600 2.4 Ghz (Win64).
Will do also a measurement for 3man - 2man (also based on no databases at all, so starting from scratch).
Hope i can get some clues if this is slow or extremely slow (don't want to wait for 200 days, to find out that a trivial speed-up is possible).
Will also run a profile do get a better insight whats going on under the surface.
Bert
What kind of profile do you use ?
Gérard
-
Ed Gilbert
- Posts: 859
- Joined: Sat Apr 28, 2007 14:53
- Real name: Ed Gilbert
- Location: Morristown, NJ USA
- Contact:
Bert,
How does the indexing work in your database builder? I'm assuming that you subdivide by at least the piece counts (nbm, nbk, nwm, nwk). Do you also subdivide by additional constraints, like the rank of the most advanced black and white man, which is how Schaeffer does it?
Also, are you saving the MTC data during the build? This is great stuff and it comes basically for free during the non-conversion passes. Saving it takes almost 0 time, and if you omit the uninteresting positions with MTC values smaller than 10 plies then it takes very little disk space.
What are you getting for an average build speed in positions/sec? A very simple addition to your program is to print out after each subdivision is completed the average build speed for that subdivision. With that info you can easily estimate how long it will take to build everything.
-- Ed
How does the indexing work in your database builder? I'm assuming that you subdivide by at least the piece counts (nbm, nbk, nwm, nwk). Do you also subdivide by additional constraints, like the rank of the most advanced black and white man, which is how Schaeffer does it?
Also, are you saving the MTC data during the build? This is great stuff and it comes basically for free during the non-conversion passes. Saving it takes almost 0 time, and if you omit the uninteresting positions with MTC values smaller than 10 plies then it takes very little disk space.
What are you getting for an average build speed in positions/sec? A very simple addition to your program is to print out after each subdivision is completed the average build speed for that subdivision. With that info you can easily estimate how long it will take to build everything.
-- Ed
-
Ed Gilbert
- Posts: 859
- Joined: Sat Apr 28, 2007 14:53
- Real name: Ed Gilbert
- Location: Morristown, NJ USA
- Contact:
Bert,
Another suggestion: log the WLD counts while you are building, so that you can compare the counts with Gerard's or mine. It is much faster to log them while you are building than to get them afterwards. My counts are non-capture counts, because I do not store capture positions in the finished database, so I suggest you do the same to make the comparisons possible. You will also want to do comparisons against our counts while your build is progressing (as opposed to waiting until you are all done), since one error can spoil all subsequent results and make months of calculations worthless.
-- Ed
Another suggestion: log the WLD counts while you are building, so that you can compare the counts with Gerard's or mine. It is much faster to log them while you are building than to get them afterwards. My counts are non-capture counts, because I do not store capture positions in the finished database, so I suggest you do the same to make the comparisons possible. You will also want to do comparisons against our counts while your build is progressing (as opposed to waiting until you are all done), since one error can spoil all subsequent results and make months of calculations worthless.
-- Ed
Ed, herewith some answers on your questions:
Bert,
How does the indexing work in your database builder? I'm assuming that you subdivide by at least the piece counts (nbm, nbk, nwm, nwk). Do you also subdivide by additional constraints, like the rank of the most advanced black and white man, which is how Schaeffer does it?
Yes i do so
Also, are you saving the MTC data during the build? This is great stuff and it comes basically for free during the non-conversion passes. Saving it takes almost 0 time, and if you omit the uninteresting positions with MTC values smaller than 10 plies then it takes very little disk space.
I dont save this during build (yet)
What are you getting for an average build speed in positions/sec? A very simple addition to your program is to print out after each subdivision is completed the average build speed for that subdivision. With that info you can easily estimate how long it will take to build everything.
Good remark, the 3m - 2m databases (and all required sub-divides took me 1hour 43min, didnt do the calculation for positions/sec., but the feed-back I got from gerard is that I have still work to do
Bert,
How does the indexing work in your database builder? I'm assuming that you subdivide by at least the piece counts (nbm, nbk, nwm, nwk). Do you also subdivide by additional constraints, like the rank of the most advanced black and white man, which is how Schaeffer does it?
Yes i do so
Also, are you saving the MTC data during the build? This is great stuff and it comes basically for free during the non-conversion passes. Saving it takes almost 0 time, and if you omit the uninteresting positions with MTC values smaller than 10 plies then it takes very little disk space.
I dont save this during build (yet)
What are you getting for an average build speed in positions/sec? A very simple addition to your program is to print out after each subdivision is completed the average build speed for that subdivision. With that info you can easily estimate how long it will take to build everything.
Good remark, the 3m - 2m databases (and all required sub-divides took me 1hour 43min, didnt do the calculation for positions/sec., but the feed-back I got from gerard is that I have still work to do
-
Ed Gilbert
- Posts: 859
- Joined: Sat Apr 28, 2007 14:53
- Real name: Ed Gilbert
- Location: Morristown, NJ USA
- Contact:
Bert,
Here's a table of slice sizes which you may find useful. The notation for the slices goes bBwW, so for example slice 1230 means 1 bm, 2 bk, 3 wm, and 0 wk. You can ignore the right hand side of the table. The 'with gaps' numbers are for indexing functions that do not remove interferences between black men and white men. The 'no gaps' positions column include both black-to-move and white-to-move counts, except for symmetric slices where there is no need to compute for both sides to move. The mbytes column is not particularly useful for this indexing by slice, but it gives the buffer space needed to hold the positions in a 2 bits per position format during db builds. I use the same format to print out tables for a different indexing scheme I use during builds.
So if I didn't make any typos with my calculator, you built 1,047,473,160 5 piece positions, 41,971,200 4-piece position, 806,300 3-piece positions, and 8,845 2-piece positions, a total of 1,090,259,505 positions in 6180 seconds which works out to 176,417 positions/sec. This is not fast but it's not unreasonable either. If you can maintain that pace you can build the 7pc db in 65.6 days with one instance running, or 32.8 days with 2 instances running, etc.
-- Ed
Here's a table of slice sizes which you may find useful. The notation for the slices goes bBwW, so for example slice 1230 means 1 bm, 2 bk, 3 wm, and 0 wk. You can ignore the right hand side of the table. The 'with gaps' numbers are for indexing functions that do not remove interferences between black men and white men. The 'no gaps' positions column include both black-to-move and white-to-move counts, except for symmetric slices where there is no need to compute for both sides to move. The mbytes column is not particularly useful for this indexing by slice, but it gives the buffer space needed to hold the positions in a 2 bits per position format during db builds. I use the same format to print out tables for a different indexing scheme I use during builds.
So if I didn't make any typos with my calculator, you built 1,047,473,160 5 piece positions, 41,971,200 4-piece position, 806,300 3-piece positions, and 8,845 2-piece positions, a total of 1,090,259,505 positions in 6180 seconds which works out to 176,417 positions/sec. This is not fast but it's not unreasonable either. If you can maintain that pace you can build the 7pc db in 65.6 days with one instance running, or 32.8 days with 2 instances running, etc.
-- Ed
Code: Select all
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
0101 2,450 0 2,450 0
0110 4,410 0 4,410 0
1010 1,985 0 2,025 0
2 piece totals
8,845 0 8,885 0
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
0201 117,600 0 117,600 0
0210 105,840 0 105,840 0
1101 211,680 0 211,680 0
1110 190,560 0 194,400 0
2001 95,040 0 95,040 0
2010 85,580 0 89,100 0
3 piece totals
806,300 0 813,660 0
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
0202 1,381,800 0 1,381,800 0
0211 4,974,480 1 4,974,480 1
0220 2,233,440 0 2,233,440 0
0301 1,842,400 0 1,842,400 0
0310 1,658,160 0 1,658,160 0
1111 4,478,160 1 4,568,400 1
1120 4,022,260 0 4,187,700 0
1201 4,974,480 1 4,974,480 1
1210 4,478,160 1 4,568,400 1
2020 903,440 0 980,100 0
2101 4,466,880 1 4,466,880 1
2110 4,022,260 0 4,187,700 0
3001 1,333,860 0 1,333,860 0
3010 1,201,420 0 1,277,100 0
4 piece totals
41,971,200 10 42,634,900 10
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
0302 42,375,200 10 42,375,200 10
0311 76,275,360 18 76,275,360 18
0320 34,246,080 8 34,246,080 8
0401 21,187,600 5 21,187,600 5
0410 19,068,840 4 19,068,840 4
1202 114,413,040 27 114,413,040 27
1211 205,995,360 49 210,146,400 50
1220 92,511,980 22 96,317,100 22
1301 76,275,360 18 76,275,360 18
1310 68,665,120 16 70,048,800 16
2102 102,738,240 24 102,738,240 24
2111 185,023,960 44 192,634,200 45
2120 83,116,480 19 90,169,200 21
2201 102,738,240 24 102,738,240 24
2210 92,511,980 22 96,317,100 22
3002 30,678,780 7 30,678,780 7
3011 55,265,320 13 58,746,600 14
3020 24,833,360 5 28,096,200 6
3101 61,357,560 14 61,357,560 14
3110 55,265,320 13 58,746,600 14
4001 13,707,540 3 13,707,540 3
4010 12,350,030 2 13,409,550 3
5 piece totals
1,570,600,750 374 1,609,693,590 383
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
0303 317,814,000 75 317,814,000 75
0312 1,716,195,600 409 1,716,195,600 409
0321 1,541,073,600 367 1,541,073,600 367
0330 460,181,700 109 460,181,700 109
0402 476,721,000 113 476,721,000 113
0411 858,097,800 204 858,097,800 204
0420 385,268,400 91 385,268,400 91
0501 190,688,400 45 190,688,400 45
0510 171,619,560 40 171,619,560 40
1212 2,317,447,800 552 2,364,147,000 563
1221 4,163,039,100 992 4,334,269,500 1,033
1230 1,243,469,700 296 1,321,798,500 315
1302 1,716,195,600 409 1,716,195,600 409
1311 3,089,930,400 736 3,152,196,000 751
1320 1,387,679,700 330 1,444,756,500 344
1401 858,097,800 204 858,097,800 204
1410 772,482,600 184 788,049,000 187
2121 1,870,120,800 445 2,028,807,000 483
2130 1,117,501,200 266 1,264,329,000 301
2202 2,311,610,400 551 2,311,610,400 551
2211 4,163,039,100 992 4,334,269,500 1,033
2220 1,870,120,800 445 2,028,807,000 483
2301 1,541,073,600 367 1,541,073,600 367
2310 1,387,679,700 330 1,444,756,500 344
3030 166,991,800 39 201,356,100 48
3102 1,380,545,100 329 1,380,545,100 329
3111 2,486,939,400 592 2,643,597,000 630
3120 1,117,501,200 266 1,264,329,000 301
3201 1,380,545,100 329 1,380,545,100 329
3210 1,243,469,700 296 1,321,798,500 315
4002 308,419,650 73 308,419,650 73
4011 555,751,350 132 603,429,750 143
4020 249,799,900 59 295,010,100 70
4101 616,839,300 147 616,839,300 147
4110 555,751,350 132 603,429,750 143
5001 109,958,310 26 109,958,310 26
5010 99,098,230 23 109,958,310 26
6 piece totals
46,198,758,750 11,014 47,890,038,930 11,417
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
0403 6,991,908,000 1,667 6,991,908,000 1,667
0412 18,878,151,600 4,500 18,878,151,600 4,500
0421 16,951,809,600 4,041 16,951,809,600 4,041
0430 5,061,998,700 1,206 5,061,998,700 1,206
0502 4,195,144,800 1,000 4,195,144,800 1,000
0511 7,551,260,640 1,800 7,551,260,640 1,800
0520 3,390,361,920 808 3,390,361,920 808
1303 25,170,868,800 6,001 25,170,868,800 6,001
1312 67,978,468,800 16,207 69,348,312,000 16,533
1321 61,057,906,800 14,557 63,569,286,000 15,156
1330 18,237,555,600 4,348 19,386,378,000 4,622
1402 18,878,151,600 4,500 18,878,151,600 4,500
1411 33,989,234,400 8,103 34,674,156,000 8,266
1420 15,264,476,700 3,639 15,892,321,500 3,789
2203 33,903,619,200 8,083 33,903,619,200 8,083
2212 91,586,860,200 21,836 95,353,929,000 22,734
2221 82,285,315,200 19,618 89,267,508,000 21,283
2230 24,585,026,400 5,861 27,815,238,000 6,631
2302 33,903,619,200 8,083 33,903,619,200 8,083
2311 61,057,906,800 14,557 63,569,286,000 15,156
2320 27,428,438,400 6,539 29,755,836,000 7,094
3103 20,247,994,800 4,827 20,247,994,800 4,827
3112 54,712,666,800 13,044 58,159,134,000 13,866
3121 49,170,052,800 11,723 55,630,476,000 13,263
3130 14,695,278,400 3,503 17,719,336,800 4,224
3202 30,371,992,200 7,241 30,371,992,200 7,241
3211 54,712,666,800 13,044 58,159,134,000 13,866
3220 24,585,026,400 5,861 27,815,238,000 6,631
4003 4,523,488,200 1,078 4,523,488,200 1,078
4012 12,226,529,700 2,915 13,275,454,500 3,165
4021 10,991,195,600 2,620 12,980,444,400 3,094
4030 3,285,915,500 783 4,228,478,100 1,008
4102 13,570,464,600 3,235 13,570,464,600 3,235
4111 24,453,059,400 5,830 26,550,909,000 6,330
4120 10,991,195,600 2,620 12,980,444,400 3,094
5002 2,419,082,820 576 2,419,082,820 576
5011 4,360,322,120 1,039 4,838,165,640 1,153
5020 1,960,491,260 467 2,419,082,820 576
7 piece totals
995,625,506,360 237,375 1,049,398,464,840 250,196
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
3131 315,948,485,600 75,327 380,965,741,200 90,829
3140 141,294,366,500 33,687 181,824,558,300 43,350
4040 15,802,050,675 3,767 22,199,510,025 5,292
4121 472,621,410,800 112,681 558,159,109,200 133,075
4130 141,294,366,500 33,687 181,824,558,300 43,350
5021 84,301,124,180 20,098 104,020,561,260 24,800
5030 25,210,705,660 6,010 34,673,520,420 8,266
8 piece totals
1,196,472,509,915 285,261 1,463,667,558,705 348,965
..........no gaps........... .......with gaps...........
slice positions mbytes positions mbytes
----- ----------------- ------ ---------------- ------
4140 1,327,372,256,700 316,470 1,864,758,842,100 444,593
5031 1,058,849,637,720 252,449 1,456,287,857,640 347,206
5040 236,918,517,630 56,485 364,071,964,410 86,801
9 piece totals
2,623,140,412,050 625,405 3,685,118,664,150 878,600
2- 9 totals
4,863,050,574,170 1,159,441 6,247,727,877,660 1,489,574I Ed, Bert
By the way I calculated all these figures with excel, before running the generation process. That way I could also very the figures you give for more than 7 pieces.
What will be really interesting for Bert will be to compare the number of win,loss or draw positions with ours. No problem for me to exchange on this point.
Gérard
The figure above (1,047,473,160) concernes only the 3 pieces against 2 pieces. It does not include 4 pieces against 1 piece positions. The total number of 5 pieces positions is found in your table (1,570,600,750).Ed Gilbert wrote:Bert,
So if I didn't make any typos with my calculator, you built 1,047,473,160 5 piece positions
By the way I calculated all these figures with excel, before running the generation process. That way I could also very the figures you give for more than 7 pieces.
What will be really interesting for Bert will be to compare the number of win,loss or draw positions with ours. No problem for me to exchange on this point.
Gérard
-
Ed Gilbert
- Posts: 859
- Joined: Sat Apr 28, 2007 14:53
- Real name: Ed Gilbert
- Location: Morristown, NJ USA
- Contact:
Hi Gerard,
-- Ed
Yes, I explicitly excluded the 4x1 positions because Bert said he only built 3x2 and successors. But maybe I misunderstood?TAILLE wrote:The figure above (1,047,473,160) concernes only the 3 pieces against 2 pieces. It does not include 4 pieces against 1 piece positions. The totalEd Gilbert wrote:Bert,
So if I didn't make any typos with my calculator, you built 1,047,473,160 5 piece positions
-- Ed
Ed/Gerard,
and all the others having already a 7p database.
With my latest computer purchase (so now 2 64bit quad systems, and 1 "normal 32bit system) i now started also with the 7p generation.
I will work in parallel on generating the databases and improving the speed/efficiency of the algorithms.
The first (simple) database I will generate is the 5k - 2K database.
To check if the result is ok, can some-one give me the WDL counts for this database.
Based on my knowledge the total "unique" piece count is:
comb(50,5) * comb(45,2) = 2,097,572,400
Bert
and all the others having already a 7p database.
With my latest computer purchase (so now 2 64bit quad systems, and 1 "normal 32bit system) i now started also with the 7p generation.
I will work in parallel on generating the databases and improving the speed/efficiency of the algorithms.
The first (simple) database I will generate is the 5k - 2K database.
To check if the result is ok, can some-one give me the WDL counts for this database.
Based on my knowledge the total "unique" piece count is:
comb(50,5) * comb(45,2) = 2,097,572,400
Bert
For this database it is possible to take into account the 2 obvious existing symmetries (though it is not really necessary). In Damy I handle for this data base only 709, 784, 600 positions.BertTuyt wrote:Ed/Gerard,
and all the others having already a 7p database.
With my latest computer purchase (so now 2 64bit quad systems, and 1 "normal 32bit system) i now started also with the 7p generation.
I will work in parallel on generating the databases and improving the speed/efficiency of the algorithms.
The first (simple) database I will generate is the 5k - 2K database.
To check if the result is ok, can some-one give me the WDL counts for this database.
Based on my knowledge the total "unique" piece count is:
comb(50,5) * comb(45,2) = 2,097,572,400
Bert
Gérard
-
Ed Gilbert
- Posts: 859
- Joined: Sat Apr 28, 2007 14:53
- Real name: Ed Gilbert
- Location: Morristown, NJ USA
- Contact:
Bert, here are the counts sans capture positions.
0502b: wins 489226234, draws 59194628, losses 0
0502w: wins 175224, draws 80452496, losses 376519010
If you throw out capture positions during compression as most people do, then it is convenient to log this data during compression.
I expect your new i7 will make quick work of the 7pc db. Good luck!
-- Ed
0502b: wins 489226234, draws 59194628, losses 0
0502w: wins 175224, draws 80452496, losses 376519010
If you throw out capture positions during compression as most people do, then it is convenient to log this data during compression.
I expect your new i7 will make quick work of the 7pc db. Good luck!
-- Ed
Hi Bert,Ed
Just a small question for you.
We all saw that in some major tournments between humans the notion of "advantageaous draw" were used. For computer tournment we of course want to use the same rules as human's but here we have a difficulty : it appears to me that the programmers having some experience in generating endgame database will have a great advantage over those who are only using an existant/available database and so, the subject is a little delicate and I imagine that it exists on this subject a lot of resistances!
In the past I generated the 5 pieces corresponding database and my feeling is that generating such database will only multiply the generating time and the necessary disk space by a factor between 2 and 3. As a consequence it is quite feasible to generate at least up to the 7 pieces database.
What is your feeling for the future ? Do you think that we cannot escape to use also such rule ?
In any case I consider it is a fun job for me and I certainly will tackle the problem before the end of the year.
Gérard
Just a small question for you.
We all saw that in some major tournments between humans the notion of "advantageaous draw" were used. For computer tournment we of course want to use the same rules as human's but here we have a difficulty : it appears to me that the programmers having some experience in generating endgame database will have a great advantage over those who are only using an existant/available database and so, the subject is a little delicate and I imagine that it exists on this subject a lot of resistances!
In the past I generated the 5 pieces corresponding database and my feeling is that generating such database will only multiply the generating time and the necessary disk space by a factor between 2 and 3. As a consequence it is quite feasible to generate at least up to the 7 pieces database.
What is your feeling for the future ? Do you think that we cannot escape to use also such rule ?
In any case I consider it is a fun job for me and I certainly will tackle the problem before the end of the year.
Gérard
