8 pieces endgame database

[TAILLE]

Hi Michel,

Good luck to you Gerard!

I'll race you , I just started the same computation last week with the computation of 5v3 on a quadcore q6700, and in a couple of days my other computer will join in with 6 cores on 4v4.

It's a bit difficult to compare my database builder to yours, since I don't know how much time it took for the 7 piece endings.

But i estimate it would be about 3 days on 6 cores. 5Kv3K took about 6.2 hours. As you, i don't exploit the symmetries.

Why don't you use all the cores in your system? I use all 12 on my system. If the computer slows down, it is mostly because of the hard-disk access and huge memory use. I also have a 'pause' button so that i can use my computer normally when i need it.

Stats until now:

Code: Select all

Non-capture
DB     Win             Draw                Lose                    Win           Draw             Lose
0503     704.925.456    3.560.143.014                0    0305       3.281.314    3.196.650.674      417.551.618 
0512   8.273.581.968    2.181.353.594                0    1205      10.284.244    7.293.125.775    8.055.044.119 
0521   8.415.019.196       66.685.206                0    2105       7.142.747    4.600.829.936   17.204.584.803 
0530   2.276.443.113           54.807                0    3005         879.372      533.378.154    9.830.307.636 
1403   3.183.280.592   21.081.806.976                1    0314      58.934.290   14.406.335.089    1.041.842.747 

0404      30.823.865    4.245.737.717          560.081    0404      30.823.865    4.245.737.717          560.081 
0413     548.631.694   13.794.024.089          257.207    1304     109.951.875   21.450.905.536       57.739.438 
0422   2.892.283.317   15.045.959.691           36.546    2204     125.521.931   39.917.605.934    1.046.702.123 
1304     109.951.875   21.450.905.536       57.739.438    0413     548.631.694   13.794.024.089          257.207 
1313   1.905.191.192   71.565.238.320       58.503.429    1313   1.905.191.192   71.565.238.320       58.503.429 
2204     125.521.931   39.917.605.934    1.046.702.123    0422   2.892.283.317   15.045.959.691           36.546 


All
0503  17.229.615.878   12.829.002.132        6.586.390    0305     339.953.720   27.805.830.388    1.919.420.292 
0512  66.082.013.381   15.082.985.562       11.052.937    1205     534.976.536   60.357.137.712   20.283.937.632 
0521  68.313.484.823    4.574.803.712        4.492.745    2105     214.808.099   28.192.355.922   44.485.617.259 
0530  21.718.714.925       47.855.119           24.366    3005       2.007.953    1.271.028.889   20.493.557.568 
1403  66.486.418.486   68.684.445.125      122.556.189    0314   3.439.336.510   25.702.283.377    6.151.799.913 

0404   3.916.965.780   33.608.681.251       55.858.469    0404   3.916.965.780   33.608.681.251       55.858.469 
0413  30.076.181.317   05.093.200.110      124.038.373    1304  10.075.690.738   24.282.093.375      935.635.687 
0422  79.922.300.425   02.218.929.773       90.723.002    2204   8.710.678.358   67.741.575.565    5.779.699.277 
1304  10.075.690.738   24.282.093.375      935.635.687    0413  30.076.181.317   05.093.200.110      124.038.373 
1313  80.587.901.533   04.430.429.608    2.160.695.259    1313  80.587.901.533   04.430.429.608    2.160.695.259 
2204   8.710.678.358   67.741.575.565    5.779.699.277    0422  79.922.300.425   02.218.929.773       90.723.002 

Michel

I am progressing very slowly comparing to you but I progress!
BTW you have some troncation problems with the table above (see my counts below).

ALL
0413 draw 105.093.200.110
1304 draw 124.282.093.375
1313 draw 404.430.429.608

Up to now no difference between us.
Before beginning this 8 pieces db generation I generated the 6x1 db in order to complete the 7db. If you have it I will be happy to compare my counts with yours

[MichelG]

Hi Michel,
I am progressing very slowly comparing to you but I progress!
BTW you have some troncation problems with the table above (see my counts below).

ALL
0413 draw 105.093.200.110
1304 draw 124.282.093.375
1313 draw 404.430.429.608

Up to now no difference between us.
Before beginning this 8 pieces db generation I generated the 6x1 db in order to complete the 7db. If you have it I will be happy to compare my counts with yours

Good to not see any differences I fixed the truncation issue and updated the table. Next in line is 1322, but with almost 2 trillion positions in it, it is going to take a few more days.

Dragon isn't actually capable of creating 6x1, although it would be straightforward to build it in. I might add it at a later stage.

Michel

[TAILLE]

Hi Michel,

Good to not see any differences I fixed the truncation issue and updated the table. Next in line is 1322, but with almost 2 trillion positions in it, it is going to take a few more days.
Michel

I guess you have already generated the 1322 and 2213 db.
My counts for these generations are the following :

1322
win without capture : 10 048 283 993
draw without capture : 83 274 268 705
loss without capture : 14 241 250
win with capture : 219 428 881 427
draw with capture : 341 944 266 846
loss with capture : 1 662 555 879

2213
win without capture : 2 128 648 799
draw without capture : 137 975 831 241
loss without capture : 1 493 070 805
win with capture : 69 602 799 163
draw with capture : 433 038 022 130
loss with capture : 12 134 125 962

Michel and Ed. : do you have the same figures ?

BTW can you give us your counters for the other genrations ?

[Ed Gilbert]

Gerard, you will remember that when we were comparing 7-piece counts, there was one slice where we disagreed by 1 count on the total number of noncapture positions, and that turned out to be an error in my count statistics but not in the actual position values. I think we have a similar situation with db1322. In the black to move positions I have one more win count than you, but the same counts as you for draws and losses, and the same counts as you for all the white to move non-captures. So one of us has an error of one count in the black win statistics, but I suspect not in the actual position. I think it's likely to be an error in the statistics like we had earlier for 2 reasons; 1) if it was a real position error then you might see it propagate to other positions in the same slice, but that apparently didn't happen, and 2) I self verify the positions after each subdivision build, which should catch most value errors, but I don't self verify the count statistics, so that's the place where memory errors will most likely go unnoticed.

I started up a program to count the number of non-capture positions in each of the ~5000 build subdivisions within this slice and compare that to the sum of the WLD counts. This slice is huge, and the program is not multi-threaded so this is going to take a few days.

-- Ed

[TAILLE]

Hi Ed,

Gerard, you will remember that when we were comparing 7-piece counts, there was one slice where we disagreed by 1 count on the total number of noncapture positions, and that turned out to be an error in my count statistics but not in the actual position values. I think we have a similar situation with db1322. In the black to move positions I have one more win count than you, but the same counts as you for draws and losses, and the same counts as you for all the white to move non-captures. So one of us has an error of one count in the black win statistics, but I suspect not in the actual position. I think it's likely to be an error in the statistics like we had earlier for 2 reasons; 1) if it was a real position error then you might see it propagate to other positions in the same slice, but that apparently didn't happen, and 2) I self verify the positions after each subdivision build, which should catch most value errors, but I don't self verify the count statistics, so that's the place where memory errors will most likely go unnoticed.

I started up a program to count the number of non-capture positions in each of the ~5000 build subdivisions within this slice and compare that to the sum of the WLD counts. This slice is huge, and the program is not multi-threaded so this is going to take a few days.

-- Ed

It seems that one position is seen as a capture position for me and a non capture position for you. If that is case the quickest verifying process is certainly to simply count the capture or non capture positions. That way you do not have to access the db and you save a lot of time.
I hope Michel statistics will help us to find the problem.
For the moment I am waiting for the result of your investigations.
The 2222db is now under generation

[MichelG]

I got the same numbers 2213, 1322, both capture and non-capture

My data up til now:

Code: Select all

Non-capture
DB     Win             Draw                Lose                    Win           Draw             Lose
0404       30.823.865    4.245.737.717          560.081   0404       30.823.865    4.245.737.717          560.081 
1304      109.951.875   21.450.905.536       57.739.438   0413      548.631.694   13.794.024.089          257.207 
1313    1.905.191.192   71.565.238.320       58.503.429   1313    1.905.191.192   71.565.238.320       58.503.429 
2204      125.521.931   39.917.605.934    1.046.702.123   0422    2.892.283.317   15.045.959.691           36.546 
2213    2.128.648.799  137.975.831.241    1.493.070.805   1322   10.048.283.993   83.274.268.705       14.241.250 
2222   11.367.135.074  170.353.316.894      505.968.094   2222   11.367.135.074  170.353.316.894      505.968.094 
3104       48.729.931   23.742.050.981   10.995.615.714   0431    7.386.983.954    2.527.311.452                0 

0503      704.925.456    3.560.143.014                0   0305        3.281.314    3.196.650.674      417.551.618 
0512    8.273.581.968    2.181.353.594                0   1205       10.284.244    7.293.125.775    8.055.044.119 
0521    8.415.019.196       66.685.206                0   2105        7.142.747    4.600.829.936   17.204.584.803 
0530    2.276.443.113           54.807                0   3005          879.372      533.378.154    9.830.307.636 
1403    3.183.280.592   21.081.806.976                1   0314       58.934.290   14.406.335.089    1.041.842.747 
1412   42.534.667.073   17.911.160.723                2   1214      166.172.640   37.716.098.563   28.762.809.911 

All
0404    3.916.965.780   33.608.681.251       55.858.469   0404    3.916.965.780   33.608.681.251       55.858.469 
1304   10.075.690.738  124.282.093.375      935.635.687   0413   30.076.181.317  105.093.200.110      124.038.373 
1313   80.587.901.533  404.430.429.608    2.160.695.259   1313   80.587.901.533  404.430.429.608    2.160.695.259 
2204    8.710.678.358  167.741.575.565    5.779.699.277   0422   79.922.300.425  102.218.929.773       90.723.002 
2213   71.731.447.962  571.013.853.371   13.627.196.767   1322  229.477.165.420  425.218.535.551    1.676.797.129 
2222  214.945.345.223  658.851.889.295   10.769.903.882   2222  214.945.345.223  658.851.889.295   10.769.903.882 
3104    2.518.041.015   85.017.523.421   21.297.407.614   0431   80.096.763.016   28.713.780.220       22.428.814 

0503   17.229.615.878   12.829.002.132        6.586.390   0305      339.953.720   27.805.830.388    1.919.420.292 
0512   66.082.013.381   15.082.985.562       11.052.937   1205      534.976.536   60.357.137.712   20.283.937.632 
0521   68.313.484.823    4.574.803.712        4.492.745   2105      214.808.099   28.192.355.922   44.485.617.259 
0530   21.718.714.925       47.855.119           24.366   3005        2.007.953    1.271.028.889   20.493.557.568 
1403   66.486.418.486   68.684.445.125      122.556.189   0314    3.439.336.510  125.702.283.377    6.151.799.913 
1412  279.702.401.953   85.472.535.138      209.332.709   1214    5.588.412.150  287.132.364.905   72.663.492.745 

Count is busy on 3113, computation on 3122

[TAILLE]

Hi Ed,

I got the same numbers 2213, 1322, both capture and non-capture

Now we are sure that the statistic counter problem is in your program. I do not know if I can help you. If yes don't hesitate to tell me. At least you have now the correct number of capture and non-capture positions.

[Ed Gilbert]

Hi Gerard,

It seems that one position is seen as a capture position for me and a non capture position for you. If that is case the quickest verifying process is certainly to simply count the capture or non capture positions. That way you do not have to access the db and you save a lot of time.

Yes, that is exactly what I'm doing. It still takes a lot of time, but fortunately the subdivision with the discrepency was close to the beginning of the list and it was found overnight. I will regenerate the statistics on the subdivision this weekend.

-- Ed

[TAILLE]

Hi Michel,

I got the same numbers 2213, 1322, both capture and non-capture

My data up til now:

Code: Select all

Non-capture
DB     Win             Draw                Lose                    Win           Draw             Lose
0404       30.823.865    4.245.737.717          560.081   0404       30.823.865    4.245.737.717          560.081 
1304      109.951.875   21.450.905.536       57.739.438   0413      548.631.694   13.794.024.089          257.207 
1313    1.905.191.192   71.565.238.320       58.503.429   1313    1.905.191.192   71.565.238.320       58.503.429 
2204      125.521.931   39.917.605.934    1.046.702.123   0422    2.892.283.317   15.045.959.691           36.546 
2213    2.128.648.799  137.975.831.241    1.493.070.805   1322   10.048.283.993   83.274.268.705       14.241.250 
2222   11.367.135.074  170.353.316.894      505.968.094   2222   11.367.135.074  170.353.316.894      505.968.094 
3104       48.729.931   23.742.050.981   10.995.615.714   0431    7.386.983.954    2.527.311.452                0 

0503      704.925.456    3.560.143.014                0   0305        3.281.314    3.196.650.674      417.551.618 
0512    8.273.581.968    2.181.353.594                0   1205       10.284.244    7.293.125.775    8.055.044.119 
0521    8.415.019.196       66.685.206                0   2105        7.142.747    4.600.829.936   17.204.584.803 
0530    2.276.443.113           54.807                0   3005          879.372      533.378.154    9.830.307.636 
1403    3.183.280.592   21.081.806.976                1   0314       58.934.290   14.406.335.089    1.041.842.747 
1412   42.534.667.073   17.911.160.723                2   1214      166.172.640   37.716.098.563   28.762.809.911 

All
0404    3.916.965.780   33.608.681.251       55.858.469   0404    3.916.965.780   33.608.681.251       55.858.469 
1304   10.075.690.738  124.282.093.375      935.635.687   0413   30.076.181.317  105.093.200.110      124.038.373 
1313   80.587.901.533  404.430.429.608    2.160.695.259   1313   80.587.901.533  404.430.429.608    2.160.695.259 
2204    8.710.678.358  167.741.575.565    5.779.699.277   0422   79.922.300.425  102.218.929.773       90.723.002 
2213   71.731.447.962  571.013.853.371   13.627.196.767   1322  229.477.165.420  425.218.535.551    1.676.797.129 
2222  214.945.345.223  658.851.889.295   10.769.903.882   2222  214.945.345.223  658.851.889.295   10.769.903.882 
3104    2.518.041.015   85.017.523.421   21.297.407.614   0431   80.096.763.016   28.713.780.220       22.428.814 

0503   17.229.615.878   12.829.002.132        6.586.390   0305      339.953.720   27.805.830.388    1.919.420.292 
0512   66.082.013.381   15.082.985.562       11.052.937   1205      534.976.536   60.357.137.712   20.283.937.632 
0521   68.313.484.823    4.574.803.712        4.492.745   2105      214.808.099   28.192.355.922   44.485.617.259 
0530   21.718.714.925       47.855.119           24.366   3005        2.007.953    1.271.028.889   20.493.557.568 
1403   66.486.418.486   68.684.445.125      122.556.189   0314    3.439.336.510  125.702.283.377    6.151.799.913 
1412  279.702.401.953   85.472.535.138      209.332.709   1214    5.588.412.150  287.132.364.905   72.663.492.745 

Count is busy on 3113, computation on 3122

No doubt at all you will finish 8 pieces generation some month before me.
As a consolation I can say that I am the first (and the only one as far as I know) to have the really complete 7 pieces db because I have already generated the 6x1 db.
My intention is to generate :
4x4 db : about 50 days
5x3 db : about 60 days
6x2 db : about 30 days
So I need less than 5 months for all these (=> February 2011 if no incident)

[Ed Gilbert]

Hi Michel,

I can confirm your new results on 2222 and 0431. However I have these results for 1412:

1412b: wins 42534667073, draws 17911160723, losses 2
1412w: wins 166172640, draws 37716098562, losses 28762809912

In 1412w I have one more loss and you have one more draw. I'm leaving for work now, but more later.

-- Ed

[TAILLE]

Hi Ed, Michel

Hi Michel,

I can confirm your new results on 2222 and 0431. However I have these results for 1412:

1412b: wins 42534667073, draws 17911160723, losses 2
1412w: wins 166172640, draws 37716098562, losses 28762809912

In 1412w I have one more loss and you have one more draw. I'm leaving for work now, but more later.

-- Ed

Very surprising. Obviously there is a bug in Ed. or in Michel db. Because we already saw that Ed. counts are not 100% correct it may be possible that all positions are correct in your databases but Ed. statistics are wrong
For your information I will generate this db in about 1.5 month. In other word, in order to avoid a very time consuming investigation, you may also wait a little to see my own counts hoping it wil not be a third figure!

[MichelG]

I'll run a verify and recount of 1412 during the weekend.

[Ed Gilbert]

I have identified the subdivision within db1322 that had the count discrepency, and I have rerun the statistics script on it to get the counts again. This time I get counts that agree with Michel and Gerard. So there was no db data error, only an error in the statistics.

The db1412 discrepency is another matter. This weekend I will reload the build database files from archive disks and recreate the build environment so I can rebuild that slice for verification.

Edit: I started up the build using 8 cores, and turned off the verify passes so it will run faster. I think it should be done by next weekend. Michel, how long do you estimate your verify + recount will take?

-- Ed

[MichelG]

The db1412 discrepency is another matter. This weekend I will reload the build database files from archive disks and recreate the build environment so I can rebuild that slice for verification.

Edit: I started up the build using 8 cores, and turned off the verify passes so it will run faster. I think it should be done by next weekend. Michel, how long do you estimate your verify + recount will take?

-- Ed

Ed, I did a verify on 1412, and it spotted an error in my databases. I rebuild the corresponding slice and that turned 1 draw into a loss, so the count is now seems the same.

Since the code did not change, it must have been a memory error of some sort rather than a bug.

I am rebuilding 1412 as a whole again because the error might have propagated through the database, and then do another count. The whole process will take about 7-8 days because it is running on my secondary computer.

Michel

[Rein Halbersma]

The db1412 discrepency is another matter. This weekend I will reload the build database files from archive disks and recreate the build environment so I can rebuild that slice for verification.

Edit: I started up the build using 8 cores, and turned off the verify passes so it will run faster. I think it should be done by next weekend. Michel, how long do you estimate your verify + recount will take?

-- Ed

Ed, I did a verify on 1412, and it spotted an error in my databases. I rebuild the corresponding slice and that turned 1 draw into a loss, so the count is now seems the same.

Since the code did not change, it must have been a memory error of some sort rather than a bug.

I am rebuilding 1412 as a whole again because the error might have propagated through the database, and then do another count. The whole process will take about 7-8 days because it is running on my secondary computer.

Michel

What kind of memory do you have? There are some articles on Wiki and dedicated computer magazines on the benefits of using ECC RAM. This will catch and correct single bit errors. Owing to cosmic radiation, the base level error rate might be something like 1 bit per day per 1Gb of memory. Of course, if you use ECC RAM, you then might also have to upgrade to a server motherboard and appropriate CPU to actually make use of the more expensive RAM. Perhaps Ed can provide us with more details about his pc setup?