Damage does not have (yet) the 7 Piece Endgame Databases

Ed Gilbert · Post by **Ed Gilbert** » Wed Apr 29, 2009 00:40

BertTuyt wrote:Ed, my counts for the 5K - 2K and 2K - 5K are the same, thanks for the info.

Will now start with the 7p containing 1 man (on the older machine).
On the new one i will start with the 4-3 part.

Do you have slice info of the 7p databases containing 1 man, so i can check results, or do you only have the info for the total slice package.

I have counts for every slice, which is what I call a separate configuration of bBwW. Here are the counts for the next slices you will need:

0403b: wins 22947628, draws 624882484, losses 0
0403w: wins 1712978, draws 599580394, losses 2251058
0412b: wins 283913660, draws 1332445036, losses 0
0412w: wins 4648489, draws 2283871173, losses 92442071
0511b: wins 885686145, draws 1562537, losses 0
0511w: wins 210906, draws 125860616, losses 1236942539

Next to that, as i will start on the new i7, i need to transfer files.
Do you apply a check (64bit xor or something like that), to make sure that the databases are not corrupted during transfer.
As these calculations are time consuming i don't want to take the risk.

Do you have a bad experience with this , so is it wise to to these checks?

I designate one machine as the master. When I finish building a file, I put it in the master's db and get a 32-bit CRC for it which is also stored as part of the db. When a satellite machine takes a file from the master, it also computes the CRC to verify it. I have had errors from transfers! They might have been due to low disk space on the target disk, I cannot remember now, but they were caught by the CRC check. I have also had errors elsewhere in the build process when I was using machines that do not have ECC memory. These errors were always caught during self-verification, except once when I was comparing counts against Schaeffer's and found it that way. I have not seen any errors at all during the 8pc db build, and both machines I'm using have ECC memory.

The CRC calculation time is negligible. On a db file that takes hours to build you can calculate the CRC in just a few seconds.

In your case you don't have to worry about an error going undetected for a long time and ruining all subsequent calculations because you can verify your counts against mine as you go. That means you can afford to be less vigilant about errors.

-- Ed

64_bit_checkers_engine · Wed Apr 29, 2009 06:04

Hey Ed,

What position is the longest conversion win you have computed for 10x10 draughts so far?

Or is that a guarded secret until after your next tournament?

[img]images/smilies/icon_smile.gif[/img]

Ed Gilbert · Post by **Ed Gilbert** » Wed Apr 29, 2009 20:48

64_bit_checkers_engine wrote:Hey Ed,

What position is the longest conversion win you have computed for 10x10 draughts so far?

Or is that a guarded secret until after your next tournament?

[img]images/smilies/icon_smile.gif[/img]

No secret at all. The longest position up to 7 pieces is around 115 plies to conversion, and very few are over 100. It is posted somewhere on this forum topic a few months back. As you know this is quite different from other forms of checkers. In English checkers there is a 9-piece positions that is 190-something, and in Italian checkers there are positions that are greater than 210 plies. I guess with flying kings, what might take several king walk moves in checkers can be done in one draughts move.

I have not scanned the 8-piece db yet for long MTC positions, so there is the possibility that a longer one will pop up when I do that.

-- Ed

TAILLE · Post by **TAILLE** » Wed Apr 29, 2009 22:53

Ed Gilbert wrote: No secret at all. The longest position up to 7 pieces is around 115 plies to conversion, and very few are over 100. It is posted somewhere on this forum topic a few months back.
-- Ed

Ed is right we already discussed on this position on this forum.

For Damy this position needs 107 plies before the first conversion.

BertTuyt · Post by **BertTuyt** » Sat May 02, 2009 15:40

I have find a good utility (incl. source code) to compare files (based on CRC) which each other.

I need this as I move Database files between computers, and want to be sure that files are not corrupted. The same CRC does not 100% guarantee that files are identical, but the odds are small that with the same CRC multiple file errors yield the same result.

You can find the utility here : http://www.codeproject.com/KB/files/fileinfo.aspx

Codeproject, CFileInfoArray, A class for gathering file information recursively trough directories.

Bert

BertTuyt · Post by **BertTuyt** » Sun May 03, 2009 00:04

Ed, could you post the WDL counts for the slice:

Whiteman 1
WhiteKing 4
Blackman 0
BlackKing 2

White to Move,

Thanks in advance,

Bert

Ed Gilbert · Post by **Ed Gilbert** » Sun May 03, 2009 05:14

Bert, here are all the 7-piece counts.

Notation: bBwW
0403b: wins 22947628, draws 624882484, losses 0
0403w: wins 1712978, draws 599580394, losses 2251058
0412b: wins 283913660, draws 1332445036, losses 0
0412w: wins 4648489, draws 2283871173, losses 92442071
0421b: wins 1138292886, draws 197838832, losses 0
0421w: wins 2808661, draws 1573165088, losses 1561645302
0430b: wins 363662706, draws 2156778, losses 0
0430w: wins 296975, draws 168572305, losses 1211671344
0502b: wins 489226234, draws 59194628, losses 0
0502w: wins 175224, draws 80452496, losses 376519010
0511b: wins 885686145, draws 1562537, losses 0
0511w: wins 210906, draws 125860616, losses 1236942539
0520b: wins 355950583, draws 277, losses 0
0520w: wins 41269, draws 21121224, losses 999519077
1303b: wins 75041376, draws 2964492283, losses 26
1303w: wins 31976341, draws 2023408571, losses 2553027
1312b: wins 893300142, draws 6813961789, losses 25
1312w: wins 80008423, draws 8013813737, losses 139511210
1321b: wins 4981650641, draws 1499012585, losses 10
1321w: wins 46295542, draws 6370190391, losses 4574983194
1330b: wins 1775068306, draws 31491058, losses 1
1330w: wins 4275925, draws 705003928, losses 4185372540
1402b: wins 2420945170, draws 478103942, losses 0
1402w: wins 2919505, draws 505610056, losses 1481333376
1411b: wins 4758418717, draws 17191554, losses 0
1411w: wins 3455934, draws 650775239, losses 5351633668
1420b: wins 1953253420, draws 4689, losses 0
1420w: wins 556644, draws 96725305, losses 4452095825
2203b: wins 78200925, draws 5281029059, losses 45804
2203w: wins 167951930, draws 2450549147, losses 703986
2212b: wins 925342523, draws 12867400378, losses 13442
2212w: wins 401236248, draws 10176739540, losses 56094484
2221b: wins 7019056270, draws 4761276537, losses 7709
2221w: wins 231885784, draws 10071366000, losses 4086903333
2230b: wins 3167431065, draws 170903135, losses 1861
2230w: wins 20578346, draws 1269512002, losses 5198855329
2302b: wins 4198230472, draws 1945895748, losses 47
2302w: wins 16862729, draws 1637041099, losses 1798168572
2311b: wins 10182489629, draws 110029711, losses 46
2311w: wins 19578663, draws 1420700913, losses 9112328689
2320b: wins 4285400830, draws 67685, losses 4
2320w: wins 2580705, draws 173577085, losses 7913400965
3103b: wins 27509423, draws 4159173566, losses 17997625
3103w: wins 308558016, draws 1165822996, losses 6169
3112b: wins 331139648, draws 10626204330, losses 12840692
3112w: wins 699473807, draws 5376500875, losses 2944008
3121b: wins 2207921855, draws 7295616081, losses 1359296
3121w: wins 390368689, draws 7449981971, losses 503075535
3130b: wins 2222661901, draws 511643243, losses 36832
3130w: wins 32063859, draws 1438257550, losses 2341507574
3202b: wins 2766598999, draws 3752544678, losses 1448
3202w: wins 39277759, draws 2265684610, losses 678102698
3211b: wins 10629556269, draws 462037384, losses 2014
3211w: wins 44712860, draws 1739337215, losses 7456542312
3220b: wins 4691761177, draws 2864648, losses 617
3220w: wins 4975720, draws 174292186, losses 6993155833
4003b: wins 1944214, draws 708582014, losses 526780449
4003w: wins 251924262, draws 57727265, losses 0
4012b: wins 23901292, draws 2171713568, losses 1073416435
4012w: wins 819983024, draws 477436143, losses 3838
4021b: wins 136394440, draws 2249042761, losses 484459502
4021w: wins 674883270, draws 1125942653, losses 6475631
4030b: wins 422782725, draws 397218117, losses 17051943
4030w: wins 69994641, draws 495944034, losses 271109700
4102b: wins 729618536, draws 2729677088, losses 162285
4102w: wins 39952296, draws 1169006275, losses 74833267
4111b: wins 5193794066, draws 776854157, losses 22047
4111w: wins 43479839, draws 1167655012, losses 2820935999
4120b: wins 2548903337, draws 16608246, losses 1178
4120w: wins 4088648, draws 131122372, losses 3035223931
5002b: wins 53579246, draws 601212602, losses 78987756
5002w: wins 58375294, draws 159922205, losses 1784475
5011b: wins 669863526, draws 605186934, losses 8116476
5011w: wins 30414574, draws 444409694, losses 226409920
5020b: wins 526663112, draws 32319102, losses 31098

BertTuyt · Post by **BertTuyt** » Sun May 03, 2009 15:49

Ed, thanks.

I think you missed the 5020w, as the total slice count should be 76 .

For the specific DB :

Whiteman 1
WhiteKing 4
Blackman 0
BlackKing 2

White to Move,

I checked the WDL this morning and I have the same number, so will do the Black to Move slice. Hope I don't see differences, so far so good.

Also expect to finish the 4K x 3K and 3K x 4K today.

So in total then 6 7P DB's and a long way to go

The new machine runs at 300KPositions/second.

I also think I could accelerate the speed a little, will try later today, don't halt a program when in the middle of a calculation.

Don't think I will finish the 7P work before the next tournament.

Anyway, I have now booked the hotel for the Saturday, as some other people did, so maybe we meet for diner in the evening, otherwise see you on Sunday.

Bert

Ed Gilbert · Post by **Ed Gilbert** » Sun May 03, 2009 17:55

Bert, you're right about missing the counts for the last slice -- off-by-one error :-)

5020w: wins 1880835, draws 80693549, losses 476298599

300Kpos/sec sounds like a reasonable speed. To get things done quickly you need to run a build instance on each core. I have 12 instances running in parallel right now on the 8pc db and it is nearly 3/4 finished. That's 8 instances on one machine and 4 on another. Of course to do that you have to automate everything. These things run for weeks at a time without any intervention from me. The only intervention I do is to restart if a power outage or if I had to stop to use a machine for some memory-intensive draughts search or install a Windows update.

-- Ed

Ed Gilbert · Post by **Ed Gilbert** » Sun May 03, 2009 19:36

BertTuyt wrote:Anyway, I have now booked the hotel for the Saturday, as some other people did, so maybe we meet for diner in the evening, otherwise see you on Sunday.

Bert, I'm glad to hear that you will attend, so we will finally get to meet in person! Gerard has very kindly offered me a ride to the tournament from Paris, and we are planning to drive up Sunday morning.

-- Ed

BertTuyt · Post by **BertTuyt** » Sun May 31, 2009 11:24

7p Database generation is progressing on my new i7 machine.
Although i get some crashes now and then ( every 2 - 3 days), like during the tournament, i don't bother as in general i don't loose that much work.

I did not check the results so far , based on the WDL values , provided by Ed.
Will do so when the total 4 -3 ( and 3 - 4 ) build. is ready, then switch to the 5 - 2 ( and 2- 5 ) build.
The 4 - 3 is now calculating databases with 2 man on each side, so still a long way to go.

I only run 1 instance, when i started 2 i got a crash, i don't know why, but didn't go into details.

On the other machine ( also 64bit OS, and quad Q6600 2.4 Ghz), im trying to develop a faster DB-program, for the longer-term future (going to 8p).

I now use a standard database i calculate as a reference the 4k - 2K (and parallel calculated the 2K - 4K).

Ed/Gerard, do you know your times for the generation of this database, so not the total chain needed to come to this result, so only the 4K - 2K ?

On my i7 the general speed is 300K/positions solved/sec.
My development target is above the 1M, so some work to do

Thanks,

Bert

Ed Gilbert · Post by **Ed Gilbert** » Sun May 31, 2009 13:13

Hi Bert,

Ed/Gerard, do you know your times for the generation of this database, so not the total chain needed to come to this result, so only the 4K - 2K ?

On my i7 the general speed is 300K/positions solved/sec.

I don't know what it is for that particular slice. For the 8pc db the average build speed is running around 200kpos/sec. This does not include verification, which averages to be a little slower than building! So my overall progress for building and verifying using a single instance is ~100kpos/sec or perhaps even a bit less. I was puzzled at first by the slow speed of verifying, as it looks at first like verification should be much less work, but I now understand that it has to do with how often capture positions get (re)evaluated during verification. This was never a problem with 8x8 checkers, where verification is typically 2x faster than building. If I ever try to go beyond the 5x4 all men slice I will cache the capture positions during verification to speed it up. But back to your question, I think 300kpos/sec is a reasonable build speed. Your I7 is faster than my quad cores, and I think our speeds are roughly the same when normalized for processor speed. I'm also running 8 instances of the builder in parallel on the one machine, and each instance is given only 1/8th of the total ram to use, so probably if I just ran one instance and gave it all the ram in the machine it would run a little faster.

Does your builder compress the db slices as they are built, or do you store uncompressed slices during building and do lookups on dependency successors in these uncompressed slices? It has been a few years since I looked at the Grimminck/Jetten builder but I thought it kept all the data in uncompressed form during the build. Martin Fierz's builder also does this. My builder compresses as it builds. This makes the builder run a little slower, not because compression takes much time, but because the lookups on dependency positions take longer on the compressed positions. But for building large databases like the 8pc db I think its the only practical way to do it. Otherwise you have to store several TB of uncompressed data.

My development target is above the 1M, so some work to do

Of course the best way and maybe the only way to accomplish that is to get all your cores building in parallel.

-- Ed

BertTuyt · Post by **BertTuyt** » Sun May 31, 2009 16:34

Ed, i based my builder on the work of Grimmick/Jetten, but modified it completely, so the Builder uses now Bitboards and all Routines are Threadsafe.

I still use uncompressed data, for 7p still no issue, but when going to 8p , i agree with you and I have to go to real-time compression.

I think using 1 Builder with all cores activated is more efficient, then using multiple instances running. Also while the memory is not shared, and efficiency improved when there is much direct RAM-access.

I also think (and hope) that the database algorithms should be far better scalable , compared with the search-routines.

So im now in the process in creating a parallel version.

This week i have seen the announcement of a new Intel Xeon processor based on the i7-Nehalem architecture, but with 8 cores (and 16 threads), which could be used in a 8-processor environment.
There is already a demonstration machine available with 128 parallel threads !
Think also the memory is huge.

I don't think this machine is affordable, but with 1 week access to such hardware, i think we could generate 8p in such a short term-frame (if my idea holds, that DB-generation is rather well scalable).

Keep you all posted, hope to have some results within 1 month.

Bert

BertTuyt · Post by **BertTuyt** » Mon Aug 10, 2009 12:00

For the moment I had to stop the 7p Database generation.
As i have posted in the past, I use a very memory-consuming algorithm, where all required non-compressed databases are pre-loaded in memory.
For whatever reason (to be verified), i even run into memory problems with my 12GByte i7 system.

Before I continue with the modification of my program/algorithm i will check if the results so far are good. For this I can rely on the WDL counts as communicated by Ed.

Keep you posted,

Bert

TAILLE · Post by **TAILLE** » Tue Aug 11, 2009 10:30

Hi Bert,

BertTuyt wrote:For the moment I had to stop the 7p Database generation.
As i have posted in the past, I use a very memory-consuming algorithm, where all required non-compressed databases are pre-loaded in memory.
For whatever reason (to be verified), i even run into memory problems with my 12GByte i7 system.

Before I continue with the modification of my program/algorithm i will check if the results so far are good. For this I can rely on the WDL counts as communicated by Ed.

Keep you posted,

Bert

For your information I also work on new algorithms for the generation process and also for the db access. I have a lot of new ideas (including non-compress db and parallellism) and I intend to test them till the end of the year by regenerating the 7p db. I hope to begin 8p db generation early in 2010.