Kingsrow - databases

[Krzysztof Grzelak]

I have a question for you, Ed. Can you calculate how much space for base 9 and 10 pieces.

[Ed Gilbert]

If I assume only 5x4 and 5x5 configurations (no 6x3 or 6x4), and assume that the size for 9 pieces will be the 7-piece size times the ratio of 9-piece positions to 7-piece positions (so assuming the same compression ratio for 9 pieces as was achieved for 7 pieces), then the size for 9 pieces would be 764 GB.

Using similar assumptions applied to 10 pieces relative to 8 pieces, the size for 10 pieces would be 4852 GB.

To use the 9-piece slices, you would of course also need the slices for 2 through 8 pieces, so you would have to add 55 GB to the 764 GB to get the total size of 2 through 9 pieces. And to use the 10-piece slices, you would also need the 2 through 9 pieces, so you would have to add (55 + 764) to the 4852 GB to get the total size of 2 through 10 pieces.

-- Ed

[Krzysztof Grzelak]

Thanks Ed. And you can describe very precisely how to generate the database 9 pieces. Here a question comes to my mind - after generating the 9 pieces database, you will have to change the program code to handle this database.How do you think Ed how long would it kidnapped.

[Ed Gilbert]

Thanks Ed. And you can describe very precisely how to generate the database 9 pieces. Here a question comes to my mind - after generating the 9 pieces database, you will have to change the program code to handle this database.How do you think Ed how long would it kidnapped.

I have no plans to build the 9-piece database.

[BertTuyt]

Ed, although you don't have the intention to build a 9p DB, would you have a gutfeel for the computing time needed on a 16-core machine?
Would resources be sufficient, or do you expect a memory overload?
The SSD most likely is no problem (as these days 4TB SSD are available).
In the end I also do not believe it will add to the playing strength, but for perfect analysis it might bring something.

Bert

[Krzysztof Grzelak]

I understand that. Can I do it myself with your help? The 764 GB is not much.

[Krzysztof Grzelak]

Ed, although you don't have the intention to build a 9p DB, would you have a gutfeel for the computing time needed on a 16-core machine?
Would resources be sufficient, or do you expect a memory overload?
The SSD most likely is no problem (as these days 4TB SSD are available).
In the end I also do not believe it will add to the playing strength, but for perfect analysis it might bring something.

Bert

You need to connect two 4 GB SSDs in Raid. I am thinking to change the processor to an even faster one to draughts core=32 - threads=64 .

[Ed Gilbert]

I built 2 through 8 pieces in 2008 - 2009. It took 5.6 core-years to build and verify. See http://edgilbert.org/InternationalDraug ... hts_db.htm

The number of positions for 9 pieces is about 11.5 times as many as 2 through 8 pieces.

So if you were to build 9 pieces on 2008 hardware, you could expect it to take 11.5 * 5.6 = 64.4 core-years.

But 2021 hardware is much faster. The CPUs are faster, and now we have SSDs that are much faster than hard drives. In my build the time was dominated by CPU speeds, but SSDs might have sped it up by 25% or so.

Let's say that with 2021 hardware a single core can build dbs at a rate 5x faster than with 2008 hardware. To build 9 pieces would then take (64.4 / 5) = 12.9 core-years. That would be 9 or 10 months on a 16-core machine.

[Krzysztof Grzelak]

And what version database are you writing about. And here I have a question - is it possible to connect two computers to make the database generation faster.

[Rein Halbersma]

IIRC, the 10-pc dbs are in the same ballpark as the chess 7-pc dbs. One key bottleneck would be generating the 5v5 kings sub db, which has 2.6 trillion (10^12) positions. That requires 640+Gb of RAM with 2 bits per position for a single side to move. For 6v4 kings you need at least 1 Tb of RAM. That will set you back at least $6-10K in RAM alone. For chess someone actually had access to two 1.5 Tb machines with 192/384 cores/threads ($90K each) to generate the 18 Tb on disk in 4 months back in 2018. So give it some resources

[Krzysztof Grzelak]

IIRC, the 10-pc dbs are in the same ballpark as the chess 7-pc dbs. One key bottleneck would be generating the 5v5 kings sub db, which has 2.6 trillion (10^12) positions. That requires 640+Gb of RAM with 2 bits per position for a single side to move. For 6v4 kings you need at least 1 Tb of RAM. That will set you back at least $6-10K in RAM alone. For chess someone actually had access to a 1.5 Tb machine with 384 cores to generate the 18 Tb on disk in 4 months back in 2018. So give it some resources

Only there is 18TB here. And few chess testers have this base. Ed could make a base 5x5. After all, he has a good computer.

[Rein Halbersma]

IIRC, the 10-pc dbs are in the same ballpark as the chess 7-pc dbs. One key bottleneck would be generating the 5v5 kings sub db, which has 2.6 trillion (10^12) positions. That requires 640+Gb of RAM with 2 bits per position for a single side to move. For 6v4 kings you need at least 1 Tb of RAM. That will set you back at least $6-10K in RAM alone. For chess someone actually had access to a 1.5 Tb machine with 384 cores to generate the 18 Tb on disk in 4 months back in 2018. So give it some resources

Only there is 18TB here. And few chess testers have this base. Ed could make a base 5x5. After all, he has a good computer.

It would take 8 calendar years to compute the 5x5 with 16 cores, but that assumes already ~1Tb of memory which is not nearly what even a high-end workstation has. I think Ed's resources (both cores and RAM) need to be increased 8-fold in order for this to be feasible within 1 calendar year. Assuming Moore's law (doubling every 3 years nowadays) 5x5 will not happen this decade.

[Krzysztof Grzelak]

It would take 8 calendar years to compute the 5x5 with 16 cores, but that assumes already ~1Tb of memory which is not nearly what even a high-end workstation has. I think Ed's resources (both cores and RAM) need to be increased 8-fold in order for this to be feasible within 1 calendar year. Assuming Moore's law (doubling every 3 years nowadays) 5x5 will not happen this decade.

I wonder over something like that.I just don't know anything about it.Is there any online service - rent a supercomputer to do it, for example, in three months.

[Rein Halbersma]

I wonder over something like that.I just don't know anything about it.Is there any online service - rent a supercomputer to do it, for example, in three months.

Of course: https://aws.amazon.com/ec2/pricing/on-demand/

For the peak memory load you want r5b.24xlarge with 96 cores and 768 Gb RAM which costs $7 per hour. That would be enough to run 5v5 all kings and it should be doable to do that within a single day (~$175). But computing all the other 5v5 endgame databases will take ~14.4 months even with that supercomputer. So that's 7 x 14.4 / 12 x 8760 = ~$72K. Let me know if you're interested

So only for peak computations with very high memory demands (like all kings dbs) you want to rent something for a day. I think with Ed's super-finegrained slicing techniques, he could generate the 9 pieces dbs with his current workstations in a year. But for analysis purposes, balanced 10 piece dbs with equal number of pieces on both sides are much more useful. That would take 8 years with current technology (and a solveable bottleneck for peak memory load).

[BertTuyt]

Rein, I assume that at least from a memory point of view the 9p would be doable-ish.
My guestimate is that 256 GByte would be sufficient, and at least the Threadripper could address this.
Also with Threadripper mother boards with 8 DIMM slots and 32 GByte Dimm seems not to be impossible.
Last but not least the costs for 256 GByte would be between 1000 - 1200 euro.

Certainly expensive, but not totally out of reach (as this is also my wish next PC system, 32 cores )

Bert