Years ago I developed an OpenMPI based EGTB generator that in principle could also generate 8-piece DTM EGTBs by partitioning the EGTBs by (man)rows. I generated a couple of 8-piece EGTBs but in the end it took too much time.
GWD uses a ZSTD page-compressed representation of the EGTB: each position has two 16-bit entries (not needed for up to 7-piece EGTBs, but required for 8-piece EGTBs): DTM white-to-move and DTM black-to-move. A page consists of 256 entries (1024 bytes). Pages are ZSTD block-compressed. When an entry is needed the page is uncompressed and stored in a cache (if not already available in the cache). The compressed size of all 2-,3-,4-,5- and 6-piece EGTBs is 22GB and loaded into RAM during game-play. A couple of years ago I added an uncompressed WDL representation that GWD now uses during game-play. The WDL representation is also around 22GB. When the root-position has transitioned into a known EGTB GWD uses the DTM representation. I know that the WDL representation can also be block-compressed to about 20%. It has always been on my to-do list to rewrite my EGTB generator for up to 7-pieces only (so I can use 8-bit entries) and to add the compressed WDL representation but the code was just too hard to modify.
Yesterday I recreated my EGTB generator from scratch using Codex in a couple of hours. I never succeeded to develop a compact EGTB index for positions without holes, but Codex could do that. You also need the inverse, so converting an index back to a position. The benchmark for indexing and inversion that Codex generated shows the following on my AMD 5950X:
Code: Select all
Pieces Largest material Positions Inversions/s Indices/s
2 WK1 BK1 2,450 41.57M 137.96M
3 WM1 WK1 BK1 105,840 35.34M 140.01M
4 WM1 BM1 WK1 BK1 4,478,160 34.45M 106.74M
5 WM1 BM1 WK2 BK1 102,997,680 32.75M 87.54M
6 WM1 BM1 WK2 BK2 2,317,447,800 31.03M 68.53M
7 WM1 BM2 WK2 BK2 45,793,430,100 29.38M 60.33MThe multi-threaded design that Codex proposed was very complex, also because of the cache for the uncompressed pages. But I already solved that problem in my EGTB generator: you allocate blocks of indices aligned at a page boundary to the threads called a slice. Each thread owns the entries corresponding to the slice. Backtracking uses won-in-N and won-in-at-most-N bitmaps, each thread creates its bitmap slice before backtracking. The full bitmap is used by all threads during backtracking. A thread only considers (and updates) predecessors that are part of its slice.
I have published the project on GitHub: https://github.com/gwiesenekker/GWDEGTB. The README has been generated by Codex.
The code has been developed and tested on Linux, you can use a Linux VM on Windows to compile and test it, or have Codex port the code to Windows if you want.
GW
