EGTB generator

[gwiesenekker]

Hi,

The results of 4 kings vs 4 kings by the latest version (3.401), available on GitHub:

Code: Select all

generated 4wX-0wO-4bX-0bO.dtm: material=4 0 4 0 positions=37581505500 maximum-index=37581505499 passes=39 maximum-dtm=39 threads=16
self-consistency: passes=1 updates=0/0
final read-only consistency verification: threads=16 cache=8192 MiB total positions-checked=75163011000 positions-skipped=0
consistency current-DB cache:
  Metric                                Value
  Lookups                                   0
  Hits                                      0
  Misses                                    0
  Hit rate                              0.00%
  Decompressions                            0
  Dirty evictions                           0
  Compressed writes                         0
generator dependency caches:
  Metric                                Value
  Lookups                        202632056820
  Hits                           182197619298
  Misses                          20434437522
  Hit rate                             89.92%
  Decompressions                  17348537797
  Dirty evictions                           0
  Compressed writes                         0
final verification current-DB cache:
  Metric                                Value
  Lookups                        289514820826
  Hits                           288976583537
  Misses                            538237289
  Hit rate                             99.81%
  Decompressions                    498554521
  Dirty evictions                           0
  Compressed writes                         0
final verification dependency caches:
  Metric                                Value
  Lookups                        202632056820
  Hits                           182217224193
  Misses                          20414832627
  Hit rate                             89.93%
  Decompressions                  17325691054
  Dirty evictions                           0
  Compressed writes                         0
generator dependency caches by material (summed across workers):
  Database                            Lookups          Misses  Decompressions    Hit %      Full MiB Mode
  4wX-0wO-1bX-0bO.dtm             17907193560               0               0   100.00         40.41 resident
  4wX-0wO-2bX-0bO.dtm             71365116672        42998985        42073008    99.94        909.27 shared
  4wX-0wO-3bX-0bO.dtm            113359746588     20391438537     17306464789    82.01      13336.01 shared
final verification dependency caches by material (summed across workers):
  Database                            Lookups          Misses  Decompressions    Hit %      Full MiB Mode
  4wX-0wO-1bX-0bO.dtm             17907193560               0               0   100.00         40.41 resident
  4wX-0wO-2bX-0bO.dtm             71365116672               0               0   100.00        909.27 shared
  4wX-0wO-3bX-0bO.dtm            113359746588     20414832627     17325691054    81.99      13336.01 shared
optimistic dependency cache activity (whole run):
  Busy reads                           5341866
  Invalidated reads                      32351
  Publication conflicts                5258279
  Pages published                  40844011870
dependency memory at report time (shared allocations counted once):
  Database                        Full MiB  Payload MiB  Allocated MiB Coverage % Mode
  4wX-0wO-2bX-0bO.dtm               909.27       909.27         937.69     100.00 dense-lazy
  4wX-0wO-1bX-0bO.dtm                40.41        40.41          40.41     100.00 resident
  4wX-0wO-3bX-0bO.dtm             13336.01      2333.60        2406.53      17.50 cached
  Coverage is capacity/full size, not occupancy or hit rate. Allocated includes shared-slot metadata; excludes contexts and private views.
shared dependency residency: budget=4096 MiB maximum-database=512 MiB used=40.41 MiB loaded=1 cached=2
optimistic dependency caches: initial-database=64 MiB used=3242.88 MiB databases=2
adaptive shared-cache budget: 4096 MiB allocated=3344.21 MiB growths=6 (metadata included)
shared-cache growth stalls: 6.793300 s total
shared-cache coordinator: shrinks=0 transfers=0 rollbacks=0 recovery-reserve=0.00 MiB assessing=no
shared-cache growth admission: minimum-load-share=0.500% minimum-timed-loads=16 growth-factor=1.500 one-growth-per-checkpoint
WTM: wins=3916965780 losses=55858469 draws=33608681251
WTM DTM statistics:
     DTM            Frequency
     -36                   20
     -34                  188
     -32                  916
     -30                 2384
     -28                 2936
     -26                 4278
     -24                 6042
     -22                 7470
     -20                 9810
     -18                16866
     -16               132301
     -14               598460
     -12              1624982
     -10              1832427
      -8              6566184
      -6              7089677
      -4             18292002
      -2             19671526
      -1          33608681251
       1            577942958
       3            355101700
       5            384795448
       7            305759996
       9            514490445
      11           1075233279
      13            599088625
      15             69563208
      17             11516247
      19              7707348
      21              6309378
      23              4212676
      25              2233408
      27              1422964
      29              1011554
      31               382120
      33               132036
      35                50642
      37                10472
      39                 1276
BTM: wins=3916965780 losses=55858469 draws=33608681251
BTM DTM statistics:
     DTM            Frequency
     -36                   20
     -34                  188
     -32                  916
     -30                 2384
     -28                 2936
     -26                 4278
     -24                 6042
     -22                 7470
     -20                 9810
     -18                16866
     -16               132301
     -14               598460
     -12              1624982
     -10              1832427
      -8              6566184
      -6              7089677
      -4             18292002
      -2             19671526
      -1          33608681251
       1            577942958
       3            355101700
       5            384795448
       7            305759996
       9            514490445
      11           1075233279
      13            599088625
      15             69563208
      17             11516247
      19              7707348
      21              6309378
      23              4212676
      25              2233408
      27              1422964
      29              1011554
      31               382120
      33               132036
      35                50642
      37                10472
      39                 1276
DTM example positions:
  WTM longest win      W:WK23,K28,K48,K49:BK33,K43,K46,K47 {39}
  WTM longest loss     W:WK27,K38,K39,K41:BK15,K16,K20,K50 {-36}
  BTM longest win      B:WK33,K43,K46,K47:BK23,K28,K48,K49 {39}
  BTM longest loss     B:WK15,K16,K20,K50:BK27,K38,K39,K41 {-36}
  draw                 W:WK43,K44,K45,K46:BK47,K48,K49,K50 {-1}
storage: raw=150326022000 payload=9938147555 file=10945232659 bytes overall=7.28% (13.73:1)
wall-clock timings:
  setup                             0.048 s
  initialization                20335.486 s
  backpropagation                1884.287 s
  frontier compilation            112.729 s
  generation subtotal           22334.703 s
  verify + statistics/fallback  21340.013 s
  storage metadata                  0.414 s
  durable publication               0.002 s
  total                         43675.317 s

GW

[gwiesenekker]

FYI, the audit of the generation of all up to 7 piece EGTBs by version 3.401.

**All 134 databases match Michel Grimminck’s statistics—including every published mate-distance frequency for both WTM and BTM.**

I selected the latest completed run for each material from 476 logs. All selected runs use **revision 3.401**; no databases were missing.

### Correctness

- **268 WTM/BTM comparisons: no differences.**
- Win/draw/loss totals and full DTM histograms match Michel’s “All positions” tables, with moves converted to plies and BTM compared through mirroring.
- Every log reports **`passes=1 updates=0/0`**.
- Logged verification covers **1,172,183,085,800 side-positions**, with **nothing skipped**.

The [full audit report](/tmp2/gwies/endgame7/reports/2026-09-16-all-2-to-7-piece-audit.md) links every selected log and its two reference pages. This was a log audit, not another scan of the DTM files.

### Cache performance

| Seven-piece family | Databases | Summed runtime | Dependency hit rate: generation / verification |
|---|---:|---:|---:|
| 4×3 | 20 | 17h59m | 98.886% / 99.049% |
| 5×2 | 18 | 11h26m | 99.760% / 99.961% |
| 6×1 | 14 | 4h23m | 99.925% / 99.950% |

All 134 jobs together took **35h05m of summed job time**.

**One dependency is the clear hotspot:** `2wX-1wO-1bX-2bO.dtm`. It accounts for **17.06 billion decompressions—70.5% of all dependency decompressions**.

Its worst behaviour occurs in:

| Database being generated | Hot dependency hit rate, generation / verification | Hot dependency decompressions, both phases |
|---|---:|---:|
| `1wX-3wO-2bX-1bO` | 75.77% / 75.74% | 11.21 billion |
| `2wX-2wO-2bX-1bO` | 87.52% / 87.67% | 5.73 billion |

Those nearly unchanged verification miss rates show **persistent cache pressure, not merely warm-up**.

Other findings:

- **Growth stalls are small:** 443 seconds overall, about **0.35%** of runtime.
- Coordinator totals: **182 shrinks, 19 rollbacks, zero recorded transfers**. Idle reclamation works, but these runs do not demonstrate successful paired memory transfers.
- Current-database verification still incurred **5.35 billion decompressions**.
- Frontier compilation is now only **2.15% of total runtime**—that earlier bottleneck is substantially resolved.

### My recommendation

Benchmark those two hotspot jobs with more shared-cache capacity, or with the hot dependency resident. It needs **7,940 MiB uncompressed**, but the current **1,024 MiB per-database residency limit excludes it**.

In the worst job, the shared cache is almost full while the separate 8 GiB residency allowance uses only **2.28 GiB**. That makes **better allocation between the two budgets** a promising next step—without blindly increasing total memory consumption.

**Bottom line: correctness is excellent; the remaining dependency-cache problem is highly concentrated and gives us a concrete benchmark target.**

GW