Each match consisted of 12,640 games at 75 moves in 60 seconds time control, GAYP start position. Matches were played on a 16-core Xeon. Each match took about 1 day to complete. I will append the detailed results per core below, along with all the engine settings, but here is the bottom line result:
books on: kingsrow vs scan: +2.3 elo
kr book off: kingsrow vs scan +1.2 elo
So a 1.1 elo drop in kingsrow performance when its opening book is disabled. That's a pretty small difference, and I'm not even sure that 12.6K games is enough to measure that small a difference with any kind of confidence.
I also ran a test to see how many book moves each engine makes on average in a GAYP match. For this test I played another match of around 2000 games, and I used the -t option in tmgr to record the move times in the pdn results. For each engine I made a histogram of the number of book moves made. I assumed that quick moves under 150 msec were book moves. This is not a perfect way to detect book moves, as it also picks up forced capture moves, but these probably average out the same for both engines, so the difference in book depths should still be accurate.
The column headings don't line up very well below, but the table gives the counts of the number of games that each engine had the various number of book moves. For example, scan played 19 games where it dropped out of book after the 3rd move, 348 games where it dropped out of book after the fourth move, etc.
Code: Select all
book moves scan kingsrow
0 0 0
1 0 0
2 0 0
3 19 0
4 348 36
5 566 26
6 882 136
7 463 482
8 143 1067
9 157 642
10 32 220
11 0 4
12 0 0
13 3 0
The average number of book moves for kingsrow is 8.0
-----------------------------------------------------------------
Match stats Kingsrow 1.61 vs. Scan 3.1 (both engines books enabled)
[ 1]: 0.503 score, 790 games, 11 wins, 6 losses, 773 draws, 0 unk
[ 2]: 0.504 score, 790 games, 10 wins, 3 losses, 777 draws, 0 unk
[ 3]: 0.500 score, 790 games, 10 wins, 10 losses, 770 draws, 0 unk
[ 4]: 0.506 score, 790 games, 10 wins, 1 losses, 779 draws, 0 unk
[ 5]: 0.501 score, 790 games, 9 wins, 7 losses, 773 draws, 1 unk
[ 6]: 0.505 score, 790 games, 11 wins, 3 losses, 776 draws, 0 unk
[ 7]: 0.504 score, 790 games, 9 wins, 2 losses, 779 draws, 0 unk
[ 8]: 0.500 score, 790 games, 5 wins, 5 losses, 780 draws, 0 unk
[ 9]: 0.501 score, 790 games, 8 wins, 7 losses, 774 draws, 1 unk
[10]: 0.502 score, 790 games, 9 wins, 6 losses, 775 draws, 0 unk
[11]: 0.504 score, 790 games, 10 wins, 4 losses, 776 draws, 0 unk
[12]: 0.501 score, 790 games, 8 wins, 7 losses, 775 draws, 0 unk
[13]: 0.501 score, 790 games, 9 wins, 8 losses, 773 draws, 0 unk
[14]: 0.508 score, 790 games, 14 wins, 2 losses, 774 draws, 0 unk
[15]: 0.506 score, 790 games, 17 wins, 7 losses, 766 draws, 0 unk
[16]: 0.506 score, 790 games, 17 wins, 7 losses, 766 draws, 0 unk
total 0.503 score, 12640 games, 167 wins, 85 losses, 12386 draws, 2 unk
elo diff 2.3
LOS 1.00
Match stats Kingsrow 1.61 vs. Scan 3.1 (kingsrow book disabled, scan book enabled)
[ 1]: 0.499 score, 790 games, 6 wins, 8 losses, 775 draws, 1 unk
[ 2]: 0.500 score, 790 games, 6 wins, 6 losses, 778 draws, 0 unk
[ 3]: 0.501 score, 790 games, 5 wins, 4 losses, 781 draws, 0 unk
[ 4]: 0.506 score, 790 games, 15 wins, 6 losses, 768 draws, 1 unk
[ 5]: 0.507 score, 790 games, 18 wins, 7 losses, 765 draws, 0 unk
[ 6]: 0.498 score, 790 games, 6 wins, 9 losses, 774 draws, 1 unk
[ 7]: 0.501 score, 790 games, 8 wins, 7 losses, 775 draws, 0 unk
[ 8]: 0.502 score, 790 games, 11 wins, 8 losses, 770 draws, 1 unk
[ 9]: 0.496 score, 790 games, 7 wins, 13 losses, 770 draws, 0 unk
[10]: 0.501 score, 790 games, 10 wins, 8 losses, 771 draws, 1 unk
[11]: 0.500 score, 790 games, 6 wins, 6 losses, 778 draws, 0 unk
[12]: 0.506 score, 790 games, 13 wins, 4 losses, 773 draws, 0 unk
[13]: 0.501 score, 790 games, 7 wins, 5 losses, 778 draws, 0 unk
[14]: 0.504 score, 790 games, 13 wins, 7 losses, 770 draws, 0 unk
[15]: 0.503 score, 790 games, 15 wins, 10 losses, 765 draws, 0 unk
[16]: 0.503 score, 790 games, 11 wins, 6 losses, 773 draws, 0 unk
total 0.502 score, 12640 games, 157 wins, 114 losses, 12364 draws, 5 unk
elo diff 1.2
LOS 1.00
tmgr.ini
---------
[General]
start-pos-file = d:\intl_book\gayp.pdn
games = 790
parallel-matches = 16
time = 60
moves = 75
#inc = 0.5
egdb-wld-path = d:\db_intl\wld_v2
[Engine]
engine-dir = d:\kr_hub
engine-command = d:\kr_hub\kr_hub
draw-count = 5
draw-pieces = 8
draw-score = 0.07
win-score = 20
set-param egdb-wld-pieces = 8
set-param egdb-wld-mem = 5500
set-param egdb-wld-path = d:\db_intl\wld_v2
set-param threads = 1
set-param tt-mem = 256
set-param tt-load = false
set-param book = false
set-param book-quality = best
[Engine]
engine-command = "\projects\scan_31\scan.exe hub"
engine-dir = \projects\scan_31
draw-count = 5
draw-pieces = 7
draw-score = 0.0
win-score = 80
set-param tt-size = 24
set-param bb-size = 6
set-param book = true
set-param book-ply = 4
set-param book-margin = 2
set-param threads = 1
