Internet engine matches

[BertTuyt]

In the 2nd match the opposite occurred.
In game 150 Ares could draw with 48. 34-30, but played 20-15 and lost.
Maybe good to check with both engines.

Bert

[Joost Buijs]

Krzysztof,

In the mean time I made a small change that makes the engine somewhat better at tactics. I'm testing it right now, when it is really better according to my tests, I will bump the version to 1.53d and send you the new version.

I'm also working on a simple GUI (Windows only) to make the engine easier to use, this will take some time because Draughts is not the primary thing I'm working on.

As requested I will take a look at making a SSE version for your old PC's, the drawback is that it will probably be slower.

The book that Ares uses is generated with a very old and weaker version of Ares, it contained some weaker lines, I tried to fix this by filtering these out, of course this is far from optimal. I simply didn't have the time to generate a new book with the latest version of Ares yet.

Joost

[Joost Buijs]

The win (and only win in this match) of Ares in game 123 is a little strange.
The move from KR 43. 20-15 was definitely wrong and losing, whereas 43. 28-22 27x18 44. 42-37 41x43 45. 49x38 is a draw.

I expect that with 16-cores KR would not have search problems.
Ed, can you give some light, is this a bug, or something you also can not reproduce?

Krzysztof, anyway thanks for organizing and sharing.

Bert

It just happens that engines sometimes overlook something because of futility pruning or LMR, even on 16 or 32 cores, better move ordering could improve this. In my experience history (like I'm using in Ares) doesn't work very well with Draughts, maybe a Policy network (like you are already working on) could improve this. Although engines are very strong at the moment they are not infallible yet. It is my gut feeling that there still is room for improvement.

Determining engine strength by playing matches against one or two engines only is definitively a failure. In chess I often see engine A always beating engine B in a match, while in a gauntlet engine B always ends on top. So playing gauntlets against a bunch of different engines (also weaker ones) is in my view more accurate. Unfortunately there are for Draughts no tools available to do things like this in a convenient way.

[Krzysztof Grzelak]

Krzysztof,

In the mean time I made a small change that makes the engine somewhat better at tactics. I'm testing it right now, when it is really better according to my tests, I will bump the version to 1.53d and send you the new version.

I'm also working on a simple GUI (Windows only) to make the engine easier to use, this will take some time because Draughts is not the primary thing I'm working on.

As requested I will take a look at making a SSE version for your old PC's, the drawback is that it will probably be slower.

The book that Ares uses is generated with a very old and weaker version of Ares, it contained some weaker lines, I tried to fix this by filtering these out, of course this is far from optimal. I simply didn't have the time to generate a new book with the latest version of Ares yet.

Joost

I understand and thanks for the answer Joost.

[Ed Gilbert]

The win (and only win in this match) of Ares in game 123 is a little strange.
The move from KR 43. 20-15 was definitely wrong and losing, whereas 43. 28-22 27x18 44. 42-37 41x43 45. 49x38 is a draw.

I expect that with 16-cores KR would not have search problems.
Ed, can you give some light, is this a bug, or something you also can not reproduce?

My 16-core PC is not available at the moment, but I looked at it with a single thread. It takes about 7 seconds for kingsrow to switch from 20-15 to 28-22, so it's not that surprising that kingsrow could not see that with 16 threads given the short time controls that give an average 0.5 sec/move. I've also seen that when using a lot of parallel threads, the nodes/sec speed of YBW increases with longer search times. It's not very good with search times under 1 second. This could be a case where lazy SMP is better than YBW. Doesn't lazy SMP also tend to broaden the search somewhat compared to a single thread, more so than YBW? I would think it would, but I haven't played with lazy SMP so I don't really know if that's true. This position may be a case where the 28-22 path is reduced at first, making it take longer to find.

-- Ed

[Krzysztof Grzelak]

Match KINGSROW - ARES (2-move ballots)

Kingsrow 1.63 vs Ares 1.53c 0 wins, 0 losses, 158 draws, 0 unknowns

Kingsrow 1.63 x64

Threads = 16
Opening Book = Best Moves
Pondering = On
HashTable Size = 512 MB
The base ends = 6 Pieces
Time = 5 Min / 100 Moves

Ares 1.53c x64

Threads = 16
Book = On
TT-size = 32
Bucket-size = 4
Ponder = On
The base ends = 6 Pieces
Time = 5 Min / 100 Moves

Match played on a computer with the equipment.

Processor - AMD Ryzen Threadripper 1950X
Hard disc - Samsung 860 Pro SSD 1 TB
Memory of frames - 128 GB DDR4 2400
System - Windows 10 64 bit Pro

[Ed Gilbert]

My 16-core PC is not available at the moment, but I looked at it with a single thread. It takes about 7 seconds for kingsrow to switch from 20-15 to 28-22, so it's not that surprising that kingsrow could not see that with 16 threads given the short time controls that give an average 0.5 sec/move. I've also seen that when using a lot of parallel threads, the nodes/sec speed of YBW increases with longer search times. It's not very good with search times under 1 second. This could be a case where lazy SMP is better than YBW. Doesn't lazy SMP also tend to broaden the search somewhat compared to a single thread, more so than YBW? I would think it would, but I haven't played with lazy SMP so I don't really know if that's true. This position may be a case where the 28-22 path is reduced at first, making it take longer to find.

The above tests were done using only a 6pc db, the same conditions as the match. I just looked at it using the 8pc db, and using that kingsrow sees 28-22 instantly. The kingsrow eval has no positional terms for kings, only material, so it really relies on the egdb to play endgames well. That is a big difference between the kingsrow eval and NNUE which does have positional king inputs. I don't know why Krzysztof configured kingsrow that way for the match.

-- Ed

[Krzysztof Grzelak]

My 16-core PC is not available at the moment, but I looked at it with a single thread. It takes about 7 seconds for kingsrow to switch from 20-15 to 28-22, so it's not that surprising that kingsrow could not see that with 16 threads given the short time controls that give an average 0.5 sec/move. I've also seen that when using a lot of parallel threads, the nodes/sec speed of YBW increases with longer search times. It's not very good with search times under 1 second. This could be a case where lazy SMP is better than YBW. Doesn't lazy SMP also tend to broaden the search somewhat compared to a single thread, more so than YBW? I would think it would, but I haven't played with lazy SMP so I don't really know if that's true. This position may be a case where the 28-22 path is reduced at first, making it take longer to find.

The above tests were done using only a 6pc db, the same conditions as the match. I just looked at it using the 8pc db, and using that kingsrow sees 28-22 instantly. The kingsrow eval has no positional terms for kings, only material, so it really relies on the egdb to play endgames well. That is a big difference between the kingsrow eval and NNUE which does have positional king inputs. I don't know why Krzysztof configured kingsrow that way for the match.

-- Ed

I wanted the ending bases to be equal to both programs.

[Ed Gilbert]

I wanted the ending bases to be equal to both programs.

Yes, that is clear, but the question is, what are you trying to measure with these engine matches? If it is to measure the relative strength of engine A vs engine B in normal match conditions, it's not doing that, because kingsrow uses an 8pc db in normal match conditions.

[Krzysztof Grzelak]

I wanted the ending bases to be equal to both programs.

Yes, that is clear, but the question is, what are you trying to measure with these engine matches? If it is to measure the relative strength of engine A vs engine B in normal match conditions, it's not doing that, because kingsrow uses an 8pc db in normal match conditions.

I will write like this - of course the program has 8pc db. But when using 6pc db the should play as well as 8pc db.

[Joost Buijs]

The win (and only win in this match) of Ares in game 123 is a little strange.
The move from KR 43. 20-15 was definitely wrong and losing, whereas 43. 28-22 27x18 44. 42-37 41x43 45. 49x38 is a draw.

I expect that with 16-cores KR would not have search problems.
Ed, can you give some light, is this a bug, or something you also can not reproduce?

My 16-core PC is not available at the moment, but I looked at it with a single thread. It takes about 7 seconds for kingsrow to switch from 20-15 to 28-22, so it's not that surprising that kingsrow could not see that with 16 threads given the short time controls that give an average 0.5 sec/move. I've also seen that when using a lot of parallel threads, the nodes/sec speed of YBW increases with longer search times. It's not very good with search times under 1 second. This could be a case where lazy SMP is better than YBW. Doesn't lazy SMP also tend to broaden the search somewhat compared to a single thread, more so than YBW? I would think it would, but I haven't played with lazy SMP so I don't really know if that's true. This position may be a case where the 28-22 path is reduced at first, making it take longer to find.

Indeed, YBW is not so good at shallow depths because you have to split several plies from the leaves to keep the overhead under control, at low depths it effectively does no SMP at all. Many available threads and small number of available moves (like often happens with Draughts) is also a problem with YBW.

Lazy-SMP has the tendency to broaden the search much more than YBW does, this can help to mitigate errors that occur due to LMR, at least that is the idea. Nowadays most chess engines use lazy-SMP, with more than 8 threads it usually performs better.

[Joost Buijs]

Krzysztof,

In the mean time I made a small change that makes the engine somewhat better at tactics. I'm testing it right now, when it is really better according to my tests, I will bump the version to 1.53d and send you the new version.

I'm also working on a simple GUI (Windows only) to make the engine easier to use, this will take some time because Draughts is not the primary thing I'm working on.

As requested I will take a look at making a SSE version for your old PC's, the drawback is that it will probably be slower.

The book that Ares uses is generated with a very old and weaker version of Ares, it contained some weaker lines, I tried to fix this by filtering these out, of course this is far from optimal. I simply didn't have the time to generate a new book with the latest version of Ares yet.

Joost

I understand and thanks for the answer Joost.

Although the engine does somewhat better a tactics I don't see any difference in strength, it remains about the same. I will run some more tests to see if I can get a conclusive result. The performance mainly seems to depend upon the quality of the network, and not so much upon the search.

[Krzysztof Grzelak]

Match DAMAGE - ARES (2-move ballots)

Damage 16.1 NNUE vs Ares 1.53c NNUE 0 wins, 3 losses, 155 draws, 0 unknowns

Damage 16.1 NNUE x64

Book = 0n
Threads = 16
NNUE-file = nn_20210328.gnn
TT-persistent = On
TT-size = 24
BB-cache = 5
BB-size = 6
BB-preload = 6
The base ends = 6 Pieces
Time = 1 Min / 120 Moves


Ares 1.53c NNUE x64

Threads = 16
Book = On
TT-size = 32
Bucket-size = 4
The base ends = 6 Pieces
Time =1 Min / 120 Moves

Match played on a computer with the equipment.

Processor - AMD Ryzen Threadripper 1950X
Hard disc - Samsung 860 Pro SSD 1 TB
Memory of frames - 128 GB DDR4 2400
System - Windows 10 64 bit Pro

[Joost Buijs]

Hi Krzysztof,

There is a new Ares version 1.53d on the FTP-server. In my tests it is about 4 Elo stronger than the previous version. The N/s went down somewhat, this is because it enters the quiescence search less frequently, and not because the engine is slower.

Joost

[Krzysztof Grzelak]

Hi Krzysztof,

There is a new Ares version 1.53d on the FTP-server. In my tests it is about 4 Elo stronger than the previous version. The N/s went down somewhat, this is because it enters the quiescence search less frequently, and not because the engine is slower.

Joost

Thank you Joost.