Reinforcement learning

[Joost Buijs]

An experiment I've wanted to try for a while now is training an international draughts network using reinforcement learning. Since it's a one-off project and coding everything from scratch takes quite a bit of time, I just never got around to it. However, with the help of Claude, it’s now actually possible to run these kinds of experiments without it eating up an excessive amount of time.

Luckily, I had already written most of the core software needed for this experiment in the past:

A game runner that can play up to 32 games in parallel using multithreading.
A game converter that translates the games into labeled positions to use as input for the network trainer.
A network trainer that uses those labeled positions to optimize a neural network.

I asked Claude to build a wrapper around these routines so the whole setup could function as a reinforcement learning system.

Here is how the loop works:

A: Play 10 x 495 ballot3 games (4,950 games total) using only basic material evaluation.
B: Translate these games into a batch of labeled positions (roughly 460,000).
C: Use these positions as input for the trainer, and train a random network for 10 epochs using this batch.

For all subsequent iterations of 4,950 games, the neural network takes over as the evaluator instead of material value, and steps B and C are repeated. The games start out with 20K nodes per move. If the network hits a plateau and stops improving, the number of nodes per move is doubled until a specific maximum is reached.

So far, this setup is working surprisingly well! On my 32-core machine, I’m already getting a decent-playing network after just 5 iterations (about 33 Elo below Kingsrow). After 46 iterations (which took about 4 hours) it's already more or less on par with Kingsrow. Of course, to train a truly strong network, it’s going to need a lot more iterations (probably around 1,000).

I've attached a PDF with a full, detailed description of the whole project.

[BertTuyt]

Joost, a very interesting result. Really curious where this will end. Without any doubt, Claude could also program the game runner, game converter and network training tools. So basically only the Draughts program remains.
Think we really grow towards a "game, set and match" situation is several years.
But anyway in the meantime it was fun....

Bert

[Joost Buijs]

Bert,

I didn't expect it to play a reasonable game after just 5 iterations, which amounted to 24,750 games at 20k nodes per move. Naturally, the process is slowing down now because it doubles the nodes per move whenever the network's performance plateaus.

It has currently completed 80 iterations and still shows signs of improvement, though that doesn't guarantee it will actually play any stronger. I plan to let it run for a few more days just to see how the network behaves with extended training. That said, a draw is still a draw—there is no changing that.

Joost

[BertTuyt]

Another way of looking at it, the computer was able to learn high-level draughts by just playing against itself and started with a mediocre level.
It has never seen any game by humans.
And after 4 hours it reached almost world champion level (or at least grandmaster++).
Many years the verdict was, the computer only copies what the programmer told him/her to do, and therefore can never be better then its creator.
Think we reach a new era. Even mathematics is no longer safe, and the only question is "who/what is next".

Bert

[Joost Buijs]

To be honest, I believe that the current top 10 International draughts engines will never lose a match against the world champion. In fact, there is a very high chance these engines would win, even though the game is notoriously drawish. This shows that a computer can reach a level through just a few hours of self-play that likely surpasses the human world champion.

Of course, a search engine is also part of the equation and needs to be developed too; a neural network alone is incapable of playing a single game.

Joost