Reinforcement learning

[Joost Buijs]

An experiment I've wanted to try for a while now is training an international draughts network using reinforcement learning. Since it's a one-off project and coding everything from scratch takes quite a bit of time, I just never got around to it. However, with the help of Claude, it’s now actually possible to run these kinds of experiments without it eating up an excessive amount of time.

Luckily, I had already written most of the core software needed for this experiment in the past:

A game runner that can play up to 32 games in parallel using multithreading.
A game converter that translates the games into labeled positions to use as input for the network trainer.
A network trainer that uses those labeled positions to optimize a neural network.

I asked Claude to build a wrapper around these routines so the whole setup could function as a reinforcement learning system.

Here is how the loop works:

A: Play 10 x 495 ballot3 games (4,950 games total) using only basic material evaluation.
B: Translate these games into a batch of labeled positions (roughly 460,000).
C: Use these positions as input for the trainer, and train a random network for 10 epochs using this batch.

For all subsequent iterations of 4,950 games, the neural network takes over as the evaluator instead of material value, and steps B and C are repeated. The games start out with 20K nodes per move. If the network hits a plateau and stops improving, the number of nodes per move is doubled until a specific maximum is reached.

So far, this setup is working surprisingly well! On my 32-core machine, I’m already getting a decent-playing network after just 5 iterations (about 33 Elo below Kingsrow). After 46 iterations (which took about 4 hours) it's already more or less on par with Kingsrow. Of course, to train a truly strong network, it’s going to need a lot more iterations (probably around 1,000).

I've attached a PDF with a full, detailed description of the whole project.