Flits self-learning mode

[Sidiki]

I want to ask, i suppose that you have Flits, if it's the case, please, whitout the 2 moves or 3 ballots dxp system, in regular game, in 10 games what's the draw rate of Flits on your computer?

It is pointless to speak of the draw rate in 10 games as the number of games is far too low. I played a quick match Maximus-Flits on my laptop, 100 games from the initial position (no ballots), 50/50 white/black, 75 moves in 1 minute. Flits using pondering, Maximus using 4 threads and no pondering. Results from Maximus' point of view: 19 wins, 1 loss, 77 draws, 3 unknown. No doubt Kingsrow will do better. Is this what you mean? (Games with more time will give more draws but Maximus can be set to use more threads and pondering.)

Jan-Jaap

Hi Jaap

Yes, but with this book, time and core haven't any kind of importance. To win a draughts game among computers world, you must take advantage on your opponent into the 20 first moves. and this file it's for the moment done to resist to Scan and Kingsrow into the 20 first moves.

Friendly, Sidiki.

[jj]

Hi Bert

i sent a mail to Ed, Jan Jaap and you icluding the file.
As mentionned; there's no any intention behind this work, just for fun.

I am waiting for your tests ie Ed, Jaap and you.

Friendly, Sidiki.

Using the same settings as before, results of Maximus vs Flits+SidikiBook: 19 wins, 3 losses, 75 draws, 3 unknown. So Flits did a little bit better but not significantly. I understand that the book is not yet trained for Maximus. I'm curious if there will be a significant difference against Kingsrow.

[BertTuyt]

Out of curiosity Damage (recent version, not yet posted), played a 158 games match (2-ballot move) against Flits (not the Hub version which I do not have), with the flits-dxp-server and the new openings book.

Conditions 1 core each (so did not use the multi-core option of Damage), 1 min / 75 moves, Flits 256 M Byte Hash and 6p DB.
Damage used the 7p DB, and a 512 M Byte Hash (2 ^ 25 entries).

Results (perspective Damage), 63 Win, 0 lost and 95 draw, which translates into an ELO difference of around 147.

So below Kingsrow (as expected), and above Maximus.

As my processor is faster (as the system Ed uses) in single core events (4.3 GHZ) this could favor the weaker program (in this case Flits), but still think Kingsrow is just the better program (kudos to Ed !!).


Bert

[jj]

The point is to play only from the initial position and not to use ballots. Then to compare the results against Flits-book with the results against Flits+book. But really this applies only to Kingsrow and Scan at the moment.

[Sidiki]

The point is to play only from the initial position and not to use ballots. Then to compare the results against Flits-book with the results against Flits+book. But really this applies only to Kingsrow and Scan at the moment.

Exactly, and i will do the 2 moves ballots soon, before i wil release before tomorrow the book for all ie Kingsrow, Scan, Damage and Maximus.

PS: only for initial position, for 2 moves ballots will come just after.

Thanks, Sidiki

[Sidiki]

Out of curiosity Damage (recent version, not yet posted), played a 158 games match (2-ballot move) against Flits (not the Hub version which I do not have), with the flits-dxp-server and the new openings book.

Conditions 1 core each (so did not use the multi-core option of Damage), 1 min / 75 moves, Flits 256 M Byte Hash and 6p DB.
Damage used the 7p DB, and a 512 M Byte Hash (2 ^ 25 entries).

Results (perspective Damage), 63 Win, 0 lost and 95 draw, which translates into an ELO difference of around 147.

So below Kingsrow (as expected), and above Maximus.

As my processor is faster (as the system Ed uses) in single core events (4.3 GHZ) this could favor the weaker program (in this case Flits), but still think Kingsrow is just the better program (kudos to Ed !!).


Bert

I am surprised of this number of draws with 2 moves ballots, but it's for initial position at this time and against Kingsrow and Scan. As i write to Jaap, the 2 moves ballots book will come soon and will include all the program.
But before, i will release the book (initial position): Kingsrow, Scan, Damage, Maximus tomorrow.

PS: once again, it's only for fun.

Friendly, Sidiki

[jj]

PS: once again, it's only for fun.

Well it could be a gain of insight if it works as you say. But scientists want to see significant proof first.

[BertTuyt]

I know that in Chess opening books play an important role, I have (indeed) no evidence yet that this also applies to Computer Draughts.
On the other hand I see that Damage struggles with specific openings (in the 2 - move ballot).
As an example the 1st 2-move ballot with Damage as black to move, where both Scan as Kingsrow create at the beginning a wing-lock, is very difficult for Damage.
So you could indeed create a book to find a way out, or avoid these openings (when you play free style without ballots).

So far I did not start with this, as I first want Damage to narrow the gap (with the famous 2) on its own strength, before doing so.

Also what I don't know if Scan and Kingsrow randomize root-move order after the opening book phase.
I know they do it while in the book, but not 100% sure they still do it after the book.
Because when you apply random root-move order after the book phase than the effects of a book could be reduced.

Bert

[Sidiki]

I know that in Chess opening books play an important role, I have (indeed) no evidence yet that this also applies to Computer Draughts.
On the other hand I see that Damage struggles with specific openings (in the 2 - move ballot).
As an example the 1st 2-move ballot with Damage as black to move, where both Scan as Kingsrow create at the beginning a wing-lock, is very difficult for Damage.
So you could indeed create a book to find a way out, or avoid these openings (when you play free style without ballots).

So far I did not start with this, as I first want Damage to narrow the gap (with the famous 2) on its own strength, before doing so.

Also what I don't know if Scan and Kingsrow randomize root-move order after the opening book phase.
I know they do it while in the book, but not 100% sure they still do it after the book.
Because when you apply random root-move order after the book phase than the effects of a book could be reduced.

Bert

Exactly, at the end of the book phase, if the book it's smartly and deeply done, you will be very safe after his end. Because, if it's focused on good moves that avoid losing a piece or get locked on the wing or in the center, you don't need to be afraid of your book.

Sidiki

[Sidiki]

If all go well, i will post/send the Maximus/Damage file before 0:00 GMT

Sidiki

[Ed Gilbert]

I don't fully understand how this book was created, and I'm not sure that I think this is an acceptable thing to do from a "fair competition" point of view. You could probably copy the entire kingsrow opening book off 1.5M positions with a little bit of programming, and use it in another engine. Do the programmers think that would be acceptable? What is the essential difference between coping an opening book and what is being done to create this flits book? As a practical matter, I don't think the opening book is very important for 10x10 engines, and I would be surprised if flits became very much stronger if it had a large opening book. But still, I would like to understand how this book is being created.

-- Ed

[BertTuyt]

Interesting question what is fair.
As Kingsrow and Scan are freely available, you could play zillions of games, and prepare a dedicated opening book for this purpose (not for winning, but avoiding losses).
Assume this is common practice in computer chess.
I wont do it anyway, .....

Bert

[jj]

Although I can imagine how to create such an opening book with Flits, yes can you elaborate on that, Sidiki? Also, can you send me the anti-Maximus book for Flits so I can test it?

Ed, can you share the results of Kingsrow vs Flits+SidikiBook? Like you, I would be surprised if Flits became very much stronger but why leave it up to speculation?

Sidiki doing this for fun and for personal use is fine, and even sharing the -perhaps surprising- results. Adding this book to Flits to compete in Krzysztof's tournament is another matter. Maybe opening books should have authors too, like in computer chess. TCEC uses human-selected opening positions which are played with both colors, and without books I think.

In general, testing against available engines to improve your own is common practice, and that includes the opening book, I suppose (I am not there yet). But yes, taking this to the extreme it starts to look like reverse engineering. To copy the entire Kingsrow opening book with a little bit of programming and use it in another engine is of course not acceptable. Where to draw the line? And the problem with forbidding things is how to enforce them. A possibility is to play a tournament without opening books, provided all programs have this option.

I think the matter of fairness becomes a matter of whether engines are publicly available and whether they play public matches. Anybody can download Kingsrow, Scan, Damage, a few people have Maximus, less people have Sjende Blyn, etc. Maximus played a number of public matches, Sjende Blyn effectively refuses to play matches. The more private an engine is, the more advantage it potentially has to public engines.

Personally, I don't offer Maximus for public download (yet) because of the extra work it generates. Such as making it user proof, making documentation and keeping it up to date, disabling research features, etc. For the average user installing 64-bit Java and making sure Maximus uses the correct Java version (not 32-bit) is challenging and they need support -sometimes not in English- and every user has a few questions, requests for change, etc. Maybe I will publish a hub version of Maximus when I get around to that. For those interested, I am currently working on an update of the Maximus app for iOS/Android which will have a completely new user interface.

Another thing about "fair competition" is the fact that Ed shared his optimization program only with Bert and with no-one else. With this program and the explanation offered by Ed every programmer can generate a very strong evaluation function in a short period of time. In my opinion Bert now has an advantage over other programmers who don't have access to the optimization program. Bert did share the games he generated but not the optimization program, this is Ed's to share of course. The generated games are not that useful from a research point of view, as they were generated by a black box using a simplified Kingsrow evaluation function. Evaluation functions derived from these games indirectly derive from Kingsrow evaluation and with the use of Ed's optimization program we might speak of "Damage powered by Kingsrow". What do the other programmers think, is this acceptable?

I am not asking for myself because I generated my own games from zero knowledge and made my own optimization program. This I did using the public information given by Fabien and Ed on this forum and by studying neural network theory. I have a few small innovations (some of which are also used by Bert now) but in spirit Maximus' evaluation function is indebted to Scan 3.0 and up for future research. In 2018 Ed and I compared notes, we have many differences so a comparison would still be interesting. Understanding this subject and getting it all to work is a rite of passage as Fabien calls it and it is a lot of work. In my opinion, giving one person access to such a powerful tool and not making it publicly available makes for an unfair competition. It is unfair to the people who don't have access to an optimization program and in a way also to the people who put in the work to make their own.

Jan-Jaap

[Krzysztof Grzelak]

I wonder what the new book for the program will give Flits. The book ends sometime and the engine must think for itself. Only in the initial phase will the engine save time thinking.

[Ed Gilbert]

JJ, thank you for your comments.

Ed, can you share the results of Kingsrow vs Flits+SidikiBook? Like you, I would be surprised if Flits became very much stronger but why leave it up to speculation?

I didn't run the experiment, for several reasons.
- I didn't fully understand how to enable and disable this additional book. From the video it looks like you just click something in a menu, and that does something. Can it be undone, or turned off? It doesn't seem to need to be told the filename of the additional book file, from what I could see.
- I ran the first brief match using the hub version of flits, and I don't know if that hub version supports opening books.
- I kind of lost interest in the experiment when I heard what the self-learning was all about. I agree that doing this just for fun, as an experiment to see how much flits can improve with an "anti-kingsrow" book is perfectly acceptable. I guess the point of my post, although maybe I didn't make it very well, is that it doesn't feel right to make a book that specifically counters every move of another program's book, and then use that in competition, as that sounds like it's basically a form of copying a book. It's just my opinion, and I was interested to see what other engine developers thought about it.

Although I'm not planning to do it, Sidiki or anyone else can download kingsrow and run the experiment.

Another thing about "fair competition" is the fact that Ed shared his optimization program only with Bert and with no-one else. With this program and the explanation offered by Ed every programmer can generate a very strong evaluation function in a short period of time. In my opinion Bert now has an advantage over other programmers who don't have access to the optimization program.

I never thought about it that way, but I can see your point. I shared the code with Bert because he asked me for it, and we had collaborated on other projects in the past, like the dxp wrappers for truus and flits, and Bert had shared some of his language localization code with me. I don't think there's anything proprietary or new in the code. Gradient descent is a well documented algorithm, and there's a lot of information available describing how to implement it. It is a lot of work though to develop and debug, or at least it was for me. Perhaps I should think about publishing the source.

-- Ed