jj wrote: Thu May 21, 2020 19:21
In general, testing against available engines to improve your own is common practice, and that includes the opening book, I suppose (I am not there yet). But yes, taking this to the extreme it starts to look like reverse engineering. To copy the entire Kingsrow opening book with a little bit of programming and use it in another engine is of course not acceptable. Where to draw the line? And the problem with forbidding things is how to enforce them. A possibility is to play a tournament without opening books, provided all programs have this option.
I think there is a big difference between directly interfacing (through de-compilation or other white box techniques) the Kingsrow book (with the positions and Kingsrow eval scores) that comes with its public version and black box reverse engineering. The former is against tournament rules, but the latter requires a serious programmer effort: writing a testing framework, building your own opening variation tree, using your own eval and dropout expansion to refine the book etc. I don't see anything wrong with that.
I think the matter of fairness becomes a matter of whether engines are publicly available and whether they play public matches. Anybody can download Kingsrow, Scan, Damage, a few people have Maximus, less people have Sjende Blyn, etc. Maximus played a number of public matches, Sjende Blyn effectively refuses to play matches. The more private an engine is, the more advantage it potentially has to public engines.
This is true, but the same is true for human play. Sijbrands has 2000+ games in Turbo Dambase, and anyone preparing for a match against him can use that. In tournament play with serious stakes, the top chess programs also devote serious CPU hours to opening preparation beyond what is publicly available. But opening preparation in draughts is overrated in any case.
Another thing about "fair competition" is the fact that Ed shared his optimization program only with Bert and with no-one else. With this program and the explanation offered by Ed every programmer can generate a very strong evaluation function in a short period of time. In my opinion Bert now has an advantage over other programmers who don't have access to the optimization program. Bert did share the games he generated but not the optimization program, this is Ed's to share of course. The generated games are not that useful from a research point of view, as they were generated by a black box using a simplified Kingsrow evaluation function. Evaluation functions derived from these games indirectly derive from Kingsrow evaluation and with the use of Ed's optimization program we might speak of "Damage powered by Kingsrow". What do the other programmers think, is this acceptable?
Yes, I have absolutely no problems with it. First of all, the optimization program is not part of the engine itself, but merely tooling to create a strong eval function. Scan doesn't release it's own ML pipeline used to compute the eval weights. Authors of different engines can collaborate on this, either by exchanging ideas or even full source code. This is entirely different from copying another engine's weight files (that is against tournament rules).
I am not asking for myself because I generated my own games from zero knowledge and made my own optimization program. This I did using the public information given by Fabien and Ed on this forum and by studying neural network theory. I have a few small innovations (some of which are also used by Bert now) but in spirit Maximus' evaluation function is indebted to Scan 3.0 and up for future research. In 2018 Ed and I compared notes, we have many differences so a comparison would still be interesting. Understanding this subject and getting it all to work is a rite of passage as Fabien calls it and it is a lot of work. In my opinion, giving one person access to such a powerful tool and not making it publicly available makes for an unfair competition. It is unfair to the people who don't have access to an optimization program and in a way also to the people who put in the work to make their own.
Well, I'm sure Ed / Bert collaborating and exchanging such tooling is mutually beneficial to both of them. They have a long history of fruitful collaboration (the whole DXP tooling around Flits/Truus e.g.) Why would one of them systematically give the other tools that put him at a disadvantage? But even if that were true, tough luck! And sure, that puts other authors at a disadvantage. It's the same when a small group of human GMs work together in training to mutually improve at the detriment of their competitors. Unless you can add your own unique added value, you have no right to be included in such a training group. Similarly, anyone looking for a handout on ML optimization tooling, should have something worth trading for.
And for years, Ed had a huge competitive advantage from his decades of endgame building experience. After Bert, Michel and Gerard caught up with him and managed to build 7 and 8 pc dbs, Ed voluntarily open sourced the Kingsrow drivers to his dbs. But not (yet?) the db building tools. Maybe the same will happen with ML tools, or maybe not.
I won't be using any handwritten gradient descent code from any draughts author however. The publicly available professional libraries (TensorFlow, PyTorch, or even SciPy) are far superior to what any of us can write here (and yes, I have written gradient descent code in my studies, 25 years ago). If I ever get around to it (into Stratego nowadays, little time for draughts), I would release any tooling that transforms draughts positions into a format that can be used by such professional ML libraries (since all my code is open source anyway).
).