ENGINE SPEED AND STRONG

[Joost Buijs]

Really I don't know what you mean, every line of code is original and written by me.

Code is slowly becoming obsolete. This is the ML era; data is everything. Replace my data with zeroes (AAAA with your encoding), and you can claim authorship. Not before.

Haha, than I think we have a different opinion, without coding your data won't go anywhere, remove the search from your program and you will see how good it plays.

[Fabien Letouzey]

Haha, than I think we have a different opinion, without coding your data won't go anywhere, remove the search from your program and you will see how good it plays.

Program = code + data. If the data isn't yours, how can the program be? I own both Scan's code and data, so I can call it "my program".

[Joost Buijs]

Haha, than I think we have a different opinion, without coding your data won't go anywhere, remove the search from your program and you will see how good it plays.

Program = code + data. If the data isn't yours, how can the program be? I own both Scan's code and data, so I can call it "my program".

Continuing this discussion seems rather useless to me, you seem to think that your data is the holy grail and that others are not capable of calculating weights themselves. My intention has always been to replace your data with data of my own before I will use the program in external events, now I'm going to do it even quicker, rather today than tomorrow.

Have a good day.

[BertTuyt]

I hope I'm able to bring some quietness to this discussion.

So what do we now so far.
Joost used the data-file of Fabien.
But he assumed, all knew about it.
He did not hide the fact when asked, was not into doing something secretly, and apologized to Fabien.

As some people were interested to test the program, he shared it.
He will no longer share the program (which is indeed his code and Fabien's data), at least that what I conclude from the post exchange.

And I guess as soon as he has an own data file, we can put this all to bed....

I don't see any reason to continue with mutual finger pointing, and hope we can continue in the good spirit of our small community.

Bert

[Catherine]

I hope I'm able to bring some quietness to this discussion.

So what do we now so far.
Joost used the data-file of Fabien.
But he assumed, all knew about it.
He did not hide the fact when asked, was not into doing something secretly, and apologized to Fabien.

As some people were interested to test the program, he shared it.
He will no longer share the program (which is indeed his code and Fabien's data), at least that what I conclude from the post exchange.

And I guess as soon as he has an own data file, we can put this all to bed....

I don't see any reason to continue with mutual finger pointing, and hope we can continue in the good spirit of our small community.

Bert

Hi all programmers, Joost, Bert, Fabien, Ed, Taille, Rein, Michel

First of all, i want to apologize everyone of you, particulary Fabien.
Like Bert and Ed said, Joost intention was to bring some ideas to our community, and the final users of all yours engine are US.
2 times Joost was in a certain measure " oblige" to share his project even the fact that he alway said that it was unfinished and contained bug.

He used ED egdb, this wasn't a really problem. everybody knows that his project was based on Scan 4x4 patterns idea. His real intention was, like he said, to make more pattern before release it. But, it somewhere me who was insisted to see how the project was.
Just to say that if we don't take things easy, our little and so kind draughts computer community will disappear.

I said since the beginning that it was a "good war" to try to get engines more stronger.
Someone can bring a new idea that can be used or can inspirate others to create something else more powerfull, the world run on this principle of innovation.
We, i pointed us others engines fans and users need to have programmers agree.

God bless you all!!!!!

Catherine.

[Joost Buijs]

To avoid any further dreadful conflicts I changed the pattern evaluation to use 6x4 patterns combined with 4x6 patterns, 5x5 patterns cannot overlap equally or I have to let them overlap for 4 fields which seems overkill and I don't like the fact that they not always contain the same number of bits, the alternative of using 6x6 patterns is a little bit too much to my taste. Collecting the data and calculating the weights with the scheme I use now takes 12.5 GB of memory, this can already be done on a standard PC with 16 GB memory.

First preliminary results indicate that this scheme works fine, I still have to play with the way I collect games, using material only with a random function added at the root to increase diversity gives the best results till now, but there are other ways I still want to try.

With the scheme I use to calculate the indices the size of the patterns has no influence on the speed of index calculation, the size of the weight table increased to 21 MB though and I expected a lot of cache pollution but somehow the speed didn't change in a significant way on the hardware I use. I expect the slowdown to be considerable on more standard hardware though.

Joost

[Rein Halbersma]

To avoid any further dreadful conflicts I changed the pattern evaluation to use 6x4 patterns combined with 4x6 patterns, 5x5 patterns cannot overlap equally or I have to let them overlap for 4 fields which seems overkill and I don't like the fact that they not always contain the same number of bits, the alternative of using 6x6 patterns is a little bit too much to my taste. Collecting the data and calculating the weights with the scheme I use now takes 12.5 GB of memory, this can already be done on a standard PC with 16 GB memory.

First preliminary results indicate that this scheme works fine, I still have to play with the way I collect games, using material only with a random function added at the root to increase diversity gives the best results till now, but there are other ways I still want to try.

With the scheme I use to calculate the indices the size of the patterns has no influence on the speed of index calculation, the size of the weight table increased to 21 MB though and I expected a lot of cache pollution but somehow the speed didn't change in a significant way on the hardware I use. I expect the slowdown to be considerable on more standard hardware though.

Joost

The patterns are not owned by anybody. They are just abstract features. Although using irregular features such as Scan's 6x3 patterns is something that should be mentioned in a program's description. If you look at the theory of convolutional neural networks, you will quickly discover that the geometry of the input arithmetically constrains the type of patterns you can use. With a 10x10 board, you can use 4x4 patterns, but there are many variations even for this size (Google for receptive field, stride and padding). You can even make 5x5 patterns work (hint: look up the general transformation formula, then take stride = 3, padding = 2), but this probably gives too many weights. 3x3 patterns are also an option.

In any case, setting up an end-to-end pipeline to get good weights from a generated set of games is where the real effort lies. There are many things to consider: how to extract the training positions from the games, what to do about non-quiet positions, endgame positions, start positions, regularization, error function, optimization routine etc., that all have an effect on the end result. Not everything is done out-of-the-box by a library. Of course, at least 3 persons have succeeded before you, but it took them many months of effort. Given the dreadful conflict you referred to, I'd be interested in hearing your journey to getting to that same point.

[Joost Buijs]

To avoid any further dreadful conflicts I changed the pattern evaluation to use 6x4 patterns combined with 4x6 patterns, 5x5 patterns cannot overlap equally or I have to let them overlap for 4 fields which seems overkill and I don't like the fact that they not always contain the same number of bits, the alternative of using 6x6 patterns is a little bit too much to my taste. Collecting the data and calculating the weights with the scheme I use now takes 12.5 GB of memory, this can already be done on a standard PC with 16 GB memory.

First preliminary results indicate that this scheme works fine, I still have to play with the way I collect games, using material only with a random function added at the root to increase diversity gives the best results till now, but there are other ways I still want to try.

With the scheme I use to calculate the indices the size of the patterns has no influence on the speed of index calculation, the size of the weight table increased to 21 MB though and I expected a lot of cache pollution but somehow the speed didn't change in a significant way on the hardware I use. I expect the slowdown to be considerable on more standard hardware though.

Joost

The patterns are not owned by anybody. They are just abstract features. Although using irregular features such as Scan's 6x3 patterns is something that should be mentioned in a program's description. If you look at the theory of convolutional neural networks, you will quickly discover that the geometry of the input arithmetically constrains the type of patterns you can use. With a 10x10 board, you can use 4x4 patterns, but there are many variations even for this size (Google for receptive field, stride and padding). You can even make 5x5 patterns work (hint: look up the general transformation formula, then take stride = 3, padding = 2), but this probably gives too many weights. 3x3 patterns are also an option.

In any case, setting up an end-to-end pipeline to get good weights from a generated set of games is where the real effort lies. There are many things to consider: how to extract the training positions from the games, what to do about non-quiet positions, endgame positions, start positions, regularization, error function, optimization routine etc., that all have an effect on the end result. Not everything is done out-of-the-box by a library. Of course, at least 3 persons have succeeded before you, but it took them many months of effort. Given the dreadful conflict you referred to, I'd be interested in hearing your journey to getting to that same point.

Indeed, atm. my main concern is which positions to include and which to omit, start positions don't present a problem, but what to do with kings on the board, at the moment I solely use positions that occur after a game-phase change which is based on men count and location only (I treat each game-phase differently), when kings arrive on the board things get very uncertain and there is a possibility that I have to omit these positions. Including or not including non quiet positions is also something I have to experience by using trial and error.

I noticed that the program is very permissive for errors in the evaluation weights, introducing large errors on purpose doesn't seem to hurt playing strength by much, my guess is that this has to do with the 'always draw' tendency of draughts.

Using a general transformation formula is out of the question because this will slow the program down to a crawl, I want to try to keep the speed as high as possible, atm. the optimized BMI2 version runs ~100 mnps on my main computer and if possible I would like to keep it this way.

Many months of effort seems a bit exaggerated, most techniques used by ML are well-known and are already in use for many years, I guess it solely depends upon how much time you can afford to spend on it, when you have a full time job and you can spend only a few hours a week on it, I agree it can take months, I'm already in the lucky position that I'm retired from work and that I can spend several hours per day on it if I want to. Not that I want to, zero sum game programming is a major hobby of me but not the only one, and there are other and more important things in life as well.

Joost