engine tournament

[BertTuyt]

As we all know engine tournaments take lot of personal time.
For this reason I started with the programming of a tournament manager in the Damage GUI.
It is still in embryonic phase, but the basics work.

You can define a tournament between engines, for examples 4 engines round-robin (but any number will do).
The GUI will take care of all, so starting the programs, and providing the result in the end, stopping the programmings.
Then if in the same round another pairing is waiting the next 2 engines are automatically started, and the sequence continues.
Caveat, all engines should use the HUB protocol (for now), as communication is with pipes and Hub.

Today i was also able to broadcast automatically, a game from my Damage GUI towards toernooibase.kndb.nl with the help of Hendrik Veenstra, and Piet Bouma.
As already mentioned all still in an embryonic phase, but i hope it will be the starting point that in the future these tournaments could run completely automatically.

Bert

[FrankMesander]

Very interesting!
Can you please post a preliminary/embryonic version of the minimal set of HUB commands that the Draughts engine must support?

Frank.

[BertTuyt]

Frank, only a limited set is needed.
Examples derived from the protocol.txt document from Fabien.

The first command the GUI will sent is hub, and the engine should sent:
id name=Scan version=3.1 author="Fabien Letouzey" country=France

Only the id and version are used, the Damage GUI does nothing (yet) with author and country.
In the protocol from Fabien hereafter the "capabilities" are sent (like param name=variant value=normal type=enum values="normal killer bt frisian losing").

In the current implementation nothing is done with these, as the GUI expect the base configuration be defined in an .ini file (read by the engine during startup).

The engine should send now wait
The GUI responds with init, as a signal that time-consuming activities can be started (like loading databases, or whatever).
When the engine has finalized these actions, the ready is submitted by the engine.

The GUI can start the game between the 2 engines, when both engines (which are automatically loaded and unloaded by the GUI) have signaled ready.

Then the GUI sends to the engine which turn it is:
pos pos=<position>
level [moves=<remaining moves>] time=<remaining time>
go think

example:
pos pos=BeeeWWeeeeeeeeeeeeeeeeeeeweeeeeeeeewweBeeeeeeeeeBee
level moves=9 time=291.587
go think

In the current implementation the moves= after the pos= message is not used.

The engine then should sent info moves to display the search process (although this is not exactly necessary).
info depth=21 mean-depth=20.8 score=-0.01 nodes=31261613 time=3.906 nps=8.0 pv="32-28 17-22 28x17x22 12x21x17 33-28 7-12 39-33 19-23 28x19x23 14x23x19 44-39 9-14 50-44 4-9 34-29 23x34x29 39x30x34 21-26 44-39 11-17 31-27 6-11 37-31 26x37x31 41x32x37 20-24 30x19x24 13x24x19"

Basically one is free to change the layout of the info command, as it does not affect the actual game play (gives only insight in the engine search process).

When the search is finalized the done move= is sent (here the ponder= is not used), so can be omitted.
done move=32-28 ponder=17-22

As I now execute the engine tournament on 1 computer i don't use pondering.

Think that is all, i started with a simple implementation to demonstrate the principles.
If there is strong need to add something, let me know.
Maybe some engine which already have implemented hub don't work with this minimal set, in which case i need to update and modify the implementation.

If there are specific (more) questions please let me know.....

Bert

[FrankMesander]

Bert,

Two interrelated additions regarding pos pos=<position>:

The tournament engine should always give HUB commands pos pos=<position> in which moves are possible. Otherwise the engine can't reply with done move=<move>.

Only the tournament engine is able to terminate the game by sending a HUB quit command. The engine may want to end the game, but that situation is not defined in the HUB protocol. The engine always has to reply with done move=<move>.

Agree with these two additions?

Frank.

[BertTuyt]

Frank, in my implementation the engines are equal, there is no master-slave relation or whatever.
All the logic and decisions are handled by the GUI, and every engine is connected with its own pipe.
Basically I have a 3th engine running in parallel, which acts as an arbiter.
This 3th engines has access to the Kingsrow databases, and therefore can decide if based on the databases what the final result is, and if needed terminate the game in an earlier stage already..
So no need to change hub, in any way.
At least that is my 5 cents....

Bert

[FrankMesander]

Bert,

A misunderstanding: I meant additions to your description, no additions to the HUB protocol.

In short: a game is ended exclusively and solely by the HUB GUI, by sending a quit command to the engine.

Frank.

[Harm Jetten]

Bert, an excellent initiative! I'm currently working on implementing the hub protocol in Moby Dam. Just the basics, no pondering or analyzing yet.

Frank, a quit command terminates the engine, not just the game. A game is also ended with a new-game command.

[Harm Jetten]

Bert, in Fabien's protocol.txt it isn't clear whether the hub GUI must send a new-game before setting up the first position after the init...ready phase. His hub2.1 happens to do so. Is it required?

[FrankMesander]

Bert, an excellent initiative! I'm currently working on implementing the hub protocol in Moby Dam. Just the basics, no pondering or analyzing yet.

Frank, a quit command terminates the engine, not just the game. A game is also ended with a new-game command.

I know the HUB protocol, so I know what quit and new-game do.
I am talking in the context of Bert's (description of the) tournament engine. And my comments are on that description, not on the HUB protocol itself.
It should be clear in the description how Bert's implementation of the tournament engine will end each game. I understood (maybe wrong) it will only be done with the quit command.

Frank.

[BertTuyt]

Harm, not sure it is required, as basically every position is "new" for engines with a hub protocol.
So sending the start position will not break or crash the engine.
The only difference is that with a new-game you can do additional clean-up, like clearing the hash-tables, setting the move-counter to zero, and whatever....

In the current set-up, if the game is finished (or adjudicated by the arbiter) and there is another round to play, i first unload the 2 engines, and then load the engines for the specific round.
In a better implementation, i could maybe keep one of the 2 engines, if this is also required by the new round.

I also so-far encountered some challenges with the quit command, which seem to be related to windows. Sometimes it takes extremely long before the engine-process really terminates after the quit command. For this reason i use the next code, to make all more robust:

Code: Select all

int unloadengine(int iengine)
{
	int iresult = 0;

	gecomm::OnGUIMessage_ENGINE_QUIT(iengine);

	gui::enginestatus[iengine].bloaded = false;
	--gui::machinestatus.ixengines;

	// test if process has stopped

	int irepeat = 3;

	while (irepeat > 0)
	{
		HANDLE hprocess = piprocinfo[iengine].hProcess;  // process handle from createprocess
		DWORD dwresult = WaitForSingleObject(hprocess, 20000);  // wait max 20 seconds for process termination

		if (dwresult == WAIT_OBJECT_0)
		{
			// the child process has terminated
			iresult = 1;
			break;
		}
		else if (dwresult == WAIT_TIMEOUT)
		{
			// timeout expired, handle this case
			iresult = 0;

			--irepeat;
		}
		else
		{
			iresult = -1;
		}
	}

	Sleep(5000); // wait 5 seconds...

	return iresult;
}

Today I played a test round-robin tournament between 6 engines (3 engines were basically Damage, but i made a copy and changed the name ,
and all seemed to work this time (including a pdf.file with all games).

I will do further testing, and then try to live broadcast a test-tournament via toernooibase, but that will take some more weeks.

For those interested I can issue a version of the Damage GUI, with several engines, so one can do a tournament test out-of-the-box.
Think I needs some 2 weeks for this, and will inform all when this option becomes available.
The only thing i will (most likely) not enable is to broadcast a tournament via Toernooibase, as I got a private key to get access to this website. But a local tournament is then possible.

Bert

[Sidiki]

Hi Bert and everybody.

This's a great thing that will help us.
Same thing as Arena for chess. Thanks to all.

I will run a tournament soon.

Take care.

Friendly, Sidiki

[FrankMesander]

For those interested I can issue a version of the Damage GUI, with several engines, so one can do a tournament test out-of-the-box.
Think I needs some 2 weeks for this, and will inform all when this option becomes available.
The only thing i will (most likely) not enable is to broadcast a tournament via Toernooibase, as I got a private key to get access to this website. But a local tournament is then possible.

Bert

Bert,

2 weeks have passed now. Any progress?

Frank.

[BertTuyt]

Frank, yes I'm still in testing phase.
But I have a version I can share with some people, as a starting point, and for feedback.
I will do my best, to put a version in the cloud for download this evening, with 4 hub-engines, so you can test it.
Will include some notes how to use it.

Bert

[BertTuyt]

I have uploaded a test version, and Frank will be the first who will be able to test it.
If it also works on his computer, and most likely some initial bugs, or issues are resolved, I invite others for further testing.
Just drop me a line during the week.....

Bert

[Sidiki]

Hi everybody,

hope you feel good.
I plan to run a tournament the 20th august for fun.

It's a while that engines haven't played against each others.

If someone has an update working well, no problem, it will make the business. Just for fun.

I am free now, for a month and no problem to run even 2 tournaments for fun.

Friendly, Sidiki.