Hub-1 protocol

[Fabien Letouzey]

Hi all,

I tried to describe the Hub protocol in a text file. The forum refuses my upload ("The extension txt is not allowed"), so I copy/paste it below.

The protocol is in a sorry state if one needs extensions. For this reason, I am designing a more ambitious one that I should present next week-end. The downside is that it will be more difficult to parse; commands will have (optional) named arguments like "info ..." in UCI and I also want quotes to handle spaces gracefully. However, once the initial effort is made, the protocol should be easier to extend. In any case, the exercise is interesting.

Fabien.

===

Hub protocol, version 1 (Hub-1)
Used in Scan 2.0 (engine) and Hub 1.0 (GUI).

---

Introduction

This document is very informal and assumes experience with text-based protocols. To make Hub extensible, a lot of changes are needed. For this reason I prefer keeping this description short, relying mostly on examples, and prepare a more complete Hub-2 proposal for later (perhaps to become UDI in the future).

A good reason to use Hub-1, however, is to interact with Scan 2.0 in an automated way. A few marked sections describe specific Scan behaviour that is not officially part of the protocol.

Note: Scan needs to be launched in Hub mode using a "hub" command-line argument. On Unix systems, that would be "./scan hub". As the protocol uses text, you can experiment manually.

---

Protocol overview.

Hub-1 is basically a subset of UCI with commands split into smaller pieces for easy parsing. It is text-based (even line-based) and doesn't specify how information is transmitted. The engine uses standard I/O, so it is up to the GUI (or anything that plays a similar role) to set up communication.

Shortly after connection between the GUI and the engine, an initialisation phase takes place. Then, from the engine point of view, a game (or problem solving, or analysis) is merely a sequence of searches. A search is handled similarly to a remote procedure call (RPC): the GUI sends the parameters in several steps, and the engine starts computing. The latter outputs intermediate information (anytime algorithm), until a final answer is produced due to depth/time constraint or interruption by the GUI.

---

The initialisation phase looks like this:

hub
name <string>
author <string>
country <string> (capitalised, no abbreviation)
var <name> <current value>
var <name> <current value>
...
wait
set <name> <new value>
set <name> <new value>
...
init
(the engine should initialise its data structures, now that is has the values)
(it might display some information here, that the GUI should ignore)
(this may take a while ...)
ready

The engine answers are indented for illustrative purposes only; spaces have no special meaning in the protocol.

Note: Scan 2.0 doesn't send "var" (the engine parameters).

Note: the GUI will set the engine's variables before asking for "init". They might affect long operations like initialisation/loading of complex data structures.

The protocol does not specify whether variables can be set after initialisation. The GUI is free to do it on behalf of the user. The engine is free to accept or ignore, on a per-variable basis. The GUI user is supposed to be informed of which variables can be changed at run time, for each engine. Scan 2.0 accepts most "dynamic changes", but can misbehave for some: changing the TT size and increasing the BB size. Hub 1.0 never sets the variables after initialisation.

---

A search is launched in 3 steps:
1) position
2) time/depth limit
3) search mode, which launches the search

Basic example:
pos wbbbbbbbbbbbbbbbbbbbbeeeeeeeeeewwwwwwwwwwwwwwwwwwww
time 1000
go

Example with a few moves:
pos beeeWWeeeeeeeeeeeeeeeeeeeweeeeeeeeewweBeeeeeeeeeBee
move 38-15
move 5-41
move 48-26
level 9 291587 0
ponder


1) position

A "position" is actually more like a short game. There is a starting position which could be anything, and a sequence of moves can be played from it. Sending moves is necessary to allow the engine to detect draw by repetition during search. Hub 1.0 only sends king moves, as they are usually reversible. Because kings appear late in the game, most of the time no moves are transmitted. So the overhead compared to a protocol requiring an "incremental board update" is very low.

syntax:
pos <position>
move <move>
move <move>
...

Instead of "pos <position>", "start" can be used to designate the standard starting position. Also, moves are sent one per line.

Note: engines are supposed to handle any position without the complete move list from the beginning. I.e. <position> might not be the standard starting position. If that happens to be too restrictive, a compatibility mode in GUIs could be added to always send the full game (but this is not always possible).

The position format is one character for the side to move (w/b) + one character per square in standard order. So, 51 in total.
For each square:
'w': white man
'b': black man
'W': white king
'B': black king
'e': empty

The move format is the same as standard notation for quiet moves, for example "32-28". For captures, all the captured pieces (but not the pivots) are included in any order like this: from x to x captured-1 x captured-2 ... without any spaces. For example "28x19x23".


2) time/depth limit

There are 3 commands which control this:
depth <depth>
time <time>
level <moves> <time> <inc>

All the arguments are integers.
<time> and <inc> are in milliseconds.
The engine is allowed to assume that at most one will be sent for a given search.
I.e. it is not necessary to handle combinations.
For "infinite" searches (most likely analysis), no limit command will be sent; the GUI will interrupt the engine later with "stop".

"time <time>" means a fixed time limit for the search.

"level <moves> <time> <inc>" is used for more complex time management, as in tournaments.
The arguments correspond to movestogo, wtime/btime, and winc/binc in UCI; I'm not giving a detailed explanation here. The engine can assume that at least one argument will be 0. The standard combinations are:
- <time> "alone" (the others are 0): sudden death for the whole game
- <moves> + <time>: x/y time control
- <time> + <inc>: x+y (Fischer) time control

Note: the increment is added before every move. While not critical, this is mathematically cleaner.


3) search mode

A search is launched with one of 3 commands, without any arguments: "go", "ponder", or "analyse".

"go" is the normal, engine is thinking, search. The GUI wants the engine to produce a move to play.

"ponder" works like in UCI: the expected opponent answer, chosen by the engine during the previous search, is already present in the position. The engine's game clock is not running at this point, so it can wait for "ponder-hit" or "stop" before measuring time.

With "analyse", the GUI cares more about search information such as the score and PV. The engine can assume that it is not playing a game, and is allowed to behave differently. For example spending time searching a single legal move, or not trying to avoid draws.

Note: the GUI will never ask the engine to search a "finished" position, even for pondering. So the engine can assume that there is at least one legal move.

Note: contrary to UCI, the engine is allowed to stop searching at anytime and produce a move, even in ponder and analysis modes (infinite searches). The GUI will do the buffering if necessary.


4) the engine is searching

The engine will (optionally) answer with progressive search information and then, eventually, its move:
info ...
info 18 22.8 16 2638461 0.52 5.1 30-24 19x30 34x25 17-21 26x17 11x22 28x17
move 30-24 19x30x24

The last line contains:
- the engine’s move
- optionally, the expected answer from the opponent (used for pondering but safe to output regardless)

Note: the GUI can interrupt the search at any time with "stop". The engine is not expected to react instantly, but must still produce a "move" answer.

Note: the GUI can also send "ping" during search. The engine should answer "pong" and keep searching. Hub 1.0 does not rely on this mecanism for normal use; the user can manually request a ping.

The only commands that the engine can receive during search are:
- ping
- ponder-hit
- stop

The "info" format that contains search information is not specified in the protocol.

Scan 2.0 sends:
- depth
- average leaf depth (real number)
- score in cp for the side to move
- number of nodes (64-bit integer)
- time in seconds (real number)
- speed in MNPS (real number)
- PV in standard notation (not Hub's move format); moves are separated by spaces

---

Additional commands that the GUI can send:
- "new" indicates that the next search is unrelated to previous ones (=> clear TT)
- "ping": must be answered by "pong"
- "quit"

Additional answers that the engine can send:
- "error <message>" (unused in Hub 1.0)

Note: if the standard input is closed, the engine should assume that the GUI has crashed and, therefore, quit as directly as possible. It is a common mistake in chess engines, especially on Windows, to poll the input in an infinite loop that consumes CPU time.

---

That is all for now, today is 2017-04-19.

Fabien Letouzey.

[BertTuyt]

Fabien, thanks for sharing.

Here a few questions (in italic) for clarification, taken from your text (in a later stage I also will react on your other mail).

hub is hub a first reply from the engine?, should it also include a protocol parameter as answer, to inform the gUI that a specific protocol has been implemented, for future use, example hub 2.4, so the GUI knows...
name <string>
author <string>
country <string> (capitalised, no abbreviation)
assume that the engine first reads its ini file, before posting the var names and values
var <name> <current value>
var <name> <current value>
...
wait assume that this command is send by the engine
set <name> <new value>
set <name> <new value>
...
init assume this is send by the GUI
(the engine should initialise its data structures, now that is has the values)
(it might display some information here, that the GUI should ignore)
(this may take a while ...)
ready assume send by the engine, and now ready for other commands from the GUI

I think the first part is relatively straightforward (only the var and set needs some additional thoughts, and although i have implemented it in a different way, it is not a problem to modifiy my GUI accordingly.

Bert

[Fabien Letouzey]

Hi Bert,

hub is hub a first reply from the engine?, should it also include a protocol parameter as answer, to inform the gUI that a specific protocol has been implemented, for future use, example hub 2.4, so the GUI knows...

I don't have enough experience to think clearly about future compatibility. In this thread I described the protocol as I implemented it in 2015.

There are two aspects that I'm considering to simplify this matter:
- assume that GUIs are always up-to-date: old engines are supposed to keep working but not necessarily old GUIs.
- in the next version, the engine will add some feature/capability declaration during the initial dialog, similarly to CECP (= xboard) and GUIDE. UCI uses a few standard engine parameters like "Ponder" partly with the same objective. Would this be enough to cover what you have in mind?

I also think that, since the protocol needs a new syntax anyway, it would be painful to keep compatibility with the version from 2015. So I'm not considering this.

Instead of thinking too much about the future, I'll try to cover what is most commonly needed now (which I can find in existing protocols).

assume that the engine first reads its ini file, before posting the var names and values

That's what Scan does indeed, but my feeling is that it's better to leave this out of the protocol specifications. It could be part of the engine documentation, though.

In chess, it is common to have a stand-alone executable. The initial value of engine parameters is often hard-coded in the source code. And that's actually OK because they can be changed externally.

For me the protocol, when it is used, replaces the engine INI file (or equivalent mecanism). The variables should be saved on the GUI side in this case.

I think the first part is relatively straightforward (only the var and set needs some additional thoughts, and although i have implemented it in a different way, it is not a problem to modifiy my GUI accordingly.

Sure. If your way is better, we should discuss it.

Fabien.

[Fabien Letouzey]

Here a few questions (in italic) for clarification, taken from your text (in a later stage I also will react on your other mail).

I see what happened. Since the forum refused my text file (possibly because of line endings), I copy/pasted it into my post. But I didn't think about the indentation being removed.

In the text file, it looks like this:

Code: Select all

hub
    name <string>
    author <string>
    country <string> (capitalised, no abbreviation)
    var <name> <current value> (engine parameter)
    var <name> <current value>
    ...
    wait
set <name> <new value>
set <name> <new value>
...
init
    (the engine should initialise its data structures, now that is has the values)
    (it might display some information here, that the GUI should ignore)
    ready

So that should already be clearer.

[mig]

Fabien, in the next version of Hub you are working on, could you provision something for variants ?

Ideally we need a command to list the supported variants, and one to set the actual variant.

[Fabien Letouzey]

Fabien, in the next version of Hub you are working on, could you provision something for variants ?

Ideally we need a command to list the supported variants, and one to set the actual variant.

Yes, something simple.

There are two parts of the protocol that will cover this: "engine features" (like in CECP and GUIDE) and standard engine parameters (like in UCI). For now they are overlapping a lot, but I have the feeling that the duplication is going to pay off later.

The goal is to group concepts that multiple engines have in common, such as pondering, threads, and variants, and give them standard names. The engine would declare early what it supports, something like:

Code: Select all

feature ponder threads=8 variants="xxx yyy"

And the GUI could use normal engine-parameter commands, with standardised names; this is what UCI does. Or perhaps send the information with every search for some of the features.

One aspect of Hub will help engines: the variant will be sent before the engine is asked to initialise. In the development version of Scan for instance, it affects the endgame tables to load. By contrast, UCI does not guarantee that engine parameters will be set early enough; they can be changed at any time.

So far, for this to work the GUI needs to know the variant and that's what I have in mind. There is an alternative view that is common in chess (with hundreds of variants I guess) and apparently used in CECP and GUIDE. The GUI doesn't know the variant, but the engines provide additional information to cover that knowledge (at least enough for the GUI to display the positions correctly). I feel that this is outside the scope of a first protocol version. The size could increase by a lot, reducing adoption. It could be part of "possible extensions", though.

There is an elegant way to do the same without increasing engine complexity: a "referee engine" whose purpose is only applying game rules like move legality. It would be connected to the GUI in addition to the player(s). I have no short-term plans for this either, but like the idea.

Fabien.

[mig]

This sounds great !

You wrote:

...the variant will be sent before the engine is asked to initialise. In the development version of Scan for instance, it affects the endgame tables to load.

Do you mean there is already a version of Scan that supports some variants other than International Draughts ? That would be excellent news. My program supports International Draughts, English Draughts, Brazilian Draughts, Spanish Draughts, German Draughts, Thai Draughts, Turkish Draughts and Alquerque, but for now it can only be engine-played on International Draughts (through Scan/Hub, Scan/DXP or Mobydam/DXP), other variants have to deal with the native generic AI which is much weaker than specialized engines.

[Fabien Letouzey]

Do you mean there is already a version of Scan that supports some variants other than International Draughts ? That would be excellent news. My program supports International Draughts, English Draughts, Brazilian Draughts, Spanish Draughts, German Draughts, Thai Draughts, Turkish Draughts and Alquerque, but for now it can only be engine-played on International Draughts (through Scan/Hub, Scan/DXP or Mobydam/DXP), other variants have to deal with the native generic AI which is much weaker than specialized engines.

Yes, but my interest in variants differs from yours and is restricted to 10x10 for now.
International draughts is ridiculously drawish at high level and I'm not sure that improving the state of the art would be observable in tournament conditions. Even opening positions with a major positional imbalance tend to end in a draw (this is from test games I ran while working on the new protocol). Therefore I needed an escape hatch in order to keep interest in the game.

I have supported Killer draughts for a while, and recently implemented the one we call "Breakthrough": the first one to make a king wins. The name comes from this great draughts-like game: https://en.m.wikipedia.org/wiki/Breakth ... oard_game). Both variants required very little work as far as the rules are concerned. I have a semi-automatic procedure that creates the data files I need in about 2 weeks: first build endgame tables, then generate a lot of games, and finally learn evaluation features. An opening book can also be constructed.

Until recently I had never seriously considered 8x8 variants. I see them as too easy and they would require major changes in the code. However I recently modified square numbering in Scan, and made the related code more flexible. I'm therefore one big step closer to being compatible with 8x8. It's still not part of my plans, but at least I'm starting to consider it as a thought experiment.

[mig]

I guess i could easily support Killers and Breakthrough variants too.

Would the changes you are thinking about allow 12x10 or 12x12 configurations ?

[Fabien Letouzey]

I guess i could easily support Killers and Breakthrough variants too.

I forgot to say, the next ones in my list are "suicide" which looks fun and the orthogonal captures from Frisian draughts.

Would the changes you are thinking about allow 12x10 or 12x12 configurations ?

I'm not sure whether you mean the protocol, or Scan.

For the protocol, I haven't thought about board size. An additional variable (or two) is all it takes, I guess. The protocols are generally game-independent except for position and move encodings. I have used Hub-1 in Othello, and UCI has been "translated" to Shogi (USI) and chinese chess (UCCI). For draughts variants, I suppose a single encoding can be found.

For Scan, 12x12 would require 128-bit integers. That's what I use in Shogi (9x9) and 10x10 Othello (since 8x8 is too easy). GCC and Clang provide such a type, but I think the native Windows compilers don't. It's a bit slower of course, but few code changes are necessary. Scan's eval would be different, however, for every rule set (assuming there is a big effect) and every board size. A convolutional neural network (CNN), while much slower, wouldn't share the last restriction.

[mig]

What ? You have a hub-interfaced Othello engine ? I'd be more than happy to integrate it into my program which also supports Othello (and a few variants around).
Where can i get a code i can compile on linux ? Is it open source ?

[Fabien Letouzey]

What ? You have a hub-interfaced Othello engine ? I'd be more than happy to integrate it into my program which also supports Othello (and a few variants around).
Where can i get a code i can compile on linux ? Is it open source ?

It's a fork of Scan that I wrote last year. It makes more sense than it looks because Scan's technology comes from Othello. The name is Decapus and it plays 10x10. There is no public release, as I mostly intended to experiment with it.

Earlier this year, I was convinced to convert it to 8x8 so that it could enter a tournament in March. The entry was named Octopus and I have a Linux binary (+ data files). It would be much more work to make it open source, as I remove a lot of stuff that's not needed to play games (mostly learning) when I distribute an engine, and have to rearrange the code after that. It's doable, though, depending on whether your project is part of a commercial venture. It should take a week-end.

There are even stronger open-source engines like Edax, and this is what I recommend for you. At least on Mac, it uses a very simple protocol that comes with Cassio, a Mac-only GUI: http://cassio.free.fr/engine-protocol.htm
Note that the GUI does the iterative deepening and time management, so it would be more work for you. But Edax is state of the art in 8x8.

I think the rest will quickly become off-topic here; we can continue by email.

[mig]

PM'ed you

[Maurits Meijer]

Thanks Fabien
I think there is a bug in de move format implementation.
In this position

dambordstand.jpg (18.63 KiB) Viewed 13540 times

I got
move 23x25x28x30x38x39 21-16
captures sorted from small to big

I expected 23x25x28x38x39x30
captures in order

[Fabien Letouzey]

Hi Maurits,

I got
move 23x25x28x30x38x39 21-16
captures sorted from small to big

I expected 23x25x28x38x39x30
captures in order

Most programs use bitboards internally and have no notion of "capture order". Therefore the protocol leaves it open:

The move format is the same as standard notation for quiet moves, for example "32-28". For captures, all the captured pieces (but not the pivots) are included in any order like this: from x to x captured-1 x captured-2 ... without any spaces. For example "28x19x23".

Fabien.