Bayeselo

[Ed Gilbert]

I have recently started using a tool called Bayeselo to calculate elo estimates and "likelihood of superiority" (LOS) statistics from the results of draughts engine matches. Bayeselo is an open source program written by Remi Coulom. Since there is not much documentation about the commands, the program can be a bit difficult to use at first, so I will briefly describe how I use it to get elo and LOS stats. I have also added a new command to the program, and I will describe how to do this.

Bayeselo is a command line program, so I run it from a "Command Prompt" window. It reads commands from standard input, which is the keyboard unless you redirect it. After it reads and executes a command, it continues to read more commands until you give it a command to exit. It has a hierarchy of command levels, what Remi calls interfaces.

When you first run it you are in the ResultSet interface. The commands in this set are all related to inputting match results. You can ask for a brief summary of the commands at any interface by type a question mark (?). If you do that at the ResultSet inferface you get this output:

Code: Select all

ResultSet commands
~~~~~~~~~~~~~~~~~~
players ......... list players
names ........... alphabetical list of player names
results ......... list game results
pgn ............. write results in PGN format

reset ........... reset results and players
addplayer <pn> .. add one player with name <pn>
addresult w b r . add result (w = white, b = black, r = result)
                   r = 0 (b wins), 1 (draw), or 2 (w wins)
removeresult n .. remove result number
removeplayer n .. remove games of player n
removerare n .... remove games of players with less than n games
pack ............ pack players (remove players with 0 games)
readpgn <file>... read PGN file
connect [p] [fr]  remove players not connected to p [fr=forbidden result]

elo ............. open Elo-estimation interface

The primary command for entering match results is "readpgn". Although Bayeselo seems to have some settings designed specifically for chess, it can be used for other games, and I think the readpgn command will work fine with checkers pdn files, as I think it ignores the game moves and only reads the headers for the players names, colors, and results.

A number of commands allow you to specify a player or list of players as parameters. For these commands, you do not input the player names, you refer to them by a number. Player numbers begin with 0 and seem to be assigned in the order that they are first encountered by Bayeselo. You can get a list of the players names and their reference numbers using the "players" command.

There are also commands to add players and results directly, without reading them from a pgn file. However these commands would be tedious to use for inputting many games as you can only enter one result at a time. The command that I added is the ability to input the number of wins, losses, and draws of an entire match between two players. The help line that I added for this new command is:

Code: Select all

addwld a b nw nl nd .. add WLD results between players a and b (from the point of view of a)

For example, let's suppose that a match between a test version of kingsrow against a baseline version ended the results +25, -10, =123. To enter this info into Bayeselo I can type

Code: Select all

addplayer test
addplayer baseline
addwld 0 1 25 10 123

Once I have input the raw data into the ResultSet interface, I use the "elo" command to open the EloRating interface. Two commands that I find useful here are "ratings" and "los" (likelihood of superiority). But before I can use these, there are a few set up commands that have to be given. Bayeselo has a default value that white has a 32 elo advantage over black, apparently to make it convenient to use for chess. For checkers there is no significant color advantage, so I set this to 0 using the "advantage 0" command. Then I give the "mm" command and "exactdist" commands, which causes Bayeselo to do its number crunching. Commands typed:

Code: Select all

elo
advantage 0
mm
exactdist

I can then give the "ratings" command, which for this example prints the following output

Code: Select all

Rank Name       Elo    +    - games score oppo. draws
   1 test        12   20   20   158   55%   -12   78%
   2 baseline   -12   20   20   158   45%    12   78%

The table shows a 24 elo difference between these two programs. I believe that the "+" and "-" columns show e.g. that the actual elo of "test" is between -8 and +32 (offset from some nominal level, which is 0 in this case) with a 95% confidence level. You can change the 95% confidence threshold that is used in bayeselo with the "confidence" command.

The "los" command takes 3 parameters -- the lowest player number, the number of players, and the number of digits to use in displaying superiority. Typing the command

Code: Select all

los 0 2 4

I get the output

Code: Select all

          tes bas
test          879
baseline  120

You can think of the numbers as probabilities with an implied decimal point to the left of the most significant digit. It says that test is stronger than baseline with a probability of .879.

Since Bayeselo reads from standard input, you can put the commands in a text file instead of typing them interactively, and redirect standard input when you run the program. To automate the commands in this example, you would create a text file with

Code: Select all

addplayer test
addplayer baseline
addwld 0 1 25 10 123
elo
advantage 0
mm
exactdist
ratings
los 0 2 4
x
x

The first "x" command exits the EloRating interface, and the next one exits the ResultSet interface and returns to the command line. If these commands are saved in a file named EloCmds.txt, then you can execute these by giving the command

Code: Select all

bayeselo <EloCmds.txt

To add the "addwld" command to Bayeselo, follow these steps if you're using MS Visual Studio.
- Download the bayeselo source from Remi Coulom's web site here: http://remi.coulom.free.fr/Bayesian-Elo/ and unzip it to a convenient directory.
- Create a new project for Bayeselo of type "console application". Add only one file to the project, bayeselo.cpp. This file includes all the other source files.
- There is a statistical function named erfc() used by bayeselo that is missing from the Microsoft libraries. I did some searching and found an open source math library package that had this function called "cephes". There are a number of packages in the cephes library. The one that has erfc is called "double". I downloaded it from here: http://netlib.org/cephes/doubldoc.html.
- Add the follwing files from the cephes library to the directory where the bayeselo project is located: mconf.h const.c exp2.c expx2.c isnan.c mtherr.c ndtr.c polevl.c. Add these files to the Visual Studio project.
- Edit mconf.h and comment out the lines that define the macros INFINITIES and NANS.
- Add the preprocessor definition _CRT_SECURE_NO_WARNINGS to the bayeselo project settings, or you will get a ton of warnings. You will still get some warnings when you compile, but this eliminates a large number of them.
- In the source file CResultSetCUI.cpp, find the declaration of the array

Code: Select all

const char * const CResultSetCUI::tszCommands[] =

and add this additional initializer to the list.

Code: Select all

 "addwld",

In the same file, find the implemetation of the member function CResultSetCUI::ProcessCommand, and add

Code: Select all

IDC_AddWLD,

to the list of anonymous enums. Then in this same function, add the help line

Code: Select all

out << "addwld a b nw nl nd .. add WLD results between players a and b (from the point of view of a)\n";

immediately after the help string for "addresult".

- In the same function as above, and in the same switch statement, add these lines:

Code: Select all

	case IDC_AddWLD:
	{
		int i;
		unsigned White = 0;
		unsigned Black = 0;
		unsigned nwins = 0;
		unsigned nlosses = 0;
		unsigned ndraws = 0;
		std::istringstream(pszParameters) >> White >> Black >> nwins >> nlosses >> ndraws;
		if (White < vecName.size() && Black < vecName.size()) {
			for (i = 0; i < nwins; ++i)
				rs.Append(White, Black, 2);		/* white wins. */
			for (i = 0; i < nlosses; ++i)
				rs.Append(White, Black, 0);		/* black wins. */
			for (i = 0; i < ndraws; ++i)
				rs.Append(White, Black, 1);		/* draws. */
		}
		else
			out << "Error: no such player\n";
	}
	break;

- In the file CBradleyTerry.cpp, near the beginning of the file just after the #includes, add this line:

Code: Select all

extern "C" double erfc(double a);

You should now be able to compile and run bayeselo.

-- Ed

[BertTuyt]

Thanks ED, think I also will need to dig into this.

Maybe interesting to check with this tool what the elo-differences are between the 2 programs Damage and Horizon as a function of search-depth differences, as i summarized in the blog earlier today.

Bert