Internet engine matches

Discussion about development of draughts in the time of computer and Internet.
Post Reply
Ed Gilbert
Posts: 854
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Internet engine matches

Post by Ed Gilbert » Tue Jan 08, 2008 02:30

Hello draughts programmers,

This post is addressed to the authors of international draughts programs, and particularly those that have participated in discussions on this forum -- Bert, Gerard, Feike, Michel, etc. I would like to run some automated engine matches between our programs over the internet. Using the convenience of the internet and the DamExchange protocol, we should be able to run automated matches with large numbers of games, many more than are possible within the constraints of the manual, interactive tournaments like the Culemborg event. For me the purpose of such automated matches are to learn more about my program's strengths and weakness, to identify areas needing improvement, and to get a clearer idea of its playing strength relative to other strong programs. Probably you are all aware of Frank Mesander's DamExchange protocol and its convenient implementation in a library of function calls. This library makes is quite easy to add to a draughts program the capability of running automated matches against another program on a remote machine over LAN. I have been using this to run matches between two different configurations of Kingsrow, and also between Kingsrow and Dam2.2. I usually run these matches using 2-move ballots as the openings, in order to get good variation in games and prevent the programs from playing the same games over and over again. There are 81 of these 2-move ballots, but 2 of them (31-27 16-21 and 32-27 16-21) lose a man immediately so I exclude those. For each ballot, one game is played with Kingsrow as white and then a second game as black, so a total of 158 games are played. When played at time controls averaging 7 or 8 seconds per move these matches take a little longer than one day to complete. We could easily run one of these over a weekend, which would allow us to watch some of the games as they are played. If the match is interrupted, Kingsrow is set up to continue a partially completed match at the game where it left off, so we would not have to start all over at the first ballot.

I am most interested in running these matches with both engines configured with equal endgame databases, so for example if your engine uses a 6-piece db then I would run Kingsrow with a 6-piece db. I'm interested to see where Kingsrow gets itself into trouble rather than to see how the larger databases can get it out of trouble. If anyone has the interest to see what kind of a difference they make, we could also run additional matches with the larger databases.

Again, my purpose for proposing these matches is so that we can learn from them and improve our engines, and the results are only for my own information. I don't intend to make them public, but neither would I mind if you are pleased with your result and you want to publish it, discuss it on the forum, etc. My experience is that these things give lots of valueable information, and they're also fun! Why don't we give it a try?

-- Ed

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Tue Jan 08, 2008 20:08

Ed, good initiative !
Lets start the preparation for this.

We can start from my perspective such a tournament soon (i have to modify an old Engine for this purpose) (although i as a result of a busy job , focus my draughts activities sin the weekends). If you want you can also use my private email h.tuyt@chello.nl

My purpose is to learn from this , and also i hope that we can share more ideas/thoughts regarding programming in this forum (like the chess community does).

Bert

Ed Gilbert
Posts: 854
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Post by Ed Gilbert » Tue Jan 08, 2008 23:26

Hi Bert,

Great, thanks for agreeing to try this experiment. It should be some fun. We can run this whenever you are ready. I think any weekend for the next 3 or 4 weeks is ok for me. You will have to open up port 27531 on your router to allow TCP traffic originating on the WAN to pass through to your PC. I tried this here and then I was able to run a DamExchange match between two of my computers by giving the router's WAN IP address, 71.251.206.21, as the remote machine. If you're not familiar with this I can give you some help in setting it up. In DamExchange terms, I will be the initiator and you will be the follower. This will allow my machine to control the start position of each game so that we can run through all the 2-move ballots.

Do you have an "instant messaging" account on one of the big portals, like Yahoo or AOL? We could communicate that way when we are starting up the match.

-- Ed

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Thu Jan 10, 2008 22:40

Ed, I examined the DamExchange and will start with a "simple" implementation in a "older" Engine (where i already use sockets , so i will try to built the DXP interface from the beginning).

I expect to have something working this weekend, so we could try to connect to each other late Saturday or somewhere Sunday, to play 1 game (so i know the basics work).
Can you write to me via my email if this is possible?

I don't know the time difference, and your time windows.
Next to that i can try to activate MSN, if you do the same (and again further details via Email), we can give this a try.

Im looking foward to it, and to be honest i don't care who will win, al least it will be fun , and we most definitely will learn.

Bert

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Mon Jan 14, 2008 22:20

So this weekend , yesterday 13-January-2008, a historical moment (at least i guess so).

The first (or did some-one tried this before) cross-continental 10x10 draughts engine match (or at least the initial beginning), between Damage and KingsRow.

So far we played 10 games, till a DXP implementation-bug forced us to stop.

Will be continued.

Bert

Ed Gilbert
Posts: 854
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Internet match, Kingsrow vs Damage

Post by Ed Gilbert » Mon Jan 21, 2008 16:09

This past weekend Bert and I ran a long engine match between kingsrow and damage. It was run over the internet using the DamExchange protocol. I think the match was a great success. The programs played 158 games with almost no operator involvement necessary. We could watch the action as it played out, and for part of the time we used Yahoo messenger to chat about the games and compare notes about what the engines were thinking. Some of the 2-move openings leave one side in a cramped position, and it was interesting to see how each program attacked and defended these. This is a great way to learn about the strengths and weaknesses of a draughts program, and I hope the other programmers will participate in similar matches.

Some details about the match mechanics: we had previously run some test games over the internet and discovered that Bert's internet connection through a wireless access point was not reliable, so for the full match he ran an ethernet cable from his router to the computer. This solved the problem and we had a solid LAN socket connection all weekend. Of course running this many games exercises the programs in ways they don't normally see, and sometimes exposes latent bugs. We discovered that Damage repeatedly crashed after 30 consecutive games, so Bert had to restart it about 5 times during the match. He thinks it is a string buffer overflowing in an output window and should be easy to fix. We started the match with time controls of 75 moves in 12 minutes. But after about 40 games we realized that kingsrow was running on a faster computer and this would make it difficult to interpret the results, so for the rest of the match I had kingsrow set to 75 moves in 5 minutes. Both programs were configured equally with 6-piece endgame databases, and pondering turned off. The match ran from Saturday morning to Monday morning. It would have finished some time on Sunday, but there were delays because we were not always present to restart when the 30 game crashes occured. Each game was concluded when Kingsrow saw a database win or loss, or if it saw database draws on 6 consecutive moves. Most of the games were 45 to 55 moves long. There is only 1 game that lasted for 75 moves without a conclusive result.

-- Ed

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Mon Jan 21, 2008 19:25

I would like to add the game result, as I understand why Ed did not provide the info but i think i have the right to do so !!

Damage was really crushed by Kingsrow, so 40 Wins for Kingsrow, 2 wins for Damage, 115 Draws, and 1 game was so complex that after 75moves there was not yet a decisive results.

I would like to compliment Ed with this result, and it motivates me, to learn from this defeat, so Damage will return stronger as ever.

Keep in mind that Ed did not play with 9p databases, so he really raised the bar, so lets' take the challenge.

Bert

Eric van Dusseldorp
Posts: 4245
Joined: Tue Sep 30, 2003 22:11

Post by Eric van Dusseldorp » Mon Jan 21, 2008 20:00

BertTuyt wrote: (...) and 1 game was so complex that after 75moves there was not yet a decisive results. (...)
I don't understand that. Why did you stop the 'complex game' after 75 moves? What was the position?

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Mon Jan 21, 2008 20:36

We played totally automatic, without operator involvement.
So every game was automatically started with a different position, and stopped after 75moves max.

Bert

ildjarn
Posts: 1537
Joined: Tue Aug 22, 2006 15:38
Real name: Joost de Heer

Post by ildjarn » Mon Jan 21, 2008 20:37

What does Kingsrow with the extra databases think of the final position of the aborted game?
Lasst die Maschinen verhungern, Ihr Narren...
Lasst sie verrecken!
Schlagt sie tot -- die Maschinen!

Eric van Dusseldorp
Posts: 4245
Joined: Tue Sep 30, 2003 22:11

Post by Eric van Dusseldorp » Mon Jan 21, 2008 21:12

BertTuyt wrote:We played totally automatic, without operator involvement.
So every game was automatically started with a different position, and stopped after 75moves max.

Bert
Can you tell something more about the 'different positions'? Were those positions totally or nearly equal for both colors, or were any of them (very) bad for white or black?

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Mon Jan 21, 2008 21:48

The positions were all the 2move ballots , except for the ones immediately loosing a man.
Both programs played every 2 move opening, so in balance.

Bert

Ed Gilbert
Posts: 854
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Post by Ed Gilbert » Mon Jan 21, 2008 22:43

ildjarn wrote:What does Kingsrow with the extra databases think of the final position of the aborted game?
Here is the game:

[Event "Game 25, opening 32-28 20-25"]
[White "Kingsrow"]
[Black "Damage 7.1"]
[Result "*"]
1. 32-28 20-25 2. 37-32 19-24 3. 34-30 25x34 4. 39x19 14x23 5. 28x19 13x24 6. 40-34 10-14 7. 41-37 14-19 8. 46-41 5-10 9. 45-40 10-14 10. 34-30 9-13 11. 44-39 18-23 12. 40-34 12-18 13. 30-25 7-12 14. 34-29 23x34 15. 39x30 1-7 16. 50-44 18-23 17. 32-28 23x32 18. 37x28 13-18 19. 44-39 8-13 20. 42-37 4-9 21. 39-34 18-23 22. 47-42 23x32 23. 37x28 12-18 24. 34-29 18-23 25. 29x18 13x22 26. 43-39 16-21 27. 31-26 21-27 28. 39-34 2-8 29. 41-37 8-12 30. 37-31 27-32 31. 38x18 12x32 32. 48-43 9-13 33. 42-38 13-18 34. 38x27 18-22 35. 27x18 17-21 36. 26x17 11x13 37. 31-27 13-18 38. 36-31 3-9 39. 34-29 7-11 40. 29x20 15x24 41. 31-26 11-17 42. 33-28 18-23 43. 27-21 23x32 44. 21x12 32-37 45. 12-7 37-41 46. 7-1 41-47 47. 43-39 24-29 48. 1x45 9-13 49. 26-21 6-11 50. 45-1 11-17 51. 21x12 13-18 52. 12x23 19x28 53. 1-7 47-38 54. 7-12 38-20 55. 49-43 20-15 56. 12-7 15-20 57. 7-18 20-47 58. 18-45 47-20 59. 45-7 20-42 60. 7-18 42-47 61. 18-34 47-20 62. 34-40 20-47 63. 40-18 47-42 64. 18-12 42-15 65. 12-1 15-42 66. 1-34 42-15 67. 34-12 15-47 68. 12-18 47-42 69. 18-36 42-47 70. 36-31 28-32 71. 31-18 14-19 72. 18-4 47-15 73. 4-36 15-47 74. 36-22 47-42 75. 22-4 42-15 76. 4-31 *

With the 9-pc db, at move 45 kingsrow sees a white database win. But after move 47. 43-39, it no longer sees a win, and it cannot tell for certain if the position is won. If white plays 1-45 then the win is preserved, but kingsrow could not know this during the match.

-- Ed

Ed Gilbert
Posts: 854
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Another engine match

Post by Ed Gilbert » Thu Feb 07, 2008 03:48

Last weekend Bert and I ran another engine match between Kingsrow and Damage. It turns out that Bert had not loaded most of the 6-piece endgame db slices into Damage for the first match, and this time he ran the 64-bit version on his core2-duo. As before, we ran with equal time controls, single-threaded engines, pondering off, and 6-piece databases for both engines. Game times were 75 moves in 11 minutes. Bert asked me to post the results, which were less lopsided than the last time. Kingsrow vs Damage: 30 wins, 6 losses, 120 draws, 2 unknowns. We both found some areas that can be improved from this match, so we'll work on improvements for a while and then try it again in a month or two.

-- Ed

BertTuyt
Posts: 1573
Joined: Wed Sep 01, 2004 19:42

Post by BertTuyt » Thu Feb 07, 2008 20:26

Ed, thanks for your post.
Small correction I was running Damage 10.0 on a Quadcore Q6600.
Only using 1 core [img]images/smilies/icon_rolleyes.gif[/img]

Bert

Post Reply