PDN 3.0 Standard Review Request

[Wieger Wesselink]

As to character encodings.
We have several options to choose: ASCII, UTF-8, any other 1-byte per character codepages (cp1252, for Europe I guess), UTF-16, other N-byte per character codepages (UCS-2 but it seems to be deprecated almost everywhere).

If ASCII would go to the standard - the checkers developers world would be treated as dinosaurs - come on, it's 21st century. There is absolutely NO reason to use such encoding.
Any other character encoding without full Unicode coverage (cp1252, cp1251 for Cyrillic world, etc) would cause problems from the beginning - the PDN file should have an external attribute, i.e. encoding. The end user would have to choose what encoding particular PDN file has - I think this is not very good. For developers any encoding is the pain - nobody understands all details from the start.
There are several encodings with full Unicode coverage. UTF-8 is the superset of the ASCII, for this reason I would recommend this to become a standard. UTF-16 has some options (big endian, little endian), but the main problem is that every character (even ASCII one) would have 2 bytes. So, UTF-16 (and UCS-2 as its parent) would break all compatibility.

For UTF-8 to be standardized, the only change to the grammar is necessary: to treat characters from 0x80 to 0xFF as normal characters, i.e. to allow them.

There is still a question about BOM marker at the beginning of the file (BOM specifies that file is in UTF-8 or UTF-16 encoding).
The reader should allow this marker. The writer ... I'm not sure, but I think it should not write this marker to the file (for compatibility reasons).


As to dash in alphanumeric moves.
It should be optional. Disallowing the 'a1b2' is very unfriendly.

Hi Sancoder,

in PGN there is a section about character codes: http://www.saremba.de/chessgml/standard ... e.htm#c4.1. It recommends to use only ASCII characters. This seems questionable to me, so I did not copy it into the PDN standard.

It seems to me that STRING values and COMMENT values should be allowed to contain Unicode characters. Note that the current token definitions already allow this. It depends on the parser generator whether or not Unicode characters are accepted. All of the example implementations accept the following UTF-8 input:

[White "Сергей Фадеев"]
[Black "高文龙"]
1.32-28 19-23

So I believe that the grammar does not need to be changed to support Unicode. Note that the current grammar does not allow Unicode characters to be used in other places, like tag names. This was done on purpose.

On the one hand I can understand that it is useful to restrict PDN to a single format like UTF-8. On the other hand I find it questionable to disallow ASCII files. I can't imagine PDN readers that would reject a PDN file because it is encoded in ASCII. Are there more opinions about this?

[sancoder]

It seems to me that STRING values and COMMENT values should be allowed to contain Unicode characters.

That's good.

On the one hand I can understand that it is useful to restrict PDN to a single format like UTF-8. On the other hand I find it questionable to disallow ASCII files. I can't imagine PDN readers that would reject a PDN file because it is encoded in ASCII. Are there more opinions about this?

I see no problem here. ASCII is subset of UTF-8. So, if the reader accepts UTF-8, it already accepts ASCII as well.

[sancoder]

in PGN there is a section about character codes: http://www.saremba.de/chessgml/standard ... e.htm#c4.1. It recommends to use only ASCII characters. This seems questionable to me, so I did not copy it into the PDN standard.

Read the section about character encodings.
The decision to use Latin-1 encoding is the bad one. I think they had no expert opinion on that area. Latin-1 is also the superset of ASCII, i.e. all ASCII files would be the same in Latin-1 encoding, but this encoding is not used in Cyrillic world as it doesn't contain neither Cyrillic nor other (Chinese, Japanese, Greek) characters. Latin-1 is widely used in Western Europe countries.
Wikipedia tells that Latin-1 is the standard for HTML (think of this like historical point), but for XHTML the standard encoding is UTF-8.

[Wieger Wesselink]

It seems to me that STRING values and COMMENT values should be allowed to contain Unicode characters.

That's good.

On the one hand I can understand that it is useful to restrict PDN to a single format like UTF-8. On the other hand I find it questionable to disallow ASCII files. I can't imagine PDN readers that would reject a PDN file because it is encoded in ASCII. Are there more opinions about this?

I see no problem here. ASCII is subset of UTF-8. So, if the reader accepts UTF-8, it already accepts ASCII as well.

Thanks for your explanations! As far as I'm concerned UTF-8 will become the character encoding of the PDN standard.

[jkrabbenbos]

Hello Jan, thanks for your comments! The link to the DGT clock time extension document can be found in the reference section: http://10x10.org/pdn/introduction.html#references. I read somewhere that this proposal was rejected for PGN, and so I did not make it part of the PDN 3.0 draft. I only copied the syntax of embedded commands.

It did never make the 'official' standard, which has not been changed since its inception. But the extended version has been adopted by all computer chess programmers, DGT and more. And has become a de facto standard.

I will create some PGNs and PDNs from the digital boards and mail them to you.

[VincentG]

- FEN tag: how to specify an unknown turn? I use FEN to send raw board scans to software. E.g. "?:W31-50:B1-20"
- GameType: what are the codes for for Chess960 and Shogi?
- Character encoding: UTF-8 and Latin-1 are incompatible. PGN more or less specifies Latin-1; must PDN viewers also accept Latin-1 if UTF-8 parsing fails?
- Line length: what does PDN 3.0 say about this? PGN specifies 255 as maximum and <80 as recommended.
- Below is an example PDN output by RabbitQueen 1.01: it uses PGN-style comments: lines starting with '%' until end-of-line. Much more commentary is generated when some debugging options are activated.

% Game list created by DGT Rabbit
% Constructed at 2012-Feb-01 12:35:20

% Board game: International Draughts 10x10

% -- Game #1 --
% Fragment 1 of 2
[Event "?"]
[Site "?"]
[Date "2012.02.01"]
[Round "?"]
[White "Game_1"]
[Black "Fragment_1"]
[Result "*"]
[Variant "International Draughts"]
[GameType "20"]
1. 33-28 17-22 2. 28x17 11x22 3. 32-28 22x33 4. 39x28 20-24 5. 38-32 19-23
6. 28x19x30 *

% Moves with takebacks:
% 1. 33-28 17-22 2. 28x17 12x21 Takeback 11x22 3. 32-28 22x33 4. 39x28 20-24
% 5. 38-32 19-23 6. 28x19x30 *


% Fragment 2 of 2
% Transition changes: M33 M27 K30 M22 k14 .50 .43 .45 .35 .15 .2 .5
[Event "?"]
[Site "?"]
[Date "2012.02.01"]
[Round "?"]
[White "Game_1"]
[Black "Fragment_2"]
[Result "*"]
[Variant "International Draughts"]
[GameType "20"]
[SetUp "1"]
[FEN "B:W27,22,K25,16,17,18,19,11,12,15,6,7,9,1,2,3,4:B46,48,49,41,42,43,44,45,37,38,K39,31,33"]
1. ... 13-19 2. 30x13x2x11 6x17x28x39x50 *

[Rein Halbersma]

The review period for Wieger Wesselink's draft proposal for the PDN 3.0 standard ended on February 1st. There were no official votes on this forum, but I think there has been a very constructive discussion about some of the subtleties. To the best of my knowledge, Wieger has adequately and constructively answered all questions.

Therefore, it is my great pleasure to announce that this standard has been CONDITIONALLY ACCEPTED.

Let me be the first to congratulate Wieger for his very valuable contributions. Let me also thank the reviewers on the various forums and by email for their insightful comments. I have received several private and public comments on the thoroughness of the documentation and the quality of Wieger's proposal. From private discussions with Wieger last summer (prior to the publication of the draft proposal), I can also personally attest to the exhaustive testing on a huge number of existing PDN files that Wieger went through in order to come up with a quality draft proposal.

The received comments have all been considered into my conditions for the accepted standard. Three main guidelines have been used:
1) Backward compatibility with existing PDN files out there.
2) Recognition of the differences between the various draughts variants throughout the world. This criterion is reflective of the diversity that the FMJD has to support.
3) When in doubt: "do as the chess guys do", i.e. take the PGN standard as a guideline.
Below I list the conditions in a Decision/Rationale/Action format.

Code: Select all

Decision: UTF-8 will be the mandatory character set in writing.
Rationale: Backward compatibility (see the comments from sancoder and MichelG on the FMJD forum).
Action: expand the documentation to explain this.

Code: Select all

Decision: a draughts variant's "native" type is the mandatory result format in writing.
Rationale: reflective of the diversity in draughts. This means that the 1-0 1/2-1/2 0-1 result format is not to be used with international draughts (see comments by Ed Gilbert on the FMJD forum). 
Action: expand the documentation to include a table of the native result format for each draughts variant.

Code: Select all

Decision: "exotic" results type (plus-draws, 3-0 wins, fractional points, Delft and Beijing formats) are supported through a ResultFormatTag in the game header.
Rationale: reflective of the diversity in draughts. 
Action: expand the documentation to include a table of values for the ResultFormatTag.

Code: Select all

Decision: elapsed time per move is as a comment in the DGT format (i.e. using the keywords %clk, %emt, %egt, etc.).
Rationale: backward compatibility and the existing practice in chess.
Action: expand the grammar to specify the comments keywords.

Code: Select all

Decision: matches with accumulating time usage (Georgiev-idea) are treated as separate games. The resulting asymmetric time controls at the start of each game are handled through a pair of TimeControlWhite and TimeControlBlack tags. 
Rationale: backward compatibility with existing PDN files (notably TurboDambase), while also enabling the full animation of the move-by-move time progress during such matches (e.g. on Toernooibase, see the comments by Piet Bouma on the FMJD forum).
Action: expand the grammar to specify the new tags.

Code: Select all

Decision: a draughts variant's "native" notation (alphanumeric vs. numeric and the move and capture separators) is the mandatory format for writing.
Rationale: reflective of the diversity in draughts. This means that Russian and Czech draughts have to be written in the alphanumeric format, that Russian captures have to use ":", and that Thai captures have to use "-". 
Action: expand the documentation to include a table of these formats and separators.

Code: Select all

Decision: the move separators are optional for reading, mandatory for writing in numeric format and only optional for writing in alpha-numeric format.  
Rationale: backward compatibility with existing PDN files (see comment by sancoder on the FMJD forum). Note that non-conforming programs that are writing numeric moves without separators, should use double-digit numeric values to at least maintain reading conformance. So 0106 can be read but 01-06 or 1-6 are mandatory for writing.
Action: expand the grammar and the documentation.

Code: Select all

Decision: captures can contain the full capture sequence in reading, but are only allowed to do so in writing for ambiguous captures.
Rationale: backward compatibility with existing (mostly Czech) PDN files (see comments by Gerard Taille on the FMJD forum).
Action: expand the documentation to reflect these semantics.

Code: Select all

Decision: disambiguated capture sequences have to specify the entire sequence of turning points. A turning point is either the square immediately behind a captured piece, or the square where a turn of direction was made. Leaving out a turning point that is not necessary for the disambiguation is forbidden.
Rationale: backward compatibility with existing PDN files and simplicity.
Action: expand the documentation to reflect these semantics.

The course of events is now that Wieger will implement the modifications above in the coming weeks, and that the resulting PDN 3.0 grammar and documentation become the new FMJD standard.

On behalf of the World Draughts Federation FMJD,

Wieger Wesselink - PDN 3.0 draft submitter
Rein Halbersma – Review Manager
Marcel Kosters – Review Wizard
Frank Teer – FMJD Secretary General

[sancoder]

Congratulations to Wieger!

Rein, let me point that UTF-8 is the encoding for the Unicode character set. Character set answers to question what characters are allowed, the encoding answers to the question how they are stored in bytes.

[Wieger Wesselink]

Hi Vincent, thanks for your comments!

- FEN tag: how to specify an unknown turn? I use FEN to send raw board scans to software. E.g. "?:W31-50:B1-20"

This seems a useful addition to me, so I'll add it to the standard.

- GameType: what are the codes for for Chess960 and Shogi?

I have no idea.

- Character encoding: UTF-8 and Latin-1 are incompatible. PGN more or less specifies Latin-1; must PDN viewers also accept Latin-1 if UTF-8 parsing fails?

When writing PDN, UTF-8 must be used. When reading PDN, UTF-8 must be supported. But it is recommended to also support Latin-1.

- Line length: what does PDN 3.0 say about this? PGN specifies 255 as maximum and <80 as recommended.

So far PDN 3.0 says nothing about it. Personally I don't like to impose such limitations. Is there a need for this?

- Below is an example PDN output by RabbitQueen 1.01: it uses PGN-style comments: lines starting with '%' until end-of-line. Much more commentary is generated when some debugging options are activated.

This was also requested by Jan Krabbenbos. I will add these PGN-style comments to the standard.

[Wieger Wesselink]

The review period for Wieger Wesselink's draft proposal for the PDN 3.0 standard ended on February 1st. There were no official votes on this forum, but I think there has been a very constructive discussion about some of the subtleties. To the best of my knowledge, Wieger has adequately and constructively answered all questions.

Therefore, it is my great pleasure to announce that this standard has been CONDITIONALLY ACCEPTED.

Let me be the first to congratulate Wieger for his very valuable contributions. Let me also thank the reviewers on the various forums and by email for their insightful comments. I have received several private and public comments on the thoroughness of the documentation and the quality of Wieger's proposal. From private discussions with Wieger last summer (prior to the publication of the draft proposal), I can also personally attest to the exhaustive testing on a huge number of existing PDN files that Wieger went through in order to come up with a quality draft proposal.

The received comments have all been considered into my conditions for the accepted, standard. Three main guidelines have been used:
1) Backward compatibility with existing PDN files out there.
2) Recognition of the differences between the various draughts variants throughout the world. This criterion is reflective of the diversity that the FMJD has to support.
3) When in doubt: "do as the chess guys do", i.e. take the PGN standard as a guideline.
Below I list the conditions in a Decision/Rationale/Action format.

Hi Rein, thanks for your report, and also thanks to all those who commented on the draft standard! In the coming month I will adjust the standard according to your decisions, and announce it here when it is ready.

[VincentG]

- GameType: what are the codes for for Chess960 and Shogi?

I have no idea.

Well, I propose 2 for Shogi and 10 for Chess960. And I've just updated the Wikipedia gametype list to match the PDN 3.0 proposal.

- Line length: what does PDN 3.0 say about this? PGN specifies 255 as maximum and <80 as recommended.

So far PDN 3.0 says nothing about it. Personally I don't like to impose such limitations. Is there a need for this?

If there is no limitation, I will put the entire game's movetext on a single line, including all annotations, possibly spanning thousands of characters. But if a naive programmer takes When in doubt: "do as the chess guys do", i.e. take the PGN standard as a guideline literally, his application might accept only 256 characters per line. So it would be good to make it clear in the specification.

[Ed Gilbert]

If there is no limitation, I will put the entire game's movetext on a single line, including all annotations, possibly spanning thousands of characters.

That is what I do. If I format the moves into multiple lines, then that creates problems everywhere that I want to use it. If I paste it into a web page, email, or view in a text editor, these environments have automatic word wrap that works much better when there are no newlines in the moves.

-- Ed

[Wishmaster]

I just have a question about this.
Not so long ago I got the makers of jijbent.nl to include PDN notation on the site for draughts games, and only a few weeks ago this has also been added for finished games in the history pages from players.
Will it be a lot of work to update this from PDN 2.0 to PDN 3.0?
And if they are willing to update it, will all main draughts programs (Turbo Dambase, Flits, Truus, Dam 2.2 etc.) be able to read the PDN 3.0 without requiring an update? Because if they won't be able to read it, I may wait a little longer before I inform them about the new standard.

Thanks.

[Wieger Wesselink]

If there is no limitation, I will put the entire game's movetext on a single line, including all annotations, possibly spanning thousands of characters.

That is what I do. If I format the moves into multiple lines, then that creates problems everywhere that I want to use it. If I paste it into a web page, email, or view in a text editor, these environments have automatic word wrap that works much better when there are no newlines in the moves.

-- Ed

This is a good example of why I'm reluctant to put a limitation on the line length.

[Wieger Wesselink]

I just have a question about this.
Not so long ago I got the makers of jijbent.nl to include PDN notation on the site for draughts games, and only a few weeks ago this has also been added for finished games in the history pages from players.
Will it be a lot of work to update this from PDN 2.0 to PDN 3.0?
And if they are willing to update it, will all main draughts programs (Turbo Dambase, Flits, Truus, Dam 2.2 etc.) be able to read the PDN 3.0 without requiring an update? Because if they won't be able to read it, I may wait a little longer before I inform them about the new standard.

Thanks.

I expect that it will not be difficult to update from PDN 2.0 to PDN 3.0. But I would advise to wait until the final version is ready. I will update the PDN checker as well, so it will be easy to test for compliance with the standard.

Most of the changes in PDN 3.0 are backward compatible with PDN 2.0. For example in PDN 3.0 the '*' is the only valid game separator, but the '*' is already valid in PDN 2.0.