Invalid EP results in bad binpack conversion

[borg323]

The attached .plain file is converted to a bad .binpack file but passes validation. Conversion to .bin works, but again converting the .bin to .binpack again results in a bad file.

test2.plain.zip

Detailed transcript:

$ ./stockfish convert test2.plain test2.binpack verify 
Stockfish 020421 by the Stockfish developers (see AUTHORS file)
info string Loaded eval file nn-c3ca321c51c9.nnue
Converting test2.plain to test2.binpack
Finished. Converted 1389 positions.
$
$ ./stockfish convert test2.binpack bad.plain verify 
Stockfish 020421 by the Stockfish developers (see AUTHORS file)
info string Loaded eval file nn-c3ca321c51c9.nnue
Converting test2.binpack to bad.plain
stockfish: ./extra/nnue_data_binpack_format.h:1238: constexpr chess::File chess::Square::file() const: Assertion `isOk()' failed.
Aborted (core dumped)
$
$ ./stockfish convert test2.plain test2.bin verify 
Stockfish 020421 by the Stockfish developers (see AUTHORS file)
info string Loaded eval file nn-c3ca321c51c9.nnue
Converting test2.plain to test2.bin
Processed 55560 bytes and 1389 positions.
Finished. Converted 1389 positions.
$
$ ./stockfish convert test2.bin good.plain verify 
Stockfish 020421 by the Stockfish developers (see AUTHORS file)
info string Loaded eval file nn-c3ca321c51c9.nnue
Converting test2.bin to good.plain
Processed 141011 bytes and 1389 positions.
Finished. Converted 1389 positions.
$
$ ./stockfish convert test2.bin test2.binpack verify 
Stockfish 020421 by the Stockfish developers (see AUTHORS file)
info string Loaded eval file nn-c3ca321c51c9.nnue
Converting test2.bin to test2.binpack
Finished. Converted 1389 positions.
$
$ ./stockfish convert test2.binpack bad.plain verify 
Stockfish 020421 by the Stockfish developers (see AUTHORS file)
info string Loaded eval file nn-c3ca321c51c9.nnue
Converting test2.binpack to bad.plain
stockfish: ./extra/nnue_data_binpack_format.h:1238: constexpr chess::File chess::Square::file() const: Assertion `isOk()' failed.
Aborted (core dumped)

[Sopel97]

triggerred by

fen Q1N1b3/5pnk/3R3p/p7/1pp1p3/PnP1P2P/2B2PP1/5RK1 b - c3 0 31
move g7f5
score -23364
ply 61
result -1
e
fen Q1N1b3/5p1k/3R3p/p4n2/1pp1p3/PnP1P2P/2B2PP1/5RK1 w - - 1 32
move a8e4
score 24189
ply 62
result 1
e

stockfish.exe convert a.plain a.binpack
stockfish.exe convert a.binpack b.plain

[Sopel97]

fen Q1N1b3/5pnk/3R3p/p7/1pp1p3/PnP1P2P/2B2PP1/5RK1 b - - 0 31
move g7f5
score -23364
ply 61
result -1
e
fen Q1N1b3/5p1k/3R3p/p4n2/1pp1p3/PnP1P2P/2B2PP1/5RK1 w - - 1 32
move a8e4
score 24189
ply 62
result 1
e

doesn't trigger it.

Somehow caused by the extraneus ep square in the fen...

[Sopel97]

This is technically an illegal fen, so... I could dig it up and find out why it causes the error, but I'm more inclined to say the the .plain writer has an error.

[Sopel97]

Okay, seems to be caused by false EP removal code not catching some illegal positions, I'll add some checks just for the sake of this.

[Sopel97]

This cannot be fixed without a version change. There were some false ep squares in the historical data that encoded correctly but requires the false ep information for decoding.

Basically the way encoding works is it uses the number of pseudo legal moves possible for a piece on the board to know how many bits to use for a move. The number of pseudo legal moves is affected by false ep square, even though such a move would never happen to be decoded, effectively making it impossible to catch in validation, but being required for decoding to work.

[borg323]

I have fixed the code that resulted in bad EP fields, so this isn't urgent for me.