fix(formatron): handle duplicate BPE token IDs and kbnf mask/accept inconsistency

[lesj0610]

Problem

Two related bugs affect constrained generation (json_schema, regex_pattern, grammar_string) with GPT-2 byte-level BPE tokenizers (e.g. EXAONE 4.x, GPT-J, GPT-NeoX, BLOOM):

Bug 1 — Duplicate token IDs dropped from vocabulary

get_vocab_dict() builds a {str: int} dict comprehension; when multiple token IDs share the same piece string, only the last ID survives. Earlier IDs are silently absent from new_vocab. When the model samples one of these dropped IDs, kbnf raises:

ValueError: The input token id is rejected and the EngineLike's internal states are not updated.

This crashes the generation loop.

Bug 2 — kbnf mask/accept inconsistency (kbnf ≤ 0.4.2)

compute_allowed_tokens() can report a token as valid while try_accept_new_token() subsequently rejects the same token with ValueError. In kbnf ≤ 0.4.2 this is triggered by GPT-2 special tokens (e.g. [PAD], id=0, bytes=b'[PAD]') whose literal ASCII bytes incidentally match valid grammar positions (e.g. inside a JSON string value).

The rejection propagates uncaught, crashing the calling process.

Fixes

1. create_engine_vocabulary() — back-fill duplicate token IDs

Build a str → bytes lookup from new_vocab, then iterate the full vocabulary and assign bytes for any ID that was dropped due to string deduplication.

2. accept_token() — defensive try/except around kbnf call

Wrap self._formatter.accept_token(token) in try/except ValueError and re-raise. This turns a process crash into a clean per-request abort that callers can handle gracefully.

This guard remains safe and beneficial even after a kbnf-side fix: it converts any future unforeseen mask/accept inconsistency into a recoverable error rather than a crash.

kbnf version note

kbnf ≥ 0.5.7 resolves the underlying mask/accept inconsistency in the Rust engine. The PyPI package is currently at 0.4.2; a pre-built wheel for Linux x86_64 is available at:

https://github.com/lesj0610/kbnf/releases/tag/v0.5.7

The setup.py comment points to this wheel until the upstream author publishes 0.5.7 to PyPI.

Affected models

Any model using a GPT-2 byte-level BPE tokenizer where special tokens are stored as literal ASCII strings — approximately 10–20% of open-source LLMs (EXAONE, GPT-J, GPT-NeoX, BLOOM, some Falcon variants).

Testing

Verified with EXAONE-4.0.1-32B + json_schema constrained generation: 10/10 requests succeed with kbnf 0.5.7; previously crashed on ~1 in 5 requests with kbnf 0.4.2.