feat: Add compatibility with chardet 6.0.0+ and fix encoding issues

[nathan-stender]

Summary

• Updates allotropy to be compatible with chardet 6.0.0+ (including chardet 7.x)
• Fixes encoding detection issues that were causing test failures
• Corrects mojibake in test data files

Changes

• Updated chardet requirement from < 6.0.0 to >= 6.0.0 in pyproject.toml
• Improved encoding detection logic in src/allotropy/parsers/utils/encoding.py:
  • Added fallback to Windows-1252 for very low confidence detections (< 0.3)
  • Better handling of single-byte special characters (en dash \x96, registered trademark \xae, micro symbol \xb5)
  • Added BOM (Byte Order Mark) stripping for UTF-16 and UTF-8 files
• Fixed test data that contained mojibake from incorrect encoding detection:
  • Corrected "®" to "®" in Luminex test JSON files
  • Fixed "�" (replacement character) to proper "µl" in test expectations

Background

Chardet 6.0.0 was released on February 22, 2026, followed by 7.0.x releases in March 2026. These versions introduced breaking changes in how they detect character encodings, particularly for:

• Short byte sequences with Windows-1252 characters
• Files with UTF-16 LE encoding (used by SoftMax Pro)
• Single-byte special characters like en dash, registered trademark, and micro symbol

Testing

• All 815 tests pass with chardet 7.0.1
• Specifically tested:
  • Luminex Intelliflex parser (UTF-8 with special characters)
  • Luminex xPONENT parser (Windows-1252 µ character)
  • MolDev SoftMax Pro parser (UTF-16 LE with BOM)
  • Encoding utility tests for Windows-1252 detection

🤖 Generated with Claude Code