Enhancement: 國際化 word boundary 檢查

[HCYT]

來源

• flashtext PR#49

問題描述

目前 non_word_boundaries 是硬編碼的 ASCII 字符集：

self.non_word_boundaries = set(string.digits + string.ascii_letters + '_')

這對非 ASCII 語言不夠友好。

建議方案

使用 Python 的 \W regex class 或 Unicode category 來判斷 word boundary：

import unicodedata

def is_word_char(char):
    category = unicodedata.category(char)
    return category.startswith(('L', 'N'))  # Letter or Number

考量

• 效能影響：需要 benchmark 測試
• 向後兼容：可能需要新增參數 unicode_boundaries=True

[HCYT]

Fixed in #4. Implemented International Word Boundaries using WordBoundarySet.

[HCYT]

Status Update: Reopened

We have reverted the "Internationalized Word Boundaries" feature (previously implemented in 1373516) due to significant performance regression.

Performance Impact:

• Baseline (v2.7): ~0.25s
• With i18n Boundaries: ~0.83s (3.5x slower)
• After Revert & Optimization: ~0.27s

Next Steps:
This feature is critical for non-English support but must be re-implemented with a "Performance First" approach. Future implementations should consider:

1. C/Rust extensions for boundary checks.
2. Pre-computed lookup tables to keep Python overhead <5%.