Skip to content

Switch to PyTorch Tokenizers #694

@IgorSwat

Description

@IgorSwat

ExecuTorch 1.0 comes with its own tokenizers implementation based on PyTorch Tokenizers. Since it is already being bundled into ExecuTorch binaries, we could get rid of tokenizers-cpp package and reduce the bundle size of our library.

Metadata

Metadata

Assignees

Labels

executorchIssues and tasks related specifically to ExecuTorchimprovementPRs or issues focused on improvements in the current codebase

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions