Java library implementing Byte-Pair Encoding Tokenization

This project implements a tokenizer based on the Byte Pair Encoding (BPE) algorithm, with additional custom tokenizers, including one similar to the GPT-4 tokenizer.

A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.

[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust

A python package to build a corpus vocabulary using the byte pair methodology and also a tokenizer to tokenize input texts based on the built vocab.

Byte Pair Encoding (BPE)

From-scratch Marathi BPE tokenizer with Flask API and web interface for encoding/decoding Marathi text using Byte Pair Encoding algorithm.

Strings Tokenization with Byte Pair Encoding (BPE).

A web app to compare pre-built or self-built tokenizers

self made byte-pair-encoding tokenizer

This repository is reimplementation of Transformer model which was introduced in 2017 NeurIPS paper "Attention is all you need"

Natural Language Processing course assignments @ University of Tehran

implementation of BPE algorithm and training of the tokens generated

Byte Pair Encoding (BPE) in the Zig programming language (0.13.0)

Byte-pair Algorithm used in GPT Tokenization based on the paper "Language Model are unsupervised multitask learners"

A pure Python implementation of Byte Pair Encoding (BPE) tokenizer. Train on any text, encode/decode with saved models, and explore BPE tokenization fundamentals.

TF-IDF Calculation

tokenizer for large-scale language models (GPT, Claude, Llama, etc.)