A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

Fully neural approach for text chunking

🍱 semantic-chunking ⇢ semantically create chunks from large document for passing to LLM workflows

🍶 llm-distillery ⇢ use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.

Token-aware, LangChain-compatible semantic chunker with PDF, markdown, and layout support

Semantic Chunking is a Python library for segmenting text into meaningful chunks using embeddings from Sentence Transformers.

Advanced semantic text chunking with custom structural markers, whole-text coherence preservation, and flexible token management. Features async processing, LangChain integration, and dynamic drift detection. Ideal for RAG systems, augmented text processing, and domain-specific document analysis.

🤖 Automated Q&A Dataset Generation Pipeline powered by LLMs. Multi-stage pipeline that searches, filters, extracts and transforms web content into high-quality question-answer datasets for LLM training. Supports multiple LLM providers (Groq, Mistral, Ollama) and search engines.

All in One-Solution for converting documents to finetune LLMs

HR Policy Assistant (RAG-based Chatbot) A conversational AI assistant for employees to query company HR policies. Built with LangChain and Qdrant, it semantically ingests HR documents, retrieves relevant policy information, reranks results with BM25/MMR, and delivers precise LLM-generated responses.Cloud-based vector storage ensure quick responses.

Lightweight, composable TypeScript library for semantic chunking, workflow pipelining, and LLM orchestration.

Cutting-edge semantic text processing system that uses hierarchical clustering and advanced language models to automatically organize and summarize large volumes of text.

``retrieval is all you need`` All in 1 repo for different levels of chunking along with their main logic and reusable code. No API keys used. Highly portable and pluggable

A Sidecar service for applications that need vector database functionality to augment their LLMs. This service provides embeddings and retrieval capabilities by abstracting embeddings generation (LiteLLM) and vector storage and search (Qdrant).

Retrieval-Augmented Generation (RAG) Fundamentals and Semantic Chunking

An exploration of advanced text splitting strategies in LangChain for RAG, from basic character splitting to state-of-the-art semantic chunking.