feat: add code chunking functionality

[bridgetmcg]

This PR introduces code chunking functionality to docling-core, enabling intelligent parsing and chunking of source code files across multiple programming languages. The implementation leverages tree-sitter for accurate parsing and provides language-specific chunkers for Python, TypeScript, JavaScript, Java, and C.

Features

Core

• CodeChunker - Base abstract class for code chunking with Tree-sitter integration
• Language-specific chunkers - Specialized implementations for 5 major programming languages
• Smart chunk splitting - Automatic splitting of large functions while preserving context

Language Support

• Python (PythonFunctionChunker) - Functions, classes, imports, module variables
• TypeScript (TypeScriptFunctionChunker) - Functions, classes, interfaces, imports
• JavaScript (JavaScriptFunctionChunker) - Inherits from TypeScript chunker
• Java (JavaFunctionChunker) - Methods, constructors, classes, enums, interfaces
• C (CFunctionChunker) - Functions, structs, macros, preprocessor definitions

Testing

• test_code_chunker.py - multi-language, real code samples

[github-actions]

✅ DCO Check Passed

Thanks @bridgetmcg, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

[dosubot]

Related Documentation

Checked 3 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[vagenas]

Nice! Let me share some first thoughts, mostly on how these capabilities can be packaged and exposed:

I see the PR contains various language-specific chunkers, e.g. Java, Python etc.
My recommendation would be:

• to encapsulate these capabilities under a single component, which would include the language detection inside it, and,
• to ensure composability with existing chunkers, instead of introducing a new chunker, I'd rather provide this as a capability pluggable into the HierarchicalChunker (good fit because (1) it follows Item boundaries, so CodeItems can be nicely delegated, and (2) is itself composable into the HybridChunker). Let us still define the interface specifics, but it could look like an optional kwarg in HierarchicalChunker.chunk(), e.g. code_chunking_strategy, adhering to a matching interface.

A more minor comment is regarding CodeDocMeta, which I see inherits from BaseMeta. Some application code may expect to interact with DocMeta (also a BaseMeta child), so if possible we should best extend that.
This point goes hand in hand with the interface specifics TBD above.

(For now I would focus on the present PR & the points above — the idea of introducing a code backend as per the docling repo PR can be discussed at a second step.)

[bridgetmcg]

hi @vagenas, I added some logic to address your suggestions.

[vagenas]

Hi @bridgetmcg, many thanks for the new iteration, incorporating feedback from above!

I only have a couple last points from my side (@dolfim-ibm I don't know if you want to add anything):

1. I see the actual integration meanwhile occurs via HierarchicalChunker -> DefaultCodeChunkingStrategy -> CodeChunkingStrategyFactory which returns a CodeChunker (subclass of BaseChunker). Since the chunker primitive should be runnable on any document, it could be confusing to expose a component that operates only on e.g. Python, as a "chunker". Ideally one could just satisfy the newly introduced interface/protocol (and not BaseChunker which may require additional points), but perhaps the fastest way to address this point is just to mark the CodeChunker class hierarchy "internal" by prepending with _. Then it's clear all these classes are implementation internals, and users only need to care about the strategy they can optionally pass to HierarchicalChunker.
2. the defined CodeChunkingStrategy(Protocol) does not seem to be used anywhere — shouldn't this be somewhere in the typing of field code_chunking_strategy within HierarchicalChunker?
3. to allow for extensibility we support the notion of customizable serializers; while this perhaps makes little sense in CodeItems, for consistency, in HierarchicalChunker we should still best use the result from doc_serializer (instead of just item.text)

Hope that makes sense. Otherwise we can also have a quick call to clarify & finalize.

[dolfim-ibm]

to avoid users thinking this is a generic ChunkType, we could rename the class to CodeChunkType.

[dolfim-ibm]

Looking at the discussion and the proposed code_chunking_strategy, I'm think we want to consider how this will scale to other types. For example, multiple chunking strategies for tables are a recurrent topic.

[vagenas]

Looking at the discussion and the proposed code_chunking_strategy, I'm think we want to consider how this will scale to other types. For example, multiple chunking strategies for tables are a recurrent topic.

What about the obvious way to expand this, i.e. adding an optional table_chunking_strategy to the HierarchicalChunker?

[dolfim-ibm]

Looking at the discussion and the proposed code_chunking_strategy, I'm think we want to consider how this will scale to other types. For example, multiple chunking strategies for tables are a recurrent topic.

What about the obvious way to expand this, i.e. adding an optional table_chunking_strategy to the HierarchicalChunker?

I was just looking up what we do in the serializers, there we are indeed using this approach. So let's go with it.

[bridgetmcg]

@vagenas I believe I addressed your comments! Let me know if not. Many thanks!

[vagenas]

@bridgetmcg I made some in-line comments incl. code suggestions.
Please also install the pre-commit hooks, so all checks are verified locally before pushing.
(E.g. I think some tests are still not up-to-date. FYI to generate the data, set env var DOCLING_GEN_TEST_DATA=1, e.g. DOCLING_GEN_TEST_DATA=1 uv run pytest)

[bridgetmcg]

@vagenas I ran all the pre-commit checks which showed that now with the language identification we can correctly label some code snippets from other tests. Those were updated in 814dc61

[vagenas]

@bridgetmcg sounds good, now we still need to address the conflicts on uv.lock (not possible manually), which means:

1. branch needs to get up-to-date with latest main
2. uv.lock needs to be regenerated
3. (if 1. is done by rebasing, a force-push would be needed)

Can you take care of these? Otherwise let me know and I could try to look into it.

[bridgetmcg]

@vagenas I had to restrict the tree-sitter versioning due to python compatibility. treesitter > 0.24 requires python 3.10+ and 0.23 requires all the treesitter language libraries to be <0.24 as well.

[codecov]

Codecov Report

❌ Patch coverage is 85.46410% with 166 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...s/chunker/code_chunking/_language_code_chunkers.py	83.73%	157 Missing ⚠️
...ng_core/transforms/chunker/code_chunking/_utils.py	91.93%	5 Missing ⚠️
...r/code_chunking/standard_code_chunking_strategy.py	90.00%	3 Missing ⚠️
docling_core/transforms/chunker/doc_chunk.py	97.14%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[vagenas]

These backticks look like a bug, perhaps to best be addressed on a different level.

[vagenas]

Can make a BaseCodeChunkingStrategy(ABC) out of this, with abstract method

[vagenas]

is this not used currently?

[vagenas]

@bridgetmcg can you fix the DCO?

[vagenas]

Hi @bridgetmcg, I did some refactoring & made couple improvements — details in 641ea9c commit message (also removed the previous change of doc_items type in CodeDocMeta).

[PeterStaar-IBM]

wonderful! 🎖️