Reconstructing chunks to html file

[gideonpeters]

Recently came across this tool, and I must say it is game changer for LLMs in performing tasks that need html understanding. I'm considering using this for a study, but I was wondering if this package has the ability to reconstruct the chunks after they are split?

Or at least to identify the unique portions of every chunk.

[KLGR123]

Hi, thank you so much for your interest! 👋

What do you mean by "identifying the unique portions of every chunk"? If you're looking to reconstruct the chunks processed and split by html_chunking back into the complete HTML, the merge_html_chunk method fully supports that. You can try using:

html = merge_html_chunk(html_1, html_2)

to get the merged HTML of html_1 and html_2. This isn’t just a simple merge; it identifies their common (root node) paths and merges them according to the DOM structure. I didn’t include this function in the README because it's a simple implementation, without error-handling mechanisms, etc. However, it should work properly.

Regarding the unique portion identification you mentioned, my understanding is that it’s possible. The package can reverse the chunks back into larger chunks or the entire HTML.

[KLGR123]

Oh, and just a note that you can cite our work in your study, which I’ve already included in the README. Best of luck with your study! 😊