Deprecate and remove Modest based parser in favor of Lexbor

[pygarap]

This issue proposes to remove the Modest based parser from selectolax, and to keep only the Lexbor based parser as the recommended and supported option.

The project already supports the Lexbor parser (https://github.com/lexbor/lexbor), and the documentation and examples guide users to use the Lexbor based parser as the default choice.

In addition, the Modest project (https://github.com/lexborisov/Modest) clearly suggests that users should move to Lexbor. Its README says that Lexbor is stable, has more features, and is very fast. Because of this, Modest looks like a legacy choice now.

Given all of the above, it seems like a good time to start deprecating Modest support in selectolax, and later to remove it completely.

Proposed plan

1. Start a clear deprecation phase

   • Mark all public Modest based parser entry points as deprecated.
   • Use the new @warnings.deprecated decorator on Python 3.13 and newer.
   • For older Python versions, use @deprecated.deprecated as a fallback, so that users still get a clear warning.

   The deprecation message can explicitly recommend switching to the Lexbor based parser.

2. Update documentation and examples

   • Remove Modest based examples from the README and docs.
   • Clearly state that Lexbor is the only recommended HTML parser backend.
   • Mention that Modest support is deprecated and will be removed in a future release.

3. Plan final removal

   • After at least one full release with deprecation warnings, remove the Modest based parser code and any related configuration or tests.
   • Clean up any remaining references to Modest in the codebase and documentation.

This change would simplify the codebase, reduce maintenance, and align selectolax with the guidance from the Modest project itself, while giving users a clear and gradual path to migrate to the Lexbor based parser.

[rushter]

We can't really use a decorator. We can only add a warning when importing an old module. Some people call the parser millions of times, and that would result in millions of log messages.

[john-friedman]

I am against deprecating Modest. On my workloads, selectolax's modest outperforms lexbor. Could be that I'm doing something wrong, overhead in bindings, or lexbor might just be slower?

Name	Lexbor	Modest
iter	0.1038	0.0918
traverse	0.0234	0.0131
traverse+memid	0.0738	0.0640
traverse signals	0.0851	0.0749

data: https://www.sec.gov/Archives/edgar/data/19617/000001961725000270/jpm-20241231.htm

from selectolax.lexbor import LexborHTMLParser
from selectolax.parser import HTMLParser

from time import time
def walk(node):
    yield ("start", node)
    for child in node.iter(include_text=True):
        yield from walk(child)
    yield ("end", node)

def traverse_signals(node):
    """Yields (signal, node) tuples like walk() but using native traverse()"""
    stack = []
    
    for current_node in node.traverse(include_text=True):
        parent = current_node.parent
        parent_mem_id = parent.mem_id if parent else None
        
        # Emit "end" signals for nodes we've exited
        while stack and stack[-1][0] != parent_mem_id:
            _, exited_node = stack.pop()
            yield ("end", exited_node)
        
        # Emit "start" signal for current node
        yield ("start", current_node)
        stack.append((current_node.mem_id, current_node))
    
    # Emit remaining "end" signals
    while stack:
        _, exited_node = stack.pop()
        yield ("end", exited_node)

with open('examples/html/jpm-20241231.html','r') as f:
    content = f.read()


def benchmark(body):
    s = time()
    nodeslen = 0
    for signal, node in walk(body):
        nodeslen+=1

    print(f"iter time:{time()-s}:{nodeslen/2}")

    s = time()
    nodeslen = 0
    for  node in body.traverse(include_text=True):
        nodeslen+=1
    print(f"traverse time:{time()-s}:{nodeslen}")

    s = time()
    nodeslen = 0
    stack = []  # Just store mem_ids

    for node in body.traverse(include_text=True):

        # Check if we need to pop (moved to sibling or ancestor)
        # Pop until we find node's parent in stack, or stack is empty
        parent_mem_id = node.parent.mem_id if node.parent else None
        while stack and stack[-1] != parent_mem_id:
            nodeslen += 1  # "end"
            stack.pop()
        
        nodeslen += 1  # "start"
        stack.append(node.mem_id)

    # Emit remaining "end"s
    while stack:
        stack.pop()
        nodeslen += 1

    elapsed = time() - s
    print(f"traverse+memid time: {elapsed:.6f}s | signals: {nodeslen} | actual nodes: {nodeslen/2}")

    s = time()
    nodeslen = 0
    for signal, node in traverse_signals(body):

        nodeslen+=1

    print(f"traverse sginals time:{time()-s}:{nodeslen/2}")


print("Lexbor")
parser = LexborHTMLParser(content)
body = parser.body
benchmark(body)
print("Modest")
parser = HTMLParser(content)
body = parser.body
benchmark(body)

[john-friedman]

Also thank you for noting: "that would result in millions of log messages". I frequently process tens of millions of html files using selectolax. I very much appreciate this package.

[lexborisov]

@john-friedman

I am against deprecating Modest. On my workloads, selectolax's modest outperforms lexbor. Could be that I'm doing something wrong, overhead in bindings, or lexbor might just be slower?

Please don't use Modest at all.
I am the developer of Modest and Lexbor. The HTML specification has been updated many times, and Lexbor constantly supports the current specification. Modest has not been updated since approximately 2018. It is now 2026.
You are comparing apples and oranges.

Regarding performance.
The specifications (HTML) are dynamic and constantly changing; some aspects become more complex, while others improve. I would not compare them in this manner.

Once again, please do not use Modest (I really like this name; it was a reference to Modest Petrovich Mussorgsky (I am a pianist :))).

[john-friedman]

Very good to hear from you @lexborisov. I owe you a debt of gratitude.

I think for my use case, processing terabytes of government html (sec corpus), newer spec compliance doesn't apply. For example, SEC filings still use '.htm' extensions. But I am an edge case, so if deprecation makes sense for the whole, it makes sense.

I was actually thinking of reaching out to you via email earlier. Will follow up with some questions there, to avoid polluting this thread.

I did not realize Modest was a pianist, I assumed the literal English meaning. Very cool to learn! I probably have heard his work via Fantasia.

[lexborisov]

I think for my use case, processing terabytes of government html (sec corpus), newer spec compliance doesn't apply. For example, SEC filings still use '.htm' extensions. But I am an edge case, so if deprecation makes sense for the whole, it makes sense.

HTML (well, many are used to HTML5, but in fact it is already a Living Standard) fully supports older HTML formats, so I don't think you need to worry about that.

terabytes of government html

That's very impressive

I was actually thinking of reaching out to you via email earlier. Will follow up with some questions there, to avoid polluting this thread.

I am always open to communication. I receive many messages via email or Telegram directly. I am often busy and cannot respond immediately, but I try to respond to everyone.

I did not realize Modest was a pianist, I assumed the literal English meaning. Very cool to learn! I probably have heard his work via Fantasia.

Subtle reference)

[john-friedman]

No worries, will try email and see if fortune smiles on me. Trying to decide whether it makes sense for me to do a minimal lexbor port to Rust.

[rushter]

I am against deprecating Modest. On my workloads, selectolax's modest outperforms lexbor. Could be that I'm doing something wrong, overhead in bindings, or lexbor might just be slower?

Name Lexbor Modest
iter 0.1038 0.0918
traverse 0.0234 0.0131
traverse+memid 0.0738 0.0640
traverse signals 0.0851 0.0749
data: https://www.sec.gov/Archives/edgar/data/19617/000001961725000270/jpm-20241231.htm

from selectolax.lexbor import LexborHTMLParser
from selectolax.parser import HTMLParser

from time import time
def walk(node):
    yield ("start", node)
    for child in node.iter(include_text=True):
        yield from walk(child)
    yield ("end", node)

def traverse_signals(node):
    """Yields (signal, node) tuples like walk() but using native traverse()"""
    stack = []
    
    for current_node in node.traverse(include_text=True):
        parent = current_node.parent
        parent_mem_id = parent.mem_id if parent else None
        
        # Emit "end" signals for nodes we've exited
        while stack and stack[-1][0] != parent_mem_id:
            _, exited_node = stack.pop()
            yield ("end", exited_node)
        
        # Emit "start" signal for current node
        yield ("start", current_node)
        stack.append((current_node.mem_id, current_node))
    
    # Emit remaining "end" signals
    while stack:
        _, exited_node = stack.pop()
        yield ("end", exited_node)

with open('examples/html/jpm-20241231.html','r') as f:
    content = f.read()


def benchmark(body):
    s = time()
    nodeslen = 0
    for signal, node in walk(body):
        nodeslen+=1

    print(f"iter time:{time()-s}:{nodeslen/2}")

    s = time()
    nodeslen = 0
    for  node in body.traverse(include_text=True):
        nodeslen+=1
    print(f"traverse time:{time()-s}:{nodeslen}")

    s = time()
    nodeslen = 0
    stack = []  # Just store mem_ids

    for node in body.traverse(include_text=True):

        # Check if we need to pop (moved to sibling or ancestor)
        # Pop until we find node's parent in stack, or stack is empty
        parent_mem_id = node.parent.mem_id if node.parent else None
        while stack and stack[-1] != parent_mem_id:
            nodeslen += 1  # "end"
            stack.pop()
        
        nodeslen += 1  # "start"
        stack.append(node.mem_id)

    # Emit remaining "end"s
    while stack:
        stack.pop()
        nodeslen += 1

    elapsed = time() - s
    print(f"traverse+memid time: {elapsed:.6f}s | signals: {nodeslen} | actual nodes: {nodeslen/2}")

    s = time()
    nodeslen = 0
    for signal, node in traverse_signals(body):

        nodeslen+=1

    print(f"traverse sginals time:{time()-s}:{nodeslen/2}")


print("Lexbor")
parser = LexborHTMLParser(content)
body = parser.body
benchmark(body)
print("Modest")
parser = HTMLParser(content)
body = parser.body
benchmark(body)

If you need some extra speed gains, we can think about on how to improve it for your tasks. Some things in the wrapper are still not optimized well.
Usage of mem_id hints that this can be done faster, most likely.

[john-friedman]

Hi @rushter, I would appreciate that. My parsing algorithm needs start/exit signals for nodes.

Benchmark for specific 13mb file:

• .traverse() : 2ms
• .iter() modified for start/exit: 12ms
  -. traverser() modified with mem id for start/exit: 10ms

Is this something worth optimizing? I think am an edge user, so don't want to waste your time.

I'm currently looking into rewriting my parser in rust where I think the overhead from tracking nodes will be much less. (Working on a minimal lexbor port to rust, as lexbor > scrapper by ~2x)

Edit: Looks like I can implement parser in Cython. I've forked selectolax, and am writing the modified traverse as a util.

[lexborisov]

Benchmark for specific 13mb file:

• .traverse() : 2ms
• .iter() modified for start/exit: 12ms
  -. traverser() modified with mem id for start/exit: 10ms

What are your requirements, i.e., what speeds are you aiming for? What is the issue with performance?

[john-friedman]

Hi @lexborisov, I forked selectolax and implemented a cython traverse() that emitted enter/exit signals for nodes. It is ~30% slower than traverse(), so the overhead was due to the python interfacing.

I'm now implementing my document parser in cython on top of lexbor, using selectolax's binding as guidance. So far, my document parser gets 100+ megabytes per second--enough to saturate residential bandwidth on a single thread.

Seventy percent of this is Lexbor constructing the tree. I'm considering forking Lexbor in the future, and using it for inspiration to write a specialized parser for my desired behavior. Hoping to maybe squeeze enough to saturate database bandwidth (I use AWS).

Don't need any help right now, but just wanted to thank you again. Is there a proper place to ask you questions related to lexbor that aren't directly related to the package? I don't want to clog up threads.

P.S. Lexbor is really good. I benchmarked a couple other parsers while doing my rewrite, and wow. Really really good performance.

[lexborisov]

Hi @john-friedman,

Thanks,
You can open a discussion on GitHub. I mean, it's better to discuss questions and problems publicly; someone else might need it or find it useful.