Proteus: A Deterministic Entropy Benchmark for Scraper Testing

[copyleftdev]

Hi everyone! 👋

I built Proteus, a specialized benchmark target to be a "Crash Test Dummy" for scrapers and anti-bot systems.

Unlike standard "scraper test sites" that are static, Proteus implements a Deterministic Entropy Engine. By modifying URL parameters, you can control the exact layout chaos:

• ?seed=123: Guarantees 100% reproducible DOM structure (great for regression testing).
• ?entropy=1.0: Maximizes DOM mutation (shuffled nodes, class injection, text obfuscation).

Why use it with Crawlee?
It answers the question: "Is my scraper breaking because of a real anti-bot update, or just brittle selectors?"

I also generated a test_cases.json suite active on the repo if you want to plug it into your CI:
Proteus Test Suite (Gist)

Would love feedback from the community on what other mutation patterns break your crawlers! 🕷️

[janbuchar]

Pretty cool! Couple of questions:

1. Are you planning to open source it at some point?
2. Can you elaborate on how this should be used with Crawlee? I can imagine using it for integration tests, as a scraping playground of sorts - like https://books.toscrape.com/. What I don't understand is how Proteus helps scrapers fix brittle selectors - care to explain? 🙂