Allow providing custom katana configuration file or Developer Docs prefix URL string #4401
Description
Hi,
I think it would be pretty helpful to be able to provide some customization to the katana behavior when indexing a custom endpoint provided in the Developer Docs section.
The best solution would probably be to have a per-URL configuration that allows changing the katana behavior but looking at the current implementation I think an easy "fix" would be to allow specifying a custom prefix URL and, globally, allow creating an optional config.katana.txt or config.katana.flags alongside the config.toml file that gets loaded using katana's --config flag if detected. This would allow full customization of the katana behavior without increasing complexity of the current WebCrawlerJob implementation.
Additional context
See katana config flag and current crawl_url implementation.
Please reply with a 👍 if you want this feature.