A minor modification of the spiderable package with caching. This
helps reduce CPU load on servers that serve a potentially large number
of pages, and also drastically reduces page load time, which is
essential for SEO.
Caching happens in two forms:
- by using the
--disk-cache=trueoption of phantomjs - by actually caching the rendered html in the database
Set Spiderable.CACHE_TIME to the desired lifetime of the database
cache in ms. Default if unset: 30 days.