Decrease memory footprint of scrapy

[rimbi]

This can be done holding Request objects in DB instead of memory.

[sardok]

my investigation shows that scrappy is not collecting request object after got response for it but the request object are aggregated in schedular queue. scheduler is not fast enough as download of course and its queue is just growing.

[rimbi]

I've found the place wherein Request objects are stored in dequeue structures. I tried to insert an hook there but it's not an easy change. I thought I could use pickle module to store Request objects as strings in database but I 'm getting errors.
Also, this change requires changes in scrapy framework. So, we need the screapy repo as well.
By the way, during these development efforts I did realize the advantages of statically typed languages over dynamically typed languages in debugging errors. :)