project-scraper

Code for scraping communities and other interesting projects from public websites.

Project structure

• data: extracted datasets
• project_scraper: Scrapy project for data harvesting

Installing

1. Make sure you have GIT and GIT-LFS installed.
2. Clone this repo: git clone https://github.com/dsrp/project-scraper.git.
3. Make sure you have pipenv available; pip install pipenv
4. Clone repo: git clone https://github.com/dsrp/project-scraper.git and change into project directory.
5. Locally install dependencies: pipenv install

Running

1. pipenv shell
2. scrapy crawl <fic|gen>