GitHub - FrontPages/scraper: The lambda functions that scrape websites and upload their screenshots/headlines to Front Pages

Deployment

This repo uses Serverless Framework for deployment.

You must have a .env file in the root directory of the repo with the following environment variables set:

BUCKET_NAME: The name of the S3 bucket to store screenshots in.
FRONT_PAGES_API_KEY: The API key used to post snapshots to Front Pages.
FRONT_PAGES_BASE_URL (optional): The base URL for Front Pages API requests. If not provided, http://front-pages-sandbox.herokuapp.com will be used by default.

This file will be used by serverless-dotenv-plugin to publish these variables to production on deploy.

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
src/scripts		src/scripts
.env.example		.env.example
.gitignore		.gitignore
.prettierrc.yaml		.prettierrc.yaml
README.md		README.md
package.json		package.json
scraper.ts		scraper.ts
selector-tester.ts		selector-tester.ts
serverless.yml		serverless.yml
trigger.ts		trigger.ts
tsconfig.json		tsconfig.json
types.ts		types.ts
yarn.lock		yarn.lock