tcdb-spider

1. Introduction

This is a web crawler for www.tcdb.com

There are 3 scripts that handle the entire crawl process:

Select a ball class like Football

Get all the years from this site https://www.tcdb.com/ViewAll.cfm/sp/Football?MODE=Years,

Loop through these years and start from https://www.tcdb.com/ViewAll.cfm/sp/Football/year/2024, crawl this page,

The program traverses all years, a json file for each year, store the set list for that year in the json file.

This list of json files will be used as a configuration file later in the official crawler

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
__pycache__		__pycache__
alembic		alembic
ebay		ebay
patch		patch
.dockerignore		.dockerignore
.gitignore		.gitignore
README.md		README.md
alembic.ini		alembic.ini
constant.py		constant.py
deal_s3.py		deal_s3.py
deal_s3_v2.py		deal_s3_v2.py
headers.py		headers.py
http_utils.py		http_utils.py
image_downloader.py		image_downloader.py
log.py		log.py
main_do_err.py		main_do_err.py
main_v2.py		main_v2.py
main_v3.py		main_v3.py
migration.py		migration.py
model_v2.py		model_v2.py
requirements.txt		requirements.txt
save_db.py		save_db.py
set_meta.py		set_meta.py
update_and_insert_meta.py		update_and_insert_meta.py
year_meta.py		year_meta.py