Skip to content

Cardkingdom import #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 59 commits into
base: master
Choose a base branch
from
Open

Cardkingdom import #5

wants to merge 59 commits into from

Conversation

@srhbutts
Copy link
Contributor

@srhbutts srhbutts commented Jun 1, 2019

Submitted initial version for review / suggestions?

@krainboltgreene
Copy link
Member

krainboltgreene commented Jun 1, 2019

This should probably be two services, one for buy list and one retail?

srhbutts and others added 23 commits June 1, 2019 04:15
@srhbutts
fixed missing end
9782c10
@srhbutts
started retail parsing function, will split to separate services later
5daf05b
@srhbutts
Merge branch 'master' of https://github.com/difference-engineers/fars…
1024327 
…ight-functions into cardkingdom_import
@srhbutts
added pricing parsing
01b5d39
@srhbutts
added function to return an array of all relevant pages
db45f2b
@srhbutts
next_page function no longer needed
7678950
@srhbutts
added skeleton of function for update buylist/retail
2682f9c
@srhbutts
added processing for separate foil pages
6ddc2cb
@srhbutts
corrected string pushed to every_page url
63b9677
@srhbutts
Merge branch 'master' of https://github.com/difference-engineers/fars…
22e88c0 
…ight-functions into cardkingdom_import
@srhbutts
no need for a default/testing url
e053177
@srhbutts
fixed redundant/unnecessary categories definition
349c8fc
@srhbutts
fixing whitespace
3ac0e2c
@srhbutts
started separate functionn for buylist vs retail url array
2b9c668
@srhbutts
need the url to be customized based on our array of categories
ec02028
@krainboltgreene
Merge branch 'master' of github.com:difference-engineers/farsight-fun…
7e444af 
…ctions into pr-5/difference-engineers/cardkingdom_import
@krainboltgreene
These are now handled automatically
3d0a699
@krainboltgreene
Updating from latest designs
07bb15a
@srhbutts
Merge branch 'master' of https://github.com/difference-engineers/fars…
af063c6 
…ight-functions into cardkingdom_import
@srhbutts
Merge branch 'master' of https://github.com/difference-engineers/fars…
46f8396 
…ight-functions into cardkingdom_import
@srhbutts
Merge branch 'master' of https://github.com/difference-engineers/fars…
5a97cb1 
…ight-functions into cardkingdom_import
@srhbutts
Merge branch 'cardkingdom_import' of https://github.com/difference-en…
7690ba4 
…gineers/farsight-functions into cardkingdom_import
@srhbutts
Let god sort them out
e9902e2
srhbutts added 17 commits July 18, 2019 22:23
@srhbutts
rewrote categories function to avoid syntax errors
9570bb7
@srhbutts
fixed missing end of block
7693b60
@srhbutts
cleaning up whitespace
06c436e
@srhbutts
fixed whitespace
14afa93
@srhbutts
ensure categories returns the sets array
25c71e0
@srhbutts
fixed syntax error
280264f
@srhbutts
fixed syntax error
391d11b
@srhbutts
changed categories to return set_ids as that's all we actually need
d45e1a2
@srhbutts
changed categories to return set_ids as that's all we actually need
5dd6e2c
@srhbutts
Merge branch 'cardkingdom_import' of https://github.com/difference-en…
8d114ff 
…gineers/farsight-functions into cardkingdom_import
@srhbutts
category_urls needed to replace a substring slice to make the array
e2e2ea8
@srhbutts
no more need for sample url
0686ef5
@srhbutts
update_buylist now has a sample url it needed to start
09dca04
@srhbutts
updated the retail function to use the new category_urls function
a993290
@srhbutts
make sure we return the array
1341bb4
@srhbutts
have to pass every_page a starter url
ac87ed0
@srhbutts
use the specific retail/buylist functions instead of "cards"
6807368
@srhbutts
Copy link
Contributor Author

srhbutts commented Jul 28, 2019 •
edited
Loading

Requesting a review / more comments

Things I still have to add:

  • Some sort of output/logging to make sure it's actually processing
  • Still need to handle CardKingdom's goofy formatting for special cards (buy a box promo, judge promo, etc)
  • Split into retail/buylist services (if that's still suggested-- they'll both be using a lot of the same code)
  • Find a way to update certain categories more frequently than others; for example, 4th Edition buylist/retail prices are less likely to change than those from the latest set.
  • Make sure we time out requests to not hammer their servers / get banned...
  • Try to make our requests look more human?
  • Actually insert stuff into the DB

krainboltgreene
krainboltgreene previously requested changes Jul 28, 2019
View reviewed changes
@srhbutts srhbutts dismissed krainboltgreene’s stale review August 1, 2019 02:26

(already added this after you suggested the change)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@krainboltgreene krainboltgreene krainboltgreene left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@srhbutts @krainboltgreene