creating api key on data-gov-in
- register and login
- go to my account section(logged In) and
- just click on create API key; there you have it
- this key can be used for any datasset
- to see how to use it look at
important open data websites
- India
- USA
- All Open Data Sites or this
- Visit any site using above link.
avoid headers problem
- some websites like
amazonuse systems that can detect bot/automated(basically not through a web-browser, rather done programmatically) requests, - to avoid them we can use headers parameter in out get request
- to get your user-agent visit
- or to learn more about headers go to requests docs and also look into network-inspection tab to know more about them for a particular website
- we also can use slenium/helium automation but that is resource heavy even with headless running
- Notes:
- we also can use other user agents like for other web-browsers(safari,firefox etc) and try what sort of info we get
- you also can trying to rotate through different user-agents when facing problem your usual user-agent; might help when there is some IP blocks etc