This IP has been automatically blocked

[bernardo-borges]

I'm getting this error message ,from every record I pull

[Dellybro]

Samee...

[struong]

Any luck on resolving this?

[bernardo-borges]

I used puppeteer to bypass this problem , using the listing to get the url and then using the puppeteer to access the html and get the values.
But consumes more resources and the search is slower.

[brozeph]

@Dellybro, @bevensteven and @kishido - I spoke with a colleague who has a great deal of experience with craigslist scraping and he mentioned that it might be worth trying to load an image from the posting. Using puppeteer would actually accomplish this as a side effect of how it works - apparently the request to the CDN is somehow used to throttle.

[jtara1]

Requesting an image before using call the list() method from the node-craiglist api isn't working for me

const { Request } = require('reqlib');

(async () => {
  try {
    await new Request().get('https://images.craigslist.org/00707_lgLO2XI9fW5z_0CI0t2_300x300.jpg');
  } catch (e) {
    console.error(e);
  }
})();

I did get a response from the site just using puppeteer alone though, and their site list api is simple enough to not need http wrapper api like this tbh.