POC with Playwright (headless) to capture product list

[daniloab]

Create a scraping module to fetch a category in the target e-commerce, paginate, and return {id|sku, title, price, url, imageUrl}.

Plan

Before starting to code, write down an implementation plan, for example, what you will use, which libs, and how to connect with the NextJS backend.

If needed, the scraper can be a backend nodejs application. To do that, issue #2 must be implemented, or you should implement it before moving on to a backend solution.

Once you have your plan, comment here in this issue and share it with the group to discuss and decide if we should move forward.

[Felsilva15]

Build a POC scraper that fetches products from a category page in the target e-commerce site, paginates up to a small number of pages, and returns normalized data:

{ id|sku, title, price, url, imageUrl }

Tech Stack:

• Playwright (Chromium, headless) for navigation and scraping
• Next.js API Route (Option A) or standalone Node service (Option B)
• Optional: zod (validation), dotenv (config), p-limit (throttling)

Architecture Options:

• Option A: Inside Next.js: Fastest POC, add API route that calls scraper module.
• Option B: Separate Node Service: Cleaner isolation, but requires additional infra (issue Implement monorepo architecture #2 prerequisite).

Data Flow:

• Request hits /api/scrape/category with { path, maxPages }.
• API calls Playwright scraper.
• Scraper navigates category, extracts product cards, follows pagination.
• Returns JSON { items, count } to frontend or consumer.

Stability & Compliance:

• Limit default pages to 3 and add polite delays (1–2s).
• Set user agent, locale, timezone for realism.
• Use robust selectors and fallbacks.
• Respect site ToS / robots.txt; keep volume low.

Next Steps for Team Decision:

• Confirm Option A (Next.js route) vs Option B (separate service).
• Approve default maxPages and delay values.
• Validate target site selectors.
• Decide if persistence (e.g. SQLite/Prisma) is required in POC or just return JSON.

[daniloab]

What would be the pros and cons of Option A and Option B?

[Felsilva15]

@daniloab

Option A – Inside Next.js:

• Pros: Quickest to build, single repo/deploy, easy to call from the app.
• Cons: Limited by serverless/runtime constraints, heavier deploy, harder to scale or add scraping-specific features.
• Good for: POC, low-volume scraping.

Option B – Separate Node Service:

• Pros: Isolated scaling, easier to add retries/proxies, cleaner logging, future-proof.
• Cons: Requires extra infra + coordination, depends on monorepo/infra setup (issue Implement monorepo architecture #2).
• Good for: Higher volume, long-term scraping.

[daniloab]

@daniloab

Option A – Inside Next.js:

• Pros: Quickest to build, single repo/deploy, easy to call from the app.
• Cons: Limited by serverless/runtime constraints, heavier deploy, harder to scale or add scraping-specific features.
• Good for: POC, low-volume scraping.

Option B – Separate Node Service:

• Pros: Isolated scaling, easier to add retries/proxies, cleaner logging, future-proof.
• Cons: Requires extra infra + coordination, depends on monorepo/infra setup (issue Implement monorepo architecture #2).
• Good for: Higher volume, long-term scraping.

I prefer option 2. And you?