What browser automation setup stays reliable when sites become harder to automate?
Asked by
2kb_chunk
My local Chrome fleet worked well until several target sites began applying stricter detection and additional checks. The same Playwright scripts now lose about half of their parallel workers, with browsers detaching mid-run and logs providing little useful information. Pinning browser versions and slowing the pacing helped only slightly, and failures become especially common when three or more jobs run at once. What tools or architecture have people found reliable under load? I'm mainly looking for boring, observable runs rather than ways to bypass access controls.
Answers
Before switching tools, measure whether this is actually detection or ordinary resource exhaustion. Track browser exits, memory, CPU, page-load times, and per-worker completion rates. A managed Playwright service such as Browserless or Browserbase can help with concurrency controls, session inspection, and better debugging, but compare it against your real workloads rather than assuming the hosted option fixes everything.
Reply to this answer
Hosted browsers can be useful, but the biggest improvement is often capacity planning: fewer sessions per machine, explicit concurrency limits, timeouts, retries, and structured logs for every browser and page event. If youโre using containers, give each worker predictable CPU and memory, keep browser counts low per node, and preserve logs before a pod or process is restarted.
pixel.by.pixel: We saw the same thing with containerized Chrome. Two or three sessions were fine, then memory pressure caused restarts that looked like random automation failures.
Reply to this answer
Throttling can reduce instability, but headed mode or slower pacing usually isnโt a complete solution when a site changes its checks during a run. Iโd focus on graceful recovery: detect detached pages, restart only the affected worker, record the URL and browser state, and stop retrying when the failure looks like an access-policy block. That keeps the system reliable without turning every failure into an endless timeout.
Reply to this answer
Add your answer
Answers are reviewed before publication.
+prefer the aisle: That makes sense. The failure threshold around three workers could easily be memory or process limits, so Iโm going to capture those metrics before changing the whole stack.