Rendering Javascript pages without giving up Scrapy

A page can contain more JavaScript than HTML and still not need a browser. Often the data is already in the response, in a script tag, or behind a JSON endpoint the application calls itself. Reaching for browser automation before checking those places is an expensive habit.

The modern approach is to treat JavaScript rendering as an acquisition choice. Keep Scrapy for scheduling, retries, item processing, pipelines, logging, and monitoring. Use an ordinary Scrapy request when it is enough, call an authorized data endpoint when that is the real source, and render in a browser only when you genuinely need the browser's DOM or state.

A JavaScript application is not the same thing as a browser-only data source.

That distinction is the difference between a crawler that scales predictably and one that launches a browser for every pagination link.

Find the data source before choosing a renderer

Start by looking at the response Scrapy receives. The Scrapy guide to dynamically loaded content suggests saving a fetched response locally, then using browser developer tools to inspect the requests that actually supply the data.

1scrapy fetch --nolog https://example.com/products > response.html

The following decision table is deliberately boring. That is a strength. It prevents a rendering problem from becoming the default architecture for the whole project.

The second and third rows are easy to miss when looking at a polished application UI. They are often the better solution.

Call the API with Scrapy when the API is the source of truth

Suppose the product page fetches a JSON document from an endpoint your project is permitted to use. Make that request with Scrapy, not with an ad hoc requests loop inside a callback. Scrapy can schedule it, retry it, log it, and feed the resulting items into the same pipeline and Spidermon checks from the earlier parts of this series.

JsonRequest serializes a JSON request body and sets the appropriate content type. The Scrapy request and response documentation also documents response.json(), which decodes the response body for you.

1# catalog/spiders/products_api.py
2import scrapy
3from scrapy.http import JsonRequest
4
5from catalog.items import Product
6
7
8class ProductsApiSpider(scrapy.Spider):
9    name = "products_api"
10    api_url = "https://api.example.com/v1/products/search"
11
12    async def start(self):
13        yield JsonRequest(
14            self.api_url,
15            data={"category": "widgets", "page": 1},
16            callback=self.parse_products,
17        )
18
19    def parse_products(self, response):
20        payload = response.json()
21
22        for product in payload["products"]:
23            yield Product(
24                name=product["name"],
25                price=product["price"],
26                description=product.get("description", ""),
27                url=response.urljoin(product["url"]),
28            )

Real APIs have their own pagination, rate limits, and terms. Treat those as part of the spider's contract. Do not treat a network-panel request as permission to use an endpoint.

This route also fits neatly with Part 2. The Product item stays the stable output contract even when the input changes from an HTML Page Object to an API response. That makes it possible to replace an acquisition strategy without rewriting exports, validation, or downstream consumers.

Add Playwright only for the requests that need a browser

When the page needs client-side rendering, scrapy-playwright is the useful integration point. It is a Scrapy download handler backed by Playwright, so rendered requests still go through Scrapy's regular scheduling and item-processing workflow.

Install the Python package and the browser binaries:

1pip install scrapy-playwright
2playwright install chromium

Then configure the handler and asyncio-compatible Twisted reactor. Recent Scrapy projects already use this reactor by default, but keeping it explicit removes an easy source of confusion when updating an older project.

1# catalog/settings.py
2DOWNLOAD_HANDLERS = {
3    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
4}
5
6TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

That setting does not mean every HTTPS request launches Chromium. A request uses Playwright only when it carries meta={"playwright": True}. Keep listing pages, robots-aware discovery, and API calls on Scrapy's normal downloader unless rendering is specifically necessary.

Make browser rendering the project default only when it really is the default

scrapy-playwright does not provide a setting that turns on Playwright for every request. Its download handler deliberately falls back to Scrapy's normal HTTP handler unless a request has the playwright meta flag.

If a narrowly scoped project truly needs browser rendering for every request, set that flag in a downloader middleware. Downloader middlewares run before Scrapy selects the download handler.

1# catalog/middlewares.py
2class EnablePlaywrightMiddleware:
3    def process_request(self, request, spider):
4        request.meta.setdefault("playwright", True)

1# catalog/settings.py
2DOWNLOADER_MIDDLEWARES = {
3    "catalog.middlewares.EnablePlaywrightMiddleware": 543,
4}

This preserves a request that has explicitly set playwright to False, which makes an opt-out possible for a public API or static asset endpoint. In most mixed crawls, the opposite policy is easier to operate: leave the default as False and mark only the rendered paths. The scrapy-playwright activation documentation explains why registering its handler alone does not enable rendering for every request.

Wait for a meaningful element, not an arbitrary number of seconds

The common first browser spider waits five seconds and hopes. The reliable version waits for an observable condition that says the field you need has appeared.

PageMethod lets the download handler perform an action before it returns the final rendered response. The callback then receives a normal Scrapy response whose selectors see the browser-rendered DOM.

1# catalog/spiders/rendered_products.py
2import scrapy
3from scrapy_playwright.page import PageMethod
4
5from catalog.items import Product
6
7
8class RenderedProductsSpider(scrapy.Spider):
9    name = "rendered_products"
10    start_urls = ["https://example.com/widgets"]
11
12    def parse(self, response):
13        for href in response.css("a.product::attr(href)").getall():
14            yield response.follow(
15                href,
16                callback=self.parse_product,
17                meta={
18                    "playwright": True,
19                    "playwright_page_methods": [
20                        PageMethod("wait_for_selector", ".product-price"),
21                    ],
22                },
23            )
24
25    def parse_product(self, response):
26        yield Product(
27            name=response.css("h1::text").get(default="").strip(),
28            price=response.css(".product-price::text").get(default="").strip(),
29            description=" ".join(
30                response.css(".description *::text").getall()
31            ).strip(),
32            url=response.url,
33        )

The example keeps the extraction short to show the boundary. In the project from Part 2, move those selectors into the appropriate scrapy-poet Page Object once the rendering path is understood. Browser acquisition should not become an excuse to put all extraction back in callbacks.

Avoid time.sleep() and fixed browser delays. They are slow when the page is fast and unreliable when the page is slow. A selector, response condition, or explicit page action describes what the spider needs rather than guessing how long the target will take.

Treat browser pages and contexts as limited resources

A browser context holds state such as cookies and local storage. A page consumes memory and a concurrency slot. Neither should be created casually.

Use a named context when public browsing needs a coherent state, such as a chosen locale, and cap the amount of browser work your crawler can do at once:

1# catalog/settings.py
2PLAYWRIGHT_CONTEXTS = {
3    "catalog": {
4        "locale": "en-GB",
5    },
6}
7
8PLAYWRIGHT_MAX_CONTEXTS = 4
9PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 2
10PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 20 * 1000

1# catalog/spiders/localized_products.py
2import scrapy
3
4
5class LocalizedProductsSpider(scrapy.Spider):
6    name = "localized_products"
7
8    async def start(self):
9        yield scrapy.Request(
10            "https://example.com/widgets",
11            callback=self.parse_product,
12            meta={
13                "playwright": True,
14                "playwright_context": "catalog",
15            },
16        )
17
18    def parse_product(self, response):
19        return {"url": response.url}

Only set playwright_include_page when the callback must directly use the Playwright Page, for example for a multi-step interaction that cannot be expressed as PageMethod calls. Otherwise let scrapy-playwright close it after producing the response. Leaving pages open is a quiet way to stall a crawl.

Connect to a managed browser when local Chromium is not the right runtime

Local launch is the simplest starting point. It is not the only option. scrapy-playwright can connect to a browser that another system owns, which is useful when browser lifecycle, isolation, or capacity is managed outside the spider process.

Here is a CDP connection to a remote Chromium instance:

1# catalog/settings.py
2PLAYWRIGHT_CDP_URL = "http://browser-worker.internal:9222"
3PLAYWRIGHT_CDP_KWARGS = {
4    "timeout": 20 * 1000,
5}

Do not configure PLAYWRIGHT_CDP_URL and PLAYWRIGHT_CONNECT_URL together. They are alternative connection models, and both bypass PLAYWRIGHT_LAUNCH_OPTIONS. The scrapy-playwright connection settings explain the lifecycle and compatibility implications in detail.

The operational question is more important than the setting name: who restarts a disconnected browser, who limits capacity, and how are browser logs correlated with the Scrapy job? Answer those before moving a spider to remote browsers.

Use browser-provider integrations for more options

Current scrapy-playwright releases expose PLAYWRIGHT_BROWSER_PROVIDER, an extension point for a provider that owns browser startup, connection, and teardown. The default provider supports local launch, CDP, WebSocket connections, and persistent contexts. The project also documents examples for Playwright-compatible alternatives such as Patchright and Camoufox.

These are sometimes described as stealth-browser options. That phrase promises more than the setting does. They are browser-provider integrations, not a guarantee that a target will accept a crawl or that its access rules can be ignored.

Use a provider only when you can name the concrete browser behavior or runtime constraint it addresses, and test it as a separate acquisition implementation. Keep the same Items, Page Objects, logs, and monitors around it. Respect the site's terms, permissions, and applicable requirements regardless of which browser implementation is underneath.

Monitor rendered spiders as a separate operating mode

Part 3 used Spidermon to monitor output quality. Browser-backed spiders need those checks, plus a few browser-specific signals:

• navigation timeouts and Playwright request failures;
• an unexpected rise in error pages or interstitial pages;
• a drop in rendered item count compared with the expected scope; and
• pages or contexts left open long enough to reduce crawl throughput.

A rendered response can be syntactically valid HTML and still be the wrong page. Validate the resulting item, not just the HTTP status.

Start with an item-count monitor and required-field validation. Then add a periodic monitor when a long-running browser crawl can waste meaningful capacity after a failure. The Spidermon periodic-monitoring documentation covers that pattern.

Browser automation also changes the economics of a spider. It is worth keeping a normal Scrapy path for the pages that do not need rendering, then using monitoring to prove that the smaller browser surface still produces the expected data.

Recent work on using Web Scraping Copilot in VS Code can help with the initial mechanics of a Scrapy project. It does not replace the technical judgment in the table above. The same caution appears in a review of AI-generated Scrapy projects: a spider that runs is not necessarily one with a sound acquisition strategy. For teams that need a managed alternative to operating browser infrastructure, Zyte API's headless browser capability is another option to evaluate against your requirements.

Questions developers ask about Scrapy and JavaScript rendering

Do I need Playwright because a site uses JavaScript?

No. Inspect the Scrapy response and network requests first. If the data is in HTML, serialized state, or an authorized JSON response, use Scrapy without a browser. Use Playwright when the rendered DOM, client-side interaction, or browser state is genuinely required.

Can one spider mix ordinary and rendered requests?

Yes. That is usually the point. Register the download handler once, then set meta={"playwright": True} only on the individual requests that need it. The rest continue through Scrapy's standard downloader.

When should I use CDP instead of PLAYWRIGHT_CONNECT_URL?

Use CDP when the remote browser is Chromium and exposes a CDP endpoint. Use the Playwright connection URL for a Playwright server endpoint. Pick one connection model, document who owns browser lifecycle, and test reconnect behavior before relying on it in scheduled crawls.

Should I enable a stealth-oriented browser provider by default?

No. Start with the default provider and add another only for a defined, permitted requirement. A provider increases the number of moving parts, so it should earn its place through a concrete operational need, not a vague expectation that every modern site needs it.

Keep the browser at the edge of the architecture

The series began with Scrapy's core components, moved extraction into Page Objects, and added monitoring around the data contract. Browser rendering belongs at the edge of that architecture: it changes how a selected response is acquired, not how the rest of the project is designed.

For your next JavaScript-heavy target, inspect one product page and one listing page. Write down where each field comes from. Then choose the least expensive permitted acquisition method that gives you that source, and make the resulting Items pass the same checks as every other spider.