PINGDOM_CHECK

Web Scraping Copilot is live. Build Scrapy spiders 3× faster, free in VS Code.

Install Now
  • Data Services
  • Pricing
  • Login    Sign up👋 Contact Sales

Zyte Developers

Coding tools & hacks straight to your inbox

Become part of the community and receive a bi-weekly dosage of all things code.

Join us
    • Zyte Data
    • News & Articles
    • Search
    • Social Media
    • Product
    • Data for AI
    • Job Posting
    • Real Estate
    • Zyte API - Ban Handling
    • Zyte API - Headless Browser
    • Zyte API - AI Extraction
    • Web Scraping Copilot
    • Zyte API Enterprise
    • Scrapy Cloud
    • Solution Overview
    • Blog
    • Webinars
    • Case Studies
    • White Papers
    • Documentation
    • Web Scraping Maturity Self-Assesment
    • Web Data compliance
    • Meet Zyte
    • Jobs
    • Terms and Policies
    • Trust Center
    • Support
    • Contact us
    • Pricing
    • Do not sell
    • Cookie settings
    • Sign up
    • Talk to us
    • Cost estimator

The Scrapy tutorial part VI: Scraping Infinite Scroll Pages With Python

Developed by Pablo Hoffman and Shane Evans, Scrapy is an open-source python framework built specifically for web data extraction. With Scrapy spiders, you are able to download HTML, parse and process the data and save it in either CSV, JSON, or XML file formats.

This video shows how to find and use underlying APIs that power AJAX-based infinite scrolling mechanisms in web pages.

After watching this video, you will know:

  • How to inspect the network requests from your browser
  • How to reverse engineer network requests
  • How to extract data from a JSON-based HTTP API

If you haven't yet, we recommend you to first watch part I, part II, part III, part IV, and part V of our tutorial series.

If you like what you saw, we can recommend you to also check out the links below:

  • The Scrapy tutorial course part VII: How to submit forms in your spiders
  • The Scrapy tutorial course part VIII: How to scrape JS pages with Splash
  • Chrome DevTools - Networking and the Console
  • Web Scraping - Discovering Hidden APIs

G2.com

Capterra.com

Proxyway.com

EWDCI logoMost loved workplace certificateZyte rewardISO 27001 iconG2 rewardG2 rewardG2 reward
© Zyte Group Limited 2026