How to Scrape Multiple Pages With a Web Scraping API (Step-by-Step Guide)

You've built a scraper that pulls data from a single page — product listings, job postings, research results — and it works. Then you realize the site has 47 pages of results and your scraper only got page one. Handling pagination is the step that turns a proof of concept into a complete data extraction pipeline.

Scraping multiple pages with a web scraping API means automatically navigating through a site's paginated content — following page numbers, offset parameters, next-page links, or dynamically loaded batches — and collecting data from each page into a single unified dataset. The approach differs based on how a site implements pagination: URL-based parameters, cursor tokens, or JavaScript-driven infinite scroll each require different handling. This guide covers every major pagination pattern with working Python code, so you can handle whatever structure your target site uses.

What Is Multi-Page Scraping?

Multi-page scraping is the automated collection of data across multiple pages of a website — following the site's pagination mechanism to retrieve records that are distributed across many pages rather than served all at once.

Most sites that display lists of items — products, jobs, articles, listings — paginate their results: rather than loading all 10,000 records in a single page response, they load 20 or 50 at a time and provide a way to navigate to the next batch. A scraper that only requests the first page collects only the first batch, missing everything else. Multi-page scraping automates the navigation through all subsequent pages until the full dataset has been collected.

The challenge is that pagination isn't standardized — different sites implement it in meaningfully different ways, and the code that handles one implementation often fails on another. URL-based page parameters, offset and limit query strings, "next page" links embedded in HTML, API cursor tokens, and JavaScript-driven infinite scroll all require different handling in your scraper.

How Pagination Works Across Different Sites

Understanding how a site implements pagination tells you exactly which pattern your scraper needs to handle:

URL-based page numbers are the most common pattern. The page number appears directly in the URL as a query parameter (?page=2) or a path segment (/listings/page/2/). Each page number maps to a distinct URL, making it straightforward to construct and request each page's URL in sequence.

Offset and limit parameters are common on sites with API-style URL structures. Rather than a page number, the URL includes ?offset=20&limit=20 (meaning "skip the first 20, return the next 20"). To navigate to the third page, you'd use ?offset=40&limit=20. Common in e-commerce and database-backed listing sites.

Next-page links embedded in HTML appear when the site's pagination renders as a "Next" button or link with an href pointing to the next page URL. Rather than constructing the next URL yourself, you extract it from the current page's HTML and follow it.

API cursor tokens appear in JSON APIs where the response includes a cursor, next_token, or after field that you pass with your next request to retrieve the following batch. Pagination that can't be reduced to page numbers — because the underlying data can change between requests — often uses cursors.

JavaScript-driven infinite scroll and load-more buttons load additional content dynamically in response to user interaction (scrolling or clicking), without changing the URL. This requires a browser automation tool rather than a plain HTTP client.

Step-by-Step Guide: Handling Each Pagination Pattern

Step 1: Identify the Pagination Pattern

Before writing any scraping code, determine which pattern the target site uses. Open the browser developer tools and watch the URL bar and Network tab as you navigate to the next page:

URL changes to ?page=2: URL-based pagination
URL changes to ?offset=20: offset-based
URL stays the same but Network tab shows a new XHR request: JavaScript/API pagination
A "Load More" button exists: button-triggered dynamic loading

You can also inspect the current page's HTML source for links containing page=2, next, or similar.

Step 2: Handle URL-Based Page Number Pagination

The simplest pattern — construct each page URL and collect data from all pages:

import requests
from bs4 import BeautifulSoup
import time
import random

BASE_URL = "https://example-site.com/listings"
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/124.0.0.0 Safari/537.36"
    )
}

def scrape_all_pages_by_number(base_url: str,
                                 page_param: str = "page",
                                 max_pages: int = 50) -> list[dict]:
    """Scrape all pages of a site using URL page number parameters."""
    all_records = []

    for page_num in range(1, max_pages + 1):
        url = f"{base_url}?{page_param}={page_num}"
        response = requests.get(url, headers=HEADERS, timeout=15)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        records = extract_records(soup)  # Your extraction function

        if not records:
            print(f"No records on page {page_num} — reached end of results.")
            break  # Stop when a page returns no data

        all_records.extend(records)
        print(f"Page {page_num}: {len(records)} records collected.")
        time.sleep(random.uniform(1.5, 3.0))  # Polite delay between pages

    return all_records

The empty-results check (if not records: break) is how you detect the last page without knowing in advance how many pages exist — it's more reliable than assuming the page count.

Step 3: Handle Offset-Based Pagination

Same principle but computing the offset instead of incrementing a page number:

def scrape_all_pages_by_offset(base_url: str,
                                 limit: int = 20,
                                 max_records: int = 5000) -> list[dict]:
    """Scrape all records using offset+limit pagination."""
    all_records = []
    offset = 0

    while offset < max_records:
        url = f"{base_url}?offset={offset}&limit={limit}"
        response = requests.get(url, headers=HEADERS, timeout=15)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        records = extract_records(soup)

        if not records:
            break  # No more records — collection complete

        all_records.extend(records)
        print(f"Offset {offset}: {len(records)} records collected.")

        if len(records) < limit:
            break  # Received fewer records than requested — last page

        offset += limit
        time.sleep(random.uniform(1.0, 2.5))

    return all_records

The len(records) < limit check catches a common edge case: the last page often returns fewer records than the requested limit, which is a cleaner signal than getting zero records.

Step 4: Follow Next-Page Links From HTML

For sites where the next page URL isn't constructable from a pattern, extract it from the current page:

def scrape_following_next_links(start_url: str) -> list[dict]:
    """Follow next-page links from each page until no next link exists."""
    all_records = []
    current_url = start_url

    while current_url:
        response = requests.get(current_url, headers=HEADERS, timeout=15)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        records = extract_records(soup)
        all_records.extend(records)

        # Find the "Next" page link — adjust selector to match target site
        next_link = soup.select_one('a[rel="next"], a.pagination-next, a:contains("Next")')
        if next_link and next_link.get("href"):
            href = next_link["href"]
            # Handle relative URLs
            if href.startswith("http"):
                current_url = href
            else:
                from urllib.parse import urljoin
                current_url = urljoin(current_url, href)
        else:
            current_url = None  # No next link found — last page reached

        print(f"Collected {len(records)} records. Next URL: {current_url}")
        time.sleep(random.uniform(1.5, 3.0))

    return all_records

Step 5: Handle JavaScript-Based Pagination With Playwright

For infinite scroll, load-more buttons, or any pagination that requires browser interaction, Python's HTTP client can't help — you need a browser automation tool:

from playwright.sync_api import sync_playwright
import time

def scrape_with_load_more_button(url: str,
                                   load_more_selector: str = "button.load-more",
                                   max_clicks: int = 20) -> str:
    """
    Interact with a 'Load More' button to reveal all paginated content,
    then return the fully loaded page HTML for extraction.
    """
    with sync_playwright() as pw:
        browser = pw.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle", timeout=30_000)

        for click_count in range(max_clicks):
            load_more = page.query_selector(load_more_selector)
            if not load_more or not load_more.is_visible():
                print(f"No more 'Load More' button after {click_count} clicks.")
                break

            load_more.click()
            # Wait for new content to load after clicking
            page.wait_for_load_state("networkidle", timeout=10_000)
            time.sleep(1.0)  # Additional pause for content rendering

        full_html = page.content()
        browser.close()

    return full_html

def scrape_infinite_scroll(url: str, scroll_count: int = 15) -> str:
    """
    Scroll down repeatedly to trigger infinite scroll content loading.
    Returns the fully loaded HTML after all scroll events.
    """
    with sync_playwright() as pw:
        browser = pw.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle", timeout=30_000)

        previous_height = 0
        for _ in range(scroll_count):
            page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            page.wait_for_timeout(2000)  # Wait for new content to load

            current_height = page.evaluate("document.body.scrollHeight")
            if current_height == previous_height:
                print("Page height stopped growing — all content loaded.")
                break
            previous_height = current_height

        html = page.content()
        browser.close()
    return html

For JavaScript-rendered pagination on bot-protected sites, a managed scraping API that handles both rendering and anti-bot bypass is more practical than self-managed Playwright. MrScraper's Scraping Browser handles JavaScript execution, including scroll-triggered content loading and button interaction, through its API — making JavaScript pagination collection available without managing browser processes. Documentation at https://docs.mrscraper.com.

Step 6: Combine and Export the Full Dataset

After collecting records from all pages, combine and export:

import csv
from datetime import datetime

def export_all_pages(records: list[dict],
                      filename: str | None = None) -> str:
    """Export the combined multi-page dataset to CSV."""
    if not records:
        print("No records to export.")
        return ""

    filename = filename or f"scraped_data_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv"
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=records[0].keys())
        writer.writeheader()
        writer.writerows(records)

    print(f"Exported {len(records)} total records to {filename}")
    return filename

Best Tools for Multi-Page Scraping

Python + requests + BeautifulSoup — the standard stack for URL-based and offset-based pagination on static HTML sites. Zero cost, maximum flexibility, handles all non-JavaScript pagination patterns well. Use this for any target that doesn't require browser rendering.

Playwright (Python) — handles JavaScript-based pagination, infinite scroll, and load-more button patterns that HTTP clients can't reach. Required for any target where pagination requires user interaction or dynamic content loading.

Scrapy — Python's purpose-built scraping framework with built-in support for following links and paginating through sites. The CrawlSpider with Rule objects handles many common pagination patterns without custom loop logic. Documentation at https://docs.scrapy.org. Best for large-scale multi-page crawling operations where framework-level pipeline management is valuable.

MrScraper — for multi-page scraping against bot-protected or JavaScript-rendered targets where managing browser processes and anti-bot bypass independently adds significant complexity. MrScraper's API handles rendering and protection bypass per request; you implement the pagination loop in your code and call MrScraper for each page's content, rather than running Playwright yourself.

Free vs. Paid: What Each Approach Supports

Free Python tooling (requests, BeautifulSoup, Playwright) handles all pagination patterns at no tool licensing cost. Infrastructure costs — a server to run your scraper — are real but modest for most scraping operations. This is the right starting point for developers comfortable with Python.

Managed scraping APIs (like MrScraper) add per-page cost but handle rendering and anti-bot bypass per page request — valuable when your pagination requires navigating through bot-protected pages where a plain Python scraper would be blocked.

Scrapy Cloud and similar platforms provide scheduling and infrastructure for Scrapy spiders, adding managed hosting at a subscription cost. Relevant for production-grade, continuously running multi-page scrapers that need uptime guarantees without managing your own servers.

Key Features to Look For in a Multi-Page Scraping Tool

Pagination pattern support: Does the tool handle URL-based, offset-based, next-link, cursor-based, and JavaScript-driven pagination, or only a subset?
JavaScript rendering: For dynamically loaded pagination (infinite scroll, load-more), a browser execution layer is required — confirm whether the tool provides this.
Request pacing and rate control: Multi-page scraping produces many requests to the same domain — configurable delays between page requests are essential for polite, sustainable access.
Failure handling and resumability: Long multi-page runs encounter occasional failures. Can the scraper resume from where it left off, or does it restart from page one?
Duplicate detection: If the target site changes the order of results between page requests (due to real-time inventory updates, for example), deduplication by a unique record identifier prevents duplicates in your output.
Structured output directly from multi-page collection: Does the tool aggregate records from all pages into one output file, or does it require post-collection merging?

When Should You Build a Multi-Page Scraping Pipeline?

Build a multi-page pipeline when:

The dataset you need spans multiple pages and you need complete coverage — a partial first-page extraction is insufficient for your use case
The target site has stable pagination that won't require frequent structural updates to your scraper
Your analysis requires data from many pages simultaneously rather than a sample from the first page

A single-page approach is sufficient when:

You only need the most recent or featured items that appear on page one
You're testing or exploring the data before committing to full extraction
The site publishes a downloadable export or provides an API that delivers the full dataset more efficiently than pagination scraping

Common Challenges and Limitations

Detecting the last page requires a stopping condition. Not all sites signal the last page clearly. The most reliable stopping condition is: stop when a page returns zero records. A secondary check — stop when you receive fewer records than the expected page size — catches the last partial page. Without either check, your scraper continues requesting non-existent pages indefinitely, generating 404 errors or empty pages until it hits a maximum you've configured.

Pagination structures change without notice. A site that uses ?page=2 today may switch to ?p=2 or a cursor-based approach after a redesign. Multi-page scrapers that rely on URL patterns need monitoring — a sudden drop to zero records per page is often the signal that the pagination structure changed, not that the site ran out of data.

Bot detection intensifies on deep pagination. The deeper into a site's pagination you scrape, the more requests you've made to the same domain in the same session — which accumulates detection signals faster than single-page access. Rate limiting, delays between pages, and proxy rotation (fresh IPs across the pagination sequence) reduce this accumulation and extend how far you can scrape before triggering detection.

JavaScript infinite scroll requires a different measurement of "completeness." Unlike URL-based pagination where you know you've reached the last page when no next link exists, infinite scroll doesn't have a natural termination signal. The scroll height comparison in the code above (stop when page height stops growing) is the standard approach, but fast-loading pages may need longer wait times between scrolls to accurately detect content completion.

Conclusion

Multi-page scraping is the step that makes web scraping practically useful at scale — a first-page collection is rarely complete enough for real analysis, and automating pagination is what transforms a single-page extraction into a full dataset. The pattern that's right for your target depends on how its pagination is implemented: URL parameters, offset arithmetic, HTML next-page links, API cursors, or JavaScript interaction each have a corresponding Python approach that handles them cleanly.

Start by identifying your target's pagination pattern using browser DevTools, match it to the corresponding code pattern in this guide, and add empty-results detection as your termination condition. For JavaScript-driven targets that require browser interaction, add Playwright to the stack — or use a managed scraping API if maintaining browser processes yourself adds more complexity than your project warrants.

What We Learned

Identifying the pagination pattern first is the critical step: URL-based, offset, next-link, cursor, and JavaScript pagination each require different code — the wrong approach silently fails or only collects page one.
Empty-results detection is the most reliable stopping condition: Checking whether a page returned any records is more robust than knowing the total page count in advance, which most sites don't expose.
JavaScript pagination requires browser automation, not HTTP clients: Infinite scroll and load-more buttons only work in a browser context — requests can't click buttons or respond to scroll events.
Rate control is essential for multi-page scraping: Many requests to the same domain in rapid succession accumulate detection signals faster than single-page access — paced delays between page requests are required, not optional.
Pagination structure changes require monitoring: A multi-page scraper that stops returning data is often reacting to a site redesign that changed its URL structure, not to a site that ran out of content.
Combining all-pages output into one file is the final step: Each page's records must be accumulated into a single dataset — don't let them live as separate page-level files unless your downstream workflow specifically requires it.

FAQ

How do I scrape multiple pages in Python?

Build a loop that increments the page number (or offset) parameter in the target URL, requests each page, extracts records, and stops when a page returns no records. For URL-based pagination, this is for page_num in range(1, max_pages + 1). For offset-based, increment by the page size on each iteration. For next-link pagination, extract the href of the next-page link and follow it until no next link is found.
How do I know when I've reached the last page?

The most reliable signal is checking whether a page returned any records: if the extraction function returns an empty list, you've gone past the last page. A secondary check — did the page return fewer records than the expected page size? — catches the final partial page. For next-link pagination, the absence of a next-page link in the HTML is the natural termination signal.
How do I scrape infinite scroll pages in Python?

Use a browser automation tool — Playwright or Puppeteer — to control a headless browser that scrolls the page programmatically. After each scroll event, wait for new content to load (using page.wait_for_load_state("networkidle") or a fixed delay), then check whether the page height increased. When the height stops growing, all content is loaded and you can extract from the final rendered page.
Can I use a web scraping API to handle pagination automatically?

A web scraping API handles rendering and content delivery per page request — it doesn't typically manage the pagination loop itself. You implement the loop in your code, calling the API for each page's content. However, some managed scraping platforms provide built-in crawling workflows that follow pagination links automatically, removing the need to implement the loop manually. Check your provider's documentation for any such features.
What is the difference between page number and offset pagination?

Page number pagination uses an integer page number in the URL (?page=3) where the server calculates which records correspond to that page. Offset pagination uses a numeric skip value (?offset=40&limit=20) that explicitly tells the server how many records to skip before returning the next batch. They're functionally equivalent — both return different slices of the same dataset — but offset pagination requires computing the offset value (offset = (page_number - 1) * page_size) rather than just incrementing a page counter.