How to Web Scrape a Table in Python: From Static HTML to Dynamic Pages

To scrape tables in Python, use pandas.read_html() for static HTML or Selenium to scrape dynamic websites where content is rendered via JavaScript. For high-volume needs, direct API inspection or the Scrapy framework provides more efficient, scalable data extraction.

1. The Quick Way: Using pandas.read_html()

The easiest method for scraping tables is with [pandas.read_html()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html), which automatically detects and converts HTML tables into Pandas DataFrames.

import pandas as pd

url = "https://en.wikipedia.org/wiki/Demographics_of_India"
tables = pd.read_html(url, match="Population distribution")
df = tables[0]
print(df.head())

• This method uses BeautifulSoup and lxml under the hood.
• The match parameter helps target a specific table.

Pros: Extremely fast and simple. Cons: Only works on static HTML tables.

2. More Control: BeautifulSoup + Requests

If you need more control, you can use a reliable approach. You can combine requests with BeautifulSoup. This also helps you clean data while you extract it.

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "https://datatables.net/examples/styling/stripe.html"
resp = requests.get(url)
soup = BeautifulSoup(resp.text, "html.parser")
table = soup.find("table", class_="stripe")

rows = []
for tr in table.tbody.find_all("tr"):
    cells = [td.get_text(strip=True) for td in tr.find_all("td")]
    rows.append(cells)

df = pd.DataFrame(rows, columns=[th.get_text() for th in table.thead.find_all("th")])
print(df.head())

This is helpful when:

• The table is nested inside custom HTML structures.
• You want to customize how rows and columns are parsed.

3. Scraping Dynamic Tables with Selenium

Static HTML parsers cannot capture tables generated via JavaScript or AJAX. To scrape dynamic websites well, you must utilize a tool like Selenium. It renders the full DOM like a standard web browser.

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
import time

# Initialize the Chrome driver
driver = webdriver.Chrome()
driver.get("https://example.com/dynamic_table")

# Wait for JavaScript to populate the table
time.sleep(3)

# Capture the rendered source and parse with BeautifulSoup
html = driver.page_source
soup = BeautifulSoup(html, "html.parser")

# Locate the table and convert to a DataFrame
target_table = soup.find("table", id="myTable")
df = pd.read_html(str(target_table))[0]

# Clean up the session
driver.quit()
print(df.head())

• Pros: Successfully renders JavaScript-heavy content and handles user interactions.
• Cons: Slower execution speeds and requires specific browser drivers like ChromeDriver.

Handling Dynamic Elements with Playwright

Modern single-page applications often load data asynchronously after the initial page load.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example.com/dynamic-table')
    
    # Wait for the table body to contain at least one row
    page.wait_for_selector('table tr td')
    
    rows = page.query_selector_all('tr')
    data = [row.inner_text() for row in rows]
    print(data)
    browser.close()

4. Accessing Hidden APIs Behind Tables

Sometimes the table content is not hardcoded into the HTML but fetched from an API in the background. This is actually a more efficient way to extract data:

1. Open DevTools → Network → XHR/Fetch
2. Locate the API URL used to load table data
3. Use requests.get() to retrieve JSON data

import requests
import pandas as pd

api = "https://www.levantineceramics.org/vessels/datatable.json"
data = requests.get(api).json()
df = pd.DataFrame(data["data"])
print(df.head())

Pros: Fast and clean. Cons: Requires inspecting the site’s network calls.

5. Scalable Scraping with Scrapy

For projects requiring high-performance crawling or asynchronous processing at scale, Scrapy provides a robust framework. It excels at navigating complex site structures while managing data pipelines and concurrent requests efficiently.

import scrapy

class TableSpider(scrapy.Spider):
    name = "table_spider"
    start_urls = ["https://example.com/page_with_table"]

    def parse(self, response):
        # Iterate through each row in the table using XPath
        for row in response.xpath('//table//tr'):
            yield {
                'column1': row.xpath('td[1]/text()').get(),
                'column2': row.xpath('td[2]/text()').get()
            }

While it offers excellent scalability and built-in export pipelines, the initial setup and learning curve are more demanding than standalone libraries. It is best suited for production-grade web scraping tasks.

Comparison Table

Responsible Scraping

Before scraping, always:

• Check robots.txt and the site’s Terms of Service
• Use rate limiting to avoid overloading the server
• Add headers like user-agent to mimic a browser
• Use proxies or headless browsing to avoid blocks

No-Code Scraping with MrScraper

If coding isn’t your thing-or you need to extract tables from difficult or protected websites-use MrScraper.

MrScraper is a visual, AI-powered web scraping tool that makes it easy to:

• Extract tables with just a few clicks
• Scrape JavaScript-rendered pages
• Export to CSV or JSON
• Use proxy rotation and CAPTCHA bypass automatically

Whether you're scraping product lists, public records, or movie data, MrScraper handles the hard part for you-no code required.

Conclusion

Learning how to web scrape a table in Python opens up a world of possibilities for data analysis, automation, and research. Whether you’re scraping a static table from Wikipedia or a dynamic one from an e-commerce site, Python offers flexible tools to make the job easier.

And for those who want a simple, efficient solution, MrScraper helps you collect structured data from any website. You can do it without writing code.

Ready to scrape your first table? Try MrScraper today.

What We Learned

Selecting the right extraction method depends on whether the target data is in the first server response. It may also require client side execution. While Pandas is best for static tables, modern web apps often need browser automation or direct API calls. This helps access content hidden by async scripts.

• Static Extraction: Use Pandas or BeautifulSoup for high performance and low resource overhead.
• Dynamic Content: Employ Selenium when data requires JavaScript execution or user interaction.
• Hidden Endpoints: Inspect network traffic to find JSON sources for the cleanest data retrieval.
• Infrastructure: Implement Scrapy for complex, multi page projects requiring concurrent processing.
• Reliability: Always include custom User Agent headers and respectful delays to maintain access.

For developers managing high volumes or sophisticated anti scraping measures, rotating proxies and solving challenges becomes a significant engineering hurdle. Tools like ScraperAPI, ScrapingBee, Bright Data, Apify, and Oxylabs offer managed infrastructure to handle these complexities at scale.

import pandas as pd

def quick_extract(url, match_text):
    try:
        tables = pd.read_html(url, match=match_text)
        return tables[0]
    except ValueError:
        return "Table not found."

# Example usage for a quick summary extraction
df = quick_extract('https://example.com/data', 'Revenue')
print(df.head())

Handling Infinite Scroll and Lazy Loading

import time
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com/dynamic-table")

last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

rows = driver.find_elements(By.TAG_NAME, "tr")
print(f"Extracted {len(rows)} rows after scrolling.")

For production environments where sites use anti-bot tools to block automated scrolling, specialized APIs can help. Providers like ScrapingBee, Bright Data, or Oxylabs can run these browser actions on the server side.

Streamline Your Data Extraction

Accelerate your development by accessing our comprehensive library of scraping resources and high-performance infrastructure designed to handle complex table structures effortlessly.

Get Started

Frequently asked questions

What is the fastest way to scrape a basic HTML table in Python?

The fastest method is using the pandas.read_html() function. It automatically finds table elements in HTML and converts them into DataFrames, but it only works with static content.

How do you handle tables that only appear after a page loads?

For tables rendered with JavaScript, use Selenium to simulate a browser. Wait for the elements to load. Or inspect the Network tab in DevTools to find the JSON API. Then call the API directly.

When should I use Scrapy instead of BeautifulSoup for table scraping?

Scrapy works best for large projects that need async processing and multi-page crawling. BeautifulSoup is better for simple, one-time extraction tasks with custom HTML.