Scrape Bing Search: A Practical Technical Guide

To scrape Bing search, send paginated requests to Bing. Parse result fields with Python. Plan for delays, retries, proxies, rate limits, blocking, and layout changes.

Understanding Bing Search URLs

The basic Bing search endpoint looks like this: `

Useful parameters:

Parameter	Description
q	Search query
first	Offset for pagination
count	Results per page
setmkt	Market/language

This returns the second page of results. Pagination usually increments the first parameter by the number of results per page.

Basic Python Bing Scraper

Start with a simple scraper using requests and BeautifulSoup.

Install dependencies:

bash

pip install requests beautifulsoup4

Copy

Basic example:

python

import requests
from bs4 import BeautifulSoup

url = "https://www.bing.com/search?q=python+scraping"

headers = {
    "User-Agent": "Mozilla/5.0"
}

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

results = soup.select("li.b_algo")

for r in results:
    title = r.select_one("h2").get_text(strip=True)
    link = r.select_one("h2 a")["href"]
    desc = r.select_one(".b_caption p")

    description = desc.get_text(strip=True) if desc else ""

    print(title)
    print(link)
    print(description)
    print("-" * 40)

Copy

Typical organic results appear inside li.b_algo. Each result usually contains:

h2 → title
h2 a → link
.b_caption p → description

This structure has stayed relatively stable across Bing versions.

Adding Pagination

A real scraper needs more than one page.

Example:

python

import requests
from bs4 import BeautifulSoup
import time

query = "python scraping"
results_per_page = 10

headers = {
    "User-Agent": "Mozilla/5.0"
}

for page in range(0, 50, results_per_page):
    url = f"https://www.bing.com/search?q={query}&count=10&first={page+1}"
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")

    for r in soup.select("li.b_algo"):
        title = r.select_one("h2").get_text(strip=True)
        link = r.select_one("h2 a")["href"]
        print(title, link)

    time.sleep(2)

Copy

The delay is important. Without it, Bing will start rejecting requests. Bing applies rate limiting and bot detection that can block aggressive scrapers.

Using Proxies

To scrape Bing search at moderate volume, route requests through proxies. As volume grows, manually managing IP addresses becomes difficult. IP rotation helps avoid sending repeated queries from one address. The example assumes url and headers are defined earlier.

python

import requests

proxies = {
    "http": "http://user:pass@proxy-ip:port",
    "https": "http://user:pass@proxy-ip:port"
}

response = requests.get(
    url,
    headers=headers,
    proxies=proxies,
    timeout=20
)

Without rotation, responses may eventually include:

CAPTCHA pages
HTTP 403 errors
Empty responses

Search engines can track request patterns and block repeat queries from one IP. They may return CAPTCHA pages, HTTP 403 errors, or empty responses.

Handling Timeouts and Retries

Network failures happen often during scraping. Always retry.

Example:

python

import requests
import time

def fetch(url):
    for attempt in range(5):
        try:
            r = requests.get(
                url,
                headers={"User-Agent": "Mozilla/5.0"},
                timeout=10
            )
            if r.status_code == 200:
                return r.text
        except requests.exceptions.RequestException:
            pass
        time.sleep(2)
    return None

Copy

This avoids crashes when requests fail.

Scraping Bing Images

To scrape Bing image search results, call Bing’s async image endpoint with your query. Include the offset and async mode parameters. Image results load asynchronously, and each image’s metadata appears in the HTML element’s JSON attributes. The selector returns structured metadata.

python

import requests
from bs4 import BeautifulSoup

def scrape_bing_images(query="cats", first=1, selector="a.iusc"):
    url = "https://www.bing.com/images/async"
    params = {"q": query, "first": first, "mmasync": 1}
    headers = {"User-Agent": "Mozilla/5.0"}
    response = requests.get(url, params=params, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    for element in soup.select(selector):
        print(element.get("m"))

scrape_bing_images()

Conclusion

Scraping Bing search results is straightforward at a small scale. A few lines of Python can extract titles and URLs in minutes. Scaling is the hard part. Blocking, rate limits, and layout changes appear quickly once the request volume grows.

Once these pieces are in place, Bing becomes a reliable data source. You can use it for ranking tools, research projects, and analytics pipelines.

What We Learned

Bing search results can provide titles, URLs, descriptions, and ranking positions for SEO monitoring, research, and competitive analysis.
A basic Python scraper can use Bing’s search endpoint with pagination to collect results across multiple pages.
Delays, proxies, and IP rotation become important as scraping volume increases because Bing applies rate limits and bot detection.
Retries help stop network failures from crashing a scraper. Bing Images needs a separate endpoint and a different way to handle metadata.
Scraping is relatively straightforward at small scale, but blocking, rate limits, and layout changes make scaling more difficult.