What Is a Proxy? Types, How They Work & Scraping Uses

A proxy server is an intermediary computer that receives your internet requests and forwards them to the destination on your behalf. The destination sees the proxy's IP address, not yours. Proxies are used for privacy, network control, geo-specific access, and above all for large-scale web scraping.

Most pages that define the word stop there. This one answers the three questions that actually decide whether your scraper runs: which type to buy, how many addresses you need, and why a clean IP still gets you blocked.

Key takeaways

• A proxy changes the IP address a website records against your request, and nothing else about your traffic.
• Proxies are classified two ways at once: by how much they reveal (transparent through elite) and by where the IP comes from (datacenter, residential, ISP, mobile).
• Datacenter IPs are identifiable from a single ASN lookup, which is why they fail on defended sites and cost a fraction of the alternatives.
• Rotate for anonymous read-only fetching; use a sticky session for anything with a login, a cart, or multi-page state.
• The number of proxies you need comes from your request rate divided by the target's safe rate per IP, not from your total page count.
• A proxy is one signal among many: TLS and browser fingerprints will still identify automation from a perfectly clean residential address.
• Plain HTTP proxies do not encrypt anything, and free public proxy lists are run by operators who can log and modify your traffic.

What is a proxy server?

Proxy server: A proxy server is an intermediary computer that receives your internet requests and forwards them to the destination on your behalf, so the destination sees the proxy's IP address instead of yours.

Proxy IP: The public IP address a proxy uses to make requests. It is the address the target website records and rate-limits, not your real one.

A proxy is a forwarding address for your internet traffic: the recipient only ever writes down the address on the envelope.

What the target website can see:

• The proxy's IP address
• The ASN that IP is registered to, which identifies the network operator
• The geolocation associated with that IP
• Everything else about your request: headers, TLS handshake, timing, cookies

What the target website cannot see:

• Your real IP address

That second list has exactly one item on it, and the gap between the two lists is where most scraping projects go wrong.

How does a proxy server work?

A proxy works by terminating your connection and opening a second one of its own. Five steps:

1. Your client sends the request to the proxy's IP and port instead of the target.
2. The proxy authenticates you by credentials or IP whitelist.
3. The proxy opens its own connection to the target using its own IP.
4. The target responds to the proxy.
5. The proxy passes the response back to you.

Step 3 is the whole mechanism. Because the proxy makes a fresh TCP connection, the target's access log records the proxy's address, and any per-IP rate limit counts against the proxy rather than against you.

How your headers change is what separates a usable scraping proxy from a useless one. A transparent proxy appends your real address to the X-Forwarded-For header and announces itself with Via. An elite proxy sends neither.

X-Forwarded-For: The HTTP header proxies use to pass along the original client IP address. Its presence is one of the simplest ways a website detects that a request came through a proxy.

The extra hop costs latency. Budget roughly 50 to 150 ms of added round-trip time on a datacenter proxy and several hundred milliseconds on a mobile one, which is why concurrency matters more than per-request speed.

What are the main types of proxies?

Proxies are classified on two independent axes. A single proxy has a position on both: a residential elite proxy and a datacenter elite proxy are equally anonymous in their headers and wildly different in how quickly they get blocked.

Table A: by anonymity level

Transparent proxy: A proxy that intercepts traffic without any client configuration and still passes your real IP in the X-Forwarded-For header. It provides no anonymity. Most corporate and school filters work this way.

Anonymous proxy: A proxy that hides your real IP but still identifies itself as a proxy through headers such as Via.

Elite (high-anonymity) proxy: A proxy that hides your real IP and sends no proxy-identifying headers, so the request looks like it came from an ordinary client.

For scraping, only the bottom row is worth buying. Anonymous and distorting proxies flag themselves in the headers, and bot detection treats a self-declared proxy as an automated client. Anything sold for scraping should be elite by default; test it if the product page does not say so.

Table B: by IP source

Datacenter proxy: An IP address owned by a cloud or hosting provider and registered to a datacenter ASN. Fast and cheap, but easy for target sites to identify with a simple ASN lookup.

Residential proxy: An IP address assigned by a consumer internet provider to a real home connection. Because it looks like ordinary household traffic it is far harder to block, and it is usually billed per gigabyte.

ISP proxy (static residential): An IP registered to a consumer ISP but hosted in a datacenter. It combines datacenter speed and stability with a residential-looking ASN, and it does not change between requests.

Mobile proxy: An IP from a mobile carrier's 4G or 5G network. Carriers share these addresses across many real subscribers via CGNAT, so blocking one address risks blocking real customers. Highest trust, highest cost.

The detection-risk column is really an ASN column. Every IP is announced to the global routing table by an autonomous system, and that mapping is public: one lookup tells an anti-bot vendor whether your address belongs to a hosting company or to Comcast. No header tuning changes that answer, which is why residential addresses cost more. Our residential proxy vs datacenter proxy comparison covers the cost side.

Mobile sits at the extreme end because of CGNAT: thousands of real subscribers share one carrier address, so blocking it blocks paying customers alongside your scraper. That shared blast radius is what buys the trust. See mobile proxies vs residential proxies.

What is the difference between a forward proxy and a reverse proxy?

A forward proxy represents the client. A reverse proxy represents the server. Same machinery, opposite ends of the connection, and the word "proxy" on its own almost always means the forward kind.

Forward proxy: A forward proxy sits in front of the client. It hides who is making the request. This is what people mean by "a proxy" in scraping and privacy contexts.

Reverse proxy: A reverse proxy sits in front of the server. It hides which backend machine answers a request and handles TLS termination, caching and load balancing. NGINX, HAProxy and Cloudflare are common examples.

When you scrape a site sitting behind Cloudflare, both kinds are in play at once: your forward proxy is hiding you, and their reverse proxy is deciding whether to serve you. An API gateway is a reverse proxy with authentication, rate limiting and routing added on top.

Proxy vs VPN — what's the difference?

A VPN encrypts every packet leaving your device and sends it through one tunnel to one exit address. A proxy redirects specific requests through an address you choose per request.

The row that matters for scraping is the third. A VPN gives you one address at a time, so a hundred concurrent workers share a single rate-limit bucket. Use a VPN to protect yourself, use proxies to distribute your traffic. The residential proxy vs VPN breakdown covers the less obvious cases.

What are HTTP, HTTPS and SOCKS5 proxies?

The protocol determines what the proxy understands about your traffic and what it can do with it.

HTTP proxy: A proxy that understands the HTTP protocol and can read, cache, and modify requests. For HTTPS traffic, it opens a blind tunnel using the CONNECT method and cannot read the contents.

SOCKS5 proxy: A lower-level proxy that forwards raw TCP and UDP packets without interpreting them. It works with any protocol, not just web traffic, and supports authentication.

For an HTTPS URL, an HTTP proxy issues a CONNECT request and then relays encrypted bytes it cannot inspect. That is why "HTTP proxy" and "HTTPS proxy" usually describe the same endpoint doing two different jobs. SOCKS5, defined in RFC 1928, sits lower and forwards packets without caring what is in them.

In a scraper config, the protocol shows up in the scheme. Most tooling reads the http_proxy and https_proxy environment variables, and a socks5:// or socks5h:// scheme switches to SOCKS.

# pip install "requests[socks]" for the socks5h:// scheme
import requests

# One endpoint serves both schemes; the CONNECT tunnel handles the HTTPS case.
proxies = {
    "http":  "http://user:pass@proxy.example.com:8080",
    "https": "http://user:pass@proxy.example.com:8080",
}

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
exit_ip = response.json()["origin"]          # assert on the payload, not the status code
print(f"Target sees: {exit_ip}")

# -x sets the proxy. socks5h:// resolves DNS at the proxy instead of locally,
# which stops your resolver from leaking the hostnames you are scraping.
curl -x http://user:pass@proxy.example.com:8080 https://httpbin.org/ip
curl -x socks5h://user:pass@proxy.example.com:1080 https://httpbin.org/ip

Prefer socks5h over socks5: the h moves hostname resolution to the proxy, so your local DNS server stops seeing every domain you request. Fuller walkthrough in our guide to using a SOCKS5 proxy server.

What is a rotating proxy, and when do you need sticky sessions?

A rotating proxy gives you a different exit IP on every request. A sticky session holds one IP for a fixed window.

Rotating proxy: A proxy setup that assigns a different exit IP from a pool on every request, or after a fixed time, so no single address accumulates enough requests to be rate-limited.

Sticky session: A proxy configuration that keeps the same exit IP for a set period, so multi-step flows such as logins, carts and paginated results are not broken by a mid-session IP change.

Backconnect proxy: A single endpoint (one host and port) that routes your requests to a rotating pool behind it, so you never manage individual IP addresses yourself.

Proxy pool: The full set of IP addresses a provider can route your traffic through. Pool size and geographic spread determine how long you can scrape before addresses repeat.

Rotate for anonymous, read-only page fetching; use sticky for anything with a login, cart, or multi-page state. That single rule resolves most of the configuration questions people ask about proxies.

A site issues a session cookie to the address that logged in. Rotate on the next request, and the cookie arrives from a stranger, so the site bounces you back to the login page.

Sticky session windows are set by the provider and commonly run from a few minutes to around half an hour. Check your provider's documented limit rather than assuming a number: a session that expires mid-checkout looks identical to a block. Sticky vs rotating residential proxies covers jobs that need both.

Why do web scrapers need proxies?

Five specific reasons, each of which shows up as a different symptom in your logs:

• Sites rate-limit per IP; a single address hits HTTP 429 Too Many Requests quickly, and the 429 rate-limit guide covers the retry behaviour that follows.
• Concurrency is capped by IP count. A hundred parallel workers on one IP is one blocked IP, not a hundred times the throughput.
• Geo-specific data such as pricing, search results, and availability only appears from a local IP.
• IP reputation: datacenter ASNs are pre-flagged by most anti-bot vendors before your first request arrives.
• Retry and failover: a pool lets a failed request retry from a clean address instead of hammering the one that just got blocked.

How many addresses you need is arithmetic, not a package tier. Divide your target request rate by the safe rate per IP for that site. A job running 600 requests per minute against a site that tolerates one request every three seconds per IP needs at least 30 addresses working continuously, before any allowance for retries or burned IPs. Pool size and its effect on success rate is measured in more depth in our analysis of residential proxy pool size.

How do you choose the right proxy for your use case?

Geo-targeting: Selecting the country, state, city, or carrier of your exit IP so the target site returns the regional version of its content, pricing or search results.

Proxy authentication: How a provider verifies you before routing traffic: either username-and-password credentials sent with each request, or IP whitelisting, where only pre-approved source addresses are accepted.

Start one tier cheaper than you think you need and measure the success rate. Paying per gigabyte for residential bandwidth to fetch a public sitemap is a common and expensive mistake, and the reverse mistake costs you a week of debugging blocks that were never going to clear.

What are the limits and risks of using a proxy?

• No encryption by default. A plain HTTP proxy can read unencrypted traffic. Use HTTPS end to end so the operator only ever sees a CONNECT tunnel.
• Free proxy lists are dangerous. Unknown operators can log, inject, or modify traffic, and many public lists exist specifically to harvest what passes through them.
• Proxies don't beat fingerprinting. Clean IPs still fail against TLS/JA3 and browser fingerprint checks, so a real browser environment is also needed. Browser fingerprinting and what it means for scrapers shows what the checks actually read.
• Per-GB billing surprises. Residential bandwidth adds up fast on image-heavy pages; block images, fonts and media so you pay for HTML rather than for product photography.
• Legal and ToS considerations. A proxy changes your IP, not your obligations. Respect robots.txt, rate limits and applicable law, and settle the question before a commercial project starts with our guide to whether web scraping is legal.

Proxy chaining: Routing a request through two or more proxies in sequence. It adds layers of separation at the cost of latency and reliability.

The third bullet is the one that surprises people who have just paid for a good pool. The IP is one signal among roughly a dozen, and a pristine residential address attached to a default python-requests TLS signature is still obviously automation.

How do you check whether you're using a proxy?

Proxy settings live in a different place on every platform:

• Windows: Settings → Network & Internet → Proxy
• macOS: System Settings → Network → your connection → Details → Proxies
• iOS and Android: the Wi-Fi network's settings, under HTTP Proxy
• Browsers and shells: the http_proxy, https_proxy and no_proxy environment variables, which most command-line tools read automatically

The settings screen tells you what is configured, not what the internet actually sees. Verify the exit address directly:

# Compare the two. If they match, your traffic is not going through the proxy.
curl -s https://httpbin.org/ip                                   # your real exit IP
curl -s -x http://user:pass@proxy.example.com:8080 https://httpbin.org/ip

When a Wi-Fi network asks you to "configure proxy", it is asking you to route traffic through a machine that the network operator controls. On corporate and campus networks, that is usually a transparent proxy doing filtering and caching, and it sees every unencrypted request you make. See how to test if your residential proxy is working for the step-by-step version.

How MrScraper handles proxies for you

Everything above is the manual path: pick a type, buy a pool, wire up rotation, handle session state, and retry the failures yourself. MrScraper's Residential Proxy collapses that into one backconnect endpoint where the configuration lives in the username string.

The mechanism, concretely:

• One endpoint for every mode: proxy.mrscraper.com:10000. Rotation, geo-targeting and session persistence are all selected by how you build the username, so switching behaviour never means changing infrastructure.
• Rotation is the default. Omit a session ID and every request exits from a different address.
• Sessions are opt-in per request via sessid, with the window set by sesstime in minutes.
• Geo-targeting uses ISO 3166 country codes across 100+ countries.
• Billing is per gigabyte, starting at $2.5/GB, so the cost lever is what you download rather than how many addresses you touch.

The checkable number: the US pool alone is listed at 10,353,360 addresses, against a published 99.95% success rate.

# Rotating: no sessid, so every request leaves from a new US address.
curl -x "http://user-country-us:pass123@proxy.mrscraper.com:10000" https://httpbin.org/ip

# Sticky: same exit IP for 30 minutes, keyed to the session ID you choose.
curl -x "http://user-country-us-sessid-test1-sesstime-30:pass123@proxy.mrscraper.com:10000" https://httpbin.org/ip

import requests

# Replace `user` and `pass123` with the credentials from your MrScraper dashboard.
USER, PASSWORD = "user", "pass123"

def proxy_for(country: str, session_id: str | None = None, minutes: int = 10) -> dict:
    """Build the proxy dict. Passing session_id pins one exit IP; omitting it rotates."""
    username = f"{USER}-country-{country}"
    if session_id:
        username += f"-sessid-{session_id}-sesstime-{minutes}"
    url = f"http://{username}:{PASSWORD}@proxy.mrscraper.com:10000"
    return {"http": url, "https": url}

# A sticky session: both requests should report the same exit address.
session = proxy_for("us", session_id="checkout-42", minutes=30)
first  = requests.get("https://httpbin.org/ip", proxies=session, timeout=30).json()["origin"]
second = requests.get("https://httpbin.org/ip", proxies=session, timeout=30).json()["origin"]
assert first == second, f"session did not hold: {first} then {second}"

Full parameter reference, plus Node.js, Selenium and Puppeteer examples, is in the MrScraper residential proxy documentation.

The honest limitation: this solves the IP half of the problem. A site that fingerprints your TLS handshake will still identify a plain requests client through a perfect residential address, which is why proxies and a real browser environment are separate purchases.

Frequently asked questions

What is a proxy in simple terms?

A proxy is a middleman server for your internet traffic. Instead of your computer contacting a website directly, it sends the request to the proxy, and the proxy contacts the website for you. The website sees the proxy's address rather than yours.

What does a proxy server actually do?

A proxy server receives your request, opens its own connection to the target site, retrieves the response, and passes it back to you. Along the way, it can hide your IP address, cache content, filter what you're allowed to reach, and make your traffic appear to come from a different country.

Is a VPN the same as a proxy?

No. A VPN encrypts all traffic from your entire device through one tunnel and one IP address. A proxy usually routes traffic from a single app or request and does not encrypt on its own. VPNs are for personal privacy; proxies are for distributing many requests across many IPs.

What is the difference between a forward proxy and a reverse proxy?

A forward proxy sits next to the client and hides who is sending the request. A reverse proxy sits next to the server and hides which backend machine answers it. Corporate filters and scraping proxies are forward proxies; NGINX, HAProxy, and Cloudflare act as reverse proxies.

What is the difference between residential and datacenter proxies?

A datacenter proxy uses an IP owned by a hosting company, so it is fast and cheap but easy to identify from its ASN. A residential proxy uses an IP that a consumer internet provider assigned to a real home connection, so it blends in with normal traffic, but it is slower and usually billed per gigabyte.

Which type of proxy is best for web scraping?

It depends on the target. Use datacenter proxies for public, lightly defended pages. Use rotating residential proxies for e-commerce, travel, and other sites with strong anti-bot systems. Use ISP or sticky residential proxies when a session must persist across a login. Use mobile proxies for the hardest mobile-first platforms.

Do proxies encrypt your traffic?

Not by themselves. A plain HTTP proxy forwards traffic as-is, so unencrypted requests remain readable to the proxy operator. Your traffic is protected only if the connection to the target site uses HTTPS, which the proxy tunnels blindly. If you need encryption from your device outward, use a VPN.

Are free proxies safe?

Generally no. Free public proxies are run by unidentified operators who can log your requests, inject content into unencrypted pages, or resell your bandwidth. Many are slow, already blocked by major sites, or deliberately set up to harvest data. For anything involving credentials or business data, use a paid provider.

Are proxies legal?

Using a proxy is legal in most countries, and businesses run them routinely for security, caching, and testing. What matters is what you do through one. A proxy changes your IP address; it does not change a website's terms of service, copyright law, or data protection rules. Check the target site's terms and the law in your jurisdiction.

Why am I still getting blocked even though I'm using proxies?

Because an IP address is only one of several signals. Sites also check TLS fingerprints, browser fingerprints, user-agent consistency, header order, request timing, and behaviour. A clean residential IP paired with an obvious automation fingerprint will still fail. Fixing blocks usually means changing the browser environment, not just the IP.

What is a rotating proxy?

A rotating proxy gives you a different exit IP from a pool on each request, or after a set time period. This spreads your requests across many addresses so no single one hits a site's rate limit. You connect to one fixed endpoint and the provider handles the rotation behind it.

When should I use a sticky session instead of rotation?

Use a sticky session whenever the site needs to remember you between requests: logging in, adding to a cart, moving through paginated results, or completing a multi-step form. Rotating mid-flow makes the site see a different visitor on each step and usually breaks the session.

How do I find my proxy server address?

On Windows, open Settings → Network & Internet → Proxy. On macOS, open System Settings → Network → your connection → Details → Proxies. On iOS and Android, open the Wi-Fi network's settings and look for the HTTP Proxy field. To confirm what a website actually sees, request an IP-echo endpoint through your proxy and compare it with your real IP.

How many proxies do I need for web scraping?

It depends on your request rate and the target's rate limit, not on total page count. Divide your requests per minute by the safe request rate per IP for that site. A site tolerating roughly one request every few seconds per IP needs dozens of addresses for moderate volume, and pools in the thousands for large jobs.

Ready to stop managing IP pools and go back to writing parsers? Try MrScraper free. 1,000 tokens, no credit card required.