How to Use cURL with Proxy and Mrscraper API

The source does not establish a best web scraping API. It explains how to use the Mrscraper API with cURL and a proxy. It covers proxy authentication and verbose debugging.

What Is cURL?

cURL is a command-line tool for making HTTP requests. It is commonly used to interact with web servers, test APIs, download files, and send data to online services. It supports HTTP methods like GET, POST, PUT, and DELETE. It can retrieve resources, submit data, and update information. This guide explains how to use cURL with a proxy when interacting with a web scraping API. It presents a step-by-step setup based on the API's JSON documentation, including proxy configuration and authentication. It also describes how proxy routing can support anonymity or access to content affected by geographic or network restrictions. The examples focus on sending API requests from the command line and using available endpoints for web scraping. A proxy is an intermediary between the client and destination server. When cURL is configured to use one, its connection can be routed through that intermediary. The guide explains useful commands to set this up. It also shows how to manage proxy credentials. It explains how to use cURL to collect data through APIs.

Why Use a Proxy?

A proxy routes your request through a different server instead of sending it directly from your connection. This can help you:

Mask your IP address.
Bypass geographic restrictions.
Improve security and anonymity.

When evaluating a web scraping API with built-in proxies, consider whether it lets you control where requests originate. Combining a proxy with a scraping API can provide more flexibility over request routing while you collect data.

Built-In Proxy Rotation

Manual rotation requires tracking proxy health, credentials, failures, and geographic assignment in your own code. When comparing ScraperAPI, ScrapingBee, Bright Data, Apify, or Oxylabs, verify rotation behavior, location controls, and pricing in each service’s documentation.

bash

#!/usr/bin/env bash
set -euo pipefail

: "${SCRAPING_API_URL:?Set SCRAPING_API_URL}"
: "${API_KEY:?Set API_KEY}"

curl --fail --silent --show-error \
  --request POST "$SCRAPING_API_URL" \
  --header "Authorization: Bearer $API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com"}'

The request contains no proxy address or rotation loop because the service controls that layer. This pattern reduces client-side state, while manual proxies remain useful when you need direct control over individual endpoints.

Step-by-Step Guide: Using cURL with a Proxy and Mrscraper API

The following steps explain how to send an API request with cURL and route the connection through a proxy.

Step 1: Understanding the API from Mrscraper.json

The API documentation in the mrscraper.json file defines several endpoints. A typical scrape request uses the POST method and sends JSON to the /api/v1/scrape endpoint. The Authorization header sends your API key as a bearer token. The Content-Type header tells the server the body is JSON. In the body, url identifies the website to scrape. The options object can include custom request headers, like a browser User-Agent. In this example, it is set to Mozilla/5.0.

json

{
  "url": "/api/v1/scrape",
  "method": "POST",
  "headers": {
    "Authorization": "Bearer your_api_key",
    "Content-Type": "application/json"
  },
  "body": {
    "url": "https://targetwebsite.com",
    "options": {
      "headers": {
        "User-Agent": "Mozilla/5.0"
      }
    }
  }
}

Step 2: Setting Up the cURL Command

Use this POST pattern to send a target URL and request options in a JSON body:

curl -X POST \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://targetwebsite.com","options":{"headers":{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36"}}}' \
  "https://api.example.com/v1/scrape"

The -H options add the authorization and content-type headers. The -d option supplies the JSON request body, including a browser-style User-Agent. The final argument identifies the API endpoint receiving the request.

API-Managed Proxy Routing

There is no universal best choice; prefer an API with documented proxy fields and predictable request behavior.

bash

curl --fail-with-body --request POST "$SCRAPING_API_ENDPOINT" \
  --header "Authorization: Bearer $API_KEY" \
  --header "Content-Type: application/json" \
  --data @request.json

This approach keeps cURL as a repeatable client and avoids confusing API-managed routing with a separate network proxy.

Step 3: Adding a Proxy to cURL

Route a cURL request through a proxy by adding the -x or --proxy option. The example below sends a JSON scraping request. It includes bearer authentication and Content-Type headers. It also provides a browser-style User-Agent in the request options.

bash

curl --proxy "http://proxy-server-address:8080" \
  --header "Authorization: Bearer your_api_key" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://targetwebsite.com","options":{"headers":{"User-Agent":"Mozilla/5.0"}}}' \
  "https://api.example.com/v1/scrape"

The -x and --proxy options select the proxy server. Replace proxy-server-address with its IP address or hostname and 8080 with the proxy's port, such as 8080.

Step 4: Handling Proxy Authentication

If the proxy requires authentication, use -U with its username and password. The request can also include the API key, JSON content type, target URL, and a browser-style User-Agent.

bash

curl -x "http://proxy-server-address:port" -U "proxyUsername:proxyPassword" \
  -H "Authorization: Bearer your_api_key" -H "Content-Type: application/json" \
  -d '{"url":"https://targetwebsite.com","options":{"headers":{"User-Agent":"Mozilla/5.0"}}}' \
  "https://api.example.com/v1/scrape"

Step 5: Debugging the Request

Add the -v (verbose) option when you need detailed information about a request. cURL then prints connection details, request headers, and the server response. It gives a step-by-step view that can help find errors.

bash

curl -v \
  -x http://proxy-server-address:port \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://targetwebsite.com","options":{"headers":{"User-Agent":"Mozilla/5.0"}}}' \
  https://api.example.com/v1/scrape

The following complete example sends a scrape request through an authenticated proxy. It uses the proxy address and credentials from the proxy provider. It sends the API key in the Authorization header. It includes a browser-style User-Agent.

bash

curl -x http://your-proxy.com:8080 \
  -U myProxyUser:myProxyPass \
  -H "Authorization: Bearer my_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","options":{"headers":{"User-Agent":"Mozilla/5.0"}}}' \
  https://api.example.com/v1/scrape

The -x option routes the cURL connection through http://your-proxy.com:8080.
The -U option authenticates with myProxyUser and myProxyPass.
The API receives the request to scrape https://example.com through the proxy.
The User-Agent value identifies the request as a browser-style request.

Conclusion

Combining cURL, a proxy, and a web scraping API provides control over where requests originate. Routing requests through a proxy can help with location blocks and privacy needs. The API can still handle the data request. Structure the cURL command with care. Then test different proxy and API setups to find what works for your scraping task. If you are choosing a web scraping API with built-in proxies, compare proxy options, request flow, and docs to your needs.

Do not assume one service is best for every use case. For further guidance, read the article on converting cURL commands to Python for efficient web scraping. It explains how to move from cURL commands to Python code when your workflow needs greater automation or scalability. Use the examples in this guide as a starting point. Check each proxy and API setting before you send production requests.

What We Learned

The key distinction is whether proxy routing is built into the scraping API or configured separately with cURL. Choose based on documented authentication, geographic controls, response handling, and operational cost rather than a universal “best” label.

Use cURL’s proxy options when you need to route the API connection itself.
Use an API with built-in proxies when proxy selection belongs in the scraping workflow.
Verify endpoint, authentication, and proxy behavior against current documentation.
For deeper command-line reference, consult the curl project’s “Everything curl.”