Python cURL: Making HTTP Requests with pycurl

When working with web APIs or scraping, making HTTP requests is essential. While Python’s requests library is the most common tool for this, sometimes you might need the raw power and performance that comes with cURL. Fortunately, Python has pycurl, a Python interface for the well-known cURL library, allowing you to manage more complex HTTP interactions with greater control.

What is cURL?

cURL is a command-line tool used for transferring data using various network protocols, including HTTP, HTTPS, FTP, and more. It's widely known for its flexibility in handling complex HTTP requests, such as handling custom headers, cookies, authentication, and proxies.

In Python, pycurl gives you access to the same features while offering better performance in some cases.

Installing pycurl

First, you'll need to install pycurl. You can do this using pip:

pip install pycurl

However, depending on your system, you may need to install libcurl as well: On Ubuntu/Debian:

sudo apt-get install libcurl4-openssl-dev

On macOS (using Homebrew):

brew install curl

Basic Example: HTTP GET Request

Let’s start with a basic GET request using pycurl. This is equivalent to using curl in the command line to fetch a URL.

import pycurl
from io import BytesIO

# Create a buffer to store the response
response_buffer = BytesIO()

# Initialize a Curl object
curl = pycurl.Curl()

# Set the URL for the request
curl.setopt(curl.URL, 'https://jsonplaceholder.typicode.com/posts')

# Write the response data to the buffer
curl.setopt(curl.WRITEDATA, response_buffer)

# Perform the request
curl.perform()

# Get the HTTP response code
status_code = curl.getinfo(pycurl.RESPONSE_CODE)

# Cleanup
curl.close()

# Get the response data and decode it
response_body = response_buffer.getvalue().decode('utf-8')

print(f"Status Code: {status_code}")
print(f"Response Body: {response_body}")

Key Components:

• curl.setopt(): This sets various options for the request, such as the URL and where to write the response data.
• curl.perform(): This actually sends the request and waits for the response.
• curl.getinfo(pycurl.RESPONSE_CODE): Fetches the HTTP response code from the server.
• Buffer (BytesIO): Used to capture the raw response data.

Adding Custom Headers

Custom headers are often required when interacting with APIs. You can easily set headers using pycurl.

curl.setopt(curl.HTTPHEADER, [
    'Content-Type: application/json',
    'Authorization: Bearer YOUR_ACCESS_TOKEN'
])

This allows you to send authorization tokens, content types, or any other required headers in your requests.

Handling POST Requests

For POST requests, pycurl lets you send data along with the request. Here's an example of how to send JSON data via POST.

import json

curl = pycurl.Curl()

data = json.dumps({
    'title': 'foo',
    'body': 'bar',
    'userId': 1
})

curl.setopt(curl.URL, 'https://jsonplaceholder.typicode.com/posts')
curl.setopt(curl.POST, 1)
curl.setopt(curl.POSTFIELDS, data)

# Set the content type to application/json
curl.setopt(curl.HTTPHEADER, ['Content-Type: application/json'])

response_buffer = BytesIO()
curl.setopt(curl.WRITEDATA, response_buffer)

curl.perform()
status_code = curl.getinfo(pycurl.RESPONSE_CODE)
curl.close()

response_body = response_buffer.getvalue().decode('utf-8')

print(f"Status Code: {status_code}")
print(f"Response Body: {response_body}")

In this example:

• curl.POSTFIELDS: Sends the JSON data as part of the POST request.
• curl.POST: Specifies that it’s a POST request.

Handling SSL Certificates

When making HTTPS requests, you often need to handle SSL certificates. By default, pycurl checks SSL certificates, but sometimes in development or testing environments, you might want to skip this check.

curl.setopt(curl.SSL_VERIFYPEER, 0)  # Disable SSL certificate verification
curl.setopt(curl.SSL_VERIFYHOST, 0)

In production, it’s crucial to ensure SSL verification is enabled to avoid man-in-the-middle (MITM) attacks.

Proxy Support

To make requests via a proxy server, you can use the following options in pycurl:

curl.setopt(curl.PROXY, 'http://proxy.example.com:8080')

For proxies that require authentication:

curl.setopt(curl.PROXYUSERPWD, 'username:password')

Timeout and Retry Mechanism

In real-world applications, timeouts and retry mechanisms are essential to handle slow or failing network connections.

curl.setopt(curl.TIMEOUT, 10)  # Set timeout to 10 seconds
curl.setopt(curl.CONNECTTIMEOUT, 5)  # Set connection timeout to 5 seconds

For retries, you’ll need to manually implement a retry mechanism in Python by wrapping the request logic in a loop.

Downloading Files

pycurl can also be used for downloading files, which is a common use case for cURL.

curl = pycurl.Curl()

with open('downloaded_file.zip', 'wb') as f:
    curl.setopt(curl.URL, 'https://example.com/file.zip')
    curl.setopt(curl.WRITEDATA, f)
    curl.perform()
    curl.close()

Conclusion

While requests is the go-to library for HTTP requests in Python, pycurl offers more power and flexibility, especially when you need fine control over your HTTP requests or when working with large files or low-level networking features.

Whether you’re sending custom headers, handling SSL certificates, or managing complex POST requests, pycurl gives you access to cURL’s rich feature set in Python, making it an excellent choice for more technical use cases.