Authenticate Web Data Tools for AI Agents Without Exposing Secrets

An AI agent needs credentials to call a web data API. The hard part is keeping those credentials away from the agent itself.

Anything the model can see, it can repeat. A key in a system prompt can show up in an answer. A key in browser code can be copied by anyone who opens the developer tools. A key in a log file can be read by anyone with log access.

Scraped pages add another risk. A web page can contain text that tries to get the agent to reveal its instructions or secrets. If the model never has the key, it cannot leak it.

This guide shows how to set up authentication so the agent can use web data tools without ever seeing a secret.

Two ways to connect

Most teams use one of two patterns:

1. Your backend owns the key. Your server holds a MrScraper API token. The agent calls your tools, and your server calls MrScraper.
2. Each user connects their own account. A person signs in to MrScraper through their MCP client, such as Claude Code or Codex. The client handles the sign-in, and the agent uses that person's access. The next section covers when to use each one.

When should you use API keys or OAuth?

API keys for your backend

MrScraper API tokens are created in the dashboard under API Tokens. Each token has a name and an expiration date. You can create more than one, so you can give each environment or app its own token. See the API token docs for the steps.

The REST API reads the token from the x-api-token header. The Scraper API examples also pass it as a token query parameter. Your server sends both. The agent never sees either.

OAuth for people using MCP clients

The hosted MrScraper MCP server at https://mcp.mrscraper.com/mcp supports OAuth 2.1. The client opens a browser, the person signs in, and no key is copied anywhere.

To add it in Claude Code:

claude mcp add --transport http --scope user mrscraper https://mcp.mrscraper.com/mcp

To add it in Codex:

codex mcp add mrscraper --url https://mcp.mrscraper.com/mcp

OAuth access uses scopes:

• scrape:read covers the scraperAPI and webSearch tools.
• account:read covers the status tool. If a client calls a tool without the right scope, the server refuses it with a 403 insufficient_scope error. Grant only the scopes a person needs.

When a client does not support OAuth

Some MCP clients cannot do the browser sign-in. For those, the MCP server accepts an API key as a bearer token in the Authorization header. If you run it locally over stdio, it reads the key from the MRSCRAPER_API_KEY environment variable.

Point the client at the variable name, not the value, when it lets you. Codex supports this with a flag:

codex mcp add mrscraper --url https://mcp.mrscraper.com/mcp \
  --bearer-token-env-var MRSCRAPER_API_KEY

If a client config file must hold the key, keep that file out of version control and readable only by you.

Where should credentials live?

Keep credentials on the server, in one place, loaded once.

Safe places

• An environment variable on your server
• Your cloud provider's secrets manager
• Your CI system's secret store, for batch jobs

Unsafe places

• System prompts, user messages or tool results
• Tool input schemas, where the model would fill in the key
• Browser code, including front-end environment variables that get bundled into the page
• Your git repository
• Logs, error messages and analytics events MrScraper's docs say the same thing: never expose your API token in client-side code.

Load the key once and keep it out of tool inputs

Read the key when the server starts. If it is missing, fail right away. Your tool functions use it directly, so it never appears in a tool schema or in anything the model sends.

import os
import requests
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ.get("MRSCRAPER_API_TOKEN")
if not TOKEN:
    raise RuntimeError("MRSCRAPER_API_TOKEN is not set")
 
 
def fetch_markdown(url: str) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, "markdown": "true"},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={"url": url, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    return response.json()

Notice that fetch_markdown takes only a URL. There is no api_key argument for the model to fill in.

Scrub secrets from anything that leaves the server

Because the Scraper API examples put the token in the URL, a raw request URL can contain your key. Error messages from HTTP libraries often include that URL. So scrub every string before it goes to the model, a log or the browser.

import re
 
 
def scrub(text: str) -> str:
    text = text.replace(TOKEN, "[REDACTED]")
    return re.sub(r"([?&]token=)[^&\s]+", r"\1[REDACTED]", text)
 
 
def safe_error(error: Exception) -> str:
    return scrub(str(error))

Use safe_error whenever you turn an exception into a tool result or a log line. Also check that your web server, load balancer and monitoring tools do not log full query strings for these requests.

Put a tool gateway between the browser and the API

If your agent runs in a web app, the browser should talk to your server, never to MrScraper. Your server checks who the user is, then runs the tool.

Browser  ->  Your server (checks user, runs tool)  ->  MrScraper API
            holds MRSCRAPER_API_TOKEN

The browser sends its own session cookie or token to your server. Your server holds the MrScraper token. These are two different credentials, and that difference is the key to the next section.

How do you separate user and service access?

There are two kinds of access in an agent app:

• User access answers "who is asking?" Your app's own login handles this.
• Service access answers "what is allowed to call MrScraper?" Your server's MrScraper token handles this. Never pass the service token to a user, and never let a user's request choose which token to use.

Decide what each user can do

Your app should decide which tools each user can run and how much they can use. MrScraper sees one token. Your app knows which person is behind each call.

ROLE_TOOLS = {
    "viewer": {"fetch_markdown"},
    "analyst": {"fetch_markdown", "extract_page"},
}
DAILY_TOKEN_LIMIT = {"viewer": 200, "analyst": 2000}
 
 
class AccessDenied(Exception):
    pass
 
 
def authorize(user: dict, tool: str, used_today: int) -> None:
    if tool not in ROLE_TOOLS.get(user["role"], set()):
        raise AccessDenied(f"{user['role']} cannot use {tool}")
    if used_today >= DAILY_TOKEN_LIMIT[user["role"]]:
        raise AccessDenied("Daily usage limit reached")

Each MrScraper response reports token_usage. Add it to the user's daily total after every call. That way one user cannot use up the whole account.

Use separate tokens for separate jobs

Create one MrScraper token for each environment and each app. For example:

• prod-research-agent
• staging-research-agent
• nightly-batch-jobs This gives you three benefits. A leak in staging does not affect production. Each token's name shows up in your audit log. And you can replace one token without touching the others.

Set an expiration date on each token, and plan the swap before it expires. Create the new token, deploy it, check that calls succeed, and then retire the old one.

When users bring their own accounts

Sometimes each user should use their own MrScraper account. In that case, skip the shared service token. Have each person connect through the hosted MCP server with OAuth. Their usage stays on their account, and the scopes limit what the client can do.

How do you audit tool calls?

An audit log answers "who called which tool, on what, and what happened?" It is how you spot misuse, debug failures and explain costs.

Log what helps, skip what hurts

Wrap every tool call

This wrapper checks access, runs the tool and writes one JSON line per call.

import hashlib
import json
import time
import uuid
from urllib.parse import urlparse
 
TOKEN_NAME = os.environ.get("MRSCRAPER_TOKEN_NAME", "unnamed")
 
 
def audited_call(user: dict, tool: str, fn, url: str, used_today: int, **kwargs):
    entry = {
        "time": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "request_id": uuid.uuid4().hex,
        "user_id": user["id"],
        "tool": tool,
        "target": urlparse(url).netloc + urlparse(url).path,
        "args_hash": hashlib.sha256(
            json.dumps({"url": url, **kwargs}, sort_keys=True).encode()
        ).hexdigest()[:16],
        "token_name": TOKEN_NAME,
    }
    try:
        authorize(user, tool, used_today)
        result = fn(url, **kwargs)
        entry.update(status="ok", token_usage=result.get("token_usage"),
                     runtime=result.get("runtime"))
        return result
    except AccessDenied as e:
        entry.update(status="denied", error=str(e))
        raise
    except requests.HTTPError as e:
        code = e.response.status_code if e.response is not None else None
        entry.update(status="error", http_status=code, error=safe_error(e))
        raise
    finally:
        with open("tool_audit.jsonl", "a") as log:
            log.write(json.dumps(entry) + "\n")

The finally block writes the entry whether the call worked, failed or was denied.

Review the log

Check for these patterns on a regular schedule:

• 401 errors. A token is missing, wrong or expired. Our guide to 401 Unauthorized vs 403 Forbidden explains what each code means.
• Usage spikes. One user or one token suddenly uses far more than normal.
• New targets. The agent starts calling hosts your team has not approved.
• Many denied calls. A user, or a prompt, keeps trying tools it should not use.

FAQs

When should you use API keys or OAuth?

Use an API key when your own server calls MrScraper for your app, such as a backend tool or a batch job. Use OAuth when a person connects their own MrScraper account through an MCP client. The hosted MrScraper MCP server supports OAuth 2.1 sign-in with scopes.

How do you audit tool calls?

Wrap every tool call in code that writes one log entry. Record the time, user, tool, target host and path, an argument hash, the result and the usage. Never record the token or full URLs with query strings. Review the log for 401 errors, usage spikes and unexpected targets.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. It provides named API tokens with expiration dates, a REST API that reads the token from a request header, and a hosted MCP server with OAuth 2.1 sign-in and scopes. Each response reports token usage and runtime.

Your application owns everything around that. This includes where the key is stored, the tool gateway, user login, roles and limits, scrubbing secrets from errors and logs, the audit log and token swaps. Your team also decides which sources it has permission to collect from.