Cache Busting

Cache busting is any technique that makes a browser, proxy or CDN fetch a newer copy of a resource instead of reusing a cached one. Its classic use is in web development: renaming app.js to app.3f9a1c.js when the file changes, so browsers download the new version.

In web data collection, the same question comes up when a cached page might show an old price, stock level or headline. Cache busting affects which cache answers your request. It can't guarantee the data is current, because the origin server decides what it returns.

How HTTP caching works

Websites tell caches how to treat each response through headers the site operator sets:

• Cache-Control (max-age, s-maxage, no-store, private) says whether a response can be reused and for how long.
• Age is the cache's estimate, in seconds, of how long ago the response was generated or last validated at the origin.
• ETag and Last-Modified identify a version, so a client can later ask whether it changed.
• Cache-status headers such as X-Cache or CF-Cache-Status often report a hit or miss. Names and values vary by provider. Because operators configure these, two sites can treat the same kind of request very differently.

Common techniques and their limits

Validators: the lighter way to check freshness

Conditional requests confirm whether content changed without downloading it again when it hasn't. Save the ETag or Last-Modified value from a response, send it back as If-None-Match or If-Modified-Since next time, and treat a 304 as "your copy is still current."

This saves bandwidth for you and work for the site, so it's usually a better first step than forcing a full fetch.

Tradeoffs

• Origin load. Every skipped cache hit is a request the origin has to serve. Forcing misses at volume adds cost for the site operator and can trigger rate limits.
• Changed responses. Extra parameters can alter what a server returns, or break the page.
• No end-to-end guarantee. Reaching the origin still returns whatever the origin currently has, which may itself come from an internal cache or a delayed feed.
• Slower, costlier runs. Uncached fetches take longer, and paid collection usually costs per request.

Checking freshness in practice

• Inspect Age, Date, Last-Modified and any cache-status header on the response.
• Store a fetch timestamp with every record so downstream users know how old the data is.
• Refresh at the rate the data actually changes, not as fast as possible.
• Spot-check a sample against a source you trust when accuracy matters. With MrScraper, you can schedule collection runs to keep datasets up to date, and each Web Fetching response includes the target site's status code in an x-status-code header, so you can spot failed or redirected fetches.

Frequently asked questions

Does cache busting guarantee fresh data?

No. It can steer a request past a cache, but the origin server decides what content it returns.

Why didn't my cache-busting parameter work?

The site's CDN may be configured to ignore query strings, or to ignore request Cache-Control headers. Check the response's cache-status and Age headers.

Is it the same as clearing my browser cache?

No. Clearing a browser cache only removes local copies. CDN and server caches are unaffected.