If you’ve noticed a spike of 404 errors for files like ads.txt, security.txt, sellers.json, or various /.well-known/ paths, the culprit is likely the WebPageTest.org bot. Unlike simple crawlers, WebPageTest uses real Chromium browsers to test websites, which means JavaScript executes and your error monitoring captures the failed network requests.
These errors don’t indicate problems with your website. They’re the result of WebPageTest probing for standard configuration files that most websites don’t have.
Important: These errors come from an external testing service, not from your users or your application code.
The Problem
WebPageTest.org is a popular web performance testing tool that uses real browsers to analyze websites. When WebPageTest (or services built on it, like HTTP Archive) scans your site, it checks for various well-known configuration files. If those files don’t exist, the requests return 404 errors.
Because WebPageTest runs in a real Chromium browser, your JavaScript error monitoring picks up these failed network requests just like it would for any other browser. You’ll see 404 errors like:
GET /dsrdelete.jsonGET /ads.txtGET /sellers.jsonGET /.well-known/security.txtGET /.well-known/apple-app-site-association
Key point: These are bot-generated requests checking for files your site may legitimately not have. The 404 response is correct behavior.
Understanding the Root Cause
WebPageTest bot errors appear in your JavaScript monitoring for a specific reason:
Real Browser Execution
Most web crawlers make simple HTTP requests without executing JavaScript. WebPageTest is different. It launches actual Chromium browser instances to render pages, run JavaScript, and measure real-world performance. This means:
- Your analytics and error monitoring scripts execute
- Network requests (including failed ones) are visible to client-side tools
- TrackJS captures these 404 errors because they happen in a real browser context
What Files Does WebPageTest Check?
WebPageTest probes for numerous standard configuration files:
Advertising Files:
/ads.txt- Authorized Digital Sellers/app-ads.txt- Authorized Sellers for Apps/sellers.json- Supply chain transparency/dsrdelete.json- IAB Data Deletion Request Framework
Security Files:
/.well-known/security.txt- Security contact information/.well-known/change-password- Password change URL
Authentication and Identity:
/.well-known/passkey-endpoints- Passkey configuration/.well-known/webauthn- WebAuthn configuration/.well-known/web-identity- Federated identity
Mobile App Integration:
/.well-known/apple-app-site-association- iOS Universal Links/.well-known/assetlinks.json- Android App Links
Privacy and Platform:
/.well-known/gpc.json- Global Privacy Control/.well-known/privacy-sandbox-attestations.json- Chrome Privacy Sandbox/.well-known/related-website-set.json- Related Website Sets
Other:
/llms.txt- LLM crawler instructions/.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200/- HTTP compliance test
If you’re seeing 404 errors for several of these files around the same time, WebPageTest is almost certainly the source.
Who Uses WebPageTest?
Beyond individual developers testing their sites, WebPageTest infrastructure powers larger projects:
- HTTP Archive runs monthly crawls of millions of websites using a private WebPageTest instance
- Research projects use WebPageTest for web standards adoption studies
- Enterprise monitoring services build on WebPageTest’s capabilities
This means your site might be scanned by WebPageTest even if you’ve never used it directly.
How to Fix WebPageTest Bot Errors
Since these errors come from external bots rather than your users, the solution is filtering them from your monitoring.
Step 1: Identify the Pattern
Check your error timeline for clusters of 404 errors affecting multiple well-known files. If you see a burst of errors for ads.txt, security.txt, and similar files within a short window, that’s the signature of a WebPageTest scan.
Step 2: Filter Bot Traffic
In TrackJS, you can filter out WebPageTest bot errors using ignore rules. If you don’t use TrackJS, you can filter it by user-agent string, but it’s not at all obvious that this is webpagetest:
Mozilla/5.0 (Linux; Android 8.1.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Mobile Safari/537.36 PTST/251202.154650
You can also use the broader bot filtering approach described in our search engine spiders and bots guide.
Step 3: Consider Which Files You Actually Need
Before dismissing all these 404s, consider whether you should implement any of these files:
Recommended for most sites:
/.well-known/security.txt- Helps security researchers contact you responsibly
Required for specific use cases:
ads.txt/sellers.json- If you run programmatic advertising/.well-known/apple-app-site-association- If you have an iOS app with Universal Links/.well-known/assetlinks.json- If you have an Android app with App Links
Optional but useful:
/.well-known/change-password- Helps password managers direct users to your password reset page/.well-known/gpc.json- If you want to honor Global Privacy Control signals
For most websites, the majority of these files aren’t needed, and 404 responses are perfectly appropriate.
When to Ignore These Errors
WebPageTest bot errors should almost always be ignored because:
- Bot traffic: These requests come from automated testing, not real users
- No user impact: Your actual visitors aren’t requesting these files
- Correct behavior: Returning 404 for files you don’t have is the right response
- Temporary spikes: Bot scans produce bursts of errors, not ongoing issues
However, pay attention if:
- Errors persist continuously: Might indicate something other than WebPageTest
- You need these files: If you run ads or have mobile apps, missing files could be a real problem
- Patterns change: Sudden increases might indicate new scanning activity worth understanding
Summary
WebPageTest.org bot errors appear in your JavaScript monitoring because WebPageTest uses real Chromium browsers that execute your tracking scripts. The bot probes for standard configuration files like ads.txt, security.txt, and various /.well-known/ paths. When these files don’t exist, you see 404 errors in your monitoring.
The fix is straightforward: filter WebPageTest bot traffic from your error monitoring using user agent or URL-based ignore rules. These errors don’t affect your users and don’t indicate problems with your site.
Remember: Bots using real browsers are increasingly common. When you see bursts of 404 errors for well-known configuration files, check the user agent before assuming something is broken.
