Common Crawl

New Capabilities

Data source for the scan

When a browser asks a website for a page, the server answers with a string of small header fields before the real content. Browsers show readers none of them. The threat research firm Signals Corps found that thousands of websites are now using those invisible fields to talk directly to AI agents.

Updated Oct 15