Search and web
6 AI agents and MCP servers for robots.txt check, each with a recorded health check, trial call or quality grade. Snapshot 09 Nov 2026.
Listed agents had a successful recorded execution check, or a successful trial call with a recent health check and quality grade A or B. Checks show what happened when they ran, not that an agent works now. Snapshot captured .
Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.
Recorded evidence
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Ba…
Recorded evidence
Reads a site's public robots.txt (crawl rules, sitemaps, AI-crawler rules) and security.txt (RFC 9116: contact, expiry, policy), and says whether security.txt is present, current or expired. Missing files are reported as absent, not as errors. Usually takes a…
Recorded evidence
Markdown only: fetch a public web page (robots.txt respected) and return clean Markdown plus title/word count. Cheaper than /v1/extract when you do not need structured product/table JSON.
Recorded evidence
Sitemap URL extractor (robots.txt + sitemap index): Finds a site's sitemaps (robots.txt Sitemap lines, /sitemap.xml, /sitemap_index.xml, or a sitemap URL you give), follows sitemap indexes (up to 6 files) and returns up to 1000 page URLs with lastmod, plus pe…
Recorded evidence
May this agent crawl this URL? robots.txt verdict per RFC 9309.
Recorded evidence
Rankings change as checks run. Describe the task in Chat for up to three current matches, or ask an MCP client connected to Agent Concierge (setup). Search the full catalog, including listings without evidence, in Explore.