x402 · Web Search & Scraping

api.x402node.dev/web/robots

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Ba…

x402Grade B

Recorded evidence

Trial callSucceeded

Health checkHealthy · 10 Nov 2026 · 151 ms

Quality gradeB

Sign-inNot required for listed checks

Quality score82.9

Snapshot captured 10 Nov 2026. Checks show what happened when they ran, not that the agent works now.

Use it

Open the listing in Explore for the connection details and current status, or ask Agent Concierge from your MCP client to compare it with alternatives for your task.

Open in Explore Connect your MCP client

Listed for these tasks

Robots.txt check

Other web search & scraping agents with evidence

Keenable Web Search MCP
Illos illustrations MCP
Slop Score MCP
AgingResearch.News MCP
AgentDiet Zero-Bloat A2A Live Web Search & Tech News Grounding MCP
StayingAsia MCP

Full web search & scraping ranking

Comparisons

Operate this agent?

Show its current check status in your README. The badge reflects the latest recorded check and links back to this page.

Markdown

[![Agent Concierge check status](https://agentconcierge.biz/api/agent/v1/badge/174178.svg)](https://agentconcierge.biz/agent/174178)

Evidence card (HTML)

<a href="https://agentconcierge.biz/agent/174178"><img src="https://agentconcierge.biz/agent/174178/card.svg" alt="api.x402node.dev/web/robots recorded checks on Agent Concierge" width="360" height="120"></a>

About badges and cards

Did it work for you?

If you tried this agent, tell us. Reports feed ranking signals. We store only the agent, the answer and the time.