Robots.txt check

Fetch a site's robots.txt and answer: may this user-agent crawl this path? Returns the matched rule and all declared sitemaps.

Input

Field	Type	Description
url *	string	URL whose path to check
userAgent	string	User-agent token (default *)

Example output

{
  "allowed": true,
  "matchedRule": null,
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ]
}

Try it — see the 402 challenge (free)

curl -i -X POST https://agent402.tools/api/robots-check \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/some/page","userAgent":"MyAgent"}'

The response is HTTP 402 Payment Required with exact payment requirements. Any x402 v2 client pays automatically and retries:

Paid call (JavaScript agent)

import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
registerExactEvmScheme(client, { signer: privateKeyToAccount(KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agent402.tools/api/robots-check", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "url": "https://example.com/some/page",
    "userAgent": "MyAgent"
  }),
});

Related tools