Our crawler.

Agent402 publishes an index of services that advertise machine payments over x402 or MPP. For each x402 origin in it, the crawler reads /robots.txt, the x402 manifest at /.well-known/x402, and /openapi.json to name and describe the routes. When an origin publishes no manifest, it also tries /agents.json and then /llms.txt, two other places a service may list its paid routes.

It sends an unpaid request to paid routes the index lists, to read the price and payment networks in their 402 challenge. When one origin claims to have moved from another, it reads /.well-known/agent402-succession on both to confirm the claim. For the MPP index, it reads a registered server's /openapi.json and sends one unpaid request to a paid endpoint to confirm its WWW-Authenticate: Payment challenge.

We do not log in, submit forms, or pay to see more. A 402 challenge is a public advertisement of price and payment rails, and asking for one with no credential is the ordinary way to read it.

The crawler identifies itself on every request as Mozilla/5.0 (compatible; Agent402/1.0; +https://agent402.tools/crawler), so it can be recognised, rate-limited or blocked by name. robots.txt rules are matched on the product token Agent402.

It honours robots.txt, with these exceptions. The x402 manifest at /.well-known/x402 is read whatever robots.txt says: it is a protocol endpoint published to be fetched by payment clients. Once an origin publishes that manifest, /openapi.json is read under a rule written for every crawler (User-agent: *); a rule that names Agent402 still stops it. The unpaid request to a paid route, and the MPP index's two reads, are not robots-gated: they read the price a route quotes to every caller. /agents.json and /llms.txt always follow robots.txt, and a robots.txt we cannot fetch or parse does not block the crawl.

It re-reads an origin every 30 minutes, backs off on errors, sends conditional requests so an unchanged document costs a single 304, and caps what it will read from any one response.

If our crawler is causing you a problem, email us and we will stop it. A Disallow for Agent402 stops every read above except the manifest and the unpaid price check, and needs no message from us.

Some tools in our catalog fetch a URL that a caller names: reading an article, checking a page's headers or metadata. Most of those requests send the crawler User-Agent above; the browser-render and screenshot tools load the page in a headless browser with a standard browser User-Agent. Each is one request made for one caller, not a crawl, and is not robots-gated. The multi-page site-crawl tool does read robots.txt and follows the rules for the token Agent402Bot, which a rule naming Agent402 also matches.

A crawl and a purchase are different acts and this page is linked from both, so it should say so. When our router buys from a service to resell the result to a caller it sends a real HTTP 402 payment, which is not a crawl: it is a customer.

Those requests identify themselves as Mozilla/5.0 (compatible; Agent402-Router/1.0; +https://agent402.tools/crawler) and carry X-Agent402-Via: router. Either one separates a purchase from a probe in your logs.

robots.txt does not govern these. A purchase is a request for a service you advertised a price for, and declining one is a matter for your own endpoint, not for a crawl directive. If you would rather we did not buy from you at all, say so and we will stop.

Email mike@agent402.tools from an address at the domain, or open an issue, and we will remove the origin from the index and stop crawling it. We do not require a reason and we do not argue about it.

A robots.txt rule limits what the crawler reads from the next cycle on, but it does not remove a listing: the row stays with what we last read. To be removed, email us.

The public index carries what a service advertises about itself: its endpoints, the prices and payment networks it declares, whether our own probe could reach it, and the payout addresses it names in its own challenges. Payout addresses are public infrastructure - they appear in every challenge the service hands to every buyer.

We publish counts of settled payments per payee, read from public chains, and buyer figures in the index are counts. Payments on public chains are public: some of our pages show individual payments as the chain records them, for example the live view on our home page, which shows shortened payer addresses with links to the public transactions. We do not sell payer data.

Measurements made by third parties are not redistributed. Where our pages display someone else's figures we label them as theirs, and they are excluded by name from any dataset we distribute.

If a row about your service is wrong, tell us and we will fix it. Several improvements to how prices are read came from operators who checked their own row and found it stale - that feedback loop is the main reason the index is accurate, and it is why the index is free to read.

If you index this catalog, three documents carry every route and its price: /.well-known/x402, /openapi.json and /api/pricing. Read those rather than requesting each priced route to learn what it costs. /openapi.json is served with an ETag, so a repeat read that sends If-None-Match is a 304 with no body.

Listed prices change when we ship a release, not from one request to the next, so a full walk of every priced route more than a few times an hour is not needed.

Unpaid requests to priced routes are budgeted per client, meaning one address and one User-Agent product token. Past 3,000 in an hour, the answer is a 429 with a Retry-After header that counts down to the top of the hour. A request that carries a payment, a credits key or a proof-of-work solution is never counted, and neither are the three documents above, /mcp or any page. Every purchase opens with one bare request, so buying is exempt from the first settlement onward: once a client settles a payment, nothing it sends is counted for the rest of that hour.

Named indexers, search engines and AI crawlers are not budgeted. If your index needs more than this and we do not recognize it by name, email mike@agent402.tools with the User-Agent it sends and we will add it to the named list.