Site map (URL discovery)
POST /api/site-mapDiscover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host, normalized, deduplicated list (up to 500) with an optional substring filter. Send POST /api/site-map with the required field url and pay $0.005 per call over x402 or MPP (there is no free tier). It returns a JSON object with url, host, total, urls, sources and 7 more.
Hard budgets: at most 6 fetches, 15 seconds, 5 MB. Use it to pick which pages to crawl or extract next.
Parameters
| Name | Type | Required | Description |
|---|---|---|---|
url | string | yes | Start URL (the site's homepage or any page on it) Also accepted as link, uri, href, page. |
limit | integer | no | Max URLs to return, 1-500 (default 100) |
includeSubdomains | boolean | no | Also keep URLs on subdomains of the start site (default false; www and bare host always count as one site) |
search | string | no | Optional case-insensitive substring filter applied to the discovered URLs |
Example request
curl -i -X POST https://agent402.tools/api/site-map \
-H "Content-Type: application/json" \
-d '{"url":"https://agent402.tools","limit":50}'Without payment this returns HTTP 402 Payment Required with the exact price for site-map; any x402 v2 or MPP client pays it and retries.
Watch a call get paid
POST /api/site-map$0.005 USDCPAYMENT-SIGNATUREon 14 chains200 OKAgent402-ReceiptEd25519Example response
{
"url": "https://agent402.tools/",
"host": "agent402.tools",
"total": 1106,
"urls": [
"https://agent402.tools/",
"https://agent402.tools/decide",
"https://agent402.tools/reports",
"https://agent402.tools/monitors"
],
"sources": {
"sitemap": 1028,
"links": 78
},
"sitemapsRead": 4,
"truncated": true,
"search": null,
"fetches": 6,
"warnings": [],
"source": "robots.txt, sitemap(s) and start-page links, fetched live",
"fetchedAt": "2026-08-22T00:00:00.000Z"
}| Field | Type | Always present | In the example |
|---|---|---|---|
url | string | yes | https://agent402.tools/ |
host | string | yes | agent402.tools |
total | number | yes | 1106 |
urls | array of string | yes | 4 items in the example |
sources | object | yes | 2 fields: sitemap, links |
sitemapsRead | number | yes | 4 |
truncated | boolean | yes | true |
search | null | no | null |
fetches | number | yes | 6 |
warnings | array | yes | 0 items in the example |
source | string | yes | robots.txt, sitemap(s) and start-page links, fetched live |
fetchedAt | string | yes | 2026-08-22T00:00:00.000Z |
From an MCP client
catalog.call {
"slug": "site-map",
"params": {
"url": "https://agent402.tools",
"limit": 50
}
}The hosted connector at https://agent402.tools/mcp needs a payment for site-map; the stdio package pays it from a wallet or from an AGENT402_CREDITS_KEY issued earlier. Local install: npx -y agent402-mcp@0.14.0.
Errors and behavior
urlis required. An input the tool rejects returns an HTTP 4xx whose body carrieserror,tool,expected,requiredandexample, so the caller can correct it.- A paid call that ends in any status of 400 or above is not charged over x402, MPP or a prepaid credits key: settlement runs after the tool and only on a successful response. The exception is a Tempo push credential, a transfer the buyer sent before the call: it settles before the tool runs, so if the tool then fails the payment is recorded as a refund owed to the paying wallet.
- Wallet-only: this tool reaches the network or stored state, so it has no proof-of-work tier. A prepaid card-credits key issued earlier (
Authorization: Bearer a402_...) also pays it. - A
GETorHEADto /api/site-map returns the same 402 quote, so the price can be read without a body. - An
Idempotency-Keyheader makes a retried paid call replay the first 200 instead of charging again (an answer larger than 1 MB is not replayed).
Paid call (JavaScript agent)
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";
const client = new x402Client();
client.setSpendControls?.({ maxAmountPerPayment: "$0.005" }); // refuse any quote above this tool's price
registerExactEvmScheme(client, { signer: privateKeyToAccount(KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);
const res = await payFetch("https://agent402.tools/api/site-map", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
"url": "https://agent402.tools",
"limit": 50
}),
});Related tools
Site crawl (pages to markdown)
Crawl a website breadth-first from a start URL over its internal links and return each page as clean markdown (or plain.
Sitemap reader
Fetch and parse a sitemap.xml (or sitemap index): returns up to 500 URLs with lastmod, or the child sitemaps.
Wayback Machine snapshot
Look up a URL in the Internet Archive's Wayback Machine: returns the archived snapshot closest to an optional timestamp.
Site favicon grabber
Fetch a site's favicon/logo: reads the page's declared <link rel=icon…> tags (all variants with their sizes and types).
Farcaster trending channels
Trending Farcaster channels ranked by casting activity over 1d, 7d or 30d, with cast counts for all three windows.
RSS/Atom feed parser
Fetch an RSS 2.0, Atom, or RSS 1.0 feed URL and parse it into normalized JSON: feed title/description/link plus items.