Site map (URL discovery)

Discover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host, normalized, deduplicated list (up to 500) with an optional substring filter. Hard budgets: at most 6 fetches, 15 seconds, 5 MB. Use it to pick which pages to crawl or extract next.

TRY IN PLAYGROUND →

Input

Example output

{
  "url": "https://www.iana.org/",
  "host": "www.iana.org",
  "total": 120,
  "urls": [
    "https://www.iana.org/",
    "https://www.iana.org/domains",
    "https://www.iana.org/numbers",
    "https://www.iana.org/protocols"
  ],
  "sources": {
    "sitemap": 96,
    "links": 24
  },
  "sitemapsRead": 1,
  "truncated": true,
  "search": null,
  "fetches": 3,
  "warnings": [],
  "source": "robots.txt, sitemap(s) and start-page links, fetched live",
  "fetchedAt": "2026-08-22T00:00:00.000Z"
}

Try it - see the 402 challenge (free)

curl -i -X POST https://agent402.tools/api/site-map \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.iana.org","limit":50}'

The response is HTTP 402 Payment Required with exact payment requirements. Any x402 v2 client pays automatically and retries:

Paid call (JavaScript agent)

import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
registerExactEvmScheme(client, { signer: privateKeyToAccount(KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agent402.tools/api/site-map", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "url": "https://www.iana.org",
    "limit": 50
  }),
});

Related tools