Dedupe lines

Remove duplicate lines, preserving first-seen order. Returns the deduped text and how many were removed.

Input

Field	Type	Description
text *	string
ci	boolean	Case-insensitive

Example output

{
  "result": "a\nb\nc",
  "removed": 2,
  "kept": 3
}

Try it — see the 402 challenge (free)

curl -i -X POST https://agent402.tools/api/dedupe-lines \
  -H "Content-Type: application/json" \
  -d '{"text":"a\nb\na\nc\nb"}'

The response is HTTP 402 Payment Required with exact payment requirements. Any x402 v2 client pays automatically and retries:

Paid call (JavaScript agent)

import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client();
registerExactEvmScheme(client, { signer: privateKeyToAccount(KEY) });
const payFetch = wrapFetchWithPayment(fetch, client);

const res = await payFetch("https://agent402.tools/api/dedupe-lines", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    "text": "a\nb\na\nc\nb"
  }),
});

No wallet? Pay with compute

This is a pure-CPU tool, so an agent without a wallet can pay with proof-of-work instead of USDC: fetch a challenge, solve it (16 leading zero bits), and resend with the X-Pow-Solution header.

import { createHash } from "node:crypto";
const lz = (b) => { let t = 0; for (const x of b) { if (!x) { t += 8; continue; } t += Math.clz32(x) - 24; break; } return t; };
const c = await (await fetch("https://agent402.tools/api/pow/challenge?slug=dedupe-lines")).json();
let n = 0;
while (lz(createHash("sha256").update(c.challenge + ":" + n).digest()) < c.difficulty) n++;
await fetch("https://agent402.tools/api/dedupe-lines", { method: "POST", headers: { "X-Pow-Solution": c.token + ":" + n, "Content-Type": "application/json" }, body: JSON.stringify({"text":"a\nb\na\nc\nb"}) });

Related tools