# RagScrape API reference

Base URL: https://rag-scrape-api.owerryking.workers.dev

Auth: send your key as a bearer token on every scrape/crawl/llms-txt call.
Error shape (all failures): {"success":false,"error":{"code":…,"message":…}}

## POST /scrape

Turn any public URL into clean, LLM-ready Markdown. Optional: heading-aligned chunks, embeddings, token slimming, change detection.

Request (JSON):
- url (string, required) — http(s) URL
- chunk (boolean) — return chunks[] aligned to headings (code fences never split)
- chunkSize (number, 200–16000, default 4000) — target max chars per chunk
- embed (boolean) — implies chunk; each chunk gains embedding: number[384]
- stripLinks (boolean) — unwrap links, keep text (fewer tokens)
- stripImages (boolean) — drop image URLs, keep alt text
- ifNoneHash (string) — contentHash from a previous call; if the page is unchanged the response is {unchanged:true, contentHash} with no content

```json
POST /scrape {"url":"https://example.com/docs","chunk":true,"embed":true,"stripLinks":true}
```

Response 200: {success:true, markdown, metadata{title,byline,siteName,excerpt,wordCount,sourceUrl,scrapedAt,contentLength}, chunks?[{index,content,headingPath[],charCount,embedding?}], contentHash, unchanged?, embedError?}
Errors: 401 UNAUTHORIZED (bad key) · 402 RATE_LIMIT_EXCEEDED (quota, upgrade hint in message) · 403 KEY_INACTIVE / blocked URL · 429 DEMO_LIMIT_EXCEEDED · 400 validation · 504 timeout · 502 unreachable.
Rate-limit headers: X-RateLimit-Limit / -Remaining / -Reset.

## POST /crawl

Same-host BFS crawl of a documentation site. 1 quota unit per successfully crawled page. Hard cap 100 pages per call.
- url (string, required) · maxPages (1–100, default 20) · includePaths (string[] prefixes) · excludePaths (string[] prefixes)
Response 200: {success:true, pages:[{url,title,markdown,wordCount}], llmsTxt, stats{crawled,failed,requested}, errors[]}

## POST /llms-txt

Generate llms.txt for any site WITHOUT page payloads (cheap). Same input as /crawl. Response: {success:true, llmsTxt, stats, errors}.

## POST /register

Free API key, no card. {"email":"you@example.com"} → {success:true, apiKey:"rsk_…", tier:"free", limit:50}

## POST /create-checkout

Hosted subscription checkout (Starter $9 / Pro $19 / Unlimited $49). Body: {email, apiKey?, plan?}. Response: {success:true, checkoutUrl}. Payment upgrades the key automatically via webhook.

## GET /

Landing page. GET /convert — free no-signup converter (5 uses per IP). GET /health — liveness. GET /openapi.json — this API as OpenAPI 3.1.

## Plans

free: 50 req/mo · Starter $9: 2,000 req/mo · Pro $19: 10,000 req/mo · Unlimited $49: 1,000,000 req/mo. Monthly counters reset on the 1st.

## MCP server

```json
{"mcpServers":{"rag-scrape":{"command":"npx","args":["-y","rag-scrape","mcp"],"env":{"RAG_SCRAPE_API_KEY":"rsk_…"}}}}
```

Tools: rag_scrape {url, chunk?, chunk_size?}, rag_crawl {url, max_pages?, include_paths?, exclude_paths?}.