docs

Architecture, the web-search upstreams, crawling, the HTTP API, and how to deploy your own copy. The full guides live in the repo’s docs/ folder.

architecture

Three pieces on Cloudflare:

The worker (src/index.ts) — Hono app, D1 database, the crawler, cron triggers (twice daily), the merging engine for web search, and the AI (Groq) endpoints. It serves /api/* only.
Pages — main site (pages/static) — the public search UI. A single Pages Function, functions/api/[[path]].ts, proxies every /api/* request to the worker, so the UI is plain static files with zero CORS headaches.
Pages — console (sitesearch-console, separate project) — console-sitesearch.pages.dev: sign in with the admin token, manage sites, trigger crawls, browse pages, edit runtime settings, create/revoke API keys.

Full guide: docs/architecture.md · console walkthrough: docs/console.md

web search upstreams

Upstream	Key?	Notes
DuckDuckGo (HTML)	no	HTML scrape of `html.duckduckgo.com`, falls back to `lite.duckduckgo.com`. Skip if bot-checked.
Wikipedia (MediaWiki API)	no	Always available; grounds results with real encyclopedia facts.
Internet Archive	no	Full-text search over books, texts and captures. Always available.
Google Books	no	Book corpus; keyless tier is rate-limited — opt-in only.
Brave Search API	yes	Optional `BRAVE_API_KEY` adds a commercial-grade independent index.

Configure with the UPSTREAMS var. Full guide: docs/upstreams.md

crawling “your sites”

From the console, add a site with a sitemap.xml URL and/or seed links. The crawler discovers sitemap-index files, follows same-domain links, respects robots.txt and per-site include/exclude patterns, re-crawls stale pages, and dedupes by content hash. Text lands in a D1/SQLite FTS5 index; search is BM25-ranked with highlighted snippets.

Full guide: docs/deployment.md

http api

Endpoint	Auth	Purpose
`GET /api/web?q=&page=`	— / key	Aggregated web search
`GET /api/ai/suggest?q=`	— / key	AI query suggestions (Groq)
`GET /api/ai/answer?q=`	— / key	AI answer citing web + your crawl index
`GET /api/search?q=&site=&page=`	— / key	Search your crawl index
`GET /api/sites`	— / key	Indexed-site list for the filter
`GET /api/health`	—	Liveness
`/api/admin/*`	token	Sites, crawls, runs, pages, stats, settings, keys

By default the API is open; set the api_auth setting to keys to require an API key (create them in the console). Admin auth: Authorization: Bearer <ADMIN_TOKEN>. Full reference: docs/api.md · keys guide: docs/api-keys.md · settings: docs/settings.md

deploy your own

npm install
npx wrangler d1 create sitesearch        # paste database_id into wrangler.toml
npx wrangler secret put ADMIN_TOKEN
npm run db:migrate
npm run deploy                            # worker + cron

npx wrangler pages project create sitesearch --production-branch main
npx wrangler pages deploy pages/static --project-name sitesearch --commit-dirty

Done — yours lives at *.pages.dev. Full guide: docs/deployment.md

← back to search