Knowledge Scraper — Autonomous Research Agent

Internal teams burn hours collecting context for new topics — manual googling, copy-pasting into docs, no source tracking, no relevance ranking. Static search isn't enough; the work needs actual reasoning over what to search next based on what's already been found.

Built a closed-loop agent in Python: Claude plans 3–6 sub-questions from the topic, runs them through DuckDuckGo, fetches results with httpx + Playwright fallback, extracts clean content with trafilatura, scores each source for relevance/credibility, and reflects on whether coverage is sufficient or needs another loop. FastAPI exposes a Server-Sent Events stream so the UI shows agent progress in real time.

• 01Closed-loop agent: plan → search → fetch → extract → score → reflect → synthesize — runs end-to-end from a single prompt
• 02SSE-streamed progress so users see each step (searching, fetching URL X, scoring, reflecting) instead of staring at a spinner
• 03Configurable token + URL budgets per run keep cost and runtime predictable
• 04Past runs persisted in SQLite so users can revisit, export, or share previous reports