Large Scale Web Searching with Agents
Challenges & Solutions
01
Searching through HTML for info wastes many tokens.
1.1
HTML-strip function tuned to common domain patterns — keeps only essential content.
1.2
Sub-agent hierarchy splits work between Sonnet (cheap) and Opus (reasoning).
02
Consistent results across hundreds of sites.
2.1
Did the first 100 by hand to spot recurring patterns → wrote per-family .md guides the agent uses to classify each site and follow family-specific instructions.
2.2
Custom MCP functions centralise flags and shared logic so every agent behaves identically.
03
Long-running tasks fill up the agent's context.
3.1
Delegate to sub-agents — each one handles a single focused task so the parent's context stays small.
04
Too many websites to implement.
4.1
Scheduled overnight queries to maximise 5-hour usage windows.
4.2
Harness fans out sub-agents in parallel, one per website.