SnagaBot

The crawler of Snaga, an independent web search for AI agents.

If you see SnagaBot in your logs, it is us: Snaga, run by Squad AI Inc. We build our own index of the web so that AI agents can find pages. Each result links to your page and shows what your page says about itself. We don't sell your content: agents follow the link to read it.

The user agent is:

Mozilla/5.0 (compatible; SnagaBot/0.1.0; +https://snaga.ai/bot)

What it fetches

When	What	How often
Crawling, to keep our index	`/robots.txt`, then HTML pages it allows. Other files (PDFs, images, video) are skipped.	One request at a time per host, at least a second apart. Your `Crawl-delay` is used when it's longer.
Fetching a page an agent asked for	That one page, after checking your `/robots.txt`.	At most one request per second per host. A page is fetched again no sooner than 15 minutes later.

We don't log in, accept cookies, fill in forms, solve CAPTCHAs or get around paywalls or blocks. We don't read more than 2 MB of a page when crawling, or 5 MB when an agent asks for it.

A page is fetched on demand when an agent asks for it, and only then: through POST /api/web_fetch, or through the web_fetch tool of our Model Context Protocol server. Those requests carry the same user agent as the crawl and follow the same rules, so blocking snagabot in your robots.txt stops them too.

Rules it follows

robots.txt, as RFC 9309 defines it. We use the group for snagabot if you have one, otherwise the one for *. If your robots.txt returns a server error, we fetch nothing from the host until it can be read. We read it again every day.
noindex, in a robots meta tag or an X-Robots-Tag header: the page is not kept.
nofollow and links marked rel="nofollow", ugc or sponsored: those links are not followed or counted.
nosnippet and max-snippet:N: we show no text of the page, or at most N characters.
Slowing down. If your server takes longer to answer, we wait ten times as long between requests. When you answer 429 or 503 with Retry-After, we wait that long. A host that asks for more than an hour is left alone for the rest of the crawl.

Directives addressed to snagabot, such as <meta name="snagabot" content="noindex"> or X-Robots-Tag: snagabot: nosnippet, apply to us alone. Directives for other crawlers, such as googlebot, don't.

How to limit or block it

To keep us out of your whole site:

User-agent: snagabot
Disallow: /

To keep us out of part of it, and slow us down:

User-agent: snagabot
Disallow: /private/
Crawl-delay: 10

Part of our index also comes from Common Crawl. Blocking CCBot keeps your pages out of their future crawls, and so out of ours.

Removal and contact

To have pages or a whole domain removed from our index, or never crawled, write to us. We keep a list of domains we never fetch, and act on requests promptly.

Contact: [email protected]

We don't collect personal data and don't build profiles of people. Text we keep from pages is used to find them, has e-mail addresses and phone numbers removed, and is never served in full.