Crawling
--crawl-links archives a site by following its links. A sitemap target archives the pages it lists, as --crawl-url-is-sitemap controls. Both write WARC, WACZ or SQLite.
Pacing
Page loads on one host wait 1 second apart. --crawl-delay sets the wait, and 0 removes it. A host whose robots.txt asks for a longer Crawl-delay is paced by that instead; --skip-robot-instructions ignores it.
A crawl runs one browser per CPU, and --crawl-concurrency sets the count. The delay holds across every browser, so more browsers speed a crawl only across several hosts or with a shorter delay.
Robot Instructions
A crawl honors the directives a page gives in <meta name="robots">, <meta name="zshot"> or the X-Robots-Tag header. A directive naming another agent, such as X-Robots-Tag: googlebot: noindex, is ignored.
| Directive | What the crawl does |
|---|---|
nofollow | Follows none of the page’s links |
noindex | Archives the page but leaves it out of SQLite’s full-text search and the WACZ page list |
noarchive | Leaves the page, and everything it loaded, out of the archive |
none | noindex and nofollow |
llm_text, llm_json and --navigate refuse a page whose robots meta tag or X-Robots-Tag says noai, or whose robots.txt says Content-Signal: ai-input=no, exiting the process or returning HTTP 403. noimageai keeps the page’s screenshot from the model. A screenshot, PDF or other render that gives no model the page ignores these directives.
Respect Site Ownership
ZShot follows robots.txt by default, its pacing included. Crawl at a rate the site’s owner would accept.