The shop-watch crawler

shop-watch is a restock notifier. People add specific products from Shopify stores, and when a sold-out product becomes available again, shop-watch posts an alert in their Discord server with a link to the store's cart. It never adds to cart, checks out or buys anything on anyone's behalf.

What it requests

• The store's /robots.txt, and the public product JSON for products someone is tracking: https://<store>/products/<handle>.js. While we measure rate limits, a small capped test tool also requests /products.json. It does not crawl pages, follow links or read HTML.
• Each product is fetched once, however many people track it, and the result is shared.
• Requests are paced per store with jittered intervals, at most 6 requests per minute and 8,640 per day to any one store. There is never more than one request in flight per store. It honours Retry-After and backs off on 429 responses.

robots.txt

• shop-watch reads each store's /robots.txt before anything else, rereads it daily, and does not request any path it disallows. It follows the group for shop-watch if there is one, otherwise the * group.
• It honours Crawl-delay as the minimum time between requests to that store.
• If robots.txt can't be read (a 401, 403, 429 or 5xx, a redirect, or a network error), shop-watch makes no other requests to that store and tries robots.txt again an hour later. A 404 means there are no rules.
• To stop shop-watch from requesting your store, add this to robots.txt:

  User-agent: shop-watch
  Disallow: /

How to recognise it

• User-Agent: shop-watch/1.0 (+https://shopwatch.mevcaus.dev/bot)
• Requests are signed with Web Bot Auth (RFC 9421 HTTP Message Signatures, Ed25519), with Signature-Agent: "https://shopwatch.mevcaus.dev".
• Key directory: https://shopwatch.mevcaus.dev/.well-known/http-message-signatures-directory
• Current key ID (JWK thumbprint): fBK5mO5AU4FCwPzH6gIgcBbx2jVR-IzgfgDqWns84b0

Contact

If you run a store and want to ask about or limit this crawler, email bot@mevcaus.dev.