Skip to content

Web reader

The Recognly web reader

Recognly helps companies see how AI answer engines describe them. To do that, our servers read a small number of public web pages. This page explains what we read, how often, and how to ask us to stop.

  • Every request says who it is: Recognly, with a link to this page.
  • robots.txt is read first and followed. A disallowed page is never requested.
  • At most one request per second to a site, or slower if your Crawl-delay asks for it.
  • No sign-in, no forms, no scripts, no cookies kept, and a block is never worked around.

How to recognise it

Every request our servers make to a public website carries this User-Agent header:

Recognly/<version> (+https://app.recognly.com/bot; <purpose>)

The product token is Recognly. We never use another name, a browser's user agent or rotating addresses to get a page. The purpose at the end says why the page was read:

PurposeWhat was read
page readA page an AI answer engine cited when answering a question about one of our customers' markets. We read what the page says so our customer can see where an answer came from.
listing checkA customer's own public profile on a review site or marketplace (and at most its pricing page), to check that the facts shown there (pricing, location, founding year) match what the customer has approved. A customer's Wikidata item and Wikipedia article are read through the official Wikimedia APIs, never as web pages, following Wikimedia's User-Agent and maxlag rules.
directory searchA public search page of an assistant, plugin or connector directory, to see whether a customer's listing appears for a keyword. One search per keyword per week, shared by all customers.
robots checkA customer's own robots.txt, once a day per site, to record changes that affect AI crawlers. Only the customer's own sites are read, and the customer can switch it off.
key directoryAn AI operator's published Web Bot Auth key directory (/.well-known/http-message-signatures-directory), read only for operators in our agent registry (never for an address taken from a request), kept for 24 hours, at most 64 KB.
agent rangesThe IP range files AI operators publish for their crawlers (for example https://openai.com/gptbot.json), once a week.
disclosure checkA public post a customer's employee wrote, to check that the post still says who they work for.
agent auditA customer's own website, read the way a shopping or research agent would, at the customer's request. Every request also carries an X-PM-Audit header with the customer's own audit token, so the customer can recognise audits. Only on a site the customer has verified it owns may the audit use a browser and, when the customer allows it, fill in the customer's own sign-up form with a test identity the customer provided. It never solves a CAPTCHA or bot challenge and never pays.

What it reads

Only single public pages that a customer's work needs, one at a time. We do not crawl other people's sites, follow links from page to page, or build an index of the web. Before any page, we read the site's robots.txt and follow it (RFC 9309): rules written for Recognly apply first, then rules for RecognlyBot, then the rules for every agent (*). If robots.txt cannot be read because the server has an error, we read nothing from that site.

The one crawl we do is of a customer's own website, when the customer asks for it: a few dozen pages found through its sitemap and links, with the same User-Agent (without a purpose), the same robots.txt rules and at least one second between requests. If that site is yours and you did not expect it, write to us (below).

We keep a page's text only as long as the feature that read it needs it (30 days for a cited page), plus a fingerprint of the text and the few facts we checked (for example, the price shown on a customer's own listing). We do not collect personal data about the people who wrote a page or who visit it.

What it never does

  • Outside an agent audit of a customer's own verified site, it never signs in, fills in or submits a form, runs scripts, or keeps cookies between pages.
  • It never retries around a block. A login wall, a 403, a 429 or a bot challenge ends the attempt, and a person can check the page in their own browser instead.
  • It never reads LinkedIn, Facebook, Instagram or Threads, whatever their robots.txt says, because those pages are about members.
  • It never posts, comments, votes or writes anything anywhere. Recognly does not operate accounts on other sites (see our responsible engagement policy).
  • It never requests a private or internal network address.

How often

  • At most one request per second to one site across all of our servers, or less often when your robots.txt sets a Crawl-delay. A site's name with or without www. counts as one site. When a site asks for more than a minute between requests we try again at a later run instead of waiting.
  • We keep a site's robots.txt for up to 24 hours before reading it again (one hour if it could not be read).
  • A cited page read for a customer is kept for 30 days before it is read again for that customer.
  • A customer's listing is read once a week, plus at most one read a day when the customer asks for one. Requests to a listing site are at least 10 seconds apart (longer if its Crawl-delay asks for it). After a block or a robots.txt that disallows it, we do not try that listing again for 7 days, and each listing site has a daily limit across all customers.
  • A directory search runs at most once a week per keyword, for all customers together.

How to ask us to stop

Add these lines to your robots.txt. We follow them within 24 hours, without any other action from you:

User-agent: Recognly
Disallow: /

You can also limit us to part of your site, for example Disallow: /forum/, or slow us down with Crawl-delay: 10. If you would rather not edit robots.txt, or you want us to stop sooner, email us (below) with your domain and we will add it to our list of sites we never read.

An operator can also switch all of these reads off for the whole platform at once, and each feature that reads public pages has its own switch.

Contact

Questions, a page we should not have read, or a request to stop: write to our contact page. Please include your domain and, if you can, the date and path of the request from your logs. We answer every message.