Web reader
The Recognly web reader
Recognly helps companies see how AI answer engines describe them. To do that, our servers read a small number of public web pages. This page explains what we read, how often, and how to ask us to stop.
- Every request says who it is: Recognly, with a link to this page.
- robots.txt is read first and followed. A disallowed page is never requested.
- At most one request per second to a site, or slower if your Crawl-delay asks for it.
- No sign-in, no forms, no scripts, no cookies kept, and a block is never worked around.
How to recognise it
Every request our servers make to a public website carries this User-Agent header:
Recognly/<version> (+https://app.recognly.com/bot; <purpose>)
The product token is Recognly. We never use another name, a browser's user agent or rotating addresses to get a page. The purpose at the end says why the page was read:
| Purpose | What was read |
|---|---|
| page read | A page an AI answer engine cited when answering a question about one of our customers' markets. We read what the page says so our customer can see where an answer came from. |
| listing check | A customer's own public profile on a review site or marketplace (and at most its pricing page), to check that the facts shown there (pricing, location, founding year) match what the customer has approved. A customer's Wikidata item and Wikipedia article are read through the official Wikimedia APIs, never as web pages, following Wikimedia's User-Agent and maxlag rules. |
| directory search | A public search page of an assistant, plugin or connector directory, to see whether a customer's listing appears for a keyword. One search per keyword per week, shared by all customers. |
| robots check | A customer's own robots.txt, once a day per site, to record changes that affect AI crawlers. Only the customer's own sites are read, and the customer can switch it off. |
| key directory | An AI operator's published Web Bot Auth key directory (/.well-known/http-message-signatures-directory), read only for operators in our agent registry (never for an address taken from a request), kept for 24 hours, at most 64 KB. |
| agent ranges | The IP range files AI operators publish for their crawlers (for example https://openai.com/gptbot.json), once a week. |
| disclosure check | A public post a customer's employee wrote, to check that the post still says who they work for. |
| agent audit | A customer's own website, read the way a shopping or research agent would, at the customer's request. Every request also carries an X-PM-Audit header with the customer's own audit token, so the customer can recognise audits. Only on a site the customer has verified it owns may the audit use a browser and, when the customer allows it, fill in the customer's own sign-up form with a test identity the customer provided. It never solves a CAPTCHA or bot challenge and never pays. |
What it reads
Only single public pages that a customer's work needs, one at a time. We do not crawl other people's sites, follow links from page to page, or build an index of the web. Before any page, we read the site's robots.txt and follow it (RFC 9309): rules written for Recognly apply first, then rules for RecognlyBot, then the rules for every agent (*). If robots.txt cannot be read because the server has an error, we read nothing from that site.
The one crawl we do is of a customer's own website, when the customer asks for it: a few dozen pages found through its sitemap and links, with the same User-Agent (without a purpose), the same robots.txt rules and at least one second between requests. If that site is yours and you did not expect it, write to us (below).
We keep a page's text only as long as the feature that read it needs it (30 days for a cited page), plus a fingerprint of the text and the few facts we checked (for example, the price shown on a customer's own listing). We do not collect personal data about the people who wrote a page or who visit it.
What it never does
- Outside an agent audit of a customer's own verified site, it never signs in, fills in or submits a form, runs scripts, or keeps cookies between pages.
- It never retries around a block. A login wall, a 403, a 429 or a bot challenge ends the attempt, and a person can check the page in their own browser instead.
- It never reads LinkedIn, Facebook, Instagram or Threads, whatever their robots.txt says, because those pages are about members.
- It never posts, comments, votes or writes anything anywhere. Recognly does not operate accounts on other sites (see our responsible engagement policy).
- It never requests a private or internal network address.
How often
- At most one request per second to one site across all of our servers, or less often when your robots.txt sets a
Crawl-delay. A site's name with or withoutwww.counts as one site. When a site asks for more than a minute between requests we try again at a later run instead of waiting. - We keep a site's robots.txt for up to 24 hours before reading it again (one hour if it could not be read).
- A cited page read for a customer is kept for 30 days before it is read again for that customer.
- A customer's listing is read once a week, plus at most one read a day when the customer asks for one. Requests to a listing site are at least 10 seconds apart (longer if its Crawl-delay asks for it). After a block or a robots.txt that disallows it, we do not try that listing again for 7 days, and each listing site has a daily limit across all customers.
- A directory search runs at most once a week per keyword, for all customers together.
How to ask us to stop
Add these lines to your robots.txt. We follow them within 24 hours, without any other action from you:
User-agent: Recognly Disallow: /
You can also limit us to part of your site, for example Disallow: /forum/, or slow us down with Crawl-delay: 10. If you would rather not edit robots.txt, or you want us to stop sooner, email us (below) with your domain and we will add it to our list of sites we never read.
An operator can also switch all of these reads off for the whole platform at once, and each feature that reads public pages has its own switch.
Contact
Questions, a page we should not have read, or a request to stop: write to our contact page. Please include your domain and, if you can, the date and path of the request from your logs. We answer every message.