Brocklorebot

Why it is on your site

Brocklore is an AI-assisted customer support tool. When a support ticket comes in, it investigates by reading the product documentation its customer pointed it at, so it can answer from the real docs instead of guessing.

Brocklorebot only fetches a site that one of our customers has explicitly added as a documentation source, by entering its URL in their workspace settings. There is no discovery crawl, no index of the open web, and nothing is fetched speculatively. If you are seeing Brocklorebot and you do not think anyone would have configured your site, tell us and we will trace it - see contact.

What it fetches

• Pages on the same origin as the configured URL, at or below its path. A source set to docs.example.com/guide never reaches the rest of the host.
• /robots.txt and /sitemap.xml first. If a sitemap is published we use it and skip link-following entirely, which is fewer requests for you and better coverage for us.
• The HTML only. Brocklorebot does not run JavaScript, submit forms, log in, accept cookies, or request images, fonts, stylesheets or other assets.
• Separately, when a customer configures an OpenAPI specification, we fetch that single URL directly. That is one request for one file they named, not a crawl.

robots.txt

Brocklorebot reads /robots.txt at the start of every crawl and honours Disallow rules written for the product token Brocklorebot or for *. Disallowed URLs are not requested.

It does not currently read Crawl-delay. The fixed one-second gap between requests applies instead. If that is still too fast for your origin, email us and we will slow it down for your host.

How to block it

Add this to your robots.txt. It takes effect on the next crawl, which is at most a week away.

User-agent: Brocklorebot
Disallow: /

To keep the docs open but hold back one area, disallow just that path:

User-agent: Brocklorebot
Disallow: /internal/

Blocking by User-Agent at your CDN or WAF also works and takes effect immediately. Brocklorebot treats a refused request as a failed page and moves on - it will not retry around the block. Either way the customer who configured the source sees the crawl come back empty, which is usually enough for them to come and ask you about it.

One caveat worth stating plainly: a User-Agent string is self-reported and anyone can copy ours. If traffic claiming to be Brocklorebot is behaving in a way nothing on this page describes, it is worth assuming it is not us. Send us the log lines and we will confirm either way.

Contact

Questions, complaints, a request to slow down or stay off entirely - email [email protected]. Include the hostname and a few log lines with timestamps if you have them. We would rather hear from you than have you find out we were a problem from your graphs.