Skip to main content
SpektraBot

For website owners

The SpektraBot crawler

SpektraBot is a guidance service for families and professionals working with children who have special educational needs and disabilities (SEND) in England. It is run by Spectrum Dynamics CIC, a community interest company. If you have seen our crawler in your logs, this page explains what it is doing.

How to recognise it

Every request carries this user agent:

SpektraBot/1.0 (+https://spektrabot.co.uk/bot; support@spektrabot.co.uk)

We do not pretend to be a web browser. When a page needs JavaScript to show its content, we load it in a headless browser that sends the same user agent.

What it fetches

  • Local authority SEND Local Offer pages and the documents they link to.
  • National SEND guidance from government and NHS sites, and from a short list of national charities.
  • Published school policy documents, such as SEND information reports.
  • Published decisions of the Local Government and Social Care Ombudsman.

It only follows links to SEND-related pages on an allowlist of public sites. It never submits forms, logs in, or tries to get past a CAPTCHA or other challenge. If your site challenges us, we stop and record the page as blocked.

How it behaves

  • It crawls one council site at a time, at about one request per second.
  • It reads your robots.txt before fetching anything and obeys it.
  • It honours a Crawl-delay line, up to 30 seconds between requests.
  • It uses ETag and Last-Modified headers, so unchanged pages are not downloaded again.
  • Crawls are started by hand, typically overnight, not continuously.

How to limit or block it

Add a group for our product token, SpektraBot, to your robots.txt. To keep us out of your whole site:

User-agent: SpektraBot
Disallow: /

To keep us out of one section, or to slow us down:

User-agent: SpektraBot
Disallow: /search
Crawl-delay: 5

If there is no SpektraBot group we follow your User-agent: * rules. Changes take effect within an hour. If your robots.txt cannot be read because of a server error, we treat the whole site as off limits until it can.

Contact

If the crawler is causing you a problem, or you would rather we used a different page as the start of your Local Offer, email support@spektrabot.co.uk. We can also stop crawling your site at our end straight away.