CrawlwrenBot

If you saw this User-Agent in your logs, this page explains who we are, what we fetch, and how to reach us or opt out.

CrawlwrenBot/0.1 (<tool>; +https://crawlwren.com/bot)

<tool> names the tool making the request: uk-eu-tenders, global-tenders, sam-gov-opportunities or gleif-lei-lookup.

Who we are

Crawlwren makes small data tools on official open data: public tender notices and LEI records. The tools run on the Apify platform when someone starts a run.

What we fetch

Official APIs and open-data files that publishers provide for reuse: UK Contracts Finder and Find a Tender, TED, SAM.gov’s public extract and API, GLEIF, CanadaBuys open data, BOAMP, TenderNed, AusTender, the EU Funding & Tenders search API and the World Bank procurement APIs.
Only the records a run asks for (a date window and filters), not whole websites.
We don’t log in, solve CAPTCHAs, rotate IPs or proxies, or impersonate a browser. If a source blocks us, the run stops for that source and reports it. We don’t work around blocks.

Rate limits

Host	Pacing
Default, per host	One request at a time, at least 1 second apart
UK Contracts Finder, Find a Tender	At most 24 requests per minute per run, 2 at a time
GLEIF API	At least 1.1 seconds apart
World Bank, EU Funding & Tenders	At least 1.5 seconds apart
SAM.gov (keyless)	One public-extract download per run

On HTTP 429 we wait for Retry-After (up to 2 minutes) before retrying; on 5xx and network errors we retry up to 3 times with backoff.

Opt out or slow us down

Email [email protected] from your domain with the host name. We’ll stop fetching it or lower the rate, and confirm by email.
Or block the User-Agent token CrawlwrenBot at your server or CDN. Our tools stop on a block rather than retrying around it.

Contact

[email protected] reaches a person.