Plainwren Link Checker

This is a link checker run by Plainwren. Someone starts a run on Apify, lists the pages to check, and confirms the sites are theirs or that they are allowed to check them. The checker reads the links on those pages and sends a HEAD request to each unique link. It does not crawl the rest of the site, follow on from a link target, or read sitemaps.

User agent

Every request uses this user agent:

PlainwrenLinkChecker/1.0 (+https://tools.plainwrenstudio.com/bots/link-checker/)

robots.txt

The checker reads robots.txt once per host and obeys the rules for the token PlainwrenLinkChecker. A 4xx response (other than 429) is treated as no rules. A 429 or 5xx response means that host is not requested for the rest of the run. Crawl-delay is honoured up to 10 seconds.

To opt out, add rules like this to your robots.txt:

User-agent: PlainwrenLinkChecker
Disallow: /

How often it requests

It sends at most one request at a time to a given host. The default gap is one request per second. On the site being checked, the person who starts the run can set that between 0.2 and 4 requests per second; hosts outside that site stay at one request per second. Up to four hosts are contacted at once by default, and never more than ten.

Questions or opt-out

Email [email protected] if you have questions or want to opt out.