SEO

Robots.txt Checker

Fetch and validate robots.txt rules.

Enter a domain (example.com) or a full URL. The file is fetched at https://domain/robots.txt from the FKINFAST server — not from your browser — and the result is not stored.

Results appear here. Enter a domain or URL, then select Fetch robots.txt.

Domain Health Report

This page checks one thing. The report checks thirteen — and tells you what to fix first.

A one-off €14.90 report, delivered by email in minutes. Every finding comes with a concrete fix, prioritized.

  • DNS records (A/AAAA)
  • Nameservers (NS)
  • Mail servers (MX)
  • SPF record and policy
  • DMARC record and policy
  • DKIM record (common selectors)
  • HTTPS availability
  • HTTP to HTTPS redirect
  • TLS certificate (issuer, validity, expiry)
  • Security headers (HSTS, CSP…)
  • CAA records
  • robots.txt availability
  • sitemap.xml availability
Get the Domain Report€14.90 · One-time payment. No subscription.

How the robots.txt check works

  1. Enter a domain (example.com) or a full URL, and submit the form.
  2. The FKINFAST server requests https://domain/robots.txt, following up to 5 redirects, and reads the response.
  3. The result shows the HTTP status, the file size, a content preview and a PASS, WARNING or UNKNOWN classification.

The check runs from FKINFAST's server, not from your browser. The domain is not stored and no resolved IP address is returned to this page.

Result states

PASS
The site answered with a 2xx status and the file content is shown in the preview.
WARNING
No robots.txt was found (4xx): crawlers then assume they may crawl everything.
UNKNOWN
No clear answer was read: timeout, connection or TLS failure, an unfollowable redirect, a server error (5xx), or a rejected destination.

Page-level errors are separate from these results. A validation error, rate limit, busy response, unavailable service or network failure is shown as an error, not as a WARNING result.

URL rules and limits

  • Enter a domain or a URL; the file is always fetched at https://domain/robots.txt on the standard port.
  • Loopback, private, link-local, multicast and cloud metadata destinations are blocked before any connection is made.
  • Up to 5 redirects are followed, with a 15-second deadline.
  • The file is read up to 64 KiB; the preview shows the first 2,048 characters.
  • Query strings are omitted from the URLs shown.
  • Limited to 10 checks per 60 seconds per client IP.

What the check does not do

The check fetches and previews the file only. It does not interpret the rules for a specific crawler, does not test whether your pages are actually blocked or allowed, and does not verify the sitemap line the file may reference.

robots.txt checker FAQ

What is robots.txt?

robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may or may not visit. It is a convention, not a security mechanism — private content must be protected by authentication, not by robots.txt.

Why is a missing robots.txt only a warning?

Crawlers treat a missing file (or a 4xx answer) as permission to crawl everything. That is often fine for a public site, which is why it is reported as a warning rather than an error.

Why is the preview cut off?

Very large robots.txt files are rare; the preview shows the first 2,048 characters so the page stays fast, while the size statistic reflects the full response up to 64 KiB.

Should my robots.txt include a sitemap line?

Yes — adding a Sitemap: https://example.com/sitemap.xml line is a common and recommended way to point crawlers at your sitemap. Use the Sitemap Checker to verify the URL it points to.

Related tools