Can AI search crawlers reach your pages?

Pick a public page you want people to find. Test the response anyone can get, read the rules for the crawler you care about, then look for evidence that it actually visited.

1. Save the public response

Try this on Betterloop's checker, then replace the origin and page with your own. The commands make ordinary public GET requests and save the response in a new temporary folder.

origin='https://betterloopai.com'
page="$origin/check"
evidence=$(mktemp -d)

curl -sS --max-time 15 \
  -o "$evidence/robots.txt" -w 'Robots HTTP %{http_code}\n' \
  "$origin/robots.txt"

curl -sSL --compressed --max-time 15 --max-redirs 5 \
  -D "$evidence/headers.txt" -o "$evidence/page.html" \
  -w 'Page HTTP %{http_code} %{url_effective}\nType %{content_type}\n' \
  "$page"

printf 'Evidence saved in %s\n' "$evidence"

Read the final status, URL and content type. A successful response should contain the page you intended to serve. Open page.html as text and find its title, main heading and a sentence that answers the reader's question. A sign-in screen, challenge or empty JavaScript shell can also return HTTP 200. Read the final response headers and HTML for indexing directives. In the Betterloop example, look for the “Free website check” heading in the saved HTML. Repeat with a representative article or product page.

2. Read the rule for that crawler and path

Use robots.txt on the final page's origin. Record a failed or redirected robots response separately and check the crawler's fetch rules. Look for its named user-agent group. Combine repeated groups for that agent and use the wildcard group only when no named group matches. Apply the most specific matching path rule. The standard prefers Allow when equivalent Allow and Disallow rules match. See the robots matching rules.

For ChatGPT search, check OAI-SearchBot. The separate GPTBot has a training purpose. Treat the result for each crawler as a policy decision for the exact page. See OpenAI's crawler definitions.

3. Keep three kinds of evidence

Policy

Record the agent, origin, path, matching robots rule and the date you read it.

Observed fetch

Look in your CDN or server logs for the named crawler and exact path. Verify the connecting IP against the crawler's published ranges or its documented verification method. Record the timestamp and status. A User-Agent string alone can be imitated. For this example, compare with the published OAI-SearchBot ranges.

Citation and visit

For a citation, record the actual query, date and cited URL from the answer. Count any referral visits separately in your analytics when a source is available.