Skip to content
ARKMONT
Blog

· ARKMONT

Let AI crawlers in

Many sites block AI crawlers without knowing it. If they can't read your page, they're unlikely to recommend you.

Today's tip

  1. Open yourdomain.com/robots.txt in your browser.
  2. Look for lines with GPTBot, OAI-SearchBot, Google-Extended, ClaudeBot or PerplexityBot followed by Disallow: /.
  3. If they're there and you want AI to find you, ask whoever runs your site to remove them.
  4. Also check your site isn't entirely behind a login or a pop-up.

ARKMONT's free check asks one AI whether it names you. Pro asks five engines (ChatGPT, Gemini, Claude, Perplexity and Grok) and gives you fixes ready to paste.

Run a free check