# Technical SEO for SaaS and AI crawlers

Indexing, JavaScript rendering, Core Web Vitals, schema, internal links, GPTBot, PerplexityBot, ClaudeBot and llms.txt, from primary docs. We automate it.

Canonical: https://ignura.com/technical-seo
Last updated: 2026-10-27
Author: Ignura

## In short

- A page can’t rank or be cited until it’s crawled, rendered and indexed. Google handles JavaScript in three phases, crawl, render and index, and most AI crawlers don’t render JavaScript at all.
- robots.txt doesn’t keep a page out of Google; Google says use noindex for that. And each AI company runs separate crawlers for training and for search, so one blanket rule can hide you.
- Speed is judged at the 75th percentile of real visits: LCP within 2.5 seconds, INP 200 ms or less, CLS 0.1 or less.
- We fix what costs you most, first. Start with a free Growth Check.

## Crawling and indexing

When half your pages aren’t indexed, it’s usually one of three things. Google hasn’t found them, because nothing links to them and they’re not in your sitemap. It found them and chose not to index them, because they’re thin or duplicates. Or something blocks it: a noindex, a canonical pointing elsewhere, a page that’s empty before JavaScript runs. Search Console’s page indexing report tells you which.

robots.txt controls crawling, not indexing. Google says it isn’t a way to keep a page out of Google; use noindex or a password.

## Rendering

Google runs JavaScript with an evergreen version of Chromium, in a separate rendering step after the crawl. Most AI crawlers skip that step. Vercel found ChatGPT’s and Claude’s crawlers fetch JavaScript files without running them; only Google’s and Apple’s crawlers rendered pages.

If your product pages, pricing or docs only exist after a script runs, render them on the server or pre-render them.

**None** of the major AI crawlers Vercel studied in late 2024 ran JavaScript. Only Google’s and Apple’s rendered pages.

## Speed

Core Web Vitals are measured on real visits, at the 75th percentile, on mobile and desktop separately: LCP within 2.5 seconds, INP of 200 ms or less, CLS of 0.1 or less. We fix the template, not one page at a time, so every page gets it.

## Schema

Structured data helps Google read a page, and some types earn rich results. Some don’t any more: since August 2023 FAQ rich results only show for well-known government and health sites, and HowTo results were dropped that September. Google also says no special schema is needed for AI features.

We add Organization, Product or SoftwareApplication with real offers, Article with dates and BreadcrumbList, written from what’s true on the page.

## Internal links

Google uses links between pages to work out what each page is about. Every important page should be a few clicks from the home page, linked with words that say what it is. Orphan pages, linked from nowhere, are the ones that don’t get indexed.

## llms.txt, honestly

llms.txt is a proposal from September 2024 for a markdown guide to a site at /llms.txt, now aimed at agents. Google's AI optimization guide says an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search". None of OpenAI, Anthropic or Perplexity says its crawlers read them. It takes an hour, so we'll add one if you want. It isn't a ranking or citation fix, and we won't bill it as one.

## Mistakes we see

- **One rule that blocks every AI bot.** You meant to stop training and you left ChatGPT and Perplexity search too.
- **robots.txt to hide pages.** Blocked pages can still be indexed from links. Use noindex.
- **Content that only exists after JavaScript.** Google gets it eventually. Most AI crawlers never do.
- **Schema that doesn’t match the page.** Markup must describe what’s visible. Anything else is a liability.

## What our AI does

- Crawls your site like Googlebot and like an AI crawler, with and without JavaScript, and shows the difference.
- Reads your robots.txt, sitemap and CDN rules against every crawler.
- Pulls Search Console’s indexing report and groups the reasons.
- Watches Core Web Vitals from real-visit data where it exists.

## What we do

- Decide which fixes matter and in what order.
- Work with your engineers, or ship the fix ourselves where we have access.
- Write the schema by hand from what the page says.
- Check again after every release.

## The first four weeks

1. Full crawl as Google and as AI crawlers, the indexing report read, the robots and CDN rules checked.
2. The blockers fixed: rendering, noindex and canonical mistakes, crawler rules.
3. Speed on the templates that carry the most pages; schema and internal links.
4. Indexing and crawler access checked again, and the list for month 2.

## The deal

Named in 5 of your 20 buyer questions by week 4, or you don’t pay the second half. $8,000 for four weeks, half to start. Then $4,000 a month, with a month’s notice to stop.

## Questions

### Why hasn’t Google indexed our pages?

It hasn’t found them, it chose not to index them, or something blocks it. Search Console’s page indexing report says which, page by page.

### Should we block GPTBot?

That’s a business call: it only affects training, and OpenAI says it’s independent of OAI-SearchBot. Don’t block OAI-SearchBot unless you want out of ChatGPT search.

### Does llms.txt help?

Not with Google. Google's AI optimization guide says an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search". No major AI company says it reads it. It's quick to add.

### Do we need to rebuild the site?

Rarely. Most fixes are settings, templates and content. When a framework only renders in the browser, server rendering is usually a contained change.

## Sources

- Google Search Central: [Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)
- Vercel: [The rise of the AI crawler (17 Dec 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- web.dev: [Web Vitals](https://web.dev/articles/vitals)
- Google Search Central: [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- Google Search Central: [Changes to HowTo and FAQ rich results (8 Aug 2023)](https://developers.google.com/search/blog/2023/08/howto-faq-changes)
- Google Search Central: [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- Google Search Central: [AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
- Google Search Central: [A guide to Google Search ranking systems](https://developers.google.com/search/docs/appearance/ranking-systems-guide)
- OpenAI: [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- Perplexity: [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- Anthropic: [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- Google Search Central: [Google’s common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- llmstxt.org: [The /llms.txt file (Jeremy Howard, 3 Sep 2024)](https://llmstxt.org/)

Book a 15-minute call about your website: https://ignura.com/book