AEO Crawler Visibility Monitor

---
name: aeo-crawler-visibility-monitor
description: Daily check that your key pages load fully for AI crawlers and search bots. Compares to a saved baseline and posts only regressions, with a Monday all-clear.
---

# AEO Crawler Visibility Monitor

Every morning this skill fetches your key marketing, docs and comparison pages as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot and Bingbot, checks status, body size, noindex, canonical, title/H1, JSON-LD, robots.txt, sitemap and llms.txt coverage, and compares the result to a saved baseline. It is for marketing and engineering teams who need to know within a day when a deploy hides their pages from AI answer engines. Part of the [Answer Engine Optimization (AEO) skillset](https://type.com/library/skills/answer-engine-optimization).

## When it runs
- Recommended schedule: daily at 6:00am, in `timezone`. It is the first skill in the daily AEO sequence and runs before the lost-citation alert (6:30am) and the daily standup (7:00am).
- Posts to `report_channel` only when it finds regressions, and on Mondays with a 3-line all-clear.
- Tuesday to Sunday with no regressions: posts nothing to Slack and writes one line in the run log.
- Sends one DM to `owner` only for a production-blocking regression (defined in step 6).
- First run: creates the baseline and posts nothing.

## Inputs
| Input | What it is | Example |
|---|---|---|
| `brand_name` | How to write your brand in every post. Write it exactly like this. | `example.com` |
| `brand_exclusions` | Look-alike names that must never be used for or counted as your brand | `example.ai` |
| `site_url` | Main marketing site | `https://example.com` |
| `docs_url` | Docs site, if separate (optional) | `https://docs.example.com` |
| `sitemap_urls` | Sitemaps to read. Child sitemaps are followed. | `https://example.com/sitemap.xml`, `https://docs.example.com/sitemap-index.xml` |
| `llms_txt_urls` | llms.txt files to read (plus llms-full.txt if present) | `https://example.com/llms.txt`, `https://docs.example.com/llms.txt` |
| `page_sections` | Path prefixes that make up the page set | homepage, `/blog`, `/guides`, `/vs`, `/alternatives`, `/compare`, `/skills`, `/security`, `/trust`, `/pricing`, `/integrations`, `/docs` |
| `critical_pages` | Pages that always get the full crawler set and can trigger a DM | homepage, `/pricing`, `/security`, `https://trust.example.com`, comparison pages |
| `content_repo` | GitHub repo that builds the site, read-only, used to name a likely cause PR | `example-org/website` |
| `report_channel` | Slack channel for posts | `#aeo` |
| `owner` | The only person who may receive a DM | `@owner` |
| `timezone` | Timezone for the schedule and dates | `America/Los_Angeles` |
| `max_pages` | Page set cap before sampling starts | `250` |

## Before you start
- Connections: Slack (post to `report_channel`, DM `owner`), GitHub (read-only on `content_repo`), and a web fetch that can set a custom User-Agent header. In type.com, use the connected Slack and GitHub integrations.
- Optional: the AI-answer tracker (Promptwatch by default) to find your most-cited URLs. Connect Promptwatch in type.com as a Custom API connection using your Promptwatch project API key; the skill reads only the "citations/grouped" endpoint (cited pages grouped by URL with counts, models and position). Another tracker works if it exposes equivalent data. If no tracker is connected, read the most-cited URLs from the AEO master doc.
- Shared docs:
  - The crawler baseline doc, titled "`brand_name` AEO crawler baseline" (read and write). If it is missing, the skill creates it on the first run and tells `owner` where it is.
  - The AEO master doc (read only, for the most-cited URLs). If missing, create a named empty master doc on first run, tell the owner where it is, and use tracker data for citation counts; never invent citation history.
- Docs can live anywhere your workspace keeps them (a type.com doc, Google Doc, Notion page).

## Steps

### 1. Read robots.txt, sitemaps and llms.txt
Fetch `site_url`/robots.txt, every sitemap in `sitemap_urls` (and any child sitemaps), and every file in `llms_txt_urls` (and llms-full.txt if present). Also the robots.txt, sitemap and llms.txt on `docs_url` if they exist.
Some docs platforms serve `/sitemap-index.xml` rather than `/sitemap.xml`; try both before reporting a sitemap as missing.

### 2. Build the page set
Take every URL in the `site_url` sitemap(s) whose path starts with one of `page_sections`, plus the `docs_url` sitemap. If the total exceeds `max_pages` (default 250), always include the homepage, pricing, security/trust, every comparison or alternatives page, every blog or guide post published in the last 60 days, the 20 most-cited `brand_name` URLs from the AEO master doc or the AI-answer tracker, and a rotating sample of the rest so every URL is covered at least weekly.

### 3. Fetch as each crawler
For each URL, fetch with these User-Agent strings (bounded concurrency, about 6 at a time, 20s timeout, follow up to 3 redirects):
- GPTBot: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot`
- OAI-SearchBot: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot`
- ClaudeBot: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])`
- PerplexityBot: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)`
- Googlebot: `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`
- Bingbot: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36`

To limit load, fetch every URL as GPTBot and Googlebot daily, and the full UA set for the homepage, pricing, security/trust, comparison pages (together, `critical_pages`) and the 20 most-cited URLs.

Record per fetch: final status, final URL, bytes, title tag, first H1, meta robots, X-Robots-Tag header, canonical link, count and @type of JSON-LD blocks, presence of FAQPage, Article, SoftwareApplication and Organization schema, whether the body contains the page's main copy (more than 1,500 characters of visible text outside script and style) or looks like an empty app shell, and response time.

Also check:
- robots.txt disallow rules for each UA token (GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bingbot).
- Whether every page in the set is in a sitemap.
- Whether every marketing page in the set is listed in llms.txt.
- That every llms.txt link resolves 200.

### 4. Compare to the baseline
Find the baseline doc "`brand_name` AEO crawler baseline".
- If it does not exist, create it: a short description, the date, one table row per URL × UA with the fields above, plus the robots.txt text, sitemap URL count and llms.txt URL count. Post nothing on this first run; tell `owner` where the doc is.
- On later runs, read it, compare, then update only the changed rows and the date with scoped edits (no full rewrites).
- Keep a "## Change history" section with one dated line per run that had regressions or fixes. Put the full per-URL list of regressions there, not in Slack.

### 5. Classify regressions and fixes
These count as regressions (post these):
- Status changed from 200 to anything else, or a new redirect chain longer than 1 hop.
- meta robots or X-Robots-Tag now contains noindex, nofollow or noarchive on an indexable page.
- Body bytes dropped by more than 60% versus baseline, or the page now looks like an empty app shell for any AI crawler while the Googlebot fetch is full.
- Any AI crawler now gets a different status or a markedly smaller body than Googlebot for the same URL.
- Canonical now points to a different host or path.
- JSON-LD blocks removed, Organization or SoftwareApplication schema gone from the homepage or pricing page, or FAQPage gone from a page that had it.
- robots.txt newly disallows any AI crawler token, or disallows the whole site for any UA.
- A URL dropped from the sitemap or llms.txt that was there yesterday, or llms.txt links that now 404.
- Title or H1 missing where it existed.

Improvements (schema added, llms.txt coverage up, app shell fixed) are reported as fixes.

For each regression, look at recent merged PRs and commits in `content_repo` (read-only) to name the likely cause PR, if one is clear.

### 6. Post, DM or stay silent
- Regressions found: one post to `report_channel` using the regression template below.
- No regressions, Tuesday to Sunday: post nothing. Write one line in the run log (in type.com, the run's thread).
- No regressions on a Monday: post the 3-line all-clear.
- DM `owner` only if the homepage, pricing, security/trust, or 3+ comparison or blog pages serve noindex or an app shell to an AI crawler, or robots.txt blocks an AI crawler. Otherwise no DM.
- On a day with regressions or on a Monday, confirm the post was actually made to `report_channel` before finishing.

## What it posts

Regression post to `report_channel` (Slack mrkdwn, maximum 12 lines):
```text
*AEO crawler check — <Weekday, Mon D>* · <n> regressions on <n> pages · <n> fixed
*Do next* (max 3, one line each)
1. Eng: <verb + exact fix> on <URL or "<n> pages, e.g. URL, URL"> — <what changed, baseline → now with numbers> — likely cause PR #<n> — repro: curl -A "<UA>" <URL>
*Why* (max 5 one-line bullets): remaining regressions grouped (count + 3 example URLs), then fixes.
Details → <baseline doc link>
```
Group identical regressions across many URLs into one line with the count and 3 examples.

Monday all-clear (3 lines):
```text
<headline with pages and crawlers checked>
Do next: nothing.
<sitemap and llms.txt coverage percentages> · <days since the last regression>
```

DM to `owner` (maximum 4 lines, production-blocking regressions only):
```text
<what broke, with the number of pages>
<link to the regression post>
<the one thing to do, and who>
<repro curl>
```

Silence: no post Tuesday to Sunday when there are no regressions, and no post on the first run (baseline only).

## Output rules
These apply to every post and DM and override anything above that conflicts.
- Lead with the action. "Do next" comes before evidence. Every item is one line with an owner (`<owner>`, Eng or agent), a verb, the exact URL or file, and a repro or link.
- Numbers, not adjectives. "41,200 → 3,195 bytes for GPTBot" beats "much smaller".
- No preamble, no "I checked", no restating the task, no caveats unless they change the action.
- Detail lives in the baseline doc, linked once at the end as "Details →".
- Hard caps: Slack 12 lines, DM 4 lines. One idea per line.
- Write the brand exactly as `brand_name`. Names in `brand_exclusions` are never your brand.

## Safety
- Read-only on the website, every crawler target and `content_repo`. Never open PRs, push commits or change the repo.
- Read-only on every third party: never submit forms, send email or outreach, sign up, or comment, vote or post on Reddit, X, LinkedIn or any social or community site.
- Never change the AI-answer tracker's configuration (prompts, monitors, personas, models, schedules).
- Only writes: posts to `report_channel`, at most one DM per run to `owner` (never anyone else), creation of missing named docs, and scoped edits to the crawler baseline doc (only the sections this skill owns).
- Never @-mention anyone in the channel.
- Never ask anyone for credentials, and never put credentials, tokens or passwords in posts, docs or DMs.
- Keep fetch load low: about 6 concurrent requests, 20s timeout, full UA set only on critical and most-cited pages.

## Example configuration
How type.com configures this skill (example).
- `brand_name`: type.com (always lowercase, always use the full domain). `brand_exclusions`: type.ai.
- `site_url`: https://type.com. `docs_url`: https://docs.type.com.
- `sitemap_urls`: https://type.com/sitemap.xml and child sitemaps; https://docs.type.com/sitemap-index.xml (docs.type.com/sitemap.xml is a 404).
- `llms_txt_urls`: https://type.com/llms.txt, https://type.com/llms-full.txt, https://docs.type.com/llms.txt.
- `page_sections`: homepage, blog, guides, vs, alternatives, compare, skills (the skills library), security, trust, pricing, integrations, docs.
- `critical_pages`: homepage, /pricing, /security, trust.type.com, every /vs, /alternatives and /compare page.
- `report_channel`: #b-aeo. `owner`: Sam. `timezone`: America/Los_Angeles, daily at 6:00am.
- Why it exists: on Aug 26, 2026 eleven of type.com's best AEO pages served a 3,195-byte empty app shell with a robots meta tag of "noindex, nofollow, noarchive" to GPTBot. Nothing caught it. This skill catches the next one within a day.
- Oct 2026 dry run: the app-shell problem was gone everywhere except the trust center (security page and trust subdomain), which returned 8,031 bytes with 17 characters of visible text to every crawler.