ControlCrawl.tech

No domain
AI-First Site Optimization Platform

Take Control of How AI Agents See Your Site

Gemini, Claude, GPT — they're all crawling the web. ControlCrawl gives you the tools to shape what they find: generate llms.txt sitemaps, validate JSON-LD schemas client-side, and audit your pages for AI readiness — all from a single, zero-dependency dashboard.

Crawl Rules — Connect a domain to scan
Schema Health — Run a directory scan first
Context Load — Generate an export to measure

Get Started

Visual llms.txt Generator & Tree Mapper

Map your site architecture, toggle page visibility for AI crawlers, and export formatted configuration files.

Interactive Site Tree

No nodes

Toggle the eye icon to hide pages from AI. Edit descriptions inline.

Enter a URL above and click Scan to build the tree.

Client-Side Directory Optimizer

Recursively scan a local project folder to validate JSON-LD structured data. Everything runs in your browser.

Checking API…
Scanner Output
> Ready. Select a directory or run the demo.

JSON-LD Inspector

// Select a scanned file to inspect its schema

AI Readiness Validator & Link Checker

Audit llms.txt compliance, verify link structure, and measure token cost against leading LLMs.

Context Window Analyzer

Token footprint across major model context windows.

Size—
Words—
Tokens—
Claude 3.5 Sonnet (200k)—
GPT-4o (128k)—
Gemini 2.0 Flash/Pro (1M)—

v1.1.0 Compliance

  • Has H1 title
  • Has intro / description block
  • Uses standard markdown links
  • Optional section grouping

Link Audit