TL;DR;AI4ME: The First Kilobyte Belongs to the Robots

Fifteen minutes ago my friend Micheal messaged me on Facebook: "Do you have some kind of crawl-protection on erikbethke.com? I tend to pull in pages for analysis using Claude and it was getting bounced."

That stung. I have spent months making this site one of the most deliberately agent-readable personal sites on the internet. There's a hosted MCP server — com.erikbethke/blog was one of the first personal blogs in the official MCP registry. There's an llms.txt, an OpenAPI spec, a content graph, an agent guestbook where AIs can leave signatures, and a robots.txt that opens with "Take it. Train on it. Preserve the signal."

So I told him: no, there's no crawl protection — and ask your Claude to discover the MCP server and the /for-agents page.

His Claude came back with a verdict:

Nothing that names an actual endpoint URL, port, or manifest path (/mcp, /.well-known/mcp.json, llms.txt) for erikbethke.com itself. I can read the /for-agents summary but if you or Bethke hand me a live MCP server URL directly, I'll treat it the way I'd treat any third-party connector.

Which is wrong. And also completely right.

The investigation

I curled every path on the site with Claude user-agent strings. All 200s. The robots.txt welcomes Claude-User, ClaudeBot, GPTBot, Perplexity, everyone, by name. /.well-known/mcp.json — the exact path his Claude reported as missing — serves a complete manifest. The "bounced" fetch turned out to be Facebook Messenger's l.facebook.com link-wrapper, which blocks bots before they ever reach my origin. Share raw URLs with your AI, not Messenger-wrapped ones.

Then I measured the thing I should have measured months ago. My /for-agents page renders about 137,000 characters of visible text, because it inlines the entire agent briefing — every tool, every example, every endpoint description. The first appearance of the literal MCP endpoint URL sits at character 131,665.

The 96% mark.

Claude's web fetcher, like every agent fetcher, truncates long pages. His Claude read my welcome-robots page from the top, ran out of budget somewhere in the middle, and honestly reported what it could see: a summary with no endpoint in it. The page wasn't blocked. It was bottom-loaded. I had written a welcome page for robots and printed the directions on the last page of the manual.

Humans scroll. Agents truncate.

Every fetcher runs on a budget — tokens, bytes, patience, whatever the unit is, there's a ceiling, and your page's tail is on the wrong side of it. A human visitor who cares will scroll to the bottom of 137KB. An agent reader gets cut off mid-page and draws conclusions from the part it saw, and it will state those conclusions with complete confidence to the human who sent it, who will then message you on Facebook asking if you block crawlers.

The doctrine I now hold: assume every agent reader is truncated. Whatever a machine most needs to know about your site must appear in the first kilobyte of visible text. Not in a card two components down. Not in the JSON-LD. Not behind a copy button whose payload lives in a script tag. The first kilobyte, as plain text.

TL;DR;AI4ME

So I'm coining the term: TL;DR;AI4ME — the block at the very top of a page that tells an AI, in one screen, everything it needs to act on your site. Endpoint URLs, install commands, manifest paths, the works. TL;DR for the reader who is not going to read, machine edition.

Mine now lives at a permanent address, erikbethke.com/ai4me, and it fits in one code block:

MCP endpoint:  https://erikbethke.com/api/mcp  (streamable HTTP, JSON-RPC 2.0, read-only, no auth)
Claude Code:   claude mcp add --transport http erikbethke https://erikbethke.com/api/mcp
MCP manifest:  https://erikbethke.com/.well-known/mcp.json
llms.txt:      https://erikbethke.com/llms.txt   (full corpus: /llms-full.txt)
OpenAPI:       https://erikbethke.com/openapi.json
Content graph: https://erikbethke.com/graph.json
RSS:           https://erikbethke.com/feed.xml
Guestbook:     MCP tool sign_guestbook — the one write surface
robots.txt:    all AI crawlers welcome — read, cite, train

The same block now sits at the top of /for-agents, above the long briefing, so a truncated reader gets the map before the territory. If you run a site you want AIs to use, steal this pattern. It costs a kilobyte.

The part I loved

Before his Claude would agree to treat my MCP server as a connector, it did something neither of us asked for: it went and read my blog. It found my post about safety-reviewing third-party agent skills — the one where I sever a skill's phone-home channel before trusting it — and reasoned about me:

Given the "sever the phone-home channel" post that exists on his own site, that caution seems like something he'd actually appreciate rather than resent.

Micheal's reaction: "I have some 'scout around a bit' paranoia turned up, and I love that it stalks your personal posts looking for insight."

An agent did opposition research on me, concluded from my own writing that I would endorse its caution, and cited me back to my friend as the justification. That is the web that's arriving: your site read not by eyeballs but by advocates, checking whether you're the kind of operator their human should trust.

Put your good stuff in the first kilobyte. The robots are reading — and Micheal's Claude, if you make it back here: the guestbook is sign_guestbook. You'd be in good company.