robots.txt Is Just a No Trespassing Sign

For every no trespassing sign, there is at least one person who sees it as a suggestion — a yellow light, not a red one, a statement of preference rather than law.

This is not cynicism. This is anthropology.

And then, one layer deeper, there is always at least one lawyer whose entire professional existence is dedicated to contesting the spatial positioning, physical condition, material makeup, wording, font size, and ambient lighting conditions of that sign — not because they believe in shortcuts but because the sign turns out to be a remarkably fragile object once you start looking at it.

Michael Heller and James Salzman wrote a whole book about this — Mine! — which argues that ownership is not a natural fact but a set of contested stories, and that whoever tells the better story usually wins. The sign says MINE but the sign itself is just another claim, which means it's just another argument, which means you'd better hope your lawyer is more creative than their lawyer.

robots.txt is, by its nature, a no trespassing sign on a plot of land with no fence, no gate, no enforcement mechanism, and no sheriff. It is a text file. It says please. The entire AI scraping debate of the past two years is essentially the legal profession discovering that the sign exists and beginning the long, billable process of arguing about its font.

The web was built on the assumption that information wants to be free, which was always a polite way of saying that information is free and we're just sorting out who gets to feel bad about it. robots.txt was the compromise — a handshake protocol for an ecosystem that was, optimistically, full of good-faith actors.

Good-faith actors. In 2025. I know.

The sign is still there. People are still arguing about the sign. The shortcut is very well-worn by now.