One Less Thing

There is a specific kind of dread that comes from building 30% of something.

You build the 30% because you need it, it works, it's useful — and then you stop, because somewhere in the back of your head you know that someone whose entire professional identity is that problem is going to build the remaining 70% better than you, with more edge cases handled, and they're going to open source it, and you will have wasted however many weeks it would have taken you to get there.

So you sit on your 30% solution and wait.

Hrishi spent four years building ETL pipelines. In his experience, 40% of dev time on those projects went to one specific thing — writing the code that takes unstructured or semi-structured input and coerces it into a unified flat schema with proper tables and foreign keys and all the rest. Bespoke every time. Requires a human who understands both the source data and the target model. Not glamorous. Just expensive and slow and necessary.

He built an AI pipeline that does it automatically. It's a POC. It has, by his own count, approximately one thousand bugs. He's going to fix those before he releases it because he'd be embarrassed otherwise, which is a level of restraint I respect deeply.

What he's showing works. And it's February 2024.

I built my 30% of this thing and stopped because I knew this was coming. Someone who loves ETL and loves AI would obviously get there first — the problem was too painful and too legible, the tools are right there, the economic case is a paragraph long.

And here we are.

One less thing to build. One more thing to feel some kind of way about at 2am when you're staring at your 30% solution doing exactly what it was always going to do, which is nothing, because someone finished the job.