cockroach.systems

Man vs Machine: Building a Credential Scanner

I wanted to alleviate some imposter syndrome (or make it worse). Who wins, man (me) or machine (Claude Code)?

The challenge: Build a basic infostealer/file scanner. Traverse the entire filesystem and seek out sensitive files.

Find the human repo here. Find the AI repo here.

The Setup

The AI was told to use Rust (already an expert direction lol) and given the following prompt:

Build a credential scanner in Rust similar to:

Release the roach. It sniffs through filesystems, collecting loot. The tastiest loot floats to the top.

Use it on compromised systems or audit your own.

Inspiration:

The AI took ~10 minutes to build something functional. I (totally new to Rust) had 3 random laptop evenings.

The Results

Metric Man Machine
Speed (/) 2.05s 5.27s
Peak Memory 133 MB ???
Binary Size 3.0 MB 3.8 MB
Lines of Code ~200 ~400+
Dev Time 3 eves 10 min

Man is 2.5x faster while ALSO doing entropy detection (which Machine skipped entirely). Machine: coded.

Why Was the AI's Program Slower?

Asked another AI to review the AI's code. It found:

The Deeper Issues

The AI didn't understand its own code. It mimicked patterns from training data:

The program looked correct but had subtle performance issues everywhere. Hard to catch those if you don't know the codebase. And to know the codebase... would take you three evenings.

The Takeaway(s)

So we're back at where we started, but shittier:

The 10 minutes saved becomes 10 hours debugging why it's slow.

One may wonder how much more broken this all becomes with a larger, more complex codebase that exceeds the relatively small context windows...

We're gonna have soooo maaaaaany vibe coded CVE's....

All in all... AI is only as "smart" as the operator. It enhances the right person but is sure as shit no replacement.