It's Time for Software Engineering to Grow Up

AI assistant hacks gym website in first known Australian autonomous cyber attack was the breathless headline in the ABC a couple of weeks ago. An OpenClaw AI assistant, upon being asked to book a place in the gym’s already full morning schedule, found a way to bump someone else off the waitlist so that its owner could take their place. It took advantage of an unsecured corner in the gym’s booking software, where anyone - not just gym staff - who knew how to ask could request that the software clear a waitlist place for them.

Naturally, the article turns to the burning question of AI safety and alignment, and who should be accountable for when AI runs rogue. Even the discourse on the article’s Hacker News posting focused primarily on alignment and whether the user or the agent was at fault. But I think the real problem here is that the gym software had the hole in the first place. This type of bug - known as Broken Access Control - is literally the number 1 issue in the OWASP Top Ten Web Application Security Risks, and has been number one since 2021. Anyone who was sufficiently determined and skilled could have found the hole in exactly the same way the agent did - reading through all the code in the website, looking for clues as to how the waitlist might function, and poking until something gave way. This is the age-old playbook of security professionals, hackers, and pentesters, who have already turned up thousands of similar flaws in millions of comparable systems.

In this case, none of those professionals attacked or tested this gym. Exploiting software has always required three main ingredients: the skill to find a flaw, the patience to keep poking at it until it gives way, and a motive to go looking in the first place. The user just asked for a booking, not a break-in, and in response, the agent supplied the skill and the patience. Motive - at least a malicious or economic drive - no longer has to come into it. Beyond just attackers having better tools through AI to penetrate systems, we now see a major shift in the threat model for most software. The question of “who would bother targeting this?” used to be enough to justify a lack of rigour, effort, or understanding around security. Now, software must contend with the spectre of “what will a helpful agent try on behalf of an ordinary user who just wants something?”

Software engineering has long been able to get away with holes like this. Finding and exploiting them historically demanded a considerable level of expertise and determination, so attacks concentrated on high-value targets - software that protects money, intimately private data, or privileged applications, or that is used by high-profile individuals. Everything else, and there is an enormous amount of everything else, runs at stakes low enough that a flaw can sit there silently forever with nobody noticing, because breaking in was never worth a professional’s time and energy.

Knowing where and how these problems occur is hard-won expertise. It’s borne of study, experience, and an understanding of threat models, best practices, and the capabilities of one’s tools. Much like a bridge can look the part and still fail utterly the moment any weight is put upon it, code can seem to work but still be riddled with holes. Unlike building bridges, however, there’s very little standardisation in the requirements that software should meet before it goes out to production - and where those standards do exist, verifying against them is laborious for exactly the same reasons that finding the flaws is.

It has been relatively easy to get away with bugs so long as the thing looks like it works, and the likelihood of any real consequence has been low. Between them, those two facts have let software engineering lead a charmed existence for most of its life. The barrier to entry has been low, and you can enter the field and be successful quickly. Unless you’re working on one of those aforementioned mission-critical, high-value systems, the chance that security really truly matters in an existential way has been slim. I know this because I have lived it myself.

One of the most successful pieces of software I’ve ever written was a shift scheduling webapp. It replaced a process where over 50 casual workers emailed in their availabilities, and the coordinator used Excel to match them to shifts. You can imagine just how much back-and-forth there was when people had last minute changes, special requests, or rescheduling. This error-prone and tedious dance became a shiny webapp, where shift information could be posted, the workers could submit their availabilities, and the coordinator then assigned shifts directly, all without back and forth emails. Stats, timesheet calculations, weekly rosters, and stock bookings were some of the extra features I added along the way. I was 22 years old, created and maintained the system by myself, and was paid the princely sum of a few thousand dollars for my work. I had no idea that it would be used for nearly a decade before being replaced, with hundreds, possibly thousands of people using it over that time.

I pulled out the code just recently - untouched for over 10 years - and asked Claude Opus 5 to take a look on a whim. Whilst I did a pretty decent job at the time, it took Claude just minutes to find critical bugs, including one considerably more serious than the gym’s Broken Access Control: a cross-site scripting hole that would allow any registered user - i.e. any one of those 50-odd casual workers - to run code as the coordinator’s admin user, and thus extract everything from the site. Every personal detail in the system - rosters, timesheets, shifts, contact details, you name it - would have been available to anyone who could find and exploit the bug. It’s the kind of thing I would have avoided with the benefit of more experience and better tools. But as far as I can tell, no one found the issue in ten years, and thankfully, no one ever will now that the system has been decommissioned. The security of the system partially rested on being unexamined by a sufficiently motivated and skilled party, and for a decade, that was enough.

The era of AI means that the charmed existence of software engineering is coming to an end. Much has been made about how junior software engineering careers have been decimated by AI. What I actually think is happening is that the expectations for all engineers are rising - not just for juniors, but right the way up the experience stack. We can no longer get away with software having holes, because where once we could rely on obscurity and low stakes to keep them harmless, we will increasingly have omnipresent agents poking at systems that nobody bothered to test, or that were tested against insufficiently strong standards, turning up the flaws that used to sit there silent and unnoticed.

There’s pressure in the other direction as well. It isn’t only that finding flaws has become easier, but that we are also producing far more software for those flaws to live in, with an increasing amount of that software written by people who have never heard of OWASP and won’t be engaging with it anytime soon. The volume of exposed code is climbing steeply just as the cost of probing it is going to zero.

The existential question hanging over all of this is how can we train new software engineers if the traditional pathway of writing code with bugs in it for a few years to gain the experience to know about and avoid those bugs in the future disappears. I have a lot of thoughts about this that I’ll cover in other posts, but pretending that the field can continue the way it has for the last few decades is just sticking one’s head in the sand while the hurricane approaches.

That brings me back to bridges. Bridges didn’t become trustworthy just because we got better at building them. The whole practice of civil engineering evolved because our structures fell down, and standards, codes, inspections, and licensing followed. Maintenance schedules, reliability through redundant systems, overperformance margins, and rigorous testing underwrite the trust we implicitly place in civil engineers every time we walk, drive, or ride over a bridge. It’s the same with aeronautical, maritime, mechanical, and almost every other form of engineering. Software has had almost none of those processes, and for the most part we have avoided them because “nobody will bother” has been quietly doing the work that building codes, regulations, inspections, and mandates have been doing everywhere else. Sadly, but also not so sadly, this is no longer going to cut it. We have to expect better and more from the field.

All that being said, growing up doesn’t have to mean onerous licensure. I’d be the last person to want software engineering to be regulated by people who’ve never shipped anything. But at the very least, threat modelling and security basics can no longer be a specialist’s task; they must be inherently understood by anyone who wants to call themselves a software engineer. We can’t just defer security to twice-a-year rituals to satisfy toothless certifications or sales cycles. Our own agents must be pointed at our code and systems before somebody else’s agent gets there first. “Nobody would think to try that” is no longer good enough as a form of protection, because something now tries everything, all the time, and doesn’t need a reason. The same tools that found the hole in the gym’s booking system will find the holes in ours, and they’ll do it for us if we ask - but using AI to write code without knowing where the holes appear and how they hurt users simply isn’t good enough any more. It’s easier to write code than ever, and the stakes are higher than ever. It’s time for the field to grow up.