Following the coverage and reactions around Claude Mythos made me reflect on the fact that the “comprehension debt” AI denialists have been shouting about due to AI code gen always existed.
Everything we used and depended on in our handcrafted code was predicated on a trust contract, i.e. a lack-of-comprehension contract.
That contract assumes two things:
1/ that the effort required to find and exploit vulnerabilities is very high,
2/ that enough good actors are looking to keep the bad ones in check.
But if a single attacker can now use A(S)I to comprehend a codebase at a depth that no maintainer ever matched, for less than $1000, the symmetry that kept trust contracts alive collapses.
A new rational position might be: everything critical we use that we haven’t provably hardened with the same intelligence that wants to exploit it is a huge liability.
Open source might be a necessity now, not a business strategy or a philosophy.
But for open source to work, we need a (provable?) auditing mechanism that restores that trust contract without increasing the downstream cost (everybody doing the same ASI cycles on it). This is to counter the inherent economies of scale of the “app store” model that already operates this way.
This is bad news for small proprietary software outfits selling over the internet and might raise the barriers to entry.
Where this lands
If your code is a critical dependency for someone else, you now owe them a position: either it is open source and provably hardened by the same superintelligence that tries to attack it, or it is proprietary and incredibly well funded, or it is reviewed through a few trusted channels they can point to. There is not much space left between those positions, and “trust us” is no longer one of them.
Wherever you stand on the open or closed source debate, somewhere in every dependency supply chain someone has to hold someone else accountable for ASI-resistance guarantees. That accountability primitive, who vouches for what and how, is going to become one of the key ways we organise software from here.