On a single weekend, sixteen security researchers pointed AI systems at 390 Bitcoin-related open-source repositories. The output was 4,962 findings. Seven hundred and twenty of those findings were labelled critical or high severity. The campaign was organized by developer Calle, with support from OpenSats, OpenCode, and an AI-inference sponsor. It was fast. It was broad. It was also, if you read the arithmetic carefully, less coherent than the headlines suggest.
Here is the calculation that should stop every security engineer in their tracks. Sixteen people working for thirty hours produce four hundred and eighty person-hours. Divide 4,962 findings by 480 and you get 10.3 findings per person-hour. Divide 720 critical or high findings by 480 and you get 1.5 findings per person-hour. The campaign's own claim of 2.3 critical or high findings per person-hour does not match that math. If the claim actually meant 2.3 per researcher per hour, the implied total would be 1,104 critical findings, not 720. If it meant per person-hour, the implied effective labour would be only 313 hours, not 480. Either way, the published rate cannot be derived from the published totals.
That gap is not an academic footnote. It is a signal. I have spent years in and around blockchain security, and I have learned that every security report is a story told through metrics. The metrics are not neutral. They are shaped by tooling, by triage choices, and by the quiet pressure to produce a compelling narrative. The campaign may well have found hundreds of exploitable issues. But the difference between 1.5 and 2.3 is a tell. Somewhere in the pipeline, hours disappeared, definitions shifted, or the denominator was chosen to make the result look better. None of those possibilities should be ignored.
The architecture of trust is built, not inherited. If we are going to rely on AI-assisted security, we need to be able to audit the auditors.
What Actually Happened
Let me reconstruct the facts. A group of sixteen security researchers, led by Calle, ran an AI-assisted security sweep of 390 Bitcoin-adjacent open-source repositories. The work took thirty hours. The campaign had institutional backing from OpenSats, which funds open-source Bitcoin development, and OpenCode, which builds open tools for AI coding. An unnamed sponsor provided AI inference compute. Each researcher used a different set of prompts and methods. Organizers deliberately encouraged that diversity, arguing that a single prompt strategy would miss weaknesses that another prompt strategy could catch.
This is not a product launch. It is not a bug bounty. It is an experiment in applied AI auditing, executed at a scale that no traditional audit firm would attempt in a weekend. Traditional security audits treat one project as one engagement. A competent firm spends one to four person-weeks on a medium-sized codebase. That is forty to one hundred and sixty person-hours per project. The Bitcoin campaign had about four hundred and eighty total person-hours. Spread across 390 repositories, each project received, on average, slightly more than one hour of aggregate human attention. On a per-project basis, that is shallower than any professional audit I have seen. But the exercise was never designed to replace deep audits. It was designed to maximize coverage.
Bitcoin is a difficult target for automated security work. It is not a single codebase. It is an ecosystem: wallets, indexers, sidechains, hardware-signing libraries, oracle bridges, second-layer protocols, and a long tail of tooling. Many of these projects are maintained by small teams or volunteers. A critical flaw in a widely used wallet library can drain funds across thousands of users. Yet the ecosystem has historically underfunded security review. Most open-source maintainers are too busy shipping features to run a disciplined audit program.
That funding gap is the context for this campaign. OpenSats and OpenCode are not paying for a one-off publicity stunt. They are testing whether LLMs can become a cost-effective first line of defense for projects that cannot afford Trail of Bits or OpenZeppelin. The question is not whether AI can find bugs. It clearly can. The question is whether the bugs it finds are real, exploitable, and prioritizable.
The Methodology: Humans Are the Killer Feature
Let me be explicit about where I stand. I have been auditing blockchain infrastructure since 2017. I have watched static-analysis tools promise the same thing that this campaign now promises: scale, speed, and safety. Slither, Mythril, Securify, and a dozen others all claimed to find vulnerabilities faster than humans. They all produced long lists of findings. They all required a human to separate signal from noise. The difference now is that LLMs understand intent, not just structure. That is meaningful. It is not magical.
Human-guided LLM auditing is a distinct category from autonomous AI scanning. The human writes prompts, steers the model toward suspicious code paths, and verifies results. The process is closer to pair programming with an extremely aggressive code reviewer. The model reads code as language, not as syntax trees. This allows it to catch logic errors that pattern-matching tools miss. It also produces confident, articulate wrong answers. The campaign's decision to use many different prompts is an ensemble-learning strategy. Each prompt defines a different prior. The intersection of their mistakes is smaller than the union of their blind spots. In my view, that is the only honest way to use AI for security at scale. A single model with a single system prompt is a single point of failure.
But ensemble methods reduce variance. They do not remove bias. If the underlying models share similar training data, their blind spots are correlated. The 4,962 findings are likely clustered around classes of bugs that LLMs are good at finding: reentrancy, unchecked return values, arithmetic overflows, access-control mistakes, unsafe deserialization. They are less likely to catch subtle economic logic, cross-protocol composability risks, or incentive misalignments. That is where human auditors earn their fees. The campaign was a volume screening. It was not a substitute for judgment.
I have seen this pattern before. In 2020, when yield farming went mainstream, teams rushed to copy audited code from Compound and Aave. Many of those copies contained deliberate or accidental deviations. Static analyzers could not tell them which deviations mattered. A skilled auditor could. The same will be true with LLMs. The AI will find the cloned pattern faster than a human. But only a human will understand which pattern change breaks the economic model.
The Numbers: What They Do and Do Not Prove
Let me walk through the data without rhetorical inflation.
The core numbers are simple: 390 projects, 16 researchers, 30 hours, 4,962 findings, 720 critical or high findings. At a campaign level, that is 165.4 findings per hour. That number checks out. 4,962 divided by 30 is 165.4. It is an exact match with the reported 166 findings per hour. But 'findings per wall-clock hour' is a throughput metric, not a quality metric. A finding is not a vulnerability. A finding is a candidate. The word 'issue' in the original report is doing a lot of work. An unused variable is an issue. A missing input limit is an issue. Neither is necessarily exploitable. The jump from 4,962 issues to 'nearly 5,000 software problems' is a narrative leap.
Now look at the per-researcher numbers. Each researcher produced, on average, 310 findings and 45 critical or high findings over the weekend. That is roughly ten findings and one and a half critical findings per hour of logged time. Those are plausible numbers for an AI-assisted workflow. A human alone cannot read that much code in thirty hours. A human using a model that can scan entire repositories in minutes can. But the model is not verifying. The model is suggesting. The human is supposed to verify. If the human spends only a few minutes per finding, the verification is shallow. If the human verifies deeply, the number of findings should be close to the real triage output. The report does not tell us how many findings survived verification.
That missing ground truth is the largest weakness of the entire exercise. The report does not disclose the specific AI model, the prompt templates, the repository versions, or the classification criteria. Without those details, the campaign cannot be reproduced. In security, reproducibility is not a luxury. It is the entire point. A finding without a commit hash, a prompt, and a result trace is an anecdote. An audit without a public methodology is a marketing document.
I do not say this to dismiss the organizers. I say it because I have made the opposite mistake. Early in my career, I trusted a summary of a test report and almost deployed a broken upgrade. The summary said all tests passed. The actual log showed that the test suite had been skipped. From then on, I have insisted on raw data. The campaign should publish the raw data: the prompts, the model outputs, the human decisions, the PoCs, and the maintainer responses. That would be the most valuable contribution this exercise could make.
The Coverage Paradox
There is an uncomfortable asymmetry in the campaign. It found 720 critical or high issues in 390 projects. That means an average of 1.85 critical or high findings per project. That is not a remarkably high number for most codebases. A single complex smart contract can contain more than one critical issue. The surprise is not that AI found issues. The surprise is that the volume was so manageable.
But 'manageable' depends on who does the work. The campaign compressed weeks of human effort into 30 hours of wall-clock time. It did not compress the effort required to fix the issues. The bottleneck moved from discovery to triage. If even ten percent of the 4,962 findings require detailed manual review, that is nearly 500 tickets. The maintainers of these 390 projects, many of them unpaid, now have to process a flood of potential vulnerabilities. Without a standard for deduplication, severity, and reproduction, the output of a 30-hour campaign can become a 300-hour backlog. That is a security tax on the people least able to pay it.
This is the part of the AI-audit story that nobody in the celebration wants to discuss. We are building tools that generate more work for the exact people we are trying to protect. The fix is not to find fewer bugs. The fix is to build a better triage pipeline. The campaign should have shipped with a dashboard, not just a list. It should have grouped findings by project, by commit, and by exploitability. It should have provided one-click reproduction where possible. Instead, the public got a count.
I am not proposing that the organizers did nothing. I am proposing that the real security insight is in the follow-through. The only positive signal in the report is that researchers sent concrete PoC re-tests to maintainers and that many maintainers quickly confirmed the reports. That gives me confidence that a portion of the findings are real. But 'confirmed' is not 'patched'. We do not know how many confirmations led to fixes, how many fixes introduced regressions, or how many critical findings are still sitting in a maintainer's inbox.
The Maintainer's Dilemma
Imagine the actual morning after the campaign. A maintainer of a small Bitcoin library wakes up to an issue tracker containing fifteen AI-generated findings. Three have PoC video re-tests attached. One is marked critical. The maintainer has a day job. The project has no paid contributors. The critical finding looks plausible but needs careful review. The maintainer must decide whether to drop everything, trace the vulnerable path, construct a patch, run the test suite, publish a release, and coordinate with every downstream project that pins the old version.
That is not a bug report. That is a project plan. Most open-source maintainers will not be able to complete it quickly. Some will abandon their projects. Others will make rushed fixes that introduce new vulnerabilities. This is the hidden external cost of high-volume AI auditing. The campaign generated work without generating the resources to do the work.
A responsible campaign would have done two additional things. It would have created patch suggestions, not just findings. It would have built a prioritized remediation path for each project. Without patches, the campaign is like a doctor who sends a list of diseases to a patient but no treatment plan. The patient still needs surgery. The patient still cannot afford the surgery.
This brings me to the deepest problem with the public perception of AI audits. A large count of findings is treated as a success. In reality, success is measured by observed fixes. If 4,962 findings lead to fewer than, say, one hundred merged patches, the campaign was not a security breakthrough. It was a research exercise with a good press release. That may be acceptable, but it should be labelled accurately.
The Missing Categories
Let me also name the vulnerabilities that AI audits will routinely miss. LLMs are language models, not execution engines. They are trained on text and code. They do not run the code. They reason about it. That means they are excellent at known patterns and weak at emergent behavior.
A reentrancy attack is a known pattern. Arithmetic overflow is a known pattern. But a vulnerability that only appears after a specific sequence of transactions, or after an unexpected interaction with a third protocol, is not a pattern. It is a simulation. The AI cannot see it unless it has been trained on something similar. The human can see it only by constructing a mental model of the system and asking what is missing.
During the DeFi summer, I watched a protocol with a clean audit report lose millions because the economic incentive was misaligned. The code was correct. The mechanism was broken. No static analyzer would have caught it because the bug was in the design, not the implementation. AI audits will not catch design-level vulnerabilities. They may even lull teams into believing the design is safe because the implementation passed.
The campaign's 4,962 findings are probably concentrated in certain categories. If the organizers publish a category breakdown, I predict a heavy concentration in input validation, access control, and error handling. I predict a much lower concentration in economic reasoning, state-machine ordering, and cross-contract composability. Those are the categories where human experts still matter most. The asymmetry is not a criticism of AI. It is a map of where we should spend our remaining human attention.
The False Positive Problem
The report does not mention false positives. It is the most important number in any AI-audit exercise. Traditional static analysis tools have false-positive rates between 50% and 90%. LLM-based security tools, in my reading, land somewhere in the 30% to 60% range when used with human prompts. If the campaign had a 50% false-positive rate, then 2,481 findings were real. If it had a 90% false-positive rate, only 496 were real. The difference changes the severity of the story. With 720 critical or high findings, a 50% false-positive rate leaves 360 critical issues. A 90% rate leaves 72. That is the difference between an emergency and a busy week. The organizers should have published enough data to narrow that range.
This is not an attack. It is a request for a denominator. Security is not a count of findings. It is a ratio of true positives to effort spent verifying them. Without a false-positive rate, we cannot compute precision. Without a re-audit, we cannot compute recall. Without precision and recall, the campaign is not a security assessment. It is a suggestion engine with a press release.
I have a personal rule, born from the ICO era and refined through the DeFi summer: the value of any security product is inverse to the ease with which its findings can be gamed. If you can produce a report with a thousand findings, you can also produce a report with zero findings by adjusting the prompt. The question is not what the AI can see. The question is what the AI does not see. And the only way to know that is to test it on known vulnerabilities with a known baseline.
What AI Auditing Gets Right
I have been critical. Let me now be fair. The campaign got something fundamentally right: it treated AI as a tool for humans, not as a replacement for them. The use of multiple prompts is exactly the right instinct. The inclusion of PoC demonstrations is exactly the right instinct. The willingness to publish numbers, incomplete as they are, is more than most security firms do. The market for security audits is informationally opaque. Firms rarely publish their false-positive rates. They almost never publish their prompt strategies. This campaign has already pushed the conversation forward by forcing people to ask what the numbers mean.
The economics are also worth taking seriously. A traditional audit of a medium-sized protocol can cost anywhere from fifty thousand to a hundred thousand dollars. At that price, the Bitcoin ecosystem cannot audit everything. A human-guided AI sweep, even with a high false-positive rate, could become an affordable first pass. Projects could submit themselves for a low-cost AI audit before paying for a deep human audit. That would raise the floor for security across the entire ecosystem. The 390 projects in this campaign are the first evidence that the floor can be raised.
But there is a danger in that vision. If AI audits become cheap, they will also become superficial. Companies will point to an AI audit report as proof of security. Regulators might even start accepting AI audit reports as compliance evidence. That would be a catastrophic mistake. An AI audit is a hypothesis generator. It is not a guarantee. The only valid use of an AI audit is as an input to a human review process. The moment it becomes an output, it becomes a liability.
I read security reports the way I read financial statements: I look for footnotes. A number like 4,962 has to be explained. What was the source? What was the model? What was the exact definition of critical? What happened after each report was sent? In 2017, I built a ledger for ICO whitepapers. I rejected eleven of twelve projects because the details did not survive basic scrutiny. The one that survived returned 40x. The lesson was not that I had good instincts. The lesson was that forcing a claim to defend its own details is the fastest way to learn if it is true. The campaign report, as it stands, would not survive my ledger.
Dependency Amplification
Security severity is not an isolated property of a line of code. It is a function of the dependency graph. A logical error in a small helper library can become a critical issue if that library is imported by hundreds of wallets. The campaign's raw count of 720 critical or high findings does not tell us how much downstream damage each issue could cause. The organizers should have mapped every finding to its transitive dependents. A critical issue in a zero-download repository is less dangerous than a medium issue in a widely used package. Without a dependency-aware severity score, the numbers are flat. In Bitcoin, where software is deeply interconnected, flat severity is dangerous. The next iteration of this campaign should publish a graph, not just a list. The graph is the story.
The Contrarian Angle: The Moat Is the Data, Not the Bugs
Here is the contrarian takeaway that most readers will miss. The campaign's most important output is not the 4,962 findings. It is the curated dataset of human-guided prompts, model outputs, human decisions, and confirmed PoCs. That dataset, not the bug list, has the highest long-term value. It will be used to train the next generation of security models. It will teach models to distinguish between an LLM guess and a verified exploit. It will give future teams a benchmark to measure precision and recall.
The organizers may not even realize how valuable this dataset is. Or perhaps they do, and that is why they have not released it. If OpenSats and OpenCode have access to the full traces, they have a proprietary asset. The next auditors will not need to start from zero. They will have a corpus of 390 projects, 16 researchers, and 4,962 labelled findings. That is a gold mine.
There is a darker side to that gold mine. AI security research is dual-use. The same prompt-diversity strategy that finds bugs in Bitcoin projects can be repurposed to hunt for exploitable vulnerabilities in Bitcoin infrastructure. The same ensemble technique can be used to build autonomous offensive agents. Every improvement in AI-assisted defense is also an improvement in AI-assisted offense. The industry does not like to think about this. It should. The campaign has demonstrated that a small team with AI inference and good prompts can scan 390 critical projects in a weekend. A malicious actor with the same tools can scan the same projects and keep the vulnerabilities private.
That is the real arms race. It is not Bitcoin versus regulators. It is AI-audited developers versus AI-augmented attackers. The attacker has an advantage because they only need one vulnerability. The defender must find all of them. This campaign has shortened the discovery side for both sides. It has not shortened the exploitation side for defenders. That asymmetry should worry everyone who celebrates the 4,962 number.
Another contrarian point: the campaign may have created a false sense of security precisely because it was so large. A number like 4,962 is reassuring. It suggests that the AI is working. It suggests that someone is watching. But security is not a count of findings. It is a measure of residual risk. After all the findings are triaged and patched, there will still be vulnerabilities that the AI could not find. The existence of a large audit report does not reduce the need for continuous review. If anything, it increases the risk of complacency.
I have seen this dynamic in the NFT market. In 2021, projects with 'audited' badges traded at a premium. Then the audited projects got drained. The audit had been performed by a firm that used pattern-matching tools and missed the one logic flaw that mattered. The same thing will happen with AI audits. A badge saying 'AI audited' will be worse than no badge, because it will create trust without providing safety. The architecture of trust cannot be purchased with a single weekend of scanning. It is built over time, through patches, retests, and honest incident reports.
What Must Come Next
The organizers need to publish the full methodology. That means the model versions, the prompt templates, the repository commit hashes, and the classification rules. The data should be queryable. Anyone should be able to re-run a subset of the audit and compare results. If the methodology is not reproducible, it is not science. It is a press release.
The industry needs a shared, open-source triage format for AI findings. Each finding should include a project, a commit, a file, a line number, a severity rationale, a PoC status, a maintainer response, and a patch recommendation. This would allow thousands of findings to be deduplicated, prioritized, and tracked to resolution. It would also create a public benchmark over time. We could finally answer the question: how many AI findings are real?
The next campaign should be a re-audit, not a first audit. Take the same 390 projects, run the same methodology, and measure what changed. Which findings were fixed? Which were false positives? Which were missed in the first pass? That measurement is the only metric that matters. Without it, we are still in the era of finding counts. With it, we enter the era of security outcomes.
I have been asked whether AI will replace security researchers. My answer has not changed. AI will replace the mechanical parts of the job, and it will expose the judgment parts. This campaign is a perfect demonstration. The mechanical part is reading code and generating candidate issues. The non-mechanical part is deciding what matters, proving it, and fixing it without breaking the system. The AI did the first part. The humans did the second part. The campaign succeeded because of the humans, not despite the AI.
The next narrative in crypto security will not be about finding more bugs. It will be about managing the flood. The winners will be teams that build triage infrastructure, public datasets, and reproducible pipelines. The losers will be projects that treat an AI audit report as a shield.
The Takeaway
We should thank the organizers for pushing the frontier. They have shown that a small team with good prompts and enough compute can scan an entire ecosystem in a weekend. That is a real capability. But a capability is not a safety guarantee. The campaign has shifted the bottleneck from discovery to triage. The next phase is not more AI. It is better process.
The original report was framed as a story about AI finding 5,000 issues. That is not the story. The story is that we do not yet have a system to handle 5,000 findings responsibly. The architecture of trust is built, not inherited. Bitcoin's security has always been an ongoing process of patches, review, and honest accounting. AI can make that process faster. It cannot make it optional.
So here is my question for the organizers, for the maintainers, and for every project that will now claim AI-assisted security: where are the fixes? Show me the commits. Show me the regression tests. Show me the re-audit. That is the only ledger that counts. The findings were the easy part. The hard part starts now.
