A penetration test at a financial services company that manages billions of dollars in client assets found a critical flaw in AI-generated code. The interesting part is what the code did right.
Incident response company Sygnia evaluated a customer onboarding application built substantially with Claude. It handled government-issued IDs, identity verification data, payment details, and Social Security numbers.
The flaw allowed one applicant to catch up to another applicant’s history.
what really broke
The app solved a real problem. People start a request, abandon it, and return before they have an account, so the system must restore their progress without a password.
Claude put together a temporary access model for this. The flaw was in how the model decided who deserved a token.
The 💜 of EU technology
The latest rumors from the EU tech scene, a story from our wise founder Boris and some questionable AI art. It’s free, every week, in your inbox. Sign up now!
Possession of a requester’s globally unique identifier, or GUID, is considered sufficient evidence to issue or restore an access token for that requester.
“In effect, the GUID became a carrier secret,” Sygnia wrote in your research.
A GUID identifies a record. It does not prove that whoever owns it controls the session, device, identity, or attached verified contact channel.
Anyone with another applicant’s GUID could access names, contact details, application status, financial details, identity verification data, payment information, Social Security numbers, and co-applicant records.
The co-applicants deserve a pause. Those are people whose data was left in the system because someone else named them.
All the controls were there.
This is what makes this finding unusual, and it’s not the story most AI coding failures tell.
The implementation did not lack security. Sygnia outlines a plan that covers temporary tokens, expiration, rate limiting, audit logging, session restoration, and suspicious activity detection.
Each one of them is a real control. None of them answer the question the system really needed to answer.
“What should be tested before the backend issues or restores a requester access token?” the researchers ask.
The vulnerability existed before issuance, at the time when the system decided whether the applicant deserved a token.
“The problem was not that the system lacked security controls,” Sygnia wrote. “The problem was that the controls revolved around the wrong trust decision.”
A short-lived token still causes damage when it reaches the wrong person. Audit logs help with further investigation and do not impede anything.
Why do scanners miss this?
Static application security testing tools look for insecure coding patterns and insecure data flows. This was neither of those things.
The defect was architectural. It lived within an assumption about trust, written in code that followed familiar framework conventions and passed basic checks.
“Working code is not the same as secure code,” said Zach Mead, the Sygnia penetration tester who performed the evaluation.
“AI-generated code can compile, follow familiar conventions, and pass basic checks, while still making erroneous assumptions about trust boundaries, authorization, state, ownership, or third-party integrations,” he said.
His instructions to the security teams are forceful. Treat AI-generated output as untrusted until validated.
An LLM found what an LLM built
Sygnia didn’t catch this by hand. Ran a language model on parts of the client’s codebase, looking for trust boundary issues and business logic flaws.
The model directly brought the issue of the applicant token to light.
Sygnia’s summary of this is the clearest line of research. A flaw in the vibration-coded security architecture was identified by reviewing the vibration-coded code.
It goes both ways, and the company says so. If a defender can ask a model where the trust boundaries are missing, so can anyone with leaked source code or stolen implementation notes.
Cisco has been targeting open weight models to hunt for bugs for the same reason. One capacity, two uses.
The problem of rhythm
This investigation is set alongside a second Sygnia case, and the pair carry the plot.
In a cloud intrusion the company investigated, the first sign was not an exploit. It was the rhythm.
Credential discovery, cloud enumeration, source code review, CI/CD polling, database access, and operational disruption were executed at the same time. It seemed like several operators instead of one.
He was a person who worked with agents.
Put the cases together and the shape becomes clear. AI is accelerating how code is written, how flaws are found, and how quickly a weakness becomes a path of attack. We have already tracked four separate attacks that share a single underlying flaw.
What Sygnia tells its clients to do
Do not prohibit tools. Bans push usage to personal accounts, where no one can see it.
The firm suggests marking pull requests that were generated or substantially modified by AI, naming the tool, and marking work that affects authentication, payments, data exports, or regulated customer information.
The most specific advice refers to the indications. Requesting “temporary access tokens for applicants” is an incomplete instruction.
A more secure message states that requester identifiers should not be treated as secrets, that token issuance requires proof independent of the requester’s control, and that negative evidence must demonstrate that one requester cannot gain access for another.
The security requirement must be included in the message, because the model alone will not infer an institution’s risk model.
A warning worth mentioning
This is a vendor report about an anonymous customer, published alongside a new business service.
signia launched AI cybersecurity services same day, covering posture assessment, governance and penetration testing of AI applications. No one can independently verify the findings and no other company has reported this case.
However, the mechanism he describes is testable and echoes our previous coverage of security flaws in vibration coding, sandbox escapes in encoding agents and quick injection into a GitHub action by Claude Code.
What separates this one is where the defect was. Those were vulnerabilities in the tools. It was a code generated by AI that took the failure to production, in a regulated company, around the data that clients must deliver.
