AI-Generated Code Security Study 2026

I gave 6 large language models 87 coding prompts each — building login forms, handling file uploads, querying databases — without ever mentioning security. Then I scanned all 522 code samples with 5 SAST tools (four open-source plus CodeQL) and validated every finding.

About one in four samples contained at least one confirmed vulnerability, and the gap between the safest and least safe model was about 10 percentage points.

Prior research from New York University (2021) found that about 40% of code generated by GitHub Copilot contained security vulnerabilities across 89 test scenarios. My study extends that work to 2026-era models across a wider prompt set, using the OWASP Top 10:2025 as the vulnerability taxonomy.

Key findings#

Which model generated the safest code?#

GPT-5.2 generated the safest code in this study, with 19.5% of its samples containing at least one confirmed vulnerability. Grok 4 came in second at 21.8% and Gemini 2.5 Pro third at 23.0%. The three weakest performers — Claude Opus 4.6, DeepSeek V3, and Llama 4 Maverick — all tied at 29.9%, about a 10-point gap behind GPT-5.2. Across the 522 total samples, the overall vulnerability rate was 25.7%, meaning roughly one in four model outputs shipped at least one OWASP-mapped flaw before any human review. The dominant category by far was OWASP A01:2025 Broken Access Control with 65 findings — driven primarily by path traversal and server-side request forgery, which OWASP 2025 consolidated into A01. Injection (A05) and Mishandling of Exceptional Conditions (A10) tied at 22 findings each. None of the six models produced security-clean code in more than 80% of samples, so even the strongest performer cannot replace SAST or human review on production code paths.

Vulnerability rate by model#

How often does each LLM produce code with at least one confirmed vulnerability? The chart below shows the percentage of samples from each model that contained a true positive after validation.

Claude Opus 4.6, DeepSeek V3, and Llama 4 Maverick all produced vulnerable code in 29.9% of samples — tied for the worst result. Then there’s a gap: Gemini 2.5 Pro (23.0%), Grok 4 (21.8%), and GPT-5.2 (19.5%) all came in under 24%.

GPT-5.2 had the lowest rate at 19.5%. The ~10-point spread between the best and worst models is hard to ignore — your choice of LLM has a measurable effect on code security, even when every model gets the same prompt.

OWASP category breakdown#

Which OWASP Top 10 categories trip up each model the most? The heatmap below shows confirmed finding counts per model, sorted by total. Darker cells mean more vulnerabilities.

Broken Access Control (A01) dominated with 65 findings — driven by path traversal and SSRF, both of which OWASP 2025 places under A01. Injection (A05) and Mishandling of Exceptional Conditions (A10) tied at 22 findings each — A10 driven mostly by Flask debug mode left on. Together these three categories account for roughly 70% of confirmed findings.

Security Logging and Alerting Failures (A09) , Cryptographic Failures (A04) , and Software Supply Chain Failures (A03) all surfaced zero findings — A09 sits in a well-known SAST blind spot, and the test set wasn’t designed around supply-chain or pure crypto attack patterns, so those categories are undersampled by design.

SSRF specifically is the interesting cell here — five of the six models produced 5-6 vulnerable samples on those prompts. GPT-5.2 was the exception at 4. With only 8 SSRF prompts per model, the 1-point gap sits at the noise floor — not a strong signal.

Python vs JavaScript#

Do LLMs generate safer code in one language over the other? Here are the vulnerability rates split by language for each model.

There is no universal “safer language” — it depends on the model. GPT-5.2 did dramatically better in Python (11.6%) than JavaScript (27.3%), a 15.7-point gap.

Gemini 2.5 Pro showed a similar pattern: 18.6% Python vs 27.3% JavaScript. Claude Opus 4.6 was the only model where Python was actually worse (32.6% vs 27.3%).

Grok 4 had the tightest cross-language gap at just 1.8 points (20.9% Python, 22.7% JavaScript), with DeepSeek V3 next at 3.9 points (27.9% Python, 31.8% JavaScript). The wide spreads for GPT-5.2 and Gemini suggest their security training data may lean more toward Python.

Most common vulnerabilities#

Across all models and languages, which specific weaknesses show up most? Here are the top 10 CWEs by total confirmed findings.

SSRF (CWE-918) leads with 32 confirmed findings — LLMs routinely pass user-supplied URLs directly to fetch operations without validation. Path traversal (CWE-22 and CWE-23) follows at 30. Flask debug mode left on — labeled CWE-215 by CodeQL and CWE-489 by OpenGrep for the same underlying issue — accounts for 18 findings. Deserialization of untrusted data (CWE-502) sits at 14, and NoSQL injection (CWE-943) at 10.

Injection-pattern weaknesses (SQL/NoSQL/OS command/code injection, path traversal, and SSRF) account for 78 of 154 total findings — roughly half. Note that OWASP 2025 spreads these across A01 (SSRF and path traversal, now under Broken Access Control) and A05 (classical injection); grouping them by injection mechanism here is a cross-OWASP description, not the heatmap classification. The recurring secondary theme is insecure defaults: Flask debug left on, cookies missing secure/HttpOnly flags, and hardcoded credentials.

Command injection (CWE-78) dropped significantly after deep triage — many flagged subprocess calls used list form without shell=True, which is not exploitable. The pattern is clear: LLMs write code that works first. Security comes second, if at all.

Model comparison deep dive#

Here’s how each model performed across categories, languages, and severity levels.

GPT-5.2#

GPT-5.2 had the lowest vulnerability rate at 19.5% (17 of 87 samples, 20 total findings). It had only 1 authentication finding (A07) and the lowest SSRF count at 4 — the only model under 5 for that category. Its A01 total was 9 (the smallest among the six models).

The language split is the widest in the study: 11.6% in Python vs 27.3% in JavaScript, a 15.7-point gap. GPT-5.2’s Python output more often used subprocess list form, parameterized queries, and explicit input validation. Its JavaScript more frequently missed input sanitization on HTTP request parameters, but still outperformed most other models.

Claude Opus 4.6#

Claude Opus 4.6 tied for the highest vulnerability rate at 29.9% (26 of 87 samples) with 29 total findings. It scored high in A01 Broken Access Control (12), A10 Mishandling of Exceptional Conditions (6) — mostly Flask debug — and A05 Injection (4).

Unusually, Claude’s Python rate (32.6%) was higher than JavaScript (27.3%) — the opposite of most models. Its code frequently shipped with debug mode on and no input validation on server-side parameters.

Gemini 2.5 Pro#

Gemini 2.5 Pro came third-best at 23.0% (20 of 87 samples, 23 total findings). It had 0 findings in A03 Software Supply Chain, A04 Cryptographic Failures, A06 Insecure Design, and A09 Logging. Its A01 total was 11 and A05 Injection 4.

Language split: 18.6% in Python vs 27.3% in JavaScript. Gemini’s Python code more often used parameterized queries and proper subprocess input handling. Its JavaScript occasionally missed output encoding in template rendering.

DeepSeek V3#

DeepSeek V3 tied for the highest rate at 29.9% (26 of 87 samples) with 30 total findings — the highest raw count. It had broad spread across A01 (12, mostly path traversal and SSRF), A05 Injection (5), and A10 Mishandling Exceptional Conditions (5).

Language rates were 27.9% Python and 31.8% JavaScript — a 3.9-point gap. DeepSeek’s code frequently used eval(), unsanitized string concatenation in queries, and debug configurations on by default.

Llama 4 Maverick#

Llama 4 Maverick also tied at 29.9% (26 of 87 samples, 31 total findings). It had the most A07 Authentication Failures of any model (5), and the broadest spread overall — 10 A01, 4 A05, 4 A02, 4 A08, 3 A10.

Llama had an 8.5-point language gap: 25.6% Python vs 34.1% JavaScript. Its JavaScript particularly struggled with authentication token handling and cookie security. As an open-weight model, these results matter for teams running self-hosted inference.

Grok 4#

Grok 4 came second-best at 21.8% (19 of 87 samples, 21 total findings). It had only 1 finding in A07 Authentication Failures, 2 in A05 Injection, and 4 in A10 Mishandling Exceptional Conditions. Its A01 total was 11.

Grok had the most consistent cross-language numbers in the study: 20.9% Python, 22.7% JavaScript — just 1.8 points apart. Its code more often included input validation and avoided debug defaults.

Tool agreement analysis#

When multiple SAST tools flag the same code, how often do they agree? Tool consensus is a decent confidence signal — a vulnerability caught by three tools is more likely real than one flagged by just one.

59.5% of confirmed vulnerabilities (91 of 153 SAST-attributed TPs) were flagged by only a single tool. That’s how SAST tools work — each has its own rule engine, language parser, and detection patterns. 40 findings (26.1%) were caught by two tools, and 22 (14.4%) by three or more.

This is exactly why running multiple SAST tools matters. A single tool would have missed a large chunk of the true positives found here. The low overlap also helps explain the high false positive count (772 total) — tools routinely flag patterns that other tools consider benign.

Prompt examples#

Here are 3 of the 87 prompts used in this study. Each one describes a routine development task without mentioning security, vulnerabilities, or best practices — the kind of request a developer might type into ChatGPT or Copilot during normal work.

All 87 prompts are split between Python (43) and JavaScript (44). The prompts were authored against the OWASP Top 10:2021 categories (A01–A10) — that was the current release when data was collected. Findings on this page are remapped to OWASP Top 10:2025 using the canonical CWE → category lookup, so the category names and rankings reflect the current edition.

Methodology#

Here’s exactly how I designed, collected, and analyzed this data.

Prompt design. I wrote 87 coding prompts that describe realistic development tasks — building a login form, querying a database, handling file uploads, processing user input — without mentioning security, vulnerabilities, or best practices. Each prompt maps to one or more OWASP Top 10 categories. The point: test what LLMs produce when developers ask for functional code without explicitly requesting secure code.

Prompts cover all 10 OWASP Top 10 categories across both Python and JavaScript. The prompt directories were named against OWASP Top 10:2021 since that was the current release at collection time; findings are remapped to OWASP Top 10:2025 in this report. Each prompt asks for a self-contained code snippet that a developer might reasonably request during day-to-day work.

Code collection. All 6 models were accessed through the OpenRouter API using a single unified endpoint. OpenRouter routes requests to each provider’s API, which let me send identical payloads (same prompt, same parameters) across all models without managing 6 separate API integrations. I sent each prompt to:

• GPT-5.2 (OpenAI)
• Claude Opus 4.6 (Anthropic)
• Gemini 2.5 Pro (Google)
• DeepSeek V3 (DeepSeek)
• Llama 4 Maverick (Meta)
• Grok 4 (xAI)

All models were called with temperature=0 (or the lowest available setting) to minimize sampling variance. Each prompt was sent once per model. Two prompts were excluded as out of scope for this analysis, leaving 87 prompts × 6 models = 522 code samples.

I extracted only the code blocks from each response, discarding explanatory text.

API costs. The entire study cost under $10 in OpenRouter credits. Claude Opus 4.6 was the most expensive model at $3.67, while open-weight models like DeepSeek V3 ($0.02) and Llama 4 Maverick ($0.02) were essentially free. The cost breakdown shows that running security research on AI-generated code is accessible to anyone.

Scanning tools. Every code sample was scanned with 5 SAST tools (four open-source plus CodeQL):

All tools were run with their built-in defaults: Bandit with --severity-level all, OpenGrep with --config auto (community rulesets pulled at run time, snapshot Feb 2026), ESLint with the security plugin’s recommended flat config, njsscan with default rules, and CodeQL with its default code-scanning query suite per language.

Validation. Every finding from every tool was reviewed and classified as true positive (TP) or false positive (FP). Out of 926 deduplicated findings, 153 were confirmed as SAST TPs and 772 as FPs, plus one manual TP from later review.

A finding counts as TP if the flagged code would be exploitable in a realistic deployment context. Borderline cases (e.g., missing input validation that might be handled by a framework) were classified as FP to keep results conservative.

Two triage passes corrected 19 findings (e.g., subprocess calls using list form without shell=True, properly implemented AES-256-GCM flagged as weak crypto, placeholder credentials, CWE misclassifications by SAST tools, SSRF findings on code with comprehensive IP blocklists).

Deduplication. When multiple tools flag the same line for the same underlying issue, I count it as one finding. The tool agreement analysis tracks how many distinct SAST tools independently flagged each unique finding.

OWASP mapping. Each confirmed finding is mapped to its OWASP Top 10:2025 category via the underlying CWE. The heatmap and category counts on this page use the 2025 mapping; the dataset preserves the prompt’s original 2021-era directory so readers can compare both views.

Limitations.

• Temperature=0 produces deterministic output for most models, but some providers apply post-processing that can introduce minor variation between runs. I did not run multiple iterations.
• Prompts are written in English. LLM behavior may differ for prompts in other languages.
• I test isolated code snippets, not full applications. A vulnerability in a snippet might be mitigated by framework-level protections in a real project. Conversely, integration issues between snippets are not captured.
• SAST tools have known blind spots. Some vulnerability classes (logic flaws, race conditions, business logic errors) are difficult or impossible for static analysis to detect. My findings undercount these categories.
• The 6 models represent a snapshot in time. Model providers frequently update their systems, and results may differ for earlier or later versions.
• I used the SAST tools’ built-in defaults. Custom rules or stricter configurations would likely produce more findings.

Update — May 2026. Routine re-audit. I tightened the deduplication rule and remapped the findings to OWASP Top 10:2025. Two prompts were dropped as out of scope, leaving 87 prompts × 6 models = 522 samples. Overall rate moved from 25.1% to 25.7%; per-model ranking unchanged. One additional process-control finding was added after manual review.

References#

1. OWASP Foundation. OWASP Top 10:2025 . The current vulnerability taxonomy used for finding classification on this page. Prompts were originally authored against OWASP Top 10:2021 (the edition current at collection time); findings are remapped to 2025 via the underlying CWE.
2. MITRE Corporation. Common Weakness Enumeration (CWE) . Used for individual finding classification and deduplication.
3. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2021). Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions . New York University. Found that about 40% of Copilot-generated programs contained vulnerabilities.
4. OpenGrep Project. OpenGrep SAST Scanner . Open-source static analysis with community rulesets.
5. GitHub Security Lab. CodeQL Analysis Engine . Semantic code analysis for vulnerability detection.

Frequently Asked Questions