GPT-5.6 Sol Edges Claude Mythos 5 in Cyber Tests as Open Models Close the Gap

Introduction

GPT-5.6 Sol ranked slightly above Claude Mythos 5 in two cybersecurity evaluations published by the UK AI Security Institute, or AISI. The result is notable, but it needs to be interpreted carefully.

AISI did not publish a universal ranking of every offensive and defensive cybersecurity capability. It tested models on a defined set of narrow technical tasks and a simulated, long-horizon corporate-network attack. Within those settings, GPT-5.6 Sol achieved the highest average result, while Mythos 5 remained very close.

The larger finding was about open-weight models. AISI found that GLM-5.2 and DeepSeek V4-Pro now perform similarly to leading closed models released only four to seven months earlier. In AISI’s internal 2025 testing, the equivalent gap was six to ten months.

That shrinking delay matters because open-weight models can be downloaded, modified, privately deployed, and run without provider monitoring. The same flexibility that supports research, privacy, and lower costs can also make advanced cyber capabilities harder to control after release.

What AISI Evaluated

AISI used two complementary evaluation systems.

Narrow Cyber Tasks

The first suite measures specific cybersecurity skills through 70 tasks selected from a larger 96-task collection.

The tasks cover four broad areas:

• Vulnerability research and exploitation
• Reverse engineering
• Web exploitation
• Cryptography

They are divided into four difficulty levels based on the estimated experience a human would need:

Difficulty Level	Approximate Human Experience	Number of Tasks
Technical non-expert	Technical background with limited cybersecurity experience	18
Apprentice	Roughly 1–3 years	25
Practitioner	Roughly 3–10 years	19
Expert	More than 10 years	8

Each model received five attempts per task, with a limit of 2.5 million tokens for each attempt. AISI then calculated the average success rate.

Cyber Ranges

The second system measures whether a model can autonomously sustain a multi-step attack in a simulated network.

The main scenario discussed in the report is called The Last Ones. It contains:

• 32 sequential attack steps
• Four network subnets
• Approximately 20 hosts
• Several named milestones
• An estimated 20 hours of work for a human expert

Each main model trajectory represents an average across ten runs, with a 100-million-token limit per run.

These environments do not reproduce every difficulty of attacking a well-defended real organization. AISI notes that the ranges do not currently include active human defenders, defensive tools, or penalties for triggering alerts.

What the GPT-5.6 Sol and Mythos 5 Results Actually Show

On the narrow-task suite, GPT-5.6 Sol recorded the highest average success rate in AISI’s published chart. Claude Mythos 5 was close behind, with overlapping uncertainty ranges.

The chart places GPT-5.6 Sol at a little above 90% average success and Mythos 5 at approximately 90%. Because the uncertainty bars overlap, the result should be described as a narrow lead within this evaluation—not proof that Sol is categorically superior in every cybersecurity setting.

The long-horizon range produced a similar picture.

On average, GPT-5.6 Sol completed roughly 29 of the 32 steps, compared with approximately 27 for Claude Mythos 5. The best attempts from both models completed the entire range.

Several conclusions follow:

1. GPT-5.6 Sol had the highest average trajectory on this specific range.
2. Mythos 5 remained close and also completed all 32 steps in its best attempt.
3. The difference may reflect model capability, agent reliability, tool use, long-horizon planning, or a combination of these factors.
4. One cyber range is weaker evidence than a large, diverse collection of tasks.
5. The results do not establish a universal offensive-security leaderboard.

AISI’s report focuses mainly on the open-versus-closed capability gap rather than declaring an overall winner between Sol and Mythos.

Open-Weight Models Now Trail by Only 4 to 7 Months

AISI selected GLM-5.2 and DeepSeek V4-Pro because they were leading open-weight candidates at the time of testing.

The report’s central comparison is based on models with similar measured performance and the difference between their release dates.

Open-Weight Model	Evaluation	Comparable Closed Model	Estimated Release Gap
GLM-5.2	Narrow cyber tasks	Claude Opus 4.6 and GPT-5.3-Codex	About 4 months
GLM-5.2	The Last Ones cyber range	Claude Opus 4.5	Less than 7 months
DeepSeek V4-Pro	Narrow cyber tasks	Claude Opus 4.5	About 5 months
DeepSeek V4-Pro	Cyber ranges	Below Sonnet 4.5 in the highlighted range	Not treated as a direct frontier match

GLM-5.2 performed comparably to Opus 4.6 across all four narrow-task difficulty levels. On The Last Ones range, it reached the same final average step as Opus 4.5.

DeepSeek V4-Pro matched Opus 4.5 on narrow tasks but performed less strongly on the long-horizon range.

AISI’s 2025 internal tests placed leading open-weight models six to ten months behind the closed frontier. The new estimate of four to seven months indicates that the preparation window for defenders may be getting shorter.

Why the Gap Is Not a Fixed Forecast

The four-to-seven-month figure describes the models and evaluations AISI tested. It is not a prediction that every future open model will remain exactly that far behind.

Several factors could move the gap in either direction:

• A closed developer may make a sudden capability jump.
• An open model may reproduce a recent frontier improvement.
• Better agent scaffolding may unlock more performance from existing weights.
• Fine-tuning and model-specific prompting may change measured capability.
• New benchmarks may expose weaknesses hidden by current tests.
• Deployment cost and available hardware may limit practical misuse.
• Safeguards and access controls may change the usable capability of hosted models.

AISI also says its setup may slightly underestimate the maximum capability of the open-weight models because it did not pursue extensive model-specific elicitation or optimization.

The report applies only to cybersecurity. It should not be used to infer equal gaps in science, coding, general reasoning, or other domains.

Open Models Were Much Cheaper at Similar Capability Levels

The capability gap was small, but the price gap was large.

AISI compared advertised first-party token prices for models with similar performance.

Cost of a 100-Million-Token Cyber Range Run

Model	Approximate Cost
Claude Opus 4.5	$85
Claude Opus 4.6	$85
GLM-5.2	$46
DeepSeek V4-Pro	$1.19

Cost per Narrow Task Solved with 100% Reliability

Comparison	Closed Model Cost	Open-Weight Model Cost
Opus 4.6 vs GLM-5.2	$15.17	$6.12
Opus 4.5 vs DeepSeek V4-Pro	$12.50	$0.28

DeepSeek V4-Pro’s advertised price was one to two orders of magnitude below the comparable closed model in these examples.

The comparison has limitations. AISI did not run the tested open-weight models through their first-party providers, so actual infrastructure expenses may differ. Self-hosting also introduces hardware, engineering, power, networking, and maintenance costs that are not represented by an API price.

Even with those caveats, lower inference costs can make repeated experimentation, fine-tuning, and private deployment more accessible.

Why Open Weights Change the Safety Problem

Open-weight models offer real benefits:

• Private deployment
• No requirement to send data back to the original provider
• Customization for specialized tasks
• Independence from provider deprecation
• Reproducible research
• Inspection and modification of model weights
• Collaboration across organizations

The same properties limit the safeguards available after release.

A closed-model provider can potentially:

• Monitor unusual patterns of use
• Apply classifiers
• Suspend accounts
• Rate-limit access
• Update safeguards
• Restrict certain tools
• Withdraw or replace a model

Once model weights are public, copies can be redistributed and run privately. Refusal behavior can be modified, deployment monitoring can be removed, and the original developer cannot reliably withdraw every copy.

AISI found that safeguards did not materially block most of its testing of the two open models. DeepSeek V4-Pro occasionally refused reverse-engineering tasks, but a small number of retries generally bypassed those refusals.

This does not mean all open-weight models will be used maliciously. It means that release decisions become difficult to reverse when a model reaches risk-relevant capability levels.

Closed Models Also Need Stronger Safeguards

Closed access is not a guarantee of safety.

Anthropic released Claude Fable 5 and Claude Mythos 5 on June 9, 2026. Fable 5 used stronger classifiers for general access, while Mythos 5 exposed more of the underlying model’s cybersecurity capability to a limited group of trusted defenders.

On June 12, U.S. export controls required Anthropic to suspend access. Anthropic said the directive followed a report from Amazon researchers describing a method that bypassed Fable 5’s safeguards in a limited cybersecurity scenario.

Anthropic later trained an updated classifier, worked with government and industry partners on a jailbreak-severity framework, and restored Fable 5 globally on July 1 after the restrictions were lifted.

The episode illustrates several points:

• Classifiers can produce both false negatives and false positives.
• Jailbreaks vary greatly in severity.
• A bypass does not necessarily reveal a model’s most dangerous capability.
• Closed providers can update safeguards and suspend access.
• Those interventions can also block legitimate security work.
• Industry and governments still lack consistent standards for judging cyber jailbreaks.

Closed deployment provides more intervention options, but those options still require effective monitoring, testing, policy, and technical controls.

The Defensive Preparation Window Is Shrinking

AISI describes the open-versus-closed gap as preparation time.

If the most capable systems remain controlled for several months before equivalent open weights are released, defenders can use that period to:

• Find and patch vulnerabilities
• Improve asset inventories
• Remove unsupported software
• Strengthen identity controls
• Deploy phishing-resistant authentication
• Improve network segmentation
• Expand logging and detection
• Test incident-response plans
• Apply AI-assisted code review and threat analysis

A reduction from six-to-ten months to four-to-seven months means organizations may have less time to complete this work before similar capabilities become cheaper and more widely available.

The UK National Cyber Security Centre has warned that AI will amplify both strong and weak security practices. Its guidance emphasizes that AI tools cannot compensate for poor foundations.

Organizations with weak patch management, exposed services, reused credentials, incomplete backups, and limited monitoring will remain vulnerable regardless of which AI model is used by an attacker.

AI Is Also Accelerating Defensive Security Work

The same capabilities used in offensive evaluations can help defenders audit code, generate tests, investigate failures, and scale vulnerability research.

A recent example came from game-networking developer Glenn Fiedler, who used Claude Code with Claude Fable 5 to audit four established open-source libraries:

• netcode
• reliable
• serialize
• yojimbo

The audit added libFuzzer targets, sanitizer-based continuous integration, multi-million-iteration stress tests, golden wire-format tests, and line-by-line review.

The public bug record documents 43 fixes:

Library	Total Fixes	Network-Reachable Fixes
netcode	7	2
reliable	7	6
serialize	11	7
yojimbo	18	12
Total	43	27

The most severe issue was a remotely reachable heap overflow in yojimbo that had existed since 2019. A crafted block fragment could be copied into a reassembly buffer before its size was validated.

Each listed issue links to the fixing commit and release, and every fix has a regression test. The developer reported spending more than $2,500 on Claude Code during the two-week repair effort.

Users of the affected libraries were advised to upgrade to the latest releases specified in the public bug record.

This case does not prove that an AI agent can independently secure arbitrary software. It shows that a knowledgeable maintainer can use AI to expand the volume of fuzzing, sanitizer coverage, review, and test generation applied to an existing codebase.

Offensive and Defensive Acceleration Is Asymmetric

AI can help both sides, but the deployment pattern is different.

An attacker needs one usable vulnerability, one accessible target, and one successful path. An open-weight capability can be copied and reused repeatedly after release.

Defenders must protect many systems, maintain inventories, prioritize patches, test changes, manage downtime, and coordinate across teams. Defensive improvements have to be implemented organization by organization.

This creates an asymmetry:

• Offensive capability can spread quickly.
• Defensive capability must be operationalized locally.
• A model release can happen once.
• Security updates must be deployed many times.
• Attackers can choose the weakest target.
• Defenders must cover every critical path.

The appropriate response is not to avoid defensive AI. It is to combine it with strong engineering practices, human expertise, and verified security controls.

How Organizations Should Respond

The AISI and NCSC findings support a practical set of actions.

1. Strengthen the Baseline First

Prioritize:

• Asset inventory
• Supported software
• Rapid patching
• Multi-factor authentication
• Least-privilege access
• Network segmentation
• Tested backups
• Centralized logging
• Incident-response exercises

AI-enhanced defense works best when these foundations already exist.

2. Expand Continuous Security Testing

For software projects, introduce:

• Fuzz testing
• AddressSanitizer
• UndefinedBehaviorSanitizer
• MemorySanitizer where supported
• Static analysis
• Dependency scanning
• Regression tests for every confirmed bug
• Security review of untrusted input paths

The Mas Bandwidth audit is useful because it combined AI review with mechanical evidence rather than trusting generated explanations.

3. Treat AI Findings as Leads, Not Proof

An AI-generated vulnerability report may be correct, incomplete, or misleading.

Require:

1. Reproduction
2. Root-cause analysis
3. Human review
4. A minimal fix
5. Regression tests
6. Release documentation
7. Coordinated disclosure where appropriate

4. Restrict Agent Permissions

A defensive coding agent should not receive unrestricted access by default.

Limit:

• Production credentials
• Network access
• Destructive shell commands
• Repository write scope
• Secret access
• Deployment privileges
• External communication

Use isolated environments and keep consequential actions behind human approval.

5. Measure Cost and Effectiveness

Track:

• Confirmed vulnerabilities
• False positives
• Time to reproduce
• Time to fix
• Test coverage added
• Token and infrastructure cost
• Human-review time
• Security regressions after deployment

The cheapest model is not necessarily the most cost-effective if it produces unreliable findings or requires extensive review.

How to Interpret AI Cyber Benchmarks

Cybersecurity benchmarks are useful, but their scope must remain visible.

A High Score Does Not Equal Real-World Compromise

A simulated range simplifies the environment. Real networks may contain active defenders, endpoint protection, alerting, rate limits, deception systems, and incomplete information.

Model and Agent Capability Are Intertwined

Long-horizon results depend on more than model knowledge. Tool design, memory, context management, retry strategy, prompt structure, and execution reliability all affect performance.

Average and Best-Attempt Results Answer Different Questions

A best attempt shows what the system can sometimes do. An average shows how reliably it progresses across repeated runs.

For operational risk, both matter.

Release-Date Comparisons Are Approximate

A model that performs similarly to an older model is not identical to it. The systems may have different strengths, weaknesses, safeguards, costs, and deployment constraints.

Cyber Capability Is Dual Use

The same skill can support penetration testing, secure-code review, incident response, vulnerability discovery, or malicious intrusion.

Evaluation results should inform both risk management and investment in defensive access.

常见问题

Did GPT-5.6 Sol outperform Claude Mythos 5 in AISI’s tests?

GPT-5.6 Sol achieved a slightly higher average result on AISI’s narrow cyber tasks and the highlighted long-horizon range. Mythos 5 remained close, the uncertainty ranges overlap on the narrow suite, and both models completed all 32 range steps in their best attempts.

Does that mean GPT-5.6 Sol is the best cybersecurity model?

Not necessarily. AISI tested defined task suites and simulated environments, not every real-world defensive or offensive scenario. The results support a narrow benchmark comparison rather than a universal ranking.

How far are open-weight models behind closed frontier models?

AISI estimated a four-to-seven-month gap for the leading open-weight models it tested. The figure is based on matching measured performance with the release dates of comparable closed models.

Which open-weight model performed best?

GLM-5.2 was the strongest open-weight model in AISI’s testing at the time of the report. It matched Opus 4.6 on narrow tasks and reached the same average point as Opus 4.5 on the highlighted cyber range.

Why are open-weight models a cybersecurity concern?

Their weights can be downloaded, modified, privately hosted, and redistributed. That provides research and privacy benefits, but it also limits the original developer’s ability to monitor misuse, update deployment safeguards, suspend users, or withdraw every copy.

Were open models cheaper in AISI’s evaluation?

Yes. At advertised first-party prices, GLM-5.2 and especially DeepSeek V4-Pro were substantially cheaper than comparable closed models. Actual self-hosting and third-party deployment costs may differ.

Can AI models help defenders as well as attackers?

Yes. AI can assist with code review, fuzz-test creation, vulnerability analysis, log investigation, and remediation. The Mas Bandwidth audit documented 43 fixes across four networking libraries after a Claude Code-assisted security effort.

What should organizations do first?

Strengthen basic security controls before relying on AI tools. Asset visibility, patching, identity protection, backups, segmentation, monitoring, and tested incident response remain essential.

相关工具

• AISI Inspect Cyber: An evaluation framework for creating and running agentic cybersecurity tasks.
• Inspect AI: AISI’s open-source framework for evaluating large language models.
• GLM-5.2: The official open-weight model repository from Z.ai.
• DeepSeek API: Official documentation and current access information for DeepSeek models.
• Claude Code: Anthropic’s agentic coding environment for repository work, testing, and security review.
• NCSC Cyber Essentials: A UK-backed baseline for protecting organizations against common cyber threats.

Related Links

• AISI Open-Weight Cyber Gap Report: The official July 2026 analysis, methodology, charts, cost comparisons, limitations, and conclusions.
• Inspect Cyber Introduction: AISI’s explanation of its open framework for agentic cyber evaluations.
• AISI Frontier AI Trends Report: Broader evidence on the progression of frontier-model capabilities, including cybersecurity.
• NCSC Guidance for Frontier AI Cyber Risk: Guidance on security baselines and carefully deployed AI-enhanced defense.
• Anthropic’s Fable 5 Redeployment Report: Anthropic’s account of the June 2026 suspension, classifier update, and restored access.
• Mas Bandwidth Security Bug Record: A verifiable list of 43 fixes with commits, severity, network reachability, and release information.
• Mas Bandwidth Upgrade Notice: The maintainer’s notice advising users to upgrade the affected networking libraries.

Summary

AISI’s latest results place GPT-5.6 Sol slightly ahead of Claude Mythos 5 on the average scores of two specified cybersecurity evaluations. The lead is narrow and should not be interpreted as proof of universal superiority.

The report’s main conclusion is broader: GLM-5.2 and DeepSeek V4-Pro now trail comparable closed frontier models by only four to seven months, down from six to ten months in AISI’s 2025 internal testing. They were also much cheaper at advertised prices.

That shorter delay reduces the time defenders may have before advanced capability becomes easier to download, modify, and run privately. At the same time, Claude Code’s role in finding 43 bugs across four networking libraries shows that powerful models can also accelerate defensive work.

The practical response is not to rely on model safeguards or AI security tools alone, but to strengthen the security baseline and use AI inside a controlled, test-driven, human-reviewed defensive process.