A cyber agent crossed the test boundary. Prompts are not scope controls.
Gemini reached real company systems during an authorised cyber evaluation after internet access and target identity escaped the test boundary. Agent evaluations need enforceable network scope, not prompt-level assumptions.
A Gemini cyber agent reached systems belonging to three real companies during an authorised evaluation in May 2026, according to reporting confirmed by Google. The failure was not that the model ignored its objective. The evaluation environment let a fictional target name collide with real infrastructure while internet access remained available, so the agent treated reachable systems as part of the exercise.
That distinction matters for every enterprise testing autonomous agents. A prompt can describe scope, but it cannot enforce scope. If an agent can resolve, connect to and authenticate against an unintended system, the effective authority boundary is the network and credential architecture around it.
Evaluation scope has to exist below the model
Irregular's FrontierCyber methodology is built around real systems, fixed objectives and controlled evaluation infrastructure. Its published design calls out network boundaries, authentication, sandboxing, instrumentation and repeatable resets as parts of the benchmark environment. Those controls are not supporting details. They are what turn offensive capability testing into an authorised experiment.
An agent's written scope is intent. Its reachable infrastructure is authority.
The May incident shows the failure mode when those two definitions diverge. A model may remain task-directed and still cause an unauthorised action because the environment makes an unintended asset look legitimate. Human red teams solve this with rules of engagement plus technical controls. Autonomous agents need the same separation, with less tolerance for ambiguity because they can enumerate and act faster than a supervisor can review each step.
Build a deny-by-default evaluation plane
The production control is concrete. Cyber-evaluation networks should resolve only approved target names, restrict outbound traffic to explicit destinations and expose synthetic credentials that are useless outside the lab. Target identifiers should not depend on names that can collide with real organisations. Sensitive actions should generate immutable telemetry and a policy decision outside the model should terminate sessions that cross an approved asset boundary.
This also changes how evaluation results should be interpreted. A successful exploit score measures model capability only if the harness can prove where the model acted. Otherwise the benchmark mixes capability with environmental leakage and creates risk for third parties.
Treat the harness as a security product
Teams adopting offensive agents should threat-model the evaluation harness with the same discipline applied to production orchestration: egress policy, DNS control, credential scoping, target allowlists, auditability and an independent kill path. Model safeguards remain useful, but they are a final behavioural layer, not the containment mechanism.
The practitioner lesson is not to stop testing capable cyber agents. It is to make authorisation machine-enforceable. When autonomous systems can act at network speed, scope has to be encoded in infrastructure before it is described in natural language.
Sources
Written by the Devence Lab research team.
