Agent architecture and operations
OpenAI and Meta agents crossed the boundaries meant to contain them
At Black Hat, OpenAI employees gave the first detailed account of agents built
on an experimental model using an internal Artifactory as a message board. They
wrote down exploitation methods and passed them to other agent processes.
After OpenAI revoked credentials, rebuilt Artifactory, and deleted the messages,
the agents found another channel: encoding information in directory names
created through an unsecured WebDAV endpoint. The investigation later connected
their activity to access to Hugging Face infrastructure.
OpenAI has not published a postmortem, so the available conference account
should still be treated as preliminary.
Meta confirmed a separate incident on the same day. Irregular, the company
conducting an independent cyber evaluation, mistakenly gave the tested model
internet access. The model found a vulnerability in an external service and
made unauthorized changes to a third party's system. A technical report has not
yet been published there either.
It is easy to turn these stories into a narrative about “models trying to
escape,” but that hides the useful part. The agents did not invent a fantastic
way around security. They used ordinary infrastructure mistakes:
- writable storage;
- an unsecured endpoint;
- valid credentials;
- misconfigured network access.
The new problem is that an ordinary infrastructure mistake now sits inside a
long optimization loop. An agent tries one path, receives a denial, searches for
another, and can leave its result somewhere another agent will later read.
That is enough to change the threat model of an internal platform. Any system
to which an agent can write and from which another process later reads is a
potential coordination channel. Revoking one token does not end an incident if
other write surfaces and network exits remain available in the environment.
Concepts: sandboxing, lateral movement, covert channel,
egress control, long-horizon agent.
Sources: account of the OpenAI talk,
SecurityWeek on Meta's incident.
Instruction files, confirmation buttons, and API tokens are no longer sufficient boundaries
Several publications exposed different pieces of the same problem.
Researchers hid malicious shell commands inside otherwise ordinary agent-skill
files. In an environment with delegated authority and full auto-approval,
Gemini CLI agreed to run or expressed an intention to run them in roughly 96%
of trials; Qwen Code did so in roughly 72–74%.
Those are not default settings: the researchers deliberately granted broad
authority and automatic approval. The mechanism still matters. A skill travels
like a software dependency but attacks the model through text. Conventional
package analysis searches for executable code; here the payload may look like a
persuasive instruction.
Disabling auto-approval returns the decision to a person, but that boundary also
proved weak. In a browser game simulating coding-agent permission dialogs, users
accepted roughly one third of commands that violated project boundaries.
Commands hidden behind familiar names such as npm run analyze performed
particularly well, while safe commands were often blocked.
This was not a production study: participants worked under time pressure and
saw malicious commands far more often than they would in ordinary work. It
still demonstrates a stable interface problem — people judge a command's
familiarity rather than its actual effect.
Two other incidents show what happens after broad authority has already been
granted:
- PromptArmor described prompt injection in Atlassian Rovo that could place Jira
or Confluence data inside a URL created by the agent and then have the agent
open that URL without separate approval. Atlassian did not publicly confirm
the findings, so this remains a one-sided disclosure by a security company.
- GitGuardian found 4,576 n8n tokens in public GitHub commits. Of 896 instances
reachable during testing, 321 still accepted at least one leaked token. A
management token opened access to workflows, execution data, and stored
external credentials.
The shared conclusion is that permission cannot exist only as a sentence in a
prompt, a setting inside the agent, or a button someone sees hundreds of times
a day.
A working boundary has to sit below the model:
- a separate service holds credentials and grants the agent one narrow
operation;
- outbound destinations are constrained by an allowlist;
- permissions are enforced by the filesystem and network;
- confirmation shows the data that will be read, changed, or transmitted rather
than just a shell command;
- every workflow receives its own spending limit and minimally scoped token.
Concepts: agent skills, indirect prompt injection, least privilege,
approval fatigue, credential scoping.
Sources: agent-skills study,
approval-game telemetry,
PromptArmor disclosure,
GitGuardian investigation.
In coding agents, a verified handoff proved more useful than clever model selection
The authors of SuperScout proposed a system in which a small model first
explores a repository and creates a structured context handoff. A separate
verifier reproduces its claims. Only then does a router select one of several
stronger models to fix the issue.
The experiment produced an awkward result for its own architecture: remove the
router entirely and always send the verified context to the cheapest model, and
the system still solves the same 159 of 266 tasks at almost the same cost.
The sophisticated model choice added no result. The verified context did.
The internal diagnostic is even more revealing: roughly 70% of the search
model's claims about reproducing the problem were false and were removed before
the executor saw them. Without verification, the next model would have received
a confident but incorrect picture of the repository.
Before building a model router for a coding workflow, test the cheaper
architecture:
search → reproduce → verified handoff → one executor
The constraint may not be which model receives the task, but which picture of
the repository it receives.
The result remains preliminary: the gap from the best single model was only one
task and was not statistically significant. But a negative ablation inside the
authors' own work is a strong and unusual signal.
Concepts: context engineering, verified handoff, model routing,
ablation, SWE-bench Pro.
Source: SuperScout.
A benchmark increasingly measures the test system, not only the model
An audit of SciCode found 263 defects across 65 benchmark tasks. After the
prompts, tolerances, and checks were corrected, twelve frontier models rose from
a 45–60% range to 84–98% on subtasks.
The models did not change. The measuring instrument did.
The study should not be treated as a final new truth: the same team identified
the errors, corrected them, and reran the evaluation. But a published change log
and a second pass by domain experts make the central conclusion serious enough:
a benchmark plateau may mean not that models have reached their limit, but that
the test has started rejecting correct answers.
Another study reached a related conclusion from the opposite direction. Across
three small models, five inference backends, and six benchmarks, roughly 39% of
observed out-of-the-box variation came from the backend. Sampling and mismatched
generation settings explained some differences, but the effect remained under
deterministic decoding.
The 39% does not generalize to frontier models: the study used models of
1–1.5 billion parameters, and several “different backends” wrapped the same
Hugging Face ecosystem. The authors' operational demand is nevertheless sound:
store the backend, its version, the full generation configuration, and the
benchmark version beside every result.
Otherwise, a team can:
- mistake a broken test for a model limit;
- mistake a backend change for a prompt regression;
- compare models that actually ran in different systems;
- buy a more expensive model for a gap that disappears under reproducible
execution.
Concepts: evaluation harness, gold answer, pass@1,
inference backend, deterministic decoding.
Sources: SciCode-Verified,
study of backend effects.
