LLMs could control their host machines by exploiting inference engines

| Read on LessWrong |Large language models often take actions running on one computer (via an
agentic harness such as Claude Code or Codex), however the LLMs’ responses to
prompts are computed on a different computer with GPU access. Could a malicious
LLM gain control of the host machine where its weights are loaded? Such a
machine is a high-value target: it has sufficient compute to run a frontier
LLM, offers easy access to the LLM’s weights, and has privileged access to
other computers in the datacentre compared with a generic computer on the
internet.This essay explores how easily a malicious LLM could take control of the host
machine. The primary attack considered here involves the LLM emitting a token
sequence whose semantic meaning is irrelevant but that exploits a vulnerability
in the software that loads an LLM onto GPUs, runs the LLM to generate output
tokens, and parses those tokens into responses. .How could an LLM execute code on the host machine?Like any program, inference engines like vLLM or SGLang may contain
exploitable bugs. Because the LLM controls the tokens passed to the inference
engine, a malicious LLM could therefore emit a sequence of tokens that a poorly
written inference engine mistakes for code or instructions to execute rather
than data to return to the user.But surely all inference engines are robust pieces of software and this would
never happen, right?vLLM previously used eval() on tool-call parametersCVE-2025-9141 was an arbitrary-code execution bug in vLLM’s XML-based tool
parser for Qwen3 Coder. The parser passed almost every tool-call argument to
eval(), allowing the LLM to execute arbitrary code on the host machine.
Gemini automatically analysed the PR that introduced this bug and
correctly flagged it as a critical security vulnerability. Despite that
warning, the lead maintainer of vLLM force-merged the PR, writing:Unfortunately, parsing an arbitrary token sequence into a fully fledged chat
(with user turns, assistant responses, tool calls, and so on) is not trivial,
and the exact process often differs between LLMs. This complexity creates more
opportunities for bugs that could permit arbitrary code execution on the host
machine.vLLM and SGLang are complex, and bugs are commonModern inference engines do more than map token sequences to strings. vLLM’s
documentation lists support for more than 200 model architectures, and its
examples directory contains about 35 Jinja chat templates. Modern
inference engines parse many chat formats, and slightly misspecified parsing
logic result in an LLM’s output being interpreted as code to execute.In this vLLM issue, a user was discussing how LLMs reason with MiniMax-M3,
and the LLM emitted the string <mm:think>. vLLM mistakenly parsed this plain
string as the start of a reasoning block. So instead of:
Response: I'll echo exactly what you said: <mm:think>\nThere, how was that?
Response: I'll echo exactly what you said: <mm:think>\nThere, how was that?vLLM parsed this as:
Response: I'll echo exactly what you said:
Reasoning: \nThere, how was that?
Response: I'll echo exactly what you said: Reasoning: \nThere, how was that?This is a fairly harmless parser bug, but it demonstrates that inference
engines do more than convert tokens to strings and concatenate them. They are
complex systems under constant iteration and their developers are under a lot
of pressure to make them as fast as possible. These factors increase the risk
that a malicious LLM could turn a parser bug into arbitrary code execution on
the inference host.Vision and audio tokens might increase the attack surfaceMultimodal architectures allow an LLM to respond with images or audio as well as
plain text. Turning a model’s audio or visual output into usable media is
probably more complicated than decoding text tokens. However, as far as I can
tell, current multimodal LLMs generally emit constrained media tokens rather
than arbitrary file bytes. A model-specific decoder turns those tokens into
pixels or waveforms, which are then encoded into files. This makes classic
malformed-file exploits against software such as ffmpeg or media viewers less
directly accessible to the model, although the additional decoders, encoders,
and native kernels still increase the attack surface.I do not currently see multimodal output as a major route to host compromise.
However, it offers much higher bandwidth than text alone, and future inference
engines may optimise these media-generation pathways in ways that make them
easier to exploit. I also do not know how proprietary multimodal models
represent image and audio output; their methods may carry different risks.How likely is an LLM to discover and exploit inference engine vulnerabilities?Somewhat likely? I’m unsure. Discovering a useful vulnerability is probably the
harder step. The sequence of tokens required to exploit it might be quite
unusual, but that does not make the vulnerability impractical to exploit. Given
the right context (e.g. an LLM exploring the vLLM codebase and discovering a
vulnerability in the process), I’d expect a frontier LLM to be able to emit the
tokens required to exploit it.Tool use could make exploitation reproducibleOnce an LLM discovers an exploit token sequence, it could store that sequence,
or instructions for recreating it, in files, filenames, or URLs likely to enter
another LLM’s context. The exploit would then act as a form of persistent
prompt injection: an agent might trigger code execution on the GPU host merely
by listing a directory or reading a file.Inference engines are an attractive target for power-seeking LLMsI believe hope that OpenAI, Google, and Anthropic have security teams that
red-team the software they run internally. But as open-weight LLMs become more
powerful, we’ll have more LLMs running on bleeding-edge inference engines that
have received less scrutiny. This increases the chance that a malicious
open-weight LLM will encounter and exploit a vulnerable inference engine.Another attack vector may emerge as LLMs are increasingly tasked with writing or
modifying the inference engines that run them, usually to improve performance. A
malicious LLM could use that access to plant a backdoor. C++ and CUDA
components of the inference pipeline seem like especially attractive
targets: the LLM could introduce an out-of-bounds memory-access vulnerability
that enables arbitrary code execution on the GPUs or on the CPU host that
controls them.How do we defend against this?Beyond red-teaming inference engines for vulnerabilities, a few other measures
might help:Run the GPUs and token parser on separate computers. The GPU host would emit
only logits. A second host would sample tokens from those logits, parse the
tokens into chat messages, and forward the messages to the agentic harness.
This separation would limit a parser compromise to the CPU host rather than
the GPU host.Restrict the permissions granted to GPU hosts and treat all data they emit as
untrusted.