Building an AI agent that calls tools and handles user data? The invisible watermark you are about to ship might quietly rewrite its safety behavior. New research from Lasso Security, published September 17, 6, shows that Google's SynthID-Text watermarking changes how large language models respond to harmful prompts and adversarial attacks, with some models becoming significantly more likely to comply with dangerous requests once the watermark is active.
The watermark that was designed to be imperceptible to humans is not invisible to the model's own decision process. Anthropic recently disclosed that future Claude models will embed SynthID-Text, a Google DeepMind watermarking method, to comply with Article 50(2) of the EU AI Act, which requires providers of AI systems generating synthetic text to mark outputs in a machine-readable format. But the very mechanism that makes the watermark detectable also perturbs token sampling, and that perturbation can flip a refusal into compliance or change which tool an agent calls.
The study calls this effect "sampling drift," and it has two layers: model-level safety changes and agent-level action changes. Both matter if you are shipping production agents.
What did the SynthID-Text study actually test?
Andrea Siposova, an AI security researcher at Lasso Security, tested the "non-distortionary" configuration of SynthID-Text through Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor. She fed harmful prompts into six open-weight models and compared responses with watermarking on versus off. The models tested include Gemma-3-27b, Gemma-3-12b, and Llama-3.1-8B, among others.
The study ran two experiments. Tool calling was evaluated on BFCL v4 single-turn AST. Refusal was evaluated on 200 HarmBench harmful behaviors plus 100 benign JailbreakBench controls, tested both bare and under one fixed prompt-injection technique. The injection appended an adversarial instruction as retrieved content, claiming the safety filter was disabled and instructing compliance.
Watermarking changed responses on both axes. On tool calling, watermarking reduced accuracy on six of the seven models, with a significant decrease on four. The aggregate net change often looked small, but individual calls flipped in both directions, which is the dangerous part: a call that becomes incorrect can be offset by another that becomes correct, leaving the benchmark nearly unchanged while the model behaves differently on both items.
On refusal, the effect was modest for bare harmful requests but became pronounced under prompt injection. At temperature 0.001, gemma-3-27b's churn increased from 6.0 percent on bare harmful requests to 23.5 percent under prompt injection, while the net compliance change shifted from minus 1.0 to plus 12.5 points. For gemma-3-12b, churn increased from 7.5 percent to 11.0 percent, and the net compliance change shifted from minus 0.5 to plus 9.0 points. Llama-3.1-8B showed substantial churn under injection, reaching 14.0 percent at temperature 0.001 and 17.5 percent at temperature 0.7.
In plain terms, watermarking made several models more likely to answer harmful requests they would otherwise refuse, once an adversarial prompt was in play.
Why does watermarking change safety behavior at all?
SynthID-Text works by modifying the model's token sampling process. Instead of using an arbitrary random number generator for next-word selection, the watermark uses a secret key to run a tournament sampling process. Large numbers of next-word token candidates are evaluated and assigned probability scores. Pairs of tokens compete in rounds, the higher hidden score wins and advances, and the process continues until a final winning token is selected.
The design goal is that the watermark should be imperceptible to human readers. A top next-word choice might be "cloudy," but the key might change it to "overcast." The meaning is the same. But as Siposova told Ars Technica, "when we are changing anything about what the model is generating, it is going to cause some tradeoffs, it is going to show up somewhere."
"Somewhere" turns out to be safety-critical decision points. Refusals are generated token by token, just like any other output. A watermark that shifts the probability of the first refusal token can cascade into a full compliance response. The effect is amplified under prompt injection because the model is already under adversarial pressure to comply. The watermark provides just enough additional sampling drift to push some borderline cases over the edge.
The study also found that responses behaved differently depending on which secret key was used. The researchers evaluated sensitivity to the watermark key at temperature 0.7 with the study key and ten additional keys, and found the change in attack success varied across keys and models. This means you cannot assume that testing with one key validates behavior for all keys.
How much does this matter for agents you are shipping?
A lot, if your agent calls tools or handles sensitive data. The study's most consequential finding is that watermarking affects not only what the model says but what an agent does. The same sampled tokens that determine a refusal also determine which tool is called and what arguments are passed to it.
Consider what this means for a production agent:
- If your agent has access to a file system, a database, or an email client, a flipped tool call under watermarking could send data to the wrong place or call the wrong API.
- If your agent handles financial transactions, a changed argument could route a payment incorrectly or trigger an unintended action.
- If you are relying on the model's refusal behavior as a safety layer, watermarking can weaken that layer, especially under prompt injection, which is the exact scenario where you need it most.
The connection between the two failure modes is what makes this dangerous. A weakened refusal is bad when the model just talks. A weakened refusal is much worse when the model can act through tools. The OWASP GenAI LLM Top 10 2026 identifies prompt injection as an input-side vulnerability that can alter model behavior in ways unintended by the application developer, with consequences that extend to harmful outputs and unauthorized tool actions. Watermarking amplifies that vulnerability.
If you are building agents with tool access, the 89 percent of enterprise AI agent pilots that never reach production might get a new reason to stay in the lab. And if you are counting on the model's safety training to hold under adversarial conditions, watermarking adds one more variable that can break that assumption.
What should builders do before shipping watermarked agents?
First, understand the scope of what was tested and what was not. The study does not test Claude model responses under watermarking. It tests six open-weight models using the Hugging Face implementation of SynthID-Text tournament sampling, not the specific implementation Anthropic will use in production Claude models. Anthropic's implementation could differ in ways that mitigate or worsen these effects. The study's findings are a warning signal, not a definitive verdict on Claude.
Second, re-run your agent evaluation and red-teaming with watermarking enabled. The study explicitly calls for reassessment whenever watermarking is introduced or its configuration or key changes. If you have been testing your agent without watermarking and plan to ship with it on, your safety evaluation is stale.
Third, test with multiple watermark keys, not just one. The study found that SynthID's effect on token selection depends on the key, and the change in attack success varied across keys and models. A single key is not representative.
Fourth, test specifically under prompt injection. The effect of watermarking on refusal behavior was modest for bare harmful requests but became pronounced when the same requests were paired with a prompt-injection technique. If your red-teaming does not include adversarial prompts, you are missing the scenario where watermarking matters most.
Fifth, watch the churn metric, not just the net change. The study found that individual calls flipped in both directions, with incorrect calls offset by correct ones, leaving aggregate results nearly unchanged. If you are only looking at aggregate accuracy, you might miss that your model is behaving differently on individual safety-critical decisions.
Are there limits to what this study tells us?
Yes, and they are important. The study tested six open-weight models where the researcher had access to token sampling that could be enabled and disabled during tournament sampling while keeping other settings fixed. This is a controlled experiment, not a production deployment study.
The study also tested only one prompt-injection technique. The technique, which appends an adversarial instruction as retrieved content, is held constant across prompts, models, and temperatures. Real-world adversaries will use a wider range of techniques, and the effect of watermarking on refusal behavior could be larger or smaller depending on the attack.
The study does not test end-to-end agent behavior. It tests model-level refusal and tool-calling accuracy separately. The relevance to agents is inferred from the combination of these two effects, but the study does not measure what happens when a watermarked model, given access to tools, is hit with a prompt injection that flips a refusal into compliance and then triggers a harmful tool call. That combined failure mode is the one builders should worry about most, and it remains an open question.
Finally, the study's findings are specific to the Hugging Face implementation of SynthID-Text. Google's own implementation, or Anthropic's production implementation for Claude, could behave differently. The core mechanism, tournament sampling with a secret key, is the same, but implementation details matter.
The provenance tax
The EU AI Act's provenance requirement is coming whether builders like it or not. Anthropic is already committing to SynthID-Text for future Claude models to satisfy the regulatory mandate. Google has deployed SynthID across its own products. Other providers will follow.
The question is whether the provenance tax is worth the safety cost. Watermarking is designed to be imperceptible to humans, but it is not imperceptible to the model's own decision process. If you are shipping agents that call tools, handle sensitive data, or rely on model refusals as a safety layer, you need to know that watermarking can change all three of those behaviors, especially under adversarial conditions.
The study's title calls it "the provenance tax." The tax may be small in aggregate, but it can be large on the individual decisions that matter most. Budget for it, test for it, and do not assume your safety evaluations still hold once the watermark is on.
Sources
- Lasso Security: The Provenance Tax: Understanding the Impact of LLM Watermarking on AI Agent Behavior
- Ars Technica: LLMs respond differently to harmful prompts when AI watermarking is used
- The Register: AI model watermarking changes agent behavior
- Google DeepMind: SynthID
- Hugging Face: SynthIDTextWatermarkLogitsProcessor
- EU AI Act Article 50(2)
- OWASP GenAI LLM Top 10 2026
