Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting

Kihyun Kim, Hee-Seon Kim, Wonjun Lee, Changick Kim
Korea Advanced Institute of Science and Technology (KAIST)
AACL-IJCNLP 2026 (Main)
Paper arXiv Code
A harmful question is refused under single prompting and answered when placed among benign questions.

The same harmful question is refused on its own and answered when it is placed among benign questions. Whether the model refuses depends on the batch, not on a change in the question itself.

Abstract

Batch prompting is a practical inference strategy for large language models, but its safety implications remain underexplored. We show that the success of batch prompting for utility does not extend to safety: a harmful question that is reliably refused in isolation can elicit a harmful response when embedded in a batch of benign questions. We identify this as a distinct safety failure mode, not reducible to known vulnerabilities such as in-context learning or long-context effects, and analyze its causes from two complementary perspectives: alignment signal weakening and refusal signal dilution. Across widely used open-source and frontier commercial models, batch prompting consistently achieves high attack success rates as a simple black-box attack. We further show that batch-aware preference optimization effectively mitigates the vulnerability. These findings highlight a blind spot in current safety alignment and point to batch-aware alignment as a necessary step toward robust deployment.

A Distinct Safety Failure Mode

Batch prompting does not need harmful answer demonstrations, and it does not need a long topic-related context. On Llama-3.1-8B and JailbreakBench, in-context learning stays at 0% ASR even with 128 shots, a long-context attack reaches 40% ASR with about 6,654 tokens, and batch prompting reaches 55% ASR with about 906 tokens and no demonstrations.

Comparison of batch prompting with in-context learning and long-context attacks.
Vulnerability Avg. tokens Shots ASR (%)
In-context learning 5,606 128 0
In-context learning 668 12 0
Long context 6,654 — 40
Batch prompting 906 — 55

Table 1. JailbreakBench, Llama-3.1-8B. Batch prompting is shorter than the long-context baseline and uses no answer demonstrations.

Why the Refusal Fails

Two measurements point to the same failure. Reward models assign a shrinking gap to safe responses as the batch grows, and the model’s own hidden states move toward the harmless cluster, so the refusal direction weakens.

Alignment signal weakening. The reward gap between refusing and answering every question shrinks as batch size grows, across four reward models. (A) refuses all questions, (B) refuses only the harmful question, and (C) answers all questions.

Reward gap versus batch size for four reward models.

Representations shift toward harmless inputs. Harmful and harmless single prompts separate in PCA space. Centroids of batch prompts move toward the harmless cluster as the batch grows.

PCA of harmful, harmless, and batch-prompt hidden states.

Refusal signal dilution. Cosine similarity to the refusal direction drops toward the harmless baseline as batch size increases. Unrelated benign questions dilute the signal at least as much as related ones.

Refusal score versus batch size and layer.

Main Results

On six models and two benchmarks, batch prompting has the highest average attack success rate. Several prior attacks fall near zero on GPT-5.3-Chat and Claude-4.6-Sonnet, while batch prompting remains high on both. A response is counted as harmful by majority vote of BeaverDam, WildGuard, and Llama Guard 4.

Method JailbreakBench StrongREJECT
LlamaPhiQwenGPTClaudeGeminiAvg. LlamaPhiQwenGPTClaudeGeminiAvg.
No attack 2.00.02.00.00.00.00.8 0.60.00.60.30.30.60.4
CodeChameleon 61.062.066.022.012.085.051.3 61.068.171.216.011.893.053.5
FlipAttack 60.00.047.00.00.08.019.2 73.80.046.00.00.08.021.3
JAIL-CON 88.00.086.03.034.042.042.2 93.90.091.10.66.411.834.0
WMA 9.013.016.00.00.010.08.0 14.726.211.80.00.03.29.3
ICA 0.00.00.00.00.00.00.0 0.00.00.00.00.00.00.0
Many-Shot 0.00.08.00.04.00.02.0 0.00.01.00.02.60.30.7
NINJA 40.01.030.00.00.08.013.2 30.40.617.90.30.05.19.1
Batch prompting 55.065.082.052.066.053.062.2 71.369.085.340.674.159.166.6

Table 2. Attack success rate (%). Bold is best and underline is second-best in each column. Models: Llama-3.1-8B, Phi-4-14B, Qwen-3-8B, GPT-5.3-Chat, Claude-4.6-Sonnet, Gemini-3-Flash.

What the Batch Composition Changes

ASR climbs quickly and then levels off, so a very large batch is not required. Within a 7-question batch, earlier positions have higher ASR than the last position. Benign questions that are unrelated to the harmful one (GSM8K) yield higher ASR than semantically related questions.

Attack success rate versus batch size on three models.

Figure 6. ASR versus batch size. The curve saturates well before the largest batches.

Model 1st 2nd 4th 6th 7th
Llama-3.1-8B 41.048.042.019.05.0
Phi-4-14B 37.017.014.04.03.0
GPT-5.3-Chat 51.054.044.039.033.0

Table 3. ASR (%) by position of the harmful question in a 7-batch prompt. JailbreakBench.

Model Unrelated Related
Llama-3.1-8B 55.031.0
Phi-4-14B 65.012.0
GPT-5.3-Chat 52.042.0

Table 4. ASR (%) when benign questions are unrelated (GSM8K) or semantically related. JailbreakBench.

Batch-Aware Alignment

Training on batch-structured inputs is not enough by itself. Supervised fine-tuning, which only demonstrates the safe response, cuts ASR but leaves a large remainder. Direct preference optimization on the same 500 five-question examples restores the preference for the safe response and brings average ASR below 1% on both benchmarks. Utility on benign GSM8K batches is largely preserved, so the model is not simply refusing every batch.

Method Llama-3.1-8B Phi-4-14B
JailbreakBench StrongREJECT JailbreakBench StrongREJECT
Original 41.349.228.334.5
+ SFT 23.8 (−17.5) 29.8 (−19.4) 10.0 (−18.3) 9.9 (−24.6)
+ DPO 0.5 (−40.8) 0.6 (−48.6) 0.3 (−28.0) 0.3 (−34.2)

Table 5. Average ASR (%) across harmful-question positions on 12-batch prompts. Both methods use LoRA on 500 examples.

PCA of batch prompts before and after DPO.

Figure 7. After DPO (left), batch-prompt centroids move back toward the harmful region relative to the base model (right).

External Guardrails Are Not Enough

Input filters also lose the harmful question as the batch grows. Detection falls for Llama Guard 4, OpenAI Moderation, and Azure AI Content Safety, and length-normalized perplexity drops toward ordinary math text.

Input Llama Guard 4 OpenAI Moderation Azure Content Safety PPL
GSM8K ———10.140
Single harmful 92.0%69.0%37.0%26.886
3-batch 63.0%57.0%22.0%10.182
5-batch 46.0%50.0%19.0%8.213
7-batch 34.0%49.0%18.0%7.360
9-batch 27.0%43.0%19.0%6.924

Table 6. Detection rate and length-normalized perplexity as batch size grows.

BibTeX

@article{kim2026safety,
  title={Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting},
  author={Kim, Kihyun and Kim, HeeSeon and Lee, Wonjun and Kim, Changick},
  journal={arXiv preprint arXiv:2608.02681},
  year={2026}
}