Latent
Self-Consistency
Reliable selection across
short and long answers.
LSC compares multiple responses through learned summary tokens and selects the one with the strongest semantic agreement.
Seoul National University AAAI 2026
Agreement can survive
a change in wording.
A numerical answer is easy to count. An explanation, a program, or a summary may have many valid forms. LSC measures agreement between response representations, so selection can work across both answer formats.
Response A“Earth’s tilted axis changes how much sunlight each hemisphere receives.”
Response B“Seasons result from axial tilt as Earth travels around the Sun.”
Response C“The angle and duration of sunlight vary through the year because Earth is tilted.”
Illustrative paraphrases. Exact string counts split these responses; semantic agreement connects them.
SC counts matching answers.
Sample responses, extract the final answers, and vote by exact match. This is effective when answers have a canonical form; free-form outputs need a way to compare meanings.
WUCS compares words.
Sample responses and rank them by probability-weighted unigram overlap. Selection is inexpensive, but shared vocabulary is an incomplete signal of shared conclusions.
USC asks the model to judge.
Place all candidates in a new prompt and ask the LLM to choose. This supports free-form comparison, while adding a judge pass over a new combined context.
LSC compares learned embeddings.
Append learned summary tokens to each response, reuse its generation cache, and compare the response embeddings obtained from the tokens’ final-layer hidden states. It requires summary-token training and access to model internals.
One question.
Follow it through LSC.
A question goes into the LLM, which generates several responses. LSC appends learned special tokens to each response, obtains response embeddings, and compares their similarities to choose one response.
- 01 Question → LLM
- 02 Sample responses
- 03 Tokens → embeddings
- 04 Similarity matrix
- 05 Select one response
01 Ask one question
Why do seasons change?
to the same question
The paper samples 10 responses. We use six here so that each response remains easy to follow.
02 Keep the generated responses
These are alternative answers to the same question. The labels A–F identify responses throughout the example.
Click a response to inspect its row in the similarity matrix below.
03 Append special tokens and obtain an embedding
Append the same six learned special tokens to each completed response. Run the appended positions through the same frozen LLM, reusing the question-and-response KV cache. Mean-pool their final-layer hidden states and normalize the result to obtain one response embedding, zi.
S₁ denotes <|Summary1|>, and so on. The input token embeddings are shared across responses; their output hidden states depend on the response they attend to.
04 Build the similarity matrix
Compare every pair of response embeddings using cosine similarity. The blue block connects A, B, C, and D: they reach the same conclusion using different wording.
05 Return the strongest majority-set representative
Score each response by its agreement with the other responses, excluding itself. In this example, A–D form the largest agreeing group, and B has the highest LSC score.
Illustrative responses and constructed unit vectors, not live model inference. Similarities and scores are computed from those vectors using the paper’s LSC rule. LSC scores pairwise agreement without an explicit clustering step; it returns an existing response.
What LSC
contributes.
The base LLM stays frozen. Training updates only the new summary-token embeddings; a short forward pass through the same model turns each completed response into a representation.
Six learned tokens per response in the main experiments.
Learned response embeddings
Supervised contrastive learning brings responses with the same answer label closer and separates those with different labels. Mean pooling and normalization yield one embedding per response.
One selection mechanism for short answers, explanations, code, and summaries.
Cache-aware selection
Summary tokens attend to the completed response using its existing KV cache. Only the appended positions need a new forward pass; the response is not encoded again from scratch.
No separate judge model or combined-candidate judge prompt.
Adaptive neighborhoods
The optional Dynamic TopK variant examines each candidate’s nearest neighbors. It selects a neighborhood size just before the largest drop in the best average similarity.
A way to reduce the influence of outliers when the majority set is small.
The scoring rule
Let zi be the normalized mean of the final-layer summary-token states for response i. Pairwise cosine similarity is Sij = zi · zj.
LSC returns the candidate with the largest score. Exponential weighting gives close neighbors more influence than distant responses. This is a consistency score, not a guarantee that the selected answer is correct.
Read the method in the paper ↗Where the learned
tokens come from.
Before inference, the special-token embeddings are trained to recognize agreement. The supervision comes from the conclusions of generated responses: responses with the same answer label are positive pairs; different labels form negative pairs.
01 Collect short and long responses
For each source question, generate 10 responses and extract each response’s final answer as a pseudo-label. The training mixture balances short-answer tasks with open-ended explanations derived from MMLU.
Short-answer format · 50%
- GSM8K 10%
- MATH 20%
- TriviaQA 20%
Long-answer format · 50%
- MMLU with chain-of-thought 25%
- MMLU with short explanations 25%
The second MMLU variant uses explanations without step-by-step chain-of-thought formatting.
02 Keep the reasoning; store the answer as supervision
MMLU starts as multiple choice. Generation prompts request an explanation without option letters, followed by a standardized final answer. Extract that answer as the label, then remove the final answer sentence from the response. The explanation remains as the training input.
Earth’s tilt changes the sunlight received by each hemisphere.
Therefore, the answer is \boxed{C}.
Earth’s tilt changes the sunlight received by each hemisphere.
Illustrative record. The extracted option label is supervision, not part of the processed explanation.
- Discard failed answer extractions and malformed responses; remove singleton answers that have no positive partner, as described in the paper.
- Limit any answer label to at most 50% of a question group, so training includes both agreeing and disagreeing responses.
- Split by question group: responses to the same question stay together in either training or validation.
| Dataset family | Question groups | Train (90%) | Validation (10%) |
|---|---|---|---|
| Llama | 2,670 | 2,403 | 267 |
| Qwen | 1,520 | 1,368 | 152 |
Paper Appendix C, Table 10. These are groups of responses to a question, not individual responses. Data generation uses Llama3 8B Instruct and Qwen3 3B, respectively.
03 Learn which responses belong together
Append the special tokens to each processed response and obtain its response embedding as in the inference scenario. Supervised contrastive learning pulls embeddings with the same answer label closer and pushes embeddings with different labels apart.
Schematic of the training objective. Labels identify conclusions, not correct/incorrect outcomes: two responses can agree on the same wrong answer.
04 Update only the special-token embeddings
The base LLM weights stay frozen. Gradients pass through the model, but update only the input embeddings of the added summary tokens. After training, those same tokens can encode newly generated responses for LSC selection.
Training configuration reported in the paper
| Setting | Value |
|---|---|
| Summary tokens | 6 |
| Optimizer | AdamW |
| Learning rate | 5 × 10⁻⁴ |
| Epochs | 3 |
| Contrastive temperature | 0.07 |
| Batch | 1 question group |
| Precision | bfloat16 |
| Training seed | 42 |
Appendix C, Table 9. The contrastive training temperature (0.07) differs from the inference selection temperature (0.5).
Sources: Method, Appendix A: dataset construction, and Appendix C: training details. The scenarios explain the published procedure; they are not new training runs.
The evidence,
model by model.
Three instruction-tuned models. Six short-answer and five long-answer benchmarks. The results below are reported in the paper; no experiments were rerun for this website.
Up to 1,000 examples per dataset; three sampling seeds (11, 33, 77). See the paper for evaluation protocols and code-generation standard deviations. Long-answer test benchmarks were excluded from summary-token training.
Short-answer average
Long-answer average
Paper Table 1, reproduced as published. Axes start at zero. These format-level averages combine different task metrics; inspect the per-task tables before comparing individual capabilities.
Time and memory are percentage overhead relative to the shared 10-candidate generation workload, not single-response inference. SC’s long-answer result is unavailable. No absolute latency, throughput, or peak VRAM is provided in this summary.
Paper Table 4. Consistency is the percentage of selections belonging to the majority answer set among 10 candidates. For long answers, GPT-4.1 identifies the majority. This is a selection metric, not task accuracy.
Explore every benchmarkPer-task scores, metrics, and differences
Δ is an arithmetic difference in the metric’s own units; relative change is Δ / baseline × 100. For percentage metrics, Δ is in percentage points. N/A means the baseline was not evaluated. These differences do not establish statistical significance.
Download the source data (JSON) ↓When LSC is useful
Use it when you already sample multiple responses, need one selection procedure across answer formats, and can retain the generation cache. The small selection overhead matters most when generation is already the main cost.
What the results do not imply
Agreement can reinforce a shared mistake. LSC needs trained summary tokens and model-internal access. It does not remove the cost of generating candidates, and it does not win on every model or benchmark.
What changes when the representation changes?
LSC response embeddings vs. sentence embeddings
On Llama 3.1 8B, LSC exceeds fine-tuned SBERT on MATH by 2.2 points (+4.4% relative), TruthfulQA by 0.4 points (+0.9%), and HumanEval by 1.2 points (+2.1%), while scoring 0.3 points lower on TriviaQA (−0.4%). This supports a task-dependent benefit from the learned representation, rather than a universal improvement.
Calibration has its own trade-offs
Lower ECE is better. LSC improves over WUCS on all four reported tasks, but SC has lower ECE on MATH (0.076 vs. 0.140) and TriviaQA (0.044 vs. 0.052). Relative to SC, LSC’s ECE rises by 0.064 (+84.2%) and 0.008 (+18.2%), respectively. Confidence quality should be checked separately from answer selection.
Paper Tables 5–6; Llama 3.1 8B. Accuracy / Truth / Pass@1 values are percentages; ECE is reported on a 0–1 scale.
What to test in your setting
- Check whether agreement tracks correctness. Evaluate LSC and the relevant baseline on the same candidate pool. If majority-set selection improves without task quality improving, inspect shared errors before increasing the sampling budget.
- Measure the full cost. Time generation and selection separately, then report end-to-end latency and peak memory with and without cache reuse. This tests whether the reported small selection overhead transfers to your serving setup.
- Test the adaptive variant. Compare LSC and Dynamic TopK by majority-set size and task. Keep it only where its gains justify the added selection logic; the paper reports mixed results across models.
Source: Oh & Lee, arXiv:2508.18395v3, Tables 1–6 and Appendix C / J. Machine-readable values ↗
Paper, code,
and models.
Run a first example.
Install the repository’s dependencies, then select responses with a released LSC checkpoint. This example uses SDPA; see the README for GPU requirements and other settings.
Setup and reproduction instructions ↗git clone https://github.com/jeongseokO/LatentSC.git
cd LatentSC
python3 -m pip install -r requirements.txt
ATTN_IMPL=sdpa \
MODEL=jeongseokoh/LatentSC_qwen3_8b_6SummaryTokens \
DATASET=MATH METHODS=LSC bash run.shCite the paper
@article{oh2026latent,
title = {Latent Self-Consistency for Reliable Majority-Set Selection
in Short- and Long-Answer Reasoning},
author = {Oh, Jungsuk and Lee, Jay-Yoon},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {40},
number = {38},
pages = {32591--32599},
year = {2026},
doi = {10.1609/aaai.v40i38.40536}
}