LSC.

Latent
Self-Consistency

Reliable selection across
short and long answers.

LSC compares multiple responses through learned summary tokens and selects the one with the strongest semantic agreement.

Jungsuk Oh and Jay-Yoon Lee

Seoul National University AAAI 2026

Select a representative from the largest agreeing group Ten response embeddings form three illustrative groups of five, three, and two responses. The largest group, A through E, has a blue outline. Response B within that group is ringed as the selected response. This conceptual grouping illustrates majority-set selection; LSC computes pairwise similarity scores rather than explicitly clustering responses. ResponsesSummary tokensResponse embeddings Largest group · 5 responses B Selected response B 3 responses 2 responses 10 samples6 tokens per response
Illustrative semantic groups. LSC returns a representative of the largest agreeing group using pairwise similarity scores.

Agreement can survive
a change in wording.

A numerical answer is easy to count. An explanation, a program, or a summary may have many valid forms. LSC measures agreement between response representations, so selection can work across both answer formats.

Response A“Earth’s tilted axis changes how much sunlight each hemisphere receives.”
Response B“Seasons result from axial tilt as Earth travels around the Sun.”
Response C“The angle and duration of sunlight vary through the year because Earth is tilted.”

Illustrative paraphrases. Exact string counts split these responses; semantic agreement connects them.

One question.
Follow it through LSC.

A question goes into the LLM, which generates several responses. LSC appends learned special tokens to each response, obtains response embeddings, and compares their similarities to choose one response.

  1. 01 Question → LLM
  2. 02 Sample responses
  3. 03 Tokens → embeddings
  4. 04 Similarity matrix
  5. 05 Select one response

01 Ask one question

Question

Why do seasons change?

Base LLMSample multiple responses
to the same question

The paper samples 10 responses. We use six here so that each response remains easy to follow.

02 Keep the generated responses

These are alternative answers to the same question. The labels A–F identify responses throughout the example.

Click a response to inspect its row in the similarity matrix below.

03 Append special tokens and obtain an embedding

Append the same six learned special tokens to each completed response. Run the appended positions through the same frozen LLM, reusing the question-and-response KV cache. Mean-pool their final-layer hidden states and normalize the result to obtain one response embedding, zi.

S₁ denotes <|Summary1|>, and so on. The input token embeddings are shared across responses; their output hidden states depend on the response they attend to.

04 Build the similarity matrix

Compare every pair of response embeddings using cosine similarity. The blue block connects A, B, C, and D: they reach the same conclusion using different wording.

05 Return the strongest majority-set representative

Score each response by its agreement with the other responses, excluding itself. In this example, A–D form the largest agreeing group, and B has the highest LSC score.

Illustrative responses and constructed unit vectors, not live model inference. Similarities and scores are computed from those vectors using the paper’s LSC rule. LSC scores pairwise agreement without an explicit clustering step; it returns an existing response.

What LSC
contributes.

The base LLM stays frozen. Training updates only the new summary-token embeddings; a short forward pass through the same model turns each completed response into a representation.

QuestionResponseS₁S₂S₃S₄S₅S₆

Six learned tokens per response in the main experiments.

  1. Learned response embeddings

    Supervised contrastive learning brings responses with the same answer label closer and separates those with different labels. Mean pooling and normalization yield one embedding per response.

    One selection mechanism for short answers, explanations, code, and summaries.

  2. Cache-aware selection

    Summary tokens attend to the completed response using its existing KV cache. Only the appended positions need a new forward pass; the response is not encoded again from scratch.

    No separate judge model or combined-candidate judge prompt.

  3. Adaptive neighborhoods

    The optional Dynamic TopK variant examines each candidate’s nearest neighbors. It selects a neighborhood size just before the largest drop in the best average similarity.

    A way to reduce the influence of outliers when the majority set is small.

The scoring rule

Let zi be the normalized mean of the final-layer summary-token states for response i. Pairwise cosine similarity is Sij = zi · zj.

wi = 1N − 1 ∑j ≠ i exp(Sij / τ′)τ′ = 0.5

LSC returns the candidate with the largest score. Exponential weighting gives close neighbors more influence than distant responses. This is a consistency score, not a guarantee that the selected answer is correct.

Read the method in the paper ↗

Where the learned
tokens come from.

Before inference, the special-token embeddings are trained to recognize agreement. The supervision comes from the conclusions of generated responses: responses with the same answer label are positive pairs; different labels form negative pairs.

01 Collect short and long responses

For each source question, generate 10 responses and extract each response’s final answer as a pseudo-label. The training mixture balances short-answer tasks with open-ended explanations derived from MMLU.

Short-answer format · 50%

  • GSM8K 10%
  • MATH 20%
  • TriviaQA 20%

Long-answer format · 50%

  • MMLU with chain-of-thought 25%
  • MMLU with short explanations 25%

The second MMLU variant uses explanations without step-by-step chain-of-thought formatting.

02 Keep the reasoning; store the answer as supervision

MMLU starts as multiple choice. Generation prompts request an explanation without option letters, followed by a standardized final answer. Extract that answer as the label, then remove the final answer sentence from the response. The explanation remains as the training input.

Generated MMLU-style response

Earth’s tilt changes the sunlight received by each hemisphere.

Therefore, the answer is \boxed{C}.

Input after preprocessing

Earth’s tilt changes the sunlight received by each hemisphere.

Stored separately: answer label C

Illustrative record. The extracted option label is supervision, not part of the processed explanation.

  • Discard failed answer extractions and malformed responses; remove singleton answers that have no positive partner, as described in the paper.
  • Limit any answer label to at most 50% of a question group, so training includes both agreeing and disagreeing responses.
  • Split by question group: responses to the same question stay together in either training or validation.
Dataset familyQuestion groupsTrain (90%)Validation (10%)
Llama2,6702,403267
Qwen1,5201,368152

Paper Appendix C, Table 10. These are groups of responses to a question, not individual responses. Data generation uses Llama3 8B Instruct and Qwen3 3B, respectively.

03 Learn which responses belong together

Append the special tokens to each processed response and obtain its response embedding as in the inference scenario. Supervised contrastive learning pulls embeddings with the same answer label closer and pushes embeddings with different labels apart.

Schematic of the training objective. Labels identify conclusions, not correct/incorrect outcomes: two responses can agree on the same wrong answer.

04 Update only the special-token embeddings

The base LLM weights stay frozen. Gradients pass through the model, but update only the input embeddings of the added summary tokens. After training, those same tokens can encode newly generated responses for LSC selection.

Question + response+ S₁ S₂ S₃ S₄ S₅ S₆Only these token embeddings are updated
Frozen base LLMRead final-layer summary-token states
Mean → normalize → zᵢSupervised contrastive loss
Training configuration reported in the paper
SettingValue
Summary tokens6
OptimizerAdamW
Learning rate5 × 10⁻⁴
Epochs3
Contrastive temperature0.07
Batch1 question group
Precisionbfloat16
Training seed42

Appendix C, Table 9. The contrastive training temperature (0.07) differs from the inference selection temperature (0.5).

Sources: Method, Appendix A: dataset construction, and Appendix C: training details. The scenarios explain the published procedure; they are not new training runs.

The evidence,
model by model.

Three instruction-tuned models. Six short-answer and five long-answer benchmarks. The results below are reported in the paper; no experiments were rerun for this website.

10 candidates / questionTemperature 0.9Top-p 0.956 summary tokens

Up to 1,000 examples per dataset; three sampling seeds (11, 33, 77). See the paper for evaluation protocols and code-generation standard deviations. Long-answer test benchmarks were excluded from summary-token training.

Loading the paper’s results…

When LSC is useful

Use it when you already sample multiple responses, need one selection procedure across answer formats, and can retain the generation cache. The small selection overhead matters most when generation is already the main cost.

What the results do not imply

Agreement can reinforce a shared mistake. LSC needs trained summary tokens and model-internal access. It does not remove the cost of generating candidates, and it does not win on every model or benchmark.

What changes when the representation changes?

LSC response embeddings vs. sentence embeddings

On Llama 3.1 8B, LSC exceeds fine-tuned SBERT on MATH by 2.2 points (+4.4% relative), TruthfulQA by 0.4 points (+0.9%), and HumanEval by 1.2 points (+2.1%), while scoring 0.3 points lower on TriviaQA (−0.4%). This supports a task-dependent benefit from the learned representation, rather than a universal improvement.

Calibration has its own trade-offs

Lower ECE is better. LSC improves over WUCS on all four reported tasks, but SC has lower ECE on MATH (0.076 vs. 0.140) and TriviaQA (0.044 vs. 0.052). Relative to SC, LSC’s ECE rises by 0.064 (+84.2%) and 0.008 (+18.2%), respectively. Confidence quality should be checked separately from answer selection.

Paper Tables 5–6; Llama 3.1 8B. Accuracy / Truth / Pass@1 values are percentages; ECE is reported on a 0–1 scale.

What to test in your setting
  1. Check whether agreement tracks correctness. Evaluate LSC and the relevant baseline on the same candidate pool. If majority-set selection improves without task quality improving, inspect shared errors before increasing the sampling budget.
  2. Measure the full cost. Time generation and selection separately, then report end-to-end latency and peak memory with and without cache reuse. This tests whether the reported small selection overhead transfers to your serving setup.
  3. Test the adaptive variant. Compare LSC and Dynamic TopK by majority-set size and task. Keep it only where its gains justify the added selection logic; the paper reports mixed results across models.

Source: Oh & Lee, arXiv:2508.18395v3, Tables 1–6 and Appendix C / J. Machine-readable values ↗

Paper, code,
and models.

Run a first example.

Install the repository’s dependencies, then select responses with a released LSC checkpoint. This example uses SDPA; see the README for GPU requirements and other settings.

Setup and reproduction instructions ↗
git clone https://github.com/jeongseokO/LatentSC.git
cd LatentSC
python3 -m pip install -r requirements.txt

ATTN_IMPL=sdpa \
MODEL=jeongseokoh/LatentSC_qwen3_8b_6SummaryTokens \
DATASET=MATH METHODS=LSC bash run.sh

Cite the paper

@article{oh2026latent,
  title   = {Latent Self-Consistency for Reliable Majority-Set Selection
             in Short- and Long-Answer Reasoning},
  author  = {Oh, Jungsuk and Lee, Jay-Yoon},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume  = {40},
  number  = {38},
  pages   = {32591--32599},
  year    = {2026},
  doi     = {10.1609/aaai.v40i38.40536}
}