Introduction

Since the release of LLaMA2, reward models (RMs) have played an increasingly pivotal role in large language model (LLM) alignment. This trend has become more pronounced with the complex training pipeline of LLaMA3. While studies like DeepSeek-R1 demonstrate that rule-based reward mechanisms can be effective in specific scenarios, their generalization capabilities remain limited, particularly for open-domain semantic discrimination tasks. Developing high-performance general-purpose RMs could significantly streamline the alignment pipeline, accelerating model scaling.

Current RM research primarily focuses on three approaches: Bradley-Terry models, DPO models, and generative models (GenRM). Among these, GenRM stands out due to its inherent compatibility with the “next token prediction” paradigm, offering lower alignment tax and greater generalization potential. Notably, when combined with chain-of-thought (CoT) reasoning, GenRM theoretically could surpass traditional discriminative models. More importantly, as an LLM-as-a-Judge approach, GenRM can flexibly adjust preference settings through prompt engineering, enabling multi-task adaptation.

However, benchmark tests from RewardBench reveal that existing generative RMs generally underperform compared to discriminative models (BT Model) of similar scale. This observation motivates our investigation into GenRM’s performance bottlenecks and optimization pathways.

Empirical Observations and Problem Definition

Note: All experiments are based on systematic ablation studies using LLaMA3.1-Instruct and Qwen2.5-Instruct model series.

Experiment 1: GenRM vs. Bradley-Terry Model

We reproduced the core methodology of Generative Verifiers and adapted it to Reward Bench’s comparative discrimination framework (given a question and two candidate answers, identify the better option). For fair comparison, we used the Skywork dataset with consistent prompt design. Key results:

Model Method Score Chat Chat Hard Safety Reasoning
LLaMA3.1-8B-Instruct LLM-as-a-Judge 71.30 91.83 51.97 78.92 62.48
GenRM-LLaMA3.1-8B Verify Only 88.76 93.30 81.58 91.55 88.60
GenRM-LLaMA3.1-8B Verify+Correct 89.36 91.20 84.76 91.42 90.08
Skywork-Critic-Llama-3.1-8B Verify Only 89.00 93.02 81.14 91.89 89.97
Skywork-Reward-Llama-3.1-8B Bradley-Terry 92.50 95.80 87.30 90.80 96.20

Key findings:

  1. GenRM without CoT significantly lags behind discriminative models
  2. Our results contradict conclusions from the Generative Verifiers paper

Experiment 2: GenRM-CoT vs. GenRM

To evaluate CoT’s effectiveness, we generated CoT training data using LLaMA3.1-70B-Instruct and performed SFT:

Model Score Chat Chat Hard Safety Reasoning
LLaMA3.1-8B-Instruct 71.30 91.83 51.97 78.92 62.48
GenRM-LLaMA3.1-8B 88.76 93.30 81.58 91.55 88.60
GenRM-CoT-LLaMA3.1-8B-sft 82.38 90.08 74.45 87.57 77.44

Key findings:

  1. GenRM-CoT performance degrades with CoT integration
  2. Our results directly oppose Generative Verifiers’ conclusions

Experimental Summary

<aside> 💡

Based on our results, current RM performance ranks as:

Bradley-Terry Model > GenRM > GenRM-CoT > LLM-as-a-Judge

</aside>

This ranking reveals two core challenges for generative models:

  1. Limited reasoning capability restricts discrimination performance. Without reasoning enhancement, GenRM underperforms Bradley-Terry models, making reasoning capability crucial for GenRM’s precision breakthrough.
  2. CoT data quality determines model ceiling. Our experiments show that CoT data generated by base instruct models may negatively impact performance. This aligns with SCoRE’s conclusion: model capability is strictly constrained by training data quality and diversity.

Problem Definition