1 tagged with "offline-evaluation"

Shadow Replay Punishes the Model That Would Have Changed the Conversation

Shadow replay evaluations quietly punish better models by scoring them against logged user turns that were conditioned on the old model — here is why, and what replay can still measure honestly.