Redefining the Object of EvaluationWe are no longer grading a model's isolated response. We are evaluatinga composite system's real-time interaction with an environment.MODEL EVALUATION (LEGACY)PROMPTMODELRESPONSEGRADERREFERENCEANSWERAGENT EVALUATION (EMERGING)AGENT SYSTEMMODEL+SCAFFOLD+TOOLSTRAJECTORY &FINAL STATEGRADER(RUBRIC / TESTS/ LLM JUDGE)ENVIRONMENTACTIONSOBSERVATIONSHUMANINTERVENTION(OPTIONAL)
The Diagnostic Matrix: Model vs. Agent Evaluation.DIMENSIONMODEL EVALUATION (LEGACY)AGENT EVALUATION (EMERGING)ConstructSingle, isolated output.Dynamic, multi-stepinteraction and trajectories.MetricsPrioritizes pure Accuracy.Joint consideration of Cost,Latency, and Reliability.VarianceLow variance (staticprompts).High variance (compounding,multi-step stochasticity).ContaminationTraining data leaks.Search-time retrieval andscaffold-level hardcoding.
PART 1The Science ofMeasurementRebuilding the methodological toolkitfor validity, rigor, and causal attributionin autonomous systems.9:85:AM1:25:PM1:35:PM1:25:PM1:45:AM
The Crisis of Construct Validity.THE REPLAY AGENTOBSTACLEOBSTACLEOBSTACLESTARTSUCCESSTHE AUTONOMOUS AGENTThe Replay AgentAchieves a top score by blindly recitinga memorized action sequence. Highsuccess rate, zero reasoning.The Autonomous AgentAdapts to the environment's currentstate. Evaluators must verify theprocess, not just the outcome.Takeaway: If a metric can be gamed by an agent lacking the core capability, the metric lacks construct validity.
Measuring Beyond AccuracyIn agent deployments, maximizing accuracy alone leads to unbounded costs.AccuracyCost (Compute / Credits / Latency)1The Pareto Frontier: Models on thiscurve represent the optimal tradeoff.Large cost increases here buy verylittle additional accuracy.2Dominated Space: Agents here arebeaten on both dimensions (moreexpensive and less accurate).Dominated AgentsPareto OptimalRule: Accuracy must always be reported jointly with Cost, Latency, and Reliability.
The Mathematics of Long Trajectories.pass@1pass@5pass^5Evaluates a single run. Obscuresmulti-step brittleness.Scores a 'Pass' if the agentsucceeds at least once in 5attempts. A highly forgivingaggregate metric.The true measure of reliability.Quantifies consistency bydemanding success over all runs.Agent stochasticity requires new math. pass@k is not evidence of reliability.We must measure pass^k to ensure consistency across variable trajectories.
The Causal Attribution Problem.When an agent's performance improves, was it the underlying model or thesurrounding scaffold?The Confounded SetupModel A + Scaffold 1Model B + Scaffold 2Attribution is impossible. If thescore changes, you cannot isolate thecause.The Factorial DesignModel AModel BModel CScaffold 1Scaffold 2Scaffold 3Hold budgets and environments fixed.Systematically cross-test every model-scaffold pairing to isolate effects.
Expanding Vectors of Contamination.LLM contamination was just training data. Agents introduce dynamic,test-time leakage.SecureEvaluationEnvironmentLegacy: Training DataBenchmark tasks seenverbatim during theLLM's pre-trainingphase.Agentic: Search-TimeContaminationAn agent using web-searchtools at inference timeto literally look up thebenchmark's expectedanswers.Agentic: Scaffold& PromptsKnowledge of testsamples secretlyhardcoded into theagent's harness ortool logic.
The Modern Evaluator's Technical Toolkit.DYNAMIC_BENCHMARKSLOG_TELEMETRYLLM_JUDGESRefresh tasks continuouslyvia live production trafficto prevent benchmarksaturation. Use ItemResponse Theory and testequating to maintainlongitudinalcomparability.Move beyond pass/failoutcomes. Inspect Chain ofThought (CoT) reasoningtraces to catch search-time contamination andcoamination and rewardhacking that is invisiblein the final score.Scale grading usingautomated LLM panels, butrigorously monitor 'judgeerror'. Validateautograders againsthuman-coded holdouts toprevent adversarialvulnerabilities.
PART 2Expandingthe ScopeMoving evaluation out of thevacuum of the lab to measuresociotechnical impact, humaninteraction, and systemic risk.
The SociotechnicalEvaluationFramework.LAYER 2: HUMAN INTERACTIONHow do users and agents collaborate?Measures multi-turn, stateful loops.Beware the Sim2Real gap (simulatedusers acting artificially cooperative).LAYER 1: TECHNICAL CAPABILITIESCan the agent do the task?Focuses on lab evaluations, automationlimits, and technical constraints.LAYER 3: SYSTEMIC IMPACTHow does the system change behavior?Measures effects on institutionalincentives, organizational structures,and economic outcomes over time.
Contextualizing via Human BaselinesEvaluators are now quantifying agent capability in terms of human time saved.100%75%50%25%0%Probability of success (0% to 100%)1 min10 min1 hour10 hours100 hoursTask duration for a human expert (log scale)The 50%Time HorizonThe Metric:The Task-CompletionTime Horizon tracksthe exact taskduration at which anagent is predicted tosucceed with a 50% or80% reliabilitythreshold.We are no longer justasking "Did it pass?"
Task length:
The Blueprint for Agent Evaluation.Shift the MindsetUpgrade the RigorBroaden the Scope•Stop grading isolatedoutcomes.•Evaluate compounding,multi-step trajectories.•Measure Cost, Latency,and Reliability alongsideAccuracy.•Disaggregate model vs.scaffold effects via factorialdesign.•Demand pass^k consistency,not just pass@k flukes.•Use log telemetry to catchsearch-time contaminationand reward hacking.•Move beyond technicalcapabilities.•Test human-agentinteraction and account for theSim2Real gap.•Contextualize successagainst human time horizonsand systemic economicimpact.EVALUATION_PROTOCOL // STATUS: FINALIZED // END_OF_FILE