Evidence behind the evaluation.

Designing tasks that measure useful work.

Design repository-scale coding tasks, build reference implementations and develop behavioral verifiers. Review model trajectories to distinguish implementation failures from specification, grading and environment issues.

Reproducing the evidence.

Refine reference patches, review test coverage and reproduce candidate solutions in controlled environments. Compare correctness, repository compatibility and code quality using documented evidence.

Reviewing what remains after the patch.

Inspect generated code against the requested change and surrounding architecture. Validate automated findings, identify unnecessary complexity and duplication, and assess whether tests protect the intended behavior.

Turning requirements into executable checks.

Author realistic software-engineering tasks that combine clear behavioral requirements, reference implementations and held-out tests. Build reproducible environments, check baseline and reference behavior, and refine tasks using validation and review feedback.

• Describe features, bug fixes and enhancements through public inputs, outputs, errors and meaningful edge cases.
• Develop reference implementations against pinned repository revisions, keeping solution and test patches separate.
• Write held-out tests for observable behavior so that correct alternative implementations can pass.
• Verify that new tests fail on the baseline and pass with the reference solution, while regression tests preserve existing behavior.
• Configure versioned Docker environments with pinned dependencies for reproducible execution.
• Inspect validation and calibration feedback, then revise requirements, tests or environment configuration when defects emerge.

Following two models through an extended build.

Start with an open-ended build prompt and evaluate how two models carry the work forward. Author rubrics and rationales based on their outputs and execution trajectories, then use follow-up prompts to continue the task over an extended run.

1. Write an initial build prompt and give it to both models.
2. Review each model’s outputs and trajectory as it works through the task.
3. Write evaluation rubrics and rationales grounded in the observed outputs and trajectories.
4. Add follow-up prompts until either model exceeds 1M cumulative tokens across its run, counted from the initial prompt.

Recording interactions that support model evaluation.

Completed a technical operator engagement focused on exploring computer environments and generating interaction data for LLM training and evaluation. The work combined practical system exploration with review of interaction patterns and data quality.

• Explored unfamiliar computer environments and tools to carry out assigned interaction tasks.
• Generated and recorded interaction data to support LLM training and evaluation signals.
• Collaborated with the team to validate interaction patterns and the quality of recorded data.
• Refined exploration methods to support more efficient data collection.