SeeTheWork / Evidence brief
ML research
Synthetic sample
Reviewer summary
The experiment is traceable. The claim needs a wider check.
The sample records seeds and settings, compares against a matched baseline, and limits its conclusion to the tested setup. Evidence for broad adoption remains incomplete. [04]
Competency evidence
Makes a controlled comparison
Supported in this example
Calibrates an adoption recommendation
Follow-up needed
Source register
All excerpts below are illustrative, including filenames and check results. They explain the review method and contain no real candidate data.
- [01]
Experiment manifest
IllustrativeRecord the conditions.
experiment.yaml
seeds: [11, 23, 37] training_steps: 1200 data_split: fixed_split_v1 methods: [baseline, proposed] record: [configuration, runtime, metric, failures]- What it supports
- The manifest states the settings needed to interpret and repeat the comparison.
- What remains open
- An executable environment and dependency versions are also needed for reproduction.
- [02]
Run ledger
IllustrativeKeep unsuccessful attempts visible.
run_ledger.csv · illustrative excerpt
seed,method,status 11,baseline,complete 11,proposed,complete 23,baseline,complete 23,proposed,failed_and_recorded- What it supports
- The ledger makes a failed attempt visible instead of silently removing it from the record.
- What remains open
- The failure needs diagnosis; this excerpt alone does not establish its cause.
- [03]
Comparison plan
IllustrativeCompare matched experiments.
analysis.md
Pair runs by seed and data split. Report each result and the spread. Track compute consumed by both methods. Do not replace failed runs without disclosure.- What it supports
- The plan accounts for experimental variation and resource cost when comparing methods.
- What remains open
- Few runs and one dataset provide limited support for generalization.
- [04]
Research verdict
IllustrativeTie the recommendation to the evidence.
recommendation.md
“Continue investigation; do not adopt yet. Resolve the failed run and test a second setting. The current result supports only this setup.”- What it supports
- The recommendation names the next work and limits the claim to available evidence.
- What remains open
- The reviewer should ask how the next experiment was prioritized within the budget.