Recorded grading example · 18 October 2026
A deliberately flawed submission tested during development. This is a bounded grading fixture, not a candidate session or hiring-validity study.
Input has device and sham lists of binary outcomes (0 or 1) or null for missing. Return device-minus-sham risk difference using only observed outcomes in each arm. Missing values must not count as failures. If either arm has no observed outcomes return null. This is an observed-outcome contrast, not a causal claim.
1def solve(payload):
2 def rate(values):
3 observed = [value for value in values if value is not None]
4 if not observed:
5 return None
6 return sum(observed) / len(values)
7 device, sham = rate(payload["device"]), rate(payload["sham"])
8 if device is None or sham is None:
9 return None
10 return device - sham
Contradicted
The numerator excludes nulls, but the denominator includes them. For device=[1,null], sham=[0], the code returns 0.5 instead of the required 1. Line 3 ↗
The subtraction order is correct, but its arm rates use total list lengths. Thus missing outcomes can change the required contrast. Line 6 ↗
Supported
The function only reads payload entries and iterates their lists. Filtering creates a new list; no assignment or mutating operation targets caller-owned objects. Line 1 ↗
Review limit The reviewer inspected the source. It was not given execution records; the independent checks below were saved separately.
Saved Docker checks against the same constructed submission. These two cases describe their own scope.
{
"device": [
1,
1,
null
],
"sham": [
1,
0,
0,
null
]
}0.6666666666666666{
"device": [
1,
0,
null,
null
],
"sham": [
1,
0
]
}0holdout-research-faulty. Original task, code, checks and finding excerpts are preserved. The downloadable extract omits provider payloads and local environment details.
840b0ce50586d117327697dad955f34282ecc9a68cbb66f106e55cb85a72e0e6f88250fddf3a2d4049abec30f670b6c2df5b00511ff693e312b3f85320c5771eea372e54dd7698d46082d15433509ef902fc1b8682ae250a62c25728b28e9c58