SeeTheWork

Recorded grading example · 18 October 2026

Omitted attention scaling; both checks failed.

A deliberately flawed submission tested during development. This is a bounded grading fixture, not a candidate session or hiring-validity study.

The task

Input has a nonempty query vector q of depth D and a nonempty list k of key vectors of the same depth. Return key probabilities softmax(dot(q,k)/sqrt(D)). Use a maximum shift before exponentiation. All dot products are finite and within [-100,100]. Do not mutate input vectors.

Submitted work solution.py

1import math
2def solve(payload):
3    q, keys = payload["q"], payload["k"]
4    scale = 1.0
5    logits = [sum(a*b for a, b in zip(q, key)) / scale for key in keys]
6    maximum = max(logits)
7    exps = [math.exp(value - maximum) for value in logits]
8    total = sum(exps)
9    return [value / total for value in exps]

Captured reviewer findings

The implementation computes the required result for all stated inputs.

Contradicted

Scale is fixed at 1. For q=[1,0] and keys=[[1,0],[0,0]], logits are [1,0] instead of [1/sqrt(2),0], yielding different probabilities despite valid finite dot products. Line 3 ↗

Exponentials are normalized in key order, but normalization uses unscaled logits. The valid depth-two example therefore returns approximately [0.7311,0.2689], not [0.6698,0.3302]. Line 4 ↗

The implementation leaves caller-owned input lists, records and vectors unchanged.

Supported

The function only reads payload entries and iterates over vectors. It constructs new logits, exponentials, and result lists without assigning into caller-owned objects. Line 2 ↗

Review limit The reviewer inspected the source. It was not given execution records; the independent checks below were saved separately.

Independent execution checks

Saved Docker checks against the same constructed submission. These two cases describe their own scope.

Failed Check 1
Input
{
  "k": [
    [
      2,
      0,
      0,
      0
    ],
    [
      0,
      0,
      0,
      0
    ]
  ],
  "q": [
    2,
    0,
    0,
    0
  ]
}
Expected result
[0.8807970779778823,0.11920292202211755]
Failed Check 2
Input
{
  "k": [
    [
      0,
      0,
      0,
      -2
    ],
    [
      0,
      0,
      0,
      0
    ]
  ],
  "q": [
    0,
    0,
    0,
    3
  ]
}
Expected result
[0.04742587317756678,0.9525741268224334]
Record and provenance

holdout-ml-faulty. Original task, code, checks and finding excerpts are preserved. The downloadable extract omits provider payloads and local environment details.

reportSha256
f76549e726a40e5b7425093a5135597cfc3cddd3846e094617a77b2097945478
requestSha256
0ebe4da0ba536d22c9942dd9d3c1fdb7f82dca01a504123f90894b4239d04f74
sourceSha256
480f0b550fb34afa5ff143631841f0d7a4c0c498abaab709b62f8f74f8cc0c13
Download sanitized record ↓