A quick syntax question, an unfamiliar repository, and a delicate migration ask for different things. Build your shortlist around the work you repeat. A convincing explanation is useful, but it cannot substitute for a correct change.
Make a small evaluation set
Choose three tasks from a repository you know: explain a real failure, make a bounded edit, and identify a missing edge case. Keep the starting commit, instructions, and available files identical. Use a disposable branch and record any help you give.
Keep separate scorecards
| Question | Evidence to keep |
|---|---|
| Did it solve the problem? | Reproduction before; focused check after. |
| Did it stay in scope? | The complete diff, including deletions. |
| Could you review it? | Assumptions, file references, and uncertainty. |
| Does it fit your setup? | Actual endpoint behavior and observed usage. |
Read the failures, too
A single successful run tells you little about consistency. Repeat the difficult case and note whether the model asks for missing evidence or fills the gap with a guess. Do not combine every observation into one mysterious score: a cheap answer that needs extensive repair may be the expensive choice.
Choose provisionally
Select a default for a named task class, then write down why. Revisit the choice when the model, provider configuration, or workload changes. These are your observations in one setup, not universal benchmark results.
