L4: Why Context Matters

Short Course Q&A AI Code Review
1

I was reading through src/utils/matching.py and calculate_metrics from the shared eval helpers, and I think the matching is many-to-one in a way the metric arithmetic doesn’t account for. Wanted to check whether this is a deliberate simplification for the lab or something worth fixing.

match_findings loops over expected_issues and appends the expected index:

for i, expected in enumerate(expected_issues):
    match_idx, score = find_best_match(expected, found_issue_strings, ...)
    if match_idx is not None:
        matched_indices.append(i)   # match_idx is discarded

Since find_best_match re-scans the full findings list each time and nothing marks a finding as consumed, several expected issues can all resolve to the same finding. But calculate_metrics does fp = len(findings) - tp, which only holds if the matching is one-to-one.

Minimal repro (keyword path only, use_embeddings=False):

expected = ["missing input validation",
            "missing authentication check",
            "missing rate limiting"]
findings = [{'issue': "This handler is missing several protections around untrusted input"}]
# all three match findings[0] on the shared token "missing"
# tp=3, fp = 1 - 3 = -2, fn=0  ->  precision 300%, recall 100%, F1 150%

Two things make this easy to hit. First, keyword_score is normalised by len(expected_keywords) only, so with a 3-token expected string a single shared token gives 0.33, which clears the 0.3 threshold. Second, extract_keywords drops words of length ≤ 3, so "SQL injection" reduces to {injection} and any finding containing that word scores 1.0 — including "uses dependency injection" or "no injection issues found here".

The blow-up case is rare, but the quieter version isn’t: with 5 findings and 2 expected issues collapsing onto the same finding, you get tp=2, fp=3 when 4 findings are actually unmatched. Precision comes out ~10 points high and nothing looks wrong. Since tp + fp = len(findings) by construction, precision reduces to tp / total_findings and recall to tp / total_expected — both move in the same direction when matching gets loose, so there’s no self-correcting trade-off. My worry is that this biases toward whichever agent emits more verbose findings.

Questions:

  1. Should match_findings consume findings (a used set, or scipy.optimize.linear_sum_assignment over the score matrix), and should fp be len(findings) - len(used) with an assert fp >= 0?
  2. How were the thresholds picked — 0.3 for keywords and 0.45 cosine for text-embedding-3-small? 0.45 feels low for short strings that all sit in the same technical vocabulary. Is there a labelled set of (expected, finding) pairs the thresholds were calibrated against?
  3. issues_match gates embeddings on keyword_score < threshold (0.3) while find_best_match gates on < 0.5, so the display path and the eval path can disagree on the same pair. Intentional?