exp-05Empirical AI Benchmarking · Decision Systems
11 min read · Empirical Benchmark Report“Can a general-purpose probabilistic decision primitive safely extract and retain operational decision signal from software-change metadata under distribution shift?”
We evaluated whether a general-purpose probabilistic decision primitive can extract and retain useful decision signal from software-change metadata under distribution shift. Using dependency updates as the concrete test case, JEV (TypeSafe's System One decision model) was compared against a deterministic static policy and a strong general LLM (DeepSeek Flash) on identical pre-merge state. On a 1,102-case in-distribution benchmark (551 breaking / 551 control), JEV showed substantially stronger ranking quality (AUROC 0.851) than static rules (0.602) and DeepSeek Flash (0.585). On an independently constructed 185-case out-of-distribution (OOD) set (100 breaking / 85 control, 103 independent repositories), JEV retained the highest AUROC among the three (0.605, 95% CI 0.525–0.685), but absolute performance fell sharply. The critical operational finding is negative: JEV's frozen in-distribution threshold did not transfer safely—the 0.62 threshold yielded 30 auto-merges at 50.0% precision with 15 unsafe merges. These empirical results demonstrate that while structured decision models can capture strong ranking signal, probability calibration and threshold stability remain unresolved under domain shift.