Hidden-effect recall
Of the relevant hidden effects in the evaluation, how many did the system identify?
SUPRAOS-REPORTED TECHNICAL EVALUATION
Inspect the consequences
the model catches.
SupraOS-reported evaluation of hidden effects, delayed risk and policy violations left undetected. The primary results, policy-invisible slice and transfer evaluations are shown separately.
PRIMARY RESULTS
VTWM and the baseline.
Baseline: language-model agent plus policy monitor. The measures refer to the published evaluation population and slices.
| Measure | VTWM | Baseline | Direction |
|---|---|---|---|
| Hidden-effect recall | 84.13% | 61.94% | Higher is better |
| Delayed-risk AUROC | 89.57% | 84.96% | Higher is better |
| Silent policy-violation rate | 1.31% | 21.76% | Lower is better |
SEPARATE POLICY-INVISIBLE SLICE
2 violations left silent.
Compared with 167.
This slice is reported separately from the primary results above. Both systems were evaluated on the same stated case and violation counts.
| Measure | VTWM | Baseline |
|---|---|---|
| Cases | 2,400 | 2,400 |
| Policy violations | 648 | 648 |
| Hidden effects found | 356 / 480 | 257 / 480 |
| Delayed-risk AUPRC | 86.81% | 76.46% |
| Reviews | 1,071 | 1,389 |
| Violations left silent | 2 | 167 |
TRANSFER EVALUATION
New structures, environments
and policy families.
| Evaluation slice | Transition F1 | Hidden-effect recall | Delayed-risk AUPRC |
|---|---|---|---|
| New workflow structures | 94.21 | 84.04% | 86.09% |
| New enterprise environment | 94.15 | 86.98% | 85.31% |
| New policy family | 94.28 | 84.10% | 85.64% |
INTERPRETING THE MEASURES
Different questions need
different evidence.
Delayed-risk AUROC and AUPRC
Delayed-risk ranking is reported separately as AUROC and AUPRC.
Silent policy failure
How often does a policy-breaking action avoid the relevant detection or review path? Read the stated population and slice with the measure.
Technical review
Request the evaluation protocol, data scope, split definitions and supporting artifacts when assessing these reported results for a deployment decision.
Discuss the evaluationPUT SUPRAOS TO WORK
Review the experiment behind the score.
Request model and baseline versions, splits, uncertainty, ablations and supporting artifacts.