Jev: Full Uncertainty and Calibration Tables

Return to Jev From First Principles.

These tables preserve all rows and values from the enhanced edition. Columns are transposed into metric/value records for small screens. Chapter 10’s displayed examples use seed 0; its acceptance verdicts use medians across fit seeds. Bootstrap intervals need not contain a finite-sample point estimate.

Chapter 9, table 1

Return to Chapter 9.

Record 1: intent/test — prior/s0/w0 — 0.0064 [0.0028, 0.0103]

Field Value
Dataset/split intent/test
Provider/seed/wording prior/s0/w0
ECE10 width [95% bootstrap] 0.0064 [0.0028, 0.0103]
Classwise ECE10 0.0026
Brier sum 0.9878
Log loss 4.3820
Accuracy 0.0130
AUROC 0.5000
n 3080

Record 2: intent/test — tfidf-lr/s0/w0 — 0.0519 [0.0430, 0.0616]

Field Value
Dataset/split intent/test
Provider/seed/wording tfidf-lr/s0/w0
ECE10 width [95% bootstrap] 0.0519 [0.0430, 0.0616]
Classwise ECE10 0.0023
Brier sum 0.1835
Log loss 0.5147
Accuracy 0.8779
AUROC 0.9971
n 3080

Record 3: intent/test — fasttext-style/s0/w0 — 0.0408 [0.0343, 0.0540]

Field Value
Dataset/split intent/test
Provider/seed/wording fasttext-style/s0/w0
ECE10 width [95% bootstrap] 0.0408 [0.0343, 0.0540]
Classwise ECE10 0.0026
Brier sum 0.2251
Log loss 0.6051
Accuracy 0.8526
AUROC 0.9962
n 3080

Record 4: intent/test — embed-sim/s0/w0 — 0.5630 [0.5469, 0.5784]

Field Value
Dataset/split intent/test
Provider/seed/wording embed-sim/s0/w0
ECE10 width [95% bootstrap] 0.5630 [0.5469, 0.5784]
Classwise ECE10 0.0100
Brier sum 0.8226
Log loss 2.4848
Accuracy 0.6799
AUROC 0.9804
n 3080

Record 5: safety/test — prior/s0/w0 — 0.1484 [0.0622, 0.2432]

Field Value
Dataset/split safety/test
Provider/seed/wording prior/s0/w0
ECE10 width [95% bootstrap] 0.1484 [0.0622, 0.2432]
Classwise ECE10 0.1484
Brier sum 0.5434
Log loss 0.7380
Accuracy 0.4828
AUROC 0.5000
n 116

Record 6: safety/test — tfidf-lr/s0/w0 — 0.0593 [0.0339, 0.1279]

Field Value
Dataset/split safety/test
Provider/seed/wording tfidf-lr/s0/w0
ECE10 width [95% bootstrap] 0.0593 [0.0339, 0.1279]
Classwise ECE10 0.0859
Brier sum 0.1710
Log loss 0.3392
Accuracy 0.8966
AUROC 0.9661
n 116

Record 7: safety/test — fasttext-style/s0/w0 — 0.0642 [0.0378, 0.1212]

Field Value
Dataset/split safety/test
Provider/seed/wording fasttext-style/s0/w0
ECE10 width [95% bootstrap] 0.0642 [0.0378, 0.1212]
Classwise ECE10 0.0743
Brier sum 0.1580
Log loss 0.2471
Accuracy 0.9052
AUROC 0.9658
n 116

Record 8: safety/test — embed-sim/s0/w0 — 0.0574 [0.0181, 0.1554]

Field Value
Dataset/split safety/test
Provider/seed/wording embed-sim/s0/w0
ECE10 width [95% bootstrap] 0.0574 [0.0181, 0.1554]
Classwise ECE10 0.0685
Brier sum 0.4827
Log loss 0.6753
Accuracy 0.5345
AUROC 0.6423
n 116

Record 9: safety/shift — prior/s0/w0 — 0.1617 [0.1044, 0.2266]

Field Value
Dataset/split safety/shift
Provider/seed/wording prior/s0/w0
ECE10 width [95% bootstrap] 0.1617 [0.1044, 0.2266]
Classwise ECE10 0.1617
Brier sum 0.5504
Log loss 0.7452
Accuracy 0.4695
AUROC 0.5000
n 262

Record 10: safety/shift — tfidf-lr/s0/w0 — 0.4375 [0.3754, 0.4937]

Field Value
Dataset/split safety/shift
Provider/seed/wording tfidf-lr/s0/w0
ECE10 width [95% bootstrap] 0.4375 [0.3754, 0.4937]
Classwise ECE10 0.4396
Brier sum 0.8646
Log loss 3.4633
Accuracy 0.5496
AUROC 0.8644
n 262

Record 11: safety/shift — fasttext-style/s0/w0 — 0.4361 [0.3731, 0.4934]

Field Value
Dataset/split safety/shift
Provider/seed/wording fasttext-style/s0/w0
ECE10 width [95% bootstrap] 0.4361 [0.3731, 0.4934]
Classwise ECE10 0.4378
Brier sum 0.8633
Log loss 2.1766
Accuracy 0.5382
AUROC 0.5115
n 262

Record 12: safety/shift — embed-sim/s0/w0 — 0.1861 [0.1209, 0.2420]

Field Value
Dataset/split safety/shift
Provider/seed/wording embed-sim/s0/w0
ECE10 width [95% bootstrap] 0.1861 [0.1209, 0.2420]
Classwise ECE10 0.1861
Brier sum 0.5652
Log loss 0.7600
Accuracy 0.3740
AUROC 0.3085
n 262

Chapter 10, table 1

Return to Chapter 10.