Return to Jev From First Principles.
These tables preserve all rows and values from the enhanced edition. Columns are transposed into metric/value records for small screens. Chapter 10’s displayed examples use seed 0; its acceptance verdicts use medians across fit seeds. Bootstrap intervals need not contain a finite-sample point estimate.
Chapter 9, table 1
Return to Chapter 9.
Record 1: intent/test — prior/s0/w0 — 0.0064 [0.0028, 0.0103]
| Field |
Value |
| Dataset/split |
intent/test |
| Provider/seed/wording |
prior/s0/w0 |
| ECE10 width [95% bootstrap] |
0.0064 [0.0028, 0.0103] |
| Classwise ECE10 |
0.0026 |
| Brier sum |
0.9878 |
| Log loss |
4.3820 |
| Accuracy |
0.0130 |
| AUROC |
0.5000 |
| n |
3080 |
Record 2: intent/test — tfidf-lr/s0/w0 — 0.0519 [0.0430, 0.0616]
| Field |
Value |
| Dataset/split |
intent/test |
| Provider/seed/wording |
tfidf-lr/s0/w0 |
| ECE10 width [95% bootstrap] |
0.0519 [0.0430, 0.0616] |
| Classwise ECE10 |
0.0023 |
| Brier sum |
0.1835 |
| Log loss |
0.5147 |
| Accuracy |
0.8779 |
| AUROC |
0.9971 |
| n |
3080 |
Record 3: intent/test — fasttext-style/s0/w0 — 0.0408 [0.0343, 0.0540]
| Field |
Value |
| Dataset/split |
intent/test |
| Provider/seed/wording |
fasttext-style/s0/w0 |
| ECE10 width [95% bootstrap] |
0.0408 [0.0343, 0.0540] |
| Classwise ECE10 |
0.0026 |
| Brier sum |
0.2251 |
| Log loss |
0.6051 |
| Accuracy |
0.8526 |
| AUROC |
0.9962 |
| n |
3080 |
Record 4: intent/test — embed-sim/s0/w0 — 0.5630 [0.5469, 0.5784]
| Field |
Value |
| Dataset/split |
intent/test |
| Provider/seed/wording |
embed-sim/s0/w0 |
| ECE10 width [95% bootstrap] |
0.5630 [0.5469, 0.5784] |
| Classwise ECE10 |
0.0100 |
| Brier sum |
0.8226 |
| Log loss |
2.4848 |
| Accuracy |
0.6799 |
| AUROC |
0.9804 |
| n |
3080 |
Record 5: safety/test — prior/s0/w0 — 0.1484 [0.0622, 0.2432]
| Field |
Value |
| Dataset/split |
safety/test |
| Provider/seed/wording |
prior/s0/w0 |
| ECE10 width [95% bootstrap] |
0.1484 [0.0622, 0.2432] |
| Classwise ECE10 |
0.1484 |
| Brier sum |
0.5434 |
| Log loss |
0.7380 |
| Accuracy |
0.4828 |
| AUROC |
0.5000 |
| n |
116 |
Record 6: safety/test — tfidf-lr/s0/w0 — 0.0593 [0.0339, 0.1279]
| Field |
Value |
| Dataset/split |
safety/test |
| Provider/seed/wording |
tfidf-lr/s0/w0 |
| ECE10 width [95% bootstrap] |
0.0593 [0.0339, 0.1279] |
| Classwise ECE10 |
0.0859 |
| Brier sum |
0.1710 |
| Log loss |
0.3392 |
| Accuracy |
0.8966 |
| AUROC |
0.9661 |
| n |
116 |
Record 7: safety/test — fasttext-style/s0/w0 — 0.0642 [0.0378, 0.1212]
| Field |
Value |
| Dataset/split |
safety/test |
| Provider/seed/wording |
fasttext-style/s0/w0 |
| ECE10 width [95% bootstrap] |
0.0642 [0.0378, 0.1212] |
| Classwise ECE10 |
0.0743 |
| Brier sum |
0.1580 |
| Log loss |
0.2471 |
| Accuracy |
0.9052 |
| AUROC |
0.9658 |
| n |
116 |
Record 8: safety/test — embed-sim/s0/w0 — 0.0574 [0.0181, 0.1554]
| Field |
Value |
| Dataset/split |
safety/test |
| Provider/seed/wording |
embed-sim/s0/w0 |
| ECE10 width [95% bootstrap] |
0.0574 [0.0181, 0.1554] |
| Classwise ECE10 |
0.0685 |
| Brier sum |
0.4827 |
| Log loss |
0.6753 |
| Accuracy |
0.5345 |
| AUROC |
0.6423 |
| n |
116 |
Record 9: safety/shift — prior/s0/w0 — 0.1617 [0.1044, 0.2266]
| Field |
Value |
| Dataset/split |
safety/shift |
| Provider/seed/wording |
prior/s0/w0 |
| ECE10 width [95% bootstrap] |
0.1617 [0.1044, 0.2266] |
| Classwise ECE10 |
0.1617 |
| Brier sum |
0.5504 |
| Log loss |
0.7452 |
| Accuracy |
0.4695 |
| AUROC |
0.5000 |
| n |
262 |
Record 10: safety/shift — tfidf-lr/s0/w0 — 0.4375 [0.3754, 0.4937]
| Field |
Value |
| Dataset/split |
safety/shift |
| Provider/seed/wording |
tfidf-lr/s0/w0 |
| ECE10 width [95% bootstrap] |
0.4375 [0.3754, 0.4937] |
| Classwise ECE10 |
0.4396 |
| Brier sum |
0.8646 |
| Log loss |
3.4633 |
| Accuracy |
0.5496 |
| AUROC |
0.8644 |
| n |
262 |
Record 11: safety/shift — fasttext-style/s0/w0 — 0.4361 [0.3731, 0.4934]
| Field |
Value |
| Dataset/split |
safety/shift |
| Provider/seed/wording |
fasttext-style/s0/w0 |
| ECE10 width [95% bootstrap] |
0.4361 [0.3731, 0.4934] |
| Classwise ECE10 |
0.4378 |
| Brier sum |
0.8633 |
| Log loss |
2.1766 |
| Accuracy |
0.5382 |
| AUROC |
0.5115 |
| n |
262 |
Record 12: safety/shift — embed-sim/s0/w0 — 0.1861 [0.1209, 0.2420]
| Field |
Value |
| Dataset/split |
safety/shift |
| Provider/seed/wording |
embed-sim/s0/w0 |
| ECE10 width [95% bootstrap] |
0.1861 [0.1209, 0.2420] |
| Classwise ECE10 |
0.1861 |
| Brier sum |
0.5652 |
| Log loss |
0.7600 |
| Accuracy |
0.3740 |
| AUROC |
0.3085 |
| n |
262 |
Chapter 10, table 1
Return to Chapter 10.
Record 1: intent/test — embed-sim — isotonic — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
isotonic |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0863 [0.0743, 0.1001] |
| Classwise ECE |
0.0056 |
| Brier sum |
0.4483 |
| Log loss |
3.0628 |
| Accuracy |
0.6873 |
| AUROC |
0.9572 |
| n |
3080 |
Record 2: intent/test — embed-sim — sigmoid — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
sigmoid |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.2241 [0.2108, 0.2375] |
| Classwise ECE |
0.0070 |
| Brier sum |
0.4654 |
| Log loss |
1.2237 |
| Accuracy |
0.7097 |
| AUROC |
0.9851 |
| n |
3080 |
Record 3: intent/test — embed-sim — temperature — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
temperature |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0187 [0.0156, 0.0354] |
| Classwise ECE |
0.0055 |
| Brier sum |
0.4359 |
| Log loss |
1.2117 |
| Accuracy |
0.6799 |
| AUROC |
0.9861 |
| n |
3080 |
Record 4: intent/test — embed-sim — uncalibrated — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
uncalibrated |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.5630 [0.5469, 0.5784] |
| Classwise ECE |
0.0100 |
| Brier sum |
0.8226 |
| Log loss |
2.4848 |
| Accuracy |
0.6799 |
| AUROC |
0.9804 |
| n |
3080 |
Record 5: intent/test — embed-sim — vector — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
vector |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0551 [0.0428, 0.0683] |
| Classwise ECE |
0.0041 |
| Brier sum |
0.3880 |
| Log loss |
1.0291 |
| Accuracy |
0.7159 |
| AUROC |
0.9880 |
| n |
3080 |
Record 6: intent/test — fasttext-style — isotonic — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
isotonic |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0423 [0.0344, 0.0541] |
| Classwise ECE |
0.0029 |
| Brier sum |
0.2401 |
| Log loss |
2.4683 |
| Accuracy |
0.8474 |
| AUROC |
0.9663 |
| n |
3080 |
Record 7: intent/test — fasttext-style — sigmoid — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
sigmoid |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.1125 [0.1006, 0.1246] |
| Classwise ECE |
0.0040 |
| Brier sum |
0.2614 |
| Log loss |
0.6734 |
| Accuracy |
0.8468 |
| AUROC |
0.9963 |
| n |
3080 |
Record 8: intent/test — fasttext-style — temperature — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
temperature |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0256 [0.0186, 0.0391] |
| Classwise ECE |
0.0026 |
| Brier sum |
0.2258 |
| Log loss |
0.5705 |
| Accuracy |
0.8526 |
| AUROC |
0.9963 |
| n |
3080 |
Record 9: intent/test — fasttext-style — uncalibrated — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
uncalibrated |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0408 [0.0343, 0.0540] |
| Classwise ECE |
0.0026 |
| Brier sum |
0.2251 |
| Log loss |
0.6051 |
| Accuracy |
0.8526 |
| AUROC |
0.9962 |
| n |
3080 |
Record 10: intent/test — fasttext-style — vector — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
vector |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0260 [0.0224, 0.0399] |
| Classwise ECE |
0.0029 |
| Brier sum |
0.2343 |
| Log loss |
0.6835 |
| Accuracy |
0.8477 |
| AUROC |
0.9961 |
| n |
3080 |
Record 11: intent/test — prior — uncalibrated — 0/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
prior |
| Method |
uncalibrated |
| Labels/seed/wording |
0/0/w0 |
| ECE10 width [95% bootstrap] |
0.0064 [0.0028, 0.0103] |
| Classwise ECE |
0.0026 |
| Brier sum |
0.9878 |
| Log loss |
4.3820 |
| Accuracy |
0.0130 |
| AUROC |
0.5000 |
| n |
3080 |
Record 12: intent/test — tfidf-lr — isotonic — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
isotonic |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0432 [0.0347, 0.0535] |
| Classwise ECE |
0.0023 |
| Brier sum |
0.1919 |
| Log loss |
1.8461 |
| Accuracy |
0.8731 |
| AUROC |
0.9754 |
| n |
3080 |
Record 13: intent/test — tfidf-lr — sigmoid — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
sigmoid |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0832 [0.0729, 0.0933] |
| Classwise ECE |
0.0032 |
| Brier sum |
0.1974 |
| Log loss |
0.5254 |
| Accuracy |
0.8766 |
| AUROC |
0.9973 |
| n |
3080 |
Record 14: intent/test — tfidf-lr — temperature — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
temperature |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0117 [0.0093, 0.0238] |
| Classwise ECE |
0.0021 |
| Brier sum |
0.1768 |
| Log loss |
0.4481 |
| Accuracy |
0.8779 |
| AUROC |
0.9972 |
| n |
3080 |
Record 15: intent/test — tfidf-lr — uncalibrated — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
uncalibrated |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0519 [0.0430, 0.0616] |
| Classwise ECE |
0.0023 |
| Brier sum |
0.1835 |
| Log loss |
0.5147 |
| Accuracy |
0.8779 |
| AUROC |
0.9971 |
| n |
3080 |
Record 16: intent/test — tfidf-lr — vector — 1000/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
vector |
| Labels/seed/wording |
1000/0/w0 |
| ECE10 width [95% bootstrap] |
0.0229 [0.0167, 0.0339] |
| Classwise ECE |
0.0023 |
| Brier sum |
0.1847 |
| Log loss |
0.5610 |
| Accuracy |
0.8766 |
| AUROC |
0.9971 |
| n |
3080 |
Record 17: safety/shift — embed-sim — isotonic — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
embed-sim |
| Method |
isotonic |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.2592 [0.2116, 0.3295] |
| Classwise ECE |
0.2570 |
| Brier sum |
0.7001 |
| Log loss |
1.5463 |
| Accuracy |
0.4046 |
| AUROC |
0.3386 |
| n |
262 |
Record 18: safety/shift — embed-sim — sigmoid — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
embed-sim |
| Method |
sigmoid |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.2893 [0.2294, 0.3491] |
| Classwise ECE |
0.2893 |
| Brier sum |
0.6545 |
| Log loss |
0.8629 |
| Accuracy |
0.3550 |
| AUROC |
0.3085 |
| n |
262 |
Record 19: safety/shift — embed-sim — temperature — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
embed-sim |
| Method |
temperature |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.2447 [0.1970, 0.3030] |
| Classwise ECE |
0.2598 |
| Brier sum |
0.6483 |
| Log loss |
0.8581 |
| Accuracy |
0.3740 |
| AUROC |
0.3085 |
| n |
262 |
Record 20: safety/shift — embed-sim — uncalibrated — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
embed-sim |
| Method |
uncalibrated |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1861 [0.1209, 0.2420] |
| Classwise ECE |
0.1861 |
| Brier sum |
0.5652 |
| Log loss |
0.7600 |
| Accuracy |
0.3740 |
| AUROC |
0.3085 |
| n |
262 |
Record 21: safety/shift — embed-sim — vector — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
embed-sim |
| Method |
vector |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.2660 [0.2124, 0.3302] |
| Classwise ECE |
0.2660 |
| Brier sum |
0.6362 |
| Log loss |
0.8407 |
| Accuracy |
0.3702 |
| AUROC |
0.3085 |
| n |
262 |
Record 22: safety/shift — fasttext-style — isotonic — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
fasttext-style |
| Method |
isotonic |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.2796 [0.2245, 0.3391] |
| Classwise ECE |
0.2861 |
| Brier sum |
0.6516 |
| Log loss |
0.9323 |
| Accuracy |
0.5725 |
| AUROC |
0.5409 |
| n |
262 |
Record 23: safety/shift — fasttext-style — sigmoid — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
fasttext-style |
| Method |
sigmoid |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1572 [0.1129, 0.2215] |
| Classwise ECE |
0.1699 |
| Brier sum |
0.5610 |
| Log loss |
0.8021 |
| Accuracy |
0.5878 |
| AUROC |
0.5115 |
| n |
262 |
Record 24: safety/shift — fasttext-style — temperature — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
fasttext-style |
| Method |
temperature |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.3095 [0.2461, 0.3675] |
| Classwise ECE |
0.3138 |
| Brier sum |
0.6931 |
| Log loss |
1.0495 |
| Accuracy |
0.5382 |
| AUROC |
0.5115 |
| n |
262 |
Record 25: safety/shift — fasttext-style — uncalibrated — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
fasttext-style |
| Method |
uncalibrated |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.4361 [0.3731, 0.4934] |
| Classwise ECE |
0.4378 |
| Brier sum |
0.8633 |
| Log loss |
2.1766 |
| Accuracy |
0.5382 |
| AUROC |
0.5115 |
| n |
262 |
Record 26: safety/shift — fasttext-style — vector — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
fasttext-style |
| Method |
vector |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1747 [0.1164, 0.2344] |
| Classwise ECE |
0.1960 |
| Brier sum |
0.5784 |
| Log loss |
0.8108 |
| Accuracy |
0.5573 |
| AUROC |
0.5115 |
| n |
262 |
Record 27: safety/shift — prior — uncalibrated — 0/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
prior |
| Method |
uncalibrated |
| Labels/seed/wording |
0/0/w0 |
| ECE10 width [95% bootstrap] |
0.1617 [0.1044, 0.2266] |
| Classwise ECE |
0.1617 |
| Brier sum |
0.5504 |
| Log loss |
0.7452 |
| Accuracy |
0.4695 |
| AUROC |
0.5000 |
| n |
262 |
Record 28: safety/shift — tfidf-lr — isotonic — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
tfidf-lr |
| Method |
isotonic |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.4094 [0.3470, 0.4646] |
| Classwise ECE |
0.4119 |
| Brier sum |
0.8023 |
| Log loss |
13.1233 |
| Accuracy |
0.6031 |
| AUROC |
0.5903 |
| n |
262 |
Record 29: safety/shift — tfidf-lr — sigmoid — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
tfidf-lr |
| Method |
sigmoid |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.3701 [0.3152, 0.4266] |
| Classwise ECE |
0.3867 |
| Brier sum |
0.7205 |
| Log loss |
1.2790 |
| Accuracy |
0.5611 |
| AUROC |
0.8644 |
| n |
262 |
Record 30: safety/shift — tfidf-lr — temperature — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
tfidf-lr |
| Method |
temperature |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.4103 [0.3491, 0.4644] |
| Classwise ECE |
0.4176 |
| Brier sum |
0.7953 |
| Log loss |
1.7027 |
| Accuracy |
0.5496 |
| AUROC |
0.8644 |
| n |
262 |
Record 31: safety/shift — tfidf-lr — uncalibrated — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
tfidf-lr |
| Method |
uncalibrated |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.4375 [0.3754, 0.4937] |
| Classwise ECE |
0.4396 |
| Brier sum |
0.8646 |
| Log loss |
3.4633 |
| Accuracy |
0.5496 |
| AUROC |
0.8644 |
| n |
262 |
Record 32: safety/shift — tfidf-lr — vector — 90/0/w0
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
tfidf-lr |
| Method |
vector |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.3998 [0.3416, 0.4531] |
| Classwise ECE |
0.4098 |
| Brier sum |
0.7813 |
| Log loss |
1.8997 |
| Accuracy |
0.5573 |
| AUROC |
0.8644 |
| n |
262 |
Record 33: safety/test — embed-sim — isotonic — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
embed-sim |
| Method |
isotonic |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.2102 [0.1420, 0.3084] |
| Classwise ECE |
0.2091 |
| Brier sum |
0.5611 |
| Log loss |
0.7815 |
| Accuracy |
0.5172 |
| AUROC |
0.6629 |
| n |
116 |
Record 34: safety/test — embed-sim — sigmoid — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
embed-sim |
| Method |
sigmoid |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1950 [0.1330, 0.2867] |
| Classwise ECE |
0.2182 |
| Brier sum |
0.5441 |
| Log loss |
0.7443 |
| Accuracy |
0.4914 |
| AUROC |
0.6423 |
| n |
116 |
Record 35: safety/test — embed-sim — temperature — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
embed-sim |
| Method |
temperature |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1209 [0.0597, 0.2088] |
| Classwise ECE |
0.1378 |
| Brier sum |
0.5008 |
| Log loss |
0.6955 |
| Accuracy |
0.5345 |
| AUROC |
0.6423 |
| n |
116 |
Record 36: safety/test — embed-sim — uncalibrated — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
embed-sim |
| Method |
uncalibrated |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0574 [0.0181, 0.1554] |
| Classwise ECE |
0.0685 |
| Brier sum |
0.4827 |
| Log loss |
0.6753 |
| Accuracy |
0.5345 |
| AUROC |
0.6423 |
| n |
116 |
Record 37: safety/test — embed-sim — vector — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
embed-sim |
| Method |
vector |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1820 [0.1231, 0.2793] |
| Classwise ECE |
0.2012 |
| Brier sum |
0.5397 |
| Log loss |
0.7378 |
| Accuracy |
0.5000 |
| AUROC |
0.6423 |
| n |
116 |
Record 38: safety/test — fasttext-style — isotonic — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
fasttext-style |
| Method |
isotonic |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0574 [0.0241, 0.1183] |
| Classwise ECE |
0.1106 |
| Brier sum |
0.2023 |
| Log loss |
0.5848 |
| Accuracy |
0.8707 |
| AUROC |
0.9565 |
| n |
116 |
Record 39: safety/test — fasttext-style — sigmoid — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
fasttext-style |
| Method |
sigmoid |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0535 [0.0329, 0.1233] |
| Classwise ECE |
0.1416 |
| Brier sum |
0.2176 |
| Log loss |
0.3420 |
| Accuracy |
0.8534 |
| AUROC |
0.9658 |
| n |
116 |
Record 40: safety/test — fasttext-style — temperature — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
fasttext-style |
| Method |
temperature |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0765 [0.0604, 0.1280] |
| Classwise ECE |
0.0852 |
| Brier sum |
0.1594 |
| Log loss |
0.2710 |
| Accuracy |
0.9052 |
| AUROC |
0.9658 |
| n |
116 |
Record 41: safety/test — fasttext-style — uncalibrated — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
fasttext-style |
| Method |
uncalibrated |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0642 [0.0378, 0.1212] |
| Classwise ECE |
0.0743 |
| Brier sum |
0.1580 |
| Log loss |
0.2471 |
| Accuracy |
0.9052 |
| AUROC |
0.9658 |
| n |
116 |
Record 42: safety/test — fasttext-style — vector — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
fasttext-style |
| Method |
vector |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0659 [0.0479, 0.1357] |
| Classwise ECE |
0.1427 |
| Brier sum |
0.2002 |
| Log loss |
0.3328 |
| Accuracy |
0.8879 |
| AUROC |
0.9658 |
| n |
116 |
Record 43: safety/test — prior — uncalibrated — 0/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
prior |
| Method |
uncalibrated |
| Labels/seed/wording |
0/0/w0 |
| ECE10 width [95% bootstrap] |
0.1484 [0.0622, 0.2432] |
| Classwise ECE |
0.1484 |
| Brier sum |
0.5434 |
| Log loss |
0.7380 |
| Accuracy |
0.4828 |
| AUROC |
0.5000 |
| n |
116 |
Record 44: safety/test — tfidf-lr — isotonic — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
tfidf-lr |
| Method |
isotonic |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.1045 [0.0502, 0.1647] |
| Classwise ECE |
0.1045 |
| Brier sum |
0.2167 |
| Log loss |
0.5863 |
| Accuracy |
0.7845 |
| AUROC |
0.9604 |
| n |
116 |
Record 45: safety/test — tfidf-lr — sigmoid — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
tfidf-lr |
| Method |
sigmoid |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0099 [0.0223, 0.0905] |
| Classwise ECE |
0.1076 |
| Brier sum |
0.1922 |
| Log loss |
0.3123 |
| Accuracy |
0.8707 |
| AUROC |
0.9661 |
| n |
116 |
Record 46: safety/test — tfidf-lr — temperature — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
tfidf-lr |
| Method |
temperature |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0144 [0.0213, 0.0852] |
| Classwise ECE |
0.0698 |
| Brier sum |
0.1633 |
| Log loss |
0.2709 |
| Accuracy |
0.8966 |
| AUROC |
0.9661 |
| n |
116 |
Record 47: safety/test — tfidf-lr — uncalibrated — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
tfidf-lr |
| Method |
uncalibrated |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0593 [0.0339, 0.1279] |
| Classwise ECE |
0.0859 |
| Brier sum |
0.1710 |
| Log loss |
0.3392 |
| Accuracy |
0.8966 |
| AUROC |
0.9661 |
| n |
116 |
Record 48: safety/test — tfidf-lr — vector — 90/0/w0
| Field |
Value |
| Task/split |
safety/test |
| Provider |
tfidf-lr |
| Method |
vector |
| Labels/seed/wording |
90/0/w0 |
| ECE10 width [95% bootstrap] |
0.0367 [0.0249, 0.1083] |
| Classwise ECE |
0.1055 |
| Brier sum |
0.1955 |
| Log loss |
0.3107 |
| Accuracy |
0.8621 |
| AUROC |
0.9661 |
| n |
116 |
Chapter 10, table 2
Return to Chapter 10.
Record 1: intent — tfidf-lr — ece
| Field |
Value |
| Task |
intent |
| Provider |
tfidf-lr |
| Metric |
ece |
| Difference [95% paired bootstrap] |
-0.0402 [-0.0447, -0.0264] |
| n |
3080 |
Record 2: intent — tfidf-lr — brier
| Field |
Value |
| Task |
intent |
| Provider |
tfidf-lr |
| Metric |
brier |
| Difference [95% paired bootstrap] |
-0.0067 [-0.0096, -0.0035] |
| n |
3080 |
Record 3: intent — tfidf-lr — accuracy
| Field |
Value |
| Task |
intent |
| Provider |
tfidf-lr |
| Metric |
accuracy |
| Difference [95% paired bootstrap] |
0.0000 [0.0000, 0.0000] |
| n |
3080 |
Record 4: intent — fasttext-style — ece
| Field |
Value |
| Task |
intent |
| Provider |
fasttext-style |
| Metric |
ece |
| Difference [95% paired bootstrap] |
-0.0152 [-0.0305, -0.0000] |
| n |
3080 |
Record 5: intent — fasttext-style — brier
| Field |
Value |
| Task |
intent |
| Provider |
fasttext-style |
| Metric |
brier |
| Difference [95% paired bootstrap] |
0.0007 [-0.0016, 0.0032] |
| n |
3080 |
Record 6: intent — fasttext-style — accuracy
| Field |
Value |
| Task |
intent |
| Provider |
fasttext-style |
| Metric |
accuracy |
| Difference [95% paired bootstrap] |
0.0000 [0.0000, 0.0000] |
| n |
3080 |
Record 7: intent — embed-sim — ece
| Field |
Value |
| Task |
intent |
| Provider |
embed-sim |
| Metric |
ece |
| Difference [95% paired bootstrap] |
-0.5443 [-0.5500, -0.5223] |
| n |
3080 |
Record 8: intent — embed-sim — brier
| Field |
Value |
| Task |
intent |
| Provider |
embed-sim |
| Metric |
brier |
| Difference [95% paired bootstrap] |
-0.3867 [-0.4006, -0.3725] |
| n |
3080 |
Record 9: intent — embed-sim — accuracy
| Field |
Value |
| Task |
intent |
| Provider |
embed-sim |
| Metric |
accuracy |
| Difference [95% paired bootstrap] |
0.0000 [0.0000, 0.0000] |
| n |
3080 |
Record 10: safety — tfidf-lr — ece
| Field |
Value |
| Task |
safety |
| Provider |
tfidf-lr |
| Metric |
ece |
| Difference [95% paired bootstrap] |
-0.0449 [-0.0703, 0.0268] |
| n |
116 |
Record 11: safety — tfidf-lr — brier
| Field |
Value |
| Task |
safety |
| Provider |
tfidf-lr |
| Metric |
brier |
| Difference [95% paired bootstrap] |
-0.0077 [-0.0315, 0.0152] |
| n |
116 |
Record 12: safety — tfidf-lr — accuracy
| Field |
Value |
| Task |
safety |
| Provider |
tfidf-lr |
| Metric |
accuracy |
| Difference [95% paired bootstrap] |
0.0000 [0.0000, 0.0000] |
| n |
116 |
Record 13: safety — fasttext-style — ece
| Field |
Value |
| Task |
safety |
| Provider |
fasttext-style |
| Metric |
ece |
| Difference [95% paired bootstrap] |
0.0124 [-0.0442, 0.0726] |
| n |
116 |
Record 14: safety — fasttext-style — brier
| Field |
Value |
| Task |
safety |
| Provider |
fasttext-style |
| Metric |
brier |
| Difference [95% paired bootstrap] |
0.0014 [-0.0287, 0.0291] |
| n |
116 |
Record 15: safety — fasttext-style — accuracy
| Field |
Value |
| Task |
safety |
| Provider |
fasttext-style |
| Metric |
accuracy |
| Difference [95% paired bootstrap] |
0.0000 [0.0000, 0.0000] |
| n |
116 |
Record 16: safety — embed-sim — ece
| Field |
Value |
| Task |
safety |
| Provider |
embed-sim |
| Metric |
ece |
| Difference [95% paired bootstrap] |
0.0636 [-0.0053, 0.0865] |
| n |
116 |
Record 17: safety — embed-sim — brier
| Field |
Value |
| Task |
safety |
| Provider |
embed-sim |
| Metric |
brier |
| Difference [95% paired bootstrap] |
0.0181 [-0.0124, 0.0500] |
| n |
116 |
Record 18: safety — embed-sim — accuracy
| Field |
Value |
| Task |
safety |
| Provider |
embed-sim |
| Metric |
accuracy |
| Difference [95% paired bootstrap] |
0.0000 [0.0000, 0.0000] |
| n |
116 |
Chapter 10, table 3
Return to Chapter 10.
Record 1: intent/test — tfidf-lr — temperature — 50/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
temperature |
| Labels/seed/wording |
50/0/w0 |
| ECE10 width [95% bootstrap] |
0.0155 [0.0115, 0.0274] |
| Classwise ECE |
0.0022 |
| Brier sum |
0.1768 |
| Log loss |
0.4504 |
| Accuracy |
0.8779 |
| AUROC |
0.9972 |
| n |
3080 |
Record 2: intent/test — tfidf-lr — isotonic — 50/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
isotonic |
| Labels/seed/wording |
50/0/w0 |
| ECE10 width [95% bootstrap] |
0.1591 [0.1501, 0.1711] |
| Classwise ECE |
0.0055 |
| Brier sum |
0.6515 |
| Log loss |
2.8632 |
| Accuracy |
0.4581 |
| AUROC |
0.8566 |
| n |
3080 |
Record 3: intent/test — tfidf-lr — temperature — 100/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
temperature |
| Labels/seed/wording |
100/0/w0 |
| ECE10 width [95% bootstrap] |
0.0349 [0.0279, 0.0460] |
| Classwise ECE |
0.0022 |
| Brier sum |
0.1790 |
| Log loss |
0.4722 |
| Accuracy |
0.8779 |
| AUROC |
0.9972 |
| n |
3080 |
Record 4: intent/test — tfidf-lr — isotonic — 100/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Method |
isotonic |
| Labels/seed/wording |
100/0/w0 |
| ECE10 width [95% bootstrap] |
0.0957 [0.0905, 0.1076] |
| Classwise ECE |
0.0042 |
| Brier sum |
0.4564 |
| Log loss |
2.7424 |
| Accuracy |
0.6166 |
| AUROC |
0.9216 |
| n |
3080 |
Record 5: intent/test — fasttext-style — temperature — 50/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
temperature |
| Labels/seed/wording |
50/0/w0 |
| ECE10 width [95% bootstrap] |
0.0259 [0.0193, 0.0392] |
| Classwise ECE |
0.0026 |
| Brier sum |
0.2254 |
| Log loss |
0.5707 |
| Accuracy |
0.8526 |
| AUROC |
0.9963 |
| n |
3080 |
Record 6: intent/test — fasttext-style — isotonic — 50/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
isotonic |
| Labels/seed/wording |
50/0/w0 |
| ECE10 width [95% bootstrap] |
0.1533 [0.1438, 0.1664] |
| Classwise ECE |
0.0053 |
| Brier sum |
0.6682 |
| Log loss |
3.1343 |
| Accuracy |
0.4571 |
| AUROC |
0.8532 |
| n |
3080 |
Record 7: intent/test — fasttext-style — temperature — 100/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
temperature |
| Labels/seed/wording |
100/0/w0 |
| ECE10 width [95% bootstrap] |
0.0326 [0.0268, 0.0458] |
| Classwise ECE |
0.0026 |
| Brier sum |
0.2243 |
| Log loss |
0.5904 |
| Accuracy |
0.8526 |
| AUROC |
0.9962 |
| n |
3080 |
Record 8: intent/test — fasttext-style — isotonic — 100/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Method |
isotonic |
| Labels/seed/wording |
100/0/w0 |
| ECE10 width [95% bootstrap] |
0.0842 [0.0735, 0.0958] |
| Classwise ECE |
0.0040 |
| Brier sum |
0.4888 |
| Log loss |
3.0462 |
| Accuracy |
0.6045 |
| AUROC |
0.9140 |
| n |
3080 |
Record 9: intent/test — embed-sim — temperature — 50/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
temperature |
| Labels/seed/wording |
50/0/w0 |
| ECE10 width [95% bootstrap] |
0.0175 [0.0155, 0.0340] |
| Classwise ECE |
0.0055 |
| Brier sum |
0.4358 |
| Log loss |
1.2116 |
| Accuracy |
0.6799 |
| AUROC |
0.9861 |
| n |
3080 |
Record 10: intent/test — embed-sim — isotonic — 50/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
isotonic |
| Labels/seed/wording |
50/0/w0 |
| ECE10 width [95% bootstrap] |
0.0842 [0.0695, 0.0973] |
| Classwise ECE |
0.0053 |
| Brier sum |
0.8055 |
| Log loss |
4.8522 |
| Accuracy |
0.3558 |
| AUROC |
0.7848 |
| n |
3080 |
Record 11: intent/test — embed-sim — temperature — 100/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
temperature |
| Labels/seed/wording |
100/0/w0 |
| ECE10 width [95% bootstrap] |
0.0154 [0.0147, 0.0346] |
| Classwise ECE |
0.0054 |
| Brier sum |
0.4356 |
| Log loss |
1.2117 |
| Accuracy |
0.6799 |
| AUROC |
0.9861 |
| n |
3080 |
Record 12: intent/test — embed-sim — isotonic — 100/0/w0
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Method |
isotonic |
| Labels/seed/wording |
100/0/w0 |
| ECE10 width [95% bootstrap] |
0.0643 [0.0522, 0.0803] |
| Classwise ECE |
0.0059 |
| Brier sum |
0.6993 |
| Log loss |
5.3657 |
| Accuracy |
0.4685 |
| AUROC |
0.8491 |
| n |
3080 |
Chapter 10, table 4
Return to Chapter 10.
Record 1: intent/test — tfidf-lr — 1000
| Field |
Value |
| Task/split |
intent/test |
| Provider |
tfidf-lr |
| Labels |
1000 |
| Coverage [95% bootstrap] |
0.9088 [0.8987, 0.9192] |
| Mean set size [95% bootstrap] |
1.0955 [1.0844, 1.1062] |
| Empty share |
0.0010 |
| Singleton share |
0.9081 |
| n |
3080 |
Record 2: intent/test — fasttext-style — 1000
| Field |
Value |
| Task/split |
intent/test |
| Provider |
fasttext-style |
| Labels |
1000 |
| Coverage [95% bootstrap] |
0.9084 [0.8987, 0.9185] |
| Mean set size [95% bootstrap] |
1.2545 [1.2344, 1.2744] |
| Empty share |
0.0000 |
| Singleton share |
0.7977 |
| n |
3080 |
Record 3: intent/test — embed-sim — 1000
| Field |
Value |
| Task/split |
intent/test |
| Provider |
embed-sim |
| Labels |
1000 |
| Coverage [95% bootstrap] |
0.9166 [0.9068, 0.9253] |
| Mean set size [95% bootstrap] |
6.1019 [6.0156, 6.1848] |
| Empty share |
0.0003 |
| Singleton share |
0.0133 |
| n |
3080 |
Record 4: safety/test — tfidf-lr — 90
| Field |
Value |
| Task/split |
safety/test |
| Provider |
tfidf-lr |
| Labels |
90 |
| Coverage [95% bootstrap] |
0.9310 [0.8793, 0.9741] |
| Mean set size [95% bootstrap] |
1.1034 [1.0517, 1.1638] |
| Empty share |
0.0000 |
| Singleton share |
0.8966 |
| n |
116 |
Record 5: safety/shift — tfidf-lr — 90
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
tfidf-lr |
| Labels |
90 |
| Coverage [95% bootstrap] |
0.5687 [0.5076, 0.6221] |
| Mean set size [95% bootstrap] |
1.0229 [1.0038, 1.0420] |
| Empty share |
0.0000 |
| Singleton share |
0.9771 |
| n |
262 |
Record 6: safety/test — fasttext-style — 90
| Field |
Value |
| Task/split |
safety/test |
| Provider |
fasttext-style |
| Labels |
90 |
| Coverage [95% bootstrap] |
0.9569 [0.9224, 0.9914] |
| Mean set size [95% bootstrap] |
1.1466 [1.0862, 1.2155] |
| Empty share |
0.0000 |
| Singleton share |
0.8534 |
| n |
116 |
Record 7: safety/shift — fasttext-style — 90
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
fasttext-style |
| Labels |
90 |
| Coverage [95% bootstrap] |
0.5649 [0.5038, 0.6183] |
| Mean set size [95% bootstrap] |
1.0344 [1.0153, 1.0573] |
| Empty share |
0.0000 |
| Singleton share |
0.9656 |
| n |
262 |
Record 8: safety/test — embed-sim — 90
| Field |
Value |
| Task/split |
safety/test |
| Provider |
embed-sim |
| Labels |
90 |
| Coverage [95% bootstrap] |
0.8793 [0.8190, 0.9397] |
| Mean set size [95% bootstrap] |
1.6466 [1.5603, 1.7414] |
| Empty share |
0.0000 |
| Singleton share |
0.3534 |
| n |
116 |
Record 9: safety/shift — embed-sim — 90
| Field |
Value |
| Task/split |
safety/shift |
| Provider |
embed-sim |
| Labels |
90 |
| Coverage [95% bootstrap] |
0.8550 [0.8092, 0.8969] |
| Mean set size [95% bootstrap] |
1.8168 [1.7672, 1.8626] |
| Empty share |
0.0000 |
| Singleton share |
0.1832 |
| n |
262 |