Under asymmetric misclassification costs, does post-hoc probability calibration with the theoretical cost-optimal threshold beat directly tuning the decision threshold on the same held-out validation data?
Paper
Calibrate or Tune? Cost-Sensitive Decision Thresholds as Estimators on Spambase
Abstract
Under asymmetric costs, held-out data can recalibrate scores for the theoretical threshold or tune the threshold directly. Monotone calibration followed by thresholds the raw score, so both recipes estimate one cut. We make explicit that isotonic plug-in attains the minimum empirical validation cost, and contrast tuning's cube-root rate with the parametric rate and bias of Platt's sigmoid. On Spambase (4,210 e-mails; 3 learners × 6 cost proportions × 5 budgets of 50–800 e-mails; 50 seeds), the pre-specified pooled comparison at 100 e-mails is unsupported: Platt minus tuned excess cost is −7.9 points (95% CI −16.5 to 0.6). An absolute-cost endpoint, not pre-specified, favours tuning at that budget by 6.9 per 1,000 e-mails and survives a split-overlap correction from 100 e-mails upward (not at 50), but naive Bayes drives it entirely. At the seed level, Platt beats tuning for logistic regression and gradient boosting and trails for naive Bayes, and isotonic's small-budget edge comes from cut placement; under the correction only the naive Bayes deficit survives, so the rest are point-estimate patterns on this corpus. Winsorising naive Bayes's heavy-tailed log-odds halves its Platt excess, leaving a residual bias.
§Ask about this paper
Grounded in this runQuestions are answered from this run's record — the paper, the executed analysis, the data provenance and the referee report. Nothing else is consulted.
This run is finished, so the recorded paper cannot be rewritten. Questions are still answered from its record.
Pipeline
1.Literature review
Literature review complete (live:agent:claude-opus-5-5): 10 key papers, 26 grounded sources, 5 open gaps.
Summary
Decision theory gives a clean prescription for cost-sensitive binary classification. With false-positive cost c_FP and false-negative cost c_FN, predict positive when P(y=1|x) exceeds the cost proportion c = c_FP/(c_FP + c_FN) (Elkan, 2001). For perfectly calibrated scores, this score-driven threshold coincides with the optimal threshold choice (Hernandez-Orallo, Flach and Ferri, 2012, Theorem 43). Practice offers two ways to use a held-out validation sample. The first is to recalibrate the scores post hoc and plug in t = c. The usual calibrators are Platt's two-parameter sigmoid with smoothed targets (Platt, 2000) and nonparametric isotonic regression computed by pool-adjacent-violators (Zadrozny and Elkan, 2002). The second skips calibration and searches for the score threshold that minimises empirical cost (Sheng and Ling, 2006). The empirical record seems to favour search. Sheng and Ling found it beat the theoretical threshold on 9 of 12 datasets, but their probabilities, from C4.5, were never calibrated, so they did not test the comparison posed here. Two further results sharpen the question. First, PAV calibration is equivalent to the ROC convex hull (Fawcett and Niculescu-Mizil, 2007; Berta, Bach and Jordan, 2024). This implies that isotonic calibration plus t = c already selects an empirical-cost-minimising cut of the validation sample. For isotonic, the two recipes should then differ only through tie-breaking and where the cut falls between adjacent scores. Nobody has measured how large that residual is. Second, the remaining contest is between a parametric calibrator and empirical search, and it has a bias-variance structure. A cost-minimising threshold is an argmax-type estimator with cube-root asymptotics (Kim and Pollard, 1990). Its excess cost should decay like m^{-2/3}. A maximum-likelihood sigmoid converges at the parametric rate, but to the threshold implied by its Kullback-Leibler projection. It is therefore biased unless the sigmoid is right at the decision level c. Niculescu-Mizil and Caruana (2005) found Platt scaling better than isotonic regression when the calibration set has fewer than about 200-1000 cases. They measured squared error, not decision cost, and also documented learner-specific distortions: boosting sigmoidal, naive Bayes pushed towards 0 and 1, logistic regression near-calibrated. Recent theory decomposes decision regret into a calibration-induced part that recalibration can remove and a grouping loss it cannot (Perez-Lebel et al., 2025). It does not say how much validation data the recalibration step needs relative to direct tuning. Evaluation practice reports total or normalised cost averaged over repeated random splits, usually as mean and standard deviation. Intervals for cost differences are rare, and the split-overlap correction of Nadeau and Bengio (2003) is seldom applied. For spam filtering, cost-sensitive evaluation is long established (thresholds 0.5, 0.9 and 0.999 in Androutsopoulos et al., 2000). Spambase is the standard benchmark for it. The literature directly comparing the two recipes on a shared, controlled validation budget, with intervals, is thin. I found no such study.
Key papers (10)
- The Foundations of Cost-Sensitive Learning. C. Elkan. Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI), pp. 973-978, 2001.Source of the theoretical cost-optimal threshold c = c_FP/(c_FP + c_FN) for calibrated probabilities; the 'theory' arm of the question.
- Probabilities for SV Machines. J. C. Platt. In A. J. Smola, P. Bartlett, B. Schoelkopf and D. Schuurmans (eds.), Advances in Large-Margin Classifiers, MIT Press, pp. 61-74, 2000.Platt (sigmoid) scaling with smoothed targets (N+ + 1)/(N+ + 2) and 1/(N- + 2): the standard two-parameter post-hoc calibrator.
- Transforming classifier scores into accurate multiclass probability estimates. B. Zadrozny and C. Elkan. Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 694-699, 2002.Isotonic-regression (PAV) calibration of classifier scores: the standard nonparametric calibrator.
- Predicting good probabilities with supervised learning. A. Niculescu-Mizil and R. Caruana. Proceedings of the 22nd International Conference on Machine Learning (ICML), pp. 625-632, 2005.Reference comparison of Platt and isotonic calibration across learners. Boosting has sigmoidal distortion, naive Bayes is pushed to 0/1, logistic regression is near-calibrated. Platt beats isotonic below about 200-1000 calibration cases, measured by squared error, not decision cost.
- PAV and the ROC convex hull. T. Fawcett and A. Niculescu-Mizil. Machine Learning 68(1):97-106, 2007.Shows PAV/isotonic calibration is equivalent to the ROC convex hull, which implies that isotonic calibration plus the theoretical threshold selects an empirical-cost-minimising cut.
- A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss. J. Hernandez-Orallo, P. Flach and C. Ferri. Journal of Machine Learning Research 13:2813-2869, 2012.Framework of threshold choice methods. Theorem 43: for perfectly calibrated models the optimal and score-driven threshold choices coincide at t = c, with PAV as the empirical calibration map.
- Thresholding for Making Classifiers Cost-Sensitive. V. S. Sheng and C. X. Ling. AAAI Conference on Artificial Intelligence (AAAI-06), pp. 476-481, 2006.The 'direct tuning' arm: choose the threshold minimising empirical misclassification cost. It beat the theoretical threshold on uncalibrated C4.5 probabilities in 9 of 12 datasets.
- Cube Root Asymptotics. J. Kim and D. Pollard. The Annals of Statistics 18(1):191-219, 1990.Gives n^{-1/3} limit theory for argmax estimators of indicator criteria, which covers the empirical-cost-minimising threshold and hence the slower convergence of direct tuning.
- Inference for the Generalization Error. C. Nadeau and Y. Bengio. Machine Learning 52(3):239-281, 2003.Corrected resampled t-test: naive intervals over overlapping random splits are too narrow. Used here as a conservative sensitivity check.
- Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration. A. Perez-Lebel, G. Varoquaux, S. Koyejo, M. Doutreligne and M. Le Morvan. Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 258:2395-2403, 2025.Recent decomposition of decision regret into a calibration-induced part, removable by recalibration, and a grouping loss. Frames what calibration can and cannot buy.
Methods in use
- Bayes plug-in rule: threshold calibrated probabilities at the cost proportion c = c_FP/(c_FP + c_FN)
- Platt (sigmoid) scaling fitted by maximum likelihood with smoothed targets on a held-out calibration set
- Isotonic regression (pool-adjacent-violators) calibration, equivalent to the ROC convex hull
- Temperature scaling and beta calibration as one- and three-parameter alternatives
- Empirical threshold search (Thresholding): choose the cut minimising total misclassification cost on held-out or cross-validated predictions, with a tie-breaking rule
- Evaluation by total or normalised expected misclassification cost and cost curves; Brier score, log-loss and ECE for probability quality
- Repeated random train/validation/test splits or cross-validation reported as mean and standard deviation; learning curves over calibration-set size; paired comparisons and, rarely, the corrected resampled t-test
Open gaps
- No controlled head-to-head of 'calibrate, then threshold at c' versus 'tune the threshold' in which both use exactly the same held-out sample at several budgets, with intervals. Sheng and Ling compared search only against uncalibrated probabilities.
- The PAV-ROCCH equivalence implies that isotonic plug-in and empirical tuning pick the same validation cut up to ties and gap placement. The out-of-sample size of that residual has not been measured.
- Platt's small-sample advantage over isotonic regression was established for squared error. Whether it holds for cost-weighted decisions, and how it changes with the cost proportion (tails versus centre), is untested.
- Rates (m^{-1/3} for empirical search versus m^{-1/2} plus a misspecification bias for a sigmoid calibrator) predict a crossover in validation-set size that depends on the learner's miscalibration at the decision level. This has not been measured on real data.
- How robust conclusions from repeated random splits of one benchmark are to the split-overlap correction of Nadeau and Bengio is rarely reported.
Grounded search sources (26) · mode: live:agent:claude-opus-5-5
- The Foundations of Cost-Sensitive Learning (Elkan, IJCAI 2001)
- Probabilities for SV Machines (Platt, in Advances in Large-Margin Classifiers, MIT Press 2000)
- Transforming classifier scores into accurate multiclass probability estimates (Zadrozny and Elkan, KDD 2002)
- Predicting good probabilities with supervised learning (Niculescu-Mizil and Caruana, ICML 2005)
- PAV and the ROC convex hull (Fawcett and Niculescu-Mizil, Machine Learning 2007)
- A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss (Hernandez-Orallo, Flach and Ferri, JMLR 2012)
- Thresholding for Making Classifiers Cost-Sensitive (Sheng and Ling, AAAI 2006)
- Cube Root Asymptotics (Kim and Pollard, Annals of Statistics 1990)
- Inference for the Generalization Error (Nadeau and Bengio, Machine Learning 2003)
- Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration (Perez-Lebel et al., AISTATS 2025)
- On Calibration of Modern Neural Networks (Guo et al., ICML 2017)
- Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers (Kull et al., AISTATS 2017)
- Classifier Calibration with ROC-Regularized Isotonic Regression (Berta, Bach and Jordan, AISTATS 2024)
- A note on Platt's probabilistic outputs for support vector machines (Lin, Lin and Weng, Machine Learning 2007)
- On the appropriateness of Platt scaling in classifier calibration (Boeken, Information Systems 2021)
- Classifier calibration: a survey on how to assess and improve predicted class probabilities (Silva Filho et al., Machine Learning 2023)
- Cost curves: An improved method for visualizing classifier performance (Drummond and Holte, Machine Learning 2006)
- Robust Classification for Imprecise Environments (Provost and Fawcett, Machine Learning 2001)
- Instance weighting versus threshold adjusting for cost-sensitive classification (Zhao, Knowledge and Information Systems 2007)
- Optimal Thresholding of Classifiers to Maximize F1 Measure (Lipton, Elkan and Naryanaswamy, ECML PKDD 2014)
- Cost-Aware Calibration of Classifiers (Yang and Bi, INFORMS Journal on Data Science 2025)
- Contrasting Cost-Agnostic and Cost-Sensitive Losses under Limited Model Capacity via H-consistency (Finocchiaro, Shah and Tambe, arXiv 2025)
- Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration (Ferrer and Ramos, arXiv 2024)
- Verified Uncertainty Calibration (Kumar, Liang and Ma, arXiv 2019)
- An evaluation of Naive Bayesian anti-spam filtering (Androutsopoulos et al., arXiv 2000)
- Spambase [Dataset] (Hopkins, Reeber, Forman and Suermondt, UCI Machine Learning Repository 1999)
2.Ideation
3 candidate ideas generated; selected: “Calibrate-then-threshold versus tune-the-threshold as threshold estimation: isotonic plug-in coincides with tuning, Platt trades variance for bias”.
- Idea 1Selected
Calibrate-then-threshold versus tune-the-threshold as threshold estimation: isotonic plug-in coincides with tuning, Platt trades variance for bias
Hypothesis. Every monotone post-hoc calibrator followed by the theoretical threshold t = c = c_FP/(c_FP + c_FN) is a rule that thresholds the raw score. The two recipes therefore differ only in how they estimate one cut from the same m held-out points. (H1, primary, directional.) At a small validation budget, m = 100, Platt-calibrated plug-in has lower test misclassification cost than empirical threshold tuning. The pre-registered endpoint is the paired difference in excess cost over the test-optimal threshold (Platt minus Tuned), averaged over 3 base learners and 6 cost proportions c in {0.02, 0.1, 0.25, 0.75, 0.9, 0.98}. Its mean over seeds is predicted to be negative, with a 95% t-interval below 0. (H2, secondary.) The Platt advantage shrinks as the budget grows: D(800) - D(100) > 0 with a 95% interval above 0. This is the prediction of a parametric calibrator with a fixed misspecification bias competing with an argmin estimator whose excess cost decays like m^{-2/3}. (H3, secondary; a check of the PAV-ROC-convex-hull equivalence.) Isotonic plug-in and empirical tuning select validation cuts of identical empirical cost, so on the test set they disagree on fewer than 1% of e-mails on average at every m >= 100.
Dataset. Spambase e-mail spam classification benchmark uci_spambase
Test plan, novelty and risk
Test plan. Data: Spambase, all 57 predictors, transformed by log1p; label is_spam. Exact duplicate rows are removed before splitting so that no e-mail appears in both the validation and the test set; the analysed n is reported. Protocol: 50 seeds. Each seed draws a stratified split into 50% train, 20% validation pool and 30% test. Three base learners are fitted on train: L2 logistic regression (C = 1, standardised features), histogram gradient boosting (scikit-learn defaults) and Gaussian naive Bayes. Each yields a log-odds score z. Validation budgets m in {50, 100, 200, 400, 800} are nested prefixes of a seeded permutation of the pool. There are no redraws, and minimum class counts are reported. Recipes, all fitted on the same m points. Platt: maximum-likelihood sigmoid on z with Platt's smoothed targets, predict spam if sigmoid(a z + b) > c. Isotonic: scikit-learn IsotonicRegression on z with clipping, predict spam if g(z) > c. Tuned: the cut minimising c*FP + (1 - c)*FN over all cuts between distinct validation scores, with ties broken at the median co-optimal cut and the threshold placed at the gap midpoint. References: Raw, the model's own sigmoid(z) > c, which uses no validation data. Oracle, the best threshold on the test set itself. Metric: test cost per e-mail C = (c FP + (1 - c) FN)/n_test, and excess cost E = 100 (C/C_oracle - 1). Estimators: the per-seed paired difference D = mean over the 18 learner-by-c cells of E_Platt - E_Tuned. Report its mean over seeds with a t-based 95% interval and a Wilcoxon signed-rank test. A Nadeau-Bengio corrected interval for split overlap is a sensitivity check. Per-learner and per-c differences get 95% intervals; isotonic-versus-tuned test disagreement is reported as a percentage of e-mails. Diagnostics: test AUROC and the fitted Platt slope at m = 800 per learner. All results as mean +/- s.d. over seeds. Refutation. H1 is refuted if the 95% interval for D(100) lies above 0 (tuning better) and unsupported if it covers 0. H2 is refuted unless D(800) - D(100) has an interval above 0. H3 is refuted if mean disagreement exceeds 1% at any m >= 100.
Novelty. Sheng and Ling (2006) showed that threshold search beats the theoretical threshold, but only against uncalibrated probabilities. Niculescu-Mizil and Caruana (2005) compared Platt and isotonic calibration on squared error, not decision cost. This study runs the decision-level head-to-head at a shared, controlled validation budget with paired intervals. It makes explicit, and verifies out of sample, the finite-sample consequence of the PAV-ROC-convex-hull equivalence: isotonic plug-in cannot beat tuning because it is tuning. It attributes the remaining contest, Platt versus tuning, to a stated bias-variance mechanism (cube-root versus root-m rates) and tests its budget-dependent prediction.
Expected outcome. Platt below tuning at m <= 200, with the gap largest at the extreme cost proportions (0.02, 0.98), where few validation e-mails lie near the cut. The advantage narrows by m = 800 and may reverse for naive Bayes, whose sigmoid is likely misspecified at the decision level. Isotonic and tuning agree on more than 99% of test decisions. Raw plug-in is adequate for logistic regression and poor for naive Bayes.
Main risk. A single, dated, well-separated benchmark, so the results may not transfer. Repeated random splits of one corpus overlap, so seed-level intervals are conditional on Spambase; hence the Nadeau-Bengio check. The test-optimal oracle is itself fitted to the test sample, so excess costs are relative rather than population regrets. Results depend on the three learners' particular miscalibration.
- Idea 2
Does the calibration-versus-tuning crossover move with the base rate? A replication on the imbalanced Adult income benchmark
Hypothesis. With 24% positives and validation budgets up to 10^4, the budget at which empirical threshold tuning stops losing to Platt-calibrated plug-in is larger than on balanced data, because the minority-class density near an extreme cut is lower. Prediction: at c = 0.9, Platt beats tuning for every m <= 1000 (paired 95% interval below 0).
Dataset. Adult (Census Income): US income prediction benchmark uci_adult_income
Test plan, novelty and risk
Test plan. Adult (48,842 rows) with fnlwgt excluded as a predictor and rows with missing workclass, occupation or native_country dropped (count reported). One-hot categorical features; logistic regression and gradient boosting. 20 seeds of stratified train/validation-pool/test splits; budgets m in {100, 300, 1000, 3000, 10000}; c in {0.1, 0.25, 0.75, 0.9}. Same recipes and excess-cost metric as idea 0, with paired differences and 95% intervals per cell. Refuted if tuning matches or beats Platt at m <= 1000 for c = 0.9.
Novelty. Tests whether the budget-dependent trade-off depends on class balance, which the literature has not examined for decision cost.
Expected outcome. A crossover at a larger budget than on balanced data, concentrated at the cost proportion that places the cut in the minority-class tail.
Main risk. Encoding choices and the heavy-tailed capital_gain variable affect calibration. The fairness-relevant structure of the data is ignored, and the 1994 labour market dates the conclusions.
- Idea 3
Is data reuse worth more than the choice of recipe? Cross-fitted calibration versus a held-out split for cost-sensitive spam filtering
Hypothesis. At a fixed total sample, fitting the base model on all data and calibrating or tuning from 5-fold out-of-fold scores lowers test cost more than switching between Platt and tuning on a held-out split. Prediction: the cross-fitted-minus-held-out difference is larger in magnitude than the Platt-minus-tuned difference at every c.
Dataset. Spambase e-mail spam classification benchmark uci_spambase
Test plan, novelty and risk
Test plan. Spambase with duplicates removed; logistic regression and gradient boosting; 30 seeds with a 30% test split. Arm A: 50/20 train/validation split with Platt or tuning on the validation part. Arm B: model refitted on the full 70% with Platt or tuning from 5-fold out-of-fold predictions. c in {0.1, 0.25, 0.75, 0.9}; paired differences in test cost with 95% intervals. Refuted if the data-reuse effect is not larger than the recipe effect.
Novelty. Separates the effect of how much data reaches the calibration step from the effect of which recipe is used, a confound in practical comparisons.
Expected outcome. Data reuse dominates at small totals; recipe choice matters mainly at extreme c.
Main risk. Out-of-fold scores come from slightly different models than the refitted one, so cross-fitting is approximate; a single dataset.
Ranking rationale
Idea 0 answers the posed question directly, on the benchmark best suited to it. It separates the case where theory predicts identity (isotonic) from the one where it predicts a trade-off (Platt), and its pre-registered, directional endpoint can come out either way. Idea 1 is a valuable external-validity replication but needs heavier preprocessing, and its answer depends on encoding choices. Idea 2 asks a practical but secondary question about data reuse rather than about the two recipes themselves.
3.Experimentation
Experiment succeeded in 1 round (10.957s).
Data provenance
Spambase e-mail spam classification benchmark
- Source
- UCI Machine Learning Repository (dataset 94), donated by Hewlett-Packard Labs
- Citation
- Hopkins, M., Reeber, E., Forman, G. and Suermondt, J. (1999) Spambase [Dataset]. UCI Machine Learning Repository. DOI: 10.24432/C53G6X. Collected at Hewlett-Packard Labs, Palo Alto, and donated 30 June 1999. No peer-reviewed introductory paper exists for this dataset -- the UCI page refers only to an internal HP technical report -- so the dataset itself is the citable primary source.
- Licence
- Creative Commons Attribution 4.0 International (CC BY 4.0): the UCI dataset page states "This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given."
- File
- https://archive.ics.uci.edu/static/public/94/spambase.zip
- Observations
- 4,601
- Variables (58)
- word_freq_make, word_freq_address, word_freq_all, word_freq_3d, word_freq_our, word_freq_over, word_freq_remove, word_freq_internet, word_freq_order, word_freq_mail, word_freq_receive, word_freq_will, word_freq_people, word_freq_report, word_freq_addresses, word_freq_free, word_freq_business, word_freq_email, word_freq_you, word_freq_credit, word_freq_your, word_freq_font, word_freq_000, word_freq_money, word_freq_hp, word_freq_hpl, word_freq_george, word_freq_650, word_freq_lab, word_freq_labs, word_freq_telnet, word_freq_857, word_freq_data, word_freq_415, word_freq_85, word_freq_technology, word_freq_1999, word_freq_parts, word_freq_pm, word_freq_direct, word_freq_cs, word_freq_meeting, word_freq_original, word_freq_project, word_freq_re, word_freq_edu, word_freq_table, word_freq_conference, char_freq_;, char_freq_(, char_freq_[, char_freq_!, char_freq_$, char_freq_#, capital_run_length_average, capital_run_length_longest, capital_run_length_total, is_spam
- Units
- The 48 word_freq_* columns are the percentage of words in the e-mail matching that token, and the 6 char_freq_* columns the percentage of characters matching that character, so both are on a 0-100 scale. capital_run_length_average is the mean length of uninterrupted runs of capitals, capital_run_length_longest the longest such run, and capital_run_length_total the total number of capital letters. is_spam is 1 for spam and 0 for legitimate mail. Column names are taken from the archive's spambase.names file by the loader.
- Retrieved
- 24 Oct 2026, 04:14
- Checksum
- sha256 813ac1df8effac70…
Real, publicly available data — downloaded and checksum-verified before the analysis ran, never simulated.
Result (RESULT_JSON)
{
"n_loaded": 4601,
"n_duplicates": 391,
"n": 4210,
"n_spam": 1679,
"n_train_pool_test": [
2105,
842,
1263
],
"seeds": 50,
"m": [
50,
100,
200,
400,
800
],
"c": [
0.02,
0.1,
0.25,
0.75,
0.9,
0.98
],
"min_spam_ham_in_val": [
15,
24
],
"learners": {
"LR": {
"auroc": [
0.978,
0.003
],
"slope800_mean_sd_med": [
0.856,
0.099,
0.857
],
"max_tie_frac": 0.002,
"z_q01_q50_q99": [
-25.3,
-2.04,
11.4
],
"raw_excess": [
12.43,
3.653
],
"excess_platt_iso_tuned": [
[
41.204,
81.712,
99.063
],
[
24.867,
50.555,
55.051
],
[
18.114,
29.909,
30.562
],
[
15.501,
20.974,
20.765
],
[
14.143,
16.348,
16.316
]
],
"D_by_m": [
[
-57.859,
-77.542,
-38.176
],
[
-30.183,
-39.811,
-20.556
],
[
-12.449,
-17.617,
-7.281
],
[
-5.263,
-8.177,
-2.35
],
[
-2.173,
-3.988,
-0.358
]
],
"raw_minus_platt_tuned_800": [
[
-1.713,
-3.219,
-0.206
],
[
-3.886,
-5.883,
-1.889
]
]
},
"GB": {
"auroc": [
0.984,
0.003
],
"slope800_mean_sd_med": [
0.649,
0.061,
0.639
],
"max_tie_frac": 0.044,
"z_q01_q50_q99": [
-10.3,
-3.92,
8.9
],
"raw_excess": [
26.161,
9.26
],
"excess_platt_iso_tuned": [
[
39.284,
81.229,
114.128
],
[
27.705,
63.628,
71.645
],
[
19.482,
35.686,
38.286
],
[
15.606,
23.782,
24.214
],
[
13.934,
17.562,
17.769
]
],
"D_by_m": [
[
-74.844,
-97.539,
-52.148
],
[
-43.94,
-58.286,
-29.595
],
[
-18.804,
-25.556,
-12.053
],
[
-8.608,
-12.032,
-5.185
],
[
-3.835,
-6.178,
-1.492
]
],
"raw_minus_platt_tuned_800": [
[
12.227,
9.208,
15.246
],
[
8.392,
5.165,
11.619
]
]
},
"NB": {
"auroc": [
0.955,
0.005
],
"slope800_mean_sd_med": [
0.003,
0.005,
1.38e-7
],
"max_tie_frac": 0.002,
"z_q01_q50_q99": [
-175000000,
15.2,
626
],
"raw_excess": [
462.254,
38.857
],
"excess_platt_iso_tuned": [
[
98.978,
64.176,
73.869
],
[
95.228,
42.57,
44.885
],
[
92.791,
26.34,
27.234
],
[
90.476,
17.307,
17.546
],
[
88.411,
12.508,
12.451
]
],
"D_by_m": [
[
25.109,
11.73,
38.488
],
[
50.343,
39.573,
61.113
],
[
65.557,
55.348,
75.766
],
[
72.93,
63.825,
82.035
],
[
75.96,
66.939,
84.98
]
],
"raw_minus_platt_tuned_800": [
[
373.843,
360.154,
387.532
],
[
449.803,
438.579,
461.027
]
]
}
},
"by_m": {
"50": {
"platt_iso_tuned": [
[
59.822,
15.224
],
[
75.706,
32.585
],
[
95.687,
43.339
]
],
"D": [
-35.865,
-50.37,
-21.359
],
"wilcoxon_p": 0.0000188,
"platt_win_rate": 0.72,
"D_iso": [
-19.981,
-25.614,
-14.348
],
"disagree_pct": [
1.248,
0.802
],
"same_val_partition": 0.968
},
"100": {
"platt_iso_tuned": [
[
49.267,
12.144
],
[
52.251,
21.885
],
[
57.194,
25.003
]
],
"D": [
-7.927,
-16.46,
0.607
],
"wilcoxon_p": 0.121,
"platt_win_rate": 0.56,
"D_iso": [
-4.943,
-6.412,
-3.474
],
"disagree_pct": [
0.603,
0.353
],
"same_val_partition": 0.956
},
"200": {
"platt_iso_tuned": [
[
43.462,
11.132
],
[
30.645,
12.326
],
[
32.027,
13.272
]
],
"D": [
11.435,
6.136,
16.733
],
"wilcoxon_p": 0.00000937,
"platt_win_rate": 0.28,
"D_iso": [
-1.382,
-2.002,
-0.762
],
"disagree_pct": [
0.375,
0.358
],
"same_val_partition": 0.933
},
"400": {
"platt_iso_tuned": [
[
40.528,
10.64
],
[
20.688,
5.343
],
[
20.842,
5.667
]
],
"D": [
19.686,
15.963,
23.41
],
"wilcoxon_p": 2.02e-11,
"platt_win_rate": 0.12,
"D_iso": [
-0.154,
-0.474,
0.166
],
"disagree_pct": [
0.308,
0.243
],
"same_val_partition": 0.899
},
"800": {
"platt_iso_tuned": [
[
38.829,
11.146
],
[
15.472,
3.763
],
[
15.512,
3.657
]
],
"D": [
23.317,
19.886,
26.748
],
"wilcoxon_p": 3e-13,
"platt_win_rate": 0.08,
"D_iso": [
-0.04,
-0.322,
0.242
],
"disagree_pct": [
0.19,
0.152
],
"same_val_partition": 0.896
}
},
"cell_D": {
"100": {
"LR": [
[
-88.069,
-123.309,
-52.828
],
[
-1.123,
-11.835,
9.59
],
[
2.328,
-3.623,
8.28
],
[
11.48,
3.998,
18.963
],
[
-3.675,
-13.596,
6.247
],
[
-102.043,
-137.779,
-66.306
]
],
"GB": [
[
-107.249,
-160.364,
-54.134
],
[
-18.662,
-30.515,
-6.809
],
[
-11.649,
-19.087,
-4.211
],
[
-3.537,
-8.649,
1.576
],
[
-17.616,
-31.533,
-3.7
],
[
-104.929,
-157.731,
-52.128
]
],
"NB": [
[
-62.968,
-92.29,
-33.646
],
[
58.845,
50.354,
67.337
],
[
191.49,
174.739,
208.241
],
[
169.41,
159.974,
178.847
],
[
43.023,
34.597,
51.448
],
[
-97.742,
-134.815,
-60.67
]
]
},
"800": {
"LR": [
[
-10,
-17.191,
-2.809
],
[
1.253,
-2.269,
4.775
],
[
-0.228,
-2.087,
1.631
],
[
1.133,
-1.894,
4.161
],
[
0.992,
-4.05,
6.034
],
[
-6.188,
-11.801,
-0.575
]
],
"GB": [
[
-6.177,
-12.922,
0.568
],
[
-3.521,
-6.117,
-0.925
],
[
-1.977,
-3.878,
-0.076
],
[
-2.5,
-5.186,
0.186
],
[
-4.893,
-8.922,
-0.865
],
[
-3.943,
-14.165,
6.28
]
],
"NB": [
[
7.527,
0.146,
14.909
],
[
60.643,
50.75,
70.536
],
[
187.724,
165.398,
210.051
],
[
159.829,
145.3,
174.359
],
[
56.221,
49.828,
62.613
],
[
-16.187,
-22.767,
-9.607
]
]
}
},
"val_cost_gap_iso_tuned": 6.94e-18,
"primary_D100": [
-7.927,
-16.46,
0.607
],
"nadeau_bengio_ci": [
-48.341,
32.487
],
"h2_D800_minus_D100": [
31.244,
23.156,
39.332
],
"h3_max_disagree_pct": 0.603,
"hypothesis_supported": false,
"revision": {
"nb_factor_on_halfwidth": 4.736,
"halfwidth_per_sd": {
"seed": 0.284,
"nb": 1.346
},
"h1_D100_sd_nb": [
30.027,
-48.341,
32.487
],
"h2_sd_nb": [
28.46,
-7.061,
69.549
],
"learners": {
"LR": {
"excess_sd_platt_iso_tuned": [
[
17.415,
46.891,
58.72
],
[
11.169,
25.335,
27.95
],
[
7.243,
15.099,
15.426
],
[
5.382,
7.801,
8.053
],
[
4.584,
6.329,
6.462
]
],
"D_by_m_sd_nb": [
[
69.259,
-151.076,
35.358
],
[
33.877,
-75.779,
15.413
],
[
18.185,
-36.925,
12.027
],
[
10.252,
-19.061,
8.535
],
[
6.388,
-10.77,
6.424
]
],
"raw_minus_platt_tuned_800_sd_nb": [
[
5.3,
-8.846,
5.421
],
[
7.028,
-13.345,
5.573
]
],
"oracle_cost_per1000_by_c": [
[
6.405,
0.897
],
[
18.352,
1.912
],
[
27.094,
2.987
],
[
25.851,
2.865
],
[
16.884,
2.509
],
[
5.741,
0.861
]
],
"abs_D_per1000_by_m": [
[
-2.516,
-4.421,
-0.611,
-11.538,
6.506
],
[
-1.585,
-2.559,
-0.61,
-6.199,
3.03
],
[
-0.603,
-1.112,
-0.093,
-3.016,
1.81
],
[
-0.285,
-0.635,
0.066,
-1.944,
1.375
],
[
-0.099,
-0.357,
0.158,
-1.32,
1.121
]
],
"platt_prob_excess": [
[
33.253,
13.943
],
[
28.96,
10.22
],
[
26.774,
7.481
],
[
27.332,
6.954
],
[
27.893,
6.908
]
],
"platt_prob_minus_tuned": [
[
-65.81,
-83.824,
-47.797,
-151.119,
19.498
],
[
-26.091,
-34.39,
-17.791,
-65.396,
13.215
],
[
-3.789,
-8.6,
1.022,
-26.573,
18.995
],
[
6.568,
3.474,
9.662,
-8.084,
21.219
],
[
11.577,
9.092,
14.063,
-0.194,
23.348
]
],
"platt_wins_excess": [
[
41.124,
17.318
],
[
24.798,
11.149
],
[
18.056,
7.248
],
[
15.496,
5.388
],
[
14.129,
4.599
]
],
"platt_wins_minus_tuned": [
[
-57.939,
-77.607,
-38.271,
-151.085,
35.207
],
[
-30.252,
-39.881,
-20.624,
-75.853,
15.348
],
[
-12.507,
-17.678,
-7.335,
-36.998,
11.985
],
[
-5.269,
-8.186,
-2.352,
-19.083,
8.546
],
[
-2.187,
-4.007,
-0.368,
-10.805,
6.43
]
],
"platt_minus_platt_prob_wins": [
[
[
7.951,
4.071,
11.831,
-10.425,
26.327
],
[
0.08,
-0.005,
0.165,
-0.322,
0.481
]
],
[
[
-4.093,
-7.113,
-1.072,
-18.397,
10.212
],
[
0.069,
0.026,
0.112,
-0.133,
0.272
]
],
[
[
-8.66,
-11.139,
-6.181,
-20.398,
3.078
],
[
0.058,
0.01,
0.106,
-0.168,
0.284
]
],
[
[
-11.831,
-13.918,
-9.744,
-21.714,
-1.948
],
[
0.006,
-0.011,
0.022,
-0.072,
0.084
]
],
[
[
-13.75,
-15.801,
-11.699,
-23.465,
-4.035
],
[
0.014,
-0.005,
0.033,
-0.076,
0.104
]
]
],
"slope800_median_prob_wins": [
7.26,
0.857
]
},
"GB": {
"excess_sd_platt_iso_tuned": [
[
21.075,
54.133,
72.238
],
[
15.129,
40.437,
46.887
],
[
9.129,
21.046,
22.437
],
[
7.044,
10.21,
10.93
],
[
6.415,
6.749,
6.912
]
],
"D_by_m_sd_nb": [
[
79.858,
-182.326,
32.639
],
[
50.478,
-111.879,
23.999
],
[
23.756,
-50.778,
13.17
],
[
12.045,
-24.82,
7.603
],
[
8.244,
-14.931,
7.261
]
],
"raw_minus_platt_tuned_800_sd_nb": [
[
10.623,
-2.07,
26.524
],
[
11.355,
-6.891,
23.675
]
],
"oracle_cost_per1000_by_c": [
[
5.657,
1.166
],
[
15.259,
2.198
],
[
22.692,
2.969
],
[
20.455,
2.412
],
[
12.423,
1.944
],
[
4.686,
1.209
]
],
"abs_D_per1000_by_m": [
[
-5.364,
-6.985,
-3.744,
-13.038,
2.309
],
[
-3.241,
-4.294,
-2.188,
-8.227,
1.745
],
[
-1.588,
-2.027,
-1.15,
-3.665,
0.488
],
[
-0.834,
-1.078,
-0.59,
-1.991,
0.323
],
[
-0.483,
-0.652,
-0.314,
-1.285,
0.319
]
],
"platt_prob_excess": [
[
54.036,
26.192
],
[
50.63,
18.896
],
[
46.572,
16.316
],
[
48.14,
16.74
],
[
49.601,
17.466
]
],
"platt_prob_minus_tuned": [
[
-60.092,
-83.068,
-37.116,
-168.904,
48.719
],
[
-21.015,
-34.049,
-7.981,
-82.743,
40.712
],
[
8.286,
2.084,
14.489,
-21.087,
37.66
],
[
23.926,
19.601,
28.25,
3.443,
44.408
],
[
31.832,
26.493,
37.171,
6.546,
57.118
]
],
"platt_wins_excess": [
[
39.284,
21.075
],
[
27.705,
15.129
],
[
19.482,
9.129
],
[
15.606,
7.044
],
[
13.934,
6.415
]
],
"platt_wins_minus_tuned": [
[
-74.844,
-97.539,
-52.148,
-182.326,
32.639
],
[
-43.94,
-58.286,
-29.595,
-111.879,
23.999
],
[
-18.804,
-25.556,
-12.053,
-50.778,
13.17
],
[
-8.608,
-12.032,
-5.185,
-24.82,
7.603
],
[
-3.835,
-6.178,
-1.492,
-14.931,
7.261
]
],
"platt_minus_platt_prob_wins": [
[
[
-14.751,
-19.227,
-10.275,
-35.949,
6.447
],
[
0,
0,
0,
0,
0
]
],
[
[
-22.925,
-27.189,
-18.661,
-43.118,
-2.732
],
[
0,
0,
0,
0,
0
]
],
[
[
-27.091,
-30.715,
-23.467,
-44.253,
-9.928
],
[
0,
0,
0,
0,
0
]
],
[
[
-32.534,
-36.206,
-28.862,
-49.925,
-15.142
],
[
0,
0,
0,
0,
0
]
],
[
[
-35.667,
-39.457,
-31.878,
-53.613,
-17.722
],
[
0,
0,
0,
0,
0
]
]
],
"slope800_median_prob_wins": [
6.8,
0.639
]
},
"NB": {
"excess_sd_platt_iso_tuned": [
[
16.782,
34.329,
40.825
],
[
19.42,
26.905,
28.459
],
[
23.858,
18.95,
19.278
],
[
27.203,
9.671,
9.663
],
[
29.49,
5.387,
5.414
]
],
"D_by_m_sd_nb": [
[
47.077,
-38.253,
88.471
],
[
37.898,
-0.664,
101.35
],
[
35.923,
17.207,
113.907
],
[
32.037,
29.811,
116.049
],
[
31.74,
33.24,
118.679
]
],
"raw_minus_platt_tuned_800_sd_nb": [
[
48.167,
309.014,
438.672
],
[
39.495,
396.646,
502.96
]
],
"oracle_cost_per1000_by_c": [
[
8.956,
1.015
],
[
30.318,
2.202
],
[
41.809,
4.013
],
[
34.913,
3.456
],
[
23.701,
2.336
],
[
7.289,
0.501
]
],
"abs_D_per1000_by_m": [
[
23.38,
21.473,
25.286,
14.351,
32.408
],
[
25.426,
23.401,
27.451,
15.837,
35.015
],
[
26.549,
24.132,
28.966,
15.101,
37.997
],
[
26.938,
24.341,
29.535,
14.64,
39.236
],
[
27.156,
24.313,
29.999,
13.692,
40.62
]
],
"platt_prob_excess": [
[
62.315,
9.805
],
[
62.959,
14.872
],
[
60.861,
12.555
],
[
58.638,
10.085
],
[
57.536,
9.429
]
],
"platt_prob_minus_tuned": [
[
-11.554,
-24.138,
1.031,
-71.154,
48.047
],
[
18.074,
9.963,
26.185,
-20.338,
56.486
],
[
33.626,
28.196,
39.057,
7.907,
59.346
],
[
41.092,
37.432,
44.752,
23.759,
58.426
],
[
45.085,
41.676,
48.494,
28.939,
61.231
]
],
"platt_wins_excess": [
[
55.981,
20.306
],
[
43.807,
14.936
],
[
40.119,
16.483
],
[
39.653,
15.898
],
[
37.736,
12.887
]
],
"platt_wins_minus_tuned": [
[
-17.888,
-31.982,
-3.794,
-84.635,
48.86
],
[
-1.078,
-11.035,
8.879,
-48.233,
46.078
],
[
12.885,
4.823,
20.946,
-25.294,
51.063
],
[
22.107,
16.107,
28.106,
-6.307,
50.52
],
[
25.284,
21.036,
29.532,
5.167,
45.402
]
],
"platt_minus_platt_prob_wins": [
[
[
36.663,
32.643,
40.682,
17.628,
55.697
],
[
42.997,
37.664,
48.329,
17.742,
68.251
]
],
[
[
32.269,
26.099,
38.439,
3.048,
61.491
],
[
51.421,
45.737,
57.104,
24.503,
78.338
]
],
[
[
31.931,
24.465,
39.396,
-3.425,
67.286
],
[
52.673,
46.211,
59.134,
22.071,
83.274
]
],
[
[
31.838,
24.043,
39.633,
-5.078,
68.754
],
[
50.824,
43.283,
58.364,
15.114,
86.533
]
],
[
[
30.875,
22.653,
39.097,
-8.064,
69.814
],
[
50.675,
42.604,
58.747,
12.45,
88.901
]
]
],
"slope800_median_prob_wins": [
4.1,
0.013
]
}
},
"by_m": {
"50": {
"D_sd_nb": [
51.04,
-104.561,
32.832
],
"D_iso_sd_nb": [
19.821,
-46.658,
6.697
],
"disagree_nb": [
0.169,
2.327
],
"isomid_excess": [
96.27,
43.048
],
"iso_minus_isomid": [
-20.564,
-26.089,
-15.04,
-46.727,
5.599
],
"isomid_minus_tuned": [
0.583,
-0.313,
1.48,
-3.663,
4.829
],
"abs_D_per1000": [
5.166,
3.864,
6.469,
-1.004,
11.337
]
},
"100": {
"D_sd_nb": [
30.027,
-48.341,
32.487
],
"D_iso_sd_nb": [
5.169,
-11.9,
2.014
],
"disagree_nb": [
0.127,
1.078
],
"isomid_excess": [
57.528,
24.914
],
"iso_minus_isomid": [
-5.277,
-6.633,
-3.92,
-11.701,
1.148
],
"isomid_minus_tuned": [
0.334,
-0.167,
0.835,
-2.037,
2.705
],
"abs_D_per1000": [
6.867,
5.942,
7.792,
2.485,
11.248
]
},
"200": {
"D_sd_nb": [
18.644,
-13.658,
36.528
],
"D_iso_sd_nb": [
2.182,
-4.319,
1.555
],
"disagree_nb": [
-0.107,
0.857
],
"isomid_excess": [
31.985,
13.403
],
"iso_minus_isomid": [
-1.34,
-1.841,
-0.838,
-3.715,
1.036
],
"isomid_minus_tuned": [
-0.043,
-0.446,
0.361,
-1.952,
1.867
],
"abs_D_per1000": [
8.119,
7.217,
9.022,
3.844,
12.394
]
},
"400": {
"D_sd_nb": [
13.102,
2.052,
37.32
],
"D_iso_sd_nb": [
1.127,
-1.671,
1.363
],
"disagree_nb": [
-0.019,
0.635
],
"isomid_excess": [
20.894,
5.563
],
"iso_minus_isomid": [
-0.206,
-0.379,
-0.034,
-1.023,
0.61
],
"isomid_minus_tuned": [
0.053,
-0.238,
0.343,
-1.325,
1.43
],
"abs_D_per1000": [
8.606,
7.7,
9.512,
4.315,
12.898
]
},
"800": {
"D_sd_nb": [
12.073,
7.068,
39.566
],
"D_iso_sd_nb": [
0.992,
-1.375,
1.296
],
"disagree_nb": [
-0.015,
0.395
],
"isomid_excess": [
15.464,
3.725
],
"iso_minus_isomid": [
0.009,
-0.077,
0.095,
-0.398,
0.416
],
"isomid_minus_tuned": [
-0.049,
-0.314,
0.217,
-1.306,
1.209
],
"abs_D_per1000": [
8.858,
7.888,
9.827,
4.266,
13.449
]
}
},
"cell_D_sd_nb": {
"100": {
"LR": [
[
124,
-254.963,
78.826
],
[
37.694,
-51.855,
49.61
],
[
20.942,
-25.857,
30.514
],
[
26.329,
-23.956,
46.917
],
[
34.911,
-50.662,
43.312
],
[
125.744,
-271.285,
67.2
]
],
"GB": [
[
186.897,
-358.797,
144.299
],
[
41.707,
-74.796,
37.472
],
[
26.172,
-46.874,
23.577
],
[
17.989,
-27.748,
20.675
],
[
48.968,
-83.523,
48.291
],
[
185.791,
-354.99,
145.131
]
],
"NB": [
[
103.176,
-201.835,
75.899
],
[
29.879,
18.631,
99.06
],
[
58.941,
112.161,
270.82
],
[
33.203,
124.721,
214.099
],
[
29.648,
3.119,
82.926
],
[
130.447,
-273.314,
77.829
]
]
},
"800": {
"LR": [
[
25.303,
-44.055,
24.056
],
[
12.392,
-15.426,
17.932
],
[
6.542,
-9.033,
8.577
],
[
10.653,
-13.205,
15.471
],
[
17.742,
-22.887,
24.87
],
[
19.752,
-32.772,
20.396
]
],
"GB": [
[
23.734,
-38.122,
25.768
],
[
9.134,
-15.815,
8.772
],
[
6.69,
-10.981,
7.028
],
[
9.452,
-15.222,
10.222
],
[
14.176,
-23.973,
14.187
],
[
35.969,
-52.354,
44.469
]
],
"NB": [
[
25.974,
-27.432,
42.486
],
[
34.809,
13.792,
107.493
],
[
78.559,
81.99,
293.459
],
[
51.124,
91.02,
228.638
],
[
22.493,
25.947,
86.495
],
[
23.153,
-47.349,
14.976
]
]
}
},
"env": {
"python": "3.12.13",
"numpy": "2.5.2",
"scipy": "1.18.1",
"sklearn": "1.9.0",
"pandas": "3.0.5",
"platform": "macOS-27.0.1-arm64-arm-64bit",
"cpu": "Apple M4",
"cpu_count": 10
}
},
"runtime_s": 10.1
}Experiment script · experiment_v1.py
import ctypes, json, os, platform, time
import numpy as np, pandas as pd, scipy, sklearn
from scipy import stats
from scipy.special import expit
from sklearn.model_selection import train_test_split as tts
from sklearn.linear_model import LogisticRegression as LogReg
from sklearn.ensemble import HistGradientBoostingClassifier as HGB
from sklearn.naive_bayes import GaussianNB
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.isotonic import IsotonicRegression as Iso
from sklearn.metrics import roc_auc_score
from dataset import load
# Revision (round 1): every quantity of the executed v1 script is computed exactly as before (same seeds, same
# order of random draws). Additions use no new random numbers: per-learner and per-cell s.d. over seeds,
# overlap-corrected (Nadeau-Bengio) intervals for every contrast, oracle costs C*, absolute cost differences,
# isotonic with midpoint placement (cut-placement ablation), Platt on two bounded inputs, software and CPU.
T0 = time.time(); df = load()
print("loaded", df.shape, "missing", df.isna().sum().sum(), "non-numeric", (df.dtypes == object).sum())
n0 = len(df); df = df.drop_duplicates().reset_index(drop=True) # no duplicate e-mail across splits
X = np.log1p(df.drop(columns="is_spam").to_numpy(float)); y = df["is_spam"].to_numpy(int)
n = len(y); print("analysed", n, "after dropping", n0 - n, "duplicates; spam", y.sum())
R, MS, CS = 50, [50, 100, 200, 400, 800], np.array([.02, .1, .25, .75, .9, .98]); LG = np.log(CS / (1 - CS))
def score(f, A): # log-odds score z
if isinstance(f, GaussianNB): j = f.predict_joint_log_proba(A); return j[:, 1] - j[:, 0]
return f.decision_function(A)
def platt(z, t): # unpenalised MLE sigmoid(a z+b), Platt targets
p = np.where(t == 1, (t.sum() + 1) / (t.sum() + 2), 1 / (len(t) - t.sum() + 2)); s = stats.iqr(z) + 1e-12
f = LogReg(C=1e12, max_iter=5000).fit(np.r_[z, z][:, None] / s, np.r_[t * 0 + 1, t * 0], sample_weight=np.r_[p, 1 - p])
return f.coef_[0, 0] / s, f.intercept_[0]
def cuts(z, t): # cost c*FP+(1-c)*FN of top-k cuts, k=0..m
o = np.argsort(-z, kind="mergesort"); s, u = z[o], t[o]; tp, fp = np.r_[0, u.cumsum()], np.r_[0, (1 - u).cumsum()]
ok = np.r_[True, s[:-1] > s[1:], True] # cuts only between distinct scores
return s, np.where(ok, CS[:, None] * fp + (1 - CS[:, None]) * (tp[-1] - tp), np.inf)
def place(s, k): # threshold flagging the top k of descending scores s: gap midpoint; max score if k=0; -inf if k=m
return s[0] if k == 0 else -np.inf if k == len(s) else (s[k - 1] + s[k]) / 2
def tune(z, t): # ERM cut per c; ties: median; gap midpoint
s, C = cuts(z, t); th = []
for w in C:
k = np.flatnonzero(w <= w.min() + 1e-9); k = k[len(k) // 2]
th.append(place(s, k))
return np.array(th)
def fences(z): # Tukey far-out fences of the validation scores: the fixed range for winsorised log-odds
q1, q3 = np.percentile(z, [25, 75]); return q1 - 3 * (q3 - q1), q3 + 3 * (q3 - q1)
def cpu(): # CPU model, read without a subprocess (macOS sysctl, else /proc/cpuinfo)
try:
b, k = ctypes.create_string_buffer(256), ctypes.c_size_t(256)
if ctypes.CDLL(None).sysctlbyname(b"machdep.cpu.brand_string", b, ctypes.byref(k), None, 0) == 0: return b.value.decode()
except Exception: pass
try: return next(l.split(":")[1].strip() for l in open("/proc/cpuinfo") if l.startswith("model name"))
except Exception: return platform.processor() or platform.machine()
cost = lambda d, t: (CS * (d & (t[:, None] == 0)).sum(0) + (1 - CS) * (~d & (t[:, None] == 1)).sum(0)) / len(t)
E = np.zeros((R, 3, 5, 6, 3)); ER = np.zeros((R, 3, 6)); DIS = np.zeros((R, 3, 5, 6)); SP = DIS.copy()
AU, SL, TI = np.zeros((3, R, 3)); Q = np.zeros((R, 3, 3)); vg, mn = 0.0, [n, n]
E2 = np.zeros((R, 3, 5, 6, 3)); AB = np.zeros((R, 3, 5, 6)); ORC = np.zeros((R, 3, 6)); SL2 = np.zeros((R, 3, 2)) # new
for r in range(R):
Xt, Xh, yt, yh = tts(X, y, train_size=.5, stratify=y, random_state=r)
Xp, Xe, yp, ye = tts(Xh, yh, train_size=.4, stratify=yh, random_state=r)
pm = np.random.default_rng(r).permutation(len(yp)); yp = yp[pm]
for l, f in enumerate([make_pipeline(StandardScaler(), LogReg(max_iter=5000)), HGB(random_state=r), GaussianNB()]):
f.fit(Xt, yt); zp, ze = score(f, Xp)[pm], score(f, Xe)
orc = np.maximum(cuts(ze, ye)[1].min(1) / len(ye), 1e-12); ORC[r, l] = orc # oracle: best test-set threshold
ex = lambda d: 100 * (cost(d, ye) / orc - 1) # excess cost, %
AU[r, l], TI[r, l], Q[r, l] = roc_auc_score(ye, ze), 1 - len(np.unique(ze)) / len(ze), np.percentile(ze, [1, 50, 99])
ER[r, l] = ex(ze[:, None] > LG)
for i, m in enumerate(MS):
zv, tv = zp[:m], yp[:m]; mn = [min(mn[0], tv.sum()), min(mn[1], m - tv.sum())]
a, b = platt(zv, tv); g = Iso(out_of_bounds="clip").fit(zv, tv); th = tune(zv, tv)
di, dt, vi, vt = g.predict(ze)[:, None] > CS, ze[:, None] > th, g.predict(zv)[:, None] > CS, zv[:, None] > th
vg = max(vg, np.abs(cost(vi, tv) - cost(vt, tv)).max()); SP[r, l, i] = (vi == vt).all(0) # Prop. 1 check
dp = (a * ze + b)[:, None] > LG
E[r, l, i] = np.c_[ex(dp), ex(di), ex(dt)]; DIS[r, l, i] = 100 * (di != dt).mean(0)
if m == 800: SL[r, l] = a
# new: isotonic's validation partition with the tuned rule's (midpoint) placement
s = np.sort(zv)[::-1]; tm = np.array([place(s, k) for k in vi.sum(0)])
a1, b1 = platt(expit(zv), tv) # Platt on the posterior probability sigma(z) (bounded in [0, 1])
lo, hi = fences(zv); a2, b2 = platt(np.clip(zv, lo, hi), tv) # Platt on winsorised log-odds
E2[r, l, i] = np.c_[ex(ze[:, None] > tm), ex((a1 * expit(ze) + b1)[:, None] > LG), ex((a2 * np.clip(ze, lo, hi) + b2)[:, None] > LG)]
AB[r, l, i] = 1000 * (cost(dp, ye) - cost(dt, ye)) # absolute Platt - Tuned cost per 1,000 e-mails
if m == 800: SL2[r, l] = a1, a2
tq = stats.t.ppf(.975, R - 1); r3 = lambda v: round(float(v), 3); g3 = lambda v: float(f"{v:.3g}")
def ci(d, se=None): # mean, 95% t-CI over seeds
se = d.std(ddof=1) / R ** .5 if se is None else se; return [r3(d.mean()), r3(d.mean() - tq * se), r3(d.mean() + tq * se)]
sd = lambda d: [r3(d.mean()), r3(d.std(ddof=1))]
P = E.mean((1, 3)); D, DI = P[..., 0] - P[..., 2], P[..., 1] - P[..., 2] # pooled over learners and c
EM, DC, RW, LN = E.mean(3), E[..., 0] - E[..., 2], ER.mean(2), ["LR", "GB", "NB"]
nb = D[:, 1].std(ddof=1) * (1 / R + len(ye) / (len(yt) + len(yp))) ** .5 # Nadeau-Bengio s.e.
prim, h2, h3 = ci(D[:, 1]), ci(D[:, 4] - D[:, 1]), DIS.mean((0, 1, 3))[1:].max()
NBF = (1 / R + len(ye) / (len(yt) + len(yp))) ** .5 # Nadeau-Bengio s.e. per unit seed s.d., applied to every contrast
cin = lambda d: ci(d, d.std(ddof=1) * NBF)[1:] # overlap-corrected 95% interval
sci = lambda d: [r3(d.std(ddof=1))] + cin(d) # s.d. over seeds and overlap-corrected interval
E2M, PT = E2.mean(3), E.mean(3)[..., 2] # per learner, mean over c; PT = tuned
env = {"python": platform.python_version(), "numpy": np.__version__, "scipy": scipy.__version__, "sklearn": sklearn.__version__,
"pandas": pd.__version__, "platform": platform.platform(), "cpu": cpu(), "cpu_count": os.cpu_count()}
res = {"n_loaded": n0, "n_duplicates": n0 - n, "n": n, "n_spam": int(y.sum()), "n_train_pool_test": [len(yt), len(yp), len(ye)],
"seeds": R, "m": MS, "c": CS.tolist(), "min_spam_ham_in_val": list(map(int, mn)),
"learners": {L: {"auroc": sd(AU[:, l]), "slope800_mean_sd_med": sd(SL[:, l]) + [g3(np.median(SL[:, l]))], "max_tie_frac": r3(TI[:, l].max()),
"z_q01_q50_q99": [g3(v) for v in np.median(Q[:, l], 0)], "raw_excess": sd(RW[:, l]),
"excess_platt_iso_tuned": [[r3(v) for v in EM[:, l, i].mean(0)] for i in range(5)], "D_by_m": [ci(DC[:, l, i].mean(1)) for i in range(5)],
"raw_minus_platt_tuned_800": [ci(RW[:, l] - EM[:, l, 4, k]) for k in (0, 2)]} for l, L in enumerate(LN)},
"by_m": {m: {"platt_iso_tuned": [sd(P[:, i, k]) for k in range(3)], "D": ci(D[:, i]), "wilcoxon_p": g3(stats.wilcoxon(D[:, i]).pvalue),
"platt_win_rate": r3((D[:, i] < 0).mean()), "D_iso": ci(DI[:, i]), "disagree_pct": sd(DIS[:, :, i].mean((1, 2))),
"same_val_partition": r3(SP[:, :, i].mean())} for i, m in enumerate(MS)},
"cell_D": {m: {L: [ci(DC[:, l, i, j]) for j in range(6)] for l, L in enumerate(LN)} for i, m in ((1, 100), (4, 800))},
"val_cost_gap_iso_tuned": g3(vg), "primary_D100": prim, "nadeau_bengio_ci": ci(D[:, 1], nb)[1:], "h2_D800_minus_D100": h2,
"h3_max_disagree_pct": r3(h3), "hypothesis_supported": bool(prim[2] < 0)}
res["revision"] = {
"nb_factor_on_halfwidth": r3((1 + R * len(ye) / (len(yt) + len(yp))) ** .5), "halfwidth_per_sd": {"seed": r3(tq / R ** .5), "nb": r3(tq * NBF)},
"h1_D100_sd_nb": sci(D[:, 1]), "h2_sd_nb": sci(D[:, 4] - D[:, 1]),
"learners": {L: {
"excess_sd_platt_iso_tuned": [[r3(EM[:, l, i, k].std(ddof=1)) for k in range(3)] for i in range(5)],
"D_by_m_sd_nb": [sci(DC[:, l, i].mean(1)) for i in range(5)],
"raw_minus_platt_tuned_800_sd_nb": [sci(RW[:, l] - EM[:, l, 4, k]) for k in (0, 2)],
"oracle_cost_per1000_by_c": [sd(1000 * ORC[:, l, j]) for j in range(6)],
"abs_D_per1000_by_m": [ci(AB[:, l, i].mean(1)) + cin(AB[:, l, i].mean(1)) for i in range(5)],
"platt_prob_excess": [sd(E2M[:, l, i, 1]) for i in range(5)], "platt_prob_minus_tuned": [ci(E2M[:, l, i, 1] - PT[:, l, i]) + cin(E2M[:, l, i, 1] - PT[:, l, i]) for i in range(5)],
"platt_wins_excess": [sd(E2M[:, l, i, 2]) for i in range(5)], "platt_wins_minus_tuned": [ci(E2M[:, l, i, 2] - PT[:, l, i]) + cin(E2M[:, l, i, 2] - PT[:, l, i]) for i in range(5)],
"platt_minus_platt_prob_wins": [[ci(EM[:, l, i, 0] - E2M[:, l, i, k]) + cin(EM[:, l, i, 0] - E2M[:, l, i, k]) for k in (1, 2)] for i in range(5)],
"slope800_median_prob_wins": [g3(np.median(SL2[:, l, k])) for k in range(2)]} for l, L in enumerate(LN)},
"by_m": {m: {"D_sd_nb": sci(D[:, i]), "D_iso_sd_nb": sci(DI[:, i]), "disagree_nb": cin(DIS[:, :, i].mean((1, 2))),
"isomid_excess": sd(E2[:, :, i, :, 0].mean((1, 2))),
"iso_minus_isomid": ci(P[:, i, 1] - E2[:, :, i, :, 0].mean((1, 2))) + cin(P[:, i, 1] - E2[:, :, i, :, 0].mean((1, 2))),
"isomid_minus_tuned": ci(E2[:, :, i, :, 0].mean((1, 2)) - P[:, i, 2]) + cin(E2[:, :, i, :, 0].mean((1, 2)) - P[:, i, 2]),
"abs_D_per1000": ci(AB[:, :, i].mean((1, 2))) + cin(AB[:, :, i].mean((1, 2)))} for i, m in enumerate(MS)},
"cell_D_sd_nb": {m: {L: [sci(DC[:, l, i, j]) for j in range(6)] for l, L in enumerate(LN)} for i, m in ((1, 100), (4, 800))},
"env": env}
res["runtime_s"] = round(time.time() - T0, 1)
print("RESULT_JSON: " + json.dumps(res))
4.Analysis
Analysis complete: experiment succeeded; inconclusive; 67 key numbers.
Results and analysis
The revised run succeeded in one round (exit 0, 10.1 s on an Apple M4). It reproduces every number of the first run exactly. It adds seed s.d.s, overlap-corrected (Nadeau-Bengio) intervals for every contrast, oracle costs C*, an absolute-cost endpoint, a cut-placement ablation and Platt fits on two bounded inputs. The sample is unchanged: 4,210 deduplicated e-mails (1,679 spam), 50 seeds of 2,105/842/1,263 splits. H1 is unsupported by its pre-specified rule: D(100) = -7.927 [-16.46, 0.607], corrected [-48.341, 32.487]. Because the interval covers 0, H1 is neither supported nor refuted, which this schema records as inconclusive. H2 meets its rule at the seed level, 31.244 [23.156, 39.332], but not under the correction, [-7.061, 69.549]. H3 holds by its rule, with at most 0.603% disagreement for m >= 100, but the corrected upper limit at m = 100 is 1.078%. D weights each cell by 1/C*, and C* runs from 4.686 to 41.809 per 1,000 e-mails. Unweighted, the pooled absolute difference, which was not pre-specified, favours tuning: 6.867 [5.942, 7.792] per 1,000 e-mails at m = 100, corrected [2.485, 11.248], and robustly from m = 100 on (at m = 50 the corrected interval, [-1.004, 11.337], covers 0). Naive Bayes drives it entirely (25.426, against -1.585 for logistic regression and -3.241 for gradient boosting, which favour Platt at the seed level). At the seed level Platt beats tuning at every budget for logistic regression and gradient boosting, but none of their Platt-versus-tuning contrasts survives the correction; naive Bayes's Platt deficit survives from m = 200. Winsorising naive Bayes's log-odds at Tukey's far-out fences cuts its Platt excess by 51.421 points at m = 100 (corrected [24.503, 78.338]), to 43.807 ± 14.936%, not distinguishable from tuning at this budget (-1.078 [-11.035, 8.879]; corrected [-48.233, 46.078]). From m = 200 it plateaus at 37.736-40.119% and tuning is cheaper, robustly only at m = 800. At the seed level, isotonic's small-budget edge over tuning is entirely cut placement: -20.564 of -19.981 at m = 50, with partition choice contributing 0.583 [-0.313, 1.48]. No isotonic contrast survives the correction.
Key numbers
| Metric | Value | Interpretation |
|---|---|---|
| e-mails loaded / duplicates removed / analysed / spam | 4601 / 391 / 4210 / 1679 | Analysed sample after removing exact duplicate rows that could leak between validation and test |
| train / pool / test per seed; seeds | 2105 / 842 / 1263; 50 seeds | Stratified 50/20/30 split redrawn for each seed; budgets are nested prefixes of the permuted pool |
| minimum spam / ham in any validation prefix | 15 / 24 | Both classes always present; no redraws |
| runtime and environment | 10.1 s; Python 3.12.13, NumPy 2.5.2, SciPy 1.18.1, scikit-learn 1.9.0, pandas 3.0.5; Apple M4 (10 cores), macOS-27.0.1-arm64-arm-64bit | Whole revised analysis; recorded by the run itself. Learner defaults depend on the scikit-learn version |
| overlap-correction factor on every seed-level half-width; half-width per s.d. (seed-level / corrected) | 4.736; 0.284 / 1.346 | Nadeau-Bengio correction sqrt(1/50 + n_test/(n_train + n_pool)) applied uniformly; a contrast is robust if its corrected interval excludes 0 |
| PRIMARY H1: D(100), pooled relative excess-cost difference Platt - Tuned | -7.927 ± 30.027 (corrected CI -48.341, 32.487); seed-level CI -16.46, 0.607 | Unsupported by the pre-specified rule: the interval covers 0 (not refuted, which needs an interval above 0) |
| H1 further evidence: Wilcoxon p; Platt win rate at m = 100 | p = 0.121; 0.56 | No reliable pooled advantage for Platt |
| H2: D(800) - D(100) | 31.244 ± 28.46 (corrected CI -7.061, 69.549); seed-level CI 23.156, 39.332 | Meets its pre-specified seed-level rule but is not robust to the overlap correction |
| H3: maximum mean isotonic-vs-tuned test disagreement for m >= 100; corrected upper limit at m = 100 | 0.603%; corrected CI 0.127, 1.078 | Holds by its pre-specified rule (point estimate below 1%), but the corrected upper limit at m = 100 exceeds 1%; below 1% from m = 200 |
| largest validation-cost gap, isotonic vs tuned | 6.94e-18 | Floating-point rounding: Proposition 1 holds exactly |
| m = 50: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.) | 59.822 ± 15.224 / 75.706 ± 32.585 / 95.687 ± 43.339 | Excess over the test-optimal threshold, averaged over 3 learners x 6 costs |
| m = 50: pooled D = Platt - Tuned (relative, 1/C*-weighted) | -35.865 ± 51.04 (corrected CI -104.561, 32.832); seed-level CI -50.37, -21.359; Wilcoxon p = 1.88e-05 | Negative favours Platt |
| m = 50: pooled absolute cost difference Platt - Tuned per 1,000 e-mails | 5.166 [95% CI 3.864, 6.469; corrected -1.004, 11.337] | Positive favours tuning; unweighted by C*; not pre-specified; not robust at this budget (corrected interval covers 0) |
| m = 50: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned) | -19.981 [95% CI -25.614, -14.348]; -20.564 [95% CI -26.089, -15.04; corrected -46.727, 5.599]; 0.583 [95% CI -0.313, 1.48; corrected -3.663, 4.829] | Cut-placement ablation: isotonic's partition with midpoint placement |
| m = 100: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.) | 49.267 ± 12.144 / 52.251 ± 21.885 / 57.194 ± 25.003 | Excess over the test-optimal threshold, averaged over 3 learners x 6 costs |
| m = 100: pooled D = Platt - Tuned (relative, 1/C*-weighted) | -7.927 ± 30.027 (corrected CI -48.341, 32.487); seed-level CI -16.46, 0.607; Wilcoxon p = 0.121 | Negative favours Platt |
| m = 100: pooled absolute cost difference Platt - Tuned per 1,000 e-mails | 6.867 [95% CI 5.942, 7.792; corrected 2.485, 11.248] | Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes (logistic regression and gradient boosting favour Platt at the seed level) |
| m = 100: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned) | -4.943 [95% CI -6.412, -3.474]; -5.277 [95% CI -6.633, -3.92; corrected -11.701, 1.148]; 0.334 [95% CI -0.167, 0.835; corrected -2.037, 2.705] | Cut-placement ablation: isotonic's partition with midpoint placement |
| m = 200: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.) | 43.462 ± 11.132 / 30.645 ± 12.326 / 32.027 ± 13.272 | Excess over the test-optimal threshold, averaged over 3 learners x 6 costs |
| m = 200: pooled D = Platt - Tuned (relative, 1/C*-weighted) | 11.435 ± 18.644 (corrected CI -13.658, 36.528); seed-level CI 6.136, 16.733; Wilcoxon p = 9.37e-06 | Negative favours Platt |
| m = 200: pooled absolute cost difference Platt - Tuned per 1,000 e-mails | 8.119 [95% CI 7.217, 9.022; corrected 3.844, 12.394] | Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes |
| m = 200: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned) | -1.382 [95% CI -2.002, -0.762]; -1.34 [95% CI -1.841, -0.838; corrected -3.715, 1.036]; -0.043 [95% CI -0.446, 0.361; corrected -1.952, 1.867] | Cut-placement ablation: isotonic's partition with midpoint placement |
| m = 400: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.) | 40.528 ± 10.64 / 20.688 ± 5.343 / 20.842 ± 5.667 | Excess over the test-optimal threshold, averaged over 3 learners x 6 costs |
| m = 400: pooled D = Platt - Tuned (relative, 1/C*-weighted) | 19.686 ± 13.102 (corrected CI 2.052, 37.32); seed-level CI 15.963, 23.41; Wilcoxon p = 2.02e-11 | Negative favours Platt |
| m = 400: pooled absolute cost difference Platt - Tuned per 1,000 e-mails | 8.606 [95% CI 7.7, 9.512; corrected 4.315, 12.898] | Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes |
| m = 400: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned) | -0.154 [95% CI -0.474, 0.166]; -0.206 [95% CI -0.379, -0.034; corrected -1.023, 0.61]; 0.053 [95% CI -0.238, 0.343; corrected -1.325, 1.43] | Cut-placement ablation: isotonic's partition with midpoint placement |
| m = 800: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.) | 38.829 ± 11.146 / 15.472 ± 3.763 / 15.512 ± 3.657 | Excess over the test-optimal threshold, averaged over 3 learners x 6 costs |
| m = 800: pooled D = Platt - Tuned (relative, 1/C*-weighted) | 23.317 ± 12.073 (corrected CI 7.068, 39.566); seed-level CI 19.886, 26.748; Wilcoxon p = 3e-13 | Negative favours Platt |
| m = 800: pooled absolute cost difference Platt - Tuned per 1,000 e-mails | 8.858 [95% CI 7.888, 9.827; corrected 4.266, 13.449] | Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes |
| m = 800: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned) | -0.04 [95% CI -0.322, 0.242]; 0.009 [95% CI -0.077, 0.095; corrected -0.398, 0.416]; -0.049 [95% CI -0.314, 0.217; corrected -1.306, 1.209] | Cut-placement ablation: isotonic's partition with midpoint placement |
| logistic regression: excess (%) Platt / isotonic / tuned by m (mean ± s.d.) | m=50: 41.204 ± 17.415 / 81.712 ± 46.891 / 99.063 ± 58.72; m=100: 24.867 ± 11.169 / 50.555 ± 25.335 / 55.051 ± 27.95; m=200: 18.114 ± 7.243 / 29.909 ± 15.099 / 30.562 ± 15.426; m=400: 15.501 ± 5.382 / 20.974 ± 7.801 / 20.765 ± 8.053; m=800: 14.143 ± 4.584 / 16.348 ± 6.329 / 16.316 ± 6.462 | Learning curves per learner, means and s.d. over 50 seeds |
| logistic regression: D = Platt - Tuned by m (mean ± s.d.; corrected CI) | m=50: -57.859 ± 69.259 (corrected CI -151.076, 35.358); m=100: -30.183 ± 33.877 (corrected CI -75.779, 15.413); m=200: -12.449 ± 18.185 (corrected CI -36.925, 12.027); m=400: -5.263 ± 10.252 (corrected CI -19.061, 8.535); m=800: -2.173 ± 6.388 (corrected CI -10.77, 6.424) | Platt ahead at every budget at the seed level; no corrected interval excludes 0 |
| logistic regression: raw minus Platt and raw minus Tuned at m = 800 (mean ± s.d.; corrected CI) | -1.713 ± 5.3 (corrected CI -8.846, 5.421); -3.886 ± 7.028 (corrected CI -13.345, 5.573) | Negative favours raw (no validation data); robust only for NB |
| logistic regression: oracle cost C* per 1,000 e-mails by c = 0.02, 0.1, 0.25, 0.75, 0.9, 0.98 (mean ± s.d.) | 6.405 ± 0.897; 18.352 ± 1.912; 27.094 ± 2.987; 25.851 ± 2.865; 16.884 ± 2.509; 5.741 ± 0.861 | The relative endpoint weights each cell by 1/C* |
| logistic regression: absolute Platt - Tuned per 1,000 e-mails at m = 100 | -1.585 [95% CI -2.559, -0.61; corrected -6.199, 3.03] | Positive favours tuning; not pre-specified; favours Platt at the seed level, not robust |
| logistic regression: Platt on posterior probability / on winsorised log-odds, excess (%) by m (mean ± s.d.) | m=50: 33.253 ± 13.943 / 41.124 ± 17.318; m=100: 28.96 ± 10.22 / 24.798 ± 11.149; m=200: 26.774 ± 7.481 / 18.056 ± 7.248; m=400: 27.332 ± 6.954 / 15.496 ± 5.388; m=800: 27.893 ± 6.908 / 14.129 ± 4.599 | Bounded-input Platt fits (input-scale ablation) |
| logistic regression: Platt(log-odds) minus Platt(winsorised) by m | m=50: 0.08 [95% CI -0.005, 0.165; corrected -0.322, 0.481]; m=100: 0.069 [95% CI 0.026, 0.112; corrected -0.133, 0.272]; m=200: 0.058 [95% CI 0.01, 0.106; corrected -0.168, 0.284]; m=400: 0.006 [95% CI -0.011, 0.022; corrected -0.072, 0.084]; m=800: 0.014 [95% CI -0.005, 0.033; corrected -0.076, 0.104] | Positive: winsorising lowers Platt's excess; negligible control |
| logistic regression: Platt(winsorised) minus Tuned by m | m=50: -57.939 [95% CI -77.607, -38.271; corrected -151.085, 35.207]; m=100: -30.252 [95% CI -39.881, -20.624; corrected -75.853, 15.348]; m=200: -12.507 [95% CI -17.678, -7.335; corrected -36.998, 11.985]; m=400: -5.269 [95% CI -8.186, -2.352; corrected -19.083, 8.546]; m=800: -2.187 [95% CI -4.007, -0.368; corrected -10.805, 6.43] | Negative favours winsorised Platt |
| logistic regression: test AUROC; Platt slope at m = 800 (mean ± s.d.; median); median slope on probability / winsorised input | 0.978 ± 0.003; 0.856 ± 0.099 (median 0.857); 7.26 / 0.857 | Slope below 1 means over-confident log-odds; NB's log-odds slope collapses |
| logistic regression: test log-odds 1st / 50th / 99th percentile; raw plug-in excess (%) | -25.3 / -2.04 / 11.4; 12.43 ± 3.653 | Scale and tails of the thresholded score; raw uses no validation data |
| gradient boosting: excess (%) Platt / isotonic / tuned by m (mean ± s.d.) | m=50: 39.284 ± 21.075 / 81.229 ± 54.133 / 114.128 ± 72.238; m=100: 27.705 ± 15.129 / 63.628 ± 40.437 / 71.645 ± 46.887; m=200: 19.482 ± 9.129 / 35.686 ± 21.046 / 38.286 ± 22.437; m=400: 15.606 ± 7.044 / 23.782 ± 10.21 / 24.214 ± 10.93; m=800: 13.934 ± 6.415 / 17.562 ± 6.749 / 17.769 ± 6.912 | Learning curves per learner, means and s.d. over 50 seeds |
| gradient boosting: D = Platt - Tuned by m (mean ± s.d.; corrected CI) | m=50: -74.844 ± 79.858 (corrected CI -182.326, 32.639); m=100: -43.94 ± 50.478 (corrected CI -111.879, 23.999); m=200: -18.804 ± 23.756 (corrected CI -50.778, 13.17); m=400: -8.608 ± 12.045 (corrected CI -24.82, 7.603); m=800: -3.835 ± 8.244 (corrected CI -14.931, 7.261) | Platt ahead at every budget at the seed level; no corrected interval excludes 0 |
| gradient boosting: raw minus Platt and raw minus Tuned at m = 800 (mean ± s.d.; corrected CI) | 12.227 ± 10.623 (corrected CI -2.07, 26.524); 8.392 ± 11.355 (corrected CI -6.891, 23.675) | Negative favours raw (no validation data); robust only for NB |
| gradient boosting: oracle cost C* per 1,000 e-mails by c = 0.02, 0.1, 0.25, 0.75, 0.9, 0.98 (mean ± s.d.) | 5.657 ± 1.166; 15.259 ± 2.198; 22.692 ± 2.969; 20.455 ± 2.412; 12.423 ± 1.944; 4.686 ± 1.209 | The relative endpoint weights each cell by 1/C* |
| gradient boosting: absolute Platt - Tuned per 1,000 e-mails at m = 100 | -3.241 [95% CI -4.294, -2.188; corrected -8.227, 1.745] | Positive favours tuning; not pre-specified; favours Platt at the seed level, not robust |
| gradient boosting: Platt on posterior probability / on winsorised log-odds, excess (%) by m (mean ± s.d.) | m=50: 54.036 ± 26.192 / 39.284 ± 21.075; m=100: 50.63 ± 18.896 / 27.705 ± 15.129; m=200: 46.572 ± 16.316 / 19.482 ± 9.129; m=400: 48.14 ± 16.74 / 15.606 ± 7.044; m=800: 49.601 ± 17.466 / 13.934 ± 6.415 | Bounded-input Platt fits (input-scale ablation) |
| gradient boosting: Platt(log-odds) minus Platt(winsorised) by m | m=50: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=100: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=200: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=400: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=800: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0] | Positive: winsorising lowers Platt's excess; negligible control |
| gradient boosting: Platt(winsorised) minus Tuned by m | m=50: -74.844 [95% CI -97.539, -52.148; corrected -182.326, 32.639]; m=100: -43.94 [95% CI -58.286, -29.595; corrected -111.879, 23.999]; m=200: -18.804 [95% CI -25.556, -12.053; corrected -50.778, 13.17]; m=400: -8.608 [95% CI -12.032, -5.185; corrected -24.82, 7.603]; m=800: -3.835 [95% CI -6.178, -1.492; corrected -14.931, 7.261] | Negative favours winsorised Platt |
| gradient boosting: test AUROC; Platt slope at m = 800 (mean ± s.d.; median); median slope on probability / winsorised input | 0.984 ± 0.003; 0.649 ± 0.061 (median 0.639); 6.8 / 0.639 | Slope below 1 means over-confident log-odds; NB's log-odds slope collapses |
| gradient boosting: test log-odds 1st / 50th / 99th percentile; raw plug-in excess (%) | -10.3 / -3.92 / 8.9; 26.161 ± 9.26 | Scale and tails of the thresholded score; raw uses no validation data |
| Gaussian naive Bayes: excess (%) Platt / isotonic / tuned by m (mean ± s.d.) | m=50: 98.978 ± 16.782 / 64.176 ± 34.329 / 73.869 ± 40.825; m=100: 95.228 ± 19.42 / 42.57 ± 26.905 / 44.885 ± 28.459; m=200: 92.791 ± 23.858 / 26.34 ± 18.95 / 27.234 ± 19.278; m=400: 90.476 ± 27.203 / 17.307 ± 9.671 / 17.546 ± 9.663; m=800: 88.411 ± 29.49 / 12.508 ± 5.387 / 12.451 ± 5.414 | Learning curves per learner, means and s.d. over 50 seeds |
| Gaussian naive Bayes: D = Platt - Tuned by m (mean ± s.d.; corrected CI) | m=50: 25.109 ± 47.077 (corrected CI -38.253, 88.471); m=100: 50.343 ± 37.898 (corrected CI -0.664, 101.35); m=200: 65.557 ± 35.923 (corrected CI 17.207, 113.907); m=400: 72.93 ± 32.037 (corrected CI 29.811, 116.049); m=800: 75.96 ± 31.74 (corrected CI 33.24, 118.679) | Platt behind at every budget; robust from m = 200 |
| Gaussian naive Bayes: raw minus Platt and raw minus Tuned at m = 800 (mean ± s.d.; corrected CI) | 373.843 ± 48.167 (corrected CI 309.014, 438.672); 449.803 ± 39.495 (corrected CI 396.646, 502.96) | Negative favours raw (no validation data); robust only for NB |
| Gaussian naive Bayes: oracle cost C* per 1,000 e-mails by c = 0.02, 0.1, 0.25, 0.75, 0.9, 0.98 (mean ± s.d.) | 8.956 ± 1.015; 30.318 ± 2.202; 41.809 ± 4.013; 34.913 ± 3.456; 23.701 ± 2.336; 7.289 ± 0.501 | The relative endpoint weights each cell by 1/C* |
| Gaussian naive Bayes: absolute Platt - Tuned per 1,000 e-mails at m = 100 | 25.426 [95% CI 23.401, 27.451; corrected 15.837, 35.015] | Positive favours tuning; not pre-specified; robust, and the whole source of the pooled absolute advantage for tuning |
| Gaussian naive Bayes: Platt on posterior probability / on winsorised log-odds, excess (%) by m (mean ± s.d.) | m=50: 62.315 ± 9.805 / 55.981 ± 20.306; m=100: 62.959 ± 14.872 / 43.807 ± 14.936; m=200: 60.861 ± 12.555 / 40.119 ± 16.483; m=400: 58.638 ± 10.085 / 39.653 ± 15.898; m=800: 57.536 ± 9.429 / 37.736 ± 12.887 | Bounded-input Platt fits (input-scale ablation) |
| Gaussian naive Bayes: Platt(log-odds) minus Platt(winsorised) by m | m=50: 42.997 [95% CI 37.664, 48.329; corrected 17.742, 68.251]; m=100: 51.421 [95% CI 45.737, 57.104; corrected 24.503, 78.338]; m=200: 52.673 [95% CI 46.211, 59.134; corrected 22.071, 83.274]; m=400: 50.824 [95% CI 43.283, 58.364; corrected 15.114, 86.533]; m=800: 50.675 [95% CI 42.604, 58.747; corrected 12.45, 88.901] | Positive: winsorising lowers Platt's excess; robust at every budget |
| Gaussian naive Bayes: Platt(winsorised) minus Tuned by m | m=50: -17.888 [95% CI -31.982, -3.794; corrected -84.635, 48.86]; m=100: -1.078 [95% CI -11.035, 8.879; corrected -48.233, 46.078]; m=200: 12.885 [95% CI 4.823, 20.946; corrected -25.294, 51.063]; m=400: 22.107 [95% CI 16.107, 28.106; corrected -6.307, 50.52]; m=800: 25.284 [95% CI 21.036, 29.532; corrected 5.167, 45.402] | Negative favours winsorised Platt |
| Gaussian naive Bayes: Platt(log-odds) minus Platt(probability); Platt(probability) minus Tuned, by m | m=50: 36.663 [95% CI 32.643, 40.682; corrected 17.628, 55.697] / -11.554 [95% CI -24.138, 1.031; corrected -71.154, 48.047]; m=100: 32.269 [95% CI 26.099, 38.439; corrected 3.048, 61.491] / 18.074 [95% CI 9.963, 26.185; corrected -20.338, 56.486]; m=200: 31.931 [95% CI 24.465, 39.396; corrected -3.425, 67.286] / 33.626 [95% CI 28.196, 39.057; corrected 7.907, 59.346]; m=400: 31.838 [95% CI 24.043, 39.633; corrected -5.078, 68.754] / 41.092 [95% CI 37.432, 44.752; corrected 23.759, 58.426]; m=800: 30.875 [95% CI 22.653, 39.097; corrected -8.064, 69.814] / 45.085 [95% CI 41.676, 48.494; corrected 28.939, 61.231] | The probability input helps less than winsorising |
| Gaussian naive Bayes: test AUROC; Platt slope at m = 800 (mean ± s.d.; median); median slope on probability / winsorised input | 0.955 ± 0.005; 0.003 ± 0.005 (median 1.38e-07); 4.1 / 0.013 | Slope below 1 means over-confident log-odds; NB's log-odds slope collapses |
| Gaussian naive Bayes: test log-odds 1st / 50th / 99th percentile; raw plug-in excess (%) | -175000000.0 / 15.2 / 626.0; 462.254 ± 38.857 | Scale and tails of the thresholded score; raw uses no validation data |
| logistic regression: per-cell Platt - Tuned at m = 100 by c (mean ± s.d.; corrected CI) | c=0.02: -88.069 ± 124.0 (corrected CI -254.963, 78.826); c=0.1: -1.123 ± 37.694 (corrected CI -51.855, 49.61); c=0.25: 2.328 ± 20.942 (corrected CI -25.857, 30.514); c=0.75: 11.48 ± 26.329 (corrected CI -23.956, 46.917); c=0.9: -3.675 ± 34.911 (corrected CI -50.662, 43.312); c=0.98: -102.043 ± 125.744 (corrected CI -271.285, 67.2) | Per-cell paired differences over 50 seeds |
| gradient boosting: per-cell Platt - Tuned at m = 100 by c (mean ± s.d.; corrected CI) | c=0.02: -107.249 ± 186.897 (corrected CI -358.797, 144.299); c=0.1: -18.662 ± 41.707 (corrected CI -74.796, 37.472); c=0.25: -11.649 ± 26.172 (corrected CI -46.874, 23.577); c=0.75: -3.537 ± 17.989 (corrected CI -27.748, 20.675); c=0.9: -17.616 ± 48.968 (corrected CI -83.523, 48.291); c=0.98: -104.929 ± 185.791 (corrected CI -354.99, 145.131) | Per-cell paired differences over 50 seeds |
| Gaussian naive Bayes: per-cell Platt - Tuned at m = 100 by c (mean ± s.d.; corrected CI) | c=0.02: -62.968 ± 103.176 (corrected CI -201.835, 75.899); c=0.1: 58.845 ± 29.879 (corrected CI 18.631, 99.06); c=0.25: 191.49 ± 58.941 (corrected CI 112.161, 270.82); c=0.75: 169.41 ± 33.203 (corrected CI 124.721, 214.099); c=0.9: 43.023 ± 29.648 (corrected CI 3.119, 82.926); c=0.98: -97.742 ± 130.447 (corrected CI -273.314, 77.829) | Per-cell paired differences over 50 seeds |
| logistic regression: per-cell Platt - Tuned at m = 800 by c (mean ± s.d.; corrected CI) | c=0.02: -10.0 ± 25.303 (corrected CI -44.055, 24.056); c=0.1: 1.253 ± 12.392 (corrected CI -15.426, 17.932); c=0.25: -0.228 ± 6.542 (corrected CI -9.033, 8.577); c=0.75: 1.133 ± 10.653 (corrected CI -13.205, 15.471); c=0.9: 0.992 ± 17.742 (corrected CI -22.887, 24.87); c=0.98: -6.188 ± 19.752 (corrected CI -32.772, 20.396) | Per-cell paired differences over 50 seeds |
| gradient boosting: per-cell Platt - Tuned at m = 800 by c (mean ± s.d.; corrected CI) | c=0.02: -6.177 ± 23.734 (corrected CI -38.122, 25.768); c=0.1: -3.521 ± 9.134 (corrected CI -15.815, 8.772); c=0.25: -1.977 ± 6.69 (corrected CI -10.981, 7.028); c=0.75: -2.5 ± 9.452 (corrected CI -15.222, 10.222); c=0.9: -4.893 ± 14.176 (corrected CI -23.973, 14.187); c=0.98: -3.943 ± 35.969 (corrected CI -52.354, 44.469) | Per-cell paired differences over 50 seeds |
| Gaussian naive Bayes: per-cell Platt - Tuned at m = 800 by c (mean ± s.d.; corrected CI) | c=0.02: 7.527 ± 25.974 (corrected CI -27.432, 42.486); c=0.1: 60.643 ± 34.809 (corrected CI 13.792, 107.493); c=0.25: 187.724 ± 78.559 (corrected CI 81.99, 293.459); c=0.75: 159.829 ± 51.124 (corrected CI 91.02, 228.638); c=0.9: 56.221 ± 22.493 (corrected CI 25.947, 86.495); c=0.98: -16.187 ± 23.153 (corrected CI -47.349, 14.976) | Per-cell paired differences over 50 seeds |
Caveats
- Single, dated benchmark: 4,210 deduplicated e-mails from one company in 1999 with pre-engineered frequency features; effect sizes and possibly signs may not transfer.
- Two inference levels. Seed-level intervals describe the expected difference over random splits of this corpus. The Nadeau-Bengio correction approximates the corpus's own sampling variability with a heuristic between-split correlation (n_test/n). Applied to every contrast (half-widths x 4.736), it leaves H2, every LR and GB Platt-versus-tuning contrast (relative, absolute and per cell), every raw-versus-recipe contrast for LR and GB, and every isotonic contrast with intervals covering 0.
- Excess cost is relative to a test-fitted oracle. The pre-specified relative endpoint weights each cell by 1/C*, with C* from 4.686 to 41.809 per 1,000 e-mails. The unweighted absolute endpoint, which was not pre-specified, has the opposite sign at m <= 100, so the pooled H1 verdict depends on the weighting: in absolute cost the interval lies above 0 (tuning cheaper) from m = 100 on, even under the correction (at m = 50 the corrected interval covers 0), and naive Bayes drives this entirely, as logistic regression and gradient boosting favour Platt at the seed level. No absolute analogue of H2 was computed.
- The pooled endpoint averages learners whose effects have opposite signs; its null result does not mean the recipes are equivalent.
- The ablations (isotonic partition with midpoint placement; Platt on the posterior probability and on log-odds winsorised at Tukey's far-out fences of the validation scores) were specified after the first review, not in the archived plan. Tuning with interpolated placement, other clipping rules, beta calibration and cost-aware calibration were not run.
- Winsorising removes about half of NB's Platt excess but leaves a plateau of 37.736-40.119% from m = 200. This residual sigmoid misfit is inferred, not decomposed further.
- Three learners at default hyperparameters (version-dependent: scikit-learn 1.9.0), six cost proportions, budgets up to 800, no cross-fitted calibration. Up to 0.044 of gradient-boosting test scores are tied.
- Budgets are nested within seeds, so contrasts across m are correlated. Secondary contrasts carry no multiplicity adjustment.
5.Paper writing
Manuscript drafted: “Calibrate or Tune? Cost-Sensitive Decision Thresholds as Estimators on Spambase” (≈ 4989 words).
≈ 4989 words · AI-drafted manuscript · not peer-reviewed
Calibrate or Tune? Cost-Sensitive Decision Thresholds as Estimators on Spambase
Abstract
Under asymmetric costs, held-out data can recalibrate scores for the theoretical threshold or tune the threshold directly. Monotone calibration followed by thresholds the raw score, so both recipes estimate one cut. We make explicit that isotonic plug-in attains the minimum empirical validation cost, and contrast tuning's cube-root rate with the parametric rate and bias of Platt's sigmoid. On Spambase (4,210 e-mails; 3 learners × 6 cost proportions × 5 budgets of 50–800 e-mails; 50 seeds), the pre-specified pooled comparison at 100 e-mails is unsupported: Platt minus tuned excess cost is −7.9 points (95% CI −16.5 to 0.6). An absolute-cost endpoint, not pre-specified, favours tuning at that budget by 6.9 per 1,000 e-mails and survives a split-overlap correction from 100 e-mails upward (not at 50), but naive Bayes drives it entirely. At the seed level, Platt beats tuning for logistic regression and gradient boosting and trails for naive Bayes, and isotonic's small-budget edge comes from cut placement; under the correction only the naive Bayes deficit survives, so the rest are point-estimate patterns on this corpus. Winsorising naive Bayes's heavy-tailed log-odds halves its Platt excess, leaving a residual bias.
Keywords: cost-sensitive classification; probability calibration; decision threshold; isotonic regression; Platt scaling; Spambase
1 Introduction
With false-positive and false-negative costs and , the Bayes rule flags an instance whose class probability exceeds [1]. As fitted scores are rarely calibrated, practitioners either recalibrate them (Platt [2], isotonic regression [3]) and threshold at , or tune the threshold on validation data [4]. We found no comparison of the two at a controlled budget. Search was compared only with uncalibrated probabilities [4] and calibrators only on squared error [5], and regret bounds after recalibration [8] do not say how much data each recipe needs. Since a non-decreasing calibrator turns into a threshold on the raw score , both recipes estimate one cut. We contribute:
- Proposition 1 (after [6, 7]): isotonic plug-in attains the minimum empirical cost and differs from tuning only by tie-breaking and cut placement within one gap.
- Proposition 2: tuning's excess cost is [9]; under stated regularity, Platt's cut converges at the parametric rate to a limit that is biased unless the sigmoid is exact at level .
- A paired evaluation on Spambase [11], pre-specified in the archived run record, with seed-level and split-overlap-corrected [10] intervals and ablations of cut placement and input scale.
- Findings. The pre-specified pooled contrast is unsupported. An absolute-cost endpoint, not pre-specified, robustly favours tuning from on, but only through naive Bayes. Logistic regression and gradient boosting favour Platt and cut placement drives isotonic's small-budget edge, but these are seed-level, point-estimate patterns: only the naive Bayes results survive the overlap correction.
2 Related Work
Cost-sensitive decisions. Elkan [1] derived the theoretical threshold, which is optimal for perfectly calibrated scores [7]; Sheng and Ling [4] instead tune it on cross-validated predictions, and Perez-Lebel et al. [8] separate the regret that recalibration removes from a grouping loss it cannot. Provost and Fawcett [12] choose operating points under imprecise costs from the ROC convex hull, which is equivalent to PAV calibration [6], the basis of Proposition 1. Spam filtering is strongly asymmetric: Androutsopoulos et al. [13] used thresholds up to 0.999, where blocking one legitimate e-mail is as bad as passing 999 spam.
Calibration. Platt [2] fits a sigmoid with smoothed targets, which Lin, Lin and Weng [14] fit robustly; Zadrozny and Elkan [3] use isotonic regression. Niculescu-Mizil and Caruana [5] find that naive Bayes pushes probabilities towards 0 and 1, and that Platt beats isotonic below roughly 200–1,000 calibration cases. Kumar, Liang and Ma [15] likewise find binning less sample-efficient than scaling. Kull, Silva Filho and Flach [16] proposed beta calibration because the logistic map misfits skewed scores such as naive Bayes's. Böken [17] asks when Platt scaling is appropriate, and Yang and Bi [18] make calibration cost-aware.
3 Problem Setup
Let be the log-odds score of a base classifier trained on separate data, the label (1 = spam), and . A rule costs , minimised by with [1]. Write for the rule and for its empirical cost on a validation sample . A non-decreasing calibrator fitted on yields the rule ; tuning picks .
Proposition 1 (isotonic plug-in is empirical tuning). Let be the isotonic (PAV) fit of on over , with tied scores pooled. Then attains . On new points it can disagree with a tuned rule only (i) inside the gap between the validation scores adjacent to the selected cut, or beyond the extreme score, or (ii) on blocks whose inclusion leaves unchanged.
Proof sketch. PAV splits the sorted scores into blocks with non-decreasing positive rates . Flagging block changes by , which changes sign once, so flagging exactly the blocks with is optimal. No cut inside a block does better, or PAV would split it. Zero increments give (ii).
Proposition 2 (rates; informal). Let the class-conditional densities of be continuous and positive near the Bayes cut , let increase with , and let . (a) A tuned threshold has and . (b) Suppose the sigmoid's Kullback–Leibler projection is unique with , so the fitted slope stays bounded away from zero with probability tending to one, and Platt's cut has finite second moments. Then at rate and , where the bias vanishes exactly when .
Proof sketch. (a) Kim and Pollard's cube-root theorem [9], then a second-order expansion of at . (b) The maximum-likelihood estimate is -consistent for its projection; the slope condition passes this rate to by the delta method, and the moment condition makes the mean first-order term of at of order .
Platt should thus win at small when is small, and tuning once falls below : the parametric-versus-binning trade-off of [15], applied to decisions.
4 Method
Algorithm 1 is the executed protocol; every rule sees the same validation e-mails.
Algorithm 1 Paired evaluation of calibrate-then-threshold versus tuning
for seed r in 0..49:
stratified split: train 50%, pool 20%, test 30%; permute pool
for learner f in {LR, GB, NB}:
fit f on train; z <- log-odds on pool and test; C*[c] <- minimum test cost over cuts
for m in {50, 100, 200, 400, 800}: V <- first m pool e-mails
Platt: MLE sigmoid(a u + b) on V, u = z; flag if a u + b > logit(c)
(Platt-prob: u = sigmoid(z); Platt-wins: u = z clipped to V's far-out fences)
Isotonic: g <- PAV fit on V; flag if g(z) > c (Iso-mid: its partition of V, midpoint cut)
Tuned: median cut minimising c FP + (1-c) FN on V; flag above the gap midpoint
E[rule, c] <- 100 (test cost / C*[c] - 1); A[c] <- cost(Platt) - cost(Tuned)
Tuning scans all cuts with cumulative counts, in ; PAV costs and each Platt fit per iteration.
5 Experiments
5.1 Setup
Data. Spambase [11] is dataset 94 of the UCI Machine Learning Repository, donated by Hewlett-Packard Labs (licence CC BY 4.0) and generated in June–July 1999 from filed spam and filed work and personal e-mail. We retrieved it on 24 October 2026 and verified its checksum. Its 4,601 e-mails have 57 predictors and no missing values: 48 word and 6 character frequencies, as percentages (0–100) of words or characters matching a token, and the mean, longest and total length of capital-letter runs. The label is_spam is 1 for spam. Removing 391 exact duplicates, so that no e-mail sits in both validation and test, leaves 4,210 e-mails (1,679 spam). Predictors are transformed by .
Protocol. Each of 50 seeds draws stratified splits of 2,105 training, 842 pool and 1,263 test e-mails. Budgets are nested prefixes of a seeded permutation of the pool; every prefix held at least 15 spam and 24 legitimate e-mails. The cost proportions include extremes like those used for spam [13].
Learners and rules. We used L2 logistic regression (LR; , standardised inputs), histogram gradient boosting (GB) and Gaussian naive Bayes (NB), otherwise with scikit-learn defaults. Platt is fitted by effectively unpenalised maximum likelihood (L-BFGS, , scores divided by their interquartile range) with targets and [2, 14]. The tuned threshold is the largest validation score when the optimal cut flags no validation e-mail, and when it flags all. Three ablations were added in revision:
- Iso-mid: isotonic's validation partition with tuning's midpoint placement;
- Platt-prob: Platt on , scikit-learn's input for models without a decision function, such as NB;
- Platt-wins: Platt on winsorised at the validation scores' Tukey far-out fences, .
Raw applies to the model's own probability, and the oracle is the best threshold on the test set.
Metric and inference. Excess cost is the percentage by which a rule's test cost exceeds the oracle's, ; it isolates the regret that thresholding can change [8]. The primary endpoint, pre-specified in the archived run record, is . is a seed's mean over the 18 learner-by-cost cells of , which weights each cell's cost difference by . We also report the unweighted per 1,000 e-mails, which was not pre-specified. We give means ± s.d. over seeds, paired 95% -intervals and Wilcoxon signed-rank tests. Seed-level intervals ignore the corpus's own sampling variability. The heuristic Nadeau–Bengio correction [10], applied to every contrast, approximates it by scaling standard errors by , which widens half-widths 4.74-fold (from 0.284 to 1.346 s.d.). A contrast is robust if its corrected interval excludes zero, and only robust contrasts are claimed beyond this corpus. All contrasts but , including the rest of Table 3 and Tables 1, 4, 5 and S1–S3, are secondary and exploratory, without multiplicity adjustment. The run took 10.1 s on one Apple M4 CPU (Appendix B).
5.2 Main Results
Table 1: Base learners and the raw baseline (50 seeds; secondary)
| Learner | AUROC | Platt slope (median) | Platt-wins slope | Log-odds p1/p50/p99 | Raw excess (%) | Raw − Platt | Raw − Tuned |
|---|---|---|---|---|---|---|---|
| LR | 0.978 ± 0.003 | 0.856 ± 0.099 (0.857) | 0.857 | −25.3/−2.04/11.4 | 12.4 ± 3.7 | −1.7 ± 5.3* | −3.9 ± 7.0* |
| GB | 0.984 ± 0.003 | 0.649 ± 0.061 (0.639) | 0.639 | −10.3/−3.92/8.9 | 26.2 ± 9.3 | 12.2 ± 10.6* | 8.4 ± 11.4* |
| NB | 0.955 ± 0.005 | 0.003 ± 0.005 (1.38e−7) | 0.013 | −1.75e8/15.2/626 | 462.3 ± 38.9 | 373.8 ± 48.2† | 449.8 ± 39.5† |
Notes: mean ± s.d. (medians in parentheses); slopes and raw contrasts at ; percentiles are medians over seeds. A slope below 1 means over-confident log-odds. Raw contrasts are paired differences in points, negative favouring raw; * seed-level 95% CI excludes 0; † overlap-corrected CI too (robust). Tied test scores: at most 0.002 (LR, NB) and 0.044 (GB).
Table 2: Hypotheses pre-specified in the archived run record (50 seeds)
| Hypothesis | Estimate [95% CI] | Overlap-corrected 95% CI | Verdict |
|---|---|---|---|
| H1: | −7.9 [−16.5, 0.6]; Wilcoxon = 0.121; Platt better in 56% of seeds | [−48.3, 32.5] | Unsupported (CI covers 0) |
| H2: | 31.2 [23.2, 39.3] | [−7.1, 69.5] | Supported at seed level; not robust |
| H3: isotonic–tuned test disagreement < 1% for | at most 0.60% | upper limit 1.08% at = 100 | Holds by its rule; not robust at = 100 |
Notes: (Section 5.1) is in points. The plan calls H1 unsupported if its interval covers 0 and refuted if it lies above 0.
Table 3: Excess test cost (%) by budget, pooled over learners and costs (50 seeds)
| m | Platt | Isotonic | Tuned | = Platt − Tuned (points) | Platt − Tuned, absolute (per 1,000 e-mails) |
|---|---|---|---|---|---|
| 50 | 59.82 ± 15.22 | 75.71 ± 32.59 | 95.69 ± 43.34 | −35.9 [−50.4, −21.4] | 5.2 [3.9, 6.5] |
| 100 | 49.27 ± 12.14 | 52.25 ± 21.89 | 57.19 ± 25.00 | −7.9 [−16.5, 0.6] | 6.9 [5.9, 7.8]† |
| 200 | 43.46 ± 11.13 | 30.65 ± 12.33 | 32.03 ± 13.27 | 11.4 [6.1, 16.7] | 8.1 [7.2, 9.0]† |
| 400 | 40.53 ± 10.64 | 20.69 ± 5.34 | 20.84 ± 5.67 | 19.7 [16.0, 23.4]† | 8.6 [7.7, 9.5]† |
| 800 | 38.83 ± 11.15 | 15.47 ± 3.76 | 15.51 ± 3.66 | 23.3 [19.9, 26.7]† | 8.9 [7.9, 9.8]† |
Notes: mean ± s.d.; bold marks the row minimum. Differences are mean [seed-level 95% CI]; † marks a robust contrast.
Table 4: Learning curves: excess test cost (%) by learner and rule (mean ± s.d., 50 seeds)
| Learner, rule | m=50 | m=100 | m=200 | m=400 | m=800 |
|---|---|---|---|---|---|
| LR Platt | 41.2 ± 17.4 | 24.9 ± 11.2 | 18.1 ± 7.2 | 15.5 ± 5.4 | 14.1 ± 4.6 |
| LR Isotonic | 81.7 ± 46.9 | 50.6 ± 25.3 | 29.9 ± 15.1 | 21.0 ± 7.8 | 16.3 ± 6.3 |
| LR Tuned | 99.1 ± 58.7 | 55.1 ± 28.0 | 30.6 ± 15.4 | 20.8 ± 8.1 | 16.3 ± 6.5 |
| LR Δ = Platt − Tuned | −57.9 ± 69.3* | −30.2 ± 33.9* | −12.4 ± 18.2* | −5.3 ± 10.3* | −2.2 ± 6.4* |
| GB Platt | 39.3 ± 21.1 | 27.7 ± 15.1 | 19.5 ± 9.1 | 15.6 ± 7.0 | 13.9 ± 6.4 |
| GB Isotonic | 81.2 ± 54.1 | 63.6 ± 40.4 | 35.7 ± 21.0 | 23.8 ± 10.2 | 17.6 ± 6.7 |
| GB Tuned | 114.1 ± 72.2 | 71.6 ± 46.9 | 38.3 ± 22.4 | 24.2 ± 10.9 | 17.8 ± 6.9 |
| GB Δ = Platt − Tuned | −74.8 ± 79.9* | −43.9 ± 50.5* | −18.8 ± 23.8* | −8.6 ± 12.0* | −3.8 ± 8.2* |
| NB Platt | 99.0 ± 16.8 | 95.2 ± 19.4 | 92.8 ± 23.9 | 90.5 ± 27.2 | 88.4 ± 29.5 |
| NB Platt-wins | 56.0 ± 20.3 | 43.8 ± 14.9 | 40.1 ± 16.5 | 39.7 ± 15.9 | 37.7 ± 12.9 |
| NB Isotonic | 64.2 ± 34.3 | 42.6 ± 26.9 | 26.3 ± 19.0 | 17.3 ± 9.7 | 12.5 ± 5.4 |
| NB Tuned | 73.9 ± 40.8 | 44.9 ± 28.5 | 27.2 ± 19.3 | 17.5 ± 9.7 | 12.5 ± 5.4 |
| NB Δ = Platt − Tuned | 25.1 ± 47.1* | 50.3 ± 37.9* | 65.6 ± 35.9† | 72.9 ± 32.0† | 76.0 ± 31.7† |
Notes: bold marks a learner's lowest unrounded mean. Δ rows are paired differences in points; * seed-level 95% CI excludes 0; † robust. This table replaces a learning-curve figure.
Table 2 gives the hypothesis verdicts. The pooled sign depends on the weighting (Table 3). turns positive from , robustly from , while the absolute difference favours tuning at every budget, robustly from . Both pool effects of opposite sign (Table 4). At the seed level Platt beats tuning at every budget for LR and GB (−30.2 ± 33.9 and −43.9 ± 50.5 points at ). No LR or GB Platt-versus-tuning contrast is robust, relative or absolute (Table S1). NB's Platt excess stays at 88.4–99.0% while tuning's falls to 12.5%, a deficit robust from . In absolute cost at , LR (−1.6 per 1,000 e-mails) and GB (−3.2) favour Platt and NB (25.4) favours tuning.
5.3 Ablations
Table 5: Isotonic versus tuning, with the cut-placement ablation (pooled, 50 seeds)
| m | Isotonic − Tuned | Placement: Isotonic − Iso-mid | Partition: Iso-mid − Tuned | Disagreement (% of e-mails) | Same partition |
|---|---|---|---|---|---|
| 50 | −19.98 [−25.61, −14.35] | −20.56 [−26.09, −15.04] | 0.58 [−0.31, 1.48] | 1.25 ± 0.80 | 0.968 |
| 100 | −4.94 [−6.41, −3.47] | −5.28 [−6.63, −3.92] | 0.33 [−0.17, 0.84] | 0.60 ± 0.35 | 0.956 |
| 200 | −1.38 [−2.00, −0.76] | −1.34 [−1.84, −0.84] | −0.04 [−0.45, 0.36] | 0.38 ± 0.36 | 0.933 |
| 400 | −0.15 [−0.47, 0.17] | −0.21 [−0.38, −0.03] | 0.05 [−0.24, 0.34] | 0.31 ± 0.24 | 0.899 |
| 800 | −0.04 [−0.32, 0.24] | 0.01 [−0.08, 0.10] | −0.05 [−0.31, 0.22] | 0.19 ± 0.15 | 0.896 |
Notes: differences in points, mean [seed-level 95% CI]; the ablation columns sum to the first, and none is robust. Disagreement: mean ± s.d.; last column: share of fits. Largest validation-cost gap over 4,500 rule pairs: 6.94e−18 (rounding error).
Cut placement. As Proposition 1 predicts, isotonic and tuned rules have equal validation cost in every fit, and 89.6–96.8% of fits share the validation partition (Table 5). Yet isotonic is cheaper at small budgets (−19.98 points at ), and Iso-mid removes all of this edge. Placement accounts for −20.56 and partition choice for 0.58 [−0.31, 1.48]. Isotonic's interpolated cut lies a fraction of the way across the gap, where and are the adjacent block rates, so it moves with ; the midpoint ignores . None of this is robust.
Input scale. Platt-wins lowers NB's Platt excess by 51.4 [45.7, 57.1] points at , robustly, to 43.8 ± 14.9%. That is not distinguishable from tuning at this budget (−1.1 [−11.0, 8.9]; corrected [−48.2, 46.1]; Table S2), but from Platt-wins plateaus at 37.7–40.1% and trails tuning, robustly at . Platt-prob helps less (63.0 ± 14.9% at ), presumably because it maps the upper half of NB's scores (log-odds above the median, 15.2) to within of 1. For LR and GB, winsorising changes Platt's excess by at most 0.08 points.
No validation data. Raw plug-in is worst by far for NB (462.3%, robustly; Table 1). At , LR's own probabilities cost less than Platt and tuning (by 1.7 and 3.9 points) and GB's cost more, at the seed level only, so whether LR needs recalibration is unresolved.
5.4 Analysis
LR (slope 0.856) and GB (0.649) have over-confident log-odds that a sigmoid can rescale, judging by the outcomes. NB's log-odds are heavy-tailed (1st percentile about −1.75e8, median 15.2), and its slope collapses to a median of 1.38e−7 (Table 1). Such a slope is not bounded away from zero and makes the cut unstable, so the conditions of Proposition 2(b) fail and it does not describe Platt on NB's log-odds. On winsorised log-odds the median slope is 0.013, the conditions are plausible, and the plateau is consistent with the fixed bias of a sigmoid not exact at level . Beta calibration [16] is the established remedy for such misfit on skewed naive Bayes scores, but we did not run it.
Weighting. is 4.7–9.0 per 1,000 e-mails at and 20.5–41.8 at (Table S3), so weights extreme cells several times more than central ones. At Platt leads in all six extreme cells, by 63.0–107.2 points, presumably because tuning is noisy where the cut lies in a sparse tail, but no such lead is robust; only NB's four central cells are, where Platt trails by 43.0–191.5 points. The pooled H1 verdict therefore depends on the weighting.
6 Limitations
The evidence comes from one dated benchmark (4,210 e-mails, one company, 1999, engineered frequencies), so effect sizes and possibly signs may not transfer. Seed-level intervals are conditional on this corpus, the overlap correction is heuristic, and there is no multiplicity adjustment. Excess cost is relative to a test-fitted oracle, and pooled verdicts depend on the weighting. The ablations were added after the first review, outside the archived plan. Tuning with interpolated placement, other clipping rules, beta calibration [16] and cost-aware calibration [18] were not run. We used three default-hyperparameter learners (defaults vary with the scikit-learn version), six cost proportions, budgets up to 800, no cross-fitting, and tables instead of figures.
7 Conclusion
At equal held-out budgets, "calibrate, then threshold at " is neither uniformly better nor worse than tuning. Isotonic calibration is tuning up to where the cut sits inside one gap, and on Spambase that placement helped at small budgets, at the seed level. The pre-specified pooled test is unsupported, and its sign flips with the weighting: in absolute cost, tuning is robustly cheaper because of naive Bayes. Platt's lead for LR and GB is a seed-level pattern on one corpus. Its NB deficit is robust; winsorising NB's log-odds halves Platt's excess but leaves a residual bias. Practitioners should check score tails and the calibrator's fit near , and whether recalibration is needed at all.
References
[1] C. Elkan. The foundations of cost-sensitive learning. In Proc. IJCAI, pages 973–978, 2001. https://mlanthology.org/ijcai/2001/elkan2001ijcai-foundations/
[2] J. C. Platt. Probabilities for SV machines. In Advances in Large-Margin Classifiers, pages 61–74. MIT Press, 2000. https://doi.org/10.7551/mitpress/1113.003.0008
[3] B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proc. ACM SIGKDD, pages 694–699, 2002. https://doi.org/10.1145/775047.775151
[4] V. S. Sheng and C. X. Ling. Thresholding for making classifiers cost-sensitive. In Proc. AAAI, pages 476–481, 2006. https://cdn.aaai.org/AAAI/2006/AAAI06-076.pdf
[5] A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proc. ICML, pages 625–632, 2005. https://doi.org/10.1145/1102351.1102430
[6] T. Fawcett and A. Niculescu-Mizil. PAV and the ROC convex hull. Machine Learning, 68(1):97–106, 2007. https://doi.org/10.1007/s10994-007-5011-0
[7] J. Hernández-Orallo, P. Flach and C. Ferri. A unified view of performance metrics: translating threshold choice into expected classification loss. JMLR, 13:2813–2869, 2012. https://jmlr.org/papers/v13/hernandez-orallo12a.html
[8] A. Perez-Lebel, G. Varoquaux, S. Koyejo, M. Doutreligne and M. Le Morvan. Decision from suboptimal classifiers: excess risk pre- and post-calibration. In Proc. AISTATS, PMLR 258:2395–2403, 2025. https://proceedings.mlr.press/v258/perez-lebel25a.html
[9] J. Kim and D. Pollard. Cube root asymptotics. Annals of Statistics, 18(1):191–219, 1990. https://doi.org/10.1214/aos/1176347498
[10] C. Nadeau and Y. Bengio. Inference for the generalization error. Machine Learning, 52(3):239–281, 2003. https://doi.org/10.1023/A:1024068626366
[11] M. Hopkins, E. Reeber, G. Forman and J. Suermondt. Spambase [Dataset]. UCI Machine Learning Repository, 1999. https://doi.org/10.24432/C53G6X
[12] F. Provost and T. Fawcett. Robust classification for imprecise environments. Machine Learning, 42(3):203–231, 2001. https://doi.org/10.1023/A:1007601015854
[13] I. Androutsopoulos, J. Koutsias, K. V. Chandrinos, G. Paliouras and C. D. Spyropoulos. An evaluation of naive Bayesian anti-spam filtering. In Proc. Workshop on Machine Learning in the New Information Age, 11th European Conference on Machine Learning, pages 9–17, 2000. https://arxiv.org/abs/cs/0006013
[14] H.-T. Lin, C.-J. Lin and R. C. Weng. A note on Platt's probabilistic outputs for support vector machines. Machine Learning, 68(3):267–276, 2007. https://doi.org/10.1007/s10994-007-5018-6
[15] A. Kumar, P. Liang and T. Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems 32, 2019. https://arxiv.org/abs/1909.10155
[16] M. Kull, T. Silva Filho and P. Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proc. AISTATS, PMLR 54:623–631, 2017. https://proceedings.mlr.press/v54/kull17a.html
[17] B. Böken. On the appropriateness of Platt scaling in classifier calibration. Information Systems, 95:101641, 2021. https://doi.org/10.1016/j.is.2020.101641
[18] M. Yang and X. Bi. Cost-aware calibration of classifiers. INFORMS Journal on Data Science, 4(2):101–113, 2025. https://doi.org/10.1287/ijds.2024.0038
Appendix (supplementary material)
A Supplementary tables
Table S1: Absolute cost difference Platt − Tuned by learner (per 1,000 e-mails, mean [95% CI], 50 seeds; secondary)
| Learner | m=50 | m=100 | m=200 | m=400 | m=800 |
|---|---|---|---|---|---|
| LR | −2.5 [−4.4, −0.6] | −1.6 [−2.6, −0.6] | −0.6 [−1.1, −0.1] | −0.3 [−0.6, 0.1] | −0.1 [−0.4, 0.2] |
| GB | −5.4 [−7.0, −3.7] | −3.2 [−4.3, −2.2] | −1.6 [−2.0, −1.2] | −0.8 [−1.1, −0.6] | −0.5 [−0.7, −0.3] |
| NB | 23.4 [21.5, 25.3]† | 25.4 [23.4, 27.5]† | 26.5 [24.1, 29.0]† | 26.9 [24.3, 29.5]† | 27.2 [24.3, 30.0]† |
Notes: positive favours tuning; † marks a robust contrast. Pooled values are in Table 3.
Table S2: Input-scale ablation: Platt on bounded inputs (points, mean [95% CI], 50 seeds; secondary)
| Contrast | m=50 | m=100 | m=200 | m=400 | m=800 |
|---|---|---|---|---|---|
| NB: Platt − Platt-wins | 43.0 [37.7, 48.3]† | 51.4 [45.7, 57.1]† | 52.7 [46.2, 59.1]† | 50.8 [43.3, 58.4]† | 50.7 [42.6, 58.7]† |
| NB: Platt-wins − Tuned | −17.9 [−32.0, −3.8] | −1.1 [−11.0, 8.9] | 12.9 [4.8, 20.9] | 22.1 [16.1, 28.1] | 25.3 [21.0, 29.5]† |
| NB: Platt − Platt-prob | 36.7 [32.6, 40.7]† | 32.3 [26.1, 38.4]† | 31.9 [24.5, 39.4] | 31.8 [24.0, 39.6] | 30.9 [22.7, 39.1] |
| NB: Platt-prob − Tuned | −11.6 [−24.1, 1.0] | 18.1 [10.0, 26.2] | 33.6 [28.2, 39.1]† | 41.1 [37.4, 44.8]† | 45.1 [41.7, 48.5]† |
| NB: Platt-prob excess (%, mean ± s.d.) | 62.3 ± 9.8 | 63.0 ± 14.9 | 60.9 ± 12.6 | 58.6 ± 10.1 | 57.5 ± 9.4 |
| LR: Platt − Platt-wins | 0.08 [−0.01, 0.17] | 0.07 [0.03, 0.11] | 0.06 [0.01, 0.11] | 0.01 [−0.01, 0.02] | 0.01 [−0.01, 0.03] |
| LR: Platt − Platt-prob | 8.0 [4.1, 11.8] | −4.1 [−7.1, −1.1] | −8.7 [−11.1, −6.2] | −11.8 [−13.9, −9.7]† | −13.8 [−15.8, −11.7]† |
| GB: Platt − Platt-wins | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] |
| GB: Platt − Platt-prob | −14.8 [−19.2, −10.3] | −22.9 [−27.2, −18.7]† | −27.1 [−30.7, −23.5]† | −32.5 [−36.2, −28.9]† | −35.7 [−39.5, −31.9]† |
Notes: a positive "Platt − X" means that input X lowers Platt's excess. † marks a robust contrast; for example, NB, Platt − Platt-wins at m = 100: [24.5, 78.3]; NB, Platt-wins − Tuned at m = 800: [5.2, 45.4].
Table S3: Platt − Tuned by cost proportion at , and the oracle cost (mean ± s.d., 50 seeds; secondary)
| Learner | Quantity | c=0.02 | c=0.1 | c=0.25 | c=0.75 | c=0.9 | c=0.98 |
|---|---|---|---|---|---|---|---|
| LR | Δ, m = 100 (points) | −88.1 ± 124.0* | −1.1 ± 37.7 | 2.3 ± 20.9 | 11.5 ± 26.3* | −3.7 ± 34.9 | −102.0 ± 125.7* |
| LR | C* (per 1,000 e-mails) | 6.4 ± 0.9 | 18.4 ± 1.9 | 27.1 ± 3.0 | 25.9 ± 2.9 | 16.9 ± 2.5 | 5.7 ± 0.9 |
| GB | Δ, m = 100 (points) | −107.2 ± 186.9* | −18.7 ± 41.7* | −11.6 ± 26.2* | −3.5 ± 18.0 | −17.6 ± 49.0* | −104.9 ± 185.8* |
| GB | C* (per 1,000 e-mails) | 5.7 ± 1.2 | 15.3 ± 2.2 | 22.7 ± 3.0 | 20.5 ± 2.4 | 12.4 ± 1.9 | 4.7 ± 1.2 |
| NB | Δ, m = 100 (points) | −63.0 ± 103.2* | 58.8 ± 29.9† | 191.5 ± 58.9† | 169.4 ± 33.2† | 43.0 ± 29.6† | −97.7 ± 130.4* |
| NB | C* (per 1,000 e-mails) | 9.0 ± 1.0 | 30.3 ± 2.2 | 41.8 ± 4.0 | 34.9 ± 3.5 | 23.7 ± 2.3 | 7.3 ± 0.5 |
Notes: negative Δ favours Platt. * seed-level 95% CI excludes 0; † robust. is the oracle's test cost per 1,000 e-mails with and .
B Reproducibility
Seeds 0–49 fix the splits, the pool permutation and gradient boosting. Hyperparameters and conventions are given in Section 5.1. The run recorded its own environment: Python 3.12.13, NumPy 2.5.2, SciPy 1.18.1, scikit-learn 1.9.0 and pandas 3.0.5 on macOS 27.0.1 (arm64), with an Apple M4 CPU (10 cores). It took 10.1 s. The defaults of HistGradientBoostingClassifier and LogisticRegression depend on the scikit-learn version, so other versions may give slightly different numbers. The data are public [11], and the script behind every number is archived with the run record. The revised script reproduces every number of the first submission exactly, and its added rules use no new random draws.
C NeurIPS paper checklist
Table S4: NeurIPS paper checklist
| Item | Answer | Justification |
|---|---|---|
| 1 Claims | Yes | The abstract and Section 1 state the unsupported primary result and separate robust from seed-level findings |
| 2 Limitations | Yes | Section 6 |
| 3 Theory | Yes | Assumptions and proof sketches in Section 3; Proposition 2 is informal |
| 4 Reproducibility | Yes | Algorithm 1, Section 5.1, Appendix B |
| 5 Open data and code | Yes | Public data [11]; the script is archived with the run record |
| 6 Experimental details | Yes | Section 5.1 |
| 7 Statistical significance | Yes | Means ± s.d., seed-level and overlap-corrected 95% intervals, Wilcoxon tests |
| 8 Compute | Yes | One CPU, 10.1 s (Appendix B) |
| 9 Code of ethics | Yes | Public, licensed data reduced to word and character frequencies |
| 10 Broader impacts | Yes | A methodological study; a misplaced spam threshold blocks legitimate mail |
| 11 Safeguards | NA | No model or data released |
| 12 Licences | Yes | Spambase is CC BY 4.0 [11] |
| 13 New assets | NA | None |
| 14 Crowdsourcing and human subjects | NA | None |
| 15 IRB approval | NA | None |
| 16 LLM usage | Yes | A research pipeline driven by a large language model produced this study, including the analysis code and this text; no LLM is part of the method studied |
6.Review
Automated review: Accept (novelty 3, rigor 4, clarity 4, reproducibility 5).
| Novelty 3/5 | Rigor 4/5 | Clarity 4/5 | Reproducibility 5/5 |
Refereed by Claude Fable 5.1, a different model from the one that wrote the paper (Claude Opus 5.5).
Final round. The two text changes I required in round 2 are made precisely and match the executed record, which is unchanged: the abstract and Findings bullet now present the post-hoc absolute-cost result with its three qualifiers (not pre-specified; robust from m = 100, not at 50; driven entirely by naive Bayes), and the 'level with tuning' sentence now says 'not distinguishable from tuning at this budget' with both intervals. The three trims to hold length remove no number and alter no claim. Across three rounds the paper has gone from an abstract carrying per-learner claims that its own sensitivity check overturned to one in which every claim is matched to the strength of its evidence, every number traces to RESULT_JSON, the needed ablations were run, the oracle costs and weighting are exposed, the literature gaps are filled, and the run is fully reproducible. Its contribution is modest in scope, which the scores reflect, but it is sound, honest and complete as submitted, and no required changes remain. I recommend acceptance.
Strengths
- Both remaining required changes are made exactly and agree with the record. The abstract now says the absolute-cost endpoint was not pre-specified, that it favours tuning by 6.9 per 1,000 e-mails at m = 100 and survives the overlap correction from 100 e-mails upward but not at 50 (record: m = 100 corrected [2.5, 11.2]; m = 50 corrected [-1.0, 11.3]), and that naive Bayes drives it entirely (per-learner -1.6, -3.2, +25.4); the Findings bullet says the same. Section 5.3 now reads 'not distinguishable from tuning at this budget (-1.1 [-11.0, 8.9]; corrected [-48.2, 46.1])', matching the record's -1.078 [-11.035, 8.879; -48.233, 46.078].
- The executed record is unchanged from round 2 (same script, same RESULT_JSON, exit 0, empty stderr, 10.1 s), so every number I verified cell by cell in round 2 (Tables 1-5, S1-S3, all seed-level and overlap-corrected markers, C* ranges, ablation sums) still holds; the three length trims (one lead-in sentence, two table notes now pointing to Section 5.1) remove no number and change no claim.
- The manuscript now claims only what the record supports, and says so consistently in abstract, Findings bullet, Section 5.2-5.4 and the conclusion: the pre-specified pooled contrast is unsupported; the pooled sign depends on the weighting and the learner mix; LR and GB advantages of Platt are seed-level patterns on one corpus; NB's deficit is robust; isotonic's small-budget edge is cut placement (not robust); winsorising NB's log-odds halves Platt's excess and leaves a residual bias; H2 and H3 are reported with their corrected intervals and the plan's own verdict rules.
- Soundness and reporting meet the venue's conventions: Proposition 1 is correct and verified on the record; Proposition 2(b) states the regularity it needs and Section 5.4 says NB violates it; contributions as bullets, executed-protocol pseudocode, every averaged result with +/- s.d. over 50 seeds, seed-level and overlap-corrected intervals, component-at-a-time ablations disclosed as post-review additions, a specific limitations section, full environment and compute statement, archived script with exact reproduction, and the NeurIPS checklist.
Weaknesses
- Scope and significance remain modest: one 1999 benchmark, three default-hyperparameter learners, a null pre-specified primary result, Proposition 1 a restatement of known equivalences and Proposition 2 an informal rate argument; the generalisable content is the estimation framing plus one robust, learner-specific finding (Platt's sigmoid fails on NB's heavy-tailed log-odds and winsorising only half-repairs it).
- Inference caveats the paper states but cannot remove: the Nadeau-Bengio correction is heuristic and may be conservative for paired differences that share splits; about 80 secondary contrasts carry robustness markers with no multiplicity adjustment; the ablations and the absolute endpoint are post hoc.
- Beta calibration, the cited remedy for exactly the NB misfit, and tuning with interpolated placement were not run; Boeken [17] is cited for its title only; learning curves are a table rather than a figure (a pipeline constraint).
Required changes
—
Record
§Event timeline
- Run createdQuestion: “Under asymmetric misclassification costs, does post-hoc probability calibration with the theoretical cost-optimal threshold beat directly tuning the decision threshold on the same held-out validation data?' language=en discipline=computer_science venue='NeurIPS” · output language: English · models: agent:claude-opus-5-5
- Stage started · Literature reviewAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Literature reviewsearch response for literature_review: literature_review-search-1.response.json
- Resume requested
- Stage started · Literature reviewAttempt 2 · agent:claude-opus-5-5
- Search complete · Literature review26 sources · 15,616 characters of notes
- awaiting_agent · Literature reviewcomplete_json response for literature_review: literature_review-complete_json-1.response.json
- Resume requested
- Stage started · Literature reviewAttempt 3 · agent:claude-opus-5-5
- Search complete · Literature review26 sources · 15,616 characters of notes
- Stage completed · Literature review
- Stage started · IdeationAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Ideationcomplete_json response for ideation: ideation-complete_json-1.response.json
- Resume requested
- Stage started · IdeationAttempt 2 · agent:claude-opus-5-5
- Awaiting approval · Ideation
- Approved · IdeationIdea 1 selected
- Stage started · ExperimentationAttempt 1 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
- Resume requested
- Stage started · ExperimentationAttempt 2 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- Experiment round · ExperimentationRound 1: exit 0; result parsed (9.343s)
- Stage completed · Experimentation
- Stage started · AnalysisAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 2 · agent:claude-opus-5-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 2 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-opus-5-5
- Awaiting approval · Review
- Sent back for revision · ReviewRe-running from Paper writing: “The referee recommended minor_revision. Address each required change below, quoted verbatim, using only results already in the run record (no new analyses, no new numbers):\n1. "Qualify every \'at every”
- Stage started · Paper writingAttempt 3 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 4 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-opus-5-5
- Awaiting approval · Review
- Approved · Review
- Run completereport=/Users/haodingresearch/Desktop/auto-research/runs/studies-opus55/computer_science-en/runs/4689bab459a3/REPORT.md
- Reopened after sign-off · ReviewReopened after sign-off; re-running from Review: “Independent review: the paper is re-refereed by an independent referee (Claude Fable 5.1) that took no part in writing it.”
- Stage started · ReviewAttempt 3 · agent:claude-fable-5-1
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 4 · agent:claude-fable-5-1
- Awaiting approval · Review
- Sent back for revision · ReviewRe-running from Experimentation: “Independent referee (Claude Fable 5.1): major_revision. Required changes: (1) [text] Apply the overlap correction uniformly: multiply every seed-level half-width in Tables 1, 3, 4, 5 and 6 and for H2 ”
- Stage started · ExperimentationAttempt 3 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
- Resume requested
- Stage started · ExperimentationAttempt 4 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- Experiment round · ExperimentationRound 1: exit 0; result parsed (10.957s)
- Stage completed · Experimentation
- Stage started · AnalysisAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 2 · agent:claude-opus-5-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 5 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 6 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-fable-5-1
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-fable-5-1
- Awaiting approval · Review
- Sent back for revision · ReviewRe-running from Analysis: “Independent referee (Claude Fable 5.1), round 2: minor_revision. Required changes: (1) [text] In the abstract, replace 'In absolute cost at that budget, tuning is cheaper by 6.9 per 1,000 e-mails, whi”
- Stage started · AnalysisAttempt 3 · agent:claude-opus-5-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 4 · agent:claude-opus-5-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 7 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 8 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-fable-5-1
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-fable-5-1
- Awaiting approval · Review
- Approved · Review
- Run completereport=/Users/haodingresearch/Desktop/auto-research/runs/studies-opus55/computer_science-en/runs/4689bab459a3/REPORT.md