Auto-Research PipelineA durable, human-gated pipeline for computational research

← All examples

Research run4689bab459a3· 24 October 2026

Under asymmetric misclassification costs, does post-hoc probability calibration with the theoretical cost-optimal threshold beat directly tuning the decision threshold on the same held-out validation data?

CompleteArchived runComputer Science — Machine Learning · Modelled on NeurIPS

Paper

Calibrate or Tune? Cost-Sensitive Decision Thresholds as Estimators on Spambase

Abstract

Under asymmetric costs, held-out data can recalibrate scores for the theoretical threshold c=cFP/(cFP+cFN)c=c_{FP}/(c_{FP}+c_{FN}) or tune the threshold directly. Monotone calibration followed by cc thresholds the raw score, so both recipes estimate one cut. We make explicit that isotonic plug-in attains the minimum empirical validation cost, and contrast tuning's cube-root rate with the parametric rate and bias of Platt's sigmoid. On Spambase (4,210 e-mails; 3 learners × 6 cost proportions × 5 budgets of 50–800 e-mails; 50 seeds), the pre-specified pooled comparison at 100 e-mails is unsupported: Platt minus tuned excess cost is −7.9 points (95% CI −16.5 to 0.6). An absolute-cost endpoint, not pre-specified, favours tuning at that budget by 6.9 per 1,000 e-mails and survives a split-overlap correction from 100 e-mails upward (not at 50), but naive Bayes drives it entirely. At the seed level, Platt beats tuning for logistic regression and gradient boosting and trails for naive Bayes, and isotonic's small-budget edge comes from cut placement; under the correction only the naive Bayes deficit survives, so the rest are point-estimate patterns on this corpus. Winsorising naive Bayes's heavy-tailed log-odds halves its Platt excess, leaving a residual bias.

Keywordscost-sensitive classificationprobability calibrationdecision thresholdisotonic regressionPlatt scalingSpambase

§Ask about this paper

Grounded in this run

Questions are answered from this run's record — the paper, the executed analysis, the data provenance and the referee report. Nothing else is consulted.

This run is finished, so the recorded paper cannot be rewritten. Questions are still answered from its record.

Pipeline

1.Literature review

Attempt 3Done

Literature review complete (live:agent:claude-opus-5-5): 10 key papers, 26 grounded sources, 5 open gaps.

Summary

Decision theory gives a clean prescription for cost-sensitive binary classification. With false-positive cost c_FP and false-negative cost c_FN, predict positive when P(y=1|x) exceeds the cost proportion c = c_FP/(c_FP + c_FN) (Elkan, 2001). For perfectly calibrated scores, this score-driven threshold coincides with the optimal threshold choice (Hernandez-Orallo, Flach and Ferri, 2012, Theorem 43). Practice offers two ways to use a held-out validation sample. The first is to recalibrate the scores post hoc and plug in t = c. The usual calibrators are Platt's two-parameter sigmoid with smoothed targets (Platt, 2000) and nonparametric isotonic regression computed by pool-adjacent-violators (Zadrozny and Elkan, 2002). The second skips calibration and searches for the score threshold that minimises empirical cost (Sheng and Ling, 2006). The empirical record seems to favour search. Sheng and Ling found it beat the theoretical threshold on 9 of 12 datasets, but their probabilities, from C4.5, were never calibrated, so they did not test the comparison posed here. Two further results sharpen the question. First, PAV calibration is equivalent to the ROC convex hull (Fawcett and Niculescu-Mizil, 2007; Berta, Bach and Jordan, 2024). This implies that isotonic calibration plus t = c already selects an empirical-cost-minimising cut of the validation sample. For isotonic, the two recipes should then differ only through tie-breaking and where the cut falls between adjacent scores. Nobody has measured how large that residual is. Second, the remaining contest is between a parametric calibrator and empirical search, and it has a bias-variance structure. A cost-minimising threshold is an argmax-type estimator with cube-root asymptotics (Kim and Pollard, 1990). Its excess cost should decay like m^{-2/3}. A maximum-likelihood sigmoid converges at the parametric rate, but to the threshold implied by its Kullback-Leibler projection. It is therefore biased unless the sigmoid is right at the decision level c. Niculescu-Mizil and Caruana (2005) found Platt scaling better than isotonic regression when the calibration set has fewer than about 200-1000 cases. They measured squared error, not decision cost, and also documented learner-specific distortions: boosting sigmoidal, naive Bayes pushed towards 0 and 1, logistic regression near-calibrated. Recent theory decomposes decision regret into a calibration-induced part that recalibration can remove and a grouping loss it cannot (Perez-Lebel et al., 2025). It does not say how much validation data the recalibration step needs relative to direct tuning. Evaluation practice reports total or normalised cost averaged over repeated random splits, usually as mean and standard deviation. Intervals for cost differences are rare, and the split-overlap correction of Nadeau and Bengio (2003) is seldom applied. For spam filtering, cost-sensitive evaluation is long established (thresholds 0.5, 0.9 and 0.999 in Androutsopoulos et al., 2000). Spambase is the standard benchmark for it. The literature directly comparing the two recipes on a shared, controlled validation budget, with intervals, is thin. I found no such study.

Key papers (10)

  1. The Foundations of Cost-Sensitive Learning. C. Elkan. Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI), pp. 973-978, 2001.
    Source of the theoretical cost-optimal threshold c = c_FP/(c_FP + c_FN) for calibrated probabilities; the 'theory' arm of the question.
  2. Probabilities for SV Machines. J. C. Platt. In A. J. Smola, P. Bartlett, B. Schoelkopf and D. Schuurmans (eds.), Advances in Large-Margin Classifiers, MIT Press, pp. 61-74, 2000.
    Platt (sigmoid) scaling with smoothed targets (N+ + 1)/(N+ + 2) and 1/(N- + 2): the standard two-parameter post-hoc calibrator.
  3. Transforming classifier scores into accurate multiclass probability estimates. B. Zadrozny and C. Elkan. Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 694-699, 2002.
    Isotonic-regression (PAV) calibration of classifier scores: the standard nonparametric calibrator.
  4. Predicting good probabilities with supervised learning. A. Niculescu-Mizil and R. Caruana. Proceedings of the 22nd International Conference on Machine Learning (ICML), pp. 625-632, 2005.
    Reference comparison of Platt and isotonic calibration across learners. Boosting has sigmoidal distortion, naive Bayes is pushed to 0/1, logistic regression is near-calibrated. Platt beats isotonic below about 200-1000 calibration cases, measured by squared error, not decision cost.
  5. PAV and the ROC convex hull. T. Fawcett and A. Niculescu-Mizil. Machine Learning 68(1):97-106, 2007.
    Shows PAV/isotonic calibration is equivalent to the ROC convex hull, which implies that isotonic calibration plus the theoretical threshold selects an empirical-cost-minimising cut.
  6. A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss. J. Hernandez-Orallo, P. Flach and C. Ferri. Journal of Machine Learning Research 13:2813-2869, 2012.
    Framework of threshold choice methods. Theorem 43: for perfectly calibrated models the optimal and score-driven threshold choices coincide at t = c, with PAV as the empirical calibration map.
  7. Thresholding for Making Classifiers Cost-Sensitive. V. S. Sheng and C. X. Ling. AAAI Conference on Artificial Intelligence (AAAI-06), pp. 476-481, 2006.
    The 'direct tuning' arm: choose the threshold minimising empirical misclassification cost. It beat the theoretical threshold on uncalibrated C4.5 probabilities in 9 of 12 datasets.
  8. Cube Root Asymptotics. J. Kim and D. Pollard. The Annals of Statistics 18(1):191-219, 1990.
    Gives n^{-1/3} limit theory for argmax estimators of indicator criteria, which covers the empirical-cost-minimising threshold and hence the slower convergence of direct tuning.
  9. Inference for the Generalization Error. C. Nadeau and Y. Bengio. Machine Learning 52(3):239-281, 2003.
    Corrected resampled t-test: naive intervals over overlapping random splits are too narrow. Used here as a conservative sensitivity check.
  10. Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration. A. Perez-Lebel, G. Varoquaux, S. Koyejo, M. Doutreligne and M. Le Morvan. Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 258:2395-2403, 2025.
    Recent decomposition of decision regret into a calibration-induced part, removable by recalibration, and a grouping loss. Frames what calibration can and cannot buy.

Methods in use

  • Bayes plug-in rule: threshold calibrated probabilities at the cost proportion c = c_FP/(c_FP + c_FN)
  • Platt (sigmoid) scaling fitted by maximum likelihood with smoothed targets on a held-out calibration set
  • Isotonic regression (pool-adjacent-violators) calibration, equivalent to the ROC convex hull
  • Temperature scaling and beta calibration as one- and three-parameter alternatives
  • Empirical threshold search (Thresholding): choose the cut minimising total misclassification cost on held-out or cross-validated predictions, with a tie-breaking rule
  • Evaluation by total or normalised expected misclassification cost and cost curves; Brier score, log-loss and ECE for probability quality
  • Repeated random train/validation/test splits or cross-validation reported as mean and standard deviation; learning curves over calibration-set size; paired comparisons and, rarely, the corrected resampled t-test

Open gaps

  • No controlled head-to-head of 'calibrate, then threshold at c' versus 'tune the threshold' in which both use exactly the same held-out sample at several budgets, with intervals. Sheng and Ling compared search only against uncalibrated probabilities.
  • The PAV-ROCCH equivalence implies that isotonic plug-in and empirical tuning pick the same validation cut up to ties and gap placement. The out-of-sample size of that residual has not been measured.
  • Platt's small-sample advantage over isotonic regression was established for squared error. Whether it holds for cost-weighted decisions, and how it changes with the cost proportion (tails versus centre), is untested.
  • Rates (m^{-1/3} for empirical search versus m^{-1/2} plus a misspecification bias for a sigmoid calibrator) predict a crossover in validation-set size that depends on the learner's miscalibration at the decision level. This has not been measured on real data.
  • How robust conclusions from repeated random splits of one benchmark are to the split-overlap correction of Nadeau and Bengio is rarely reported.
Grounded search sources (26) · mode: live:agent:claude-opus-5-5

2.Ideation

Attempt 2Done

3 candidate ideas generated; selected: “Calibrate-then-threshold versus tune-the-threshold as threshold estimation: isotonic plug-in coincides with tuning, Platt trades variance for bias”.

  1. Idea 1Selected

    Calibrate-then-threshold versus tune-the-threshold as threshold estimation: isotonic plug-in coincides with tuning, Platt trades variance for bias

    Hypothesis. Every monotone post-hoc calibrator followed by the theoretical threshold t = c = c_FP/(c_FP + c_FN) is a rule that thresholds the raw score. The two recipes therefore differ only in how they estimate one cut from the same m held-out points. (H1, primary, directional.) At a small validation budget, m = 100, Platt-calibrated plug-in has lower test misclassification cost than empirical threshold tuning. The pre-registered endpoint is the paired difference in excess cost over the test-optimal threshold (Platt minus Tuned), averaged over 3 base learners and 6 cost proportions c in {0.02, 0.1, 0.25, 0.75, 0.9, 0.98}. Its mean over seeds is predicted to be negative, with a 95% t-interval below 0. (H2, secondary.) The Platt advantage shrinks as the budget grows: D(800) - D(100) > 0 with a 95% interval above 0. This is the prediction of a parametric calibrator with a fixed misspecification bias competing with an argmin estimator whose excess cost decays like m^{-2/3}. (H3, secondary; a check of the PAV-ROC-convex-hull equivalence.) Isotonic plug-in and empirical tuning select validation cuts of identical empirical cost, so on the test set they disagree on fewer than 1% of e-mails on average at every m >= 100.

    Dataset. Spambase e-mail spam classification benchmark uci_spambase

    Test plan, novelty and risk

    Test plan. Data: Spambase, all 57 predictors, transformed by log1p; label is_spam. Exact duplicate rows are removed before splitting so that no e-mail appears in both the validation and the test set; the analysed n is reported. Protocol: 50 seeds. Each seed draws a stratified split into 50% train, 20% validation pool and 30% test. Three base learners are fitted on train: L2 logistic regression (C = 1, standardised features), histogram gradient boosting (scikit-learn defaults) and Gaussian naive Bayes. Each yields a log-odds score z. Validation budgets m in {50, 100, 200, 400, 800} are nested prefixes of a seeded permutation of the pool. There are no redraws, and minimum class counts are reported. Recipes, all fitted on the same m points. Platt: maximum-likelihood sigmoid on z with Platt's smoothed targets, predict spam if sigmoid(a z + b) > c. Isotonic: scikit-learn IsotonicRegression on z with clipping, predict spam if g(z) > c. Tuned: the cut minimising c*FP + (1 - c)*FN over all cuts between distinct validation scores, with ties broken at the median co-optimal cut and the threshold placed at the gap midpoint. References: Raw, the model's own sigmoid(z) > c, which uses no validation data. Oracle, the best threshold on the test set itself. Metric: test cost per e-mail C = (c FP + (1 - c) FN)/n_test, and excess cost E = 100 (C/C_oracle - 1). Estimators: the per-seed paired difference D = mean over the 18 learner-by-c cells of E_Platt - E_Tuned. Report its mean over seeds with a t-based 95% interval and a Wilcoxon signed-rank test. A Nadeau-Bengio corrected interval for split overlap is a sensitivity check. Per-learner and per-c differences get 95% intervals; isotonic-versus-tuned test disagreement is reported as a percentage of e-mails. Diagnostics: test AUROC and the fitted Platt slope at m = 800 per learner. All results as mean +/- s.d. over seeds. Refutation. H1 is refuted if the 95% interval for D(100) lies above 0 (tuning better) and unsupported if it covers 0. H2 is refuted unless D(800) - D(100) has an interval above 0. H3 is refuted if mean disagreement exceeds 1% at any m >= 100.

    Novelty. Sheng and Ling (2006) showed that threshold search beats the theoretical threshold, but only against uncalibrated probabilities. Niculescu-Mizil and Caruana (2005) compared Platt and isotonic calibration on squared error, not decision cost. This study runs the decision-level head-to-head at a shared, controlled validation budget with paired intervals. It makes explicit, and verifies out of sample, the finite-sample consequence of the PAV-ROC-convex-hull equivalence: isotonic plug-in cannot beat tuning because it is tuning. It attributes the remaining contest, Platt versus tuning, to a stated bias-variance mechanism (cube-root versus root-m rates) and tests its budget-dependent prediction.

    Expected outcome. Platt below tuning at m <= 200, with the gap largest at the extreme cost proportions (0.02, 0.98), where few validation e-mails lie near the cut. The advantage narrows by m = 800 and may reverse for naive Bayes, whose sigmoid is likely misspecified at the decision level. Isotonic and tuning agree on more than 99% of test decisions. Raw plug-in is adequate for logistic regression and poor for naive Bayes.

    Main risk. A single, dated, well-separated benchmark, so the results may not transfer. Repeated random splits of one corpus overlap, so seed-level intervals are conditional on Spambase; hence the Nadeau-Bengio check. The test-optimal oracle is itself fitted to the test sample, so excess costs are relative rather than population regrets. Results depend on the three learners' particular miscalibration.

  2. Idea 2

    Does the calibration-versus-tuning crossover move with the base rate? A replication on the imbalanced Adult income benchmark

    Hypothesis. With 24% positives and validation budgets up to 10^4, the budget at which empirical threshold tuning stops losing to Platt-calibrated plug-in is larger than on balanced data, because the minority-class density near an extreme cut is lower. Prediction: at c = 0.9, Platt beats tuning for every m <= 1000 (paired 95% interval below 0).

    Dataset. Adult (Census Income): US income prediction benchmark uci_adult_income

    Test plan, novelty and risk

    Test plan. Adult (48,842 rows) with fnlwgt excluded as a predictor and rows with missing workclass, occupation or native_country dropped (count reported). One-hot categorical features; logistic regression and gradient boosting. 20 seeds of stratified train/validation-pool/test splits; budgets m in {100, 300, 1000, 3000, 10000}; c in {0.1, 0.25, 0.75, 0.9}. Same recipes and excess-cost metric as idea 0, with paired differences and 95% intervals per cell. Refuted if tuning matches or beats Platt at m <= 1000 for c = 0.9.

    Novelty. Tests whether the budget-dependent trade-off depends on class balance, which the literature has not examined for decision cost.

    Expected outcome. A crossover at a larger budget than on balanced data, concentrated at the cost proportion that places the cut in the minority-class tail.

    Main risk. Encoding choices and the heavy-tailed capital_gain variable affect calibration. The fairness-relevant structure of the data is ignored, and the 1994 labour market dates the conclusions.

  3. Idea 3

    Is data reuse worth more than the choice of recipe? Cross-fitted calibration versus a held-out split for cost-sensitive spam filtering

    Hypothesis. At a fixed total sample, fitting the base model on all data and calibrating or tuning from 5-fold out-of-fold scores lowers test cost more than switching between Platt and tuning on a held-out split. Prediction: the cross-fitted-minus-held-out difference is larger in magnitude than the Platt-minus-tuned difference at every c.

    Dataset. Spambase e-mail spam classification benchmark uci_spambase

    Test plan, novelty and risk

    Test plan. Spambase with duplicates removed; logistic regression and gradient boosting; 30 seeds with a 30% test split. Arm A: 50/20 train/validation split with Platt or tuning on the validation part. Arm B: model refitted on the full 70% with Platt or tuning from 5-fold out-of-fold predictions. c in {0.1, 0.25, 0.75, 0.9}; paired differences in test cost with 95% intervals. Refuted if the data-reuse effect is not larger than the recipe effect.

    Novelty. Separates the effect of how much data reaches the calibration step from the effect of which recipe is used, a confound in practical comparisons.

    Expected outcome. Data reuse dominates at small totals; recipe choice matters mainly at extreme c.

    Main risk. Out-of-fold scores come from slightly different models than the refitted one, so cross-fitting is approximate; a single dataset.

Ranking rationale

Idea 0 answers the posed question directly, on the benchmark best suited to it. It separates the case where theory predicts identity (isotonic) from the one where it predicts a trade-off (Platt), and its pre-registered, directional endpoint can come out either way. Idea 1 is a valuable external-validity replication but needs heavier preprocessing, and its answer depends on encoding choices. Idea 2 asks a practical but secondary question about data reuse rather than about the two recipes themselves.

3.Experimentation

Attempt 4Done

Experiment succeeded in 1 round (10.957s).

Reviewer feedback applied: Independent referee (Claude Fable 5.1): major_revision. Required changes: (1) [text] Apply the overlap correction uniformly: multiply every seed-level half-width in Tables 1, 3, 4, 5 and 6 and for H2 by sqrt(1 + 50 x n_test/(n_train + n_pool)) = 4.74 (computable from the reported intervals), report the corrected intervals or a column stating which contrasts survive, and replace the Section 5.2 sentence 'narrow effects such as LR at m=800 may not survive it' with the actual consequence (no LR or GB contrast and no raw-recipe contrast survives; NB survives only from m=200; H2 does not survive). Alternatively, argue explicitly why the Nadeau-Bengio correction overstates the variance of paired differences that share splits, and keep that argument consistent across H1 and the secondary contrasts. (2) [text] Rewrite the abstract, the 'Findings' bullet and the conclusion so that 'Platt leads at every budget for LR and GB', 'logistic regression's own probabilities beat both recipes' and 'the learner sets the sign' are stated as point-estimate patterns that are not robust to the paper's own overlap correction, and hedge the Section 5.3 sentence 'LR's own probabilities beat Platt by 1.7 points and tuning by 3.9' accordingly. (3) [analysis] Record and report the s.d. over seeds for the per-learner Platt, isotonic and tuned excess costs at each m (Table 4) and for the per-cell differences (Table 6), so that every averaged result carries +/- s.d. as the venue requires; remove the note 'per-learner s.d. were not recorded'. (4) [analysis] Report the oracle cost C* (mean +/- s.d. over seeds, per learner x c) and, at least for the primary budget m=100, the pooled absolute cost difference Platt - Tuned with its interval, so the reader can see how the relative endpoint weights cells; state in Section 5.1 that the pooled D weights each cell by 1/C*. (5) [analysis] Test the input-scale attribution for NB directly: fit Platt at the same budgets on a bounded transform of the NB score (e.g. log-odds clipped to a fixed range, or the NB posterior probability) and report its excess cost next to Table 4. If this is not done, delete 'a failure of that input scale rather than of calibrated plug-in' from the abstract and the corresponding sentence in Section 5.4 and the conclusion, and present the attribution as an untested hypothesis. (6) [analysis] Isolate cut placement for the isotonic-versus-tuned gap: run tuning with the cut placed at the isotonic interpolation point (fraction (c - v_lo)/(v_hi - v_lo) across the gap), or isotonic with midpoint placement, and add the row to Table 5. If this is not done, replace the 'likely, unablated mechanism' paragraph with a plain statement that isotonic plug-in beat midpoint tuning at m <= 100 for reasons not isolated here. (7) [text] Replace 'pre-registered' throughout with 'pre-specified in the archived run record' unless a public, time-stamped registration can be cited; report the H1 verdict in the plan's own term ('unsupported': interval covers zero) or explain why 'inconclusive' is used instead. (8) [text] Cite, from the retrieved sources, Kull et al. (2017) beta calibration in Section 2 and in the NB discussion of Section 5.4 as the established remedy for sigmoid misfit; Kumar, Liang and Ma (2019) when contrasting parametric and non-parametric calibrators' sample efficiency; Lin, Lin and Weng (2007) where the Platt fit is described; Boeken (2021); Yang and Bi (2025); Provost and Fawcett (2001) next to the ROC-convex-hull discussion; and Androutsopoulos et al. (2000) to motivate the extreme cost proportions for spam. (9) [text] State the exact versions of Python, NumPy, SciPy and scikit-learn and the CPU used in the reproducibility appendix, noting that HistGradientBoostingClassifier and LogisticRegression defaults depend on the scikit-learn version; add the NeurIPS paper checklist. (10) [text] In Proposition 2(b), state the regularity needed (a-dagger bounded away from zero and finite moments of the fitted cut) and say in Section 5.4 that NB's slope collapse is a violation of these conditions, so Proposition 2(b) does not describe the NB case; in Section 5.1 include Table 3 (m != 100) and Table 5 among the uncorrected secondary contrasts; and add one figure of excess cost versus m per learner with intervals, drawn from the recorded numbers.
Succeeded1 round · exit 0 · 10.957s

Data provenance

Datasetuci_spambase

Spambase e-mail spam classification benchmark

Source
UCI Machine Learning Repository (dataset 94), donated by Hewlett-Packard Labs
Citation
Hopkins, M., Reeber, E., Forman, G. and Suermondt, J. (1999) Spambase [Dataset]. UCI Machine Learning Repository. DOI: 10.24432/C53G6X. Collected at Hewlett-Packard Labs, Palo Alto, and donated 30 June 1999. No peer-reviewed introductory paper exists for this dataset -- the UCI page refers only to an internal HP technical report -- so the dataset itself is the citable primary source.
Licence
Creative Commons Attribution 4.0 International (CC BY 4.0): the UCI dataset page states "This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given."
File
https://archive.ics.uci.edu/static/public/94/spambase.zip
Observations
4,601
Variables (58)
word_freq_make, word_freq_address, word_freq_all, word_freq_3d, word_freq_our, word_freq_over, word_freq_remove, word_freq_internet, word_freq_order, word_freq_mail, word_freq_receive, word_freq_will, word_freq_people, word_freq_report, word_freq_addresses, word_freq_free, word_freq_business, word_freq_email, word_freq_you, word_freq_credit, word_freq_your, word_freq_font, word_freq_000, word_freq_money, word_freq_hp, word_freq_hpl, word_freq_george, word_freq_650, word_freq_lab, word_freq_labs, word_freq_telnet, word_freq_857, word_freq_data, word_freq_415, word_freq_85, word_freq_technology, word_freq_1999, word_freq_parts, word_freq_pm, word_freq_direct, word_freq_cs, word_freq_meeting, word_freq_original, word_freq_project, word_freq_re, word_freq_edu, word_freq_table, word_freq_conference, char_freq_;, char_freq_(, char_freq_[, char_freq_!, char_freq_$, char_freq_#, capital_run_length_average, capital_run_length_longest, capital_run_length_total, is_spam
Units
The 48 word_freq_* columns are the percentage of words in the e-mail matching that token, and the 6 char_freq_* columns the percentage of characters matching that character, so both are on a 0-100 scale. capital_run_length_average is the mean length of uninterrupted runs of capitals, capital_run_length_longest the longest such run, and capital_run_length_total the total number of capital letters. is_spam is 1 for spam and 0 for legitimate mail. Column names are taken from the archive's spambase.names file by the loader.
Retrieved
24 Oct 2026, 04:14
Checksum
sha256 813ac1df8effac70…

Real, publicly available data — downloaded and checksum-verified before the analysis ran, never simulated.

Result (RESULT_JSON)

{
  "n_loaded": 4601,
  "n_duplicates": 391,
  "n": 4210,
  "n_spam": 1679,
  "n_train_pool_test": [
    2105,
    842,
    1263
  ],
  "seeds": 50,
  "m": [
    50,
    100,
    200,
    400,
    800
  ],
  "c": [
    0.02,
    0.1,
    0.25,
    0.75,
    0.9,
    0.98
  ],
  "min_spam_ham_in_val": [
    15,
    24
  ],
  "learners": {
    "LR": {
      "auroc": [
        0.978,
        0.003
      ],
      "slope800_mean_sd_med": [
        0.856,
        0.099,
        0.857
      ],
      "max_tie_frac": 0.002,
      "z_q01_q50_q99": [
        -25.3,
        -2.04,
        11.4
      ],
      "raw_excess": [
        12.43,
        3.653
      ],
      "excess_platt_iso_tuned": [
        [
          41.204,
          81.712,
          99.063
        ],
        [
          24.867,
          50.555,
          55.051
        ],
        [
          18.114,
          29.909,
          30.562
        ],
        [
          15.501,
          20.974,
          20.765
        ],
        [
          14.143,
          16.348,
          16.316
        ]
      ],
      "D_by_m": [
        [
          -57.859,
          -77.542,
          -38.176
        ],
        [
          -30.183,
          -39.811,
          -20.556
        ],
        [
          -12.449,
          -17.617,
          -7.281
        ],
        [
          -5.263,
          -8.177,
          -2.35
        ],
        [
          -2.173,
          -3.988,
          -0.358
        ]
      ],
      "raw_minus_platt_tuned_800": [
        [
          -1.713,
          -3.219,
          -0.206
        ],
        [
          -3.886,
          -5.883,
          -1.889
        ]
      ]
    },
    "GB": {
      "auroc": [
        0.984,
        0.003
      ],
      "slope800_mean_sd_med": [
        0.649,
        0.061,
        0.639
      ],
      "max_tie_frac": 0.044,
      "z_q01_q50_q99": [
        -10.3,
        -3.92,
        8.9
      ],
      "raw_excess": [
        26.161,
        9.26
      ],
      "excess_platt_iso_tuned": [
        [
          39.284,
          81.229,
          114.128
        ],
        [
          27.705,
          63.628,
          71.645
        ],
        [
          19.482,
          35.686,
          38.286
        ],
        [
          15.606,
          23.782,
          24.214
        ],
        [
          13.934,
          17.562,
          17.769
        ]
      ],
      "D_by_m": [
        [
          -74.844,
          -97.539,
          -52.148
        ],
        [
          -43.94,
          -58.286,
          -29.595
        ],
        [
          -18.804,
          -25.556,
          -12.053
        ],
        [
          -8.608,
          -12.032,
          -5.185
        ],
        [
          -3.835,
          -6.178,
          -1.492
        ]
      ],
      "raw_minus_platt_tuned_800": [
        [
          12.227,
          9.208,
          15.246
        ],
        [
          8.392,
          5.165,
          11.619
        ]
      ]
    },
    "NB": {
      "auroc": [
        0.955,
        0.005
      ],
      "slope800_mean_sd_med": [
        0.003,
        0.005,
        1.38e-7
      ],
      "max_tie_frac": 0.002,
      "z_q01_q50_q99": [
        -175000000,
        15.2,
        626
      ],
      "raw_excess": [
        462.254,
        38.857
      ],
      "excess_platt_iso_tuned": [
        [
          98.978,
          64.176,
          73.869
        ],
        [
          95.228,
          42.57,
          44.885
        ],
        [
          92.791,
          26.34,
          27.234
        ],
        [
          90.476,
          17.307,
          17.546
        ],
        [
          88.411,
          12.508,
          12.451
        ]
      ],
      "D_by_m": [
        [
          25.109,
          11.73,
          38.488
        ],
        [
          50.343,
          39.573,
          61.113
        ],
        [
          65.557,
          55.348,
          75.766
        ],
        [
          72.93,
          63.825,
          82.035
        ],
        [
          75.96,
          66.939,
          84.98
        ]
      ],
      "raw_minus_platt_tuned_800": [
        [
          373.843,
          360.154,
          387.532
        ],
        [
          449.803,
          438.579,
          461.027
        ]
      ]
    }
  },
  "by_m": {
    "50": {
      "platt_iso_tuned": [
        [
          59.822,
          15.224
        ],
        [
          75.706,
          32.585
        ],
        [
          95.687,
          43.339
        ]
      ],
      "D": [
        -35.865,
        -50.37,
        -21.359
      ],
      "wilcoxon_p": 0.0000188,
      "platt_win_rate": 0.72,
      "D_iso": [
        -19.981,
        -25.614,
        -14.348
      ],
      "disagree_pct": [
        1.248,
        0.802
      ],
      "same_val_partition": 0.968
    },
    "100": {
      "platt_iso_tuned": [
        [
          49.267,
          12.144
        ],
        [
          52.251,
          21.885
        ],
        [
          57.194,
          25.003
        ]
      ],
      "D": [
        -7.927,
        -16.46,
        0.607
      ],
      "wilcoxon_p": 0.121,
      "platt_win_rate": 0.56,
      "D_iso": [
        -4.943,
        -6.412,
        -3.474
      ],
      "disagree_pct": [
        0.603,
        0.353
      ],
      "same_val_partition": 0.956
    },
    "200": {
      "platt_iso_tuned": [
        [
          43.462,
          11.132
        ],
        [
          30.645,
          12.326
        ],
        [
          32.027,
          13.272
        ]
      ],
      "D": [
        11.435,
        6.136,
        16.733
      ],
      "wilcoxon_p": 0.00000937,
      "platt_win_rate": 0.28,
      "D_iso": [
        -1.382,
        -2.002,
        -0.762
      ],
      "disagree_pct": [
        0.375,
        0.358
      ],
      "same_val_partition": 0.933
    },
    "400": {
      "platt_iso_tuned": [
        [
          40.528,
          10.64
        ],
        [
          20.688,
          5.343
        ],
        [
          20.842,
          5.667
        ]
      ],
      "D": [
        19.686,
        15.963,
        23.41
      ],
      "wilcoxon_p": 2.02e-11,
      "platt_win_rate": 0.12,
      "D_iso": [
        -0.154,
        -0.474,
        0.166
      ],
      "disagree_pct": [
        0.308,
        0.243
      ],
      "same_val_partition": 0.899
    },
    "800": {
      "platt_iso_tuned": [
        [
          38.829,
          11.146
        ],
        [
          15.472,
          3.763
        ],
        [
          15.512,
          3.657
        ]
      ],
      "D": [
        23.317,
        19.886,
        26.748
      ],
      "wilcoxon_p": 3e-13,
      "platt_win_rate": 0.08,
      "D_iso": [
        -0.04,
        -0.322,
        0.242
      ],
      "disagree_pct": [
        0.19,
        0.152
      ],
      "same_val_partition": 0.896
    }
  },
  "cell_D": {
    "100": {
      "LR": [
        [
          -88.069,
          -123.309,
          -52.828
        ],
        [
          -1.123,
          -11.835,
          9.59
        ],
        [
          2.328,
          -3.623,
          8.28
        ],
        [
          11.48,
          3.998,
          18.963
        ],
        [
          -3.675,
          -13.596,
          6.247
        ],
        [
          -102.043,
          -137.779,
          -66.306
        ]
      ],
      "GB": [
        [
          -107.249,
          -160.364,
          -54.134
        ],
        [
          -18.662,
          -30.515,
          -6.809
        ],
        [
          -11.649,
          -19.087,
          -4.211
        ],
        [
          -3.537,
          -8.649,
          1.576
        ],
        [
          -17.616,
          -31.533,
          -3.7
        ],
        [
          -104.929,
          -157.731,
          -52.128
        ]
      ],
      "NB": [
        [
          -62.968,
          -92.29,
          -33.646
        ],
        [
          58.845,
          50.354,
          67.337
        ],
        [
          191.49,
          174.739,
          208.241
        ],
        [
          169.41,
          159.974,
          178.847
        ],
        [
          43.023,
          34.597,
          51.448
        ],
        [
          -97.742,
          -134.815,
          -60.67
        ]
      ]
    },
    "800": {
      "LR": [
        [
          -10,
          -17.191,
          -2.809
        ],
        [
          1.253,
          -2.269,
          4.775
        ],
        [
          -0.228,
          -2.087,
          1.631
        ],
        [
          1.133,
          -1.894,
          4.161
        ],
        [
          0.992,
          -4.05,
          6.034
        ],
        [
          -6.188,
          -11.801,
          -0.575
        ]
      ],
      "GB": [
        [
          -6.177,
          -12.922,
          0.568
        ],
        [
          -3.521,
          -6.117,
          -0.925
        ],
        [
          -1.977,
          -3.878,
          -0.076
        ],
        [
          -2.5,
          -5.186,
          0.186
        ],
        [
          -4.893,
          -8.922,
          -0.865
        ],
        [
          -3.943,
          -14.165,
          6.28
        ]
      ],
      "NB": [
        [
          7.527,
          0.146,
          14.909
        ],
        [
          60.643,
          50.75,
          70.536
        ],
        [
          187.724,
          165.398,
          210.051
        ],
        [
          159.829,
          145.3,
          174.359
        ],
        [
          56.221,
          49.828,
          62.613
        ],
        [
          -16.187,
          -22.767,
          -9.607
        ]
      ]
    }
  },
  "val_cost_gap_iso_tuned": 6.94e-18,
  "primary_D100": [
    -7.927,
    -16.46,
    0.607
  ],
  "nadeau_bengio_ci": [
    -48.341,
    32.487
  ],
  "h2_D800_minus_D100": [
    31.244,
    23.156,
    39.332
  ],
  "h3_max_disagree_pct": 0.603,
  "hypothesis_supported": false,
  "revision": {
    "nb_factor_on_halfwidth": 4.736,
    "halfwidth_per_sd": {
      "seed": 0.284,
      "nb": 1.346
    },
    "h1_D100_sd_nb": [
      30.027,
      -48.341,
      32.487
    ],
    "h2_sd_nb": [
      28.46,
      -7.061,
      69.549
    ],
    "learners": {
      "LR": {
        "excess_sd_platt_iso_tuned": [
          [
            17.415,
            46.891,
            58.72
          ],
          [
            11.169,
            25.335,
            27.95
          ],
          [
            7.243,
            15.099,
            15.426
          ],
          [
            5.382,
            7.801,
            8.053
          ],
          [
            4.584,
            6.329,
            6.462
          ]
        ],
        "D_by_m_sd_nb": [
          [
            69.259,
            -151.076,
            35.358
          ],
          [
            33.877,
            -75.779,
            15.413
          ],
          [
            18.185,
            -36.925,
            12.027
          ],
          [
            10.252,
            -19.061,
            8.535
          ],
          [
            6.388,
            -10.77,
            6.424
          ]
        ],
        "raw_minus_platt_tuned_800_sd_nb": [
          [
            5.3,
            -8.846,
            5.421
          ],
          [
            7.028,
            -13.345,
            5.573
          ]
        ],
        "oracle_cost_per1000_by_c": [
          [
            6.405,
            0.897
          ],
          [
            18.352,
            1.912
          ],
          [
            27.094,
            2.987
          ],
          [
            25.851,
            2.865
          ],
          [
            16.884,
            2.509
          ],
          [
            5.741,
            0.861
          ]
        ],
        "abs_D_per1000_by_m": [
          [
            -2.516,
            -4.421,
            -0.611,
            -11.538,
            6.506
          ],
          [
            -1.585,
            -2.559,
            -0.61,
            -6.199,
            3.03
          ],
          [
            -0.603,
            -1.112,
            -0.093,
            -3.016,
            1.81
          ],
          [
            -0.285,
            -0.635,
            0.066,
            -1.944,
            1.375
          ],
          [
            -0.099,
            -0.357,
            0.158,
            -1.32,
            1.121
          ]
        ],
        "platt_prob_excess": [
          [
            33.253,
            13.943
          ],
          [
            28.96,
            10.22
          ],
          [
            26.774,
            7.481
          ],
          [
            27.332,
            6.954
          ],
          [
            27.893,
            6.908
          ]
        ],
        "platt_prob_minus_tuned": [
          [
            -65.81,
            -83.824,
            -47.797,
            -151.119,
            19.498
          ],
          [
            -26.091,
            -34.39,
            -17.791,
            -65.396,
            13.215
          ],
          [
            -3.789,
            -8.6,
            1.022,
            -26.573,
            18.995
          ],
          [
            6.568,
            3.474,
            9.662,
            -8.084,
            21.219
          ],
          [
            11.577,
            9.092,
            14.063,
            -0.194,
            23.348
          ]
        ],
        "platt_wins_excess": [
          [
            41.124,
            17.318
          ],
          [
            24.798,
            11.149
          ],
          [
            18.056,
            7.248
          ],
          [
            15.496,
            5.388
          ],
          [
            14.129,
            4.599
          ]
        ],
        "platt_wins_minus_tuned": [
          [
            -57.939,
            -77.607,
            -38.271,
            -151.085,
            35.207
          ],
          [
            -30.252,
            -39.881,
            -20.624,
            -75.853,
            15.348
          ],
          [
            -12.507,
            -17.678,
            -7.335,
            -36.998,
            11.985
          ],
          [
            -5.269,
            -8.186,
            -2.352,
            -19.083,
            8.546
          ],
          [
            -2.187,
            -4.007,
            -0.368,
            -10.805,
            6.43
          ]
        ],
        "platt_minus_platt_prob_wins": [
          [
            [
              7.951,
              4.071,
              11.831,
              -10.425,
              26.327
            ],
            [
              0.08,
              -0.005,
              0.165,
              -0.322,
              0.481
            ]
          ],
          [
            [
              -4.093,
              -7.113,
              -1.072,
              -18.397,
              10.212
            ],
            [
              0.069,
              0.026,
              0.112,
              -0.133,
              0.272
            ]
          ],
          [
            [
              -8.66,
              -11.139,
              -6.181,
              -20.398,
              3.078
            ],
            [
              0.058,
              0.01,
              0.106,
              -0.168,
              0.284
            ]
          ],
          [
            [
              -11.831,
              -13.918,
              -9.744,
              -21.714,
              -1.948
            ],
            [
              0.006,
              -0.011,
              0.022,
              -0.072,
              0.084
            ]
          ],
          [
            [
              -13.75,
              -15.801,
              -11.699,
              -23.465,
              -4.035
            ],
            [
              0.014,
              -0.005,
              0.033,
              -0.076,
              0.104
            ]
          ]
        ],
        "slope800_median_prob_wins": [
          7.26,
          0.857
        ]
      },
      "GB": {
        "excess_sd_platt_iso_tuned": [
          [
            21.075,
            54.133,
            72.238
          ],
          [
            15.129,
            40.437,
            46.887
          ],
          [
            9.129,
            21.046,
            22.437
          ],
          [
            7.044,
            10.21,
            10.93
          ],
          [
            6.415,
            6.749,
            6.912
          ]
        ],
        "D_by_m_sd_nb": [
          [
            79.858,
            -182.326,
            32.639
          ],
          [
            50.478,
            -111.879,
            23.999
          ],
          [
            23.756,
            -50.778,
            13.17
          ],
          [
            12.045,
            -24.82,
            7.603
          ],
          [
            8.244,
            -14.931,
            7.261
          ]
        ],
        "raw_minus_platt_tuned_800_sd_nb": [
          [
            10.623,
            -2.07,
            26.524
          ],
          [
            11.355,
            -6.891,
            23.675
          ]
        ],
        "oracle_cost_per1000_by_c": [
          [
            5.657,
            1.166
          ],
          [
            15.259,
            2.198
          ],
          [
            22.692,
            2.969
          ],
          [
            20.455,
            2.412
          ],
          [
            12.423,
            1.944
          ],
          [
            4.686,
            1.209
          ]
        ],
        "abs_D_per1000_by_m": [
          [
            -5.364,
            -6.985,
            -3.744,
            -13.038,
            2.309
          ],
          [
            -3.241,
            -4.294,
            -2.188,
            -8.227,
            1.745
          ],
          [
            -1.588,
            -2.027,
            -1.15,
            -3.665,
            0.488
          ],
          [
            -0.834,
            -1.078,
            -0.59,
            -1.991,
            0.323
          ],
          [
            -0.483,
            -0.652,
            -0.314,
            -1.285,
            0.319
          ]
        ],
        "platt_prob_excess": [
          [
            54.036,
            26.192
          ],
          [
            50.63,
            18.896
          ],
          [
            46.572,
            16.316
          ],
          [
            48.14,
            16.74
          ],
          [
            49.601,
            17.466
          ]
        ],
        "platt_prob_minus_tuned": [
          [
            -60.092,
            -83.068,
            -37.116,
            -168.904,
            48.719
          ],
          [
            -21.015,
            -34.049,
            -7.981,
            -82.743,
            40.712
          ],
          [
            8.286,
            2.084,
            14.489,
            -21.087,
            37.66
          ],
          [
            23.926,
            19.601,
            28.25,
            3.443,
            44.408
          ],
          [
            31.832,
            26.493,
            37.171,
            6.546,
            57.118
          ]
        ],
        "platt_wins_excess": [
          [
            39.284,
            21.075
          ],
          [
            27.705,
            15.129
          ],
          [
            19.482,
            9.129
          ],
          [
            15.606,
            7.044
          ],
          [
            13.934,
            6.415
          ]
        ],
        "platt_wins_minus_tuned": [
          [
            -74.844,
            -97.539,
            -52.148,
            -182.326,
            32.639
          ],
          [
            -43.94,
            -58.286,
            -29.595,
            -111.879,
            23.999
          ],
          [
            -18.804,
            -25.556,
            -12.053,
            -50.778,
            13.17
          ],
          [
            -8.608,
            -12.032,
            -5.185,
            -24.82,
            7.603
          ],
          [
            -3.835,
            -6.178,
            -1.492,
            -14.931,
            7.261
          ]
        ],
        "platt_minus_platt_prob_wins": [
          [
            [
              -14.751,
              -19.227,
              -10.275,
              -35.949,
              6.447
            ],
            [
              0,
              0,
              0,
              0,
              0
            ]
          ],
          [
            [
              -22.925,
              -27.189,
              -18.661,
              -43.118,
              -2.732
            ],
            [
              0,
              0,
              0,
              0,
              0
            ]
          ],
          [
            [
              -27.091,
              -30.715,
              -23.467,
              -44.253,
              -9.928
            ],
            [
              0,
              0,
              0,
              0,
              0
            ]
          ],
          [
            [
              -32.534,
              -36.206,
              -28.862,
              -49.925,
              -15.142
            ],
            [
              0,
              0,
              0,
              0,
              0
            ]
          ],
          [
            [
              -35.667,
              -39.457,
              -31.878,
              -53.613,
              -17.722
            ],
            [
              0,
              0,
              0,
              0,
              0
            ]
          ]
        ],
        "slope800_median_prob_wins": [
          6.8,
          0.639
        ]
      },
      "NB": {
        "excess_sd_platt_iso_tuned": [
          [
            16.782,
            34.329,
            40.825
          ],
          [
            19.42,
            26.905,
            28.459
          ],
          [
            23.858,
            18.95,
            19.278
          ],
          [
            27.203,
            9.671,
            9.663
          ],
          [
            29.49,
            5.387,
            5.414
          ]
        ],
        "D_by_m_sd_nb": [
          [
            47.077,
            -38.253,
            88.471
          ],
          [
            37.898,
            -0.664,
            101.35
          ],
          [
            35.923,
            17.207,
            113.907
          ],
          [
            32.037,
            29.811,
            116.049
          ],
          [
            31.74,
            33.24,
            118.679
          ]
        ],
        "raw_minus_platt_tuned_800_sd_nb": [
          [
            48.167,
            309.014,
            438.672
          ],
          [
            39.495,
            396.646,
            502.96
          ]
        ],
        "oracle_cost_per1000_by_c": [
          [
            8.956,
            1.015
          ],
          [
            30.318,
            2.202
          ],
          [
            41.809,
            4.013
          ],
          [
            34.913,
            3.456
          ],
          [
            23.701,
            2.336
          ],
          [
            7.289,
            0.501
          ]
        ],
        "abs_D_per1000_by_m": [
          [
            23.38,
            21.473,
            25.286,
            14.351,
            32.408
          ],
          [
            25.426,
            23.401,
            27.451,
            15.837,
            35.015
          ],
          [
            26.549,
            24.132,
            28.966,
            15.101,
            37.997
          ],
          [
            26.938,
            24.341,
            29.535,
            14.64,
            39.236
          ],
          [
            27.156,
            24.313,
            29.999,
            13.692,
            40.62
          ]
        ],
        "platt_prob_excess": [
          [
            62.315,
            9.805
          ],
          [
            62.959,
            14.872
          ],
          [
            60.861,
            12.555
          ],
          [
            58.638,
            10.085
          ],
          [
            57.536,
            9.429
          ]
        ],
        "platt_prob_minus_tuned": [
          [
            -11.554,
            -24.138,
            1.031,
            -71.154,
            48.047
          ],
          [
            18.074,
            9.963,
            26.185,
            -20.338,
            56.486
          ],
          [
            33.626,
            28.196,
            39.057,
            7.907,
            59.346
          ],
          [
            41.092,
            37.432,
            44.752,
            23.759,
            58.426
          ],
          [
            45.085,
            41.676,
            48.494,
            28.939,
            61.231
          ]
        ],
        "platt_wins_excess": [
          [
            55.981,
            20.306
          ],
          [
            43.807,
            14.936
          ],
          [
            40.119,
            16.483
          ],
          [
            39.653,
            15.898
          ],
          [
            37.736,
            12.887
          ]
        ],
        "platt_wins_minus_tuned": [
          [
            -17.888,
            -31.982,
            -3.794,
            -84.635,
            48.86
          ],
          [
            -1.078,
            -11.035,
            8.879,
            -48.233,
            46.078
          ],
          [
            12.885,
            4.823,
            20.946,
            -25.294,
            51.063
          ],
          [
            22.107,
            16.107,
            28.106,
            -6.307,
            50.52
          ],
          [
            25.284,
            21.036,
            29.532,
            5.167,
            45.402
          ]
        ],
        "platt_minus_platt_prob_wins": [
          [
            [
              36.663,
              32.643,
              40.682,
              17.628,
              55.697
            ],
            [
              42.997,
              37.664,
              48.329,
              17.742,
              68.251
            ]
          ],
          [
            [
              32.269,
              26.099,
              38.439,
              3.048,
              61.491
            ],
            [
              51.421,
              45.737,
              57.104,
              24.503,
              78.338
            ]
          ],
          [
            [
              31.931,
              24.465,
              39.396,
              -3.425,
              67.286
            ],
            [
              52.673,
              46.211,
              59.134,
              22.071,
              83.274
            ]
          ],
          [
            [
              31.838,
              24.043,
              39.633,
              -5.078,
              68.754
            ],
            [
              50.824,
              43.283,
              58.364,
              15.114,
              86.533
            ]
          ],
          [
            [
              30.875,
              22.653,
              39.097,
              -8.064,
              69.814
            ],
            [
              50.675,
              42.604,
              58.747,
              12.45,
              88.901
            ]
          ]
        ],
        "slope800_median_prob_wins": [
          4.1,
          0.013
        ]
      }
    },
    "by_m": {
      "50": {
        "D_sd_nb": [
          51.04,
          -104.561,
          32.832
        ],
        "D_iso_sd_nb": [
          19.821,
          -46.658,
          6.697
        ],
        "disagree_nb": [
          0.169,
          2.327
        ],
        "isomid_excess": [
          96.27,
          43.048
        ],
        "iso_minus_isomid": [
          -20.564,
          -26.089,
          -15.04,
          -46.727,
          5.599
        ],
        "isomid_minus_tuned": [
          0.583,
          -0.313,
          1.48,
          -3.663,
          4.829
        ],
        "abs_D_per1000": [
          5.166,
          3.864,
          6.469,
          -1.004,
          11.337
        ]
      },
      "100": {
        "D_sd_nb": [
          30.027,
          -48.341,
          32.487
        ],
        "D_iso_sd_nb": [
          5.169,
          -11.9,
          2.014
        ],
        "disagree_nb": [
          0.127,
          1.078
        ],
        "isomid_excess": [
          57.528,
          24.914
        ],
        "iso_minus_isomid": [
          -5.277,
          -6.633,
          -3.92,
          -11.701,
          1.148
        ],
        "isomid_minus_tuned": [
          0.334,
          -0.167,
          0.835,
          -2.037,
          2.705
        ],
        "abs_D_per1000": [
          6.867,
          5.942,
          7.792,
          2.485,
          11.248
        ]
      },
      "200": {
        "D_sd_nb": [
          18.644,
          -13.658,
          36.528
        ],
        "D_iso_sd_nb": [
          2.182,
          -4.319,
          1.555
        ],
        "disagree_nb": [
          -0.107,
          0.857
        ],
        "isomid_excess": [
          31.985,
          13.403
        ],
        "iso_minus_isomid": [
          -1.34,
          -1.841,
          -0.838,
          -3.715,
          1.036
        ],
        "isomid_minus_tuned": [
          -0.043,
          -0.446,
          0.361,
          -1.952,
          1.867
        ],
        "abs_D_per1000": [
          8.119,
          7.217,
          9.022,
          3.844,
          12.394
        ]
      },
      "400": {
        "D_sd_nb": [
          13.102,
          2.052,
          37.32
        ],
        "D_iso_sd_nb": [
          1.127,
          -1.671,
          1.363
        ],
        "disagree_nb": [
          -0.019,
          0.635
        ],
        "isomid_excess": [
          20.894,
          5.563
        ],
        "iso_minus_isomid": [
          -0.206,
          -0.379,
          -0.034,
          -1.023,
          0.61
        ],
        "isomid_minus_tuned": [
          0.053,
          -0.238,
          0.343,
          -1.325,
          1.43
        ],
        "abs_D_per1000": [
          8.606,
          7.7,
          9.512,
          4.315,
          12.898
        ]
      },
      "800": {
        "D_sd_nb": [
          12.073,
          7.068,
          39.566
        ],
        "D_iso_sd_nb": [
          0.992,
          -1.375,
          1.296
        ],
        "disagree_nb": [
          -0.015,
          0.395
        ],
        "isomid_excess": [
          15.464,
          3.725
        ],
        "iso_minus_isomid": [
          0.009,
          -0.077,
          0.095,
          -0.398,
          0.416
        ],
        "isomid_minus_tuned": [
          -0.049,
          -0.314,
          0.217,
          -1.306,
          1.209
        ],
        "abs_D_per1000": [
          8.858,
          7.888,
          9.827,
          4.266,
          13.449
        ]
      }
    },
    "cell_D_sd_nb": {
      "100": {
        "LR": [
          [
            124,
            -254.963,
            78.826
          ],
          [
            37.694,
            -51.855,
            49.61
          ],
          [
            20.942,
            -25.857,
            30.514
          ],
          [
            26.329,
            -23.956,
            46.917
          ],
          [
            34.911,
            -50.662,
            43.312
          ],
          [
            125.744,
            -271.285,
            67.2
          ]
        ],
        "GB": [
          [
            186.897,
            -358.797,
            144.299
          ],
          [
            41.707,
            -74.796,
            37.472
          ],
          [
            26.172,
            -46.874,
            23.577
          ],
          [
            17.989,
            -27.748,
            20.675
          ],
          [
            48.968,
            -83.523,
            48.291
          ],
          [
            185.791,
            -354.99,
            145.131
          ]
        ],
        "NB": [
          [
            103.176,
            -201.835,
            75.899
          ],
          [
            29.879,
            18.631,
            99.06
          ],
          [
            58.941,
            112.161,
            270.82
          ],
          [
            33.203,
            124.721,
            214.099
          ],
          [
            29.648,
            3.119,
            82.926
          ],
          [
            130.447,
            -273.314,
            77.829
          ]
        ]
      },
      "800": {
        "LR": [
          [
            25.303,
            -44.055,
            24.056
          ],
          [
            12.392,
            -15.426,
            17.932
          ],
          [
            6.542,
            -9.033,
            8.577
          ],
          [
            10.653,
            -13.205,
            15.471
          ],
          [
            17.742,
            -22.887,
            24.87
          ],
          [
            19.752,
            -32.772,
            20.396
          ]
        ],
        "GB": [
          [
            23.734,
            -38.122,
            25.768
          ],
          [
            9.134,
            -15.815,
            8.772
          ],
          [
            6.69,
            -10.981,
            7.028
          ],
          [
            9.452,
            -15.222,
            10.222
          ],
          [
            14.176,
            -23.973,
            14.187
          ],
          [
            35.969,
            -52.354,
            44.469
          ]
        ],
        "NB": [
          [
            25.974,
            -27.432,
            42.486
          ],
          [
            34.809,
            13.792,
            107.493
          ],
          [
            78.559,
            81.99,
            293.459
          ],
          [
            51.124,
            91.02,
            228.638
          ],
          [
            22.493,
            25.947,
            86.495
          ],
          [
            23.153,
            -47.349,
            14.976
          ]
        ]
      }
    },
    "env": {
      "python": "3.12.13",
      "numpy": "2.5.2",
      "scipy": "1.18.1",
      "sklearn": "1.9.0",
      "pandas": "3.0.5",
      "platform": "macOS-27.0.1-arm64-arm-64bit",
      "cpu": "Apple M4",
      "cpu_count": 10
    }
  },
  "runtime_s": 10.1
}
Experiment script · experiment_v1.py
import ctypes, json, os, platform, time
import numpy as np, pandas as pd, scipy, sklearn
from scipy import stats
from scipy.special import expit
from sklearn.model_selection import train_test_split as tts
from sklearn.linear_model import LogisticRegression as LogReg
from sklearn.ensemble import HistGradientBoostingClassifier as HGB
from sklearn.naive_bayes import GaussianNB
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.isotonic import IsotonicRegression as Iso
from sklearn.metrics import roc_auc_score
from dataset import load

# Revision (round 1): every quantity of the executed v1 script is computed exactly as before (same seeds, same
# order of random draws). Additions use no new random numbers: per-learner and per-cell s.d. over seeds,
# overlap-corrected (Nadeau-Bengio) intervals for every contrast, oracle costs C*, absolute cost differences,
# isotonic with midpoint placement (cut-placement ablation), Platt on two bounded inputs, software and CPU.
T0 = time.time(); df = load()
print("loaded", df.shape, "missing", df.isna().sum().sum(), "non-numeric", (df.dtypes == object).sum())
n0 = len(df); df = df.drop_duplicates().reset_index(drop=True)  # no duplicate e-mail across splits
X = np.log1p(df.drop(columns="is_spam").to_numpy(float)); y = df["is_spam"].to_numpy(int)
n = len(y); print("analysed", n, "after dropping", n0 - n, "duplicates; spam", y.sum())
R, MS, CS = 50, [50, 100, 200, 400, 800], np.array([.02, .1, .25, .75, .9, .98]); LG = np.log(CS / (1 - CS))

def score(f, A):  # log-odds score z
    if isinstance(f, GaussianNB): j = f.predict_joint_log_proba(A); return j[:, 1] - j[:, 0]
    return f.decision_function(A)

def platt(z, t):  # unpenalised MLE sigmoid(a z+b), Platt targets
    p = np.where(t == 1, (t.sum() + 1) / (t.sum() + 2), 1 / (len(t) - t.sum() + 2)); s = stats.iqr(z) + 1e-12
    f = LogReg(C=1e12, max_iter=5000).fit(np.r_[z, z][:, None] / s, np.r_[t * 0 + 1, t * 0], sample_weight=np.r_[p, 1 - p])
    return f.coef_[0, 0] / s, f.intercept_[0]

def cuts(z, t):  # cost c*FP+(1-c)*FN of top-k cuts, k=0..m
    o = np.argsort(-z, kind="mergesort"); s, u = z[o], t[o]; tp, fp = np.r_[0, u.cumsum()], np.r_[0, (1 - u).cumsum()]
    ok = np.r_[True, s[:-1] > s[1:], True]  # cuts only between distinct scores
    return s, np.where(ok, CS[:, None] * fp + (1 - CS[:, None]) * (tp[-1] - tp), np.inf)

def place(s, k):  # threshold flagging the top k of descending scores s: gap midpoint; max score if k=0; -inf if k=m
    return s[0] if k == 0 else -np.inf if k == len(s) else (s[k - 1] + s[k]) / 2

def tune(z, t):  # ERM cut per c; ties: median; gap midpoint
    s, C = cuts(z, t); th = []
    for w in C:
        k = np.flatnonzero(w <= w.min() + 1e-9); k = k[len(k) // 2]
        th.append(place(s, k))
    return np.array(th)

def fences(z):  # Tukey far-out fences of the validation scores: the fixed range for winsorised log-odds
    q1, q3 = np.percentile(z, [25, 75]); return q1 - 3 * (q3 - q1), q3 + 3 * (q3 - q1)

def cpu():  # CPU model, read without a subprocess (macOS sysctl, else /proc/cpuinfo)
    try:
        b, k = ctypes.create_string_buffer(256), ctypes.c_size_t(256)
        if ctypes.CDLL(None).sysctlbyname(b"machdep.cpu.brand_string", b, ctypes.byref(k), None, 0) == 0: return b.value.decode()
    except Exception: pass
    try: return next(l.split(":")[1].strip() for l in open("/proc/cpuinfo") if l.startswith("model name"))
    except Exception: return platform.processor() or platform.machine()

cost = lambda d, t: (CS * (d & (t[:, None] == 0)).sum(0) + (1 - CS) * (~d & (t[:, None] == 1)).sum(0)) / len(t)
E = np.zeros((R, 3, 5, 6, 3)); ER = np.zeros((R, 3, 6)); DIS = np.zeros((R, 3, 5, 6)); SP = DIS.copy()
AU, SL, TI = np.zeros((3, R, 3)); Q = np.zeros((R, 3, 3)); vg, mn = 0.0, [n, n]
E2 = np.zeros((R, 3, 5, 6, 3)); AB = np.zeros((R, 3, 5, 6)); ORC = np.zeros((R, 3, 6)); SL2 = np.zeros((R, 3, 2))  # new
for r in range(R):
    Xt, Xh, yt, yh = tts(X, y, train_size=.5, stratify=y, random_state=r)
    Xp, Xe, yp, ye = tts(Xh, yh, train_size=.4, stratify=yh, random_state=r)
    pm = np.random.default_rng(r).permutation(len(yp)); yp = yp[pm]
    for l, f in enumerate([make_pipeline(StandardScaler(), LogReg(max_iter=5000)), HGB(random_state=r), GaussianNB()]):
        f.fit(Xt, yt); zp, ze = score(f, Xp)[pm], score(f, Xe)
        orc = np.maximum(cuts(ze, ye)[1].min(1) / len(ye), 1e-12); ORC[r, l] = orc  # oracle: best test-set threshold
        ex = lambda d: 100 * (cost(d, ye) / orc - 1)  # excess cost, %
        AU[r, l], TI[r, l], Q[r, l] = roc_auc_score(ye, ze), 1 - len(np.unique(ze)) / len(ze), np.percentile(ze, [1, 50, 99])
        ER[r, l] = ex(ze[:, None] > LG)
        for i, m in enumerate(MS):
            zv, tv = zp[:m], yp[:m]; mn = [min(mn[0], tv.sum()), min(mn[1], m - tv.sum())]
            a, b = platt(zv, tv); g = Iso(out_of_bounds="clip").fit(zv, tv); th = tune(zv, tv)
            di, dt, vi, vt = g.predict(ze)[:, None] > CS, ze[:, None] > th, g.predict(zv)[:, None] > CS, zv[:, None] > th
            vg = max(vg, np.abs(cost(vi, tv) - cost(vt, tv)).max()); SP[r, l, i] = (vi == vt).all(0)  # Prop. 1 check
            dp = (a * ze + b)[:, None] > LG
            E[r, l, i] = np.c_[ex(dp), ex(di), ex(dt)]; DIS[r, l, i] = 100 * (di != dt).mean(0)
            if m == 800: SL[r, l] = a
            # new: isotonic's validation partition with the tuned rule's (midpoint) placement
            s = np.sort(zv)[::-1]; tm = np.array([place(s, k) for k in vi.sum(0)])
            a1, b1 = platt(expit(zv), tv)  # Platt on the posterior probability sigma(z) (bounded in [0, 1])
            lo, hi = fences(zv); a2, b2 = platt(np.clip(zv, lo, hi), tv)  # Platt on winsorised log-odds
            E2[r, l, i] = np.c_[ex(ze[:, None] > tm), ex((a1 * expit(ze) + b1)[:, None] > LG), ex((a2 * np.clip(ze, lo, hi) + b2)[:, None] > LG)]
            AB[r, l, i] = 1000 * (cost(dp, ye) - cost(dt, ye))  # absolute Platt - Tuned cost per 1,000 e-mails
            if m == 800: SL2[r, l] = a1, a2

tq = stats.t.ppf(.975, R - 1); r3 = lambda v: round(float(v), 3); g3 = lambda v: float(f"{v:.3g}")
def ci(d, se=None):  # mean, 95% t-CI over seeds
    se = d.std(ddof=1) / R ** .5 if se is None else se; return [r3(d.mean()), r3(d.mean() - tq * se), r3(d.mean() + tq * se)]
sd = lambda d: [r3(d.mean()), r3(d.std(ddof=1))]
P = E.mean((1, 3)); D, DI = P[..., 0] - P[..., 2], P[..., 1] - P[..., 2]  # pooled over learners and c
EM, DC, RW, LN = E.mean(3), E[..., 0] - E[..., 2], ER.mean(2), ["LR", "GB", "NB"]
nb = D[:, 1].std(ddof=1) * (1 / R + len(ye) / (len(yt) + len(yp))) ** .5  # Nadeau-Bengio s.e.
prim, h2, h3 = ci(D[:, 1]), ci(D[:, 4] - D[:, 1]), DIS.mean((0, 1, 3))[1:].max()
NBF = (1 / R + len(ye) / (len(yt) + len(yp))) ** .5  # Nadeau-Bengio s.e. per unit seed s.d., applied to every contrast
cin = lambda d: ci(d, d.std(ddof=1) * NBF)[1:]  # overlap-corrected 95% interval
sci = lambda d: [r3(d.std(ddof=1))] + cin(d)  # s.d. over seeds and overlap-corrected interval
E2M, PT = E2.mean(3), E.mean(3)[..., 2]  # per learner, mean over c; PT = tuned
env = {"python": platform.python_version(), "numpy": np.__version__, "scipy": scipy.__version__, "sklearn": sklearn.__version__,
       "pandas": pd.__version__, "platform": platform.platform(), "cpu": cpu(), "cpu_count": os.cpu_count()}
res = {"n_loaded": n0, "n_duplicates": n0 - n, "n": n, "n_spam": int(y.sum()), "n_train_pool_test": [len(yt), len(yp), len(ye)],
  "seeds": R, "m": MS, "c": CS.tolist(), "min_spam_ham_in_val": list(map(int, mn)),
  "learners": {L: {"auroc": sd(AU[:, l]), "slope800_mean_sd_med": sd(SL[:, l]) + [g3(np.median(SL[:, l]))], "max_tie_frac": r3(TI[:, l].max()),
    "z_q01_q50_q99": [g3(v) for v in np.median(Q[:, l], 0)], "raw_excess": sd(RW[:, l]),
    "excess_platt_iso_tuned": [[r3(v) for v in EM[:, l, i].mean(0)] for i in range(5)], "D_by_m": [ci(DC[:, l, i].mean(1)) for i in range(5)],
    "raw_minus_platt_tuned_800": [ci(RW[:, l] - EM[:, l, 4, k]) for k in (0, 2)]} for l, L in enumerate(LN)},
  "by_m": {m: {"platt_iso_tuned": [sd(P[:, i, k]) for k in range(3)], "D": ci(D[:, i]), "wilcoxon_p": g3(stats.wilcoxon(D[:, i]).pvalue),
    "platt_win_rate": r3((D[:, i] < 0).mean()), "D_iso": ci(DI[:, i]), "disagree_pct": sd(DIS[:, :, i].mean((1, 2))),
    "same_val_partition": r3(SP[:, :, i].mean())} for i, m in enumerate(MS)},
  "cell_D": {m: {L: [ci(DC[:, l, i, j]) for j in range(6)] for l, L in enumerate(LN)} for i, m in ((1, 100), (4, 800))},
  "val_cost_gap_iso_tuned": g3(vg), "primary_D100": prim, "nadeau_bengio_ci": ci(D[:, 1], nb)[1:], "h2_D800_minus_D100": h2,
  "h3_max_disagree_pct": r3(h3), "hypothesis_supported": bool(prim[2] < 0)}
res["revision"] = {
  "nb_factor_on_halfwidth": r3((1 + R * len(ye) / (len(yt) + len(yp))) ** .5), "halfwidth_per_sd": {"seed": r3(tq / R ** .5), "nb": r3(tq * NBF)},
  "h1_D100_sd_nb": sci(D[:, 1]), "h2_sd_nb": sci(D[:, 4] - D[:, 1]),
  "learners": {L: {
    "excess_sd_platt_iso_tuned": [[r3(EM[:, l, i, k].std(ddof=1)) for k in range(3)] for i in range(5)],
    "D_by_m_sd_nb": [sci(DC[:, l, i].mean(1)) for i in range(5)],
    "raw_minus_platt_tuned_800_sd_nb": [sci(RW[:, l] - EM[:, l, 4, k]) for k in (0, 2)],
    "oracle_cost_per1000_by_c": [sd(1000 * ORC[:, l, j]) for j in range(6)],
    "abs_D_per1000_by_m": [ci(AB[:, l, i].mean(1)) + cin(AB[:, l, i].mean(1)) for i in range(5)],
    "platt_prob_excess": [sd(E2M[:, l, i, 1]) for i in range(5)], "platt_prob_minus_tuned": [ci(E2M[:, l, i, 1] - PT[:, l, i]) + cin(E2M[:, l, i, 1] - PT[:, l, i]) for i in range(5)],
    "platt_wins_excess": [sd(E2M[:, l, i, 2]) for i in range(5)], "platt_wins_minus_tuned": [ci(E2M[:, l, i, 2] - PT[:, l, i]) + cin(E2M[:, l, i, 2] - PT[:, l, i]) for i in range(5)],
    "platt_minus_platt_prob_wins": [[ci(EM[:, l, i, 0] - E2M[:, l, i, k]) + cin(EM[:, l, i, 0] - E2M[:, l, i, k]) for k in (1, 2)] for i in range(5)],
    "slope800_median_prob_wins": [g3(np.median(SL2[:, l, k])) for k in range(2)]} for l, L in enumerate(LN)},
  "by_m": {m: {"D_sd_nb": sci(D[:, i]), "D_iso_sd_nb": sci(DI[:, i]), "disagree_nb": cin(DIS[:, :, i].mean((1, 2))),
    "isomid_excess": sd(E2[:, :, i, :, 0].mean((1, 2))),
    "iso_minus_isomid": ci(P[:, i, 1] - E2[:, :, i, :, 0].mean((1, 2))) + cin(P[:, i, 1] - E2[:, :, i, :, 0].mean((1, 2))),
    "isomid_minus_tuned": ci(E2[:, :, i, :, 0].mean((1, 2)) - P[:, i, 2]) + cin(E2[:, :, i, :, 0].mean((1, 2)) - P[:, i, 2]),
    "abs_D_per1000": ci(AB[:, :, i].mean((1, 2))) + cin(AB[:, :, i].mean((1, 2)))} for i, m in enumerate(MS)},
  "cell_D_sd_nb": {m: {L: [sci(DC[:, l, i, j]) for j in range(6)] for l, L in enumerate(LN)} for i, m in ((1, 100), (4, 800))},
  "env": env}
res["runtime_s"] = round(time.time() - T0, 1)
print("RESULT_JSON: " + json.dumps(res))
Listing. experiment_v1.py

4.Analysis

Attempt 4Done

Analysis complete: experiment succeeded; inconclusive; 67 key numbers.

Reviewer feedback applied: Independent referee (Claude Fable 5.1), round 2: minor_revision. Required changes: (1) [text] In the abstract, replace 'In absolute cost at that budget, tuning is cheaper by 6.9 per 1,000 e-mails, which survives a split-overlap correction' with a sentence that says the absolute endpoint was not pre-specified and that its robust advantage for tuning (6.9 per 1,000 at m = 100, and from m = 100 upward) is driven by naive Bayes, with LR and GB favouring Platt at the seed level (Table S1); make the Findings bullet ('tuning is robustly cheaper in absolute cost') say the same. (2) [text] In Section 5.3, replace 'That is level with tuning (-1.1 [-11.0, 8.9]; Table S2)' with wording that does not assert equivalence, e.g. 'not distinguishable from tuning at this budget (-1.1 [-11.0, 8.9]; corrected [-48.2, 46.1]; Table S2)'. Keep the analysis record consistent with the paper.
InconclusiveExperiment succeeded

Results and analysis

The revised run succeeded in one round (exit 0, 10.1 s on an Apple M4). It reproduces every number of the first run exactly. It adds seed s.d.s, overlap-corrected (Nadeau-Bengio) intervals for every contrast, oracle costs C*, an absolute-cost endpoint, a cut-placement ablation and Platt fits on two bounded inputs. The sample is unchanged: 4,210 deduplicated e-mails (1,679 spam), 50 seeds of 2,105/842/1,263 splits. H1 is unsupported by its pre-specified rule: D(100) = -7.927 [-16.46, 0.607], corrected [-48.341, 32.487]. Because the interval covers 0, H1 is neither supported nor refuted, which this schema records as inconclusive. H2 meets its rule at the seed level, 31.244 [23.156, 39.332], but not under the correction, [-7.061, 69.549]. H3 holds by its rule, with at most 0.603% disagreement for m >= 100, but the corrected upper limit at m = 100 is 1.078%. D weights each cell by 1/C*, and C* runs from 4.686 to 41.809 per 1,000 e-mails. Unweighted, the pooled absolute difference, which was not pre-specified, favours tuning: 6.867 [5.942, 7.792] per 1,000 e-mails at m = 100, corrected [2.485, 11.248], and robustly from m = 100 on (at m = 50 the corrected interval, [-1.004, 11.337], covers 0). Naive Bayes drives it entirely (25.426, against -1.585 for logistic regression and -3.241 for gradient boosting, which favour Platt at the seed level). At the seed level Platt beats tuning at every budget for logistic regression and gradient boosting, but none of their Platt-versus-tuning contrasts survives the correction; naive Bayes's Platt deficit survives from m = 200. Winsorising naive Bayes's log-odds at Tukey's far-out fences cuts its Platt excess by 51.421 points at m = 100 (corrected [24.503, 78.338]), to 43.807 ± 14.936%, not distinguishable from tuning at this budget (-1.078 [-11.035, 8.879]; corrected [-48.233, 46.078]). From m = 200 it plateaus at 37.736-40.119% and tuning is cheaper, robustly only at m = 800. At the seed level, isotonic's small-budget edge over tuning is entirely cut placement: -20.564 of -19.981 at m = 50, with partition choice contributing 0.583 [-0.313, 1.48]. No isotonic contrast survives the correction.

Key numbers

MetricValueInterpretation
e-mails loaded / duplicates removed / analysed / spam4601 / 391 / 4210 / 1679Analysed sample after removing exact duplicate rows that could leak between validation and test
train / pool / test per seed; seeds2105 / 842 / 1263; 50 seedsStratified 50/20/30 split redrawn for each seed; budgets are nested prefixes of the permuted pool
minimum spam / ham in any validation prefix15 / 24Both classes always present; no redraws
runtime and environment10.1 s; Python 3.12.13, NumPy 2.5.2, SciPy 1.18.1, scikit-learn 1.9.0, pandas 3.0.5; Apple M4 (10 cores), macOS-27.0.1-arm64-arm-64bitWhole revised analysis; recorded by the run itself. Learner defaults depend on the scikit-learn version
overlap-correction factor on every seed-level half-width; half-width per s.d. (seed-level / corrected)4.736; 0.284 / 1.346Nadeau-Bengio correction sqrt(1/50 + n_test/(n_train + n_pool)) applied uniformly; a contrast is robust if its corrected interval excludes 0
PRIMARY H1: D(100), pooled relative excess-cost difference Platt - Tuned-7.927 ± 30.027 (corrected CI -48.341, 32.487); seed-level CI -16.46, 0.607Unsupported by the pre-specified rule: the interval covers 0 (not refuted, which needs an interval above 0)
H1 further evidence: Wilcoxon p; Platt win rate at m = 100p = 0.121; 0.56No reliable pooled advantage for Platt
H2: D(800) - D(100)31.244 ± 28.46 (corrected CI -7.061, 69.549); seed-level CI 23.156, 39.332Meets its pre-specified seed-level rule but is not robust to the overlap correction
H3: maximum mean isotonic-vs-tuned test disagreement for m >= 100; corrected upper limit at m = 1000.603%; corrected CI 0.127, 1.078Holds by its pre-specified rule (point estimate below 1%), but the corrected upper limit at m = 100 exceeds 1%; below 1% from m = 200
largest validation-cost gap, isotonic vs tuned6.94e-18Floating-point rounding: Proposition 1 holds exactly
m = 50: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.)59.822 ± 15.224 / 75.706 ± 32.585 / 95.687 ± 43.339Excess over the test-optimal threshold, averaged over 3 learners x 6 costs
m = 50: pooled D = Platt - Tuned (relative, 1/C*-weighted)-35.865 ± 51.04 (corrected CI -104.561, 32.832); seed-level CI -50.37, -21.359; Wilcoxon p = 1.88e-05Negative favours Platt
m = 50: pooled absolute cost difference Platt - Tuned per 1,000 e-mails5.166 [95% CI 3.864, 6.469; corrected -1.004, 11.337]Positive favours tuning; unweighted by C*; not pre-specified; not robust at this budget (corrected interval covers 0)
m = 50: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned)-19.981 [95% CI -25.614, -14.348]; -20.564 [95% CI -26.089, -15.04; corrected -46.727, 5.599]; 0.583 [95% CI -0.313, 1.48; corrected -3.663, 4.829]Cut-placement ablation: isotonic's partition with midpoint placement
m = 100: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.)49.267 ± 12.144 / 52.251 ± 21.885 / 57.194 ± 25.003Excess over the test-optimal threshold, averaged over 3 learners x 6 costs
m = 100: pooled D = Platt - Tuned (relative, 1/C*-weighted)-7.927 ± 30.027 (corrected CI -48.341, 32.487); seed-level CI -16.46, 0.607; Wilcoxon p = 0.121Negative favours Platt
m = 100: pooled absolute cost difference Platt - Tuned per 1,000 e-mails6.867 [95% CI 5.942, 7.792; corrected 2.485, 11.248]Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes (logistic regression and gradient boosting favour Platt at the seed level)
m = 100: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned)-4.943 [95% CI -6.412, -3.474]; -5.277 [95% CI -6.633, -3.92; corrected -11.701, 1.148]; 0.334 [95% CI -0.167, 0.835; corrected -2.037, 2.705]Cut-placement ablation: isotonic's partition with midpoint placement
m = 200: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.)43.462 ± 11.132 / 30.645 ± 12.326 / 32.027 ± 13.272Excess over the test-optimal threshold, averaged over 3 learners x 6 costs
m = 200: pooled D = Platt - Tuned (relative, 1/C*-weighted)11.435 ± 18.644 (corrected CI -13.658, 36.528); seed-level CI 6.136, 16.733; Wilcoxon p = 9.37e-06Negative favours Platt
m = 200: pooled absolute cost difference Platt - Tuned per 1,000 e-mails8.119 [95% CI 7.217, 9.022; corrected 3.844, 12.394]Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes
m = 200: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned)-1.382 [95% CI -2.002, -0.762]; -1.34 [95% CI -1.841, -0.838; corrected -3.715, 1.036]; -0.043 [95% CI -0.446, 0.361; corrected -1.952, 1.867]Cut-placement ablation: isotonic's partition with midpoint placement
m = 400: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.)40.528 ± 10.64 / 20.688 ± 5.343 / 20.842 ± 5.667Excess over the test-optimal threshold, averaged over 3 learners x 6 costs
m = 400: pooled D = Platt - Tuned (relative, 1/C*-weighted)19.686 ± 13.102 (corrected CI 2.052, 37.32); seed-level CI 15.963, 23.41; Wilcoxon p = 2.02e-11Negative favours Platt
m = 400: pooled absolute cost difference Platt - Tuned per 1,000 e-mails8.606 [95% CI 7.7, 9.512; corrected 4.315, 12.898]Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes
m = 400: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned)-0.154 [95% CI -0.474, 0.166]; -0.206 [95% CI -0.379, -0.034; corrected -1.023, 0.61]; 0.053 [95% CI -0.238, 0.343; corrected -1.325, 1.43]Cut-placement ablation: isotonic's partition with midpoint placement
m = 800: pooled excess (%) Platt / isotonic / tuned (mean ± s.d.)38.829 ± 11.146 / 15.472 ± 3.763 / 15.512 ± 3.657Excess over the test-optimal threshold, averaged over 3 learners x 6 costs
m = 800: pooled D = Platt - Tuned (relative, 1/C*-weighted)23.317 ± 12.073 (corrected CI 7.068, 39.566); seed-level CI 19.886, 26.748; Wilcoxon p = 3e-13Negative favours Platt
m = 800: pooled absolute cost difference Platt - Tuned per 1,000 e-mails8.858 [95% CI 7.888, 9.827; corrected 4.266, 13.449]Positive favours tuning; unweighted by C*; not pre-specified; robust, but driven entirely by naive Bayes
m = 800: isotonic - tuned; placement (isotonic - isotonic-mid); partition (isotonic-mid - tuned)-0.04 [95% CI -0.322, 0.242]; 0.009 [95% CI -0.077, 0.095; corrected -0.398, 0.416]; -0.049 [95% CI -0.314, 0.217; corrected -1.306, 1.209]Cut-placement ablation: isotonic's partition with midpoint placement
logistic regression: excess (%) Platt / isotonic / tuned by m (mean ± s.d.)m=50: 41.204 ± 17.415 / 81.712 ± 46.891 / 99.063 ± 58.72; m=100: 24.867 ± 11.169 / 50.555 ± 25.335 / 55.051 ± 27.95; m=200: 18.114 ± 7.243 / 29.909 ± 15.099 / 30.562 ± 15.426; m=400: 15.501 ± 5.382 / 20.974 ± 7.801 / 20.765 ± 8.053; m=800: 14.143 ± 4.584 / 16.348 ± 6.329 / 16.316 ± 6.462Learning curves per learner, means and s.d. over 50 seeds
logistic regression: D = Platt - Tuned by m (mean ± s.d.; corrected CI)m=50: -57.859 ± 69.259 (corrected CI -151.076, 35.358); m=100: -30.183 ± 33.877 (corrected CI -75.779, 15.413); m=200: -12.449 ± 18.185 (corrected CI -36.925, 12.027); m=400: -5.263 ± 10.252 (corrected CI -19.061, 8.535); m=800: -2.173 ± 6.388 (corrected CI -10.77, 6.424)Platt ahead at every budget at the seed level; no corrected interval excludes 0
logistic regression: raw minus Platt and raw minus Tuned at m = 800 (mean ± s.d.; corrected CI)-1.713 ± 5.3 (corrected CI -8.846, 5.421); -3.886 ± 7.028 (corrected CI -13.345, 5.573)Negative favours raw (no validation data); robust only for NB
logistic regression: oracle cost C* per 1,000 e-mails by c = 0.02, 0.1, 0.25, 0.75, 0.9, 0.98 (mean ± s.d.)6.405 ± 0.897; 18.352 ± 1.912; 27.094 ± 2.987; 25.851 ± 2.865; 16.884 ± 2.509; 5.741 ± 0.861The relative endpoint weights each cell by 1/C*
logistic regression: absolute Platt - Tuned per 1,000 e-mails at m = 100-1.585 [95% CI -2.559, -0.61; corrected -6.199, 3.03]Positive favours tuning; not pre-specified; favours Platt at the seed level, not robust
logistic regression: Platt on posterior probability / on winsorised log-odds, excess (%) by m (mean ± s.d.)m=50: 33.253 ± 13.943 / 41.124 ± 17.318; m=100: 28.96 ± 10.22 / 24.798 ± 11.149; m=200: 26.774 ± 7.481 / 18.056 ± 7.248; m=400: 27.332 ± 6.954 / 15.496 ± 5.388; m=800: 27.893 ± 6.908 / 14.129 ± 4.599Bounded-input Platt fits (input-scale ablation)
logistic regression: Platt(log-odds) minus Platt(winsorised) by mm=50: 0.08 [95% CI -0.005, 0.165; corrected -0.322, 0.481]; m=100: 0.069 [95% CI 0.026, 0.112; corrected -0.133, 0.272]; m=200: 0.058 [95% CI 0.01, 0.106; corrected -0.168, 0.284]; m=400: 0.006 [95% CI -0.011, 0.022; corrected -0.072, 0.084]; m=800: 0.014 [95% CI -0.005, 0.033; corrected -0.076, 0.104]Positive: winsorising lowers Platt's excess; negligible control
logistic regression: Platt(winsorised) minus Tuned by mm=50: -57.939 [95% CI -77.607, -38.271; corrected -151.085, 35.207]; m=100: -30.252 [95% CI -39.881, -20.624; corrected -75.853, 15.348]; m=200: -12.507 [95% CI -17.678, -7.335; corrected -36.998, 11.985]; m=400: -5.269 [95% CI -8.186, -2.352; corrected -19.083, 8.546]; m=800: -2.187 [95% CI -4.007, -0.368; corrected -10.805, 6.43]Negative favours winsorised Platt
logistic regression: test AUROC; Platt slope at m = 800 (mean ± s.d.; median); median slope on probability / winsorised input0.978 ± 0.003; 0.856 ± 0.099 (median 0.857); 7.26 / 0.857Slope below 1 means over-confident log-odds; NB's log-odds slope collapses
logistic regression: test log-odds 1st / 50th / 99th percentile; raw plug-in excess (%)-25.3 / -2.04 / 11.4; 12.43 ± 3.653Scale and tails of the thresholded score; raw uses no validation data
gradient boosting: excess (%) Platt / isotonic / tuned by m (mean ± s.d.)m=50: 39.284 ± 21.075 / 81.229 ± 54.133 / 114.128 ± 72.238; m=100: 27.705 ± 15.129 / 63.628 ± 40.437 / 71.645 ± 46.887; m=200: 19.482 ± 9.129 / 35.686 ± 21.046 / 38.286 ± 22.437; m=400: 15.606 ± 7.044 / 23.782 ± 10.21 / 24.214 ± 10.93; m=800: 13.934 ± 6.415 / 17.562 ± 6.749 / 17.769 ± 6.912Learning curves per learner, means and s.d. over 50 seeds
gradient boosting: D = Platt - Tuned by m (mean ± s.d.; corrected CI)m=50: -74.844 ± 79.858 (corrected CI -182.326, 32.639); m=100: -43.94 ± 50.478 (corrected CI -111.879, 23.999); m=200: -18.804 ± 23.756 (corrected CI -50.778, 13.17); m=400: -8.608 ± 12.045 (corrected CI -24.82, 7.603); m=800: -3.835 ± 8.244 (corrected CI -14.931, 7.261)Platt ahead at every budget at the seed level; no corrected interval excludes 0
gradient boosting: raw minus Platt and raw minus Tuned at m = 800 (mean ± s.d.; corrected CI)12.227 ± 10.623 (corrected CI -2.07, 26.524); 8.392 ± 11.355 (corrected CI -6.891, 23.675)Negative favours raw (no validation data); robust only for NB
gradient boosting: oracle cost C* per 1,000 e-mails by c = 0.02, 0.1, 0.25, 0.75, 0.9, 0.98 (mean ± s.d.)5.657 ± 1.166; 15.259 ± 2.198; 22.692 ± 2.969; 20.455 ± 2.412; 12.423 ± 1.944; 4.686 ± 1.209The relative endpoint weights each cell by 1/C*
gradient boosting: absolute Platt - Tuned per 1,000 e-mails at m = 100-3.241 [95% CI -4.294, -2.188; corrected -8.227, 1.745]Positive favours tuning; not pre-specified; favours Platt at the seed level, not robust
gradient boosting: Platt on posterior probability / on winsorised log-odds, excess (%) by m (mean ± s.d.)m=50: 54.036 ± 26.192 / 39.284 ± 21.075; m=100: 50.63 ± 18.896 / 27.705 ± 15.129; m=200: 46.572 ± 16.316 / 19.482 ± 9.129; m=400: 48.14 ± 16.74 / 15.606 ± 7.044; m=800: 49.601 ± 17.466 / 13.934 ± 6.415Bounded-input Platt fits (input-scale ablation)
gradient boosting: Platt(log-odds) minus Platt(winsorised) by mm=50: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=100: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=200: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=400: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]; m=800: 0.0 [95% CI 0.0, 0.0; corrected 0.0, 0.0]Positive: winsorising lowers Platt's excess; negligible control
gradient boosting: Platt(winsorised) minus Tuned by mm=50: -74.844 [95% CI -97.539, -52.148; corrected -182.326, 32.639]; m=100: -43.94 [95% CI -58.286, -29.595; corrected -111.879, 23.999]; m=200: -18.804 [95% CI -25.556, -12.053; corrected -50.778, 13.17]; m=400: -8.608 [95% CI -12.032, -5.185; corrected -24.82, 7.603]; m=800: -3.835 [95% CI -6.178, -1.492; corrected -14.931, 7.261]Negative favours winsorised Platt
gradient boosting: test AUROC; Platt slope at m = 800 (mean ± s.d.; median); median slope on probability / winsorised input0.984 ± 0.003; 0.649 ± 0.061 (median 0.639); 6.8 / 0.639Slope below 1 means over-confident log-odds; NB's log-odds slope collapses
gradient boosting: test log-odds 1st / 50th / 99th percentile; raw plug-in excess (%)-10.3 / -3.92 / 8.9; 26.161 ± 9.26Scale and tails of the thresholded score; raw uses no validation data
Gaussian naive Bayes: excess (%) Platt / isotonic / tuned by m (mean ± s.d.)m=50: 98.978 ± 16.782 / 64.176 ± 34.329 / 73.869 ± 40.825; m=100: 95.228 ± 19.42 / 42.57 ± 26.905 / 44.885 ± 28.459; m=200: 92.791 ± 23.858 / 26.34 ± 18.95 / 27.234 ± 19.278; m=400: 90.476 ± 27.203 / 17.307 ± 9.671 / 17.546 ± 9.663; m=800: 88.411 ± 29.49 / 12.508 ± 5.387 / 12.451 ± 5.414Learning curves per learner, means and s.d. over 50 seeds
Gaussian naive Bayes: D = Platt - Tuned by m (mean ± s.d.; corrected CI)m=50: 25.109 ± 47.077 (corrected CI -38.253, 88.471); m=100: 50.343 ± 37.898 (corrected CI -0.664, 101.35); m=200: 65.557 ± 35.923 (corrected CI 17.207, 113.907); m=400: 72.93 ± 32.037 (corrected CI 29.811, 116.049); m=800: 75.96 ± 31.74 (corrected CI 33.24, 118.679)Platt behind at every budget; robust from m = 200
Gaussian naive Bayes: raw minus Platt and raw minus Tuned at m = 800 (mean ± s.d.; corrected CI)373.843 ± 48.167 (corrected CI 309.014, 438.672); 449.803 ± 39.495 (corrected CI 396.646, 502.96)Negative favours raw (no validation data); robust only for NB
Gaussian naive Bayes: oracle cost C* per 1,000 e-mails by c = 0.02, 0.1, 0.25, 0.75, 0.9, 0.98 (mean ± s.d.)8.956 ± 1.015; 30.318 ± 2.202; 41.809 ± 4.013; 34.913 ± 3.456; 23.701 ± 2.336; 7.289 ± 0.501The relative endpoint weights each cell by 1/C*
Gaussian naive Bayes: absolute Platt - Tuned per 1,000 e-mails at m = 10025.426 [95% CI 23.401, 27.451; corrected 15.837, 35.015]Positive favours tuning; not pre-specified; robust, and the whole source of the pooled absolute advantage for tuning
Gaussian naive Bayes: Platt on posterior probability / on winsorised log-odds, excess (%) by m (mean ± s.d.)m=50: 62.315 ± 9.805 / 55.981 ± 20.306; m=100: 62.959 ± 14.872 / 43.807 ± 14.936; m=200: 60.861 ± 12.555 / 40.119 ± 16.483; m=400: 58.638 ± 10.085 / 39.653 ± 15.898; m=800: 57.536 ± 9.429 / 37.736 ± 12.887Bounded-input Platt fits (input-scale ablation)
Gaussian naive Bayes: Platt(log-odds) minus Platt(winsorised) by mm=50: 42.997 [95% CI 37.664, 48.329; corrected 17.742, 68.251]; m=100: 51.421 [95% CI 45.737, 57.104; corrected 24.503, 78.338]; m=200: 52.673 [95% CI 46.211, 59.134; corrected 22.071, 83.274]; m=400: 50.824 [95% CI 43.283, 58.364; corrected 15.114, 86.533]; m=800: 50.675 [95% CI 42.604, 58.747; corrected 12.45, 88.901]Positive: winsorising lowers Platt's excess; robust at every budget
Gaussian naive Bayes: Platt(winsorised) minus Tuned by mm=50: -17.888 [95% CI -31.982, -3.794; corrected -84.635, 48.86]; m=100: -1.078 [95% CI -11.035, 8.879; corrected -48.233, 46.078]; m=200: 12.885 [95% CI 4.823, 20.946; corrected -25.294, 51.063]; m=400: 22.107 [95% CI 16.107, 28.106; corrected -6.307, 50.52]; m=800: 25.284 [95% CI 21.036, 29.532; corrected 5.167, 45.402]Negative favours winsorised Platt
Gaussian naive Bayes: Platt(log-odds) minus Platt(probability); Platt(probability) minus Tuned, by mm=50: 36.663 [95% CI 32.643, 40.682; corrected 17.628, 55.697] / -11.554 [95% CI -24.138, 1.031; corrected -71.154, 48.047]; m=100: 32.269 [95% CI 26.099, 38.439; corrected 3.048, 61.491] / 18.074 [95% CI 9.963, 26.185; corrected -20.338, 56.486]; m=200: 31.931 [95% CI 24.465, 39.396; corrected -3.425, 67.286] / 33.626 [95% CI 28.196, 39.057; corrected 7.907, 59.346]; m=400: 31.838 [95% CI 24.043, 39.633; corrected -5.078, 68.754] / 41.092 [95% CI 37.432, 44.752; corrected 23.759, 58.426]; m=800: 30.875 [95% CI 22.653, 39.097; corrected -8.064, 69.814] / 45.085 [95% CI 41.676, 48.494; corrected 28.939, 61.231]The probability input helps less than winsorising
Gaussian naive Bayes: test AUROC; Platt slope at m = 800 (mean ± s.d.; median); median slope on probability / winsorised input0.955 ± 0.005; 0.003 ± 0.005 (median 1.38e-07); 4.1 / 0.013Slope below 1 means over-confident log-odds; NB's log-odds slope collapses
Gaussian naive Bayes: test log-odds 1st / 50th / 99th percentile; raw plug-in excess (%)-175000000.0 / 15.2 / 626.0; 462.254 ± 38.857Scale and tails of the thresholded score; raw uses no validation data
logistic regression: per-cell Platt - Tuned at m = 100 by c (mean ± s.d.; corrected CI)c=0.02: -88.069 ± 124.0 (corrected CI -254.963, 78.826); c=0.1: -1.123 ± 37.694 (corrected CI -51.855, 49.61); c=0.25: 2.328 ± 20.942 (corrected CI -25.857, 30.514); c=0.75: 11.48 ± 26.329 (corrected CI -23.956, 46.917); c=0.9: -3.675 ± 34.911 (corrected CI -50.662, 43.312); c=0.98: -102.043 ± 125.744 (corrected CI -271.285, 67.2)Per-cell paired differences over 50 seeds
gradient boosting: per-cell Platt - Tuned at m = 100 by c (mean ± s.d.; corrected CI)c=0.02: -107.249 ± 186.897 (corrected CI -358.797, 144.299); c=0.1: -18.662 ± 41.707 (corrected CI -74.796, 37.472); c=0.25: -11.649 ± 26.172 (corrected CI -46.874, 23.577); c=0.75: -3.537 ± 17.989 (corrected CI -27.748, 20.675); c=0.9: -17.616 ± 48.968 (corrected CI -83.523, 48.291); c=0.98: -104.929 ± 185.791 (corrected CI -354.99, 145.131)Per-cell paired differences over 50 seeds
Gaussian naive Bayes: per-cell Platt - Tuned at m = 100 by c (mean ± s.d.; corrected CI)c=0.02: -62.968 ± 103.176 (corrected CI -201.835, 75.899); c=0.1: 58.845 ± 29.879 (corrected CI 18.631, 99.06); c=0.25: 191.49 ± 58.941 (corrected CI 112.161, 270.82); c=0.75: 169.41 ± 33.203 (corrected CI 124.721, 214.099); c=0.9: 43.023 ± 29.648 (corrected CI 3.119, 82.926); c=0.98: -97.742 ± 130.447 (corrected CI -273.314, 77.829)Per-cell paired differences over 50 seeds
logistic regression: per-cell Platt - Tuned at m = 800 by c (mean ± s.d.; corrected CI)c=0.02: -10.0 ± 25.303 (corrected CI -44.055, 24.056); c=0.1: 1.253 ± 12.392 (corrected CI -15.426, 17.932); c=0.25: -0.228 ± 6.542 (corrected CI -9.033, 8.577); c=0.75: 1.133 ± 10.653 (corrected CI -13.205, 15.471); c=0.9: 0.992 ± 17.742 (corrected CI -22.887, 24.87); c=0.98: -6.188 ± 19.752 (corrected CI -32.772, 20.396)Per-cell paired differences over 50 seeds
gradient boosting: per-cell Platt - Tuned at m = 800 by c (mean ± s.d.; corrected CI)c=0.02: -6.177 ± 23.734 (corrected CI -38.122, 25.768); c=0.1: -3.521 ± 9.134 (corrected CI -15.815, 8.772); c=0.25: -1.977 ± 6.69 (corrected CI -10.981, 7.028); c=0.75: -2.5 ± 9.452 (corrected CI -15.222, 10.222); c=0.9: -4.893 ± 14.176 (corrected CI -23.973, 14.187); c=0.98: -3.943 ± 35.969 (corrected CI -52.354, 44.469)Per-cell paired differences over 50 seeds
Gaussian naive Bayes: per-cell Platt - Tuned at m = 800 by c (mean ± s.d.; corrected CI)c=0.02: 7.527 ± 25.974 (corrected CI -27.432, 42.486); c=0.1: 60.643 ± 34.809 (corrected CI 13.792, 107.493); c=0.25: 187.724 ± 78.559 (corrected CI 81.99, 293.459); c=0.75: 159.829 ± 51.124 (corrected CI 91.02, 228.638); c=0.9: 56.221 ± 22.493 (corrected CI 25.947, 86.495); c=0.98: -16.187 ± 23.153 (corrected CI -47.349, 14.976)Per-cell paired differences over 50 seeds

Caveats

  • Single, dated benchmark: 4,210 deduplicated e-mails from one company in 1999 with pre-engineered frequency features; effect sizes and possibly signs may not transfer.
  • Two inference levels. Seed-level intervals describe the expected difference over random splits of this corpus. The Nadeau-Bengio correction approximates the corpus's own sampling variability with a heuristic between-split correlation (n_test/n). Applied to every contrast (half-widths x 4.736), it leaves H2, every LR and GB Platt-versus-tuning contrast (relative, absolute and per cell), every raw-versus-recipe contrast for LR and GB, and every isotonic contrast with intervals covering 0.
  • Excess cost is relative to a test-fitted oracle. The pre-specified relative endpoint weights each cell by 1/C*, with C* from 4.686 to 41.809 per 1,000 e-mails. The unweighted absolute endpoint, which was not pre-specified, has the opposite sign at m <= 100, so the pooled H1 verdict depends on the weighting: in absolute cost the interval lies above 0 (tuning cheaper) from m = 100 on, even under the correction (at m = 50 the corrected interval covers 0), and naive Bayes drives this entirely, as logistic regression and gradient boosting favour Platt at the seed level. No absolute analogue of H2 was computed.
  • The pooled endpoint averages learners whose effects have opposite signs; its null result does not mean the recipes are equivalent.
  • The ablations (isotonic partition with midpoint placement; Platt on the posterior probability and on log-odds winsorised at Tukey's far-out fences of the validation scores) were specified after the first review, not in the archived plan. Tuning with interpolated placement, other clipping rules, beta calibration and cost-aware calibration were not run.
  • Winsorising removes about half of NB's Platt excess but leaves a plateau of 37.736-40.119% from m = 200. This residual sigmoid misfit is inferred, not decomposed further.
  • Three learners at default hyperparameters (version-dependent: scikit-learn 1.9.0), six cost proportions, budgets up to 800, no cross-fitted calibration. Up to 0.044 of gradient-boosting test scores are tied.
  • Budgets are nested within seeds, so contrasts across m are correlated. Secondary contrasts carry no multiplicity adjustment.

5.Paper writing

Attempt 8Done

Manuscript drafted: “Calibrate or Tune? Cost-Sensitive Decision Thresholds as Estimators on Spambase” (≈ 4989 words).

Reviewer feedback applied: Independent referee (Claude Fable 5.1), round 2: minor_revision. Required changes: (1) [text] In the abstract, replace 'In absolute cost at that budget, tuning is cheaper by 6.9 per 1,000 e-mails, which survives a split-overlap correction' with a sentence that says the absolute endpoint was not pre-specified and that its robust advantage for tuning (6.9 per 1,000 at m = 100, and from m = 100 upward) is driven by naive Bayes, with LR and GB favouring Platt at the seed level (Table S1); make the Findings bullet ('tuning is robustly cheaper in absolute cost') say the same. (2) [text] In Section 5.3, replace 'That is level with tuning (-1.1 [-11.0, 8.9]; Table S2)' with wording that does not assert equivalence, e.g. 'not distinguishable from tuning at this budget (-1.1 [-11.0, 8.9]; corrected [-48.2, 46.1]; Table S2)'. Keep the analysis record consistent with the paper.

≈ 4989 words · AI-drafted manuscript · not peer-reviewed

Calibrate or Tune? Cost-Sensitive Decision Thresholds as Estimators on Spambase

Abstract

Under asymmetric costs, held-out data can recalibrate scores for the theoretical threshold c=cFP/(cFP+cFN)c=c_{FP}/(c_{FP}+c_{FN}) or tune the threshold directly. Monotone calibration followed by cc thresholds the raw score, so both recipes estimate one cut. We make explicit that isotonic plug-in attains the minimum empirical validation cost, and contrast tuning's cube-root rate with the parametric rate and bias of Platt's sigmoid. On Spambase (4,210 e-mails; 3 learners × 6 cost proportions × 5 budgets of 50–800 e-mails; 50 seeds), the pre-specified pooled comparison at 100 e-mails is unsupported: Platt minus tuned excess cost is −7.9 points (95% CI −16.5 to 0.6). An absolute-cost endpoint, not pre-specified, favours tuning at that budget by 6.9 per 1,000 e-mails and survives a split-overlap correction from 100 e-mails upward (not at 50), but naive Bayes drives it entirely. At the seed level, Platt beats tuning for logistic regression and gradient boosting and trails for naive Bayes, and isotonic's small-budget edge comes from cut placement; under the correction only the naive Bayes deficit survives, so the rest are point-estimate patterns on this corpus. Winsorising naive Bayes's heavy-tailed log-odds halves its Platt excess, leaving a residual bias.

Keywords: cost-sensitive classification; probability calibration; decision threshold; isotonic regression; Platt scaling; Spambase

1 Introduction

With false-positive and false-negative costs cFPc_{FP} and cFNc_{FN}, the Bayes rule flags an instance whose class probability exceeds c=cFP/(cFP+cFN)c=c_{FP}/(c_{FP}+c_{FN}) [1]. As fitted scores are rarely calibrated, practitioners either recalibrate them (Platt [2], isotonic regression [3]) and threshold at cc, or tune the threshold on validation data [4]. We found no comparison of the two at a controlled budget. Search was compared only with uncalibrated probabilities [4] and calibrators only on squared error [5], and regret bounds after recalibration [8] do not say how much data each recipe needs. Since a non-decreasing calibrator gg turns g(z)>cg(z)>c into a threshold on the raw score zz, both recipes estimate one cut. We contribute:

  • Proposition 1 (after [6, 7]): isotonic plug-in attains the minimum empirical cost and differs from tuning only by tie-breaking and cut placement within one gap.
  • Proposition 2: tuning's excess cost is OP(m−2/3)O_P(m^{-2/3}) [9]; under stated regularity, Platt's cut converges at the parametric rate to a limit that is biased unless the sigmoid is exact at level cc.
  • A paired evaluation on Spambase [11], pre-specified in the archived run record, with seed-level and split-overlap-corrected [10] intervals and ablations of cut placement and input scale.
  • Findings. The pre-specified pooled contrast is unsupported. An absolute-cost endpoint, not pre-specified, robustly favours tuning from m=100m=100 on, but only through naive Bayes. Logistic regression and gradient boosting favour Platt and cut placement drives isotonic's small-budget edge, but these are seed-level, point-estimate patterns: only the naive Bayes results survive the overlap correction.

2 Related Work

Cost-sensitive decisions. Elkan [1] derived the theoretical threshold, which is optimal for perfectly calibrated scores [7]; Sheng and Ling [4] instead tune it on cross-validated predictions, and Perez-Lebel et al. [8] separate the regret that recalibration removes from a grouping loss it cannot. Provost and Fawcett [12] choose operating points under imprecise costs from the ROC convex hull, which is equivalent to PAV calibration [6], the basis of Proposition 1. Spam filtering is strongly asymmetric: Androutsopoulos et al. [13] used thresholds up to 0.999, where blocking one legitimate e-mail is as bad as passing 999 spam.

Calibration. Platt [2] fits a sigmoid with smoothed targets, which Lin, Lin and Weng [14] fit robustly; Zadrozny and Elkan [3] use isotonic regression. Niculescu-Mizil and Caruana [5] find that naive Bayes pushes probabilities towards 0 and 1, and that Platt beats isotonic below roughly 200–1,000 calibration cases. Kumar, Liang and Ma [15] likewise find binning less sample-efficient than scaling. Kull, Silva Filho and Flach [16] proposed beta calibration because the logistic map misfits skewed scores such as naive Bayes's. Böken [17] asks when Platt scaling is appropriate, and Yang and Bi [18] make calibration cost-aware.

3 Problem Setup

Let zz be the log-odds score of a base classifier trained on separate data, y∈{0,1}y\in\{0,1\} the label (1 = spam), cFP=cc_{FP}=c and cFN=1−cc_{FN}=1-c. A rule δ\delta costs R(δ)=cPr⁡(δ=1,y=0)+(1−c)Pr⁡(δ=0,y=1)R(\delta)=c\Pr(\delta=1,y=0)+(1-c)\Pr(\delta=0,y=1), minimised by 1{η(z)>c}\mathbb{1}\{\eta(z)>c\} with η(z)=Pr⁡(y=1∣z)\eta(z)=\Pr(y=1\mid z) [1]. Write R(t)R(t) for the rule 1{z>t}\mathbb{1}\{z>t\} and R^m(t)\hat R_m(t) for its empirical cost on a validation sample DmD_m. A non-decreasing calibrator g^\hat g fitted on DmD_m yields the rule 1{g^(z)>c}\mathbb{1}\{\hat g(z)>c\}; tuning picks t^∈arg⁡min⁡tR^m(t)\hat t\in\arg\min_t\hat R_m(t).

Proposition 1 (isotonic plug-in is empirical tuning). Let g^\hat g be the isotonic (PAV) fit of yy on zz over DmD_m, with tied scores pooled. Then 1{g^(z)>c}\mathbb{1}\{\hat g(z)>c\} attains min⁡tR^m(t)\min_t\hat R_m(t). On new points it can disagree with a tuned rule only (i) inside the gap between the validation scores adjacent to the selected cut, or beyond the extreme score, or (ii) on blocks whose inclusion leaves R^m\hat R_m unchanged.

Proof sketch. PAV splits the sorted scores into blocks BjB_j with non-decreasing positive rates vjv_j. Flagging block jj changes mR^mm\hat R_m by ∣Bj∣(c−vj)|B_j|(c-v_j), which changes sign once, so flagging exactly the blocks with vj>cv_j>c is optimal. No cut inside a block does better, or PAV would split it. Zero increments give (ii). □\square

Proposition 2 (rates; informal). Let the class-conditional densities of zz be continuous and positive near the Bayes cut t∗t^*, let η\eta increase with η(t∗)=c\eta(t^*)=c, and let R′′(t∗)>0R''(t^*)>0. (a) A tuned threshold has t^−t∗=OP(m−1/3)\hat t-t^*=O_P(m^{-1/3}) and R(t^)−R(t∗)=OP(m−2/3)R(\hat t)-R(t^*)=O_P(m^{-2/3}). (b) Suppose the sigmoid's Kullback–Leibler projection (a†,b†)(a^\dagger,b^\dagger) is unique with a†>0a^\dagger>0, so the fitted slope a^\hat a stays bounded away from zero with probability tending to one, and Platt's cut t^P=(logit⁡c−b^)/a^\hat t_P=(\operatorname{logit}c-\hat b)/\hat a has finite second moments. Then t^P→tP=(logit⁡c−b†)/a†\hat t_P\to t_P=(\operatorname{logit}c-b^\dagger)/a^\dagger at rate m−1/2m^{-1/2} and ER(t^P)−R(t∗)=ΔP+O(m−1)\mathbb{E}R(\hat t_P)-R(t^*)=\Delta_P+O(m^{-1}), where the bias ΔP=R(tP)−R(t∗)≥0\Delta_P=R(t_P)-R(t^*)\ge0 vanishes exactly when σ(a†t∗+b†)=c\sigma(a^\dagger t^*+b^\dagger)=c.

Proof sketch. (a) Kim and Pollard's cube-root theorem [9], then a second-order expansion of RR at t∗t^*. (b) The maximum-likelihood estimate is m\sqrt m-consistent for its projection; the slope condition passes this rate to t^P\hat t_P by the delta method, and the moment condition makes the mean first-order term of RR at tPt_P of order m−1m^{-1}. □\square

Platt should thus win at small mm when ΔP\Delta_P is small, and tuning once m−2/3m^{-2/3} falls below ΔP\Delta_P: the parametric-versus-binning trade-off of [15], applied to decisions.

4 Method

Algorithm 1 is the executed protocol; every rule sees the same mm validation e-mails.

Algorithm 1  Paired evaluation of calibrate-then-threshold versus tuning
for seed r in 0..49:
  stratified split: train 50%, pool 20%, test 30%; permute pool
  for learner f in {LR, GB, NB}:
    fit f on train; z <- log-odds on pool and test; C*[c] <- minimum test cost over cuts
    for m in {50, 100, 200, 400, 800}:  V <- first m pool e-mails
      Platt:     MLE sigmoid(a u + b) on V, u = z;  flag if a u + b > logit(c)
                 (Platt-prob: u = sigmoid(z); Platt-wins: u = z clipped to V's far-out fences)
      Isotonic:  g <- PAV fit on V;  flag if g(z) > c   (Iso-mid: its partition of V, midpoint cut)
      Tuned:     median cut minimising c FP + (1-c) FN on V;  flag above the gap midpoint
      E[rule, c] <- 100 (test cost / C*[c] - 1);  A[c] <- cost(Platt) - cost(Tuned)

Tuning scans all m+1m+1 cuts with cumulative counts, in O(mlog⁡m+m∣C∣)O(m\log m+m|C|); PAV costs O(mlog⁡m)O(m\log m) and each Platt fit O(m)O(m) per iteration.

5 Experiments

5.1 Setup

Data. Spambase [11] is dataset 94 of the UCI Machine Learning Repository, donated by Hewlett-Packard Labs (licence CC BY 4.0) and generated in June–July 1999 from filed spam and filed work and personal e-mail. We retrieved it on 24 October 2026 and verified its checksum. Its 4,601 e-mails have 57 predictors and no missing values: 48 word and 6 character frequencies, as percentages (0–100) of words or characters matching a token, and the mean, longest and total length of capital-letter runs. The label is_spam is 1 for spam. Removing 391 exact duplicates, so that no e-mail sits in both validation and test, leaves 4,210 e-mails (1,679 spam). Predictors are transformed by log⁡(1+x)\log(1+x).

Protocol. Each of 50 seeds draws stratified splits of 2,105 training, 842 pool and 1,263 test e-mails. Budgets m∈{50,100,200,400,800}m\in\{50,100,200,400,800\} are nested prefixes of a seeded permutation of the pool; every prefix held at least 15 spam and 24 legitimate e-mails. The cost proportions c∈{0.02,0.1,0.25,0.75,0.9,0.98}c\in\{0.02,0.1,0.25,0.75,0.9,0.98\} include extremes like those used for spam [13].

Learners and rules. We used L2 logistic regression (LR; C=1C=1, standardised inputs), histogram gradient boosting (GB) and Gaussian naive Bayes (NB), otherwise with scikit-learn defaults. Platt is fitted by effectively unpenalised maximum likelihood (L-BFGS, C=1012C=10^{12}, scores divided by their interquartile range) with targets (N++1)/(N++2)(N_++1)/(N_++2) and 1/(N−+2)1/(N_-+2) [2, 14]. The tuned threshold is the largest validation score when the optimal cut flags no validation e-mail, and −∞-\infty when it flags all. Three ablations were added in revision:

  • Iso-mid: isotonic's validation partition with tuning's midpoint placement;
  • Platt-prob: Platt on σ(z)\sigma(z), scikit-learn's input for models without a decision function, such as NB;
  • Platt-wins: Platt on zz winsorised at the validation scores' Tukey far-out fences, [q1−3 IQR, q3+3 IQR][q_1-3\,\mathrm{IQR},\,q_3+3\,\mathrm{IQR}].

Raw applies cc to the model's own probability, and the oracle is the best threshold on the test set.

Metric and inference. Excess cost E=100 (C/C∗−1)E=100\,(C/C^*-1) is the percentage by which a rule's test cost exceeds the oracle's, C∗C^*; it isolates the regret that thresholding can change [8]. The primary endpoint, pre-specified in the archived run record, is D(100)D(100). D(m)D(m) is a seed's mean over the 18 learner-by-cost cells of EPlatt−ETuned=100 (CPlatt−CTuned)/C∗E_{\text{Platt}}-E_{\text{Tuned}}=100\,(C_{\text{Platt}}-C_{\text{Tuned}})/C^*, which weights each cell's cost difference by 1/C∗1/C^*. We also report the unweighted CPlatt−CTunedC_{\text{Platt}}-C_{\text{Tuned}} per 1,000 e-mails, which was not pre-specified. We give means ± s.d. over seeds, paired 95% tt-intervals and Wilcoxon signed-rank tests. Seed-level intervals ignore the corpus's own sampling variability. The heuristic Nadeau–Bengio correction [10], applied to every contrast, approximates it by scaling standard errors by 1/50+ntest/(ntrain+npool)\sqrt{1/50+n_{\text{test}}/(n_{\text{train}}+n_{\text{pool}})}, which widens half-widths 4.74-fold (from 0.284 to 1.346 s.d.). A contrast is robust if its corrected interval excludes zero, and only robust contrasts are claimed beyond this corpus. All contrasts but D(100)D(100), including the rest of Table 3 and Tables 1, 4, 5 and S1–S3, are secondary and exploratory, without multiplicity adjustment. The run took 10.1 s on one Apple M4 CPU (Appendix B).

5.2 Main Results

Table 1: Base learners and the raw baseline (50 seeds; secondary)

LearnerAUROCPlatt slope (median)Platt-wins slopeLog-odds p1/p50/p99Raw excess (%)Raw − PlattRaw − Tuned
LR0.978 ± 0.0030.856 ± 0.099 (0.857)0.857−25.3/−2.04/11.412.4 ± 3.7−1.7 ± 5.3*−3.9 ± 7.0*
GB0.984 ± 0.0030.649 ± 0.061 (0.639)0.639−10.3/−3.92/8.926.2 ± 9.312.2 ± 10.6*8.4 ± 11.4*
NB0.955 ± 0.0050.003 ± 0.005 (1.38e−7)0.013−1.75e8/15.2/626462.3 ± 38.9373.8 ± 48.2†449.8 ± 39.5†

Notes: mean ± s.d. (medians in parentheses); slopes and raw contrasts at m=800m=800; percentiles are medians over seeds. A slope below 1 means over-confident log-odds. Raw contrasts are paired differences in points, negative favouring raw; * seed-level 95% CI excludes 0; † overlap-corrected CI too (robust). Tied test scores: at most 0.002 (LR, NB) and 0.044 (GB).

Table 2: Hypotheses pre-specified in the archived run record (50 seeds)

HypothesisEstimate [95% CI]Overlap-corrected 95% CIVerdict
H1: D(100)<0D(100)<0−7.9 [−16.5, 0.6]; Wilcoxon pp = 0.121; Platt better in 56% of seeds[−48.3, 32.5]Unsupported (CI covers 0)
H2: D(800)−D(100)>0D(800)-D(100)>031.2 [23.2, 39.3][−7.1, 69.5]Supported at seed level; not robust
H3: isotonic–tuned test disagreement < 1% for m≥100m\ge100at most 0.60%upper limit 1.08% at mm = 100Holds by its rule; not robust at mm = 100

Notes: D(m)D(m) (Section 5.1) is in points. The plan calls H1 unsupported if its interval covers 0 and refuted if it lies above 0.

Table 3: Excess test cost (%) by budget, pooled over learners and costs (50 seeds)

mPlattIsotonicTunedDD = Platt − Tuned (points)Platt − Tuned, absolute (per 1,000 e-mails)
5059.82 ± 15.2275.71 ± 32.5995.69 ± 43.34−35.9 [−50.4, −21.4]5.2 [3.9, 6.5]
10049.27 ± 12.1452.25 ± 21.8957.19 ± 25.00−7.9 [−16.5, 0.6]6.9 [5.9, 7.8]†
20043.46 ± 11.1330.65 ± 12.3332.03 ± 13.2711.4 [6.1, 16.7]8.1 [7.2, 9.0]†
40040.53 ± 10.6420.69 ± 5.3420.84 ± 5.6719.7 [16.0, 23.4]†8.6 [7.7, 9.5]†
80038.83 ± 11.1515.47 ± 3.7615.51 ± 3.6623.3 [19.9, 26.7]†8.9 [7.9, 9.8]†

Notes: mean ± s.d.; bold marks the row minimum. Differences are mean [seed-level 95% CI]; † marks a robust contrast.

Table 4: Learning curves: excess test cost (%) by learner and rule (mean ± s.d., 50 seeds)

Learner, rulem=50m=100m=200m=400m=800
LR Platt41.2 ± 17.424.9 ± 11.218.1 ± 7.215.5 ± 5.414.1 ± 4.6
LR Isotonic81.7 ± 46.950.6 ± 25.329.9 ± 15.121.0 ± 7.816.3 ± 6.3
LR Tuned99.1 ± 58.755.1 ± 28.030.6 ± 15.420.8 ± 8.116.3 ± 6.5
LR Δ = Platt − Tuned−57.9 ± 69.3*−30.2 ± 33.9*−12.4 ± 18.2*−5.3 ± 10.3*−2.2 ± 6.4*
GB Platt39.3 ± 21.127.7 ± 15.119.5 ± 9.115.6 ± 7.013.9 ± 6.4
GB Isotonic81.2 ± 54.163.6 ± 40.435.7 ± 21.023.8 ± 10.217.6 ± 6.7
GB Tuned114.1 ± 72.271.6 ± 46.938.3 ± 22.424.2 ± 10.917.8 ± 6.9
GB Δ = Platt − Tuned−74.8 ± 79.9*−43.9 ± 50.5*−18.8 ± 23.8*−8.6 ± 12.0*−3.8 ± 8.2*
NB Platt99.0 ± 16.895.2 ± 19.492.8 ± 23.990.5 ± 27.288.4 ± 29.5
NB Platt-wins56.0 ± 20.343.8 ± 14.940.1 ± 16.539.7 ± 15.937.7 ± 12.9
NB Isotonic64.2 ± 34.342.6 ± 26.926.3 ± 19.017.3 ± 9.712.5 ± 5.4
NB Tuned73.9 ± 40.844.9 ± 28.527.2 ± 19.317.5 ± 9.712.5 ± 5.4
NB Δ = Platt − Tuned25.1 ± 47.1*50.3 ± 37.9*65.6 ± 35.9†72.9 ± 32.0†76.0 ± 31.7†

Notes: bold marks a learner's lowest unrounded mean. Δ rows are paired differences in points; * seed-level 95% CI excludes 0; † robust. This table replaces a learning-curve figure.

Table 2 gives the hypothesis verdicts. The pooled sign depends on the weighting (Table 3). DD turns positive from m=200m=200, robustly from m=400m=400, while the absolute difference favours tuning at every budget, robustly from m=100m=100. Both pool effects of opposite sign (Table 4). At the seed level Platt beats tuning at every budget for LR and GB (−30.2 ± 33.9 and −43.9 ± 50.5 points at m=100m=100). No LR or GB Platt-versus-tuning contrast is robust, relative or absolute (Table S1). NB's Platt excess stays at 88.4–99.0% while tuning's falls to 12.5%, a deficit robust from m=200m=200. In absolute cost at m=100m=100, LR (−1.6 per 1,000 e-mails) and GB (−3.2) favour Platt and NB (25.4) favours tuning.

5.3 Ablations

Table 5: Isotonic versus tuning, with the cut-placement ablation (pooled, 50 seeds)

mIsotonic − TunedPlacement: Isotonic − Iso-midPartition: Iso-mid − TunedDisagreement (% of e-mails)Same partition
50−19.98 [−25.61, −14.35]−20.56 [−26.09, −15.04]0.58 [−0.31, 1.48]1.25 ± 0.800.968
100−4.94 [−6.41, −3.47]−5.28 [−6.63, −3.92]0.33 [−0.17, 0.84]0.60 ± 0.350.956
200−1.38 [−2.00, −0.76]−1.34 [−1.84, −0.84]−0.04 [−0.45, 0.36]0.38 ± 0.360.933
400−0.15 [−0.47, 0.17]−0.21 [−0.38, −0.03]0.05 [−0.24, 0.34]0.31 ± 0.240.899
800−0.04 [−0.32, 0.24]0.01 [−0.08, 0.10]−0.05 [−0.31, 0.22]0.19 ± 0.150.896

Notes: differences in points, mean [seed-level 95% CI]; the ablation columns sum to the first, and none is robust. Disagreement: mean ± s.d.; last column: share of fits. Largest validation-cost gap over 4,500 rule pairs: 6.94e−18 (rounding error).

Cut placement. As Proposition 1 predicts, isotonic and tuned rules have equal validation cost in every fit, and 89.6–96.8% of fits share the validation partition (Table 5). Yet isotonic is cheaper at small budgets (−19.98 points at m=50m=50), and Iso-mid removes all of this edge. Placement accounts for −20.56 and partition choice for 0.58 [−0.31, 1.48]. Isotonic's interpolated cut lies a fraction (c−vlo)/(vhi−vlo)(c-v_{\text{lo}})/(v_{\text{hi}}-v_{\text{lo}}) of the way across the gap, where vlov_{\text{lo}} and vhiv_{\text{hi}} are the adjacent block rates, so it moves with cc; the midpoint ignores cc. None of this is robust.

Input scale. Platt-wins lowers NB's Platt excess by 51.4 [45.7, 57.1] points at m=100m=100, robustly, to 43.8 ± 14.9%. That is not distinguishable from tuning at this budget (−1.1 [−11.0, 8.9]; corrected [−48.2, 46.1]; Table S2), but from m=200m=200 Platt-wins plateaus at 37.7–40.1% and trails tuning, robustly at m=800m=800. Platt-prob helps less (63.0 ± 14.9% at m=100m=100), presumably because it maps the upper half of NB's scores (log-odds above the median, 15.2) to within 10−610^{-6} of 1. For LR and GB, winsorising changes Platt's excess by at most 0.08 points.

No validation data. Raw plug-in is worst by far for NB (462.3%, robustly; Table 1). At m=800m=800, LR's own probabilities cost less than Platt and tuning (by 1.7 and 3.9 points) and GB's cost more, at the seed level only, so whether LR needs recalibration is unresolved.

5.4 Analysis

LR (slope 0.856) and GB (0.649) have over-confident log-odds that a sigmoid can rescale, judging by the outcomes. NB's log-odds are heavy-tailed (1st percentile about −1.75e8, median 15.2), and its slope collapses to a median of 1.38e−7 (Table 1). Such a slope is not bounded away from zero and makes the cut (logit⁡c−b^)/a^(\operatorname{logit}c-\hat b)/\hat a unstable, so the conditions of Proposition 2(b) fail and it does not describe Platt on NB's log-odds. On winsorised log-odds the median slope is 0.013, the conditions are plausible, and the plateau is consistent with the fixed bias ΔP\Delta_P of a sigmoid not exact at level cc. Beta calibration [16] is the established remedy for such misfit on skewed naive Bayes scores, but we did not run it.

Weighting. C∗C^* is 4.7–9.0 per 1,000 e-mails at c∈{0.02,0.98}c\in\{0.02,0.98\} and 20.5–41.8 at c∈{0.25,0.75}c\in\{0.25,0.75\} (Table S3), so DD weights extreme cells several times more than central ones. At m=100m=100 Platt leads in all six extreme cells, by 63.0–107.2 points, presumably because tuning is noisy where the cut lies in a sparse tail, but no such lead is robust; only NB's four central cells are, where Platt trails by 43.0–191.5 points. The pooled H1 verdict therefore depends on the 1/C∗1/C^* weighting.

6 Limitations

The evidence comes from one dated benchmark (4,210 e-mails, one company, 1999, engineered frequencies), so effect sizes and possibly signs may not transfer. Seed-level intervals are conditional on this corpus, the overlap correction is heuristic, and there is no multiplicity adjustment. Excess cost is relative to a test-fitted oracle, and pooled verdicts depend on the weighting. The ablations were added after the first review, outside the archived plan. Tuning with interpolated placement, other clipping rules, beta calibration [16] and cost-aware calibration [18] were not run. We used three default-hyperparameter learners (defaults vary with the scikit-learn version), six cost proportions, budgets up to 800, no cross-fitting, and tables instead of figures.

7 Conclusion

At equal held-out budgets, "calibrate, then threshold at cc" is neither uniformly better nor worse than tuning. Isotonic calibration is tuning up to where the cut sits inside one gap, and on Spambase that placement helped at small budgets, at the seed level. The pre-specified pooled test is unsupported, and its sign flips with the weighting: in absolute cost, tuning is robustly cheaper because of naive Bayes. Platt's lead for LR and GB is a seed-level pattern on one corpus. Its NB deficit is robust; winsorising NB's log-odds halves Platt's excess but leaves a residual bias. Practitioners should check score tails and the calibrator's fit near cc, and whether recalibration is needed at all.

References

[1] C. Elkan. The foundations of cost-sensitive learning. In Proc. IJCAI, pages 973–978, 2001. https://mlanthology.org/ijcai/2001/elkan2001ijcai-foundations/

[2] J. C. Platt. Probabilities for SV machines. In Advances in Large-Margin Classifiers, pages 61–74. MIT Press, 2000. https://doi.org/10.7551/mitpress/1113.003.0008

[3] B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proc. ACM SIGKDD, pages 694–699, 2002. https://doi.org/10.1145/775047.775151

[4] V. S. Sheng and C. X. Ling. Thresholding for making classifiers cost-sensitive. In Proc. AAAI, pages 476–481, 2006. https://cdn.aaai.org/AAAI/2006/AAAI06-076.pdf

[5] A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proc. ICML, pages 625–632, 2005. https://doi.org/10.1145/1102351.1102430

[6] T. Fawcett and A. Niculescu-Mizil. PAV and the ROC convex hull. Machine Learning, 68(1):97–106, 2007. https://doi.org/10.1007/s10994-007-5011-0

[7] J. Hernández-Orallo, P. Flach and C. Ferri. A unified view of performance metrics: translating threshold choice into expected classification loss. JMLR, 13:2813–2869, 2012. https://jmlr.org/papers/v13/hernandez-orallo12a.html

[8] A. Perez-Lebel, G. Varoquaux, S. Koyejo, M. Doutreligne and M. Le Morvan. Decision from suboptimal classifiers: excess risk pre- and post-calibration. In Proc. AISTATS, PMLR 258:2395–2403, 2025. https://proceedings.mlr.press/v258/perez-lebel25a.html

[9] J. Kim and D. Pollard. Cube root asymptotics. Annals of Statistics, 18(1):191–219, 1990. https://doi.org/10.1214/aos/1176347498

[10] C. Nadeau and Y. Bengio. Inference for the generalization error. Machine Learning, 52(3):239–281, 2003. https://doi.org/10.1023/A:1024068626366

[11] M. Hopkins, E. Reeber, G. Forman and J. Suermondt. Spambase [Dataset]. UCI Machine Learning Repository, 1999. https://doi.org/10.24432/C53G6X

[12] F. Provost and T. Fawcett. Robust classification for imprecise environments. Machine Learning, 42(3):203–231, 2001. https://doi.org/10.1023/A:1007601015854

[13] I. Androutsopoulos, J. Koutsias, K. V. Chandrinos, G. Paliouras and C. D. Spyropoulos. An evaluation of naive Bayesian anti-spam filtering. In Proc. Workshop on Machine Learning in the New Information Age, 11th European Conference on Machine Learning, pages 9–17, 2000. https://arxiv.org/abs/cs/0006013

[14] H.-T. Lin, C.-J. Lin and R. C. Weng. A note on Platt's probabilistic outputs for support vector machines. Machine Learning, 68(3):267–276, 2007. https://doi.org/10.1007/s10994-007-5018-6

[15] A. Kumar, P. Liang and T. Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems 32, 2019. https://arxiv.org/abs/1909.10155

[16] M. Kull, T. Silva Filho and P. Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proc. AISTATS, PMLR 54:623–631, 2017. https://proceedings.mlr.press/v54/kull17a.html

[17] B. Böken. On the appropriateness of Platt scaling in classifier calibration. Information Systems, 95:101641, 2021. https://doi.org/10.1016/j.is.2020.101641

[18] M. Yang and X. Bi. Cost-aware calibration of classifiers. INFORMS Journal on Data Science, 4(2):101–113, 2025. https://doi.org/10.1287/ijds.2024.0038

Appendix (supplementary material)

A Supplementary tables

Table S1: Absolute cost difference Platt − Tuned by learner (per 1,000 e-mails, mean [95% CI], 50 seeds; secondary)

Learnerm=50m=100m=200m=400m=800
LR−2.5 [−4.4, −0.6]−1.6 [−2.6, −0.6]−0.6 [−1.1, −0.1]−0.3 [−0.6, 0.1]−0.1 [−0.4, 0.2]
GB−5.4 [−7.0, −3.7]−3.2 [−4.3, −2.2]−1.6 [−2.0, −1.2]−0.8 [−1.1, −0.6]−0.5 [−0.7, −0.3]
NB23.4 [21.5, 25.3]†25.4 [23.4, 27.5]†26.5 [24.1, 29.0]†26.9 [24.3, 29.5]†27.2 [24.3, 30.0]†

Notes: positive favours tuning; † marks a robust contrast. Pooled values are in Table 3.

Table S2: Input-scale ablation: Platt on bounded inputs (points, mean [95% CI], 50 seeds; secondary)

Contrastm=50m=100m=200m=400m=800
NB: Platt − Platt-wins43.0 [37.7, 48.3]†51.4 [45.7, 57.1]†52.7 [46.2, 59.1]†50.8 [43.3, 58.4]†50.7 [42.6, 58.7]†
NB: Platt-wins − Tuned−17.9 [−32.0, −3.8]−1.1 [−11.0, 8.9]12.9 [4.8, 20.9]22.1 [16.1, 28.1]25.3 [21.0, 29.5]†
NB: Platt − Platt-prob36.7 [32.6, 40.7]†32.3 [26.1, 38.4]†31.9 [24.5, 39.4]31.8 [24.0, 39.6]30.9 [22.7, 39.1]
NB: Platt-prob − Tuned−11.6 [−24.1, 1.0]18.1 [10.0, 26.2]33.6 [28.2, 39.1]†41.1 [37.4, 44.8]†45.1 [41.7, 48.5]†
NB: Platt-prob excess (%, mean ± s.d.)62.3 ± 9.863.0 ± 14.960.9 ± 12.658.6 ± 10.157.5 ± 9.4
LR: Platt − Platt-wins0.08 [−0.01, 0.17]0.07 [0.03, 0.11]0.06 [0.01, 0.11]0.01 [−0.01, 0.02]0.01 [−0.01, 0.03]
LR: Platt − Platt-prob8.0 [4.1, 11.8]−4.1 [−7.1, −1.1]−8.7 [−11.1, −6.2]−11.8 [−13.9, −9.7]†−13.8 [−15.8, −11.7]†
GB: Platt − Platt-wins0.00 [0.00, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
GB: Platt − Platt-prob−14.8 [−19.2, −10.3]−22.9 [−27.2, −18.7]†−27.1 [−30.7, −23.5]†−32.5 [−36.2, −28.9]†−35.7 [−39.5, −31.9]†

Notes: a positive "Platt − X" means that input X lowers Platt's excess. † marks a robust contrast; for example, NB, Platt − Platt-wins at m = 100: [24.5, 78.3]; NB, Platt-wins − Tuned at m = 800: [5.2, 45.4].

Table S3: Platt − Tuned by cost proportion at m=100m=100, and the oracle cost C∗C^* (mean ± s.d., 50 seeds; secondary)

LearnerQuantityc=0.02c=0.1c=0.25c=0.75c=0.9c=0.98
LRΔ, m = 100 (points)−88.1 ± 124.0*−1.1 ± 37.72.3 ± 20.911.5 ± 26.3*−3.7 ± 34.9−102.0 ± 125.7*
LRC* (per 1,000 e-mails)6.4 ± 0.918.4 ± 1.927.1 ± 3.025.9 ± 2.916.9 ± 2.55.7 ± 0.9
GBΔ, m = 100 (points)−107.2 ± 186.9*−18.7 ± 41.7*−11.6 ± 26.2*−3.5 ± 18.0−17.6 ± 49.0*−104.9 ± 185.8*
GBC* (per 1,000 e-mails)5.7 ± 1.215.3 ± 2.222.7 ± 3.020.5 ± 2.412.4 ± 1.94.7 ± 1.2
NBΔ, m = 100 (points)−63.0 ± 103.2*58.8 ± 29.9†191.5 ± 58.9†169.4 ± 33.2†43.0 ± 29.6†−97.7 ± 130.4*
NBC* (per 1,000 e-mails)9.0 ± 1.030.3 ± 2.241.8 ± 4.034.9 ± 3.523.7 ± 2.37.3 ± 0.5

Notes: negative Δ favours Platt. * seed-level 95% CI excludes 0; † robust. C∗C^* is the oracle's test cost per 1,000 e-mails with cFP=cc_{FP}=c and cFN=1−cc_{FN}=1-c.

B Reproducibility

Seeds 0–49 fix the splits, the pool permutation and gradient boosting. Hyperparameters and conventions are given in Section 5.1. The run recorded its own environment: Python 3.12.13, NumPy 2.5.2, SciPy 1.18.1, scikit-learn 1.9.0 and pandas 3.0.5 on macOS 27.0.1 (arm64), with an Apple M4 CPU (10 cores). It took 10.1 s. The defaults of HistGradientBoostingClassifier and LogisticRegression depend on the scikit-learn version, so other versions may give slightly different numbers. The data are public [11], and the script behind every number is archived with the run record. The revised script reproduces every number of the first submission exactly, and its added rules use no new random draws.

C NeurIPS paper checklist

Table S4: NeurIPS paper checklist

ItemAnswerJustification
1 ClaimsYesThe abstract and Section 1 state the unsupported primary result and separate robust from seed-level findings
2 LimitationsYesSection 6
3 TheoryYesAssumptions and proof sketches in Section 3; Proposition 2 is informal
4 ReproducibilityYesAlgorithm 1, Section 5.1, Appendix B
5 Open data and codeYesPublic data [11]; the script is archived with the run record
6 Experimental detailsYesSection 5.1
7 Statistical significanceYesMeans ± s.d., seed-level and overlap-corrected 95% intervals, Wilcoxon tests
8 ComputeYesOne CPU, 10.1 s (Appendix B)
9 Code of ethicsYesPublic, licensed data reduced to word and character frequencies
10 Broader impactsYesA methodological study; a misplaced spam threshold blocks legitimate mail
11 SafeguardsNANo model or data released
12 LicencesYesSpambase is CC BY 4.0 [11]
13 New assetsNANone
14 Crowdsourcing and human subjectsNANone
15 IRB approvalNANone
16 LLM usageYesA research pipeline driven by a large language model produced this study, including the analysis code and this text; no LLM is part of the method studied

6.Review

Attempt 2Done

Automated review: Accept (novelty 3, rigor 4, clarity 4, reproducibility 5).

Accept
Novelty 3/5Rigor 4/5Clarity 4/5Reproducibility 5/5

Refereed by Claude Fable 5.1, a different model from the one that wrote the paper (Claude Opus 5.5).

Final round. The two text changes I required in round 2 are made precisely and match the executed record, which is unchanged: the abstract and Findings bullet now present the post-hoc absolute-cost result with its three qualifiers (not pre-specified; robust from m = 100, not at 50; driven entirely by naive Bayes), and the 'level with tuning' sentence now says 'not distinguishable from tuning at this budget' with both intervals. The three trims to hold length remove no number and alter no claim. Across three rounds the paper has gone from an abstract carrying per-learner claims that its own sensitivity check overturned to one in which every claim is matched to the strength of its evidence, every number traces to RESULT_JSON, the needed ablations were run, the oracle costs and weighting are exposed, the literature gaps are filled, and the run is fully reproducible. Its contribution is modest in scope, which the scores reflect, but it is sound, honest and complete as submitted, and no required changes remain. I recommend acceptance.

Strengths

  • Both remaining required changes are made exactly and agree with the record. The abstract now says the absolute-cost endpoint was not pre-specified, that it favours tuning by 6.9 per 1,000 e-mails at m = 100 and survives the overlap correction from 100 e-mails upward but not at 50 (record: m = 100 corrected [2.5, 11.2]; m = 50 corrected [-1.0, 11.3]), and that naive Bayes drives it entirely (per-learner -1.6, -3.2, +25.4); the Findings bullet says the same. Section 5.3 now reads 'not distinguishable from tuning at this budget (-1.1 [-11.0, 8.9]; corrected [-48.2, 46.1])', matching the record's -1.078 [-11.035, 8.879; -48.233, 46.078].
  • The executed record is unchanged from round 2 (same script, same RESULT_JSON, exit 0, empty stderr, 10.1 s), so every number I verified cell by cell in round 2 (Tables 1-5, S1-S3, all seed-level and overlap-corrected markers, C* ranges, ablation sums) still holds; the three length trims (one lead-in sentence, two table notes now pointing to Section 5.1) remove no number and change no claim.
  • The manuscript now claims only what the record supports, and says so consistently in abstract, Findings bullet, Section 5.2-5.4 and the conclusion: the pre-specified pooled contrast is unsupported; the pooled sign depends on the weighting and the learner mix; LR and GB advantages of Platt are seed-level patterns on one corpus; NB's deficit is robust; isotonic's small-budget edge is cut placement (not robust); winsorising NB's log-odds halves Platt's excess and leaves a residual bias; H2 and H3 are reported with their corrected intervals and the plan's own verdict rules.
  • Soundness and reporting meet the venue's conventions: Proposition 1 is correct and verified on the record; Proposition 2(b) states the regularity it needs and Section 5.4 says NB violates it; contributions as bullets, executed-protocol pseudocode, every averaged result with +/- s.d. over 50 seeds, seed-level and overlap-corrected intervals, component-at-a-time ablations disclosed as post-review additions, a specific limitations section, full environment and compute statement, archived script with exact reproduction, and the NeurIPS checklist.

Weaknesses

  • Scope and significance remain modest: one 1999 benchmark, three default-hyperparameter learners, a null pre-specified primary result, Proposition 1 a restatement of known equivalences and Proposition 2 an informal rate argument; the generalisable content is the estimation framing plus one robust, learner-specific finding (Platt's sigmoid fails on NB's heavy-tailed log-odds and winsorising only half-repairs it).
  • Inference caveats the paper states but cannot remove: the Nadeau-Bengio correction is heuristic and may be conservative for paired differences that share splits; about 80 secondary contrasts carry robustness markers with no multiplicity adjustment; the ablations and the absolute endpoint are post hoc.
  • Beta calibration, the cited remedy for exactly the NB misfit, and tuning with interpolated placement were not run; Boeken [17] is cited for its title only; learning curves are a table rather than a figure (a pipeline constraint).

Required changes

—

Record

§Event timeline

  1. Run createdQuestion: “Under asymmetric misclassification costs, does post-hoc probability calibration with the theoretical cost-optimal threshold beat directly tuning the decision threshold on the same held-out validation data?' language=en discipline=computer_science venue='NeurIPS” · output language: English · models: agent:claude-opus-5-5
  2. Stage started · Literature reviewAttempt 1 · agent:claude-opus-5-5
  3. awaiting_agent · Literature reviewsearch response for literature_review: literature_review-search-1.response.json
  4. Resume requested
  5. Stage started · Literature reviewAttempt 2 · agent:claude-opus-5-5
  6. Search complete · Literature review26 sources · 15,616 characters of notes
  7. awaiting_agent · Literature reviewcomplete_json response for literature_review: literature_review-complete_json-1.response.json
  8. Resume requested
  9. Stage started · Literature reviewAttempt 3 · agent:claude-opus-5-5
  10. Search complete · Literature review26 sources · 15,616 characters of notes
  11. Stage completed · Literature review
  12. Stage started · IdeationAttempt 1 · agent:claude-opus-5-5
  13. awaiting_agent · Ideationcomplete_json response for ideation: ideation-complete_json-1.response.json
  14. Resume requested
  15. Stage started · IdeationAttempt 2 · agent:claude-opus-5-5
  16. Awaiting approval · Ideation
  17. Approved · IdeationIdea 1 selected
  18. Stage started · ExperimentationAttempt 1 · agent:claude-opus-5-5
  19. Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
  20. awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
  21. Resume requested
  22. Stage started · ExperimentationAttempt 2 · agent:claude-opus-5-5
  23. Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
  24. Experiment round · ExperimentationRound 1: exit 0; result parsed (9.343s)
  25. Stage completed · Experimentation
  26. Stage started · AnalysisAttempt 1 · agent:claude-opus-5-5
  27. awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
  28. Resume requested
  29. Stage started · AnalysisAttempt 2 · agent:claude-opus-5-5
  30. Stage completed · Analysis
  31. Stage started · Paper writingAttempt 1 · agent:claude-opus-5-5
  32. awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
  33. Resume requested
  34. Stage started · Paper writingAttempt 2 · agent:claude-opus-5-5
  35. Stage completed · Paper writing
  36. Stage started · ReviewAttempt 1 · agent:claude-opus-5-5
  37. awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
  38. Resume requested
  39. Stage started · ReviewAttempt 2 · agent:claude-opus-5-5
  40. Awaiting approval · Review
  41. Sent back for revision · ReviewRe-running from Paper writing: “The referee recommended minor_revision. Address each required change below, quoted verbatim, using only results already in the run record (no new analyses, no new numbers):\n1. "Qualify every \'at every”
  42. Stage started · Paper writingAttempt 3 · agent:claude-opus-5-5
  43. awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
  44. Resume requested
  45. Stage started · Paper writingAttempt 4 · agent:claude-opus-5-5
  46. Stage completed · Paper writing
  47. Stage started · ReviewAttempt 1 · agent:claude-opus-5-5
  48. awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
  49. Resume requested
  50. Stage started · ReviewAttempt 2 · agent:claude-opus-5-5
  51. Awaiting approval · Review
  52. Approved · Review
  53. Run completereport=/Users/haodingresearch/Desktop/auto-research/runs/studies-opus55/computer_science-en/runs/4689bab459a3/REPORT.md
  54. Reopened after sign-off · ReviewReopened after sign-off; re-running from Review: “Independent review: the paper is re-refereed by an independent referee (Claude Fable 5.1) that took no part in writing it.”
  55. Stage started · ReviewAttempt 3 · agent:claude-fable-5-1
  56. awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
  57. Resume requested
  58. Stage started · ReviewAttempt 4 · agent:claude-fable-5-1
  59. Awaiting approval · Review
  60. Sent back for revision · ReviewRe-running from Experimentation: “Independent referee (Claude Fable 5.1): major_revision. Required changes: (1) [text] Apply the overlap correction uniformly: multiply every seed-level half-width in Tables 1, 3, 4, 5 and 6 and for H2 ”
  61. Stage started · ExperimentationAttempt 3 · agent:claude-opus-5-5
  62. Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
  63. awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
  64. Resume requested
  65. Stage started · ExperimentationAttempt 4 · agent:claude-opus-5-5
  66. Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
  67. Experiment round · ExperimentationRound 1: exit 0; result parsed (10.957s)
  68. Stage completed · Experimentation
  69. Stage started · AnalysisAttempt 1 · agent:claude-opus-5-5
  70. awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
  71. Resume requested
  72. Stage started · AnalysisAttempt 2 · agent:claude-opus-5-5
  73. Stage completed · Analysis
  74. Stage started · Paper writingAttempt 5 · agent:claude-opus-5-5
  75. awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
  76. Resume requested
  77. Stage started · Paper writingAttempt 6 · agent:claude-opus-5-5
  78. Stage completed · Paper writing
  79. Stage started · ReviewAttempt 1 · agent:claude-fable-5-1
  80. awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
  81. Resume requested
  82. Stage started · ReviewAttempt 2 · agent:claude-fable-5-1
  83. Awaiting approval · Review
  84. Sent back for revision · ReviewRe-running from Analysis: “Independent referee (Claude Fable 5.1), round 2: minor_revision. Required changes: (1) [text] In the abstract, replace 'In absolute cost at that budget, tuning is cheaper by 6.9 per 1,000 e-mails, whi”
  85. Stage started · AnalysisAttempt 3 · agent:claude-opus-5-5
  86. awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
  87. Resume requested
  88. Stage started · AnalysisAttempt 4 · agent:claude-opus-5-5
  89. Stage completed · Analysis
  90. Stage started · Paper writingAttempt 7 · agent:claude-opus-5-5
  91. awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
  92. Resume requested
  93. Stage started · Paper writingAttempt 8 · agent:claude-opus-5-5
  94. Stage completed · Paper writing
  95. Stage started · ReviewAttempt 1 · agent:claude-fable-5-1
  96. awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
  97. Resume requested
  98. Stage started · ReviewAttempt 2 · agent:claude-fable-5-1
  99. Awaiting approval · Review
  100. Approved · Review
  101. Run completereport=/Users/haodingresearch/Desktop/auto-research/runs/studies-opus55/computer_science-en/runs/4689bab459a3/REPORT.md