Under asymmetric misclassification costs, does post-hoc probability calibration with the theoretical cost-optimal threshold beat directly tuning the decision threshold on the same held-out validation data?
Paper
Calibrate or Tune the Cut-off? A Matched-Budget Contest Between Two Recipes for Cost-Sensitive Thresholding
Abstract
Two standard recipes prescribe incompatible uses of the same held-out sample when misclassification costs are asymmetric. Decision theory says to fit a post-hoc calibration map and decide at the fixed Bayes cut-off ; the thresholding literature says to skip calibration and search the cut-off minimising empirical held-out cost. We run them head to head on the real Spambase benchmark (4,601 e-mails) with matched data and tuning budgets: 15 seeds, three tuned model families, five cost ratios, four held-out sizes. They are indistinguishable. At the pre-registered operating point the paired difference is per e-mail (95% CI , ), and two one-sided tests reject any difference exceeding 5% of the operating cost (). Our hypothesis — calibration wins, and wins harder as costs grow more asymmetric — is refuted. What moves cost is whether the cut-off is moved at all, how much held-out data there is, and how distorted the base learner's scores are: with 100 held-out e-mails both recipes are significantly worse than spending none.
§Ask about this paper
Grounded in this runQuestions are answered from this run's record — the paper, the executed analysis, the data provenance and the referee report. Nothing else is consulted.
This run is finished, so the recorded paper cannot be rewritten. Questions are still answered from its record.
Pipeline
1.Literature review
Literature review complete (live:agent:claude-opus-5): 12 key papers, 14 grounded sources, 6 open gaps.
Summary
Three literatures answer the same practical question in incompatible ways, and they are almost never tested against each other on the same held-out sample. Cost-sensitive decision theory (Elkan, 2001) shows that if a model emits the true posterior, the Bayes decision under a 2x2 cost matrix is a fixed cut-off t* = c_fp/(c_fp + c_fn); nothing needs to be learned about the operating point. Provost and Fawcett (2001) give the dual, geometric account through the ROC convex hull and stress that operating conditions are usually imprecise. Against this, Sheng and Ling (2006) argue that accurate probabilities are unnecessary: one can simply search the threshold that minimises empirical cost on held-out data, and they report that this beats theoretical thresholding on UCI benchmarks. That empirical-thresholding recipe is now the software default. The calibration literature takes the opposite view of how held-out data should be spent. Zadrozny and Elkan (2002) introduced isotonic (PAV) calibration; Niculescu-Mizil and Caruana (2005) documented which learners are distorted and found isotonic superior on large calibration sets and Platt's sigmoid superior on small ones; Guo et al. (2017) revived the agenda for deep networks; Filho et al. (2023) survey the field and name cost-sensitive decision-making as the motivating application. A third strand qualifies both. Sahoo et al. (2021) show that average calibration is insufficient for threshold decisions and define threshold calibration as the exact condition, while Perez-Lebel et al. (2023) show that grouping loss survives marginal calibration; conversely Blasiok et al. (2023) explain why models trained to a proper loss are often already close to calibrated. Finally, Cawley and Talbot (2010) and Schneider et al. (2025) establish that selection over-fitting is driven by the variance of the selection criterion and is frequently as large as the differences between learning algorithms. This last point is what makes the contest decidable. Empirical threshold search is a one-dimensional empirical risk minimisation whose objective is a step function; its effective sample size is only the held-out points near the cut-off, which shrinks as the cost ratio pushes t* towards one, whereas a calibration map pools the entire held-out sample. Theory therefore predicts a crossover governed by held-out size and cost ratio, and predicts that calibration should help most for the learners whose scores are most distorted. No study appears to have measured that crossover with matched data, matched tuning budgets and paired seeds, or to have reported regret against the best achievable test threshold, which is what separates the decision rule from the quality of the underlying model.
Key papers (12)
- The Foundations of Cost-Sensitive Learning. Charles Elkan; IJCAI-01, pp. 973-978, 2001.Derives the theoretical cost-optimal cut-off t* = c_fp/(c_fp + c_fn) for a calibrated posterior, and the equivalent class-rebalancing transformation. It is the theoretical arm of the contest and is explicit that the result presupposes true conditional probabilities.
- Thresholding for Making Classifiers Cost-sensitive. Victor S. Sheng and Charles X. Ling; AAAI-06, pp. 476-481, 2006.The opposing position: search the threshold empirically on held-out instances, on the argument that this removes the need for accurate probability estimates. It is the baseline the proposed study puts under a controlled test.
- Predicting Good Probabilities With Supervised Learning. Alexandru Niculescu-Mizil and Rich Caruana; ICML-05, pp. 625-632, 2005.The standard empirical reference on which learners produce distorted scores and on the Platt-versus-isotonic trade-off as a function of calibration-set size. It motivates both the model-family ablation and the calibration-set-size ablation.
- Transforming Classifier Scores into Accurate Multiclass Probability Estimates. Bianca Zadrozny and Charles Elkan; KDD-02, pp. 694-699, 2002.Introduces isotonic-regression calibration, one of the two post-hoc maps compared here, and frames calibration explicitly as a prerequisite for cost-based thresholding.
- On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Gavin C. Cawley and Nicola L. C. Talbot; JMLR 11, pp. 2079-2107, 2010.Establishes that the variance of a selection criterion, not just its bias, drives over-fitting of the selection step, and that the effect is often as large as differences between algorithms. This is the mechanism by which empirical threshold search is expected to fail at extreme cost ratios.
- Reliable Decisions with Threshold Calibration. Roshni Sahoo, Shengjia Zhao, Alyssa Chen and Stefano Ermon; NeurIPS 34, 2021.Shows that average calibration does not guarantee accurate loss prediction for threshold decisions and defines threshold calibration as the necessary and sufficient condition, which bounds how much a marginally calibrated model can be trusted at a fixed cut-off.
- Beyond calibration: estimating the grouping loss of modern neural networks. Alexandre Perez-Lebel, Marine Le Morvan and Gael Varoquaux; ICLR 2023, 2023.Demonstrates that grouping loss survives marginal calibration, so a calibrated score can still be far from the true posterior for individual instances. It sets an expectation that calibration will not eliminate decision regret entirely.
- When Does Optimizing a Proper Loss Yield Calibration?. Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu and Preetum Nakkiran; NeurIPS 36, 2023.Gives the positive counterpart: local optimality of a proper loss under Lipschitz post-processing implies smooth calibration, predicting that log-loss-trained models will benefit less from post-hoc calibration than distorted learners.
- Robust Classification for Imprecise Environments. Foster Provost and Tom Fawcett; Machine Learning 42(3), pp. 203-231, 2001.The ROC convex hull view of choosing an operating point under a given cost slope, and the argument for evaluating across a range of cost ratios rather than a single point, which motivates the cost-ratio sweep.
- Classifier calibration: a survey on how to assess and improve predicted class probabilities. Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull and Peter Flach; Machine Learning 112, pp. 3211-3260, 2023.The current survey of post-hoc calibration methods and their assessment, and the source of the conventional claim that calibration is what enables optimal cost-sensitive decisions.
- Overtuning in Hyperparameter Optimization. Lennart Schneider, Bernd Bischl and Matthias Feurer; AutoML 2025 (arXiv:2506.19540), 2025.A modern quantification of how often and how badly a tuning step over-fits its validation signal, supporting the framing of empirical threshold search as a selection procedure with a winner's-curse penalty.
- Scikit-learn: Machine Learning in Python. Fabian Pedregosa et al.; JMLR 12, pp. 2825-2830, 2011.Provides the reference implementations of the base learners, of sigmoid and isotonic calibration, and of tuned-threshold classification, and is the reason empirical threshold tuning is the de facto default in applied work.
Methods in use
- Theoretical cost-optimal thresholding at t* = c_fp/(c_fp + c_fn) applied to a model's posterior (Elkan, 2001)
- Empirical threshold search: pick the cut-off minimising total cost on held-out or cross-validated instances (Sheng and Ling, 2006; scikit-learn's tuned-threshold classifier)
- Post-hoc calibration maps: Platt/sigmoid scaling, isotonic regression via pool-adjacent-violators, temperature scaling, beta calibration
- ROC convex hull and cost curves for choosing and visualising operating points across cost ratios (Provost and Fawcett, 2001)
- Class rebalancing, instance reweighting and resampling as alternative routes to cost sensitivity
- Train / calibration / test protocols and nested cross-validation to keep the selection step off the evaluation data (Cawley and Talbot, 2010)
- Evaluation with expected cost per instance under a stated cost matrix, plus log loss, Brier score, expected calibration error and AUROC as diagnostics
- Reporting practice: several random seeds, mean and standard deviation, paired within-seed comparisons, equal hyperparameter search budgets across model families, and ablations over the design choices
Open gaps
- No head-to-head test of the two recipes on the same held-out sample: calibrate-then-use-t* versus search-the-threshold-empirically, with identical base models, identical data budgets and paired seeds. The two literatures give opposite advice and both are widely followed.
- The predicted crossover has never been mapped. Empirical threshold search has an effective sample size equal to the held-out points near the cut-off, which shrinks as the cost ratio pushes t* towards one, so its advantage should depend jointly on held-out size and cost ratio; the location of the crossover is unknown.
- Whether the answer is conditional on the base learner. Distortion is learner-specific, so calibration should help most where scores are most distorted (boosted trees, forests) and least for log-loss-trained linear models; no study reports this interaction under a decision loss.
- Isotonic versus sigmoid guidance was derived for probabilistic metrics, not for expected cost at an extreme cut-off, where isotonic's flat steps can place substantial probability mass exactly at the threshold.
- Decision regret relative to the best achievable threshold on the test data is rarely reported, although it is what isolates the quality of the decision rule from the quality of the underlying model.
- The size of the residual penalty implied by threshold calibration and grouping loss - how far a marginally calibrated model still is from the oracle operating point on standard tabular benchmarks - has not been quantified.
Grounded search sources (14) · mode: live:agent:claude-opus-5
- Elkan, C. (2001) The Foundations of Cost-Sensitive Learning. IJCAI-01, 973-978.
- Provost, F. and Fawcett, T. (2001) Robust Classification for Imprecise Environments. Machine Learning 42(3), 203-231.
- Sheng, V.S. and Ling, C.X. (2006) Thresholding for Making Classifiers Cost-sensitive. AAAI-06, 476-481.
- Zadrozny, B. and Elkan, C. (2002) Transforming Classifier Scores into Accurate Multiclass Probability Estimates. KDD-02, 694-699.
- Niculescu-Mizil, A. and Caruana, R. (2005) Predicting Good Probabilities With Supervised Learning. ICML-05, 625-632.
- Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q. (2017) On Calibration of Modern Neural Networks. ICML-17.
- Sahoo, R., Zhao, S., Chen, A. and Ermon, S. (2021) Reliable Decisions with Threshold Calibration. NeurIPS 34.
- Perez-Lebel, A., Le Morvan, M. and Varoquaux, G. (2023) Beyond calibration: estimating the grouping loss of modern neural networks. ICLR 2023.
- Blasiok, J., Gopalan, P., Hu, L. and Nakkiran, P. (2023) When Does Optimizing a Proper Loss Yield Calibration? NeurIPS 36.
- Cawley, G.C. and Talbot, N.L.C. (2010) On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. JMLR 11, 2079-2107.
- Schneider, L., Bischl, B. and Feurer, M. (2025) Overtuning in Hyperparameter Optimization. AutoML 2025, arXiv:2506.19540.
- Filho, T.S., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M. and Flach, P. (2023) Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning 112, 3211-3260.
- Pedregosa, F. et al. (2011) Scikit-learn: Machine Learning in Python. JMLR 12, 2825-2830.
- Hopkins, M., Reeber, E., Forman, G. and Suermondt, J. (1999) Spambase [Dataset]. UCI Machine Learning Repository. DOI 10.24432/C53G6X.
2.Ideation
3 candidate ideas generated; selected: “Calibrate, or tune the cut-off? A matched-budget contest between the two standard recipes for cost-sensitive thresholding”.
- Idea 1Selected
Calibrate, or tune the cut-off? A matched-budget contest between the two standard recipes for cost-sensitive thresholding
Hypothesis. Given a fixed base classifier and a single held-out sample of size m, spending that sample on a post-hoc calibration map and then deciding at the theoretical Bayes cut-off t* = c_fp/(c_fp + c_fn) (Elkan, 2001) yields strictly lower expected test cost than spending the same sample on a direct empirical search for the cost-minimising cut-off (Sheng and Ling, 2006), and the advantage grows with the cost ratio R = c_fp/c_fn and shrinks with m. Formally: the paired difference D = Cost(empirical-threshold) - Cost(calibrate-then-t*) in test cost per e-mail, averaged over seeds, is positive, and its mean increases monotonically in R over R in {1, 2, 5, 10, 20} at fixed m. The hypothesis is refuted if D is not significantly greater than zero at R = 10 (the reference operating point) under a paired test across seeds, or if D fails to increase with R.
Dataset. Spambase e-mail spam classification benchmark uci_spambase
Test plan, novelty and risk
Test plan. Data: uci_spambase, all 4,601 e-mails, target is_spam (1,813 positives), 57 numeric predictors, no missing values; spam is the positive class, a false positive discards legitimate mail and carries cost c_fp = R, a false negative carries c_fn = 1. Design: for each of 10 seeds, a stratified 4-way split into train (45%), tune (15%), decision (20%) and test (20%). Base learners, each tuned on the tune split under an identical budget of 12 random hyperparameter draws scored by log loss (a proper loss, so selection is not entangled with the decision rule): (a) L2-regularised logistic regression on standardised features (C log-uniform in [1e-3, 1e2]), (b) random forest (n_estimators, max_depth, min_samples_leaf, max_features), (c) histogram gradient boosting (learning_rate, max_leaf_nodes, min_samples_leaf, l2_regularization). Decision rules, all fitted on the SAME decision split (subsampled to m in {100, 250, 500, full}) so the data budget is matched exactly: R0 fixed cut-off 0.5; R1 raw score with theoretical t*; R2 isotonic calibration then t*; R3 sigmoid (Platt) calibration then t*; R4 empirical cut-off minimising decision-split cost on raw scores; R5 isotonic calibration then empirical cut-off on the calibrated scores. Primary metric: expected cost per test e-mail, and regret against the oracle cut-off chosen on the test split itself, which isolates the decision rule from model quality. Secondary: log loss, Brier score, AUROC, and the fitted cut-off. Estimator and inference: mean +/- standard deviation over 10 seeds, paired per-seed differences R4 - R2, a paired two-sided t-test and a Wilcoxon signed-rank test across seeds. Ablations: cost ratio R in {1, 2, 5, 10, 20}; decision-split size m; calibration family (isotonic vs sigmoid); base learner. Refutation: if mean(D) <= 0 at R = 10 with m = full, or the paired test does not reject at the 5% level, the hypothesis is refuted and the paper reports that empirical thresholding is at least as good, which is itself a useful negative result. Compute: ~10 seeds x 36 model fits on <= 2,100 training rows plus vectorised post-hoc evaluation; well under two minutes.
Novelty. Elkan (2001) and Sheng and Ling (2006) prescribe incompatible uses of the same held-out data, and both recipes are in daily use - the second is the default in scikit-learn. The literature contains no controlled, matched-budget, paired-seed comparison of the two on identical splits, and no study reports decision regret against the oracle cut-off, which is what separates the decision rule from the underlying model. Cawley and Talbot (2010) and Schneider et al. (2025) supply the mechanism that makes the contest decidable in advance: the empirical cut-off is a selection made on a step-function criterion whose effective sample size is only the held-out points near the cut-off, and that number collapses as R pushes t* towards one.
Expected outcome. At R = 1 the two recipes should be indistinguishable. At R = 10 and R = 20 the empirical cut-off should show a visible winner's-curse penalty, particularly at m = 100-250, while calibration plus t* should be close to the oracle. The gain from calibration should be largest for the tree ensembles (whose scores are distorted) and small or absent for logistic regression trained on log loss, consistent with Niculescu-Mizil and Caruana (2005) and Blasiok et al. (2023). A plausible refutation is that Spambase is so separable (AUROC around 0.98) that all rules land near the oracle and the differences are within noise.
Main risk. Spambase is easy and its score distribution is strongly bimodal, so few test points lie near any plausible cut-off and every rule may achieve near-oracle cost, leaving differences inside the seed-to-seed standard deviation. The study is designed so that this outcome is still reportable: it would be evidence that the choice of recipe is immaterial on well-separated problems, and the m and R sweeps localise where, if anywhere, it starts to matter. A second risk is that isotonic regression at small m produces flat steps straddling t*, making its behaviour erratic; the sigmoid arm guards against attributing that artefact to calibration in general.
- Idea 2
Cost-sensitive learning at training time versus at decision time: does Elkan's rebalancing theorem hold empirically for modern tabular learners?
Hypothesis. For a fixed learner and a fixed cost ratio, making a classifier cost-sensitive by reweighting the training distribution in proportion to costs (Elkan's rebalancing rule) attains no lower expected test cost than leaving the learner cost-blind and moving the decision threshold post hoc; that is, the paired difference in test cost per case (reweighted-at-training minus cost-blind-plus-threshold) is greater than or equal to zero on average. Refuted if reweighting is significantly cheaper than post-hoc thresholding under a paired test across seeds at the reference cost ratio.
Dataset. Adult (Census Income): US income prediction benchmark uci_adult_income
Test plan, novelty and risk
Test plan. Data: uci_adult_income, 48,842 records, target income > 50K (about 24% positive), excluding fnlwgt as a predictor because it is a survey design weight; median or most-frequent imputation for the '?' entries in workclass, occupation and native_country; one-hot encoding for categorical columns. Design: 5 seeds, stratified 60/20/20 train/validation/test split, and a 12,000-row training subsample per seed to stay inside the compute budget (stated explicitly in the paper). Learners: L2 logistic regression and histogram gradient boosting, each tuned on validation by log loss under an identical budget of 10 random draws. Arms: (a) cost-blind training plus theoretical threshold t*; (b) cost-blind training plus empirical threshold on validation; (c) class-weighted training at weight ratio c_fp/c_fn plus argmax decision; (d) class-weighted training plus theoretical threshold. Cost ratios: the cost of a false positive relative to a false negative in {1, 3, 10}. Metrics: expected cost per case, mean +/- standard deviation over seeds, paired per-seed differences with a two-sided paired t-test, plus AUROC and log loss to confirm that reweighting changes calibration rather than ranking.
Novelty. Elkan's rebalancing theorem is stated for the true posterior and is widely cited as licence to use class weights instead of thresholds, but the equivalence is not guaranteed for a finite-capacity learner whose loss is only approximately proper. A direct paired test on a standard benchmark, holding the tuning budget fixed, is not in the literature in this form.
Expected outcome. Reweighting and thresholding should be close; any difference should appear mainly as a calibration shift, with AUROC essentially unchanged. The most likely finding is that post-hoc thresholding is at least as good and cheaper to compute, since it needs no refit per cost ratio.
Main risk. The training subsample and one-hot encoding may make the gradient-boosting arm noisier than the seed count can resolve, and the comparison is somewhat less sharp than idea 1 because both arms are defensible in the literature, so a null result would be less informative.
- Idea 3
Does in-distribution model selection survive a natural subpopulation shift? Selecting tabular models on white wines and deploying them on red
Hypothesis. Across a family of tuned model configurations, the ranking induced by in-distribution validation performance transfers to a natural subpopulation shift: the Spearman rank correlation between validation performance measured on held-out white wines and test performance measured on red wines is positive and large (rho >= 0.7). Refuted if the rank correlation is not significantly positive, or if the configuration selected in distribution is worse under shift than the median configuration.
Dataset. Vinho Verde wine quality: physicochemical measurements and sensory scores uci_wine_quality
Test plan, novelty and risk
Test plan. Data: uci_wine_quality, 4,898 white and 1,599 red Vinho Verde wines, 11 physicochemical predictors, outcome binarised as good (quality >= 7) versus not, which is the standard framing for this benchmark. Design: 5 seeds; train and validate only on white wines (70/30 stratified split of the 4,898), then evaluate on all 1,599 red wines as the shifted target domain, and on a held-out white test set as the in-distribution reference. Configurations: about 30 pipelines spanning L2 logistic regression at several regularisation strengths, random forests and histogram gradient boosting at several depths, all fitted under an identical budget. Metrics: balanced accuracy, AUROC and log loss in and out of distribution; Spearman rank correlation between the in-distribution validation metric and the out-of-distribution test metric across configurations, with a bootstrap confidence interval over configurations and mean +/- standard deviation across seeds; and the selection regret, defined as the out-of-distribution loss of the configuration chosen in distribution minus the best achievable out-of-distribution loss.
Novelty. The accuracy-on-the-line phenomenon is documented for vision benchmarks; whether it holds for tabular data under a naturally occurring subpopulation split, and whether validation-based model selection therefore remains safe under shift, is much less studied.
Expected outcome. Rank correlation plausibly positive but well below the near-perfect linear relationships reported in vision, with a non-trivial selection regret, because the red and white conditional distributions differ.
Main risk. There is only one shift pair, so the result is a single case study and cannot support a general claim; the red-white shift may also be so severe that all configurations collapse to near-chance out of distribution, making the rank correlation uninformative.
Ranking rationale
Idea 1 is placed first because it resolves an explicit contradiction between two heavily cited and heavily used prescriptions, it has a mechanism that predicts the direction and the moderators of the effect in advance (selection variance at extreme cut-offs), it is falsifiable in both directions, and Spambase is small enough that ten seeds, three tuned learner families, six decision rules, five cost ratios and four held-out sizes all fit comfortably inside a two-minute budget with room for the paired statistics. Idea 2 tests a real and widely misapplied theorem but both of its arms are defensible, so the most likely outcome is a null that is harder to make interesting, and the Adult pipeline costs more compute for less resolution. Idea 3 is the most speculative: it asks a fashionable question, but the wine data offer exactly one shift pair, so any answer is a single case study rather than evidence about tabular data in general.
3.Experimentation
Experiment succeeded in 1 round (62.127s).
Data provenance
Spambase e-mail spam classification benchmark
- Source
- UCI Machine Learning Repository (dataset 94), donated by Hewlett-Packard Labs
- Citation
- Hopkins, M., Reeber, E., Forman, G. and Suermondt, J. (1999) Spambase [Dataset]. UCI Machine Learning Repository. DOI: 10.24432/C53G6X. Collected at Hewlett-Packard Labs, Palo Alto, and donated 30 June 1999. No peer-reviewed introductory paper exists for this dataset -- the UCI page refers only to an internal HP technical report -- so the dataset itself is the citable primary source.
- Licence
- Creative Commons Attribution 4.0 International (CC BY 4.0): the UCI dataset page states "This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given."
- File
- https://archive.ics.uci.edu/static/public/94/spambase.zip
- Observations
- 4,601
- Variables (58)
- word_freq_make, word_freq_address, word_freq_all, word_freq_3d, word_freq_our, word_freq_over, word_freq_remove, word_freq_internet, word_freq_order, word_freq_mail, word_freq_receive, word_freq_will, word_freq_people, word_freq_report, word_freq_addresses, word_freq_free, word_freq_business, word_freq_email, word_freq_you, word_freq_credit, word_freq_your, word_freq_font, word_freq_000, word_freq_money, word_freq_hp, word_freq_hpl, word_freq_george, word_freq_650, word_freq_lab, word_freq_labs, word_freq_telnet, word_freq_857, word_freq_data, word_freq_415, word_freq_85, word_freq_technology, word_freq_1999, word_freq_parts, word_freq_pm, word_freq_direct, word_freq_cs, word_freq_meeting, word_freq_original, word_freq_project, word_freq_re, word_freq_edu, word_freq_table, word_freq_conference, char_freq_;, char_freq_(, char_freq_[, char_freq_!, char_freq_$, char_freq_#, capital_run_length_average, capital_run_length_longest, capital_run_length_total, is_spam
- Units
- The 48 word_freq_* columns are the percentage of words in the e-mail matching that token, and the 6 char_freq_* columns the percentage of characters matching that character, so both are on a 0-100 scale. capital_run_length_average is the mean length of uninterrupted runs of capitals, capital_run_length_longest the longest such run, and capital_run_length_total the total number of capital letters. is_spam is 1 for spam and 0 for legitimate mail. Column names are taken from the archive's spambase.names file by the loader.
- Retrieved
- 8 Sept 2026, 15:44
- Checksum
- sha256 813ac1df8effac70…
Real, publicly available data — downloaded and checksum-verified before the analysis ran, never simulated.
Result (RESULT_JSON)
{
"dataset": "uci_spambase",
"n_rows_loaded": 4601,
"n_analysed": 4601,
"n_features": 57,
"n_positive": 1813,
"prevalence": 0.39404477287546186,
"n_seeds": 15,
"n_train": 2070,
"n_tune": 690,
"n_decision": 920,
"n_test": 921,
"hpo_budget_per_family": 10,
"families": [
"logreg",
"random_forest",
"hist_gbdt"
],
"rules": [
"fixed_half",
"raw_tstar",
"isotonic_tstar",
"platt_tstar",
"empirical_raw",
"isotonic_empirical"
],
"cost_ratios": [
1,
2,
5,
10,
20
],
"decision_sizes": [
100,
250,
500,
920
],
"equivalence_margin_fraction": 0.05,
"primary_endpoint": {
"cost_ratio": 10,
"decision_size": 920,
"pooled_over": [
"logreg",
"random_forest",
"hist_gbdt"
],
"mean_diff": 0.00028954035468693463,
"sd_diff": 0.0027228726563596145,
"ci95_low": -0.0012183362624367197,
"ci95_high": 0.0017974169718105886,
"t_stat": 0.41183893383274817,
"p_two_sided": 0.6866935687822401,
"wilcoxon_p": 0.25575474881618,
"cohens_dz": 0.10633635547027012,
"reference_mean_cost": 0.18161418747737967,
"tost_p_equivalent": 2.7517727494105093e-9,
"seeds_a_worse": 6
},
"monotonicity": {
"mean_diff_by_cost_ratio": [
-0.00004825672578115577,
0.00009651345156231154,
-0.00012064181445288942,
0.00028954035468693463,
-0.00016889854023404704
],
"monotone_increasing": false,
"spearman_rho": -0.3,
"spearman_p": 0.623837664781073
},
"by_cost_ratio_full_decision_split": {
"1.0": {
"cost": {
"fixed_half": {
"mean": 0.06394016166003139,
"sd": 0.004922084648957818
},
"raw_tstar": {
"mean": 0.06394016166003139,
"sd": 0.004922084648957818
},
"isotonic_tstar": {
"mean": 0.06355410785378214,
"sd": 0.005875498774938411
},
"platt_tstar": {
"mean": 0.064133188563156,
"sd": 0.005135188762906649
},
"empirical_raw": {
"mean": 0.06350585112800097,
"sd": 0.005679654673053403
},
"isotonic_empirical": {
"mean": 0.06391603329714081,
"sd": 0.005593224554084621
}
},
"regret": {
"fixed_half": {
"mean": 0.004970442755459042,
"sd": 0.0015200047906292994
},
"raw_tstar": {
"mean": 0.004970442755459042,
"sd": 0.0015200047906292994
},
"isotonic_tstar": {
"mean": 0.004584388949209795,
"sd": 0.0017446944466633935
},
"platt_tstar": {
"mean": 0.005163469658583664,
"sd": 0.002108003963519388
},
"empirical_raw": {
"mean": 0.004536132223428639,
"sd": 0.0016857816626454933
},
"isotonic_empirical": {
"mean": 0.004946314392568465,
"sd": 0.002871608269308558
}
},
"regret_ratio": {
"fixed_half": 0.08428805237315874,
"raw_tstar": 0.08428805237315874,
"isotonic_tstar": 0.07774140752864155,
"platt_tstar": 0.08756137479541733,
"empirical_raw": 0.07692307692307691,
"isotonic_empirical": 0.08387888707037643
},
"oracle_cost": {
"mean": 0.05896971890457234,
"sd": 0.004790688904081222
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": -0.00004825672578115577,
"sd_diff": 0.0007600023953146506,
"ci95_low": -0.00046913202385149514,
"ci95_high": 0.0003726185722891836,
"t_stat": -0.24591698190048009,
"p_two_sided": 0.8093151976775883,
"wilcoxon_p": 0.8576043783480918,
"cohens_dz": -0.06349549169667666,
"reference_mean_cost": 0.06355410785378214,
"tost_p_equivalent": 1.1284064725275199e-10,
"seeds_a_worse": 5
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.0003860538062492476,
"sd_diff": 0.0026717658364766705,
"ci95_low": -0.001865628409873647,
"ci95_high": 0.0010935207973751518,
"t_stat": -0.559622382294945,
"p_two_sided": 0.5845799698716757,
"wilcoxon_p": 0.6492989349095808,
"cohens_dz": -0.14449387778621614,
"reference_mean_cost": 0.06394016166003139,
"tost_p_equivalent": 0.0005684302492739862,
"seeds_a_worse": 9
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.0004343105320304034,
"sd_diff": 0.0025194051737289364,
"ci95_low": -0.0018295106129645176,
"ci95_high": 0.0009608895489037108,
"t_stat": -0.6676486478538962,
"p_two_sided": 0.515213595710547,
"wilcoxon_p": 0.7266877064190556,
"cohens_dz": -0.17238613961706936,
"reference_mean_cost": 0.06394016166003139,
"tost_p_equivalent": 0.0004064403742656408,
"seeds_a_worse": 7
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": 0.00019302690312462216,
"sd_diff": 0.0018447887798294952,
"ci95_low": -0.0008285830712309275,
"ci95_high": 0.001214636877480172,
"t_stat": 0.40524421513488934,
"p_two_sided": 0.6914227351434702,
"wilcoxon_p": 0.8199228885386955,
"cohens_dz": 0.10463360642428814,
"reference_mean_cost": 0.06394016166003139,
"tost_p_equivalent": 0.000009668891590238731,
"seeds_a_worse": 7
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": 0.0005790807093738697,
"sd_diff": 0.0024842506992282264,
"ci95_low": -0.0007966514724783387,
"ci95_high": 0.001954812891226078,
"t_stat": 0.9027953355157464,
"p_two_sided": 0.3819052962756482,
"wilcoxon_p": 0.3792364621156361,
"cohens_dz": 0.23310075329908156,
"reference_mean_cost": 0.06355410785378214,
"tost_p_equivalent": 0.0005951175183072843,
"seeds_a_worse": 9
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0,
"sd_diff": 0,
"ci95_low": 0,
"ci95_high": 0,
"t_stat": null,
"p_two_sided": null,
"wilcoxon_p": null,
"cohens_dz": null,
"reference_mean_cost": 0.06394016166003139,
"tost_p_equivalent": null,
"seeds_a_worse": 0
}
},
"tstar": 0.5
},
"2.0": {
"cost": {
"fixed_half": {
"mean": 0.08968512486427797,
"sd": 0.007166019325952983
},
"raw_tstar": {
"mean": 0.09113282663771265,
"sd": 0.007018261362697014
},
"isotonic_tstar": {
"mean": 0.08748944384123539,
"sd": 0.005318257754413614
},
"platt_tstar": {
"mean": 0.09026420557365183,
"sd": 0.007271781242054124
},
"empirical_raw": {
"mean": 0.0875859572927977,
"sd": 0.005482048400264423
},
"isotonic_empirical": {
"mean": 0.08770659910725058,
"sd": 0.005520598957002077
}
},
"regret": {
"fixed_half": {
"mean": 0.008469055374592834,
"sd": 0.003574512756578393
},
"raw_tstar": {
"mean": 0.009916757148027509,
"sd": 0.0038903678392728223
},
"isotonic_tstar": {
"mean": 0.006273374351550247,
"sd": 0.0025579803961732193
},
"platt_tstar": {
"mean": 0.009048136083966703,
"sd": 0.00354876815384876
},
"empirical_raw": {
"mean": 0.0063698878031125585,
"sd": 0.0028127917810677446
},
"isotonic_empirical": {
"mean": 0.0064905296175654485,
"sd": 0.0026222800445152975
}
},
"regret_ratio": {
"fixed_half": 0.10427807486631012,
"raw_tstar": 0.12210338680926916,
"isotonic_tstar": 0.077243018419489,
"platt_tstar": 0.1114081996434938,
"empirical_raw": 0.0784313725490196,
"isotonic_empirical": 0.07991681521093286
},
"oracle_cost": {
"mean": 0.08121606948968514,
"sd": 0.006234009110628631
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0.00009651345156231154,
"sd_diff": 0.0010811706833124182,
"ci95_low": -0.000502218916137181,
"ci95_high": 0.0006952458192618041,
"t_stat": 0.34573171133407055,
"p_two_sided": 0.7346845264613661,
"wilcoxon_p": 0.822801683757703,
"cohens_dz": 0.08926754401684303,
"reference_mean_cost": 0.08748944384123539,
"tost_p_equivalent": 1.9164412525005974e-10,
"seeds_a_worse": 5
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.003643382796477261,
"sd_diff": 0.0035788726644714925,
"ci95_low": -0.00562529641767265,
"ci95_high": -0.001661469175281871,
"t_stat": -3.9427949015053416,
"p_two_sided": 0.0014724594994596882,
"wilcoxon_p": 0.003836748208591641,
"cohens_dz": -1.0180252660694469,
"reference_mean_cost": 0.09113282663771265,
"tost_p_equivalent": 0.16988753554981884,
"seeds_a_worse": 2
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.0035468693449149493,
"sd_diff": 0.0034800809757286346,
"ci95_low": -0.005474073952424522,
"ci95_high": -0.0016196647374053764,
"t_stat": -3.947312145848291,
"p_two_sided": 0.0014594469126746291,
"wilcoxon_p": 0.004266724822176128,
"cohens_dz": -1.0191916135435128,
"reference_mean_cost": 0.09113282663771265,
"tost_p_equivalent": 0.1400044959026157,
"seeds_a_worse": 2
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.0008686210640608038,
"sd_diff": 0.0034438657437666016,
"ci95_low": -0.002775770344586313,
"ci95_high": 0.001038528216464705,
"t_stat": -0.97685425785299,
"p_two_sided": 0.34521999929981917,
"wilcoxon_p": 0.3298199269621632,
"cohens_dz": -0.25222268482242904,
"reference_mean_cost": 0.09113282663771265,
"tost_p_equivalent": 0.0004931222238149207,
"seeds_a_worse": 7
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": 0.0027747617324164566,
"sd_diff": 0.0030578085040714287,
"ci95_low": 0.001081403825221618,
"ci95_high": 0.004468119639611295,
"t_stat": 3.514479721353912,
"p_two_sided": 0.003434938762747158,
"wilcoxon_p": 0.005146932125209912,
"cohens_dz": 0.9074347620924924,
"reference_mean_cost": 0.08748944384123539,
"tost_p_equivalent": 0.031119971152721252,
"seeds_a_worse": 12
},
"fixed_half_vs_raw_tstar": {
"mean_diff": -0.001447701773434673,
"sd_diff": 0.005693037426607026,
"ci95_low": -0.004600400815727601,
"ci95_high": 0.0017049972688582544,
"t_stat": -0.9848740555582616,
"p_two_sided": 0.34140168038916885,
"wilcoxon_p": 0.27994840378225916,
"cohens_dz": -0.2542933876859271,
"reference_mean_cost": 0.09113282663771265,
"tost_p_equivalent": 0.026421449976137608,
"seeds_a_worse": 6
}
},
"tstar": 0.6666666666666666
},
"5.0": {
"cost": {
"fixed_half": {
"mean": 0.16692001447701774,
"sd": 0.016913522628277538
},
"raw_tstar": {
"mean": 0.1410544094583183,
"sd": 0.011767918405270828
},
"isotonic_tstar": {
"mean": 0.13444323802629995,
"sd": 0.01289133130915582
},
"platt_tstar": {
"mean": 0.13422608276028475,
"sd": 0.011205391641119522
},
"empirical_raw": {
"mean": 0.13432259621184706,
"sd": 0.012691643387952437
},
"isotonic_empirical": {
"mean": 0.1350464470985644,
"sd": 0.012535872968822147
}
},
"regret": {
"fixed_half": {
"mean": 0.046543612015924725,
"sd": 0.011901790399413378
},
"raw_tstar": {
"mean": 0.02067800699722524,
"sd": 0.005001659726830248
},
"isotonic_tstar": {
"mean": 0.014066835565206904,
"sd": 0.005883561649464041
},
"platt_tstar": {
"mean": 0.013849680299191704,
"sd": 0.004713904814396663
},
"empirical_raw": {
"mean": 0.013946193750754014,
"sd": 0.006040495325436543
},
"isotonic_empirical": {
"mean": 0.01467004463747135,
"sd": 0.005636660608978286
}
},
"regret_ratio": {
"fixed_half": 0.3866506313890559,
"raw_tstar": 0.17177791140509122,
"isotonic_tstar": 0.11685708558829426,
"platt_tstar": 0.1150531168570856,
"empirical_raw": 0.11585488073762278,
"isotonic_empirical": 0.12186810984165164
},
"oracle_cost": {
"mean": 0.12037640246109305,
"sd": 0.010653157231696115
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": -0.00012064181445288942,
"sd_diff": 0.0009039514704129992,
"ci95_low": -0.0006212334532380293,
"ci95_high": 0.0003799498243322504,
"t_stat": -0.5168902906024486,
"p_two_sided": 0.6133013417364487,
"wilcoxon_p": 0.4995336482572515,
"cohens_dz": -0.13346049915463973,
"reference_mean_cost": 0.13444323802629995,
"tost_p_equivalent": 4.6981122223505504e-14,
"seeds_a_worse": 3
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.006611171432018339,
"sd_diff": 0.003833839916195467,
"ci95_low": -0.00873428121092199,
"ci95_high": -0.004488061653114689,
"t_stat": -6.678671362089236,
"p_two_sided": 0.000010461470324908948,
"wilcoxon_p": 0.00006103515625,
"cohens_dz": -1.7244255306776013,
"reference_mean_cost": 0.1410544094583183,
"tost_p_equivalent": 0.33118847572189625,
"seeds_a_worse": 0
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.006731813246471229,
"sd_diff": 0.0038541236371984378,
"ci95_low": -0.008866155775660576,
"ci95_high": -0.0045974707172818825,
"t_stat": -6.7647545973157825,
"p_two_sided": 0.000009099754113789593,
"wilcoxon_p": 0.00006103515625,
"cohens_dz": -1.746652126438939,
"reference_mean_cost": 0.1410544094583183,
"tost_p_equivalent": 0.3759281551483985,
"seeds_a_worse": 0
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.006828326698033538,
"sd_diff": 0.005107174668672919,
"ci95_low": -0.00965658575909227,
"ci95_high": -0.004000067636974806,
"t_stat": -5.178204643394206,
"p_two_sided": 0.0001400619383691071,
"wilcoxon_p": 0.002300512942081378,
"cohens_dz": -1.3370066898079784,
"reference_mean_cost": 0.1410544094583183,
"tost_p_equivalent": 0.4336566450525945,
"seeds_a_worse": 2
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.0002171552660151982,
"sd_diff": 0.006190330897800492,
"ci95_low": -0.0036452462533957874,
"ci95_high": 0.003210935721365391,
"t_stat": -0.1358632911072551,
"p_two_sided": 0.8938635865211003,
"wilcoxon_p": 0.682424306487939,
"cohens_dz": -0.035079750921288615,
"reference_mean_cost": 0.13444323802629995,
"tost_p_equivalent": 0.0005738912160693893,
"seeds_a_worse": 8
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0.02586560501869948,
"sd_diff": 0.012860811901397174,
"ci95_low": 0.01874352477816795,
"ci95_high": 0.03298768525923101,
"t_stat": 7.789326074049729,
"p_two_sided": 0.0000018657782833092535,
"wilcoxon_p": 0.0006533107814151373,
"cohens_dz": 2.0111953441982533,
"reference_mean_cost": 0.1410544094583183,
"tost_p_equivalent": 0.999970847195676,
"seeds_a_worse": 15
}
},
"tstar": 0.8333333333333334
},
"10.0": {
"cost": {
"fixed_half": {
"mean": 0.29564483049825074,
"sd": 0.03447310784364567
},
"raw_tstar": {
"mean": 0.19548799613946197,
"sd": 0.013655444488437304
},
"isotonic_tstar": {
"mean": 0.18161418747737967,
"sd": 0.025594991271956233
},
"platt_tstar": {
"mean": 0.18081795150199065,
"sd": 0.01598149230641794
},
"empirical_raw": {
"mean": 0.18190372783206665,
"sd": 0.025279249561334947
},
"isotonic_empirical": {
"mean": 0.18207262637230065,
"sd": 0.026900813703594165
}
},
"regret": {
"fixed_half": {
"mean": 0.13791772228254315,
"sd": 0.028557882238902644
},
"raw_tstar": {
"mean": 0.037760887923754385,
"sd": 0.008123670531240051
},
"isotonic_tstar": {
"mean": 0.023887079261672096,
"sd": 0.013716380858236643
},
"platt_tstar": {
"mean": 0.023090843286283032,
"sd": 0.008379403129291767
},
"empirical_raw": {
"mean": 0.02417661961635903,
"sd": 0.013704916190348618
},
"isotonic_empirical": {
"mean": 0.024345518156593075,
"sd": 0.0156434272180191
}
},
"regret_ratio": {
"fixed_half": 0.8744072204375096,
"raw_tstar": 0.23940645556065476,
"isotonic_tstar": 0.15144561725562183,
"platt_tstar": 0.1463974300137678,
"empirical_raw": 0.15328132170720513,
"isotonic_empirical": 0.15435214930396207
},
"oracle_cost": {
"mean": 0.1577271082157076,
"sd": 0.01482708443509974
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0.00028954035468693463,
"sd_diff": 0.0027228726563596145,
"ci95_low": -0.0012183362624367197,
"ci95_high": 0.0017974169718105886,
"t_stat": 0.41183893383274817,
"p_two_sided": 0.6866935687822401,
"wilcoxon_p": 0.25575474881618,
"cohens_dz": 0.10633635547027012,
"reference_mean_cost": 0.18161418747737967,
"tost_p_equivalent": 2.7517727494105093e-9,
"seeds_a_worse": 6
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.013873808662082288,
"sd_diff": 0.014892860808163532,
"ci95_low": -0.022121200078738093,
"ci95_high": -0.005626417245426483,
"t_stat": -3.607972342510521,
"p_two_sided": 0.002852996259674617,
"wilcoxon_p": 0.00335693359375,
"cohens_dz": -0.9315744530746806,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.8477853614529237,
"seeds_a_worse": 2
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.013584268307395352,
"sd_diff": 0.014778203711317928,
"ci95_low": -0.021768164740208644,
"ci95_high": -0.005400371874582059,
"t_stat": -3.5600838879127537,
"p_two_sided": 0.0031374692601368534,
"wilcoxon_p": 0.006395825971507906,
"cohens_dz": -0.9192097072658298,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.832500809416237,
"seeds_a_worse": 3
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.014670044637471354,
"sd_diff": 0.0131542894154307,
"ci95_low": -0.02195464730813899,
"ci95_high": -0.007385441966803716,
"t_stat": -4.319263228494632,
"p_two_sided": 0.0007067223081222766,
"wilcoxon_p": 0.0019613960794227885,
"cohens_dz": -1.1152289701230529,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.9142693776845858,
"seeds_a_worse": 2
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.0007962359753890665,
"sd_diff": 0.01780553376189409,
"ci95_low": -0.010656611910436075,
"ci95_high": 0.009064139959657946,
"t_stat": -0.17319383476909755,
"p_two_sided": 0.8649781748436566,
"wilcoxon_p": 0.97796630859375,
"cohens_dz": -0.04471845584843426,
"reference_mean_cost": 0.18161418747737967,
"tost_p_equivalent": 0.04655567289820373,
"seeds_a_worse": 8
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0.10015683435878876,
"sd_diff": 0.031205585878875525,
"ci95_low": 0.0828757569053502,
"ci95_high": 0.11743791181222732,
"t_stat": 12.430651133617545,
"p_two_sided": 5.940832914551605e-9,
"wilcoxon_p": 0.0006533107814151373,
"cohens_dz": 3.20958032153434,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.9999999889424936,
"seeds_a_worse": 15
}
},
"tstar": 0.9090909090909092
},
"20.0": {
"cost": {
"fixed_half": {
"mean": 0.5530944625407166,
"sd": 0.07007003170724961
},
"raw_tstar": {
"mean": 0.25607431535770303,
"sd": 0.020751333952441314
},
"isotonic_tstar": {
"mean": 0.24403426227530467,
"sd": 0.034157256714959304
},
"platt_tstar": {
"mean": 0.23990831222101583,
"sd": 0.02830659812412079
},
"empirical_raw": {
"mean": 0.24386536373507064,
"sd": 0.034591872385139874
},
"isotonic_empirical": {
"mean": 0.2445409578960068,
"sd": 0.03484290517895198
}
},
"regret": {
"fixed_half": {
"mean": 0.3532151043551695,
"sd": 0.06959939693046106
},
"raw_tstar": {
"mean": 0.05619495717215588,
"sd": 0.014426172712646295
},
"isotonic_tstar": {
"mean": 0.04415490408975752,
"sd": 0.02318180192205417
},
"platt_tstar": {
"mean": 0.0400289540354687,
"sd": 0.01762575316641576
},
"empirical_raw": {
"mean": 0.04398600554952347,
"sd": 0.023467360360541104
},
"isotonic_empirical": {
"mean": 0.04466159971045965,
"sd": 0.023829352121357818
}
},
"regret_ratio": {
"fixed_half": 1.767141477547078,
"raw_tstar": 0.2811443746982134,
"isotonic_tstar": 0.22090777402221143,
"platt_tstar": 0.20026557218734903,
"empirical_raw": 0.2200627716079188,
"isotonic_empirical": 0.2234427812650893
},
"oracle_cost": {
"mean": 0.1998793581855471,
"sd": 0.02014208424557557
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": -0.00016889854023404704,
"sd_diff": 0.002872259846279311,
"ci95_low": -0.0017595030256814742,
"ci95_high": 0.0014217059452133803,
"t_stat": -0.22774444811202368,
"p_two_sided": 0.8231374073739885,
"wilcoxon_p": 0.7202301969573564,
"cohens_dz": -0.05880336364860445,
"reference_mean_cost": 0.24403426227530467,
"tost_p_equivalent": 8.966959222025819e-11,
"seeds_a_worse": 5
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.012040053082398364,
"sd_diff": 0.030387720684592175,
"ci95_low": -0.028868211887754437,
"ci95_high": 0.004788105722957714,
"t_stat": -1.5345318446087317,
"p_two_sided": 0.14718413142568446,
"wilcoxon_p": 0.207763671875,
"cohens_dz": -0.3962144185596376,
"reference_mean_cost": 0.25607431535770303,
"tost_p_equivalent": 0.4619218002022348,
"seeds_a_worse": 5
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.01220895162263241,
"sd_diff": 0.030226568693722426,
"ci95_low": -0.02894786743005839,
"ci95_high": 0.00452996418479357,
"t_stat": -1.5643544190620515,
"p_two_sided": 0.14005172503874813,
"wilcoxon_p": 0.15142822265625,
"cohens_dz": -0.40391457417288695,
"reference_mean_cost": 0.25607431535770303,
"tost_p_equivalent": 0.47016591217375553,
"seeds_a_worse": 5
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.016166003136687184,
"sd_diff": 0.01968928902692962,
"ci95_low": -0.027069567966336507,
"ci95_high": -0.005262438307037862,
"t_stat": -3.1799350823430896,
"p_two_sided": 0.006681906451902317,
"wilcoxon_p": 0.013150364234284666,
"cohens_dz": -0.821055707729033,
"reference_mean_cost": 0.25607431535770303,
"tost_p_equivalent": 0.7404467131698949,
"seeds_a_worse": 2
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.004125950054288821,
"sd_diff": 0.026460473582316464,
"ci95_low": -0.018779271905240497,
"ci95_high": 0.010527371796662854,
"t_stat": -0.6039096691838293,
"p_two_sided": 0.5555637030887467,
"wilcoxon_p": 0.5614013671875,
"cohens_dz": -0.1559288060908401,
"reference_mean_cost": 0.24403426227530467,
"tost_p_equivalent": 0.12843947469460673,
"seeds_a_worse": 6
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0.29702014718301367,
"sd_diff": 0.0721867489324795,
"ci95_low": 0.2570444580786291,
"ci95_high": 0.33699583628739826,
"t_stat": 15.935806786421164,
"p_two_sided": 2.2793728788911872e-10,
"wilcoxon_p": 0.00006103515625,
"cohens_dz": 4.11460761947922,
"reference_mean_cost": 0.25607431535770303,
"tost_p_equivalent": 0.9999999997953516,
"seeds_a_worse": 15
}
},
"tstar": 0.9523809523809524
}
},
"by_decision_size_at_R10": {
"100": {
"cost": {
"fixed_half": {
"mean": 0.29564483049825074,
"sd": 0.03447310784364567
},
"raw_tstar": {
"mean": 0.19548799613946197,
"sd": 0.013655444488437304
},
"isotonic_tstar": {
"mean": 0.22726505006635303,
"sd": 0.04882995031107432
},
"platt_tstar": {
"mean": 0.20354686934491495,
"sd": 0.02273464733253259
},
"empirical_raw": {
"mean": 0.2290505489202558,
"sd": 0.0510250455430174
},
"isotonic_empirical": {
"mean": 0.23064302087103392,
"sd": 0.05031739365683369
}
},
"regret": {
"fixed_half": {
"mean": 0.13791772228254315,
"sd": 0.028557882238902644
},
"raw_tstar": {
"mean": 0.037760887923754385,
"sd": 0.008123670531240051
},
"isotonic_tstar": {
"mean": 0.06953794185064545,
"sd": 0.04899873749263044
},
"platt_tstar": {
"mean": 0.04581976112920739,
"sd": 0.021295596779759993
},
"empirical_raw": {
"mean": 0.07132344070454821,
"sd": 0.05061514555031535
},
"isotonic_empirical": {
"mean": 0.07291591265532635,
"sd": 0.05078242113400844
}
},
"regret_ratio": {
"fixed_half": 0.8744072204375096,
"raw_tstar": 0.23940645556065476,
"isotonic_tstar": 0.44087501912192145,
"platt_tstar": 0.2905002294630564,
"empirical_raw": 0.4521951965733517,
"isotonic_empirical": 0.4622915710570598
},
"oracle_cost": {
"mean": 0.1577271082157076,
"sd": 0.01482708443509974
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0.0017854988539027672,
"sd_diff": 0.008106145040271259,
"ci95_low": -0.0027035346426470593,
"ci95_high": 0.006274532350452593,
"t_stat": 0.8530821113467816,
"p_two_sided": 0.4079724429773925,
"wilcoxon_p": 0.19591441987172553,
"cohens_dz": 0.22026485401290308,
"reference_mean_cost": 0.22726505006635303,
"tost_p_equivalent": 0.00021583685552291936,
"seeds_a_worse": 8
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": 0.031777053926891057,
"sd_diff": 0.0499371306403798,
"ci95_low": 0.004122792739535332,
"ci95_high": 0.05943131511424678,
"t_stat": 2.4645388926464764,
"p_two_sided": 0.02726979665621086,
"wilcoxon_p": 0.012451171875,
"cohens_dz": 0.6363412058200181,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.9450022069145068,
"seeds_a_worse": 11
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": 0.03356255278079383,
"sd_diff": 0.05175343412882501,
"ci95_low": 0.004902456247657269,
"ci95_high": 0.06222264931393039,
"t_stat": 2.511663431892402,
"p_two_sided": 0.02489898173151024,
"wilcoxon_p": 0.01507568359375,
"cohens_dz": 0.6485087095331623,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.9516254451818398,
"seeds_a_worse": 11
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": 0.008058873205453005,
"sd_diff": 0.02312258381894709,
"ci95_low": -0.004745986906766387,
"ci95_high": 0.0208637333176724,
"t_stat": 1.3498440294696243,
"p_two_sided": 0.1984882252176535,
"wilcoxon_p": 0.306526583291276,
"cohens_dz": 0.3485282297408912,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.3890279862105513,
"seeds_a_worse": 9
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.023718180721438057,
"sd_diff": 0.05417060819848865,
"ci95_low": -0.05371686363710357,
"ci95_high": 0.006280502194227461,
"t_stat": -1.69575572420895,
"p_two_sided": 0.11204638119669616,
"wilcoxon_p": 0.13967601763991985,
"cohens_dz": -0.437842245273144,
"reference_mean_cost": 0.22726505006635303,
"tost_p_equivalent": 0.80401240160993,
"seeds_a_worse": 5
}
}
},
"250": {
"cost": {
"fixed_half": {
"mean": 0.29564483049825074,
"sd": 0.03447310784364567
},
"raw_tstar": {
"mean": 0.19548799613946197,
"sd": 0.013655444488437304
},
"isotonic_tstar": {
"mean": 0.19855229822656537,
"sd": 0.0311053810796724
},
"platt_tstar": {
"mean": 0.187598021474243,
"sd": 0.01739484761965564
},
"empirical_raw": {
"mean": 0.1963324888406322,
"sd": 0.03014548341917118
},
"isotonic_empirical": {
"mean": 0.1982627578718784,
"sd": 0.03212969699444278
}
},
"regret": {
"fixed_half": {
"mean": 0.13791772228254315,
"sd": 0.028557882238902644
},
"raw_tstar": {
"mean": 0.037760887923754385,
"sd": 0.008123670531240051
},
"isotonic_tstar": {
"mean": 0.04082519001085776,
"sd": 0.02439909808828474
},
"platt_tstar": {
"mean": 0.029870913258535418,
"sd": 0.015532624407894444
},
"empirical_raw": {
"mean": 0.0386053806249246,
"sd": 0.02430706962111215
},
"isotonic_empirical": {
"mean": 0.040535649656170826,
"sd": 0.025105508997797708
}
},
"regret_ratio": {
"fixed_half": 0.8744072204375096,
"raw_tstar": 0.23940645556065476,
"isotonic_tstar": 0.2588343276732446,
"platt_tstar": 0.18938350925500996,
"empirical_raw": 0.24476059354443935,
"isotonic_empirical": 0.2569986232216613
},
"oracle_cost": {
"mean": 0.1577271082157076,
"sd": 0.01482708443509974
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": -0.002219809385933162,
"sd_diff": 0.0039168930186059565,
"ci95_low": -0.004388912439921438,
"ci95_high": -0.00005070633194488626,
"t_stat": -2.1949245850308845,
"p_two_sided": 0.045531266374096685,
"wilcoxon_p": 0.025514556367511352,
"cohens_dz": -0.5667270909337228,
"reference_mean_cost": 0.19855229822656537,
"tost_p_equivalent": 0.000001198392038001832,
"seeds_a_worse": 3
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": 0.003064302087103384,
"sd_diff": 0.024845846219476813,
"ci95_low": -0.010694868933802133,
"ci95_high": 0.0168234731080089,
"t_stat": 0.4776649926214485,
"p_two_sided": 0.640260502059444,
"wilcoxon_p": 0.8202444727582786,
"cohens_dz": 0.12333257076594392,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.15664869914008858,
"seeds_a_worse": 6
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": 0.0008444927011702223,
"sd_diff": 0.02462410597933139,
"ci95_low": -0.012791882667720597,
"ci95_high": 0.01448086807006104,
"t_stat": 0.1328253773099947,
"p_two_sided": 0.8962217300407884,
"wilcoxon_p": 0.909488229399474,
"cohens_dz": 0.03429536495168839,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.09097829333246175,
"seeds_a_worse": 6
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.007889974665218968,
"sd_diff": 0.017711751559328744,
"ci95_low": -0.017698415747557993,
"ci95_high": 0.001918466417120058,
"t_stat": -1.7252805504883455,
"p_two_sided": 0.10647153195965434,
"wilcoxon_p": 0.072998046875,
"cohens_dz": -0.4454655226384617,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.34326677065923894,
"seeds_a_worse": 3
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.010954276752322353,
"sd_diff": 0.032262581987475836,
"ci95_low": -0.02882069914020966,
"ci95_high": 0.006912145635564956,
"t_stat": -1.315013517763736,
"p_two_sided": 0.20963847560477672,
"wilcoxon_p": 0.2804339073312231,
"cohens_dz": -0.33953503028910537,
"reference_mean_cost": 0.19855229822656537,
"tost_p_equivalent": 0.5481680376236138,
"seeds_a_worse": 6
}
}
},
"500": {
"cost": {
"fixed_half": {
"mean": 0.29564483049825074,
"sd": 0.03447310784364567
},
"raw_tstar": {
"mean": 0.19548799613946197,
"sd": 0.013655444488437304
},
"isotonic_tstar": {
"mean": 0.18972131740861384,
"sd": 0.028860235321687407
},
"platt_tstar": {
"mean": 0.1810351067680058,
"sd": 0.016107105468881118
},
"empirical_raw": {
"mean": 0.18800820364338283,
"sd": 0.02735552620172375
},
"isotonic_empirical": {
"mean": 0.18981783086017617,
"sd": 0.029480461635677187
}
},
"regret": {
"fixed_half": {
"mean": 0.13791772228254315,
"sd": 0.028557882238902644
},
"raw_tstar": {
"mean": 0.037760887923754385,
"sd": 0.008123670531240051
},
"isotonic_tstar": {
"mean": 0.031994209192906256,
"sd": 0.022988391921028466
},
"platt_tstar": {
"mean": 0.023307998552298228,
"sd": 0.007311559417887468
},
"empirical_raw": {
"mean": 0.03028109542767523,
"sd": 0.019843600858993095
},
"isotonic_empirical": {
"mean": 0.03209072264446857,
"sd": 0.02304443626726021
}
},
"regret_ratio": {
"fixed_half": 0.8744072204375096,
"raw_tstar": 0.23940645556065476,
"isotonic_tstar": 0.20284534189995407,
"platt_tstar": 0.14777420835245525,
"empirical_raw": 0.1919840905614196,
"isotonic_empirical": 0.20345724338381516
},
"oracle_cost": {
"mean": 0.1577271082157076,
"sd": 0.01482708443509974
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": -0.0017131137652310262,
"sd_diff": 0.005146594106061489,
"ci95_low": -0.0045632025830944205,
"ci95_high": 0.0011369750526323689,
"t_stat": -1.2891751216759306,
"p_two_sided": 0.2182310563886176,
"wilcoxon_p": 0.1660308270627835,
"cohens_dz": -0.33286358510638664,
"reference_mean_cost": 0.18972131740861384,
"tost_p_equivalent": 0.000021116480877045593,
"seeds_a_worse": 2
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.00576667873084812,
"sd_diff": 0.02462022997658225,
"ci95_low": -0.019400907640961407,
"ci95_high": 0.007867550179265166,
"t_stat": -0.9071503681625522,
"p_two_sided": 0.3796768273147673,
"wilcoxon_p": 0.39415050825096176,
"cohens_dz": -0.23422521789329948,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.2692776951448566,
"seeds_a_worse": 5
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.007479792496079146,
"sd_diff": 0.022161487100369116,
"ci95_low": -0.01975241498589149,
"ci95_high": 0.0047928299937331975,
"t_stat": -1.3071826650982838,
"p_two_sided": 0.21221348755863745,
"wilcoxon_p": 0.2326020367394871,
"cohens_dz": -0.33751311282511204,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.3472331468609276,
"seeds_a_worse": 5
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.014452889371456152,
"sd_diff": 0.01244418164894374,
"ci95_low": -0.021344247468521454,
"ci95_high": -0.007561531274390848,
"t_stat": -4.498150333973876,
"p_two_sided": 0.000500962501999923,
"wilcoxon_p": 0.0014632771410400872,
"cohens_dz": -1.1614174221478768,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.9162854400288828,
"seeds_a_worse": 1
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.008686210640608033,
"sd_diff": 0.025223744206918912,
"ci95_low": -0.022654654591553497,
"ci95_high": 0.005282233310337428,
"t_stat": -1.3337254325429069,
"p_two_sided": 0.20358711074849123,
"wilcoxon_p": 0.410103931438363,
"cohens_dz": -0.34436642591013084,
"reference_mean_cost": 0.18972131740861384,
"tost_p_equivalent": 0.45200005739914145,
"seeds_a_worse": 6
}
}
},
"920": {
"cost": {
"fixed_half": {
"mean": 0.29564483049825074,
"sd": 0.03447310784364567
},
"raw_tstar": {
"mean": 0.19548799613946197,
"sd": 0.013655444488437304
},
"isotonic_tstar": {
"mean": 0.18161418747737967,
"sd": 0.025594991271956233
},
"platt_tstar": {
"mean": 0.18081795150199065,
"sd": 0.01598149230641794
},
"empirical_raw": {
"mean": 0.18190372783206665,
"sd": 0.025279249561334947
},
"isotonic_empirical": {
"mean": 0.18207262637230065,
"sd": 0.026900813703594165
}
},
"regret": {
"fixed_half": {
"mean": 0.13791772228254315,
"sd": 0.028557882238902644
},
"raw_tstar": {
"mean": 0.037760887923754385,
"sd": 0.008123670531240051
},
"isotonic_tstar": {
"mean": 0.023887079261672096,
"sd": 0.013716380858236643
},
"platt_tstar": {
"mean": 0.023090843286283032,
"sd": 0.008379403129291767
},
"empirical_raw": {
"mean": 0.02417661961635903,
"sd": 0.013704916190348618
},
"isotonic_empirical": {
"mean": 0.024345518156593075,
"sd": 0.0156434272180191
}
},
"regret_ratio": {
"fixed_half": 0.8744072204375096,
"raw_tstar": 0.23940645556065476,
"isotonic_tstar": 0.15144561725562183,
"platt_tstar": 0.1463974300137678,
"empirical_raw": 0.15328132170720513,
"isotonic_empirical": 0.15435214930396207
},
"oracle_cost": {
"mean": 0.1577271082157076,
"sd": 0.01482708443509974
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0.00028954035468693463,
"sd_diff": 0.0027228726563596145,
"ci95_low": -0.0012183362624367197,
"ci95_high": 0.0017974169718105886,
"t_stat": 0.41183893383274817,
"p_two_sided": 0.6866935687822401,
"wilcoxon_p": 0.25575474881618,
"cohens_dz": 0.10633635547027012,
"reference_mean_cost": 0.18161418747737967,
"tost_p_equivalent": 2.7517727494105093e-9,
"seeds_a_worse": 6
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.013873808662082288,
"sd_diff": 0.014892860808163532,
"ci95_low": -0.022121200078738093,
"ci95_high": -0.005626417245426483,
"t_stat": -3.607972342510521,
"p_two_sided": 0.002852996259674617,
"wilcoxon_p": 0.00335693359375,
"cohens_dz": -0.9315744530746806,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.8477853614529237,
"seeds_a_worse": 2
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.013584268307395352,
"sd_diff": 0.014778203711317928,
"ci95_low": -0.021768164740208644,
"ci95_high": -0.005400371874582059,
"t_stat": -3.5600838879127537,
"p_two_sided": 0.0031374692601368534,
"wilcoxon_p": 0.006395825971507906,
"cohens_dz": -0.9192097072658298,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.832500809416237,
"seeds_a_worse": 3
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.014670044637471354,
"sd_diff": 0.0131542894154307,
"ci95_low": -0.02195464730813899,
"ci95_high": -0.007385441966803716,
"t_stat": -4.319263228494632,
"p_two_sided": 0.0007067223081222766,
"wilcoxon_p": 0.0019613960794227885,
"cohens_dz": -1.1152289701230529,
"reference_mean_cost": 0.19548799613946197,
"tost_p_equivalent": 0.9142693776845858,
"seeds_a_worse": 2
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.0007962359753890665,
"sd_diff": 0.01780553376189409,
"ci95_low": -0.010656611910436075,
"ci95_high": 0.009064139959657946,
"t_stat": -0.17319383476909755,
"p_two_sided": 0.8649781748436566,
"wilcoxon_p": 0.97796630859375,
"cohens_dz": -0.04471845584843426,
"reference_mean_cost": 0.18161418747737967,
"tost_p_equivalent": 0.04655567289820373,
"seeds_a_worse": 8
}
}
}
},
"by_family_at_R10_full": {
"logreg": {
"cost": {
"fixed_half": {
"mean": 0.36771625045240686,
"sd": 0.039263489687893095
},
"raw_tstar": {
"mean": 0.2604415490408976,
"sd": 0.02517391518771953
},
"isotonic_tstar": {
"mean": 0.25812522620340217,
"sd": 0.03734607618877389
},
"platt_tstar": {
"mean": 0.2624683315237062,
"sd": 0.030063225044027767
},
"empirical_raw": {
"mean": 0.258704306912776,
"sd": 0.03756191313746154
},
"isotonic_empirical": {
"mean": 0.25740137531668483,
"sd": 0.03737763003690322
}
},
"regret": {
"fixed_half": {
"mean": 0.13108939558450963,
"sd": 0.03767830858019167
},
"raw_tstar": {
"mean": 0.023814694173000365,
"sd": 0.011106706166130562
},
"isotonic_tstar": {
"mean": 0.02149837133550488,
"sd": 0.015257268676560472
},
"platt_tstar": {
"mean": 0.0258414766558089,
"sd": 0.01661019804411616
},
"empirical_raw": {
"mean": 0.02207745204487875,
"sd": 0.015193480592136243
},
"isotonic_empirical": {
"mean": 0.020774520448787543,
"sd": 0.016072563661285504
}
},
"regret_ratio": {
"fixed_half": 0.5539920464973999,
"raw_tstar": 0.1006423982869379,
"isotonic_tstar": 0.09085347200978888,
"platt_tstar": 0.10920770877944322,
"empirical_raw": 0.09330070357907612,
"isotonic_empirical": 0.08779443254817983
},
"oracle_cost": {
"mean": 0.23662685486789725,
"sd": 0.030979211231468275
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0.0005790807093738693,
"sd_diff": 0.0009940015379306486,
"ci95_low": 0.00002862100538078517,
"ci95_high": 0.0011295404133669534,
"t_stat": 2.256304299271065,
"p_two_sided": 0.040568593140109646,
"wilcoxon_p": 0.046113498652135015,
"cohens_dz": 0.5825752650035353,
"reference_mean_cost": 0.25812522620340217,
"tost_p_equivalent": 3.050964810780743e-17,
"seeds_a_worse": 6
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.0023163228374954827,
"sd_diff": 0.020424087578345552,
"ci95_low": -0.013626805541687551,
"ci95_high": 0.008994159866696588,
"t_stat": -0.4392401736257446,
"p_two_sided": 0.6671910867706733,
"wilcoxon_p": 0.6373518882339371,
"cohens_dz": -0.11341132516251752,
"reference_mean_cost": 0.2604415490408976,
"tost_p_equivalent": 0.030896336884812903,
"seeds_a_worse": 5
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.001737242128121613,
"sd_diff": 0.020722607831210976,
"ci95_low": -0.013213039838133833,
"ci95_high": 0.009738555581890605,
"t_stat": -0.3246845129410411,
"p_two_sided": 0.750220560715307,
"wilcoxon_p": 0.6831652607455685,
"cohens_dz": -0.08383318075947459,
"reference_mean_cost": 0.2604415490408976,
"tost_p_equivalent": 0.026712930434103586,
"seeds_a_worse": 5
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": 0.0020267824828085366,
"sd_diff": 0.012262659087964391,
"ci95_low": -0.004764051770608822,
"ci95_high": 0.008817616736225895,
"t_stat": 0.6401299054302778,
"p_two_sided": 0.532425302758895,
"wilcoxon_p": 0.5504522071367985,
"cohens_dz": 0.16528083087605294,
"reference_mean_cost": 0.2604415490408976,
"tost_p_equivalent": 0.0018661514292050009,
"seeds_a_worse": 10
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": 0.004343105320304019,
"sd_diff": 0.02532443819693942,
"ci95_low": -0.009681101103661317,
"ci95_high": 0.018367311744269355,
"t_stat": 0.6642111641550716,
"p_two_sided": 0.5173459396126855,
"wilcoxon_p": 0.3485908811649828,
"cohens_dz": 0.17149858514250887,
"reference_mean_cost": 0.25812522620340217,
"tost_p_equivalent": 0.1057074562655543,
"seeds_a_worse": 8
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0.10727470141150924,
"sd_diff": 0.04117001062991981,
"ci95_low": 0.08447550945863025,
"ci95_high": 0.13007389336438824,
"t_stat": 10.091645002739163,
"p_two_sided": 8.342290253710558e-8,
"wilcoxon_p": 0.00006103515625,
"cohens_dz": 2.605651535429739,
"reference_mean_cost": 0.2604415490408976,
"tost_p_equivalent": 0.999999797219806,
"seeds_a_worse": 15
}
},
"mean_empirical_threshold": 0.8485431093712957,
"sd_empirical_threshold": 0.07186651931852739,
"diagnostics": {
"auroc": {
"mean": 0.9678431430630848,
"sd": 0.004901772846696877
},
"logloss_raw": {
"mean": 0.2494756613257333,
"sd": 0.025920886548439385
},
"logloss_iso": {
"mean": 0.26191962204273495,
"sd": 0.05792107337419095
},
"logloss_platt": {
"mean": 0.2446106701547049,
"sd": 0.02308570210430197
},
"brier_raw": {
"mean": 0.06529402847447602,
"sd": 0.0053636935059265456
},
"ece_raw": {
"mean": 0.038580625474565027,
"sd": 0.008868646334929868
},
"ece_iso": {
"mean": 0.0254876142057846,
"sd": 0.006515783947293326
},
"ece_platt": {
"mean": 0.0405625261512769,
"sd": 0.006316486468505416
}
},
"mean_hpo_seconds_per_seed": 0.038721132278442386
},
"random_forest": {
"cost": {
"fixed_half": {
"mean": 0.2515381831342744,
"sd": 0.045857947097111613
},
"raw_tstar": {
"mean": 0.18602967788635544,
"sd": 0.017994647694036997
},
"isotonic_tstar": {
"mean": 0.13637350705754617,
"sd": 0.026819209130029553
},
"platt_tstar": {
"mean": 0.14325009048136086,
"sd": 0.02153823495948192
},
"empirical_raw": {
"mean": 0.13666304741223312,
"sd": 0.02370968816326328
},
"isotonic_empirical": {
"mean": 0.13847267462902646,
"sd": 0.028544227707752768
}
},
"regret": {
"fixed_half": {
"mean": 0.13174086138255522,
"sd": 0.04425819055623873
},
"raw_tstar": {
"mean": 0.06623235613463628,
"sd": 0.025211129180269964
},
"isotonic_tstar": {
"mean": 0.016576185305827,
"sd": 0.01328193820997756
},
"platt_tstar": {
"mean": 0.023452768729641697,
"sd": 0.01367142266183835
},
"empirical_raw": {
"mean": 0.016865725660513933,
"sd": 0.009640696950727534
},
"isotonic_empirical": {
"mean": 0.018675352877307275,
"sd": 0.015990271706701302
}
},
"regret_ratio": {
"fixed_half": 1.0996978851963746,
"raw_tstar": 0.5528700906344411,
"isotonic_tstar": 0.13836858006042294,
"platt_tstar": 0.19577039274924468,
"empirical_raw": 0.14078549848942595,
"isotonic_empirical": 0.15589123867069482
},
"oracle_cost": {
"mean": 0.11979732175171916,
"sd": 0.018277874774881436
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0.00028954035468693463,
"sd_diff": 0.007877537015793093,
"ci95_low": -0.0040728942376532734,
"ci95_high": 0.004651974947027143,
"t_stat": 0.1423522313521732,
"p_two_sided": 0.8888301419327552,
"wilcoxon_p": 0.7854947471183542,
"cohens_dz": 0.03675518808816214,
"reference_mean_cost": 0.13637350705754617,
"tost_p_equivalent": 0.0031467856492402883,
"seeds_a_worse": 2
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": -0.04965617082880927,
"sd_diff": 0.03181599035126148,
"ci95_low": -0.06727527901194381,
"ci95_high": -0.03203706264567473,
"t_stat": -6.044681323232285,
"p_two_sided": 0.000030145200115166503,
"wilcoxon_p": 0.00018310546875,
"cohens_dz": -1.5607300065339766,
"reference_mean_cost": 0.18602967788635544,
"tost_p_equivalent": 0.9998855327457123,
"seeds_a_worse": 1
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": -0.04936663047412234,
"sd_diff": 0.027727160621182648,
"ci95_low": -0.06472142022613757,
"ci95_high": -0.03401184072210711,
"t_stat": -6.895626288491428,
"p_two_sided": 0.000007375461372759578,
"wilcoxon_p": 0.0006533107814151373,
"cohens_dz": -1.7804430517998242,
"reference_mean_cost": 0.18602967788635544,
"tost_p_equivalent": 0.9999670565348524,
"seeds_a_worse": 0
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.04277958740499457,
"sd_diff": 0.02835756540263814,
"ci95_low": -0.05848348368869409,
"ci95_high": -0.02707569112129506,
"t_stat": -5.842695845876625,
"p_two_sided": 0.00004272907459920141,
"wilcoxon_p": 0.0006533107814151373,
"cohens_dz": -1.508577580535695,
"reference_mean_cost": 0.18602967788635544,
"tost_p_equivalent": 0.999782609263018,
"seeds_a_worse": 0
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": 0.006876583423814696,
"sd_diff": 0.020863810080802844,
"ci95_low": -0.004677409485648934,
"ci95_high": 0.01843057633327833,
"t_stat": 1.276511479739057,
"p_two_sided": 0.22254379037518413,
"wilcoxon_p": 0.016973975554361757,
"cohens_dz": 0.32959384681813036,
"reference_mean_cost": 0.13637350705754617,
"tost_p_equivalent": 0.5042125348539948,
"seeds_a_worse": 12
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0.06550850524791893,
"sd_diff": 0.05518519584209568,
"ci95_low": 0.03494796242292404,
"ci95_high": 0.09606904807291382,
"t_stat": 4.597489344535346,
"p_two_sided": 0.00041443912218412513,
"wilcoxon_p": 0.00030517578125,
"cohens_dz": 1.18706664438343,
"reference_mean_cost": 0.18602967788635544,
"tost_p_equivalent": 0.9992665165676242,
"seeds_a_worse": 14
}
},
"mean_empirical_threshold": 0.7437382819761851,
"sd_empirical_threshold": 0.0658125027087912,
"diagnostics": {
"auroc": {
"mean": 0.9808655140522198,
"sd": 0.00449329026383785
},
"logloss_raw": {
"mean": 0.18916904949410637,
"sd": 0.023745349160965955
},
"logloss_iso": {
"mean": 0.18593718186603084,
"sd": 0.036611292300254415
},
"logloss_platt": {
"mean": 0.1683512676395223,
"sd": 0.019320410276158183
},
"brier_raw": {
"mean": 0.047835205473738554,
"sd": 0.003340732925698663
},
"ece_raw": {
"mean": 0.04827113163892431,
"sd": 0.01125839525455648
},
"ece_iso": {
"mean": 0.017375170374021347,
"sd": 0.004823439106033693
},
"ece_platt": {
"mean": 0.024501620349534313,
"sd": 0.00526400577333199
}
},
"mean_hpo_seconds_per_seed": 2.5691203753153484
},
"hist_gbdt": {
"cost": {
"fixed_half": {
"mean": 0.267680057908071,
"sd": 0.04916723703254906
},
"raw_tstar": {
"mean": 0.13999276149113285,
"sd": 0.02496768469292639
},
"isotonic_tstar": {
"mean": 0.15034382917119077,
"sd": 0.030999319429017336
},
"platt_tstar": {
"mean": 0.13673543250090484,
"sd": 0.01883065969221564
},
"empirical_raw": {
"mean": 0.15034382917119077,
"sd": 0.0309966028673116
},
"isotonic_empirical": {
"mean": 0.15034382917119077,
"sd": 0.0309966028673116
}
},
"regret": {
"fixed_half": {
"mean": 0.15092290988056464,
"sd": 0.03961649100493011
},
"raw_tstar": {
"mean": 0.0232356134636265,
"sd": 0.014391583908981513
},
"isotonic_tstar": {
"mean": 0.0335866811436844,
"sd": 0.026147239182377015
},
"platt_tstar": {
"mean": 0.01997828447339849,
"sd": 0.010840687674165484
},
"empirical_raw": {
"mean": 0.0335866811436844,
"sd": 0.026111789360895796
},
"isotonic_empirical": {
"mean": 0.0335866811436844,
"sd": 0.026111789360895796
}
},
"regret_ratio": {
"fixed_half": 1.292622442653441,
"raw_tstar": 0.19900805951642905,
"isotonic_tstar": 0.2876627402355858,
"platt_tstar": 0.17110973341599506,
"empirical_raw": 0.2876627402355858,
"isotonic_empirical": 0.2876627402355858
},
"oracle_cost": {
"mean": 0.11675714802750636,
"sd": 0.01664396138907161
},
"contrasts": {
"empirical_raw_vs_isotonic_tstar": {
"mean_diff": 0,
"sd_diff": 0.0004103848784030699,
"ci95_low": -0.00022726357059696292,
"ci95_high": 0.00022726357059696292,
"t_stat": 0,
"p_two_sided": 1,
"wilcoxon_p": 1,
"cohens_dz": 0,
"reference_mean_cost": 0.15034382917119077,
"tost_p_equivalent": 1.3253728715168488e-19,
"seeds_a_worse": 1
},
"isotonic_tstar_vs_raw_tstar": {
"mean_diff": 0.010351067680057906,
"sd_diff": 0.02092909234086001,
"ci95_low": -0.0012390773400170196,
"ci95_high": 0.021941212700132832,
"t_stat": 1.9154921812860024,
"p_two_sided": 0.07608228680916392,
"wilcoxon_p": 0.06847552025215699,
"cohens_dz": 0.49457795452741377,
"reference_mean_cost": 0.13999276149113285,
"tost_p_equivalent": 0.7274521586824325,
"seeds_a_worse": 10
},
"empirical_raw_vs_raw_tstar": {
"mean_diff": 0.010351067680057906,
"sd_diff": 0.020868652807163105,
"ci95_low": -0.0012056070418748864,
"ci95_high": 0.0219077424019907,
"t_stat": 1.9210398060084408,
"p_two_sided": 0.07532686411057497,
"wilcoxon_p": 0.06861194191702381,
"cohens_dz": 0.4960103450714812,
"reference_mean_cost": 0.13999276149113285,
"tost_p_equivalent": 0.7280262480215751,
"seeds_a_worse": 10
},
"platt_tstar_vs_raw_tstar": {
"mean_diff": -0.0032573289902280145,
"sd_diff": 0.01815924104719371,
"ci95_low": -0.013313581490990977,
"ci95_high": 0.006798923510534948,
"t_stat": -0.6947196140788772,
"p_two_sided": 0.4985992451590715,
"wilcoxon_p": 0.67877197265625,
"cohens_dz": -0.17937583304074234,
"reference_mean_cost": 0.13999276149113285,
"tost_p_equivalent": 0.21905509539263676,
"seeds_a_worse": 8
},
"platt_tstar_vs_isotonic_tstar": {
"mean_diff": -0.01360839667028592,
"sd_diff": 0.027550450499118716,
"ci95_low": -0.028865327618494467,
"ci95_high": 0.001648534277922626,
"t_stat": -1.9130392686060709,
"p_two_sided": 0.07641848070376883,
"wilcoxon_p": 0.13222065246162956,
"cohens_dz": -0.4939446151968087,
"reference_mean_cost": 0.15034382917119077,
"tost_p_equivalent": 0.7968718457438437,
"seeds_a_worse": 5
},
"fixed_half_vs_raw_tstar": {
"mean_diff": 0.12768729641693813,
"sd_diff": 0.03273009154527431,
"ci95_low": 0.10956197586543688,
"ci95_high": 0.1458126169684394,
"t_stat": 15.109361116848252,
"p_two_sided": 4.622887442620787e-10,
"wilcoxon_p": 0.00006103515625,
"cohens_dz": 3.901220265159145,
"reference_mean_cost": 0.13999276149113285,
"tost_p_equivalent": 0.9999999995135104,
"seeds_a_worse": 15
}
},
"mean_empirical_threshold": 0.903288249583066,
"sd_empirical_threshold": 0.07223427334387789,
"diagnostics": {
"auroc": {
"mean": 0.9854540846720712,
"sd": 0.0031153960436600065
},
"logloss_raw": {
"mean": 0.14792045833129944,
"sd": 0.021234339678055856
},
"logloss_iso": {
"mean": 0.18692200153223837,
"sd": 0.05667800967007169
},
"logloss_platt": {
"mean": 0.14389687087512332,
"sd": 0.01513405103327428
},
"brier_raw": {
"mean": 0.03951648772361925,
"sd": 0.004517661162504012
},
"ece_raw": {
"mean": 0.0195317355922214,
"sd": 0.006224260524043429
},
"ece_iso": {
"mean": 0.01767469389363007,
"sd": 0.006345658906612354
},
"ece_platt": {
"mean": 0.018147414227556433,
"sd": 0.004106117349413187
}
},
"mean_hpo_seconds_per_seed": 1.413260555267334
}
},
"cheapest_rule_by_cost_ratio": {
"1.0": "empirical_raw",
"2.0": "isotonic_tstar",
"5.0": "platt_tstar",
"10.0": "platt_tstar",
"20.0": "platt_tstar"
},
"trivial_baseline_cost": {
"1.0": {
"classify_all_as_ham": 0.39404477287546186,
"classify_all_as_spam": 0.6059552271245381
},
"2.0": {
"classify_all_as_ham": 0.39404477287546186,
"classify_all_as_spam": 1.2119104542490762
},
"5.0": {
"classify_all_as_ham": 0.39404477287546186,
"classify_all_as_spam": 3.0297761356226904
},
"10.0": {
"classify_all_as_ham": 0.39404477287546186,
"classify_all_as_spam": 6.059552271245381
},
"20.0": {
"classify_all_as_ham": 0.39404477287546186,
"classify_all_as_spam": 12.119104542490762
}
},
"runtime_seconds": 61.127012729644775,
"hypothesis_supported": false
}Experiment script · experiment_v1.py
"""Calibrate, or tune the cut-off? A matched-budget contest between the two standard
recipes for cost-sensitive thresholding, on the real UCI Spambase benchmark.
Every number comes from the real e-mails loaded through `dataset.load()`; nothing is
simulated. Randomness enters only via stratified splits, the random hyperparameter
search and the sub-sampling of the decision split. All are seeded.
"""
import os
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "NUMEXPR_NUM_THREADS"):
os.environ.setdefault(_v, "1") # single-threaded: faster here, and reproducible
import json, time, warnings # noqa: E401
import numpy as np
import pandas as pd
from scipy import stats
from scipy.optimize import minimize
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from dataset import load
warnings.filterwarnings("ignore")
SEEDS = list(range(15))
COST_RATIOS = [1.0, 2.0, 5.0, 10.0, 20.0]
DEC_SIZES = [100, 250, 500, None] # None = the whole decision split
HPO_BUDGET = 10 # identical random-search budget per family
FAMILIES = ["logreg", "random_forest", "hist_gbdt"]
RULES = ["fixed_half", "raw_tstar", "isotonic_tstar", "platt_tstar",
"empirical_raw", "isotonic_empirical"]
PRIMARY_R, EQUIV_FRAC = 10.0, 0.05 # equivalence margin = 5% of the reference cost
t_start = time.time()
df = load()
n_loaded = int(df.shape[0])
num = df[[c for c in df.columns if c != "is_spam"]].astype(float)
keep = np.isfinite(num.to_numpy()).all(axis=1) & df["is_spam"].isin([0, 1]).to_numpy()
X, y = num.to_numpy(float)[keep], df["is_spam"].to_numpy(int)[keep]
n, d = X.shape
print(f"loaded {n_loaded} rows; analysed {n} x {d}; positives={int(y.sum())}; dropped={n_loaded-n}")
def evaluator(p, yy):
"""Exact, tie-safe cost of the rule 'predict positive iff score >= t'."""
order = np.argsort(p, kind="mergesort")
ps, ys = p[order], yy[order].astype(np.int64)
cum_pos = np.concatenate(([0], np.cumsum(ys)))
m, tot_pos = len(ps), int(ys.sum())
def cost(t, R):
j = np.searchsorted(ps, np.atleast_1d(np.asarray(t, float)), side="left")
fn = cum_pos[j]
return (R * ((m - j) - (tot_pos - fn)) + fn) / m
return cost, ps
def candidates(ps):
u = np.unique(ps)
mids = (u[:-1] + u[1:]) / 2.0 if len(u) > 1 else np.array([], float)
return np.concatenate(([u[0] - 1e-9], mids, [u[-1] + 1e-9]))
def empirical_threshold(p, yy, R):
cost, ps = evaluator(p, yy)
cands = candidates(ps)
c = cost(cands, R)
return float(np.median(cands[c == c.min()])) # midpoint of the flat minimum
def fit_isotonic(p, yy):
ir = IsotonicRegression(y_min=0.0, y_max=1.0, out_of_bounds="clip").fit(p, yy.astype(float))
return lambda q: np.clip(ir.predict(np.asarray(q, float)), 0.0, 1.0)
def _logit(p):
p = np.clip(np.asarray(p, float), 1e-6, 1 - 1e-6)
return np.log(p / (1 - p))
def fit_platt(p, yy):
"""Platt's sigmoid with his target smoothing, fitted on the score log-odds."""
f, npos = _logit(p), float(yy.sum())
nneg = float(len(yy) - npos)
tgt = np.where(yy == 1, (npos + 1.0) / (npos + 2.0), 1.0 / (nneg + 2.0))
def nll(ab):
z = ab[0] * f + ab[1]
return float(np.sum(tgt * np.logaddexp(0.0, -z) + (1 - tgt) * np.logaddexp(0.0, z)))
a, b = minimize(nll, np.array([1.0, 0.0]), method="L-BFGS-B").x
return lambda q: 1.0 / (1.0 + np.exp(-(float(a) * _logit(q) + float(b))))
def ece(p, yy, bins=10):
idx = np.clip(np.digitize(p, np.linspace(0, 1, bins + 1)[1:-1]), 0, bins - 1)
return float(sum((idx == b).mean() * abs(p[idx == b].mean() - yy[idx == b].mean())
for b in range(bins) if (idx == b).any()))
def draw(family, rng):
if family == "logreg":
return {"C": float(10 ** rng.uniform(-3, 2))}
if family == "random_forest":
return {"n_estimators": int(rng.choice([60, 120])),
"max_depth": [None, 8, 16][int(rng.integers(3))],
"min_samples_leaf": int(rng.choice([1, 2, 5])),
"max_features": ["sqrt", 0.3, 0.6][int(rng.integers(3))]}
return {"learning_rate": float(10 ** rng.uniform(-1.5, -0.3)),
"max_leaf_nodes": int(rng.choice([15, 31, 63])),
"min_samples_leaf": int(rng.choice([5, 20, 50])),
"l2_regularization": float(10 ** rng.uniform(-3, 1)),
"max_iter": int(rng.choice([60, 120]))}
def build(family, cfg, seed):
if family == "logreg":
return make_pipeline(StandardScaler(),
LogisticRegression(C=cfg["C"], max_iter=5000, random_state=seed))
if family == "random_forest":
return RandomForestClassifier(random_state=seed, n_jobs=1, **cfg)
return HistGradientBoostingClassifier(random_state=seed, early_stopping=False, **cfg)
rows, diag, fit_seconds = [], [], {f: 0.0 for f in FAMILIES}
for seed in SEEDS:
idx = np.arange(n)
tr, rest = train_test_split(idx, train_size=0.45, random_state=seed, stratify=y)
tu, rest2 = train_test_split(rest, train_size=0.15 / 0.55, random_state=seed, stratify=y[rest])
de, te = train_test_split(rest2, train_size=0.5, random_state=seed, stratify=y[rest2])
rng, sub_rng = np.random.default_rng(1000 + seed), np.random.default_rng(5000 + seed)
perm = sub_rng.permutation(len(de))
while len(np.unique(y[de][perm[:min(DEC_SIZES[:-1])]])) < 2: # both classes present
perm = sub_rng.permutation(len(de))
for family in FAMILIES:
best, t0 = (np.inf, None), time.time()
for _ in range(HPO_BUDGET):
mdl = build(family, draw(family, rng), seed).fit(X[tr], y[tr])
ll = log_loss(y[tu], mdl.predict_proba(X[tu])[:, 1], labels=[0, 1])
if ll < best[0]:
best = (ll, mdl)
fit_seconds[family] += time.time() - t0
model = best[1]
p_de_all, p_te = model.predict_proba(X[de])[:, 1], model.predict_proba(X[te])[:, 1]
y_de_all, y_te = y[de], y[te]
raw_cost, raw_ps = evaluator(p_te, y_te)
oracle_c = candidates(raw_ps)
for m in DEC_SIZES:
mm = len(de) if m is None else m
sel = perm[:mm]
p_de, y_de = p_de_all[sel], y_de_all[sel]
iso, platt = fit_isotonic(p_de, y_de), fit_platt(p_de, y_de)
iso_cost, _ = evaluator(iso(p_te), y_te)
platt_cost, _ = evaluator(platt(p_te), y_te)
p_de_iso = iso(p_de)
for R in COST_RATIOS:
tstar = R / (R + 1.0)
oracle = float(raw_cost(oracle_c, R).min())
t_emp, t_ie = empirical_threshold(p_de, y_de, R), empirical_threshold(p_de_iso, y_de, R)
got = {"fixed_half": (float(raw_cost(0.5, R)[0]), 0.5),
"raw_tstar": (float(raw_cost(tstar, R)[0]), tstar),
"isotonic_tstar": (float(iso_cost(tstar, R)[0]), tstar),
"platt_tstar": (float(platt_cost(tstar, R)[0]), tstar),
"empirical_raw": (float(raw_cost(t_emp, R)[0]), t_emp),
"isotonic_empirical": (float(iso_cost(t_ie, R)[0]), t_ie)}
for rule, (c, thr) in got.items():
rows.append({"seed": seed, "family": family, "m": mm, "R": R, "rule": rule,
"cost": c, "regret": c - oracle, "thr": thr, "oracle": oracle})
if m is None:
diag.append({"seed": seed, "family": family,
"auroc": float(roc_auc_score(y_te, p_te)),
"logloss_raw": float(log_loss(y_te, p_te, labels=[0, 1])),
"logloss_iso": float(log_loss(y_te, np.clip(iso(p_te), 1e-9, 1 - 1e-9), labels=[0, 1])),
"logloss_platt": float(log_loss(y_te, platt(p_te), labels=[0, 1])),
"brier_raw": float(brier_score_loss(y_te, p_te)),
"ece_raw": ece(p_te, y_te), "ece_iso": ece(iso(p_te), y_te),
"ece_platt": ece(platt(p_te), y_te)})
print(f"seed {seed} done at {time.time() - t_start:.1f}s")
res, dg = pd.DataFrame(rows), pd.DataFrame(diag)
M_FULL = int(res["m"].max())
def cell(sub, col="cost"):
g = sub.groupby("seed")[col].mean()
return {"mean": float(g.mean()), "sd": float(g.std(ddof=1))}
def contrast(sl, rule_a, rule_b):
"""Paired difference cost(rule_a) - cost(rule_b), one value per seed, with a TOST."""
a = sl[sl["rule"] == rule_a].groupby("seed")["cost"].mean()
b = sl[sl["rule"] == rule_b].groupby("seed")["cost"].mean().reindex(a.index)
dif = (a - b).to_numpy()
k, sd = len(dif), float(dif.std(ddof=1))
se, ref = sd / np.sqrt(k), float(b.mean())
same = bool(sd == 0.0 and dif.mean() == 0.0)
tt, crit, delta = stats.ttest_rel(a, b), float(stats.t.ppf(0.975, k - 1)), EQUIV_FRAC * float(b.mean())
tost = None if se <= 0 else float(max(stats.t.sf((dif.mean() + delta) / se, k - 1),
stats.t.cdf((dif.mean() - delta) / se, k - 1)))
return {"mean_diff": float(dif.mean()), "sd_diff": sd,
"ci95_low": float(dif.mean() - crit * se), "ci95_high": float(dif.mean() + crit * se),
"t_stat": None if same else float(tt.statistic),
"p_two_sided": None if same else float(tt.pvalue),
"wilcoxon_p": None if same else float(stats.wilcoxon(a, b).pvalue),
"cohens_dz": float(dif.mean() / sd) if sd > 0 else None, "reference_mean_cost": ref,
"tost_p_equivalent": tost, "seeds_a_worse": int((dif > 0).sum())}
CONTRASTS = [("empirical_raw", "isotonic_tstar"), ("isotonic_tstar", "raw_tstar"),
("empirical_raw", "raw_tstar"), ("platt_tstar", "raw_tstar"),
("platt_tstar", "isotonic_tstar"), ("fixed_half", "raw_tstar")]
def block(sl, which=None):
return {"cost": {r: cell(sl[sl["rule"] == r]) for r in RULES},
"regret": {r: cell(sl[sl["rule"] == r], "regret") for r in RULES},
"regret_ratio": {r: float(sl[sl["rule"] == r]["regret"].mean()
/ max(sl[sl["rule"] == r]["oracle"].mean(), 1e-12)) for r in RULES},
"oracle_cost": cell(sl[sl["rule"] == "raw_tstar"], "oracle"),
"contrasts": {f"{a}_vs_{b}": contrast(sl, a, b) for a, b in (which or CONTRASTS)}}
full = res[res["m"] == M_FULL]
by_R = {str(R): {**block(full[full["R"] == R]), "tstar": R / (R + 1.0)} for R in COST_RATIOS}
M_CONTRASTS = [c for c in CONTRASTS if "fixed_half" not in c] # fixed_half/raw do not depend on m
by_m = {str(int(mm)): block(res[(res["m"] == mm) & (res["R"] == PRIMARY_R)], M_CONTRASTS)
for mm in sorted(res["m"].unique())}
by_family = {}
for fam in FAMILIES:
dgf, sl = dg[dg["family"] == fam], full[(full["R"] == PRIMARY_R) & (full["family"] == fam)]
by_family[fam] = {**block(sl),
"mean_empirical_threshold": float(sl[sl["rule"] == "empirical_raw"]["thr"].mean()),
"sd_empirical_threshold": float(sl[sl["rule"] == "empirical_raw"]["thr"].std(ddof=1)),
"diagnostics": {c: {"mean": float(dgf[c].mean()), "sd": float(dgf[c].std(ddof=1))}
for c in ["auroc", "logloss_raw", "logloss_iso", "logloss_platt",
"brier_raw", "ece_raw", "ece_iso", "ece_platt"]},
"mean_hpo_seconds_per_seed": float(fit_seconds[fam] / len(SEEDS))}
prim = by_R[str(PRIMARY_R)]["contrasts"]["empirical_raw_vs_isotonic_tstar"]
diffs_by_R = [by_R[str(R)]["contrasts"]["empirical_raw_vs_isotonic_tstar"]["mean_diff"] for R in COST_RATIOS]
monotone = bool(all(diffs_by_R[i + 1] >= diffs_by_R[i] for i in range(len(diffs_by_R) - 1)))
rho, prev = stats.spearmanr(COST_RATIOS, diffs_by_R), float(y.mean())
out = {"dataset": "uci_spambase", "n_rows_loaded": n_loaded, "n_analysed": int(n), "n_features": int(d),
"n_positive": int(y.sum()), "prevalence": prev, "n_seeds": len(SEEDS),
"n_train": int(round(0.45 * n)), "n_tune": int(round(0.15 * n)), "n_decision": M_FULL,
"n_test": int(n - round(0.45 * n) - round(0.15 * n) - M_FULL),
"hpo_budget_per_family": HPO_BUDGET, "families": FAMILIES, "rules": RULES,
"cost_ratios": COST_RATIOS, "decision_sizes": [M_FULL if s is None else s for s in DEC_SIZES],
"equivalence_margin_fraction": EQUIV_FRAC,
"primary_endpoint": {"cost_ratio": PRIMARY_R, "decision_size": M_FULL, "pooled_over": FAMILIES, **prim},
"monotonicity": {"mean_diff_by_cost_ratio": [float(v) for v in diffs_by_R],
"monotone_increasing": monotone, "spearman_rho": float(rho.statistic),
"spearman_p": float(rho.pvalue)},
"by_cost_ratio_full_decision_split": by_R, "by_decision_size_at_R10": by_m,
"by_family_at_R10_full": by_family,
"cheapest_rule_by_cost_ratio": {str(R): min(RULES, key=lambda r: by_R[str(R)]["cost"][r]["mean"])
for R in COST_RATIOS},
"trivial_baseline_cost": {str(R): {"classify_all_as_ham": prev,
"classify_all_as_spam": float(R * (1 - prev))} for R in COST_RATIOS},
"runtime_seconds": float(time.time() - t_start),
"hypothesis_supported": bool(prim["mean_diff"] > 0 and monotone
and prim["p_two_sided"] is not None and prim["p_two_sided"] < 0.05)}
def clean(o):
"""Recursively cast numpy scalars and turn any non-finite float into null."""
if isinstance(o, dict):
return {k: clean(v) for k, v in o.items()}
if isinstance(o, (list, tuple)):
return [clean(v) for v in o]
if isinstance(o, (np.floating, float)):
return float(o) if np.isfinite(o) else None
return int(o) if isinstance(o, np.integer) else bool(o) if isinstance(o, np.bool_) else o
print("RESULT_JSON: " + json.dumps(clean(out), separators=(",", ":"), allow_nan=False))
4.Analysis
Analysis complete: experiment succeeded; hypothesis refuted; 9 key numbers.
Results and analysis
The run completed in 61.1 s of wall-clock time with exit code 0 on the first attempt. All 4,601 loaded e-mails passed the filters (finite values on all 57 predictors, is_spam in {0,1}); none were dropped, and 1,813 (39.4%) are spam. Each of 15 seeds used a stratified four-way split into 2,070 training, 690 tuning, 920 decision and 921 test e-mails. Three model families (L2 logistic regression, random forest, histogram gradient boosting) were each given an identical budget of 10 random hyperparameter draws selected on the tuning split by log loss; mean search time per seed was 0.04 s, 2.57 s and 1.41 s respectively. Reported costs are per test e-mail with c_fp = R (a discarded legitimate e-mail) and c_fn = 1, so they are on the scale of R, not of an error rate.
PRIMARY ENDPOINT: REFUTED. At the pre-registered operating point (R = 10, full decision split of m = 920, pooled across the three families) the paired difference D = cost(empirical cut-off) - cost(isotonic + t*) is +0.00029 +/- 0.00272 per e-mail (mean +/- s.d. over 15 seeds), 95% CI [-0.00122, +0.00180], paired t = 0.41, p = 0.687, Wilcoxon p = 0.256, Cohen's d_z = 0.11, with empirical thresholding worse in only 6 of 15 seeds. The hypothesis required D to be significantly positive; it is not. A two one-sided-tests procedure with a margin of 5% of the reference cost (0.00908 per e-mail) rejects the presence of any difference larger than that margin with p = 2.75e-9, so this is a positive finding of equivalence rather than a failure of power: the two recipes are interchangeable on this benchmark to well within 5% of the operating cost.
SECOND CLAUSE ALSO REFUTED. The predicted growth of D with the cost ratio does not appear. Across R = 1, 2, 5, 10, 20 (m = 920) the mean differences are -0.00005, +0.00010, -0.00012, +0.00029, -0.00017; the sequence is not monotone increasing and Spearman rho = -0.30 (p = 0.624). Equivalence holds in every cost-ratio cell (TOST p <= 1.9e-10 at R = 1, 2, 5, 20 and p = 2.8e-9 at R = 10) and in every decision-set cell (TOST p = 2.2e-4, 1.2e-6, 2.1e-5 and 2.8e-9 at m = 100, 250, 500, 920). One cell, m = 250 at R = 10, shows a nominally significant difference of -0.00222 +/- 0.00392 (p = 0.046, Wilcoxon p = 0.026) in the direction OPPOSITE to the hypothesis, i.e. empirical thresholding marginally cheaper; at 1.1% of that cell's cost and without correction for the twenty-odd cells examined, this should be read as noise, not as evidence for the reverse claim.
WHAT DOES MATTER (1): SETTING THE CUT-OFF AT ALL. The naive 0.5 cut-off is catastrophic under asymmetry. At R = 10 it costs 0.2956 +/- 0.0345 against 0.1955 +/- 0.0137 for the zero-parameter rule that simply applies t* = 0.909 to the raw score (paired difference +0.10016, 95% CI [+0.08288, +0.11744], p = 5.9e-9), and at R = 20 it costs 0.5531 +/- 0.0701 against 0.2561 +/- 0.0208 (difference +0.29702, p = 2.3e-10) -- a regret of 176.7% over the oracle cut-off, versus 28.1% for raw-score t*. At R = 1 and R = 2, where t* is 0.5 and 0.667, the gap is nil or small (+0.00000 and -0.00145, p = 0.34).
WHAT DOES MATTER (2): HOW MUCH HELD-OUT DATA YOU HAVE. The informative contrast is not calibration versus threshold search but either of them versus spending no data at all. At R = 10 with the full m = 920 both data-driven recipes beat raw-score t*: isotonic + t* by -0.01387 (95% CI [-0.02212, -0.00563], p = 0.0029) and the empirical cut-off by -0.01358 (95% CI [-0.02177, -0.00540], p = 0.0031), about 7% of the operating cost. At m = 100 both are significantly WORSE than using no held-out data at all: +0.03178 (95% CI [+0.00412, +0.05943], p = 0.027) for isotonic and +0.03356 (95% CI [+0.00490, +0.06222], p = 0.025) for the empirical cut-off, a 17% penalty. The crossover sits near m = 250, where neither differs from raw-score t* (+0.00306, p = 0.640, and +0.00084, p = 0.896). Both recipes therefore inherit the same finite-sample penalty, which is why they track each other so closely.
WHAT DOES MATTER (3): THE CALIBRATION FAMILY AND THE BASE LEARNER. Platt's sigmoid, with two parameters, is the only rule that is never significantly worse than raw-score t* at any m and is the cheapest rule at R = 5, 10 and 20 (0.1342 +/- 0.0112, 0.1808 +/- 0.0160, 0.2399 +/- 0.0283). At m = 100 it costs 0.2035 +/- 0.0227 against 0.2273 +/- 0.0488 for isotonic and 0.2291 +/- 0.0510 for the empirical cut-off, and at m = 500 it is the only rule that significantly beats raw-score t* (-0.01445, 95% CI [-0.02134, -0.00756], p = 0.0005). Its advantage over isotonic is not individually significant at R = 10 (-0.00080, p = 0.865); the consistency across cells is the signal. The family interaction is exactly as the calibration literature predicts. For the random forest, whose scores are the most distorted (test ECE 0.048), calibration is transformative: regret against the oracle falls from 0.0662 with raw t* to 0.0166 with isotonic (paired difference -0.04966, p = 3.0e-5) and its ECE to 0.017. For histogram gradient boosting, already nearly calibrated (ECE 0.020, log loss 0.148, AUROC 0.986), isotonic calibration HURTS: cost rises by +0.01035 (95% CI [-0.00124, +0.02194], p = 0.076) and test log loss degrades from 0.148 to 0.187. For logistic regression, trained on a proper loss, nothing moves significantly in either direction (isotonic -0.00232, p = 0.667). So the benefit of post-hoc calibration is a property of the base learner's distortion, not of the decision rule.
RESIDUAL GAP. Even the best rule stays well above the oracle threshold chosen on the test set itself: at R = 10 the cheapest rule (Platt + t*) has regret 0.0231 on an oracle cost of 0.1577, i.e. 14.6%, and at R = 20 regret 0.0400 on an oracle of 0.1999, i.e. 20.0%. No amount of held-out data spent on the operating point closed that gap in this experiment, which is consistent with the argument that marginal calibration is not the condition that governs threshold decisions.
Key numbers
| Metric | Value | Interpretation |
|---|---|---|
| Primary endpoint: paired difference cost(empirical cut-off) - cost(isotonic + t*), R = 10, m = 920, pooled over three families, 15 seeds | +0.00029 +/- 0.00272 per e-mail; 95% CI [-0.00122, +0.00180]; t = 0.41, p = 0.687; Wilcoxon p = 0.256; d_z = 0.11 | The pre-registered hypothesis required this to be significantly positive. It is not, so the hypothesis is refuted at its own reference operating point. |
| Equivalence test (TOST) on the primary endpoint, margin 5% of the reference cost = 0.00908 | p = 2.75e-9 | The null result is a demonstration of equivalence, not a lack of power: any true difference between the two recipes is smaller than 5% of the operating cost. |
| Mean paired difference by cost ratio R = 1, 2, 5, 10, 20 (m = 920) | -0.00005, +0.00010, -0.00012, +0.00029, -0.00017; not monotone; Spearman rho = -0.30, p = 0.624 | The second clause of the hypothesis, that calibration's advantage grows with the cost ratio, is refuted; the differences fluctuate around zero at every asymmetry tested. |
| Naive 0.5 cut-off versus raw-score t*, R = 10 and R = 20 (per-e-mail cost) | R = 10: 0.2956 +/- 0.0345 vs 0.1955 +/- 0.0137, difference +0.10016 (95% CI [+0.08288, +0.11744]), p = 5.9e-9. R = 20: 0.5531 +/- 0.0701 vs 0.2561 +/- 0.0208, difference +0.29702, p = 2.3e-10 | The decision that matters is moving the cut-off at all; the zero-parameter Bayes threshold captures most of the achievable gain before any held-out data is spent. |
| Data-driven rules versus raw-score t* at R = 10, small versus large decision split | m = 100: isotonic +0.03178 (95% CI [+0.00412, +0.05943], p = 0.027), empirical +0.03356 (95% CI [+0.00490, +0.06222], p = 0.025). m = 920: isotonic -0.01387 (95% CI [-0.02212, -0.00563], p = 0.0029), empirical -0.01358 (95% CI [-0.02177, -0.00540], p = 0.0031) | Both recipes carry the same finite-sample penalty: with 100 held-out e-mails each is about 17% worse than spending no data, and only past roughly 250 points does either start to pay. This, not the choice between them, is the decision that changes cost. |
| Platt (sigmoid) + t* cost at R = 5, 10, 20 with m = 920 | 0.1342 +/- 0.0112, 0.1808 +/- 0.0160, 0.2399 +/- 0.0283 (cheapest rule in all three cells); at m = 500 it beats raw-score t* by -0.01445 (95% CI [-0.02134, -0.00756], p = 0.0005) | The two-parameter sigmoid is the most robust rule tested: never significantly worse than doing nothing, and the only one that helps at moderate m. Its margin over isotonic in any single cell is not significant (R = 10: -0.00080, p = 0.865). |
| Base-learner interaction: regret against the oracle cut-off at R = 10, m = 920 | Random forest 0.0662 (raw t*) to 0.0166 (isotonic), difference -0.04966, p = 3.0e-5, ECE 0.048 to 0.017. Histogram gradient boosting 0.0232 (raw t*) to 0.0336 (isotonic), difference +0.01035, p = 0.076, ECE 0.020, log loss 0.148 to 0.187. Logistic regression 0.0238 to 0.0215, difference -0.00232, p = 0.667 | Post-hoc calibration pays exactly where the scores are distorted and can hurt where they are not; the effect belongs to the base learner, not to the decision recipe. |
| Residual regret of the cheapest rule against the test-set oracle threshold | R = 10: regret 0.0231 on oracle cost 0.1577 (14.6%). R = 20: regret 0.0400 on oracle cost 0.1999 (20.0%) | No decision rule tested came close to the best achievable operating point, so a substantial part of the cost gap is not addressable by choosing between these recipes. |
| Sample and protocol actually analysed | 4,601 of 4,601 loaded e-mails, 57 predictors, 1,813 spam (39.4%); 15 seeds; per seed 2,070 train / 690 tune / 920 decision / 921 test; 10 random hyperparameter draws per family; total wall-clock 61.1 s | No rows were dropped, the tuning budget was identical across families, and the decision-rule data budget was matched exactly because every rule was fitted on the same decision subsample. |
Caveats
- One dataset. Spambase is a single, well-separated, late-1990s corpus from one company (test AUROC 0.968-0.986 across families). On an easier or harder problem, or one with a less bimodal score distribution, the equivalence between the two recipes need not hold; nothing here licenses a general claim about tabular classification.
- The equivalence claim is bounded by the margin actually tested. TOST rejects differences larger than 5% of the reference cost, not differences of any size; a true effect of, say, 1% of the operating cost would be invisible at 15 seeds.
- Multiplicity is not controlled. Roughly twenty paired contrasts were computed across cost ratios, decision-split sizes and families. One of them (m = 250, R = 10; p = 0.046) is nominally significant in the direction opposite to the hypothesis and is best read as sampling noise.
- Costs are stipulated, not measured. The cost ratios R in {1, 2, 5, 10, 20} are assumptions about how much worse discarding a legitimate e-mail is than admitting spam; the dataset carries no cost information, and results are reported per e-mail on the scale of R.
- The seed-to-seed variation comes from overlapping resamples of one finite corpus, so the standard deviations reported measure split variability on these 4,601 e-mails, not sampling variability over e-mail corpora, and the paired tests treat the 15 seeds as independent replicates when they are not fully independent.
- Models were fitted on the 2,070-row training split only and were not refitted on train plus tune after selection, and the search budget was 10 random draws per family rather than an exhaustive search, so absolute cost levels would improve slightly under a larger budget even though the comparison between decision rules is unaffected.
- Only two calibration maps were tested (isotonic regression and Platt's sigmoid). Temperature scaling, beta calibration and cross-fitted variants such as scikit-learn's CalibratedClassifierCV were not evaluated, and a cross-fitted calibrator might use a small decision split more efficiently than the single-fit version used here.
- The oracle threshold used for the regret figures is selected on the test split itself, so it is an optimistic reference by construction; it is a diagnostic for how far every rule sits from the best achievable operating point, not an attainable baseline.
- The empirical cut-off is defined as the midpoint of the flat minimum of the validation cost curve. Other tie-breaking conventions for that plateau would give slightly different numbers, although the plateau is narrow at large m.
5.Paper writing
Manuscript drafted: “Calibrate or Tune the Cut-off? A Matched-Budget Contest Between Two Recipes for Cost-Sensitive Thresholding” (≈ 3807 words).
≈ 3807 words · AI-drafted manuscript · not peer-reviewed
Calibrate or Tune the Cut-off? A Matched-Budget Contest Between Two Recipes for Cost-Sensitive Thresholding
Abstract
Two standard recipes prescribe incompatible uses of the same held-out sample when misclassification costs are asymmetric. Decision theory says to fit a post-hoc calibration map and decide at the fixed Bayes cut-off ; the thresholding literature says to skip calibration and search the cut-off minimising empirical held-out cost. We run them head to head on the real Spambase benchmark (4,601 e-mails) with matched data and tuning budgets: 15 seeds, three tuned model families, five cost ratios, four held-out sizes. They are indistinguishable. At the pre-registered operating point the paired difference is per e-mail (95% CI , ), and two one-sided tests reject any difference exceeding 5% of the operating cost (). Our hypothesis — calibration wins, and wins harder as costs grow more asymmetric — is refuted. What moves cost is whether the cut-off is moved at all, how much held-out data there is, and how distorted the base learner's scores are: with 100 held-out e-mails both recipes are significantly worse than spending none.
1 Introduction
Turning a classifier into a decision rule under asymmetric costs forces a choice about how to spend a held-out sample. Elkan's recipe [1] fits a post-hoc calibration map on that data [4, 5] and thresholds at the constant , which needs no further estimation. Sheng and Ling's [3] skips calibration and searches the cut-off minimising cost on the same data, since accurate probabilities are then unnecessary; it is the software default [13]. We found no controlled comparison giving the two the same held-out sample, base models and tuning budget.
There is a concrete reason to expect a difference. Empirical threshold search is a one-dimensional model selection on a step-function criterion, and selection over-fitting theory [10, 11] says its penalty scales with that criterion's variance — governed by how many held-out points fall near the operating point, which collapses as pushes towards one. A calibration map instead pools the whole sample, so it should win, and win by more at extreme cost ratios and small held-out sizes. We pre-registered that hypothesis; it is false, because calibration pays the same finite-sample penalty.
Our contributions:
- A matched-budget head-to-head test of calibrate-then- against empirical threshold search on identical held-out samples: 15 seeds, three tuned model families, five cost ratios, four held-out sizes, scored by cost per instance and by regret against an oracle cut-off.
- A negative result stated as an equivalence, not a failure to reject: two one-sided tests bound the difference below 5% of the operating cost in every cell examined.
- A relocation of the question. Moving the cut-off at all gains an order of magnitude more than any difference between the recipes, and both are significantly worse than the zero-parameter rule on small held-out samples; the crossover here is near 250 examples.
- Evidence that calibration's benefit belongs to the base learner, not the decision recipe: transformative for a random forest, neutral for logistic regression, harmful for gradient boosting.
2 Related Work
Cost-sensitive decision theory. Elkan [1] derives the Bayes-optimal cut-off for a cost matrix, conditional on true posteriors; Provost and Fawcett [2] give the geometric dual through the ROC convex hull and argue that operating conditions are usually imprecise, motivating a sweep over cost ratios.
Empirical thresholding. Sheng and Ling [3] select the cut-off minimising held-out empirical cost, explicitly to avoid needing accurate probabilities — tuned-threshold classification as implemented in mainstream tooling [13], and the baseline we test.
Post-hoc calibration. Zadrozny and Elkan [4] introduced isotonic (pool-adjacent-violators) calibration; Niculescu-Mizil and Caruana [5] documented which learners distort scores, reporting that isotonic regression needs a large calibration set while the two-parameter sigmoid tolerates a small one. Guo et al. [6] revived the topic for deep networks, and Filho et al. [12] survey the field, naming cost-sensitive decisions as its motivating application.
Calibration and decisions. Sahoo et al. [7] show that average calibration does not guarantee accurate loss prediction for threshold decisions, and define threshold calibration as the exact condition; Perez-Lebel et al. [8] show that grouping loss survives marginal calibration. Conversely Blasiok et al. [9] show that near-optimality of a proper loss implies smooth calibration, so log-loss-trained models should gain least from recalibration.
Selection over-fitting. Cawley and Talbot [10] establish that a selection criterion's variance drives over-fitting of the selection step, often by as much as differences between algorithms; Schneider et al. [11] quantify this for hyperparameter optimisation.
3 Problem Setup
Let with and . A rule incurs on a false positive and on a false negative, zero otherwise; we set and . The risk is
and a score induces the family .
Proposition 1 (Elkan [1]). If , then is minimised at . Proof sketch. Given , a positive decision costs and a negative one , so positive is preferred iff ; pointwise optimality gives global optimality.
When one may instead estimate the cut-off from a held-out set by minimising .
Proposition 2 (effective sample size of the empirical cut-off). Let the scores have density on . Then is piecewise constant with jumps only at the observed scores, and the count of observations inside is , with mean . Proof sketch. Each summand changes only as crosses some , so the minimiser is fixed by the ordering, and only points inside the window discriminate between cut-offs inside it; membership is i.i.d. Bernoulli.
The informative sample for locating the cut-off is therefore , not , and is small as for a well-separated classifier. With [10, 11] this gives the pre-registered hypothesis: is positive, increases with and decreases with . Proposition 2 constrains only the search arm; whether calibration escapes the same penalty is what the experiment settles.
4 Method
Six rules are built from the same fitted model and the same held-out decision split, so the data budget is matched exactly. FIXED-0.5 thresholds the raw score at ; RAW- thresholds it at , spending no held-out data; ISO- and PLATT- fit an isotonic map [4] or Platt's sigmoid on the score log-odds with target smoothing [5], then threshold at ; EMP-RAW searches the empirical cut-off on raw scores [3]; ISO-EMP fits the isotonic map, then searches on calibrated scores.
Algorithm 1 Matched-budget evaluation of cost-sensitive decision rules
1: input D, seed k, HPO budget B, cost ratios R, decision sizes M
2: split D stratified: Tr 45%, Tu 15%, De 20%, Te 20%, seed k
3: for each model family:
4: fit B random configurations on Tr; keep f minimising log loss on Tu
5: s_De <- f(De); s_Te <- f(Te) # cost-agnostic selection
6: for each m in M:
7: S <- first m of a fixed random permutation of De
8: g_iso <- Isotonic(s_De[S], y_De[S]); g_platt <- Platt(...)
9: for each R:
10: t* <- R/(R+1)
11: t_emp <- argmin_t cost_R(t; s_De[S]) # plateau midpoint
12: t_ie <- argmin_t cost_R(t; g_iso(s_De[S]))
13: score the six rules on (s_Te, y_Te)
14: oracle <- min over cut-offs of the test cost of s_Te
15: return per-(seed, family, m, R, rule) costs and regrets
Cut-offs are evaluated exactly and tie-safely by sorting the scores once and using a cumulative count of positives, so a sweep costs , not . Per (seed, family) the cost is model fits plus for all post-hoc rules, negligible beside the fits.
5 Experiments
5.1 Setup
Data. We use the Spambase e-mail spam classification benchmark, dataset 94 of the UCI Machine Learning Repository, donated by Hewlett-Packard Labs and licensed CC BY 4.0 [14]. We retrieved the archive on 2026-09-08 (SHA-256 813ac1df8effac70…); it contains 4,601 real e-mails collected at Hewlett-Packard Labs, Palo Alto, and donated on 30 June 1999. It carries no per-message timestamp, so the sample period is known only at corpus level. Each e-mail has 57 pre-engineered predictors: 48 word_freq_* columns (percentage of words matching a token) and 6 char_freq_* columns (percentage of characters matching a character), both on a 0–100 scale, plus capital_run_length_average (mean length of an uninterrupted run of capitals), capital_run_length_longest and capital_run_length_total (counts of capitals). The target is_spam is 1 for spam, 0 for legitimate mail. Two filters — all 57 predictors finite, and is_spam — retained all 4,601 rows; none were dropped. Positives number 1,813 (39.4%). Spam is the positive class, so a false positive discards a legitimate e-mail at cost and a false negative admits spam at cost 1.
Protocol. For each of 15 seeds we draw a stratified four-way split into 2,070 training, 690 tuning, 920 decision and 921 test e-mails. Three families — L2 logistic regression on standardised features, random forest, and histogram gradient boosting [13] — each receive 10 random hyperparameter draws, selected on the tuning split by log loss, a proper rule that keeps model selection independent of the decision rule. Models are fitted on the training split only and not refitted after selection. Cost ratios are ; decision-split sizes are , the smaller ones nested prefixes of a single seeded permutation shared by all families.
Metrics and inference. The primary metric is expected cost per test e-mail; we also report regret against an oracle cut-off chosen on the test split itself, which isolates the decision rule from model quality, plus log loss, Brier score, AUROC and 10-bin expected calibration error (ECE). Results are mean standard deviation over 15 seeds; contrasts are paired within seed (family-averaged where pooled) and tested with a two-sided paired -test and a Wilcoxon signed-rank test. Equivalence uses two one-sided tests (TOST) with a margin of 5% of the reference rule's mean cost. The pre-registered primary endpoint is EMP-RAW minus ISO- at , , pooled over families. The experiment runs in 61.1 s of single-threaded CPU time.
5.2 Main Results
Table 1: Expected cost per test e-mail by decision rule and cost ratio (; mean±s.d. over 15 seeds).
| Rule | |||||
|---|---|---|---|---|---|
| FIXED-0.5 | 0.0639±0.0049 | 0.0897±0.0072 | 0.1669±0.0169 | 0.2956±0.0345 | 0.5531±0.0701 |
| RAW- | 0.0639±0.0049 | 0.0911±0.0070 | 0.1411±0.0118 | 0.1955±0.0137 | 0.2561±0.0208 |
| ISO- | 0.0636±0.0059 | 0.0875±0.0053 | 0.1344±0.0129 | 0.1816±0.0256 | 0.2440±0.0342 |
| PLATT- | 0.0641±0.0051 | 0.0903±0.0073 | 0.1342±0.0112 | 0.1808±0.0160 | 0.2399±0.0283 |
| EMP-RAW | 0.0635±0.0057 | 0.0876±0.0055 | 0.1343±0.0127 | 0.1819±0.0253 | 0.2439±0.0346 |
| ISO-EMP | 0.0639±0.0056 | 0.0877±0.0055 | 0.1350±0.0125 | 0.1821±0.0269 | 0.2445±0.0348 |
| Oracle cut-off | 0.0590±0.0048 | 0.0812±0.0062 | 0.1204±0.0107 | 0.1577±0.0148 | 0.1999±0.0201 |
Notes. Costs are per e-mail with , , so they lie on the scale of and are not error rates; each entry averages the three families within a seed, then across seeds. Bold marks the cheapest rule per column. The italic oracle row picks the cut-off on the test split itself: an unattainable lower bound, not a competitor. is 0.500, 0.667, 0.833, 0.909, 0.952 across columns; trivial baselines at cost 0.394 (all ham) and 6.060 (all spam).
Table 2: Paired contrasts at , , pooled over the three families (15 seeds).
| Contrast (A − B) | Mean diff. | 95% CI | (paired ) | Wilcoxon | TOST | |
|---|---|---|---|---|---|---|
| EMP-RAW − ISO- (primary) | +0.00029±0.00272 | [−0.00122, +0.00180] | 0.687 | 0.256 | 0.11 | |
| ISO- − RAW- | −0.01387 | [−0.02212, −0.00563] | 0.0029 | 0.0034 | −0.93 | 0.848 |
| EMP-RAW − RAW- | −0.01358 | [−0.02177, −0.00540] | 0.0031 | 0.0064 | −0.92 | 0.833 |
| PLATT- − RAW- | −0.01467 | [−0.02195, −0.00739] | 0.0007 | 0.0020 | −1.12 | 0.914 |
| PLATT- − ISO- | −0.00080 | [−0.01066, +0.00906] | 0.865 | 0.978 | −0.04 | 0.047 |
| FIXED-0.5 − RAW- | +0.10016 | [+0.08288, +0.11744] | 0.0007 | 3.21 | 1.000 |
Notes. A positive difference means A is the more expensive rule. TOST gives the larger of the two one-sided -values against a margin of 5% of B's mean cost (0.00908 for the primary row); small means the difference is bounded inside that margin. Only the primary row carries the s.d. of paired differences.
The primary endpoint (Table 2) is per e-mail, , , with empirical thresholding the worse arm in only 6 of 15 seeds; the hypothesis required it to be significantly positive, so it is refuted. The second clause fails too: mean differences across are , , , , , not monotone increasing (Spearman , ). Equivalence within the 5% margin holds at every cost ratio (TOST at ; at ), so this is an informative null, not a shortage of power. Table 1 does show a large effect of moving the cut-off at all: at the naive rule costs 0.5531 against 0.2561 for the zero-parameter Bayes rule, a difference of ().
5.3 Ablations
Table 3: Decision-split size at (cost per e-mail; mean±s.d. over 15 seeds).
| Rule | ||||
|---|---|---|---|---|
| RAW- (no data spent) | 0.1955±0.0137 (0.0378) | 0.1955±0.0137 | 0.1955±0.0137 | 0.1955±0.0137 |
| ISO- | 0.2273±0.0488 | 0.1986±0.0311 | 0.1897±0.0289 | 0.1816±0.0256 |
| PLATT- | 0.2035±0.0227 | 0.1876±0.0174 (0.0299) | 0.1810±0.0161 (0.0233) | 0.1808±0.0160 (0.0231) |
| EMP-RAW | 0.2291±0.0510 | 0.1963±0.0301 | 0.1880±0.0274 | 0.1819±0.0253 |
| ISO-EMP | 0.2306±0.0503 | 0.1983±0.0321 | 0.1898±0.0295 | 0.1821±0.0269 |
Notes. RAW- spends no decision data, so its row is constant; the oracle cost is 0.1577 throughout, and a rule's regret is its cost minus that oracle. Bold marks the cheapest rule per column. Against RAW- at , ISO- is worse by (95% CI , ) and EMP-RAW by (, ); at neither differs (, ); at both are better (, ). The recipes stay equivalent at every (TOST , , , ).
Table 4: Base learner at , (mean±s.d. over 15 seeds).
| Rule | Logistic regression | Random forest | Gradient boosting |
|---|---|---|---|
| FIXED-0.5 | 0.3677±0.0393 | 0.2515±0.0459 | 0.2677±0.0492 |
| RAW- | 0.2604±0.0252 | 0.1860±0.0180 | 0.1400±0.0250 |
| ISO- | 0.2581±0.0373 | 0.1364±0.0268 | 0.1503±0.0310 |
| PLATT- | 0.2625±0.0301 | 0.1433±0.0215 | 0.1367±0.0188 |
| EMP-RAW | 0.2587±0.0376 | 0.1367±0.0237 | 0.1503±0.0310 |
| ISO-EMP | 0.2574±0.0374 | 0.1385±0.0285 | 0.1503±0.0310 |
| Oracle cut-off | 0.2366±0.0310 | 0.1198±0.0183 | 0.1168±0.0166 |
| Raw AUROC | 0.9678±0.0049 | 0.9809±0.0045 | 0.9855±0.0031 |
| ECE, raw / isotonic | 0.0386 / 0.0255 | 0.0483 / 0.0174 | 0.0195 / 0.0177 |
| Log loss, raw / isotonic | 0.2495 / 0.2619 | 0.1892 / 0.1859 | 0.1479 / 0.1869 |
Notes. Bold marks the cheapest of the six rules per column; a rule's oracle regret is its cost minus the italic oracle row. Diagnostics use the test split with the full decision split for calibration; ECE uses 10 equal-width bins. Against RAW-, isotonic calibration cuts the random forest's cost by (95% CI , ), raises gradient boosting's by (, ), and leaves logistic regression unchanged (, ).
Platt's sigmoid is the cheapest rule at , is never significantly worse than RAW- at any , and is the only rule that significantly beats it at (, 95% CI , ). Its margin over isotonic in any one cell is not significant (, , at ): the evidence is consistency across cells, not any single contrast.
5.4 Analysis
First, the choice between the recipes does not matter, and the equivalence tests make that a positive claim: the difference is bounded below 5% of the operating cost in all eight cost-ratio and held-out-size cells and in all three per-family cells. The mechanism of Proposition 2 is real but non-differential — isotonic calibration inherits the same small-sample penalty as the threshold search, so the arms move together rather than apart.
Second, what matters is whether to spend held-out data at all. At both recipes cost about 17% more than the zero-parameter rule; the crossover is near ; only at do both pay off, by roughly 7%. That the two-parameter sigmoid stays closest to the zero-parameter rule at small matches the sample-size guidance of [5], transposed to a decision loss.
Third, calibration's benefit belongs to the base learner: large where scores are distorted (random forest, oracle regret 0.0662 to 0.0166) and absent or harmful where they are not (gradient boosting, regret 0.0232 to 0.0336), while logistic regression trained on a proper loss is unmoved — as [5] and [9] predict. Every rule still stays far from the oracle: at the cheapest leaves regret 0.0231 on an oracle cost of 0.1577 (14.6%), and 20.0% at — a gap no way of spending the decision split closed, consistent with [7] and [8].
6 Limitations
This is one dataset, and an easy one: test AUROC is 0.968–0.986 across families and the scores are strongly bimodal, so few test points sit near any plausible cut-off. Nothing here licenses a general claim about tabular classification, and on a harder problem the recipes need not be equivalent. Spambase is a late-1990s corpus from a single company, so its vocabulary and spam/ham boundary are dated and its 39.4% prevalence far exceeds a modern mail stream's.
The equivalence claim is bounded by the margin tested: TOST rejects differences above 5% of the reference cost, not any difference, and an effect of order 1% would be invisible with 15 seeds. Multiplicity is not controlled: of roughly twenty paired contrasts two are nominally significant — EMP-RAW versus ISO- at , (, ), opposite to our hypothesis, and the same contrast for logistic regression at (, ), agreeing with it — but at about 1% and 0.2% of their cells' costs, uncorrected, both read as noise. The 15 seeds resample one finite corpus, so the standard deviations measure split variability on these 4,601 e-mails, not variability across corpora, and the paired tests treat overlapping resamples as independent.
Costs are stipulated, not measured: the dataset carries none, and encodes an assumption about how much worse discarding legitimate mail is than admitting spam. Models were fitted on the 2,070-row training split only, not refitted after selection, and searched with 10 draws per family, so absolute costs would improve under a larger budget even though the between-rule comparison would not. Only two calibration maps were run; temperature scaling, beta calibration and cross-fitted calibrators were not, and a cross-fitted map might use a 100-point decision split more efficiently — the most important unexecuted check. The regret oracle is selected on the test split itself, so it is optimistic by construction: a diagnostic, not an attainable baseline. Finally, the empirical cut-off is the midpoint of the flat minimum of the validation cost curve; other tie-breaking conventions would shift the numbers slightly, though the plateau is narrow at large .
7 Conclusion
We gave two standard recipes for cost-sensitive thresholding the same held-out sample, the same tuned base models and the same budget, and they finished level: at the pre-registered operating point the paired difference is per e-mail, 95% CI , with equivalence within 5% of the operating cost at every cost ratio and held-out size tested. Elkan's fixed Bayes cut-off and Sheng and Ling's empirical search disagree, but on this benchmark the disagreement costs nothing. What does cost something is moving the cut-off away from 0.5 at all, having enough held-out data for fitting anything on it to be worthwhile — below roughly 250 examples it is not — and knowing whether the base learner's scores are distorted.
References
[1] C. Elkan. The foundations of cost-sensitive learning. In IJCAI, pages 973–978, 2001.
[2] F. Provost and T. Fawcett. Robust classification for imprecise environments. Machine Learning, 42(3):203–231, 2001.
[3] V. S. Sheng and C. X. Ling. Thresholding for making classifiers cost-sensitive. In AAAI, pages 476–481, 2006.
[4] B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In KDD, pages 694–699, 2002.
[5] A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In ICML, pages 625–632, 2005.
[6] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
[7] R. Sahoo, S. Zhao, A. Chen, and S. Ermon. Reliable decisions with threshold calibration. In NeurIPS 34, 2021.
[8] A. Perez-Lebel, M. Le Morvan, and G. Varoquaux. Beyond calibration: estimating the grouping loss of modern neural networks. In ICLR, 2023.
[9] J. Blasiok, P. Gopalan, L. Hu, and P. Nakkiran. When does optimizing a proper loss yield calibration? In NeurIPS 36, 2023.
[10] G. C. Cawley and N. L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. JMLR, 11:2079–2107, 2010.
[11] L. Schneider, B. Bischl, and M. Feurer. Overtuning in hyperparameter optimization. In AutoML, 2025. arXiv:2506.19540.
[12] T. Silva Filho, H. Song, M. Perello-Nieto, R. Santos-Rodriguez, M. Kull, and P. Flach. Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112:3211–3260, 2023.
[13] F. Pedregosa et al. Scikit-learn: machine learning in Python. JMLR, 12:2825–2830, 2011.
[14] M. Hopkins, E. Reeber, G. Forman, and J. Suermondt. Spambase [Dataset]. UCI Machine Learning Repository, 1999. DOI: 10.24432/C53G6X.
Appendix
Table 5: Additional diagnostics at , (mean±s.d. over 15 seeds).
| Quantity | Logistic regression | Random forest | Gradient boosting |
|---|---|---|---|
| Empirical cut-off selected | 0.849±0.072 | 0.744±0.066 | 0.903±0.072 |
| Brier score (raw) | 0.0653±0.0054 | 0.0478±0.0033 | 0.0395±0.0045 |
| Log loss after Platt | 0.2446±0.0231 | 0.1684±0.0193 | 0.1439±0.0151 |
| ECE after Platt | 0.0406±0.0063 | 0.0245±0.0053 | 0.0181±0.0041 |
| Search time per seed (s) | 0.04 | 2.57 | 1.41 |
Notes. The theoretical cut-off at is ; the empirical one sits close to it for gradient boosting, whose scores are near-calibrated, and well below it for the random forest, whose scores are compressed away from the extremes. Search time is the mean per-seed wall-clock time of the 10-draw search.
Reproducibility. Seeds are ; split seeds, hyperparameter-draw seeds () and decision-subsample seeds () are fixed and BLAS/OpenMP threads pinned to one, so the run is deterministic. Ranges: logistic regression on standardised features; random forest n_estimators , max_depth , min_samples_leaf , max_features ; histogram gradient boosting learning_rate , max_leaf_nodes , min_samples_leaf , l2_regularization , max_iter , early stopping off. The analysis is a single Python script using NumPy, pandas, SciPy and scikit-learn [13]; it loads the archive of Section 5.1 and finishes in 61.1 s. Script and raw result object are archived with the run record.
6.Review
Automated review: Minor revision (novelty 3, rigor 4, clarity 4, reproducibility 5).
| Novelty 3/5 | Rigor 4/5 | Clarity 4/5 | Reproducibility 5/5 |
A carefully executed, honestly reported head-to-head test of the two standard recipes for cost-sensitive thresholding -- calibrate then apply Elkan's Bayes cut-off, versus search the cut-off that minimises held-out cost -- on the real UCI Spambase corpus (4,601 e-mails, 57 predictors, 1,813 spam), with the data budget matched exactly and the tuning budget matched across three model families. The pre-registered hypothesis is refuted twice over: the paired difference at R = 10, m = 920 is +0.00029 +/- 0.00272 per e-mail (95% CI [-0.00122, +0.00180], p = 0.687) and shows no growth with the cost ratio (Spearman rho = -0.30, p = 0.624). The authors convert this into an equivalence result rather than a failure to reject, and then relocate the interesting question: moving the cut-off at all is worth ten to thirty times more than the choice between recipes, both recipes are significantly worse than spending no held-out data below about 250 examples, and the benefit of calibration is a property of the base learner (transformative for the random forest, harmful for gradient boosting). I verified every reported number, the dataset provenance, the sample size, the split sizes, the seed count and the search budget against the experiment record and found no discrepancy, and the data is nowhere described as simulated. The venue's conventions are followed: contributions as bullets, algorithm pseudocode with complexity, mean +/- s.d. over 15 seeds with paired tests and effect sizes, four ablation axes, explicit compute and reproducibility statements. The paper's ceiling is set by scope -- one easy, dated, strongly bimodal benchmark, and a single-fit rather than cross-fitted calibration arm -- and by a theoretical proposition whose density assumption is inconsistent with the atomic scores its own tie-safe evaluator exists to handle. None of the required changes needs new experiments; they are matters of qualification, statement and presentation.
Strengths
- The comparison is genuinely matched, which is what the literature has been missing: every decision rule is fitted on the same decision split, from the same selected model, with an identical 10-draw hyperparameter budget per family and a cost-agnostic (log-loss) selection criterion, so the contrast isolates the decision recipe rather than confounding it with model quality or tuning effort.
- The null result is converted into a positive equivalence claim rather than being left as a failure to reject. TOST against a margin of 5% of the reference cost gives p = 2.8e-9 at the pre-registered endpoint, and equivalence also holds in every cost-ratio cell (p <= 1.9e-10 at R = 1, 2, 5, 20), every decision-size cell (2.2e-4 to 2.8e-9) and every per-family cell. This is the right statistical treatment and it is executed correctly.
- The pre-registration is respected. The primary endpoint (EMP-RAW minus ISO-t*, R = 10, m = 920, pooled) is reported first, both clauses of the hypothesis are declared refuted, and the manuscript does not retreat to a post-hoc winner: it explicitly says the one cell favouring empirical thresholding (m = 250, p = 0.046) runs opposite to the hypothesis and should be read as noise.
- Data provenance is exact and matches the experiment record in every particular: UCI Machine Learning Repository dataset 94, Hewlett-Packard Labs, CC BY 4.0, retrieved 2026-09-08 with the SHA-256 prefix quoted, 4,601 e-mails, 57 predictors with correct units (word_freq_* and char_freq_* on a 0-100 scale, three capital-run statistics), 1,813 positives (39.4%), both filters stated, and the honest note that no per-message timestamp exists so the sample period is only known at corpus level. Nothing is described as simulated.
- Every quantity I checked reproduces the executed run exactly, including the ones easiest to get wrong: Table 1 costs and oracle row, Table 2 CIs / Wilcoxon p / d_z / TOST, Table 3 decision-size contrasts, Table 4 per-family costs, oracle means and ECE / log-loss diagnostics, and Table 5 empirical cut-offs and search times (0.04 / 2.57 / 1.41 s).
- The reframing in Section 5.4 is the paper's real contribution and it is well supported: moving the cut-off at all is worth +0.10016 at R = 10 and +0.29702 at R = 20 against the naive 0.5 rule, an order of magnitude more than any difference between the two recipes, and both data-driven recipes are significantly worse than spending no data at m = 100 (+0.03178, p = 0.027; +0.03356, p = 0.025) with the crossover near m = 250.
- Reproducibility is exemplary for an empirical paper: 15 named seeds with the derived seed offsets (1000+k, 5000+k), full hyperparameter search ranges per family, BLAS/OpenMP threads pinned to one, dataset hash and retrieval date, and a 61.1 s single-threaded wall-clock figure.
- The limitations section is specific rather than ritual. It names the uncontrolled multiplicity and quantifies both nominally significant contrasts, states that the equivalence margin bounds only differences above 5%, admits the seeds are overlapping resamples of one corpus rather than independent replicates, and identifies cross-fitted calibration as the most important check that was not run.
Weaknesses
- Single benchmark. The headline equivalence rests on one easy, strongly bimodal, late-1990s corpus (test AUROC 0.968-0.986). Proposition 2 says the informative sample for threshold search is O(m f_S(t*)), so the result is precisely the one a benchmark with few points near t* would produce; the paper acknowledges this but the abstract still asserts flatly that the two recipes 'are indistinguishable'. External validity, not internal validity, is what caps significance here.
- Proposition 2 is technically loose. It assumes the scores have a density f_S on (0,1), yet the base learners include a random forest whose scores are atomic (multiples of 1/n_estimators) and an isotonic map whose output is a step function with heavy ties -- which is exactly why the method needs the tie-safe cumulative-count evaluator described in Section 4. The proposition also proves only a binomial window count, not an excess-risk rate, so it is closer to a remark than a result and cannot by itself support the directional prediction it is credited with motivating.
- The pre-registered endpoint pools three families that Table 4 shows behave very differently under calibration (isotonic changes the random forest by -0.04966 and gradient boosting by +0.01035). Section 5.4 does report that equivalence also holds in all three per-family cells, which justifies the pooling, but Section 5.2 states the pooled conclusion before that evidence is given.
- The calibration arm is the single-fit version. Practitioners generally use a cross-fitted calibrator, which is plausibly the arm most able to exploit a 100-point decision split. The paper names this as the most important unexecuted check, but it does mean the contest is between empirical thresholding and a weaker-than-standard implementation of the calibration recipe.
- The equivalence margin is defined as 5% of the reference rule's own mean cost, so it moves from cell to cell and is never argued to be the practitioner-relevant threshold. The paper is careful to state what TOST does and does not rule out, but the choice of 5% is asserted rather than justified.
- One protocol detail is omitted: the decision-split permutation is redrawn until the smallest prefix (m = 100) contains both classes. This is a real conditioning step on the cell where the paper's most striking finding sits (both recipes worse than spending no data), and a reader cannot infer it from Section 5.1.
- No figures. A cost-versus-m curve with error bars, or reliability diagrams for the three families, would communicate the crossover and the base-learner interaction far better than Tables 3 and 4 alone; a NeurIPS reader will expect at least one.
- The manuscript runs to roughly 3,800 words against the venue's stated 2,600-3,400 bound, and Table 3's oracle-regret column was dropped, so several regret figures quoted in Section 5.4 (0.0662, 0.0166, 0.0232, 0.0336) are only derivable by subtracting the oracle row rather than being tabulated directly.
Required changes
- Qualify the abstract's flat claim that the two recipes 'are indistinguishable': state that this holds on Spambase, at the tested cost ratios and held-out sizes, and within a 5%-of-operating-cost margin. The same qualification should appear in the first sentence of Section 5.4 and in the conclusion.
- Fix Proposition 2. Either state the density assumption as an idealisation and add a sentence on the atomic/tied case that the tie-safe evaluator in Section 4 actually handles, or restate it as a Remark. Also make explicit that it yields a window count, not an excess-risk bound, and therefore motivates but does not imply the pre-registered direction.
- Move the per-family equivalence evidence (TOST p = 3.1e-17, 3.2e-3, 1.3e-19 for logistic regression, random forest and gradient boosting) into Section 5.2, next to the pooled primary endpoint, so the pooling across heterogeneous families is justified where the pooled claim is first made.
- Add the omitted protocol step to Section 5.1: the seeded permutation of the decision split is redrawn until the smallest prefix (m = 100) contains both classes. Say how this conditions the m = 100 results.
- Restore the oracle-regret figures to a table, or add a regret row to Table 4, so that the per-family regret values quoted in Section 5.4 (0.0662 -> 0.0166 for the random forest; 0.0232 -> 0.0336 for gradient boosting) appear directly rather than only as a difference the reader must compute.
- Justify or relabel the 5% equivalence margin. State why 5% of the operating cost is the smallest difference a practitioner would act on, or present it explicitly as a stipulated margin and report the smallest margin the data would support at 15 seeds.
- Bring the manuscript within the venue's length bound, and state in Section 5.1 or the appendix that no figures are included because the run produced tabular summaries only.
Record
§Event timeline
- Run createdQuestion: “Under asymmetric misclassification costs, does post-hoc probability calibration with the theoretical cost-optimal threshold beat directly tuning the decision threshold on the same held-out validation data?” · output language: English · venue: Computer Science — Machine Learning · NeurIPS · models: agent:claude-opus-5
- Stage started · Literature reviewAttempt 1 · agent:claude-opus-5
- awaiting_agent · Literature reviewsearch response for literature_review: literature_review-search-1.response.json
- Resume requested
- Stage started · Literature reviewAttempt 2 · agent:claude-opus-5
- Search complete · Literature review14 sources · 13,326 characters of notes
- awaiting_agent · Literature reviewcomplete_json response for literature_review: literature_review-complete_json-1.response.json
- Resume requested
- Stage started · Literature reviewAttempt 3 · agent:claude-opus-5
- Search complete · Literature review14 sources · 13,326 characters of notes
- Stage completed · Literature review
- Stage started · IdeationAttempt 1 · agent:claude-opus-5
- awaiting_agent · Ideationcomplete_json response for ideation: ideation-complete_json-1.response.json
- Resume requested
- Stage started · IdeationAttempt 2 · agent:claude-opus-5
- Awaiting approval · Ideation
- Approved · IdeationIdea 1 selected
- Stage started · ExperimentationAttempt 1 · agent:claude-opus-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
- Resume requested
- Stage started · ExperimentationAttempt 2 · agent:claude-opus-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
- Resume requested
- Stage started · ExperimentationAttempt 3 · agent:claude-opus-5
- Dataset provisioned · Experimentationuci_spambase · 4,601 observations · 58 variables · sha256 813ac1df8eff
- Experiment round · ExperimentationRound 1: exit 0; result parsed (62.127s)
- Stage completed · Experimentation
- Stage started · AnalysisAttempt 1 · agent:claude-opus-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 2 · agent:claude-opus-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 1 · agent:claude-opus-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 2 · agent:claude-opus-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-opus-5
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-opus-5
- Awaiting approval · Review
- Approved · Review
- Run completereport=/private/tmp/claude-501/-Users-haodingresearch-Desktop-auto-research/579004ae-5819-417d-918c-9d9409997567/scratchpad/studies/computer_science-en/runs/193af5f07258/REPORT.md