Does prespecified covariate adjustment in the Cox model really increase power over the unadjusted log-rank test in a randomised survival trial? A real-data resampling study of the NCCTG adjuvant colon cancer trial
Paper
Does prespecified Cox adjustment really buy power over the log-rank test? A real-data resampling study of the NCCTG adjuvant colon cancer trial
Abstract
Prespecified covariate adjustment is recommended to raise power in randomised survival trials, mainly on evidence from parametric simulations. We resampled the 619 observation and levamisole-plus-fluorouracil patients of the NCCTG adjuvant colon cancer trial into 1:1 trials of 150, 300 and 600 patients, removed the trial's chance covariate imbalance by entropy balancing, and compared the log-rank test with the stratified log-rank test, Cox models adjusted for three candidate stratification factors (Cox-S) or eight prognostic factors, and the covariate-adjusted log-rank test (CL) with a finite-sample correction. In null trials resampled from the observation arm alone, the model-based Cox-S Wald test rejected 5.5%-5.9% at every size (Monte Carlo SE about 0.3), against 4.7%-5.2% for the log-rank test; its null variance of z, 1.035-1.065 (Monte Carlo SE 0.02), was close to the control arm's mean squared martingale residual per event (1.024-1.040). A null world pooling both arms overstated the excess at n = 150 (6.3% and 6.6%). Cox-S estimated a conditional log hazard ratio 7%-9% further from zero than the marginal one, but with a 5%-7% larger sampling standard deviation, so its null-calibrated relative efficiency was only 1.004-1.082. Adjusting for the same three factors, the corrected CL test held its size (4.8%-5.3%) and had calibrated relative efficiency 1.095-1.127, 8.7%-11.3% fewer events for equal power, whereas the 12-stratum stratified log-rank test lost efficiency (0.816-0.961). In trials like this one, the corrected CL test is a better route to power than Cox adjustment or stratification.
§Ask about this paper
Grounded in this runQuestions are answered from this run's record — the paper, the executed analysis, the data provenance and the referee report. Nothing else is consulted.
This run is finished, so the recorded paper cannot be rewritten. Questions are still answered from its record.
Pipeline
1.Literature review
Literature review complete (live:agent:claude-opus-5-5): 54 key papers, 55 grounded sources, 6 open gaps.
Summary
Covariate adjustment in randomised survival trials has two literatures that answer different questions. The Cox-model literature (Gail et al. 1984; Struthers and Kalbfleisch 1986; Chastang et al. 1988; Ford et al. 1995; Martinussen and Vansteelandt 2013; Aalen et al. 2015; Daniel et al. 2021) establishes that the hazard ratio is non-collapsible, so adjusting for prognostic covariates changes the estimand to a conditional hazard ratio further from 1; power can rise even though precision does not (cf. Robinson and Jewell 1991). Tests based on Cox models remain valid for the null of no effect under misspecification (Lagakos and Schoenfeld 1984; DiRienzo and Lagakos 2001), but model-based variances can then be inconsistent and Wald tests supranominal (Lin and Wei 1989), and Cox Z statistics degrade at low events per variable (Peduzzi et al. 1995; Vittinghoff and McCulloch 2007). A second literature adjusts the log-rank test itself while keeping the marginal estimand (Tangen and Koch 1999; Kong and Slud 1997; Lu and Tsiatis 2008; Tsiatis et al. 2008; Moore and van der Laan 2009), culminating in the covariate-adjusted log-rank test of Ye, Shao and Yi (2024), now reviewed and applied in oncology (Bannick et al. 2026; Backenroth et al. 2026) and shown to inflate type I error in small samples with many covariates, for which Krotka and Magirr (2026) propose a finite-sample variance correction. The quantitative case that adjustment pays in survival trials rests on parametric simulations with one binary covariate (Hernandez et al. 2006: 4%-44% sample-size savings) or on outcomes simulated from models fitted to real studies (Kahan et al. 2014: median power 80% to 92.6% across 12 outcomes, two of them time-to-event). Simulation studies are reported under ADEMP with Monte Carlo standard errors (Morris et al. 2019). What is missing is a model-free check on real event and censoring times: how much prespecified Cox adjustment and the covariate-adjusted log-rank test gain over the log-rank test in a real trial with heavy censoring and moderately prognostic binary covariates, where that gain comes from (estimand shift versus variance), which test versions hold their size at realistic events per variable, and whether simple planning-stage predictors anticipate the realised gain. The NCCTG adjuvant colon cancer trial (Moertel et al. 1990, 1995; 929 stage III patients; data in the R survival package, Therneau and Grambsch 2000) offers real overall-survival and recurrence outcomes with a real treatment effect and the trial's own stratification factors.
Key papers (54)
- Randomized controlled trials with time-to-event outcomes: how much does prespecified covariate adjustment increase power?. Hernandez AV, Eijkemans MJ, Steyerberg EW. Ann Epidemiol 2006;16(1):41-48, 2006.Closest antecedent: Cox models on simulated trials with one dichotomous covariate; prespecified adjustment cut required sample size by 15-44% (covariate HR 2-5, prevalence 50%) or 4-12% (prevalence 10%) with nominal type I error.
- Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. Hernandez AV, Steyerberg EW, Habbema JD. J Clin Epidemiol 2004;57(5):454-460, 2004.Binary-outcome counterpart: sample-size reductions of 3-46% with one dichotomous covariate; type I error near nominal.
- The risks and rewards of covariate adjustment in randomized trials: an assessment of 12 outcomes from 8 studies. Kahan BC, Jairath V, Dore CJ, Morris TP. Trials 2014;15:139, 2014.12 outcomes (2 time-to-event) from 8 studies, 5,000 datasets each simulated from fitted parameters; adjustment raised median power from 80% to 92.6%; noise covariates cost at most 1.5 points.
- Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury. Turner EL, Perel P, Clayton T, Edwards P, Hernandez AV, Roberts I. J Clin Epidemiol 2012;65(5):474-481, 2012.Empirical power gains from adjustment in real trial data (binary outcome).
- Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates. Gail MH, Wieand S, Piantadosi S. Biometrika 1984;71(3):431-444, 1984.Omitting a balanced covariate biases nonlinear-model treatment estimates, including Cox, even under randomisation (non-collapsibility).
- Properties of proportional-hazards score tests under misspecified regression models. Lagakos SW, Schoenfeld DA. Biometrics 1984;40(4):1037-1048, 1984.Omitting a balanced covariate leaves the size of the PH score test intact but can cut power; misspecified covariate form costs little efficiency.
- Misspecified proportional hazard models. Struthers CA, Kalbfleisch JD. Biometrika 1986;73(2):363-369, 1986.Partial-likelihood estimator converges to an implicitly defined parameter under misspecification; effects of omitted covariates.
- A quantitative study of the bias in estimating the treatment effect caused by omitting a balanced covariate in survival models. Chastang C, Byar D, Piantadosi S. Stat Med 1988;7(12):1243-1255, 1988.Approximate attenuation and power-loss formulae for omitting balanced prognostic covariates; effect modest unless strongly prognostic or many.
- Model inconsistency, illustrated by the Cox proportional hazards model. Ford I, Norrie J, Ahmadi S. Stat Med 1995;14(8):735-746, 1995.Adjusted and unadjusted Cox models cannot both hold; their parameters differ in interpretation.
- The robust inference for the Cox proportional hazards model. Lin DY, Wei LJ. J Am Stat Assoc 1989;84(408):1074-1078, 1989.Sandwich (robust) variance for Cox under misspecification; robust score tests good in small samples; model-based tests can be supranominal.
- Effects of model misspecification on tests of no randomized treatment effect arising from Cox's proportional hazards model. DiRienzo AG, Lagakos SW. J R Stat Soc Series B Stat Methodol 2001;63(4):745-757, 2001.Score and Wald numerators have null mean zero under misspecification (given censoring conditions); model-based variances inconsistent but sizes can be near nominal; log-rank asymptotically valid.
- Robust covariate-adjusted logrank tests. Kong FH, Slud E. Biometrika 1997;84(4):847-862, 1997.Robust covariate-adjusted logrank tests built from working PH models, valid without the model being correct.
- Nonparametric analysis of covariance for hypothesis testing with logrank and Wilcoxon scores and survival-rate estimation in a randomized clinical trial. Tangen CM, Koch GG. J Biopharm Stat 1999;9(2):307-338, 1999.Nonparametric ANCOVA on logrank scores: covariate-adjusted logrank-type testing without extra model assumptions.
- Improving the efficiency of the log-rank test using auxiliary covariates. Lu X, Tsiatis AA. Biometrika 2008;95(3):679-694, 2008.Semiparametric augmentation of the log-rank test with covariates; more powerful Wald tests.
- Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: a principled yet flexible approach. Tsiatis AA, Davidian M, Zhang M, Lu X. Stat Med 2008;27(23):4658-4677, 2008.General augmentation framework for covariate adjustment in two-arm trials.
- Covariate-adjusted log-rank test: guaranteed efficiency gain and universal applicability. Ye T, Shao J, Yi Y. Biometrika 2024;111(2):691-705, 2024.The CL test: arm-specific regression of derived log-rank outcomes on covariates; explicit variance; guaranteed asymptotic efficiency gain; valid under simple and covariate-adaptive randomisation.
- Finite-sample correction for the covariate-adjusted log-rank test. Krotka P, Magirr D. arXiv:2608.10923 [stat.ME], 2026, 2026.Reports small-sample type I error inflation of the CL test with many covariates and proposes the variance multiplier Gamma = C_df x E(VIF).
- Practical considerations when using the covariate-adjusted log-rank test for the analysis of time-to-event endpoints in oncology trials. Backenroth D, Roels S, Dibaj S, Ye T, Ohrn F, Van Lancker K. Biometrics 2026;82(2):ujag109, 2026.Estimating attainable variance reduction for the CL test from historical data, prognostic scores, type I error in group sequential designs.
- Covariate adjustment in randomized clinical trials: from general theory to practical insights. Bannick MS, Yi Y, Ye T. Clin Trials 2026;23(3):272-281, 2026.Review: covariate-adjusted log-rank and Cox methods that keep the marginal estimand; RobinCar software.
- Log-rank and stratified log-rank tests. Ye T, Shao J, Yi Y. Stat Theory Relat Fields 2023;7(4):309-317, 2023.Stratified log-rank asymptotically more powerful than unstratified under simple randomisation with Cox alternatives; log-rank conservative under covariate-adaptive randomisation.
- The type I error and power of non-parametric logrank and Wilcoxon tests with adjustment for covariates - a simulation study. Jiang H, Symanowski J, Paul S, Qu Y, Zagar A, Hong S. Stat Med 2008;27(28):5850-5860, 2008.Simulation of type I error and power of covariate-adjusted nonparametric logrank tests versus Cox models.
- Power of logrank test and Cox regression model in clinical trials with heterogeneous samples. Akazawa K, Nakamura T, Palesch Y. Stat Med 1997;16(5):583-597, 1997.Power loss of simple and stratified logrank tests under patient heterogeneity, from a >6,000-patient cancer trial.
- The impact of covariate imbalance on the size of the logrank test in randomized clinical trials. Kinukawa N, Nakamura T, Akazawa K, Nose Y. Stat Med 2000;19(15):1955-1967, 2000.Baseline imbalance from patient heterogeneity and its effect on logrank size.
- Making apples from oranges: comparing noncollapsible effect estimators and their standard errors after adjustment for different covariate sets. Daniel R, Zhang J, Farewell D. Biom J 2021;63(3):528-557, 2021.Conditional versus marginal hazard ratios and their standard errors; marginalisation of conditional HRs.
- On collapsibility and confounding bias in Cox and Aalen regression models. Martinussen T, Vansteelandt S. Lifetime Data Anal 2013;19(3):279-296, 2013.Non-collapsibility of the Cox hazard ratio.
- Does Cox analysis of a randomized survival study yield a causal treatment effect?. Aalen OO, Cook RJ, Roysland K. Lifetime Data Anal 2015;21(4):579-593, 2015.Selection of risk sets breaks balance over follow-up; Cox tests of no effect valid with robust variance.
- Covariate adjustment in randomized controlled trials: general concepts and practical considerations. Van Lancker K, Bretz F, Dukes O. Clin Trials 2024;21(4):399-411, 2024.Conditional versus marginal estimands; alignment of method and estimand.
- Improving precision and power in randomized trials for COVID-19 treatments using covariate adjustment, for binary, ordinal, and time-to-event outcomes. Benkeser D, Diaz I, Luedtke A, Segal J, Scharfstein D, Rosenblum M. Biometrics 2021;77(4):1467-1481, 2021.Precision gains of covariate adjustment including time-to-event outcomes, data-derived simulation.
- Using simulation studies to evaluate statistical methods. Morris TP, White IR, Crowther MJ. Stat Med 2019;38(11):2074-2102, 2019.ADEMP structure, performance measures and Monte Carlo standard errors.
- Planning a method for covariate adjustment in individually randomised trials: a practical guide. Morris TP, Walker AS, Williamson EJ, White IR. Trials 2022;23:328, 2022.Choosing adjustment methods in a SAP; robust SEs anti-conservative at small n.
- A comparison of covariate adjustment approaches under model misspecification in individually randomized trials. Tackney MS, Morris T, White I, Leyrat C, Diaz-Ordaz K, Williamson E. Trials 2023;24:14, 2023.Small-sample robustness of adjustment methods; SE underestimation with many covariates.
- Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. Peduzzi P, Concato J, Feinstein AR, Holford TR. J Clin Epidemiol 1995;48(12):1503-1510, 1995.Cox Z statistics lose null validity at low events per variable; EPV 10 suggested.
- Relaxing the rule of ten events per variable in logistic and Cox regression. Vittinghoff E, McCulloch CE. Am J Epidemiol 2007;165(6):710-718, 2007.Coverage and type I error acceptable in many settings below 10 EPV.
- Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Pocock SJ, Assmann SE, Enos LE, Kasten LE. Stat Med 2002;21(19):2917-2930, 2002.Practice survey: inconsistent covariate adjustment in trial reports.
- Improper analysis of trials randomised using stratified blocks or minimisation. Kahan BC, Morris TP. Stat Med 2012;31(4):328-340, 2012.Analysis should adjust for factors balanced in the randomisation.
- Covariate imbalance and random allocation in clinical trials. Senn S. Stat Med 1989;8(4):467-475, 1989.Chance covariate imbalance under randomisation and the case for adjustment.
- Stratification in small randomised clinical trials and analysis of covariance: some simple theory and recommendations. Senn S, Konig F, Posch M. Stat Methods Med Res 2026;35(9):1838-1856, 2026.Variance inflation factor and degrees-of-freedom costs of covariates in small trials.
- Adjusting for partially missing baseline measurements in randomized trials. White IR, Thompson SG. Stat Med 2005;24(7):993-1007, 2005.Deterministic mean imputation or missing indicators valid for missing baseline covariates in trials.
- Sample-size formula for the proportional-hazards regression model. Schoenfeld D. Biometrics 1983;39(2):499-503, 1983.Required events proportional to 1/(log HR)^2; converts efficiency into event savings.
- Regression models and life-tables. Cox DR. J R Stat Soc Series B Stat Methodol 1972;34(2):187-202, 1972.The proportional hazards model and partial likelihood.
- Covariance analysis of censored survival data. Breslow N. Biometrics 1974;30(1):89-99, 1974.Breslow handling of tied times and baseline hazard estimator.
- Asymptotically efficient rank invariant test procedures. Peto R, Peto J. J R Stat Soc Series A 1972;135(2):185-207, 1972.The logrank test and its permutational variance.
- A solution to the problem of monotone likelihood in Cox regression. Heinze G, Schemper M. Biometrics 2001;57(1):114-119, 2001.Monotone likelihood with rare binary covariates in Cox regression.
- Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Hainmueller J. Polit Anal 2012;20(1):25-46, 2012.Exponential-tilting weights giving exact covariate-mean balance; usable to remove chance imbalance from a resampling population.
- Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma. Moertel CG, Fleming TR, Macdonald JS, Haller DG, Laurie JA, Goodman PJ, et al. N Engl J Med 1990;322(6):352-358, 1990.Primary report of the trial analysed: in stage C, Lev+5FU reduced recurrence risk 41% and death rate 33%.
- Fluorouracil plus levamisole as effective adjuvant therapy after resection of stage III colon carcinoma: a final report. Moertel CG, Fleming TR, Macdonald JS, Haller DG, Laurie JA, Tangen CM, et al. Ann Intern Med 1995;122(5):321-326, 1995.Final report on the 929 eligible stage III patients (median follow-up 6.5 years): recurrence -40%, death -33%.
- Surgical adjuvant therapy of large-bowel carcinoma: an evaluation of levamisole and the combination of levamisole and fluorouracil. Laurie JA, Moertel CG, Fleming TR, Wieand HS, Leigh JE, Rubin J, et al. J Clin Oncol 1989;7(10):1447-1456, 1989.Earlier NCCTG adjuvant trial of the same regimens.
- Modeling Survival Data: Extending the Cox Model. Therneau TM, Grambsch PM. New York: Springer; 2000, 2000.Source of the R survival package data and Cox model methods.
- Disease-free survival versus overall survival as a primary end point for adjuvant colon cancer studies: individual patient data from 20,898 patients on 18 randomized trials. Sargent DJ, Wieand HS, Haller DG, Gray R, Benedetti JK, Buyse M, et al. J Clin Oncol 2005;23(34):8664-8670, 2005.Justifies disease-free (recurrence-free) survival as an endpoint alongside overall survival.
- Using regression models to analyze randomized trials: asymptotically valid hypothesis tests despite incorrectly specified models. Rosenblum M, van der Laan MJ. Biometrics 2009;65(3):937-945, 2009.Regression-based tests that stay valid in trials under misspecification.
- Increasing power in randomized trials with right censored outcomes through covariate adjustment. Moore KL, van der Laan MJ. J Biopharm Stat 2009;19(6):1099-1131, 2009.Targeted maximum likelihood adjustment for censored outcomes.
- Toward better practice of covariate adjustment in analyzing randomized clinical trials. Ye T, Shao J, Yi Y, Zhao Q. J Am Stat Assoc 2023;118(544):2370-2382, 2023.ANHECOVA and robust variance for covariate adjustment.
- Efficiency study of estimators for a treatment effect in a pretest-posttest trial. Yang L, Tsiatis AA. Am Stat 2001;55(4):314-321, 2001.ANCOVA with and without interactions; equivalence under equal allocation.
- Some surprising results about covariate adjustment in logistic regression models. Robinson LD, Jewell NP. Int Stat Rev 1991;59(2):227-240, 1991.Adjustment lowers precision yet raises power in non-collapsible models.
Methods in use
- Unadjusted log-rank test (Peto and Peto 1972) with hypergeometric or binomial variance; equivalent to the Cox score test without covariates.
- Cox proportional hazards regression (Cox 1972) with Breslow ties (Breslow 1974); Wald, score and partial likelihood-ratio tests of the treatment coefficient; model-based or Lin-Wei sandwich variance (Lin and Wei 1989).
- Covariate-adjusted log-rank tests targeting the marginal estimand: nonparametric ANCOVA on logrank scores (Tangen and Koch 1999), semiparametric augmentation (Lu and Tsiatis 2008), the CL test with arm-specific regressions of derived outcomes (Ye, Shao and Yi 2024), with the Krotka-Magirr finite-sample variance correction (2026).
- Monte Carlo evaluation under the ADEMP framework (Morris et al. 2019): type I error and power at two-sided alpha 0.05, bias, empirical and model standard errors, coverage of 95% intervals, all with Monte Carlo standard errors; 1,000-10,000 repetitions.
- Data-generating mechanisms: fully parametric (exponential or Weibull times, one binary covariate; Hernandez et al. 2006) or parametric models fitted to real data (Kahan et al. 2014; Peduzzi et al. 1995; Benkeser et al. 2021).
- Efficiency summaries: power at fixed n, percentage sample-size reduction for equal power, relative efficiency as a ratio of squared mean z statistics, events required via Schoenfeld (1983).
- Handling partially missing baseline covariates in trials by deterministic mean imputation or missing indicators (White and Thompson 2005); adjusting for randomisation stratification factors (Kahan and Morris 2012).
Open gaps
- No model-free real-data evidence: existing quantification of the power gain from prespecified Cox adjustment in survival trials uses parametric simulation with a single binary covariate (Hernandez et al. 2006) or outcomes generated from fitted models (Kahan et al. 2014), not the real event and censoring times of a trial.
- The source of the gain is rarely decomposed: how much of the adjusted Cox z-statistic gain is estimand shift (conditional versus marginal log hazard ratio) and how much is standard-error change, compared with the variance-only gain of the covariate-adjusted log-rank test.
- Conflicting evidence on small-sample size: model-based Cox tests supranominal under misspecification (Lin and Wei 1989) versus near-nominal (DiRienzo and Lagakos 2001); Cox Z statistics invalid at low EPV (Peduzzi et al. 1995); CL inflation with many covariates (Krotka and Magirr 2026) - unresolved for Wald versus score versus likelihood-ratio versions on real trial data.
- The Krotka-Magirr correction was derived and checked with Gaussian covariates; its performance with real, binary-dominated prognostic covariates is unknown.
- Planning-stage predictors of the gain (the R^2 of derived log-rank outcomes on covariates; the ratio of conditional to marginal log hazard ratios) have not been validated against realised gains in real data.
- A single trial's reanalysis confounds expected gain with that trial's chance covariate imbalance (Senn 1989; Kinukawa et al. 2000); resampling designs need to neutralise it.
Grounded search sources (55) · mode: live:agent:claude-opus-5-5
- Randomized controlled trials with time-to-event outcomes: how much does prespecified covariate adjustment increase power?
- Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements
- The risks and rewards of covariate adjustment in randomized trials: an assessment of 12 outcomes from 8 studies
- Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury
- Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates
- Properties of proportional-hazards score tests under misspecified regression models
- Misspecified proportional hazard models
- A quantitative study of the bias in estimating the treatment effect caused by omitting a balanced covariate in survival models
- Model inconsistency, illustrated by the Cox proportional hazards model
- The robust inference for the Cox proportional hazards model
- Effects of model misspecification on tests of no randomized treatment effect arising from Cox's proportional hazards model
- Robust covariate-adjusted logrank tests
- Nonparametric analysis of covariance for hypothesis testing with logrank and Wilcoxon scores and survival-rate estimation in a randomized clinical trial
- Improving the efficiency of the log-rank test using auxiliary covariates
- Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: a principled yet flexible approach
- Covariate-adjusted log-rank test: guaranteed efficiency gain and universal applicability
- Finite-sample correction for the covariate-adjusted log-rank test
- Practical considerations when using the covariate-adjusted log-rank test for the analysis of time-to-event endpoints in oncology trials
- Covariate adjustment in randomized clinical trials: from general theory to practical insights
- Log-rank and stratified log-rank tests
- The type I error and power of non-parametric logrank and Wilcoxon tests with adjustment for covariates - a simulation study
- Power of logrank test and Cox regression model in clinical trials with heterogeneous samples
- The impact of covariate imbalance on the size of the logrank test in randomized clinical trials
- Making apples from oranges: comparing noncollapsible effect estimators and their standard errors after adjustment for different covariate sets
- On collapsibility and confounding bias in Cox and Aalen regression models
- Does Cox analysis of a randomized survival study yield a causal treatment effect?
- Covariate adjustment in randomized controlled trials: general concepts and practical considerations
- Improving precision and power in randomized trials for COVID-19 treatments using covariate adjustment, for binary, ordinal, and time-to-event outcomes
- Using simulation studies to evaluate statistical methods
- Planning a method for covariate adjustment in individually randomised trials: a practical guide
- A comparison of covariate adjustment approaches under model misspecification in individually randomized trials
- Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates
- Relaxing the rule of ten events per variable in logistic and Cox regression
- Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems
- Improper analysis of trials randomised using stratified blocks or minimisation
- Covariate imbalance and random allocation in clinical trials
- Stratification in small randomised clinical trials and analysis of covariance: some simple theory and recommendations
- Adjusting for partially missing baseline measurements in randomized trials
- Sample-size formula for the proportional-hazards regression model
- Regression models and life-tables
- Covariance analysis of censored survival data
- Asymptotically efficient rank invariant test procedures
- A solution to the problem of monotone likelihood in Cox regression
- Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies
- Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma
- Fluorouracil plus levamisole as effective adjuvant therapy after resection of stage III colon carcinoma: a final report
- Surgical adjuvant therapy of large-bowel carcinoma: an evaluation of levamisole and the combination of levamisole and fluorouracil
- Modeling Survival Data: Extending the Cox Model
- Disease-free survival versus overall survival as a primary end point for adjuvant colon cancer studies: individual patient data from 20,898 patients on 18 randomized trials
- Using regression models to analyze randomized trials: asymptotically valid hypothesis tests despite incorrectly specified models
- Increasing power in randomized trials with right censored outcomes through covariate adjustment
- Toward better practice of covariate adjustment in analyzing randomized clinical trials
- Efficiency study of estimators for a treatment effect in a pretest-posttest trial
- Some surprising results about covariate adjustment in logistic regression models
- Colon dataset (R package survival), Rdatasets archive CSV
2.Ideation
3 candidate ideas generated; selected: “Does prespecified Cox adjustment really buy power over the log-rank test? An imbalance-neutralised resampling study of the NCCTG adjuvant colon cancer trial”.
- Idea 1Selected
Does prespecified Cox adjustment really buy power over the log-rank test? An imbalance-neutralised resampling study of the NCCTG adjuvant colon cancer trial
Hypothesis. Primary (H1): in 1:1 trials of n = 150, 300 and 600 patients resampled from the observation and levamisole-plus-fluorouracil arms of the NCCTG trial, with the trial's own chance covariate imbalance removed, prespecified Cox adjustment for the three randomisation stratification factors raises the efficiency of the test of no treatment effect over the unadjusted log-rank test by at least 10% - relative efficiency RE = (mean z_adjusted / mean z_log-rank)^2 >= 1.10 - in all six endpoint-by-size cells (overall and recurrence-free survival), while its model-based Wald test keeps the type I error at or below 6.0%. Secondary (H2): the gain comes from the conditional log hazard ratio lying further from zero than the marginal one (mean-estimate ratio > 1), not from a smaller standard error (mean-SE ratio >= 1). Secondary (H3): any type I error inflation of the adjusted Wald test is confined to small trials with many covariates, is explained by the model-based standard error underestimating the empirical standard error of the treatment coefficient, and is absent from the score and likelihood-ratio versions of the same model.
Dataset. Adjuvant chemotherapy trial in stage B/C colon cancer colon_cancer_trial
Test plan, novelty and risk
Test plan. Data: colon_cancer_trial, one record per patient; analysis population = the 619 patients randomised to observation (Obs, 315) or levamisole plus fluorouracil (Lev+5FU, 304); the levamisole-alone arm is excluded. Endpoints: overall survival (death, etype 2 record) and recurrence-free survival (first of recurrence or death, from the etype 1 record and the death indicator); time in days. Covariate sets fixed in advance: S = the trial's stratification factors (node4: more than 4 positive nodes; extent of local spread as two indicators, submucosa/muscle and contiguous structures, reference serosa; surg: long interval from surgery); P = S plus obstruction, adherence, poor differentiation (differ = 3; the 13 missing values imputed deterministically as not poor, following White and Thompson), age per 10 years (centred) and sex. Perforation (2.7%) is excluded because it is absent from many small resamples. Resampling worlds, all from real patients with their real times and censoring: (i) null - both arms drawn with replacement from the pooled 619, so treatment is independent of everything (exact null); (ii) balanced alternative (primary) - each arm drawn with replacement from its own patients with entropy-balancing weights that equalise the means of the nine covariates to the pooled means, so arms differ only by the real treatment effect plus chance; (iii) trial-preserving alternative (contrast) - each arm drawn uniformly from its own patients, which carries the trial's chance imbalance into every replicate. Total sizes n = 150, 300, 600 with n/2 per arm; 3,000 replicates under the null (MCSE of a 5% rate 0.40 points), 2,000 under the balanced alternative and 1,000 under the trial-preserving alternative per endpoint and size; fixed seed. Methods: log-rank test (hypergeometric variance); Cox models with Breslow ties adjusted for S or P, tested by Wald (model-based SE), Wald (Lin-Wei robust SE), score and partial likelihood-ratio tests; the covariate-adjusted log-rank test of Ye, Shao and Yi (CL) for S and P, with and without the Krotka-Magirr finite-sample variance correction; unadjusted Cox for the marginal log hazard ratio. Estimands and performance (ADEMP): type I error and power at two-sided 0.05 with Monte Carlo SEs; for the marginal and the two conditional log hazard ratios, bias against the balanced-population value (weighted full-data Cox fit), empirical SE, mean model and robust SE and coverage of 95% intervals; RE with 95% Monte Carlo intervals (delta method on paired replicates); the decomposition of the z-ratio into mean-estimate ratio and mean-SE ratio; the large-sample RE predicted from the balanced population ((z_adjusted/z_unadjusted)^2 for Cox, V_L/V_CL for CL) compared with the realised RE; ratios of the empirical SD of the null treatment coefficient to its mean model and robust SEs. Refutation: H1 is refuted if RE for the S-adjusted Cox Wald test is below 1.10 in any of the six cells or its type I error exceeds 6.0% in any cell. H2 is refuted if the mean-SE ratio is below 1 or the estimate ratio is at most 1. H3 is refuted if score or likelihood-ratio tests also exceed 6.0% where the Wald test does, or if the empirical-to-model SE ratio is not above 1 where inflation occurs.
Novelty. Existing evidence for Cox adjustment in survival trials comes from parametric simulations with one binary covariate (Hernandez et al. 2006) or from outcomes simulated from models fitted to real studies (Kahan et al. 2014). Here no outcome is modelled: every replicate trial consists of real NCCTG patients with their real event and censoring times, and entropy-balancing weights remove the trial's own chance imbalance so the comparison measures adjustment rather than one trial's accident. The study decomposes the gain into estimand shift and precision, tests which versions of the adjusted test hold their size, checks the 2026 Krotka-Magirr correction on real binary-dominated covariates, and compares the realised gain with a large-sample prediction a planner could compute.
Expected outcome. Literature magnitudes suggest modest gains with moderately prognostic binary covariates and about 50% censoring: RE perhaps 1.05-1.20, larger for the extended set, with the Cox gain driven by estimand shift and the CL gain by variance reduction; type I error near 5% except possibly the Wald test with nine covariates at n = 150 (about 70 events). In the trial-preserving world, adjustment may appear to lose power because the control arm carries slightly worse prognostic factors.
Main risk. The resampling population is one trial: covariate prognostic strength, censoring and the effect size (hazard ratio near 0.7) are those of this trial, so gains may not transfer. Entropy balancing equalises covariate means but not every feature of the joint distribution. Bootstrap duplicates create tied times (handled by Breslow). Power differences near the ceiling at n = 600 for recurrence-free survival are uninformative; RE is the main summary.
- Idea 2
Strong continuous prognostic factors: how much power does prespecified Cox adjustment add in the Mayo PBC trial?
Hypothesis. In 1:1 trials of n = 100-300 resampled from the 312 randomised Mayo PBC patients, Cox adjustment for log bilirubin, albumin, age and oedema (strong, continuous prognostic factors) increases the efficiency of the test of a treatment effect over the log-rank test by at least 25% (RE >= 1.25) with type I error at or below 6%, and the adjusted Wald test's size is inflated at n = 100 while the score test's is not.
Dataset. Mayo Clinic primary biliary cholangitis trial mayo_pbc_trial
Test plan, novelty and risk
Test plan. Filter to the 312 randomised patients (trt not missing); endpoint death (status == 2; transplant censored). Covariates log(bili), albumin, age, edema; complete cases. Null world: permutation-style resampling of the pooled trial. Alternative world: the trial shows no D-penicillamine effect, so a treatment effect cannot be taken from the data; instead compare tests under the null only and report the precision gain (ratio of empirical SDs of z and the CL variance reduction). Refuted if the CL variance reduction is below 20% or adjusted tests exceed 6% type I error at n >= 200.
Novelty. Real-data counterpart with strong continuous covariates, bracketing the colon trial's moderate binary covariates.
Expected outcome. Large variance reductions (bilirubin is highly prognostic) and some Wald inflation at the smallest sizes.
Main risk. No real treatment effect exists in the PBC trial, so power cannot be estimated without inducing an effect, which would require modelling outcomes; the study would be limited to size and precision.
- Idea 3
Stratified log-rank test versus Cox adjustment for the randomisation stratification factors in the NCCTG colon trial
Hypothesis. In trials of n = 150-600 resampled from the NCCTG observation and Lev+5FU arms, the log-rank test stratified on node4 x extent x time-from-surgery (up to 12 strata) has lower power than Cox adjustment for the same factors when n = 150 because sparse strata waste events, and equal power at n = 600.
Dataset. Adjuvant chemotherapy trial in stage B/C colon cancer colon_cancer_trial
Test plan, novelty and risk
Test plan. Population and endpoints as in the primary idea; resample arms with replacement; compare power and type I error of the stratified log-rank test, the Cox model adjusted for the same factors and the unadjusted log-rank test with Monte Carlo SEs; refuted if the stratified test's power is within 2 points of the adjusted Cox test at n = 150.
Novelty. Tests the sparse-strata concern of Akazawa et al. (1997) against the asymptotic superiority shown by Ye, Shao and Yi (2023) with real outcomes.
Expected outcome. Small losses for the stratified test at n = 150, vanishing by n = 600.
Main risk. With only three binary or ternary factors, strata are rarely empty, so the comparison may show no difference.
Ranking rationale
Idea 0 answers the posed question directly on the only available randomised trial with a real treatment effect, uses real outcomes without any outcome model, neutralises the trial's chance imbalance, and settles three falsifiable claims (size of the gain, its source, and which tests hold their size). Idea 1 cannot estimate power without inducing an effect, because the PBC trial shows none. Idea 2 is narrower and addresses a design question with likely small differences.
3.Experimentation
Experiment succeeded in 1 round (98.375s).
Data provenance
Adjuvant chemotherapy trial in stage B/C colon cancer
- Source
- R package 'survival', mirrored by the Rdatasets archive; original data from a North Central Cancer Treatment Group / Mayo Clinic trial
- Citation
- Moertel, C.G., Fleming, T.R., Macdonald, J.S. et al. (1990) Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma. New England Journal of Medicine 322(6), 352-358. DOI: 10.1056/NEJM199002083220602. Final report: Moertel, C.G. et al. (1995) Fluorouracil plus levamisole as effective adjuvant therapy after resection of stage III colon carcinoma. Annals of Internal Medicine 122(5), 321-326. DOI: 10.7326/0003-4819-122-5-199503010-00001. See also Laurie, J.A. et al. (1989) Journal of Clinical Oncology 7(10), 1447-1456. (The survival package's own help file misprints these two as volume 332 and as a 1991 stage-II report; the corrected forms are given here.) Data: colon, R package 'survival'.
- Licence
- Redistributed in the R package 'survival' under LGPL (>= 2) via the Rdatasets archive. The package's COPYRIGHTS file reads "Copyright 2000 Mayo Foundation for Medical Education and Research. This software is accepted by users 'as is' and without warranties or guarantees of any kind." No per-dataset licence is stated.
- File
- https://vincentarelbundock.github.io/Rdatasets/csv/survival/colon.csv
- Observations
- 1,858
- Variables (16)
- id, study, rx, sex, age, obstruct, perfor, adhere, nodes, status, differ, extent, surg, node4, time, etype
- Units
- The table is in long form: etype is 1 for the recurrence record and 2 for the death record, so 1,858 rows are 929 patients. time is days to the event or censoring and status is the 0/1 event indicator. rx is the randomised arm ('Obs', 'Lev', 'Lev+5FU'). nodes is the number of positive lymph nodes, differ is tumour differentiation (1 well to 3 poor), extent is depth of invasion (1 submucosa to 4 contiguous structures), and obstruct, perfor, adhere, surg and node4 are 0/1 clinical indicators.
- Retrieved
- 24 Oct 2026, 02:20
- Checksum
- sha256 6f3472a64f696e31…
Real, publicly available data — downloaded and checksum-verified before the analysis ran, never simulated.
Result (RESULT_JSON)
{
"design": {
"seed": 20261024,
"sizes": [
150,
300,
600
],
"reps": {
"null": 5000,
"balanced": 2000,
"trial": 1000,
"perm": 2000,
"null_obs": 5000,
"null_trt": 5000,
"perm_obs": 2000
},
"alpha": 0.05,
"tests": [
"LR",
"S_wald",
"S_rwald",
"S_score",
"S_lrt",
"S_CL",
"S_CLkm",
"P_wald",
"P_rwald",
"P_score",
"P_lrt",
"P_CL",
"P_CLkm",
"SLR"
],
"covariates": [
"node4",
"extent_1_2",
"extent_4",
"surg",
"obstruct",
"adhere",
"poor_diff",
"age10",
"male"
],
"sets": {
"S": [
"node4",
"extent_1_2",
"extent_4",
"surg"
],
"P": [
"node4",
"extent_1_2",
"extent_4",
"surg",
"obstruct",
"adhere",
"poor_diff",
"age10",
"male"
]
},
"ties": "Breslow",
"block": 250,
"strata_SLR": "node4 x extent (submucosa/muscle, serosa, contiguous) x surg",
"streams": "original worlds: default_rng(seed); null_obs, null_trt, perm_obs: default_rng([seed, 1/2/3])",
"calibration_suffix": {
"": "pooled null",
"_obs": "observation-arm null",
"_trt": "Lev+5FU-arm null"
}
},
"data": {
"N": 619,
"n_trt": 304,
"n_ctl": 315,
"differ_missing_imputed": 13,
"perfor_pct_excluded": 2.7,
"median_age": 61,
"OS": {
"events": 291,
"events_trt": 123,
"events_ctl": 168,
"censored_pct": 53,
"median_fu_censored_days": 2318.5
},
"RFS": {
"events": 324,
"events_trt": 134,
"events_ctl": 190,
"censored_pct": 47.7,
"median_fu_censored_days": 2309
},
"prevalence_pct": {
"node4": 26.8,
"extent_1_2": 14.2,
"extent_4": 5,
"surg": 27,
"obstruct": 18.9,
"adhere": 13.9,
"poor_diff": 17.1,
"male": 49.6
},
"strata": {
"n_strata": 12,
"sizes_sorted": [
1,
4,
6,
7,
12,
14,
19,
33,
56,
105,
111,
251
],
"occupied_ctl": 11,
"occupied_trt": 12
}
},
"balance": {
"smd_before": {
"node4": -0.037,
"extent_1_2": -0.023,
"extent_4": -0.126,
"surg": -0.088,
"obstruct": -0.057,
"adhere": -0.061,
"poor_diff": 0.033,
"age10": 0.02,
"male": -0.127
},
"max_abs_smd_after": 1.2e-13,
"weight_ratio_range": [
0.69,
1.683
],
"ess_trt": 299.3,
"ess_ctl": 311.1
},
"phi": {
"OS": {
"pooled": {
"none": 0.997,
"S": 1.054,
"P": 1.073,
"strat": 0.964
},
"controls_only": {
"none": 0.994,
"S": 1.04,
"P": 1.036,
"strat": 0.929
},
"implied_asymptotic_size_pct": {
"none": 5,
"S": 5.6,
"P": 5.9,
"strat": 4.6
},
"implied_asymptotic_size_pct_controls": {
"none": 4.9,
"S": 5.5,
"P": 5.4,
"strat": 4.2
},
"lev5fu_only": {
"none": 0.995,
"S": 1.042,
"P": 1.055,
"strat": 0.932
},
"implied_asymptotic_size_pct_lev5fu": {
"none": 4.9,
"S": 5.5,
"P": 5.6,
"strat": 4.2
}
},
"RFS": {
"pooled": {
"none": 0.997,
"S": 1.056,
"P": 1.08,
"strat": 0.964
},
"controls_only": {
"none": 0.994,
"S": 1.024,
"P": 1.027,
"strat": 0.93
},
"implied_asymptotic_size_pct": {
"none": 5,
"S": 5.6,
"P": 5.9,
"strat": 4.6
},
"implied_asymptotic_size_pct_controls": {
"none": 4.9,
"S": 5.3,
"P": 5.3,
"strat": 4.2
},
"lev5fu_only": {
"none": 0.995,
"S": 1.067,
"P": 1.092,
"strat": 0.936
},
"implied_asymptotic_size_pct_lev5fu": {
"none": 4.9,
"S": 5.8,
"P": 6.1,
"strat": 4.3
}
}
},
"population": {
"observed": {
"OS": {
"z_LR": -3.157,
"U_logHR": -0.373,
"U_HR": 0.689,
"U_HR_ci": [
0.546,
0.869
],
"S_logHR": -0.366,
"S_HR": 0.693,
"S_HR_ci": [
0.549,
0.876
],
"P_logHR": -0.384,
"P_HR": 0.681,
"P_HR_ci": [
0.539,
0.862
],
"S_z_wald": -3.071,
"S_z_CL": -3.025,
"S_CL_var_reduction_pct": 11,
"S_RE_pred_cox": 0.946,
"S_RE_pred_CL": 1.123,
"S_est_ratio": 0.983,
"S_se_ratio": 1.004,
"P_z_wald": -3.203,
"P_z_CL": -3.094,
"P_CL_var_reduction_pct": 12,
"P_RE_pred_cox": 1.029,
"P_RE_pred_CL": 1.137,
"P_est_ratio": 1.03,
"P_se_ratio": 1.009,
"z_SLR": -2.715,
"SLR_RE_pred": 0.739
},
"RFS": {
"z_LR": -4.258,
"U_logHR": -0.477,
"U_HR": 0.621,
"U_HR_ci": [
0.498,
0.775
],
"S_logHR": -0.47,
"S_HR": 0.625,
"S_HR_ci": [
0.5,
0.781
],
"P_logHR": -0.488,
"P_HR": 0.614,
"P_HR_ci": [
0.491,
0.767
],
"S_z_wald": -4.14,
"S_z_CL": -4.164,
"S_CL_var_reduction_pct": 11.8,
"S_RE_pred_cox": 0.945,
"S_RE_pred_CL": 1.133,
"S_est_ratio": 0.986,
"S_se_ratio": 1.005,
"P_z_wald": -4.29,
"P_z_CL": -4.257,
"P_CL_var_reduction_pct": 12.3,
"P_RE_pred_cox": 1.015,
"P_RE_pred_CL": 1.141,
"P_est_ratio": 1.025,
"P_se_ratio": 1.008,
"z_SLR": -3.864,
"SLR_RE_pred": 0.823
}
},
"trial": {
"OS": {
"z_LR": -3.16,
"U_logHR": -0.373,
"U_HR": 0.689,
"U_HR_ci": [
0.546,
0.869
],
"S_logHR": -0.366,
"S_HR": 0.693,
"S_HR_ci": [
0.549,
0.876
],
"P_logHR": -0.384,
"P_HR": 0.681,
"P_HR_ci": [
0.539,
0.862
],
"S_z_wald": -3.074,
"S_z_CL": -3.029,
"S_CL_var_reduction_pct": 11,
"S_RE_pred_cox": 0.946,
"S_RE_pred_CL": 1.123,
"S_est_ratio": 0.983,
"S_se_ratio": 1.004,
"P_z_wald": -3.205,
"P_z_CL": -3.096,
"P_CL_var_reduction_pct": 12.1,
"P_RE_pred_cox": 1.029,
"P_RE_pred_CL": 1.137,
"P_est_ratio": 1.029,
"P_se_ratio": 1.009,
"z_SLR": -2.713,
"SLR_RE_pred": 0.737
},
"RFS": {
"z_LR": -4.264,
"U_logHR": -0.477,
"U_HR": 0.621,
"U_HR_ci": [
0.498,
0.775
],
"S_logHR": -0.47,
"S_HR": 0.625,
"S_HR_ci": [
0.501,
0.781
],
"P_logHR": -0.488,
"P_HR": 0.614,
"P_HR_ci": [
0.491,
0.767
],
"S_z_wald": -4.144,
"S_z_CL": -4.172,
"S_CL_var_reduction_pct": 11.8,
"S_RE_pred_cox": 0.945,
"S_RE_pred_CL": 1.134,
"S_est_ratio": 0.986,
"S_se_ratio": 1.005,
"P_z_wald": -4.295,
"P_z_CL": -4.262,
"P_CL_var_reduction_pct": 12.4,
"P_RE_pred_cox": 1.015,
"P_RE_pred_CL": 1.141,
"P_est_ratio": 1.024,
"P_se_ratio": 1.008,
"z_SLR": -3.866,
"SLR_RE_pred": 0.822
}
},
"balanced": {
"OS": {
"z_LR": -2.918,
"U_logHR": -0.345,
"U_HR": 0.708,
"U_HR_ci": [
0.561,
0.894
],
"S_logHR": -0.376,
"S_HR": 0.686,
"S_HR_ci": [
0.543,
0.867
],
"P_logHR": -0.384,
"P_HR": 0.681,
"P_HR_ci": [
0.539,
0.861
],
"S_z_wald": -3.157,
"S_z_CL": -3.089,
"S_CL_var_reduction_pct": 10.8,
"S_RE_pred_cox": 1.171,
"S_RE_pred_CL": 1.121,
"S_est_ratio": 1.091,
"S_se_ratio": 1.003,
"P_z_wald": -3.211,
"P_z_CL": -3.105,
"P_CL_var_reduction_pct": 11.7,
"P_RE_pred_cox": 1.211,
"P_RE_pred_CL": 1.132,
"P_est_ratio": 1.113,
"P_se_ratio": 1.007,
"z_SLR": -2.758,
"SLR_RE_pred": 0.893
},
"RFS": {
"z_LR": -4.003,
"U_logHR": -0.448,
"U_HR": 0.639,
"U_HR_ci": [
0.512,
0.797
],
"S_logHR": -0.483,
"S_HR": 0.617,
"S_HR_ci": [
0.494,
0.77
],
"P_logHR": -0.489,
"P_HR": 0.613,
"P_HR_ci": [
0.491,
0.766
],
"S_z_wald": -4.261,
"S_z_CL": -4.26,
"S_CL_var_reduction_pct": 11.7,
"S_RE_pred_cox": 1.133,
"S_RE_pred_CL": 1.132,
"S_est_ratio": 1.077,
"S_se_ratio": 1.004,
"P_z_wald": -4.306,
"P_z_CL": -4.27,
"P_CL_var_reduction_pct": 12.1,
"P_RE_pred_cox": 1.157,
"P_RE_pred_CL": 1.138,
"P_est_ratio": 1.092,
"P_se_ratio": 1.006,
"z_SLR": -3.957,
"SLR_RE_pred": 0.977
}
}
},
"cells": {
"OS": {
"150": {
"null_obs": {
"R": 5000,
"events_mean": 80,
"epv_P": 8,
"rej_pct": {
"LR": 5.1,
"S_wald": 5.8,
"S_rwald": 5.6,
"S_score": 6,
"S_lrt": 6,
"S_CL": 5.8,
"S_CLkm": 4.9,
"P_wald": 6.4,
"P_rwald": 5.9,
"P_score": 6.6,
"P_lrt": 6.5,
"P_CL": 6.6,
"P_CLkm": 4.8,
"SLR": 5.3
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.33,
"S_rwald": 0.33,
"S_score": 0.34,
"S_lrt": 0.34,
"S_CL": 0.33,
"S_CLkm": 0.3,
"P_wald": 0.35,
"P_rwald": 0.33,
"P_score": 0.35,
"P_lrt": 0.35,
"P_CL": 0.35,
"P_CLkm": 0.3,
"SLR": 0.32
},
"mean_z": {
"LR": -0.004,
"S_wald": -0.001,
"S_rwald": -0.001,
"S_score": -0.001,
"S_lrt": -0.001,
"S_CL": 0.002,
"S_CLkm": 0.002,
"P_wald": -0.001,
"P_rwald": -0.001,
"P_score": -0.001,
"P_lrt": -0.001,
"P_CL": -0.002,
"P_CLkm": -0.002,
"SLR": 0.006
},
"var_z": {
"LR": 1.012,
"S_wald": 1.063,
"S_rwald": 1.037,
"S_score": 1.076,
"S_lrt": 1.073,
"S_CL": 1.049,
"S_CLkm": 0.979,
"P_wald": 1.125,
"P_rwald": 1.095,
"P_score": 1.139,
"P_lrt": 1.136,
"P_CL": 1.15,
"P_CLkm": 0.996,
"SLR": 1.003
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.021,
"P_rwald": 0.021,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.022,
"P_CLkm": 0.019,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 14.2,
"P": 19.6
},
"rej_pct_converged_only": {
"LR": 5.1,
"S_wald": 5.8,
"S_rwald": 5.6,
"S_score": 6,
"S_lrt": 6,
"S_CL": 5.8,
"S_CLkm": 4.9,
"P_wald": 6.4,
"P_rwald": 5.9,
"P_score": 6.5,
"P_lrt": 6.5,
"P_CL": 6.6,
"P_CLkm": 4.8,
"SLR": 5.3
},
"null_crit_abs_z": {
"LR": 1.97,
"S_wald": 2.023,
"S_rwald": 2.003,
"S_score": 2.039,
"S_lrt": 2.039,
"S_CL": 2.01,
"S_CLkm": 1.941,
"P_wald": 2.058,
"P_rwald": 2.049,
"P_score": 2.074,
"P_lrt": 2.074,
"P_CL": 2.09,
"P_CLkm": 1.944,
"SLR": 1.977
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.0009,
"emp_se": 0.2259,
"mod_se": 0.2251,
"rmse": 0.2259,
"cov_pct": 95.2,
"ratio_emp_mod": 1.004
},
"S": {
"truth": 0,
"mean": 0,
"bias": -0.0002,
"emp_se": 0.239,
"mod_se": 0.2302,
"rmse": 0.239,
"cov_pct": 94.2,
"ratio_emp_mod": 1.038,
"rob_se": 0.2333,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.025
},
"P": {
"truth": 0,
"mean": 0,
"bias": -0.0003,
"emp_se": 0.2538,
"mod_se": 0.2377,
"rmse": 0.2538,
"cov_pct": 93.6,
"ratio_emp_mod": 1.068,
"rob_se": 0.2414,
"rcov_pct": 94.1,
"ratio_emp_rob": 1.052
}
},
"sep_flag_S": 6,
"sep_flag_P": 6,
"nonconverged": 11
},
"null_trt": {
"R": 5000,
"events_mean": 60.7,
"epv_P": 6.1,
"rej_pct": {
"LR": 5,
"S_wald": 6.1,
"S_rwald": 5.9,
"S_score": 6.4,
"S_lrt": 6.4,
"S_CL": 5.9,
"S_CLkm": 5.3,
"P_wald": 7,
"P_rwald": 6.3,
"P_score": 7.1,
"P_lrt": 7.2,
"P_CL": 7.2,
"P_CLkm": 5.3,
"SLR": 4.7
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.34,
"S_rwald": 0.33,
"S_score": 0.35,
"S_lrt": 0.35,
"S_CL": 0.33,
"S_CLkm": 0.32,
"P_wald": 0.36,
"P_rwald": 0.34,
"P_score": 0.36,
"P_lrt": 0.37,
"P_CL": 0.36,
"P_CLkm": 0.32,
"SLR": 0.3
},
"mean_z": {
"LR": -0.006,
"S_wald": -0.009,
"S_rwald": -0.009,
"S_score": -0.009,
"S_lrt": -0.009,
"S_CL": -0.008,
"S_CLkm": -0.007,
"P_wald": -0.001,
"P_rwald": -0.001,
"P_score": -0.001,
"P_lrt": 0,
"P_CL": -0.001,
"P_CLkm": -0.001,
"SLR": -0.007
},
"var_z": {
"LR": 0.993,
"S_wald": 1.071,
"S_rwald": 1.042,
"S_score": 1.09,
"S_lrt": 1.092,
"S_CL": 1.055,
"S_CLkm": 0.984,
"P_wald": 1.133,
"P_rwald": 1.077,
"P_score": 1.153,
"P_lrt": 1.158,
"P_CL": 1.155,
"P_CLkm": 1,
"SLR": 0.994
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.022,
"S_rwald": 0.021,
"S_score": 0.022,
"S_lrt": 0.023,
"S_CL": 0.022,
"S_CLkm": 0.02,
"P_wald": 0.023,
"P_rwald": 0.022,
"P_score": 0.024,
"P_lrt": 0.024,
"P_CL": 0.024,
"P_CLkm": 0.021,
"SLR": 0.021
},
"cl_var_reduction_pct": {
"S": 13.4,
"P": 19.2
},
"rej_pct_converged_only": {
"LR": 5,
"S_wald": 6.1,
"S_rwald": 5.9,
"S_score": 6.5,
"S_lrt": 6.4,
"S_CL": 5.9,
"S_CLkm": 5.3,
"P_wald": 7,
"P_rwald": 6.3,
"P_score": 7.1,
"P_lrt": 7.2,
"P_CL": 7.2,
"P_CLkm": 5.3,
"SLR": 4.7
},
"null_crit_abs_z": {
"LR": 1.953,
"S_wald": 2.055,
"S_rwald": 2.025,
"S_score": 2.077,
"S_lrt": 2.08,
"S_CL": 2.058,
"S_CLkm": 1.988,
"P_wald": 2.134,
"P_rwald": 2.071,
"P_score": 2.159,
"P_lrt": 2.156,
"P_CL": 2.137,
"P_CLkm": 1.989,
"SLR": 1.947
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.0014,
"emp_se": 0.2583,
"mod_se": 0.2592,
"rmse": 0.2583,
"cov_pct": 95.3,
"ratio_emp_mod": 0.997
},
"S": {
"truth": 0,
"mean": -0.002,
"bias": -0.0022,
"emp_se": 0.2784,
"mod_se": 0.2659,
"rmse": 0.2783,
"cov_pct": 93.9,
"ratio_emp_mod": 1.047,
"rob_se": 0.2698,
"rcov_pct": 94.1,
"ratio_emp_rob": 1.032
},
"P": {
"truth": 0,
"mean": 0,
"bias": 0.0002,
"emp_se": 0.2967,
"mod_se": 0.2751,
"rmse": 0.2966,
"cov_pct": 93,
"ratio_emp_mod": 1.078,
"rob_se": 0.2825,
"rcov_pct": 93.7,
"ratio_emp_rob": 1.05
}
},
"sep_flag_S": 222,
"sep_flag_P": 222,
"nonconverged": 236
},
"null": {
"R": 5000,
"events_mean": 70.4,
"epv_P": 7,
"rej_pct": {
"LR": 5.3,
"S_wald": 6.3,
"S_rwald": 5.8,
"S_score": 6.6,
"S_lrt": 6.5,
"S_CL": 5.6,
"S_CLkm": 4.9,
"P_wald": 7.1,
"P_rwald": 6.5,
"P_score": 7.3,
"P_lrt": 7.3,
"P_CL": 7,
"P_CLkm": 5.1,
"SLR": 5.6
},
"rej_mcse_pct": {
"LR": 0.32,
"S_wald": 0.34,
"S_rwald": 0.33,
"S_score": 0.35,
"S_lrt": 0.35,
"S_CL": 0.33,
"S_CLkm": 0.31,
"P_wald": 0.36,
"P_rwald": 0.35,
"P_score": 0.37,
"P_lrt": 0.37,
"P_CL": 0.36,
"P_CLkm": 0.31,
"SLR": 0.32
},
"mean_z": {
"LR": -0.006,
"S_wald": -0.003,
"S_rwald": -0.003,
"S_score": -0.003,
"S_lrt": -0.004,
"S_CL": -0.002,
"S_CLkm": -0.002,
"P_wald": -0.007,
"P_rwald": -0.006,
"P_score": -0.007,
"P_lrt": -0.007,
"P_CL": -0.005,
"P_CLkm": -0.005,
"SLR": -0.002
},
"var_z": {
"LR": 1.018,
"S_wald": 1.108,
"S_rwald": 1.075,
"S_score": 1.124,
"S_lrt": 1.121,
"S_CL": 1.078,
"S_CLkm": 1.005,
"P_wald": 1.184,
"P_rwald": 1.127,
"P_score": 1.202,
"P_lrt": 1.2,
"P_CL": 1.188,
"P_CLkm": 1.028,
"SLR": 1.026
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.022,
"S_rwald": 0.021,
"S_score": 0.023,
"S_lrt": 0.023,
"S_CL": 0.022,
"S_CLkm": 0.02,
"P_wald": 0.024,
"P_rwald": 0.023,
"P_score": 0.024,
"P_lrt": 0.024,
"P_CL": 0.024,
"P_CLkm": 0.021,
"SLR": 0.021
},
"cl_var_reduction_pct": {
"S": 13.6,
"P": 18.3
},
"rej_pct_converged_only": {
"LR": 5.3,
"S_wald": 6.3,
"S_rwald": 5.8,
"S_score": 6.6,
"S_lrt": 6.5,
"S_CL": 5.6,
"S_CLkm": 5,
"P_wald": 7.1,
"P_rwald": 6.5,
"P_score": 7.3,
"P_lrt": 7.3,
"P_CL": 7,
"P_CLkm": 5.1,
"SLR": 5.6
},
"null_crit_abs_z": {
"LR": 1.987,
"S_wald": 2.068,
"S_rwald": 2.02,
"S_score": 2.087,
"S_lrt": 2.081,
"S_CL": 2.022,
"S_CLkm": 1.953,
"P_wald": 2.133,
"P_rwald": 2.059,
"P_score": 2.153,
"P_lrt": 2.149,
"P_CL": 2.116,
"P_CLkm": 1.968,
"SLR": 2.013
},
"est": {
"U": {
"truth": 0,
"mean": -0.002,
"bias": -0.0018,
"emp_se": 0.2426,
"mod_se": 0.2403,
"rmse": 0.2426,
"cov_pct": 94.9,
"ratio_emp_mod": 1.01
},
"S": {
"truth": 0,
"mean": -0.001,
"bias": -0.0011,
"emp_se": 0.2614,
"mod_se": 0.2461,
"rmse": 0.2614,
"cov_pct": 93.7,
"ratio_emp_mod": 1.062,
"rob_se": 0.2501,
"rcov_pct": 94.2,
"ratio_emp_rob": 1.045
},
"P": {
"truth": 0,
"mean": -0.002,
"bias": -0.0021,
"emp_se": 0.2799,
"mod_se": 0.2543,
"rmse": 0.2799,
"cov_pct": 92.9,
"ratio_emp_mod": 1.101,
"rob_se": 0.2609,
"rcov_pct": 93.5,
"ratio_emp_rob": 1.073
}
},
"sep_flag_S": 50,
"sep_flag_P": 50,
"nonconverged": 54
},
"balanced": {
"R": 2000,
"events_mean": 69.8,
"epv_P": 7,
"rej_pct": {
"LR": 30.7,
"S_wald": 32.2,
"S_rwald": 30.7,
"S_score": 32.6,
"S_lrt": 32.8,
"S_CL": 32.8,
"S_CLkm": 30.3,
"P_wald": 31.2,
"P_rwald": 29.1,
"P_score": 31.8,
"P_lrt": 32,
"P_CL": 35.4,
"P_CLkm": 30.1,
"SLR": 25.4
},
"rej_mcse_pct": {
"LR": 1.03,
"S_wald": 1.04,
"S_rwald": 1.03,
"S_score": 1.05,
"S_lrt": 1.05,
"S_CL": 1.05,
"S_CLkm": 1.03,
"P_wald": 1.04,
"P_rwald": 1.02,
"P_score": 1.04,
"P_lrt": 1.04,
"P_CL": 1.07,
"P_CLkm": 1.03,
"SLR": 0.97
},
"mean_z": {
"LR": -1.448,
"S_wald": -1.487,
"S_rwald": -1.454,
"S_score": -1.505,
"S_lrt": -1.505,
"S_CL": -1.544,
"S_CLkm": -1.492,
"P_wald": -1.46,
"P_rwald": -1.415,
"P_score": -1.477,
"P_lrt": -1.477,
"P_CL": -1.586,
"P_CLkm": -1.476,
"SLR": -1.301
},
"var_z": {
"LR": 0.988,
"S_wald": 1.011,
"S_rwald": 0.965,
"S_score": 1.058,
"S_lrt": 1.054,
"S_CL": 1.023,
"S_CLkm": 0.955,
"P_wald": 1.082,
"P_rwald": 1.017,
"P_score": 1.128,
"P_lrt": 1.127,
"P_CL": 1.115,
"P_CLkm": 0.965,
"SLR": 0.996
},
"var_z_mcse": {
"LR": 0.032,
"S_wald": 0.033,
"S_rwald": 0.031,
"S_score": 0.035,
"S_lrt": 0.034,
"S_CL": 0.034,
"S_CLkm": 0.032,
"P_wald": 0.035,
"P_rwald": 0.033,
"P_score": 0.037,
"P_lrt": 0.037,
"P_CL": 0.037,
"P_CLkm": 0.032,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 13.4,
"P": 18
},
"rej_pct_converged_only": {
"LR": 30.8,
"S_wald": 32.3,
"S_rwald": 30.8,
"S_score": 32.7,
"S_lrt": 32.8,
"S_CL": 32.9,
"S_CLkm": 30.4,
"P_wald": 31.3,
"P_rwald": 29.2,
"P_score": 31.9,
"P_lrt": 32.1,
"P_CL": 35.4,
"P_CLkm": 30.1,
"SLR": 25.4
},
"diff_vs_LR_pct": {
"S_wald": [
1.5,
0.78
],
"S_rwald": [
0,
0.77
],
"S_score": [
1.9,
0.79
],
"S_lrt": [
2.1,
0.78
],
"S_CL": [
2.1,
0.75
],
"S_CLkm": [
-0.4,
0.73
],
"P_wald": [
0.5,
0.85
],
"P_rwald": [
-1.6,
0.85
],
"P_score": [
1.1,
0.84
],
"P_lrt": [
1.3,
0.84
],
"P_CL": [
4.7,
0.83
],
"P_CLkm": [
-0.6,
0.79
],
"SLR": [
-5.3,
0.86
]
},
"size_adj_power_pct": {
"LR": 29.5,
"S_wald": 28.5,
"S_rwald": 28.1,
"S_score": 28.5,
"S_lrt": 28.7,
"S_CL": 30.6,
"S_CLkm": 30.6,
"P_wald": 26.1,
"P_rwald": 26.1,
"P_score": 26.2,
"P_lrt": 26.2,
"P_CL": 29.9,
"P_CLkm": 29.9,
"SLR": 23.5
},
"RE_cal": {
"S_wald": [
0.97,
0.936,
1.005
],
"S_rwald": [
0.954,
0.921,
0.989
],
"S_score": [
0.979,
0.944,
1.015
],
"S_lrt": [
0.98,
0.946,
1.016
],
"S_CL": [
1.075,
1.042,
1.108
],
"S_CLkm": [
1.075,
1.042,
1.108
],
"P_wald": [
0.874,
0.839,
0.911
],
"P_rwald": [
0.862,
0.828,
0.899
],
"P_score": [
0.882,
0.846,
0.919
],
"P_lrt": [
0.883,
0.848,
0.92
],
"P_CL": [
1.028,
0.992,
1.066
],
"P_CLkm": [
1.028,
0.992,
1.066
],
"SLR": [
0.801,
0.769,
0.836
]
},
"size_adj_power_pct_obs": {
"LR": 30.3,
"S_wald": 30,
"S_rwald": 28.8,
"S_score": 30,
"S_lrt": 30,
"S_CL": 31,
"S_CLkm": 31,
"P_wald": 28.2,
"P_rwald": 26.2,
"P_score": 28.2,
"P_lrt": 28.2,
"P_CL": 30.6,
"P_CLkm": 30.6,
"SLR": 24.8
},
"RE_cal_obs": {
"S_wald": [
1.004,
0.969,
1.041
],
"S_rwald": [
0.984,
0.949,
1.02
],
"S_score": [
1.017,
0.981,
1.054
],
"S_lrt": [
1.019,
0.983,
1.056
],
"S_CL": [
1.097,
1.064,
1.131
],
"S_CLkm": [
1.097,
1.064,
1.131
],
"P_wald": [
0.914,
0.877,
0.953
],
"P_rwald": [
0.882,
0.846,
0.92
],
"P_score": [
0.925,
0.887,
0.965
],
"P_lrt": [
0.928,
0.89,
0.967
],
"P_CL": [
1.055,
1.018,
1.094
],
"P_CLkm": [
1.055,
1.018,
1.094
],
"SLR": [
0.816,
0.782,
0.85
]
},
"size_adj_power_pct_trt": {
"LR": 30.9,
"S_wald": 28.8,
"S_rwald": 27.9,
"S_score": 28.8,
"S_lrt": 28.7,
"S_CL": 29.1,
"S_CLkm": 29.1,
"P_wald": 26.1,
"P_rwald": 25.8,
"P_score": 25.9,
"P_lrt": 25.9,
"P_CL": 29.3,
"P_CLkm": 29.3,
"SLR": 25.8
},
"RE_cal_trt": {
"S_wald": [
0.978,
0.944,
1.013
],
"S_rwald": [
0.96,
0.927,
0.995
],
"S_score": [
0.984,
0.95,
1.02
],
"S_lrt": [
0.982,
0.947,
1.017
],
"S_CL": [
1.07,
1.038,
1.104
],
"S_CLkm": [
1.07,
1.038,
1.104
],
"P_wald": [
0.891,
0.855,
0.929
],
"P_rwald": [
0.88,
0.844,
0.918
],
"P_score": [
0.896,
0.86,
0.935
],
"P_lrt": [
0.893,
0.856,
0.931
],
"P_CL": [
1.031,
0.993,
1.07
],
"P_CLkm": [
1.031,
0.993,
1.07
],
"SLR": [
0.807,
0.773,
0.841
]
},
"RE": {
"S_wald": [
1.055,
1.027,
1.083
],
"S_rwald": [
1.008,
0.981,
1.035
],
"S_score": [
1.081,
1.052,
1.11
],
"S_lrt": [
1.08,
1.052,
1.109
],
"S_CL": [
1.138,
1.112,
1.163
],
"S_CLkm": [
1.061,
1.037,
1.085
],
"P_wald": [
1.017,
0.986,
1.048
],
"P_rwald": [
0.954,
0.925,
0.985
],
"P_score": [
1.041,
1.009,
1.074
],
"P_lrt": [
1.041,
1.009,
1.073
],
"P_CL": [
1.2,
1.168,
1.231
],
"P_CLkm": [
1.039,
1.011,
1.066
],
"SLR": [
0.808,
0.782,
0.834
]
},
"decomp": {
"S": {
"est_ratio": 1.068,
"se_ratio": 1.026,
"empsd_ratio": 1.064
},
"P": {
"est_ratio": 1.084,
"se_ratio": 1.061,
"empsd_ratio": 1.14
}
},
"est": {
"U": {
"truth": -0.345,
"mean": -0.35,
"bias": -0.0055,
"emp_se": 0.2422,
"mod_se": 0.2436,
"rmse": 0.2422,
"cov_pct": 95,
"ratio_emp_mod": 0.994
},
"S": {
"truth": -0.376,
"mean": -0.374,
"bias": 0.0021,
"emp_se": 0.2577,
"mod_se": 0.25,
"rmse": 0.2577,
"cov_pct": 93.5,
"ratio_emp_mod": 1.031,
"rob_se": 0.2557,
"rcov_pct": 94.3,
"ratio_emp_rob": 1.008
},
"P": {
"truth": -0.384,
"mean": -0.38,
"bias": 0.0041,
"emp_se": 0.2761,
"mod_se": 0.2584,
"rmse": 0.276,
"cov_pct": 93,
"ratio_emp_mod": 1.068,
"rob_se": 0.2673,
"rcov_pct": 93.7,
"ratio_emp_rob": 1.033
}
},
"sep_flag_S": 22,
"sep_flag_P": 22,
"nonconverged": 27
},
"trial": {
"R": 1000,
"events_mean": 70.3,
"epv_P": 7,
"rej_pct": {
"LR": 33.7,
"S_wald": 29.9,
"S_rwald": 28,
"S_score": 30.1,
"S_lrt": 30.1,
"S_CL": 32.3,
"S_CLkm": 30.8,
"P_wald": 30.4,
"P_rwald": 27.6,
"P_score": 30.8,
"P_lrt": 31,
"P_CL": 34.6,
"P_CLkm": 30.1,
"SLR": 23.5
},
"rej_mcse_pct": {
"LR": 1.49,
"S_wald": 1.45,
"S_rwald": 1.42,
"S_score": 1.45,
"S_lrt": 1.45,
"S_CL": 1.48,
"S_CLkm": 1.46,
"P_wald": 1.45,
"P_rwald": 1.41,
"P_score": 1.46,
"P_lrt": 1.46,
"P_CL": 1.5,
"P_CLkm": 1.45,
"SLR": 1.34
},
"mean_z": {
"LR": -1.543,
"S_wald": -1.404,
"S_rwald": -1.363,
"S_score": -1.42,
"S_lrt": -1.419,
"S_CL": -1.491,
"S_CLkm": -1.44,
"P_wald": -1.4,
"P_rwald": -1.352,
"P_score": -1.416,
"P_lrt": -1.416,
"P_CL": -1.555,
"P_CLkm": -1.447,
"SLR": -1.269
},
"var_z": {
"LR": 0.953,
"S_wald": 1.067,
"S_rwald": 1.006,
"S_score": 1.111,
"S_lrt": 1.106,
"S_CL": 1.058,
"S_CLkm": 0.987,
"P_wald": 1.132,
"P_rwald": 1.057,
"P_score": 1.177,
"P_lrt": 1.174,
"P_CL": 1.148,
"P_CLkm": 0.994,
"SLR": 1.001
},
"var_z_mcse": {
"LR": 0.041,
"S_wald": 0.045,
"S_rwald": 0.043,
"S_score": 0.047,
"S_lrt": 0.047,
"S_CL": 0.045,
"S_CLkm": 0.042,
"P_wald": 0.048,
"P_rwald": 0.046,
"P_score": 0.051,
"P_lrt": 0.05,
"P_CL": 0.05,
"P_CLkm": 0.043,
"SLR": 0.046
},
"cl_var_reduction_pct": {
"S": 13.6,
"P": 18.5
},
"rej_pct_converged_only": {
"LR": 33.6,
"S_wald": 29.7,
"S_rwald": 27.8,
"S_score": 29.9,
"S_lrt": 29.9,
"S_CL": 32.1,
"S_CLkm": 30.5,
"P_wald": 30.2,
"P_rwald": 27.4,
"P_score": 30.6,
"P_lrt": 30.7,
"P_CL": 34.4,
"P_CLkm": 29.8,
"SLR": 23.2
},
"diff_vs_LR_pct": {
"S_wald": [
-3.8,
1.14
],
"S_rwald": [
-5.7,
1.13
],
"S_score": [
-3.6,
1.14
],
"S_lrt": [
-3.6,
1.14
],
"S_CL": [
-1.4,
1.11
],
"S_CLkm": [
-2.9,
1.13
],
"P_wald": [
-3.3,
1.28
],
"P_rwald": [
-6.1,
1.24
],
"P_score": [
-2.9,
1.27
],
"P_lrt": [
-2.7,
1.27
],
"P_CL": [
0.9,
1.2
],
"P_CLkm": [
-3.6,
1.23
],
"SLR": [
-10.2,
1.22
]
},
"size_adj_power_pct": {
"LR": 32.8,
"S_wald": 26.4,
"S_rwald": 26.2,
"S_score": 26.5,
"S_lrt": 26.9,
"S_CL": 30.9,
"S_CLkm": 30.9,
"P_wald": 25,
"P_rwald": 24.8,
"P_score": 25,
"P_lrt": 25.1,
"P_CL": 29.4,
"P_CLkm": 29.4,
"SLR": 21.9
},
"RE_cal": {
"S_wald": [
0.76,
0.724,
0.798
],
"S_rwald": [
0.739,
0.703,
0.775
],
"S_score": [
0.767,
0.73,
0.805
],
"S_lrt": [
0.767,
0.731,
0.806
],
"S_CL": [
0.882,
0.845,
0.92
],
"S_CLkm": [
0.882,
0.845,
0.92
],
"P_wald": [
0.708,
0.669,
0.748
],
"P_rwald": [
0.694,
0.656,
0.734
],
"P_score": [
0.713,
0.674,
0.754
],
"P_lrt": [
0.714,
0.675,
0.755
],
"P_CL": [
0.871,
0.829,
0.914
],
"P_CLkm": [
0.871,
0.829,
0.914
],
"SLR": [
0.671,
0.634,
0.71
]
},
"size_adj_power_pct_obs": {
"LR": 33.5,
"S_wald": 28.2,
"S_rwald": 26.6,
"S_score": 28.2,
"S_lrt": 28.1,
"S_CL": 31.3,
"S_CLkm": 31.3,
"P_wald": 27.1,
"P_rwald": 25.2,
"P_score": 27.1,
"P_lrt": 27.2,
"P_CL": 30.5,
"P_CLkm": 30.5,
"SLR": 23.2
},
"RE_cal_obs": {
"S_wald": [
0.787,
0.75,
0.826
],
"S_rwald": [
0.761,
0.725,
0.799
],
"S_score": [
0.797,
0.759,
0.836
],
"S_lrt": [
0.798,
0.76,
0.837
],
"S_CL": [
0.9,
0.863,
0.939
],
"S_CLkm": [
0.9,
0.863,
0.939
],
"P_wald": [
0.74,
0.699,
0.783
],
"P_rwald": [
0.71,
0.671,
0.752
],
"P_score": [
0.748,
0.707,
0.792
],
"P_lrt": [
0.75,
0.709,
0.794
],
"P_CL": [
0.894,
0.851,
0.938
],
"P_CLkm": [
0.894,
0.851,
0.938
],
"SLR": [
0.683,
0.645,
0.723
]
},
"size_adj_power_pct_trt": {
"LR": 33.7,
"S_wald": 27.1,
"S_rwald": 26.2,
"S_score": 27,
"S_lrt": 26.9,
"S_CL": 29.7,
"S_CLkm": 29.7,
"P_wald": 25,
"P_rwald": 24.7,
"P_score": 24.7,
"P_lrt": 24.9,
"P_CL": 29,
"P_CLkm": 29,
"SLR": 23.9
},
"RE_cal_trt": {
"S_wald": [
0.766,
0.73,
0.804
],
"S_rwald": [
0.743,
0.708,
0.78
],
"S_score": [
0.771,
0.735,
0.809
],
"S_lrt": [
0.768,
0.732,
0.807
],
"S_CL": [
0.878,
0.842,
0.916
],
"S_CLkm": [
0.878,
0.842,
0.916
],
"P_wald": [
0.721,
0.682,
0.763
],
"P_rwald": [
0.708,
0.669,
0.75
],
"P_score": [
0.725,
0.685,
0.767
],
"P_lrt": [
0.722,
0.682,
0.764
],
"P_CL": [
0.873,
0.83,
0.917
],
"P_CLkm": [
0.873,
0.83,
0.917
],
"SLR": [
0.675,
0.637,
0.715
]
},
"RE": {
"S_wald": [
0.827,
0.793,
0.862
],
"S_rwald": [
0.78,
0.748,
0.814
],
"S_score": [
0.847,
0.811,
0.883
],
"S_lrt": [
0.845,
0.81,
0.881
],
"S_CL": [
0.933,
0.899,
0.968
],
"S_CLkm": [
0.871,
0.839,
0.903
],
"P_wald": [
0.823,
0.783,
0.864
],
"P_rwald": [
0.768,
0.73,
0.807
],
"P_score": [
0.842,
0.801,
0.884
],
"P_lrt": [
0.842,
0.801,
0.884
],
"P_CL": [
1.016,
0.973,
1.059
],
"P_CLkm": [
0.879,
0.842,
0.917
],
"SLR": [
0.676,
0.643,
0.711
]
},
"decomp": {
"S": {
"est_ratio": 0.95,
"se_ratio": 1.031,
"empsd_ratio": 1.114
},
"P": {
"est_ratio": 0.98,
"se_ratio": 1.068,
"empsd_ratio": 1.187
}
},
"est": {
"U": {
"truth": -0.373,
"mean": -0.372,
"bias": 0.0004,
"emp_se": 0.2381,
"mod_se": 0.2429,
"rmse": 0.238,
"cov_pct": 96.1,
"ratio_emp_mod": 0.98
},
"S": {
"truth": -0.366,
"mean": -0.354,
"bias": 0.0127,
"emp_se": 0.2653,
"mod_se": 0.2504,
"rmse": 0.2655,
"cov_pct": 94.3,
"ratio_emp_mod": 1.06,
"rob_se": 0.258,
"rcov_pct": 95.6,
"ratio_emp_rob": 1.029
},
"P": {
"truth": -0.384,
"mean": -0.365,
"bias": 0.0186,
"emp_se": 0.2826,
"mod_se": 0.2594,
"rmse": 0.2831,
"cov_pct": 93.3,
"ratio_emp_mod": 1.09,
"rob_se": 0.2693,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.05
}
},
"sep_flag_S": 10,
"sep_flag_P": 10,
"nonconverged": 11
}
},
"300": {
"null_obs": {
"R": 5000,
"events_mean": 160,
"epv_P": 16,
"rej_pct": {
"LR": 4.7,
"S_wald": 5.7,
"S_rwald": 5.3,
"S_score": 5.8,
"S_lrt": 5.8,
"S_CL": 5.1,
"S_CLkm": 4.8,
"P_wald": 5.6,
"P_rwald": 5.4,
"P_score": 5.7,
"P_lrt": 5.7,
"P_CL": 5.4,
"P_CLkm": 4.6,
"SLR": 4.8
},
"rej_mcse_pct": {
"LR": 0.3,
"S_wald": 0.33,
"S_rwald": 0.32,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.31,
"S_CLkm": 0.3,
"P_wald": 0.33,
"P_rwald": 0.32,
"P_score": 0.33,
"P_lrt": 0.33,
"P_CL": 0.32,
"P_CLkm": 0.3,
"SLR": 0.3
},
"mean_z": {
"LR": -0.008,
"S_wald": -0.004,
"S_rwald": -0.004,
"S_score": -0.004,
"S_lrt": -0.004,
"S_CL": -0.006,
"S_CLkm": -0.006,
"P_wald": -0.009,
"P_rwald": -0.008,
"P_score": -0.009,
"P_lrt": -0.009,
"P_CL": -0.011,
"P_CLkm": -0.011,
"SLR": -0.011
},
"var_z": {
"LR": 1.013,
"S_wald": 1.065,
"S_rwald": 1.028,
"S_score": 1.072,
"S_lrt": 1.07,
"S_CL": 1.027,
"S_CLkm": 0.993,
"P_wald": 1.079,
"P_rwald": 1.043,
"P_score": 1.085,
"P_lrt": 1.084,
"P_CL": 1.055,
"P_CLkm": 0.984,
"SLR": 1.013
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.021,
"P_rwald": 0.021,
"P_score": 0.021,
"P_lrt": 0.021,
"P_CL": 0.021,
"P_CLkm": 0.019,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 13.1,
"P": 16.7
},
"rej_pct_converged_only": {
"LR": 4.7,
"S_wald": 5.7,
"S_rwald": 5.3,
"S_score": 5.8,
"S_lrt": 5.8,
"S_CL": 5.1,
"S_CLkm": 4.8,
"P_wald": 5.6,
"P_rwald": 5.4,
"P_score": 5.7,
"P_lrt": 5.7,
"P_CL": 5.4,
"P_CLkm": 4.6,
"SLR": 4.8
},
"null_crit_abs_z": {
"LR": 1.935,
"S_wald": 2.014,
"S_rwald": 1.982,
"S_score": 2.022,
"S_lrt": 2.022,
"S_CL": 1.972,
"S_CLkm": 1.939,
"P_wald": 2.021,
"P_rwald": 1.993,
"P_score": 2.03,
"P_lrt": 2.028,
"P_CL": 1.989,
"P_CLkm": 1.921,
"SLR": 1.943
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.0013,
"emp_se": 0.1592,
"mod_se": 0.1586,
"rmse": 0.1592,
"cov_pct": 95.5,
"ratio_emp_mod": 1.004
},
"S": {
"truth": 0,
"mean": -0.001,
"bias": -0.0007,
"emp_se": 0.166,
"mod_se": 0.1604,
"rmse": 0.166,
"cov_pct": 94.3,
"ratio_emp_mod": 1.035,
"rob_se": 0.1633,
"rcov_pct": 94.7,
"ratio_emp_rob": 1.016
},
"P": {
"truth": 0,
"mean": -0.001,
"bias": -0.0015,
"emp_se": 0.1696,
"mod_se": 0.1628,
"rmse": 0.1696,
"cov_pct": 94.4,
"ratio_emp_mod": 1.042,
"rob_se": 0.1656,
"rcov_pct": 94.6,
"ratio_emp_rob": 1.024
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"null_trt": {
"R": 5000,
"events_mean": 121.3,
"epv_P": 12.1,
"rej_pct": {
"LR": 4.2,
"S_wald": 5.5,
"S_rwald": 5.2,
"S_score": 5.6,
"S_lrt": 5.6,
"S_CL": 5.1,
"S_CLkm": 4.5,
"P_wald": 6,
"P_rwald": 5.2,
"P_score": 6.2,
"P_lrt": 6.1,
"P_CL": 5.4,
"P_CLkm": 4.6,
"SLR": 4.6
},
"rej_mcse_pct": {
"LR": 0.28,
"S_wald": 0.32,
"S_rwald": 0.31,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.31,
"S_CLkm": 0.29,
"P_wald": 0.34,
"P_rwald": 0.31,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.32,
"P_CLkm": 0.3,
"SLR": 0.3
},
"mean_z": {
"LR": 0.002,
"S_wald": 0.008,
"S_rwald": 0.008,
"S_score": 0.008,
"S_lrt": 0.008,
"S_CL": 0.01,
"S_CLkm": 0.01,
"P_wald": 0.003,
"P_rwald": 0.003,
"P_score": 0.003,
"P_lrt": 0.003,
"P_CL": 0.009,
"P_CLkm": 0.009,
"SLR": 0.008
},
"var_z": {
"LR": 0.969,
"S_wald": 1.055,
"S_rwald": 1.014,
"S_score": 1.063,
"S_lrt": 1.063,
"S_CL": 1.007,
"S_CLkm": 0.974,
"P_wald": 1.104,
"P_rwald": 1.045,
"P_score": 1.113,
"P_lrt": 1.113,
"P_CL": 1.052,
"P_CLkm": 0.982,
"SLR": 0.974
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.022,
"P_rwald": 0.021,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.019
},
"cl_var_reduction_pct": {
"S": 11.8,
"P": 15.6
},
"rej_pct_converged_only": {
"LR": 4.2,
"S_wald": 5.5,
"S_rwald": 5.2,
"S_score": 5.6,
"S_lrt": 5.6,
"S_CL": 5.1,
"S_CLkm": 4.6,
"P_wald": 6,
"P_rwald": 5.2,
"P_score": 6.2,
"P_lrt": 6.1,
"P_CL": 5.4,
"P_CLkm": 4.7,
"SLR": 4.6
},
"null_crit_abs_z": {
"LR": 1.897,
"S_wald": 2.009,
"S_rwald": 1.972,
"S_score": 2.021,
"S_lrt": 2.021,
"S_CL": 1.963,
"S_CLkm": 1.93,
"P_wald": 2.048,
"P_rwald": 1.984,
"P_score": 2.06,
"P_lrt": 2.057,
"P_CL": 1.996,
"P_CLkm": 1.928,
"SLR": 1.913
},
"est": {
"U": {
"truth": 0,
"mean": 0,
"bias": 0.0003,
"emp_se": 0.1794,
"mod_se": 0.1824,
"rmse": 0.1794,
"cov_pct": 96,
"ratio_emp_mod": 0.984
},
"S": {
"truth": 0,
"mean": 0.001,
"bias": 0.0015,
"emp_se": 0.1909,
"mod_se": 0.1847,
"rmse": 0.1909,
"cov_pct": 94.5,
"ratio_emp_mod": 1.034,
"rob_se": 0.1883,
"rcov_pct": 94.8,
"ratio_emp_rob": 1.014
},
"P": {
"truth": 0,
"mean": 0.001,
"bias": 0.0005,
"emp_se": 0.1988,
"mod_se": 0.1878,
"rmse": 0.1987,
"cov_pct": 94,
"ratio_emp_mod": 1.059,
"rob_se": 0.193,
"rcov_pct": 94.8,
"ratio_emp_rob": 1.03
}
},
"sep_flag_S": 15,
"sep_flag_P": 15,
"nonconverged": 27
},
"null": {
"R": 5000,
"events_mean": 141.2,
"epv_P": 14.1,
"rej_pct": {
"LR": 4.9,
"S_wald": 5.7,
"S_rwald": 5.2,
"S_score": 5.7,
"S_lrt": 5.8,
"S_CL": 5.1,
"S_CLkm": 4.7,
"P_wald": 5.9,
"P_rwald": 5.3,
"P_score": 6.1,
"P_lrt": 6.1,
"P_CL": 5.7,
"P_CLkm": 4.7,
"SLR": 4.9
},
"rej_mcse_pct": {
"LR": 0.3,
"S_wald": 0.33,
"S_rwald": 0.31,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.31,
"S_CLkm": 0.3,
"P_wald": 0.33,
"P_rwald": 0.32,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.33,
"P_CLkm": 0.3,
"SLR": 0.31
},
"mean_z": {
"LR": -0.015,
"S_wald": -0.012,
"S_rwald": -0.011,
"S_score": -0.012,
"S_lrt": -0.012,
"S_CL": -0.013,
"S_CLkm": -0.013,
"P_wald": -0.016,
"P_rwald": -0.015,
"P_score": -0.016,
"P_lrt": -0.016,
"P_CL": -0.014,
"P_CLkm": -0.013,
"SLR": -0.012
},
"var_z": {
"LR": 0.999,
"S_wald": 1.052,
"S_rwald": 1.01,
"S_score": 1.059,
"S_lrt": 1.058,
"S_CL": 1.02,
"S_CLkm": 0.986,
"P_wald": 1.094,
"P_rwald": 1.035,
"P_score": 1.102,
"P_lrt": 1.101,
"P_CL": 1.059,
"P_CLkm": 0.988,
"SLR": 0.984
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.022,
"P_rwald": 0.02,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 12.4,
"P": 15.3
},
"rej_pct_converged_only": {
"LR": 4.9,
"S_wald": 5.7,
"S_rwald": 5.2,
"S_score": 5.7,
"S_lrt": 5.8,
"S_CL": 5.1,
"S_CLkm": 4.7,
"P_wald": 5.9,
"P_rwald": 5.3,
"P_score": 6.1,
"P_lrt": 6.1,
"P_CL": 5.7,
"P_CLkm": 4.7,
"SLR": 4.9
},
"null_crit_abs_z": {
"LR": 1.948,
"S_wald": 2.018,
"S_rwald": 1.973,
"S_score": 2.027,
"S_lrt": 2.027,
"S_CL": 1.968,
"S_CLkm": 1.934,
"P_wald": 2.052,
"P_rwald": 1.992,
"P_score": 2.06,
"P_lrt": 2.062,
"P_CL": 2.005,
"P_CLkm": 1.937,
"SLR": 1.945
},
"est": {
"U": {
"truth": 0,
"mean": -0.002,
"bias": -0.0024,
"emp_se": 0.1689,
"mod_se": 0.169,
"rmse": 0.1689,
"cov_pct": 95.3,
"ratio_emp_mod": 0.999
},
"S": {
"truth": 0,
"mean": -0.002,
"bias": -0.002,
"emp_se": 0.1763,
"mod_se": 0.171,
"rmse": 0.1763,
"cov_pct": 94.3,
"ratio_emp_mod": 1.031,
"rob_se": 0.1744,
"rcov_pct": 94.8,
"ratio_emp_rob": 1.011
},
"P": {
"truth": 0,
"mean": -0.003,
"bias": -0.0027,
"emp_se": 0.1827,
"mod_se": 0.1737,
"rmse": 0.1827,
"cov_pct": 94.1,
"ratio_emp_mod": 1.052,
"rob_se": 0.1785,
"rcov_pct": 94.7,
"ratio_emp_rob": 1.024
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"balanced": {
"R": 2000,
"events_mean": 140,
"epv_P": 14,
"rej_pct": {
"LR": 52.7,
"S_wald": 58,
"S_rwald": 56,
"S_score": 58.5,
"S_lrt": 58.7,
"S_CL": 58.2,
"S_CLkm": 57.1,
"P_wald": 58.1,
"P_rwald": 55.4,
"P_score": 58.5,
"P_lrt": 58.5,
"P_CL": 59.2,
"P_CLkm": 57,
"SLR": 47.1
},
"rej_mcse_pct": {
"LR": 1.12,
"S_wald": 1.1,
"S_rwald": 1.11,
"S_score": 1.1,
"S_lrt": 1.1,
"S_CL": 1.1,
"S_CLkm": 1.11,
"P_wald": 1.1,
"P_rwald": 1.11,
"P_score": 1.1,
"P_lrt": 1.1,
"P_CL": 1.1,
"P_CLkm": 1.11,
"SLR": 1.12
},
"mean_z": {
"LR": -2.053,
"S_wald": -2.158,
"S_rwald": -2.102,
"S_score": -2.177,
"S_lrt": -2.176,
"S_CL": -2.177,
"S_CLkm": -2.14,
"P_wald": -2.154,
"P_rwald": -2.084,
"P_score": -2.173,
"P_lrt": -2.172,
"P_CL": -2.206,
"P_CLkm": -2.131,
"SLR": -1.903
},
"var_z": {
"LR": 0.971,
"S_wald": 0.992,
"S_rwald": 0.937,
"S_score": 1.032,
"S_lrt": 1.029,
"S_CL": 0.983,
"S_CLkm": 0.95,
"P_wald": 1.04,
"P_rwald": 0.972,
"P_score": 1.08,
"P_lrt": 1.077,
"P_CL": 1.024,
"P_CLkm": 0.956,
"SLR": 0.995
},
"var_z_mcse": {
"LR": 0.031,
"S_wald": 0.031,
"S_rwald": 0.029,
"S_score": 0.032,
"S_lrt": 0.032,
"S_CL": 0.031,
"S_CLkm": 0.03,
"P_wald": 0.032,
"P_rwald": 0.03,
"P_score": 0.034,
"P_lrt": 0.034,
"P_CL": 0.032,
"P_CLkm": 0.03,
"SLR": 0.032
},
"cl_var_reduction_pct": {
"S": 12.1,
"P": 14.7
},
"rej_pct_converged_only": {
"LR": 52.7,
"S_wald": 58,
"S_rwald": 56,
"S_score": 58.5,
"S_lrt": 58.7,
"S_CL": 58.2,
"S_CLkm": 57.1,
"P_wald": 58,
"P_rwald": 55.3,
"P_score": 58.4,
"P_lrt": 58.4,
"P_CL": 59.2,
"P_CLkm": 56.9,
"SLR": 47.1
},
"diff_vs_LR_pct": {
"S_wald": [
5.3,
0.85
],
"S_rwald": [
3.3,
0.84
],
"S_score": [
5.8,
0.84
],
"S_lrt": [
6,
0.84
],
"S_CL": [
5.5,
0.77
],
"S_CLkm": [
4.4,
0.78
],
"P_wald": [
5.3,
0.88
],
"P_rwald": [
2.6,
0.89
],
"P_score": [
5.8,
0.89
],
"P_lrt": [
5.8,
0.89
],
"P_CL": [
6.6,
0.83
],
"P_CLkm": [
4.2,
0.82
],
"SLR": [
-5.6,
0.89
]
},
"size_adj_power_pct": {
"LR": 53,
"S_wald": 55.9,
"S_rwald": 55.6,
"S_score": 55.9,
"S_lrt": 55.9,
"S_CL": 58.1,
"S_CLkm": 58.1,
"P_wald": 54.9,
"P_rwald": 54,
"P_score": 54.9,
"P_lrt": 54.9,
"P_CL": 57.6,
"P_CLkm": 57.6,
"SLR": 47.5
},
"RE_cal": {
"S_wald": [
1.049,
1.02,
1.08
],
"S_rwald": [
1.038,
1.009,
1.068
],
"S_score": [
1.061,
1.031,
1.092
],
"S_lrt": [
1.061,
1.031,
1.092
],
"S_CL": [
1.101,
1.074,
1.129
],
"S_CLkm": [
1.101,
1.074,
1.129
],
"P_wald": [
1.005,
0.974,
1.037
],
"P_rwald": [
0.995,
0.963,
1.027
],
"P_score": [
1.016,
0.984,
1.048
],
"P_lrt": [
1.016,
0.984,
1.049
],
"P_CL": [
1.089,
1.059,
1.12
],
"P_CLkm": [
1.089,
1.059,
1.12
],
"SLR": [
0.872,
0.844,
0.901
]
},
"size_adj_power_pct_obs": {
"LR": 53.6,
"S_wald": 55.9,
"S_rwald": 55.3,
"S_score": 56,
"S_lrt": 56,
"S_CL": 57.9,
"S_CLkm": 57.9,
"P_wald": 56,
"P_rwald": 54,
"P_score": 56,
"P_lrt": 56,
"P_CL": 58.1,
"P_CLkm": 58.1,
"SLR": 47.5
},
"RE_cal_obs": {
"S_wald": [
1.051,
1.021,
1.082
],
"S_rwald": [
1.034,
1.004,
1.064
],
"S_score": [
1.063,
1.033,
1.094
],
"S_lrt": [
1.064,
1.034,
1.095
],
"S_CL": [
1.109,
1.082,
1.138
],
"S_CLkm": [
1.109,
1.082,
1.138
],
"P_wald": [
1.034,
1.001,
1.068
],
"P_rwald": [
1.002,
0.969,
1.035
],
"P_score": [
1.046,
1.012,
1.081
],
"P_lrt": [
1.047,
1.013,
1.082
],
"P_CL": [
1.109,
1.078,
1.142
],
"P_CLkm": [
1.109,
1.078,
1.142
],
"SLR": [
0.859,
0.832,
0.888
]
},
"size_adj_power_pct_trt": {
"LR": 55,
"S_wald": 56.2,
"S_rwald": 55.6,
"S_score": 56,
"S_lrt": 56.1,
"S_CL": 58.1,
"S_CLkm": 58.1,
"P_wald": 55.1,
"P_rwald": 54.4,
"P_score": 54.9,
"P_lrt": 55.2,
"P_CL": 57.8,
"P_CLkm": 57.8,
"SLR": 48.9
},
"RE_cal_trt": {
"S_wald": [
1.016,
0.987,
1.045
],
"S_rwald": [
1.002,
0.974,
1.031
],
"S_score": [
1.025,
0.997,
1.055
],
"S_lrt": [
1.025,
0.996,
1.054
],
"S_CL": [
1.082,
1.056,
1.11
],
"S_CLkm": [
1.082,
1.056,
1.11
],
"P_wald": [
0.967,
0.936,
0.998
],
"P_rwald": [
0.956,
0.926,
0.988
],
"P_score": [
0.975,
0.944,
1.008
],
"P_lrt": [
0.976,
0.945,
1.008
],
"P_CL": [
1.064,
1.034,
1.095
],
"P_CLkm": [
1.064,
1.034,
1.095
],
"SLR": [
0.855,
0.827,
0.884
]
},
"RE": {
"S_wald": [
1.105,
1.086,
1.124
],
"S_rwald": [
1.049,
1.031,
1.067
],
"S_score": [
1.125,
1.105,
1.145
],
"S_lrt": [
1.124,
1.104,
1.143
],
"S_CL": [
1.125,
1.108,
1.142
],
"S_CLkm": [
1.087,
1.071,
1.104
],
"P_wald": [
1.101,
1.08,
1.122
],
"P_rwald": [
1.031,
1.011,
1.052
],
"P_score": [
1.12,
1.099,
1.142
],
"P_lrt": [
1.12,
1.098,
1.142
],
"P_CL": [
1.155,
1.136,
1.175
],
"P_CLkm": [
1.078,
1.06,
1.096
],
"SLR": [
0.859,
0.841,
0.878
]
},
"decomp": {
"S": {
"est_ratio": 1.076,
"se_ratio": 1.014,
"empsd_ratio": 1.047
},
"P": {
"est_ratio": 1.093,
"se_ratio": 1.031,
"empsd_ratio": 1.09
}
},
"est": {
"U": {
"truth": -0.345,
"mean": -0.35,
"bias": -0.0047,
"emp_se": 0.1691,
"mod_se": 0.1713,
"rmse": 0.1691,
"cov_pct": 95.6,
"ratio_emp_mod": 0.987
},
"S": {
"truth": -0.376,
"mean": -0.376,
"bias": 0.0001,
"emp_se": 0.177,
"mod_se": 0.1737,
"rmse": 0.177,
"cov_pct": 94.5,
"ratio_emp_mod": 1.019,
"rob_se": 0.1782,
"rcov_pct": 95.2,
"ratio_emp_rob": 0.993
},
"P": {
"truth": -0.384,
"mean": -0.382,
"bias": 0.0021,
"emp_se": 0.1842,
"mod_se": 0.1767,
"rmse": 0.1842,
"cov_pct": 94.3,
"ratio_emp_mod": 1.043,
"rob_se": 0.1826,
"rcov_pct": 94.9,
"ratio_emp_rob": 1.009
}
},
"sep_flag_S": 1,
"sep_flag_P": 1,
"nonconverged": 1
},
"trial": {
"R": 1000,
"events_mean": 140.1,
"epv_P": 14,
"rej_pct": {
"LR": 59.7,
"S_wald": 55.9,
"S_rwald": 53.4,
"S_score": 56.4,
"S_lrt": 56.4,
"S_CL": 57.6,
"S_CLkm": 56.5,
"P_wald": 57.1,
"P_rwald": 54.7,
"P_score": 57.2,
"P_lrt": 57.2,
"P_CL": 59.4,
"P_CLkm": 57.1,
"SLR": 46.1
},
"rej_mcse_pct": {
"LR": 1.55,
"S_wald": 1.57,
"S_rwald": 1.58,
"S_score": 1.57,
"S_lrt": 1.57,
"S_CL": 1.56,
"S_CLkm": 1.57,
"P_wald": 1.57,
"P_rwald": 1.57,
"P_score": 1.56,
"P_lrt": 1.56,
"P_CL": 1.55,
"P_CLkm": 1.57,
"SLR": 1.58
},
"mean_z": {
"LR": -2.212,
"S_wald": -2.078,
"S_rwald": -2.011,
"S_score": -2.096,
"S_lrt": -2.095,
"S_CL": -2.115,
"S_CLkm": -2.079,
"P_wald": -2.126,
"P_rwald": -2.05,
"P_score": -2.144,
"P_lrt": -2.144,
"P_CL": -2.186,
"P_CLkm": -2.112,
"SLR": -1.842
},
"var_z": {
"LR": 1.006,
"S_wald": 1.039,
"S_rwald": 0.972,
"S_score": 1.076,
"S_lrt": 1.074,
"S_CL": 1.038,
"S_CLkm": 1.003,
"P_wald": 1.076,
"P_rwald": 1.004,
"P_score": 1.114,
"P_lrt": 1.112,
"P_CL": 1.1,
"P_CLkm": 1.027,
"SLR": 1.047
},
"var_z_mcse": {
"LR": 0.046,
"S_wald": 0.05,
"S_rwald": 0.046,
"S_score": 0.051,
"S_lrt": 0.051,
"S_CL": 0.05,
"S_CLkm": 0.048,
"P_wald": 0.051,
"P_rwald": 0.048,
"P_score": 0.053,
"P_lrt": 0.053,
"P_CL": 0.053,
"P_CLkm": 0.049,
"SLR": 0.05
},
"cl_var_reduction_pct": {
"S": 12.4,
"P": 15.2
},
"rej_pct_converged_only": {
"LR": 59.7,
"S_wald": 55.9,
"S_rwald": 53.4,
"S_score": 56.4,
"S_lrt": 56.4,
"S_CL": 57.6,
"S_CLkm": 56.5,
"P_wald": 57.1,
"P_rwald": 54.7,
"P_score": 57.2,
"P_lrt": 57.2,
"P_CL": 59.4,
"P_CLkm": 57.1,
"SLR": 46.1
},
"diff_vs_LR_pct": {
"S_wald": [
-3.8,
1.11
],
"S_rwald": [
-6.3,
1.13
],
"S_score": [
-3.3,
1.1
],
"S_lrt": [
-3.3,
1.1
],
"S_CL": [
-2.1,
1.14
],
"S_CLkm": [
-3.2,
1.15
],
"P_wald": [
-2.6,
1.17
],
"P_rwald": [
-5,
1.17
],
"P_score": [
-2.5,
1.17
],
"P_lrt": [
-2.5,
1.17
],
"P_CL": [
-0.3,
1.16
],
"P_CLkm": [
-2.6,
1.21
],
"SLR": [
-13.6,
1.31
]
},
"size_adj_power_pct": {
"LR": 60.2,
"S_wald": 53.4,
"S_rwald": 53.1,
"S_score": 53.4,
"S_lrt": 53.4,
"S_CL": 57.4,
"S_CLkm": 57.4,
"P_wald": 54,
"P_rwald": 53.4,
"P_score": 54.1,
"P_lrt": 53.9,
"P_CL": 57.7,
"P_CLkm": 57.7,
"SLR": 46.9
},
"RE_cal": {
"S_wald": [
0.838,
0.81,
0.867
],
"S_rwald": [
0.817,
0.79,
0.846
],
"S_score": [
0.847,
0.818,
0.876
],
"S_lrt": [
0.847,
0.819,
0.877
],
"S_CL": [
0.895,
0.868,
0.923
],
"S_CLkm": [
0.895,
0.868,
0.923
],
"P_wald": [
0.843,
0.813,
0.875
],
"P_rwald": [
0.829,
0.798,
0.861
],
"P_score": [
0.852,
0.821,
0.884
],
"P_lrt": [
0.852,
0.821,
0.885
],
"P_CL": [
0.921,
0.89,
0.953
],
"P_CLkm": [
0.921,
0.89,
0.953
],
"SLR": [
0.704,
0.676,
0.733
]
},
"size_adj_power_pct_obs": {
"LR": 60.8,
"S_wald": 53.5,
"S_rwald": 52.6,
"S_score": 53.5,
"S_lrt": 53.5,
"S_CL": 57.2,
"S_CLkm": 57.2,
"P_wald": 55.1,
"P_rwald": 53.4,
"P_score": 55,
"P_lrt": 55,
"P_CL": 58.2,
"P_CLkm": 58.2,
"SLR": 47
},
"RE_cal_obs": {
"S_wald": [
0.839,
0.811,
0.869
],
"S_rwald": [
0.814,
0.787,
0.843
],
"S_score": [
0.848,
0.82,
0.878
],
"S_lrt": [
0.85,
0.821,
0.879
],
"S_CL": [
0.902,
0.875,
0.93
],
"S_CLkm": [
0.902,
0.875,
0.93
],
"P_wald": [
0.868,
0.835,
0.901
],
"P_rwald": [
0.835,
0.803,
0.867
],
"P_score": [
0.877,
0.844,
0.912
],
"P_lrt": [
0.878,
0.845,
0.912
],
"P_CL": [
0.938,
0.906,
0.972
],
"P_CLkm": [
0.938,
0.906,
0.972
],
"SLR": [
0.694,
0.666,
0.722
]
},
"size_adj_power_pct_trt": {
"LR": 62,
"S_wald": 53.7,
"S_rwald": 53.1,
"S_score": 53.6,
"S_lrt": 53.5,
"S_CL": 57.6,
"S_CLkm": 57.6,
"P_wald": 54.1,
"P_rwald": 53.8,
"P_score": 54.1,
"P_lrt": 54.1,
"P_CL": 58.1,
"P_CLkm": 58.1,
"SLR": 47.9
},
"RE_cal_trt": {
"S_wald": [
0.811,
0.784,
0.839
],
"S_rwald": [
0.79,
0.763,
0.817
],
"S_score": [
0.818,
0.791,
0.847
],
"S_lrt": [
0.818,
0.791,
0.847
],
"S_CL": [
0.88,
0.853,
0.907
],
"S_CLkm": [
0.88,
0.853,
0.907
],
"P_wald": [
0.811,
0.781,
0.842
],
"P_rwald": [
0.797,
0.767,
0.828
],
"P_score": [
0.818,
0.788,
0.85
],
"P_lrt": [
0.818,
0.788,
0.85
],
"P_CL": [
0.9,
0.869,
0.932
],
"P_CLkm": [
0.9,
0.869,
0.932
],
"SLR": [
0.69,
0.662,
0.719
]
},
"RE": {
"S_wald": [
0.882,
0.86,
0.905
],
"S_rwald": [
0.826,
0.805,
0.848
],
"S_score": [
0.898,
0.875,
0.921
],
"S_lrt": [
0.897,
0.875,
0.92
],
"S_CL": [
0.914,
0.893,
0.935
],
"S_CLkm": [
0.884,
0.863,
0.904
],
"P_wald": [
0.924,
0.899,
0.95
],
"P_rwald": [
0.859,
0.835,
0.883
],
"P_score": [
0.94,
0.914,
0.966
],
"P_lrt": [
0.939,
0.914,
0.965
],
"P_CL": [
0.977,
0.951,
1.003
],
"P_CLkm": [
0.912,
0.888,
0.936
],
"SLR": [
0.694,
0.672,
0.716
]
},
"decomp": {
"S": {
"est_ratio": 0.965,
"se_ratio": 1.017,
"empsd_ratio": 1.052
},
"P": {
"est_ratio": 1.006,
"se_ratio": 1.036,
"empsd_ratio": 1.092
}
},
"est": {
"U": {
"truth": -0.373,
"mean": -0.376,
"bias": -0.0037,
"emp_se": 0.1722,
"mod_se": 0.1715,
"rmse": 0.1722,
"cov_pct": 95.8,
"ratio_emp_mod": 1.005
},
"S": {
"truth": -0.366,
"mean": -0.363,
"bias": 0.0029,
"emp_se": 0.1811,
"mod_se": 0.1744,
"rmse": 0.1811,
"cov_pct": 95.1,
"ratio_emp_mod": 1.038,
"rob_se": 0.1803,
"rcov_pct": 95.7,
"ratio_emp_rob": 1.004
},
"P": {
"truth": -0.384,
"mean": -0.379,
"bias": 0.0049,
"emp_se": 0.1881,
"mod_se": 0.1776,
"rmse": 0.188,
"cov_pct": 94.2,
"ratio_emp_mod": 1.059,
"rob_se": 0.1845,
"rcov_pct": 95.3,
"ratio_emp_rob": 1.019
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
},
"315": {
"perm_obs": {
"R": 2000,
"events_mean": 168,
"epv_P": 16.8,
"rej_pct": {
"LR": 5.5,
"S_wald": 6.2,
"S_rwald": 5.7,
"S_score": 6.3,
"S_lrt": 6.3,
"S_CL": 5.5,
"S_CLkm": 5.2,
"P_wald": 6.4,
"P_rwald": 5.8,
"P_score": 6.5,
"P_lrt": 6.5,
"P_CL": 5.8,
"P_CLkm": 5.1,
"SLR": 5.9
},
"rej_mcse_pct": {
"LR": 0.51,
"S_wald": 0.54,
"S_rwald": 0.52,
"S_score": 0.55,
"S_lrt": 0.55,
"S_CL": 0.51,
"S_CLkm": 0.5,
"P_wald": 0.55,
"P_rwald": 0.52,
"P_score": 0.55,
"P_lrt": 0.55,
"P_CL": 0.52,
"P_CLkm": 0.49,
"SLR": 0.52
},
"mean_z": {
"LR": 0.02,
"S_wald": 0.022,
"S_rwald": 0.022,
"S_score": 0.022,
"S_lrt": 0.022,
"S_CL": 0.026,
"S_CLkm": 0.026,
"P_wald": 0.021,
"P_rwald": 0.021,
"P_score": 0.021,
"P_lrt": 0.021,
"P_CL": 0.029,
"P_CLkm": 0.028,
"SLR": 0.03
},
"var_z": {
"LR": 1.035,
"S_wald": 1.097,
"S_rwald": 1.049,
"S_score": 1.104,
"S_lrt": 1.102,
"S_CL": 1.04,
"S_CLkm": 1.007,
"P_wald": 1.121,
"P_rwald": 1.07,
"P_score": 1.128,
"P_lrt": 1.126,
"P_CL": 1.074,
"P_CLkm": 1.005,
"SLR": 1.024
},
"var_z_mcse": {
"LR": 0.031,
"S_wald": 0.035,
"S_rwald": 0.034,
"S_score": 0.036,
"S_lrt": 0.035,
"S_CL": 0.032,
"S_CLkm": 0.031,
"P_wald": 0.036,
"P_rwald": 0.034,
"P_score": 0.036,
"P_lrt": 0.036,
"P_CL": 0.034,
"P_CLkm": 0.031,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 12.1,
"P": 14.3
},
"rej_pct_converged_only": {
"LR": 5.5,
"S_wald": 6.2,
"S_rwald": 5.7,
"S_score": 6.3,
"S_lrt": 6.3,
"S_CL": 5.5,
"S_CLkm": 5.2,
"P_wald": 6.4,
"P_rwald": 5.8,
"P_score": 6.5,
"P_lrt": 6.5,
"P_CL": 5.8,
"P_CLkm": 5.1,
"SLR": 5.9
},
"est": {
"U": {
"truth": 0,
"mean": 0.003,
"bias": 0.0031,
"emp_se": 0.1573,
"mod_se": 0.1546,
"rmse": 0.1573,
"cov_pct": 94.5,
"ratio_emp_mod": 1.017
},
"S": {
"truth": 0,
"mean": 0.004,
"bias": 0.0035,
"emp_se": 0.1643,
"mod_se": 0.1563,
"rmse": 0.1642,
"cov_pct": 93.8,
"ratio_emp_mod": 1.051,
"rob_se": 0.1599,
"rcov_pct": 94.3,
"ratio_emp_rob": 1.027
},
"P": {
"truth": 0,
"mean": 0.003,
"bias": 0.0033,
"emp_se": 0.1685,
"mod_se": 0.1586,
"rmse": 0.1685,
"cov_pct": 93.6,
"ratio_emp_mod": 1.062,
"rob_se": 0.1624,
"rcov_pct": 94.2,
"ratio_emp_rob": 1.037
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
},
"600": {
"null_obs": {
"R": 5000,
"events_mean": 320.1,
"epv_P": 32,
"rej_pct": {
"LR": 5,
"S_wald": 5.8,
"S_rwald": 5.4,
"S_score": 5.9,
"S_lrt": 5.9,
"S_CL": 5.4,
"S_CLkm": 5.3,
"P_wald": 6.3,
"P_rwald": 5.8,
"P_score": 6.3,
"P_lrt": 6.3,
"P_CL": 5.7,
"P_CLkm": 5.3,
"SLR": 5.2
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.33,
"S_rwald": 0.32,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.32,
"S_CLkm": 0.32,
"P_wald": 0.34,
"P_rwald": 0.33,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.33,
"P_CLkm": 0.32,
"SLR": 0.31
},
"mean_z": {
"LR": -0.009,
"S_wald": 0.008,
"S_rwald": 0.008,
"S_score": 0.008,
"S_lrt": 0.008,
"S_CL": 0.004,
"S_CLkm": 0.004,
"P_wald": 0.012,
"P_rwald": 0.012,
"P_score": 0.012,
"P_lrt": 0.012,
"P_CL": 0.009,
"P_CLkm": 0.009,
"SLR": 0.008
},
"var_z": {
"LR": 0.986,
"S_wald": 1.049,
"S_rwald": 1.009,
"S_score": 1.053,
"S_lrt": 1.052,
"S_CL": 1.003,
"S_CLkm": 0.986,
"P_wald": 1.063,
"P_rwald": 1.025,
"P_score": 1.067,
"P_lrt": 1.065,
"P_CL": 1.024,
"P_CLkm": 0.99,
"SLR": 1.006
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.022,
"S_rwald": 0.021,
"S_score": 0.022,
"S_lrt": 0.022,
"S_CL": 0.021,
"S_CLkm": 0.02,
"P_wald": 0.022,
"P_rwald": 0.021,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.021,
"P_CLkm": 0.021,
"SLR": 0.021
},
"cl_var_reduction_pct": {
"S": 12.6,
"P": 15.4
},
"rej_pct_converged_only": {
"LR": 5,
"S_wald": 5.8,
"S_rwald": 5.4,
"S_score": 5.9,
"S_lrt": 5.9,
"S_CL": 5.4,
"S_CLkm": 5.3,
"P_wald": 6.3,
"P_rwald": 5.8,
"P_score": 6.3,
"P_lrt": 6.3,
"P_CL": 5.7,
"P_CLkm": 5.3,
"SLR": 5.2
},
"null_crit_abs_z": {
"LR": 1.956,
"S_wald": 2.033,
"S_rwald": 1.993,
"S_score": 2.037,
"S_lrt": 2.034,
"S_CL": 2.001,
"S_CLkm": 1.984,
"P_wald": 2.071,
"P_rwald": 2.034,
"P_score": 2.076,
"P_lrt": 2.075,
"P_CL": 2.026,
"P_CLkm": 1.992,
"SLR": 1.978
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.001,
"emp_se": 0.1109,
"mod_se": 0.112,
"rmse": 0.1109,
"cov_pct": 95.1,
"ratio_emp_mod": 0.991
},
"S": {
"truth": 0,
"mean": 0.001,
"bias": 0.0009,
"emp_se": 0.1155,
"mod_se": 0.1126,
"rmse": 0.1155,
"cov_pct": 94.2,
"ratio_emp_mod": 1.026,
"rob_se": 0.1148,
"rcov_pct": 94.6,
"ratio_emp_rob": 1.007
},
"P": {
"truth": 0,
"mean": 0.001,
"bias": 0.0014,
"emp_se": 0.1171,
"mod_se": 0.1134,
"rmse": 0.1171,
"cov_pct": 93.7,
"ratio_emp_mod": 1.033,
"rob_se": 0.1154,
"rcov_pct": 94.2,
"ratio_emp_rob": 1.015
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"null_trt": {
"R": 5000,
"events_mean": 242.8,
"epv_P": 24.3,
"rej_pct": {
"LR": 5.6,
"S_wald": 6.3,
"S_rwald": 5.8,
"S_score": 6.3,
"S_lrt": 6.3,
"S_CL": 6,
"S_CLkm": 5.9,
"P_wald": 6.6,
"P_rwald": 5.8,
"P_score": 6.7,
"P_lrt": 6.7,
"P_CL": 5.8,
"P_CLkm": 5.5,
"SLR": 5.9
},
"rej_mcse_pct": {
"LR": 0.33,
"S_wald": 0.34,
"S_rwald": 0.33,
"S_score": 0.34,
"S_lrt": 0.34,
"S_CL": 0.34,
"S_CLkm": 0.33,
"P_wald": 0.35,
"P_rwald": 0.33,
"P_score": 0.35,
"P_lrt": 0.35,
"P_CL": 0.33,
"P_CLkm": 0.32,
"SLR": 0.33
},
"mean_z": {
"LR": 0.006,
"S_wald": 0.001,
"S_rwald": 0.001,
"S_score": 0.001,
"S_lrt": 0.001,
"S_CL": 0.002,
"S_CLkm": 0.002,
"P_wald": -0.001,
"P_rwald": -0.001,
"P_score": -0.001,
"P_lrt": -0.001,
"P_CL": -0.003,
"P_CLkm": -0.003,
"SLR": 0.003
},
"var_z": {
"LR": 1.034,
"S_wald": 1.107,
"S_rwald": 1.06,
"S_score": 1.112,
"S_lrt": 1.111,
"S_CL": 1.058,
"S_CLkm": 1.04,
"P_wald": 1.145,
"P_rwald": 1.08,
"P_score": 1.15,
"P_lrt": 1.15,
"P_CL": 1.083,
"P_CLkm": 1.047,
"SLR": 1.042
},
"var_z_mcse": {
"LR": 0.021,
"S_wald": 0.022,
"S_rwald": 0.021,
"S_score": 0.022,
"S_lrt": 0.022,
"S_CL": 0.021,
"S_CLkm": 0.02,
"P_wald": 0.022,
"P_rwald": 0.021,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 11,
"P": 13.8
},
"rej_pct_converged_only": {
"LR": 5.6,
"S_wald": 6.3,
"S_rwald": 5.8,
"S_score": 6.3,
"S_lrt": 6.3,
"S_CL": 6,
"S_CLkm": 5.9,
"P_wald": 6.6,
"P_rwald": 5.8,
"P_score": 6.7,
"P_lrt": 6.7,
"P_CL": 5.8,
"P_CLkm": 5.5,
"SLR": 5.9
},
"null_crit_abs_z": {
"LR": 2.003,
"S_wald": 2.063,
"S_rwald": 2.029,
"S_score": 2.069,
"S_lrt": 2.068,
"S_CL": 2.035,
"S_CLkm": 2.018,
"P_wald": 2.092,
"P_rwald": 2.027,
"P_score": 2.098,
"P_lrt": 2.096,
"P_CL": 2.035,
"P_CLkm": 2,
"SLR": 2.024
},
"est": {
"U": {
"truth": 0,
"mean": 0.001,
"bias": 0.0008,
"emp_se": 0.1306,
"mod_se": 0.1287,
"rmse": 0.1306,
"cov_pct": 94.5,
"ratio_emp_mod": 1.015
},
"S": {
"truth": 0,
"mean": 0,
"bias": 0.0002,
"emp_se": 0.1366,
"mod_se": 0.1295,
"rmse": 0.1366,
"cov_pct": 93.7,
"ratio_emp_mod": 1.055,
"rob_se": 0.1322,
"rcov_pct": 94.2,
"ratio_emp_rob": 1.033
},
"P": {
"truth": 0,
"mean": 0,
"bias": -0.0001,
"emp_se": 0.1401,
"mod_se": 0.1305,
"rmse": 0.1401,
"cov_pct": 93.4,
"ratio_emp_mod": 1.074,
"rob_se": 0.1343,
"rcov_pct": 94.2,
"ratio_emp_rob": 1.043
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"null": {
"R": 5000,
"events_mean": 282.2,
"epv_P": 28.2,
"rej_pct": {
"LR": 4.7,
"S_wald": 5.7,
"S_rwald": 5,
"S_score": 5.8,
"S_lrt": 5.7,
"S_CL": 4.9,
"S_CLkm": 4.6,
"P_wald": 5.9,
"P_rwald": 5.1,
"P_score": 5.9,
"P_lrt": 5.9,
"P_CL": 5.1,
"P_CLkm": 4.6,
"SLR": 4.6
},
"rej_mcse_pct": {
"LR": 0.3,
"S_wald": 0.33,
"S_rwald": 0.31,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.31,
"S_CLkm": 0.3,
"P_wald": 0.33,
"P_rwald": 0.31,
"P_score": 0.33,
"P_lrt": 0.33,
"P_CL": 0.31,
"P_CLkm": 0.3,
"SLR": 0.3
},
"mean_z": {
"LR": 0,
"S_wald": 0.004,
"S_rwald": 0.004,
"S_score": 0.004,
"S_lrt": 0.004,
"S_CL": 0.003,
"S_CLkm": 0.003,
"P_wald": 0.009,
"P_rwald": 0.009,
"P_score": 0.009,
"P_lrt": 0.009,
"P_CL": 0.008,
"P_CLkm": 0.008,
"SLR": 0.005
},
"var_z": {
"LR": 1.002,
"S_wald": 1.065,
"S_rwald": 1.017,
"S_score": 1.069,
"S_lrt": 1.068,
"S_CL": 1.013,
"S_CLkm": 0.996,
"P_wald": 1.101,
"P_rwald": 1.034,
"P_score": 1.105,
"P_lrt": 1.104,
"P_CL": 1.034,
"P_CLkm": 1,
"SLR": 0.994
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.02,
"P_wald": 0.022,
"P_rwald": 0.021,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 11.8,
"P": 13.8
},
"rej_pct_converged_only": {
"LR": 4.7,
"S_wald": 5.7,
"S_rwald": 5,
"S_score": 5.8,
"S_lrt": 5.7,
"S_CL": 4.9,
"S_CLkm": 4.6,
"P_wald": 5.9,
"P_rwald": 5.1,
"P_score": 5.9,
"P_lrt": 5.9,
"P_CL": 5.1,
"P_CLkm": 4.6,
"SLR": 4.6
},
"null_crit_abs_z": {
"LR": 1.941,
"S_wald": 2.009,
"S_rwald": 1.957,
"S_score": 2.014,
"S_lrt": 2.016,
"S_CL": 1.953,
"S_CLkm": 1.936,
"P_wald": 2.028,
"P_rwald": 1.978,
"P_score": 2.033,
"P_lrt": 2.032,
"P_CL": 1.966,
"P_CLkm": 1.933,
"SLR": 1.928
},
"est": {
"U": {
"truth": 0,
"mean": 0,
"bias": 0,
"emp_se": 0.1192,
"mod_se": 0.1193,
"rmse": 0.1192,
"cov_pct": 95.3,
"ratio_emp_mod": 1
},
"S": {
"truth": 0,
"mean": 0,
"bias": 0.0005,
"emp_se": 0.1241,
"mod_se": 0.12,
"rmse": 0.1241,
"cov_pct": 94.3,
"ratio_emp_mod": 1.034,
"rob_se": 0.1228,
"rcov_pct": 95,
"ratio_emp_rob": 1.01
},
"P": {
"truth": 0,
"mean": 0.001,
"bias": 0.0011,
"emp_se": 0.1271,
"mod_se": 0.1209,
"rmse": 0.1271,
"cov_pct": 94.1,
"ratio_emp_mod": 1.051,
"rob_se": 0.1248,
"rcov_pct": 94.9,
"ratio_emp_rob": 1.019
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"balanced": {
"R": 2000,
"events_mean": 279.9,
"epv_P": 28,
"rej_pct": {
"LR": 81.9,
"S_wald": 87.2,
"S_rwald": 85.7,
"S_score": 87.4,
"S_lrt": 87.4,
"S_CL": 86.2,
"S_CLkm": 86,
"P_wald": 86.7,
"P_rwald": 85.3,
"P_score": 86.8,
"P_lrt": 86.8,
"P_CL": 86.8,
"P_CLkm": 86,
"SLR": 76.4
},
"rej_mcse_pct": {
"LR": 0.86,
"S_wald": 0.75,
"S_rwald": 0.78,
"S_score": 0.74,
"S_lrt": 0.74,
"S_CL": 0.77,
"S_CLkm": 0.78,
"P_wald": 0.76,
"P_rwald": 0.79,
"P_score": 0.76,
"P_lrt": 0.76,
"P_CL": 0.76,
"P_CLkm": 0.78,
"SLR": 0.95
},
"mean_z": {
"LR": -2.852,
"S_wald": -3.059,
"S_rwald": -2.981,
"S_score": -3.081,
"S_lrt": -3.079,
"S_CL": -3.027,
"S_CLkm": -3.002,
"P_wald": -3.082,
"P_rwald": -2.989,
"P_score": -3.105,
"P_lrt": -3.104,
"P_CL": -3.055,
"P_CLkm": -3.004,
"SLR": -2.678
},
"var_z": {
"LR": 0.97,
"S_wald": 1.013,
"S_rwald": 0.96,
"S_score": 1.051,
"S_lrt": 1.047,
"S_CL": 1,
"S_CLkm": 0.983,
"P_wald": 1.045,
"P_rwald": 0.987,
"P_score": 1.084,
"P_lrt": 1.081,
"P_CL": 1.022,
"P_CLkm": 0.988,
"SLR": 1.008
},
"var_z_mcse": {
"LR": 0.032,
"S_wald": 0.032,
"S_rwald": 0.031,
"S_score": 0.034,
"S_lrt": 0.034,
"S_CL": 0.032,
"S_CLkm": 0.032,
"P_wald": 0.033,
"P_rwald": 0.031,
"P_score": 0.035,
"P_lrt": 0.034,
"P_CL": 0.033,
"P_CLkm": 0.032,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 11.4,
"P": 13.2
},
"rej_pct_converged_only": {
"LR": 81.9,
"S_wald": 87.2,
"S_rwald": 85.7,
"S_score": 87.4,
"S_lrt": 87.4,
"S_CL": 86.2,
"S_CLkm": 86,
"P_wald": 86.7,
"P_rwald": 85.3,
"P_score": 86.8,
"P_lrt": 86.8,
"P_CL": 86.8,
"P_CLkm": 86,
"SLR": 76.4
},
"diff_vs_LR_pct": {
"S_wald": [
5.3,
0.65
],
"S_rwald": [
3.8,
0.64
],
"S_score": [
5.5,
0.66
],
"S_lrt": [
5.5,
0.66
],
"S_CL": [
4.2,
0.6
],
"S_CLkm": [
4.1,
0.61
],
"P_wald": [
4.8,
0.67
],
"P_rwald": [
3.4,
0.67
],
"P_score": [
4.9,
0.67
],
"P_lrt": [
4.9,
0.67
],
"P_CL": [
4.9,
0.63
],
"P_CLkm": [
4,
0.63
],
"SLR": [
-5.5,
0.74
]
},
"size_adj_power_pct": {
"LR": 82.4,
"S_wald": 85.5,
"S_rwald": 85.9,
"S_score": 85.5,
"S_lrt": 85.5,
"S_CL": 86.3,
"S_CLkm": 86.3,
"P_wald": 85.2,
"P_rwald": 84.9,
"P_score": 85.2,
"P_lrt": 85.2,
"P_CL": 86.7,
"P_CLkm": 86.7,
"SLR": 77.7
},
"RE_cal": {
"S_wald": [
1.083,
1.056,
1.11
],
"S_rwald": [
1.078,
1.051,
1.105
],
"S_score": [
1.094,
1.067,
1.122
],
"S_lrt": [
1.094,
1.067,
1.122
],
"S_CL": [
1.115,
1.09,
1.14
],
"S_CLkm": [
1.115,
1.09,
1.14
],
"P_wald": [
1.063,
1.035,
1.093
],
"P_rwald": [
1.065,
1.036,
1.095
],
"P_score": [
1.075,
1.046,
1.105
],
"P_lrt": [
1.075,
1.046,
1.105
],
"P_CL": [
1.113,
1.086,
1.14
],
"P_CLkm": [
1.113,
1.086,
1.14
],
"SLR": [
0.89,
0.866,
0.914
]
},
"size_adj_power_pct_obs": {
"LR": 82,
"S_wald": 85.1,
"S_rwald": 84.9,
"S_score": 85.1,
"S_lrt": 85.2,
"S_CL": 85.4,
"S_CLkm": 85.4,
"P_wald": 84.2,
"P_rwald": 83.5,
"P_score": 84.2,
"P_lrt": 84.2,
"P_CL": 85.2,
"P_CLkm": 85.2,
"SLR": 75.9
},
"RE_cal_obs": {
"S_wald": [
1.082,
1.054,
1.11
],
"S_rwald": [
1.069,
1.042,
1.097
],
"S_score": [
1.094,
1.066,
1.123
],
"S_lrt": [
1.094,
1.066,
1.123
],
"S_CL": [
1.108,
1.083,
1.133
],
"S_CLkm": [
1.108,
1.083,
1.133
],
"P_wald": [
1.084,
1.054,
1.116
],
"P_rwald": [
1.058,
1.028,
1.089
],
"P_score": [
1.096,
1.065,
1.128
],
"P_lrt": [
1.097,
1.066,
1.129
],
"P_CL": [
1.105,
1.078,
1.133
],
"P_CLkm": [
1.105,
1.078,
1.133
],
"SLR": [
0.865,
0.841,
0.89
]
},
"size_adj_power_pct_trt": {
"LR": 81.5,
"S_wald": 84.4,
"S_rwald": 83.7,
"S_score": 84.2,
"S_lrt": 84.4,
"S_CL": 84.6,
"S_CLkm": 84.6,
"P_wald": 83.7,
"P_rwald": 83.6,
"P_score": 83.7,
"P_lrt": 83.7,
"P_CL": 84.9,
"P_CLkm": 84.9,
"SLR": 74.7
},
"RE_cal_trt": {
"S_wald": [
1.075,
1.049,
1.101
],
"S_rwald": [
1.067,
1.041,
1.093
],
"S_score": [
1.086,
1.06,
1.113
],
"S_lrt": [
1.085,
1.06,
1.112
],
"S_CL": [
1.102,
1.079,
1.126
],
"S_CLkm": [
1.102,
1.079,
1.126
],
"P_wald": [
1.055,
1.027,
1.084
],
"P_rwald": [
1.052,
1.024,
1.081
],
"P_score": [
1.066,
1.038,
1.095
],
"P_lrt": [
1.066,
1.038,
1.094
],
"P_CL": [
1.096,
1.071,
1.122
],
"P_CLkm": [
1.096,
1.071,
1.122
],
"SLR": [
0.876,
0.852,
0.901
]
},
"RE": {
"S_wald": [
1.151,
1.137,
1.164
],
"S_rwald": [
1.093,
1.08,
1.106
],
"S_score": [
1.168,
1.154,
1.182
],
"S_lrt": [
1.166,
1.152,
1.18
],
"S_CL": [
1.127,
1.115,
1.139
],
"S_CLkm": [
1.108,
1.097,
1.12
],
"P_wald": [
1.169,
1.154,
1.183
],
"P_rwald": [
1.099,
1.085,
1.113
],
"P_score": [
1.186,
1.171,
1.201
],
"P_lrt": [
1.185,
1.17,
1.2
],
"P_CL": [
1.148,
1.135,
1.161
],
"P_CLkm": [
1.11,
1.097,
1.123
],
"SLR": [
0.882,
0.869,
0.895
]
},
"decomp": {
"S": {
"est_ratio": 1.091,
"se_ratio": 1.009,
"empsd_ratio": 1.052
},
"P": {
"est_ratio": 1.11,
"se_ratio": 1.018,
"empsd_ratio": 1.08
}
},
"est": {
"U": {
"truth": -0.345,
"mean": -0.343,
"bias": 0.0022,
"emp_se": 0.1197,
"mod_se": 0.1209,
"rmse": 0.1197,
"cov_pct": 95.2,
"ratio_emp_mod": 0.99
},
"S": {
"truth": -0.376,
"mean": -0.374,
"bias": 0.0025,
"emp_se": 0.1259,
"mod_se": 0.1219,
"rmse": 0.1259,
"cov_pct": 93.7,
"ratio_emp_mod": 1.032,
"rob_se": 0.1251,
"rcov_pct": 94.2,
"ratio_emp_rob": 1.006
},
"P": {
"truth": -0.384,
"mean": -0.38,
"bias": 0.0037,
"emp_se": 0.1292,
"mod_se": 0.1231,
"rmse": 0.1292,
"cov_pct": 93.5,
"ratio_emp_mod": 1.05,
"rob_se": 0.127,
"rcov_pct": 94,
"ratio_emp_rob": 1.018
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"trial": {
"R": 1000,
"events_mean": 281,
"epv_P": 28.1,
"rej_pct": {
"LR": 87.5,
"S_wald": 83.4,
"S_rwald": 81.8,
"S_score": 83.5,
"S_lrt": 83.7,
"S_CL": 83.1,
"S_CLkm": 83,
"P_wald": 84.7,
"P_rwald": 82.9,
"P_score": 84.8,
"P_lrt": 84.8,
"P_CL": 84.7,
"P_CLkm": 84,
"SLR": 73
},
"rej_mcse_pct": {
"LR": 1.05,
"S_wald": 1.18,
"S_rwald": 1.22,
"S_score": 1.17,
"S_lrt": 1.17,
"S_CL": 1.19,
"S_CLkm": 1.19,
"P_wald": 1.14,
"P_rwald": 1.19,
"P_score": 1.14,
"P_lrt": 1.14,
"P_CL": 1.14,
"P_CLkm": 1.16,
"SLR": 1.4
},
"mean_z": {
"LR": -3.077,
"S_wald": -2.934,
"S_rwald": -2.835,
"S_score": -2.955,
"S_lrt": -2.955,
"S_CL": -2.942,
"S_CLkm": -2.917,
"P_wald": -3.027,
"P_rwald": -2.916,
"P_score": -3.048,
"P_lrt": -3.048,
"P_CL": -3.017,
"P_CLkm": -2.966,
"SLR": -2.576
},
"var_z": {
"LR": 1.029,
"S_wald": 1.085,
"S_rwald": 1.018,
"S_score": 1.123,
"S_lrt": 1.121,
"S_CL": 1.069,
"S_CLkm": 1.051,
"P_wald": 1.127,
"P_rwald": 1.057,
"P_score": 1.168,
"P_lrt": 1.165,
"P_CL": 1.107,
"P_CLkm": 1.07,
"SLR": 1.063
},
"var_z_mcse": {
"LR": 0.05,
"S_wald": 0.052,
"S_rwald": 0.049,
"S_score": 0.054,
"S_lrt": 0.054,
"S_CL": 0.052,
"S_CLkm": 0.051,
"P_wald": 0.054,
"P_rwald": 0.051,
"P_score": 0.056,
"P_lrt": 0.056,
"P_CL": 0.054,
"P_CLkm": 0.052,
"SLR": 0.051
},
"cl_var_reduction_pct": {
"S": 11.6,
"P": 13.6
},
"rej_pct_converged_only": {
"LR": 87.5,
"S_wald": 83.4,
"S_rwald": 81.8,
"S_score": 83.5,
"S_lrt": 83.7,
"S_CL": 83.1,
"S_CLkm": 83,
"P_wald": 84.7,
"P_rwald": 82.9,
"P_score": 84.8,
"P_lrt": 84.8,
"P_CL": 84.7,
"P_CLkm": 84,
"SLR": 73
},
"diff_vs_LR_pct": {
"S_wald": [
-4.1,
0.82
],
"S_rwald": [
-5.7,
0.88
],
"S_score": [
-4,
0.82
],
"S_lrt": [
-3.8,
0.8
],
"S_CL": [
-4.4,
0.8
],
"S_CLkm": [
-4.5,
0.79
],
"P_wald": [
-2.8,
0.87
],
"P_rwald": [
-4.6,
0.89
],
"P_score": [
-2.7,
0.86
],
"P_lrt": [
-2.7,
0.86
],
"P_CL": [
-2.8,
0.83
],
"P_CLkm": [
-3.5,
0.85
],
"SLR": [
-14.5,
1.14
]
},
"size_adj_power_pct": {
"LR": 87.8,
"S_wald": 82.2,
"S_rwald": 81.8,
"S_score": 82.2,
"S_lrt": 82.2,
"S_CL": 83.8,
"S_CLkm": 83.8,
"P_wald": 83.1,
"P_rwald": 82.3,
"P_score": 83.1,
"P_lrt": 83.1,
"P_CL": 84.5,
"P_CLkm": 84.5,
"SLR": 74.3
},
"RE_cal": {
"S_wald": [
0.855,
0.831,
0.88
],
"S_rwald": [
0.837,
0.813,
0.861
],
"S_score": [
0.864,
0.84,
0.889
],
"S_lrt": [
0.865,
0.841,
0.89
],
"S_CL": [
0.904,
0.881,
0.927
],
"S_CLkm": [
0.904,
0.881,
0.927
],
"P_wald": [
0.88,
0.854,
0.908
],
"P_rwald": [
0.87,
0.844,
0.898
],
"P_score": [
0.89,
0.863,
0.917
],
"P_lrt": [
0.89,
0.863,
0.918
],
"P_CL": [
0.931,
0.907,
0.957
],
"P_CLkm": [
0.931,
0.907,
0.957
],
"SLR": [
0.707,
0.684,
0.73
]
},
"size_adj_power_pct_obs": {
"LR": 87.5,
"S_wald": 81.9,
"S_rwald": 80.6,
"S_score": 81.9,
"S_lrt": 81.9,
"S_CL": 82.5,
"S_CLkm": 82.5,
"P_wald": 81.7,
"P_rwald": 81.1,
"P_score": 81.7,
"P_lrt": 81.8,
"P_CL": 83,
"P_CLkm": 83,
"SLR": 72.5
},
"RE_cal_obs": {
"S_wald": [
0.855,
0.83,
0.88
],
"S_rwald": [
0.83,
0.806,
0.854
],
"S_score": [
0.864,
0.839,
0.89
],
"S_lrt": [
0.865,
0.84,
0.89
],
"S_CL": [
0.898,
0.876,
0.921
],
"S_CLkm": [
0.898,
0.876,
0.921
],
"P_wald": [
0.898,
0.869,
0.927
],
"P_rwald": [
0.864,
0.837,
0.893
],
"P_score": [
0.907,
0.879,
0.937
],
"P_lrt": [
0.908,
0.88,
0.937
],
"P_CL": [
0.926,
0.9,
0.952
],
"P_CLkm": [
0.926,
0.9,
0.952
],
"SLR": [
0.687,
0.665,
0.711
]
},
"size_adj_power_pct_trt": {
"LR": 86.1,
"S_wald": 80.8,
"S_rwald": 79.9,
"S_score": 80.8,
"S_lrt": 80.8,
"S_CL": 81.9,
"S_CLkm": 81.9,
"P_wald": 81.5,
"P_rwald": 81.2,
"P_score": 81.5,
"P_lrt": 81.5,
"P_CL": 82.8,
"P_CLkm": 82.8,
"SLR": 71.1
},
"RE_cal_trt": {
"S_wald": [
0.85,
0.827,
0.873
],
"S_rwald": [
0.828,
0.805,
0.851
],
"S_score": [
0.858,
0.834,
0.882
],
"S_lrt": [
0.858,
0.835,
0.882
],
"S_CL": [
0.894,
0.872,
0.916
],
"S_CLkm": [
0.894,
0.872,
0.916
],
"P_wald": [
0.874,
0.848,
0.9
],
"P_rwald": [
0.86,
0.834,
0.886
],
"P_score": [
0.882,
0.856,
0.909
],
"P_lrt": [
0.882,
0.856,
0.909
],
"P_CL": [
0.918,
0.893,
0.943
],
"P_CLkm": [
0.918,
0.893,
0.943
],
"SLR": [
0.696,
0.673,
0.719
]
},
"RE": {
"S_wald": [
0.909,
0.893,
0.925
],
"S_rwald": [
0.848,
0.833,
0.864
],
"S_score": [
0.922,
0.906,
0.939
],
"S_lrt": [
0.922,
0.906,
0.938
],
"S_CL": [
0.914,
0.899,
0.928
],
"S_CLkm": [
0.899,
0.884,
0.913
],
"P_wald": [
0.967,
0.949,
0.986
],
"P_rwald": [
0.898,
0.881,
0.916
],
"P_score": [
0.981,
0.963,
1
],
"P_lrt": [
0.981,
0.963,
1
],
"P_CL": [
0.961,
0.945,
0.978
],
"P_CLkm": [
0.929,
0.913,
0.945
],
"SLR": [
0.701,
0.685,
0.717
]
},
"decomp": {
"S": {
"est_ratio": 0.972,
"se_ratio": 1.01,
"empsd_ratio": 1.055
},
"P": {
"est_ratio": 1.014,
"se_ratio": 1.022,
"empsd_ratio": 1.089
}
},
"est": {
"U": {
"truth": -0.373,
"mean": -0.369,
"bias": 0.0036,
"emp_se": 0.1228,
"mod_se": 0.1208,
"rmse": 0.1228,
"cov_pct": 95.2,
"ratio_emp_mod": 1.017
},
"S": {
"truth": -0.366,
"mean": -0.359,
"bias": 0.0075,
"emp_se": 0.1296,
"mod_se": 0.1221,
"rmse": 0.1298,
"cov_pct": 93.3,
"ratio_emp_mod": 1.062,
"rob_se": 0.1264,
"rcov_pct": 94,
"ratio_emp_rob": 1.025
},
"P": {
"truth": -0.384,
"mean": -0.374,
"bias": 0.0094,
"emp_se": 0.1337,
"mod_se": 0.1234,
"rmse": 0.134,
"cov_pct": 92.8,
"ratio_emp_mod": 1.084,
"rob_se": 0.1282,
"rcov_pct": 93.8,
"ratio_emp_rob": 1.043
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
},
"619": {
"perm": {
"R": 2000,
"events_mean": 291,
"epv_P": 29.1,
"rej_pct": {
"LR": 4.5,
"S_wald": 5.5,
"S_rwald": 5.3,
"S_score": 5.5,
"S_lrt": 5.5,
"S_CL": 4.7,
"S_CLkm": 4.6,
"P_wald": 5.9,
"P_rwald": 5.5,
"P_score": 5.9,
"P_lrt": 5.9,
"P_CL": 4.9,
"P_CLkm": 4.5,
"SLR": 5
},
"rej_mcse_pct": {
"LR": 0.46,
"S_wald": 0.51,
"S_rwald": 0.5,
"S_score": 0.51,
"S_lrt": 0.51,
"S_CL": 0.47,
"S_CLkm": 0.47,
"P_wald": 0.53,
"P_rwald": 0.51,
"P_score": 0.53,
"P_lrt": 0.53,
"P_CL": 0.48,
"P_CLkm": 0.47,
"SLR": 0.49
},
"mean_z": {
"LR": -0.023,
"S_wald": -0.026,
"S_rwald": -0.025,
"S_score": -0.026,
"S_lrt": -0.026,
"S_CL": -0.026,
"S_CLkm": -0.026,
"P_wald": -0.028,
"P_rwald": -0.027,
"P_score": -0.028,
"P_lrt": -0.028,
"P_CL": -0.025,
"P_CLkm": -0.025,
"SLR": -0.027
},
"var_z": {
"LR": 1.007,
"S_wald": 1.084,
"S_rwald": 1.031,
"S_score": 1.088,
"S_lrt": 1.087,
"S_CL": 1.017,
"S_CLkm": 1.001,
"P_wald": 1.114,
"P_rwald": 1.04,
"P_score": 1.118,
"P_lrt": 1.117,
"P_CL": 1.033,
"P_CLkm": 0.999,
"SLR": 1.018
},
"var_z_mcse": {
"LR": 0.031,
"S_wald": 0.035,
"S_rwald": 0.033,
"S_score": 0.035,
"S_lrt": 0.035,
"S_CL": 0.033,
"S_CLkm": 0.032,
"P_wald": 0.036,
"P_rwald": 0.034,
"P_score": 0.036,
"P_lrt": 0.036,
"P_CL": 0.033,
"P_CLkm": 0.032,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 11.2,
"P": 12.4
},
"rej_pct_converged_only": {
"LR": 4.5,
"S_wald": 5.5,
"S_rwald": 5.3,
"S_score": 5.5,
"S_lrt": 5.5,
"S_CL": 4.7,
"S_CLkm": 4.6,
"P_wald": 5.9,
"P_rwald": 5.5,
"P_score": 5.9,
"P_lrt": 5.9,
"P_CL": 4.9,
"P_CLkm": 4.5,
"SLR": 5
},
"est": {
"U": {
"truth": 0,
"mean": -0.003,
"bias": -0.0028,
"emp_se": 0.1178,
"mod_se": 0.1174,
"rmse": 0.1178,
"cov_pct": 95.5,
"ratio_emp_mod": 1.003
},
"S": {
"truth": 0,
"mean": -0.003,
"bias": -0.0032,
"emp_se": 0.1232,
"mod_se": 0.1181,
"rmse": 0.1232,
"cov_pct": 94.5,
"ratio_emp_mod": 1.044,
"rob_se": 0.1211,
"rcov_pct": 94.7,
"ratio_emp_rob": 1.018
},
"P": {
"truth": 0,
"mean": -0.003,
"bias": -0.0034,
"emp_se": 0.1259,
"mod_se": 0.119,
"rmse": 0.1259,
"cov_pct": 94.1,
"ratio_emp_mod": 1.058,
"rob_se": 0.1231,
"rcov_pct": 94.5,
"ratio_emp_rob": 1.022
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
}
},
"RFS": {
"150": {
"null_obs": {
"R": 5000,
"events_mean": 90.4,
"epv_P": 9,
"rej_pct": {
"LR": 5.2,
"S_wald": 5.6,
"S_rwald": 5.6,
"S_score": 5.8,
"S_lrt": 5.7,
"S_CL": 5.7,
"S_CLkm": 4.9,
"P_wald": 6.9,
"P_rwald": 6.6,
"P_score": 7.1,
"P_lrt": 7,
"P_CL": 6.8,
"P_CLkm": 5,
"SLR": 4.9
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.32,
"S_rwald": 0.33,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.33,
"S_CLkm": 0.3,
"P_wald": 0.36,
"P_rwald": 0.35,
"P_score": 0.36,
"P_lrt": 0.36,
"P_CL": 0.36,
"P_CLkm": 0.31,
"SLR": 0.3
},
"mean_z": {
"LR": -0.004,
"S_wald": 0.001,
"S_rwald": 0.001,
"S_score": 0.001,
"S_lrt": 0.001,
"S_CL": -0.001,
"S_CLkm": -0.001,
"P_wald": -0.001,
"P_rwald": -0.001,
"P_score": -0.001,
"P_lrt": -0.001,
"P_CL": -0.005,
"P_CLkm": -0.005,
"SLR": 0.002
},
"var_z": {
"LR": 1.005,
"S_wald": 1.061,
"S_rwald": 1.06,
"S_score": 1.073,
"S_lrt": 1.069,
"S_CL": 1.055,
"S_CLkm": 0.984,
"P_wald": 1.141,
"P_rwald": 1.134,
"P_score": 1.153,
"P_lrt": 1.149,
"P_CL": 1.167,
"P_CLkm": 1.011,
"SLR": 1.016
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.021,
"S_score": 0.022,
"S_lrt": 0.021,
"S_CL": 0.021,
"S_CLkm": 0.019,
"P_wald": 0.023,
"P_rwald": 0.023,
"P_score": 0.023,
"P_lrt": 0.023,
"P_CL": 0.023,
"P_CLkm": 0.02,
"SLR": 0.021
},
"cl_var_reduction_pct": {
"S": 15.2,
"P": 19.9
},
"rej_pct_converged_only": {
"LR": 5.2,
"S_wald": 5.6,
"S_rwald": 5.6,
"S_score": 5.8,
"S_lrt": 5.7,
"S_CL": 5.7,
"S_CLkm": 4.9,
"P_wald": 6.9,
"P_rwald": 6.6,
"P_score": 7.1,
"P_lrt": 7,
"P_CL": 6.8,
"P_CLkm": 5,
"SLR": 4.9
},
"null_crit_abs_z": {
"LR": 1.974,
"S_wald": 2.011,
"S_rwald": 2.005,
"S_score": 2.026,
"S_lrt": 2.021,
"S_CL": 2.014,
"S_CLkm": 1.945,
"P_wald": 2.072,
"P_rwald": 2.081,
"P_score": 2.087,
"P_lrt": 2.08,
"P_CL": 2.106,
"P_CLkm": 1.96,
"SLR": 1.944
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.0007,
"emp_se": 0.2112,
"mod_se": 0.2115,
"rmse": 0.2111,
"cov_pct": 95.1,
"ratio_emp_mod": 0.999
},
"S": {
"truth": 0,
"mean": 0,
"bias": 0.0002,
"emp_se": 0.2241,
"mod_se": 0.2164,
"rmse": 0.2241,
"cov_pct": 94.4,
"ratio_emp_mod": 1.036,
"rob_se": 0.2168,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.034
},
"P": {
"truth": 0,
"mean": 0,
"bias": -0.0003,
"emp_se": 0.2396,
"mod_se": 0.223,
"rmse": 0.2395,
"cov_pct": 93.1,
"ratio_emp_mod": 1.074,
"rob_se": 0.2241,
"rcov_pct": 93.4,
"ratio_emp_rob": 1.069
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 7
},
"null_trt": {
"R": 5000,
"events_mean": 66.1,
"epv_P": 6.6,
"rej_pct": {
"LR": 5,
"S_wald": 6.4,
"S_rwald": 5.9,
"S_score": 6.6,
"S_lrt": 6.7,
"S_CL": 6,
"S_CLkm": 5.1,
"P_wald": 7.3,
"P_rwald": 6.7,
"P_score": 7.4,
"P_lrt": 7.5,
"P_CL": 7.2,
"P_CLkm": 5.3,
"SLR": 5.1
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.35,
"S_rwald": 0.33,
"S_score": 0.35,
"S_lrt": 0.35,
"S_CL": 0.34,
"S_CLkm": 0.31,
"P_wald": 0.37,
"P_rwald": 0.35,
"P_score": 0.37,
"P_lrt": 0.37,
"P_CL": 0.37,
"P_CLkm": 0.32,
"SLR": 0.31
},
"mean_z": {
"LR": 0,
"S_wald": -0.007,
"S_rwald": -0.006,
"S_score": -0.006,
"S_lrt": -0.005,
"S_CL": -0.003,
"S_CLkm": -0.002,
"P_wald": -0.002,
"P_rwald": -0.002,
"P_score": -0.001,
"P_lrt": -0.002,
"P_CL": 0.005,
"P_CLkm": 0.004,
"SLR": -0.001
},
"var_z": {
"LR": 0.998,
"S_wald": 1.095,
"S_rwald": 1.056,
"S_score": 1.113,
"S_lrt": 1.125,
"S_CL": 1.062,
"S_CLkm": 0.991,
"P_wald": 1.158,
"P_rwald": 1.098,
"P_score": 1.177,
"P_lrt": 1.18,
"P_CL": 1.161,
"P_CLkm": 1.005,
"SLR": 0.99
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.022,
"S_rwald": 0.021,
"S_score": 0.023,
"S_lrt": 0.024,
"S_CL": 0.021,
"S_CLkm": 0.02,
"P_wald": 0.023,
"P_rwald": 0.022,
"P_score": 0.024,
"P_lrt": 0.024,
"P_CL": 0.024,
"P_CLkm": 0.021,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 13.8,
"P": 20.3
},
"rej_pct_converged_only": {
"LR": 5,
"S_wald": 6.5,
"S_rwald": 6,
"S_score": 6.7,
"S_lrt": 6.7,
"S_CL": 6.2,
"S_CLkm": 5.3,
"P_wald": 7.4,
"P_rwald": 6.8,
"P_score": 7.6,
"P_lrt": 7.6,
"P_CL": 7.4,
"P_CLkm": 5.5,
"SLR": 5.2
},
"null_crit_abs_z": {
"LR": 1.955,
"S_wald": 2.052,
"S_rwald": 2.021,
"S_score": 2.074,
"S_lrt": 2.075,
"S_CL": 2.038,
"S_CLkm": 1.968,
"P_wald": 2.156,
"P_rwald": 2.1,
"P_score": 2.18,
"P_lrt": 2.186,
"P_CL": 2.14,
"P_CLkm": 1.991,
"SLR": 1.968
},
"est": {
"U": {
"truth": 0,
"mean": 0,
"bias": 0.0003,
"emp_se": 0.2477,
"mod_se": 0.2481,
"rmse": 0.2477,
"cov_pct": 95.4,
"ratio_emp_mod": 0.999
},
"S": {
"truth": 0,
"mean": -0.001,
"bias": -0.0011,
"emp_se": 0.2691,
"mod_se": 0.2545,
"rmse": 0.2691,
"cov_pct": 93.6,
"ratio_emp_mod": 1.057,
"rob_se": 0.2591,
"rcov_pct": 94.1,
"ratio_emp_rob": 1.038
},
"P": {
"truth": 0,
"mean": 0,
"bias": 0.0002,
"emp_se": 0.2865,
"mod_se": 0.2633,
"rmse": 0.2865,
"cov_pct": 92.7,
"ratio_emp_mod": 1.088,
"rob_se": 0.2705,
"rcov_pct": 93.3,
"ratio_emp_rob": 1.059
}
},
"sep_flag_S": 136,
"sep_flag_P": 136,
"nonconverged": 227
},
"null": {
"R": 5000,
"events_mean": 78.4,
"epv_P": 7.8,
"rej_pct": {
"LR": 5.2,
"S_wald": 6.6,
"S_rwald": 6.3,
"S_score": 6.9,
"S_lrt": 6.9,
"S_CL": 5.8,
"S_CLkm": 5.1,
"P_wald": 7.3,
"P_rwald": 6.7,
"P_score": 7.6,
"P_lrt": 7.6,
"P_CL": 7.2,
"P_CLkm": 5.5,
"SLR": 5.4
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.35,
"S_rwald": 0.34,
"S_score": 0.36,
"S_lrt": 0.36,
"S_CL": 0.33,
"S_CLkm": 0.31,
"P_wald": 0.37,
"P_rwald": 0.35,
"P_score": 0.37,
"P_lrt": 0.38,
"P_CL": 0.37,
"P_CLkm": 0.32,
"SLR": 0.32
},
"mean_z": {
"LR": -0.018,
"S_wald": -0.013,
"S_rwald": -0.013,
"S_score": -0.013,
"S_lrt": -0.013,
"S_CL": -0.017,
"S_CLkm": -0.016,
"P_wald": -0.014,
"P_rwald": -0.014,
"P_score": -0.014,
"P_lrt": -0.015,
"P_CL": -0.02,
"P_CLkm": -0.019,
"SLR": -0.013
},
"var_z": {
"LR": 1.028,
"S_wald": 1.114,
"S_rwald": 1.093,
"S_score": 1.129,
"S_lrt": 1.126,
"S_CL": 1.088,
"S_CLkm": 1.015,
"P_wald": 1.192,
"P_rwald": 1.146,
"P_score": 1.209,
"P_lrt": 1.206,
"P_CL": 1.197,
"P_CLkm": 1.037,
"SLR": 1.037
},
"var_z_mcse": {
"LR": 0.021,
"S_wald": 0.023,
"S_rwald": 0.022,
"S_score": 0.023,
"S_lrt": 0.023,
"S_CL": 0.022,
"S_CLkm": 0.021,
"P_wald": 0.024,
"P_rwald": 0.023,
"P_score": 0.025,
"P_lrt": 0.025,
"P_CL": 0.025,
"P_CLkm": 0.021,
"SLR": 0.021
},
"cl_var_reduction_pct": {
"S": 14.3,
"P": 18.8
},
"rej_pct_converged_only": {
"LR": 5.2,
"S_wald": 6.6,
"S_rwald": 6.3,
"S_score": 6.9,
"S_lrt": 6.9,
"S_CL": 5.8,
"S_CLkm": 5.1,
"P_wald": 7.3,
"P_rwald": 6.7,
"P_score": 7.6,
"P_lrt": 7.6,
"P_CL": 7.2,
"P_CLkm": 5.5,
"SLR": 5.4
},
"null_crit_abs_z": {
"LR": 1.966,
"S_wald": 2.083,
"S_rwald": 2.067,
"S_score": 2.102,
"S_lrt": 2.103,
"S_CL": 2.038,
"S_CLkm": 1.968,
"P_wald": 2.145,
"P_rwald": 2.1,
"P_score": 2.165,
"P_lrt": 2.165,
"P_CL": 2.136,
"P_CLkm": 1.987,
"SLR": 1.993
},
"est": {
"U": {
"truth": 0,
"mean": -0.004,
"bias": -0.0043,
"emp_se": 0.2305,
"mod_se": 0.2274,
"rmse": 0.2305,
"cov_pct": 95.1,
"ratio_emp_mod": 1.014
},
"S": {
"truth": 0,
"mean": -0.003,
"bias": -0.0033,
"emp_se": 0.2478,
"mod_se": 0.233,
"rmse": 0.2478,
"cov_pct": 93.4,
"ratio_emp_mod": 1.064,
"rob_se": 0.2355,
"rcov_pct": 93.7,
"ratio_emp_rob": 1.053
},
"P": {
"truth": 0,
"mean": -0.004,
"bias": -0.0038,
"emp_se": 0.2653,
"mod_se": 0.2405,
"rmse": 0.2653,
"cov_pct": 92.7,
"ratio_emp_mod": 1.103,
"rob_se": 0.2454,
"rcov_pct": 93.3,
"ratio_emp_rob": 1.081
}
},
"sep_flag_S": 14,
"sep_flag_P": 14,
"nonconverged": 32
},
"balanced": {
"R": 2000,
"events_mean": 77.7,
"epv_P": 7.8,
"rej_pct": {
"LR": 50.4,
"S_wald": 53.5,
"S_rwald": 53.2,
"S_score": 54,
"S_lrt": 54,
"S_CL": 56.5,
"S_CLkm": 53.8,
"P_wald": 54,
"P_rwald": 52.1,
"P_score": 54.5,
"P_lrt": 54.4,
"P_CL": 57.5,
"P_CLkm": 52.4,
"SLR": 46.3
},
"rej_mcse_pct": {
"LR": 1.12,
"S_wald": 1.12,
"S_rwald": 1.12,
"S_score": 1.11,
"S_lrt": 1.11,
"S_CL": 1.11,
"S_CLkm": 1.11,
"P_wald": 1.11,
"P_rwald": 1.12,
"P_score": 1.11,
"P_lrt": 1.11,
"P_CL": 1.11,
"P_CLkm": 1.12,
"SLR": 1.11
},
"mean_z": {
"LR": -1.976,
"S_wald": -2.04,
"S_rwald": -2.016,
"S_score": -2.071,
"S_lrt": -2.066,
"S_CL": -2.118,
"S_CLkm": -2.046,
"P_wald": -2.042,
"P_rwald": -2.002,
"P_score": -2.072,
"P_lrt": -2.068,
"P_CL": -2.169,
"P_CLkm": -2.018,
"SLR": -1.892
},
"var_z": {
"LR": 1.024,
"S_wald": 1.028,
"S_rwald": 1.008,
"S_score": 1.097,
"S_lrt": 1.084,
"S_CL": 1.052,
"S_CLkm": 0.981,
"P_wald": 1.096,
"P_rwald": 1.064,
"P_score": 1.167,
"P_lrt": 1.157,
"P_CL": 1.157,
"P_CLkm": 1.001,
"SLR": 1.034
},
"var_z_mcse": {
"LR": 0.034,
"S_wald": 0.034,
"S_rwald": 0.033,
"S_score": 0.036,
"S_lrt": 0.036,
"S_CL": 0.035,
"S_CLkm": 0.033,
"P_wald": 0.036,
"P_rwald": 0.035,
"P_score": 0.039,
"P_lrt": 0.038,
"P_CL": 0.039,
"P_CLkm": 0.033,
"SLR": 0.035
},
"cl_var_reduction_pct": {
"S": 14,
"P": 18.1
},
"rej_pct_converged_only": {
"LR": 50.3,
"S_wald": 53.5,
"S_rwald": 53.2,
"S_score": 54,
"S_lrt": 54,
"S_CL": 56.4,
"S_CLkm": 53.7,
"P_wald": 53.9,
"P_rwald": 52,
"P_score": 54.4,
"P_lrt": 54.3,
"P_CL": 57.4,
"P_CLkm": 52.3,
"SLR": 46.2
},
"diff_vs_LR_pct": {
"S_wald": [
3,
0.88
],
"S_rwald": [
2.8,
0.89
],
"S_score": [
3.5,
0.88
],
"S_lrt": [
3.5,
0.87
],
"S_CL": [
6,
0.83
],
"S_CLkm": [
3.4,
0.82
],
"P_wald": [
3.5,
0.91
],
"P_rwald": [
1.7,
0.91
],
"P_score": [
4,
0.91
],
"P_lrt": [
4,
0.91
],
"P_CL": [
7,
0.89
],
"P_CLkm": [
1.9,
0.86
],
"SLR": [
-4.2,
0.94
]
},
"size_adj_power_pct": {
"LR": 50.2,
"S_wald": 48.4,
"S_rwald": 47.6,
"S_score": 48.2,
"S_lrt": 47.9,
"S_CL": 53.5,
"S_CLkm": 53.5,
"P_wald": 46.1,
"P_rwald": 46.6,
"P_score": 46,
"P_lrt": 45.8,
"P_CL": 51.5,
"P_CLkm": 51.5,
"SLR": 45.5
},
"RE_cal": {
"S_wald": [
0.985,
0.954,
1.016
],
"S_rwald": [
0.98,
0.949,
1.011
],
"S_score": [
1.001,
0.97,
1.033
],
"S_lrt": [
0.999,
0.968,
1.03
],
"S_CL": [
1.086,
1.057,
1.116
],
"S_CLkm": [
1.086,
1.057,
1.116
],
"P_wald": [
0.921,
0.89,
0.954
],
"P_rwald": [
0.922,
0.889,
0.955
],
"P_score": [
0.936,
0.903,
0.969
],
"P_lrt": [
0.934,
0.902,
0.968
],
"P_CL": [
1.035,
1.003,
1.068
],
"P_CLkm": [
1.035,
1.003,
1.068
],
"SLR": [
0.909,
0.877,
0.942
]
},
"size_adj_power_pct_obs": {
"LR": 50.1,
"S_wald": 51.7,
"S_rwald": 50.5,
"S_score": 51.7,
"S_lrt": 51.8,
"S_CL": 54.1,
"S_CLkm": 54.1,
"P_wald": 49,
"P_rwald": 47.1,
"P_score": 49.1,
"P_lrt": 49.4,
"P_CL": 52.4,
"P_CLkm": 52.4,
"SLR": 46.8
},
"RE_cal_obs": {
"S_wald": [
1.01,
0.978,
1.043
],
"S_rwald": [
0.987,
0.956,
1.02
],
"S_score": [
1.03,
0.997,
1.063
],
"S_lrt": [
1.028,
0.996,
1.061
],
"S_CL": [
1.095,
1.066,
1.126
],
"S_CLkm": [
1.095,
1.066,
1.126
],
"P_wald": [
0.941,
0.908,
0.976
],
"P_rwald": [
0.911,
0.878,
0.945
],
"P_score": [
0.959,
0.925,
0.994
],
"P_lrt": [
0.958,
0.924,
0.994
],
"P_CL": [
1.038,
1.006,
1.071
],
"P_CLkm": [
1.038,
1.006,
1.071
],
"SLR": [
0.907,
0.874,
0.94
]
},
"size_adj_power_pct_trt": {
"LR": 50.6,
"S_wald": 49.8,
"S_rwald": 49.9,
"S_score": 49.5,
"S_lrt": 49.5,
"S_CL": 53.5,
"S_CLkm": 53.5,
"P_wald": 45.5,
"P_rwald": 46.6,
"P_score": 45.3,
"P_lrt": 45,
"P_CL": 51.3,
"P_CLkm": 51.3,
"SLR": 46.1
},
"RE_cal_trt": {
"S_wald": [
0.972,
0.942,
1.003
],
"S_rwald": [
0.984,
0.954,
1.015
],
"S_score": [
0.985,
0.955,
1.017
],
"S_lrt": [
0.97,
0.938,
1.003
],
"S_CL": [
1.08,
1.051,
1.111
],
"S_CLkm": [
1.08,
1.051,
1.111
],
"P_wald": [
0.921,
0.888,
0.955
],
"P_rwald": [
0.934,
0.9,
0.969
],
"P_score": [
0.933,
0.899,
0.968
],
"P_lrt": [
0.927,
0.893,
0.961
],
"P_CL": [
1.036,
1.002,
1.071
],
"P_CLkm": [
1.036,
1.002,
1.071
],
"SLR": [
0.924,
0.892,
0.957
]
},
"RE": {
"S_wald": [
1.066,
1.045,
1.087
],
"S_rwald": [
1.041,
1.02,
1.062
],
"S_score": [
1.099,
1.077,
1.121
],
"S_lrt": [
1.093,
1.072,
1.115
],
"S_CL": [
1.149,
1.13,
1.169
],
"S_CLkm": [
1.072,
1.054,
1.091
],
"P_wald": [
1.068,
1.044,
1.092
],
"P_rwald": [
1.027,
1.004,
1.051
],
"P_score": [
1.1,
1.075,
1.125
],
"P_lrt": [
1.096,
1.071,
1.12
],
"P_CL": [
1.205,
1.182,
1.229
],
"P_CLkm": [
1.043,
1.023,
1.064
],
"SLR": [
0.916,
0.895,
0.938
]
},
"decomp": {
"S": {
"est_ratio": 1.077,
"se_ratio": 1.026,
"empsd_ratio": 1.066
},
"P": {
"est_ratio": 1.113,
"se_ratio": 1.059,
"empsd_ratio": 1.14
}
},
"est": {
"U": {
"truth": -0.448,
"mean": -0.452,
"bias": -0.0038,
"emp_se": 0.2333,
"mod_se": 0.2313,
"rmse": 0.2333,
"cov_pct": 94.7,
"ratio_emp_mod": 1.009
},
"S": {
"truth": -0.483,
"mean": -0.487,
"bias": -0.0042,
"emp_se": 0.2488,
"mod_se": 0.2374,
"rmse": 0.2487,
"cov_pct": 93.2,
"ratio_emp_mod": 1.048,
"rob_se": 0.2406,
"rcov_pct": 93.8,
"ratio_emp_rob": 1.034
},
"P": {
"truth": -0.489,
"mean": -0.503,
"bias": -0.0139,
"emp_se": 0.266,
"mod_se": 0.245,
"rmse": 0.2663,
"cov_pct": 92.5,
"ratio_emp_mod": 1.085,
"rob_se": 0.2505,
"rcov_pct": 93.4,
"ratio_emp_rob": 1.062
}
},
"sep_flag_S": 13,
"sep_flag_P": 13,
"nonconverged": 18
},
"trial": {
"R": 1000,
"events_mean": 78.2,
"epv_P": 7.8,
"rej_pct": {
"LR": 54.5,
"S_wald": 49.4,
"S_rwald": 48,
"S_score": 49.8,
"S_lrt": 49.6,
"S_CL": 53.1,
"S_CLkm": 51.6,
"P_wald": 52,
"P_rwald": 49.8,
"P_score": 52.3,
"P_lrt": 52.3,
"P_CL": 56.8,
"P_CLkm": 52.2,
"SLR": 45.2
},
"rej_mcse_pct": {
"LR": 1.57,
"S_wald": 1.58,
"S_rwald": 1.58,
"S_score": 1.58,
"S_lrt": 1.58,
"S_CL": 1.58,
"S_CLkm": 1.58,
"P_wald": 1.58,
"P_rwald": 1.58,
"P_score": 1.58,
"P_lrt": 1.58,
"P_CL": 1.57,
"P_CLkm": 1.58,
"SLR": 1.57
},
"mean_z": {
"LR": -2.092,
"S_wald": -1.941,
"S_rwald": -1.906,
"S_score": -1.969,
"S_lrt": -1.963,
"S_CL": -2.059,
"S_CLkm": -1.989,
"P_wald": -1.989,
"P_rwald": -1.937,
"P_score": -2.017,
"P_lrt": -2.011,
"P_CL": -2.15,
"P_CLkm": -2,
"SLR": -1.839
},
"var_z": {
"LR": 0.954,
"S_wald": 1.043,
"S_rwald": 1.012,
"S_score": 1.105,
"S_lrt": 1.092,
"S_CL": 1.059,
"S_CLkm": 0.988,
"P_wald": 1.115,
"P_rwald": 1.072,
"P_score": 1.181,
"P_lrt": 1.167,
"P_CL": 1.161,
"P_CLkm": 1.005,
"SLR": 1.034
},
"var_z_mcse": {
"LR": 0.041,
"S_wald": 0.045,
"S_rwald": 0.044,
"S_score": 0.048,
"S_lrt": 0.047,
"S_CL": 0.046,
"S_CLkm": 0.043,
"P_wald": 0.048,
"P_rwald": 0.047,
"P_score": 0.052,
"P_lrt": 0.051,
"P_CL": 0.051,
"P_CLkm": 0.044,
"SLR": 0.047
},
"cl_var_reduction_pct": {
"S": 14.2,
"P": 18.5
},
"rej_pct_converged_only": {
"LR": 54.4,
"S_wald": 49.3,
"S_rwald": 47.9,
"S_score": 49.7,
"S_lrt": 49.5,
"S_CL": 53,
"S_CLkm": 51.5,
"P_wald": 52,
"P_rwald": 49.8,
"P_score": 52.3,
"P_lrt": 52.3,
"P_CL": 56.9,
"P_CLkm": 52.2,
"SLR": 45.3
},
"diff_vs_LR_pct": {
"S_wald": [
-5.1,
1.24
],
"S_rwald": [
-6.5,
1.25
],
"S_score": [
-4.7,
1.23
],
"S_lrt": [
-4.9,
1.24
],
"S_CL": [
-1.4,
1.17
],
"S_CLkm": [
-2.9,
1.15
],
"P_wald": [
-2.5,
1.27
],
"P_rwald": [
-4.7,
1.27
],
"P_score": [
-2.2,
1.28
],
"P_lrt": [
-2.2,
1.27
],
"P_CL": [
2.3,
1.25
],
"P_CLkm": [
-2.3,
1.23
],
"SLR": [
-9.3,
1.35
]
},
"size_adj_power_pct": {
"LR": 54.4,
"S_wald": 44.8,
"S_rwald": 43.9,
"S_score": 44.8,
"S_lrt": 44.8,
"S_CL": 51.3,
"S_CLkm": 51.3,
"P_wald": 44.8,
"P_rwald": 44.2,
"P_score": 44.8,
"P_lrt": 44.9,
"P_CL": 51.1,
"P_CLkm": 51.1,
"SLR": 43.4
},
"RE_cal": {
"S_wald": [
0.795,
0.764,
0.826
],
"S_rwald": [
0.781,
0.751,
0.812
],
"S_score": [
0.806,
0.775,
0.839
],
"S_lrt": [
0.804,
0.773,
0.836
],
"S_CL": [
0.915,
0.885,
0.947
],
"S_CLkm": [
0.915,
0.885,
0.947
],
"P_wald": [
0.78,
0.746,
0.815
],
"P_rwald": [
0.769,
0.736,
0.805
],
"P_score": [
0.791,
0.756,
0.827
],
"P_lrt": [
0.788,
0.754,
0.824
],
"P_CL": [
0.907,
0.872,
0.943
],
"P_CLkm": [
0.907,
0.872,
0.943
],
"SLR": [
0.766,
0.733,
0.801
]
},
"size_adj_power_pct_obs": {
"LR": 54,
"S_wald": 47.7,
"S_rwald": 46.3,
"S_score": 47.7,
"S_lrt": 47.9,
"S_CL": 52.2,
"S_CLkm": 52.2,
"P_wald": 47.9,
"P_rwald": 44.6,
"P_score": 48.4,
"P_lrt": 48.4,
"P_CL": 52.2,
"P_CLkm": 52.2,
"SLR": 45.9
},
"RE_cal_obs": {
"S_wald": [
0.815,
0.783,
0.848
],
"S_rwald": [
0.787,
0.756,
0.819
],
"S_score": [
0.829,
0.797,
0.863
],
"S_lrt": [
0.828,
0.796,
0.861
],
"S_CL": [
0.923,
0.892,
0.955
],
"S_CLkm": [
0.923,
0.892,
0.955
],
"P_wald": [
0.796,
0.761,
0.833
],
"P_rwald": [
0.76,
0.726,
0.796
],
"P_score": [
0.81,
0.774,
0.848
],
"P_lrt": [
0.809,
0.773,
0.846
],
"P_CL": [
0.909,
0.874,
0.946
],
"P_CLkm": [
0.909,
0.874,
0.946
],
"SLR": [
0.764,
0.73,
0.799
]
},
"size_adj_power_pct_trt": {
"LR": 54.6,
"S_wald": 46,
"S_rwald": 45.7,
"S_score": 45.9,
"S_lrt": 45.5,
"S_CL": 51.3,
"S_CLkm": 51.3,
"P_wald": 44.2,
"P_rwald": 44.2,
"P_score": 44.1,
"P_lrt": 43.7,
"P_CL": 51,
"P_CLkm": 51,
"SLR": 44.8
},
"RE_cal_trt": {
"S_wald": [
0.784,
0.755,
0.816
],
"S_rwald": [
0.784,
0.754,
0.816
],
"S_score": [
0.794,
0.763,
0.825
],
"S_lrt": [
0.781,
0.75,
0.814
],
"S_CL": [
0.911,
0.88,
0.943
],
"S_CLkm": [
0.911,
0.88,
0.943
],
"P_wald": [
0.779,
0.744,
0.815
],
"P_rwald": [
0.78,
0.745,
0.816
],
"P_score": [
0.788,
0.753,
0.825
],
"P_lrt": [
0.782,
0.747,
0.818
],
"P_CL": [
0.907,
0.871,
0.945
],
"P_CLkm": [
0.907,
0.871,
0.945
],
"SLR": [
0.779,
0.745,
0.814
]
},
"RE": {
"S_wald": [
0.861,
0.834,
0.887
],
"S_rwald": [
0.83,
0.804,
0.856
],
"S_score": [
0.885,
0.858,
0.913
],
"S_lrt": [
0.88,
0.853,
0.908
],
"S_CL": [
0.969,
0.943,
0.995
],
"S_CLkm": [
0.904,
0.88,
0.928
],
"P_wald": [
0.903,
0.872,
0.935
],
"P_rwald": [
0.857,
0.827,
0.888
],
"P_score": [
0.929,
0.897,
0.963
],
"P_lrt": [
0.924,
0.892,
0.957
],
"P_CL": [
1.056,
1.024,
1.088
],
"P_CLkm": [
0.914,
0.886,
0.942
],
"SLR": [
0.772,
0.745,
0.8
]
},
"decomp": {
"S": {
"est_ratio": 0.972,
"se_ratio": 1.03,
"empsd_ratio": 1.11
},
"P": {
"est_ratio": 1.03,
"se_ratio": 1.065,
"empsd_ratio": 1.19
}
},
"est": {
"U": {
"truth": -0.477,
"mean": -0.477,
"bias": -0.0008,
"emp_se": 0.2252,
"mod_se": 0.2309,
"rmse": 0.2251,
"cov_pct": 96.2,
"ratio_emp_mod": 0.975
},
"S": {
"truth": -0.47,
"mean": -0.464,
"bias": 0.0059,
"emp_se": 0.2499,
"mod_se": 0.2379,
"rmse": 0.2498,
"cov_pct": 93.7,
"ratio_emp_mod": 1.051,
"rob_se": 0.2425,
"rcov_pct": 94.3,
"ratio_emp_rob": 1.031
},
"P": {
"truth": -0.488,
"mean": -0.492,
"bias": -0.0033,
"emp_se": 0.2679,
"mod_se": 0.246,
"rmse": 0.2678,
"cov_pct": 92.7,
"ratio_emp_mod": 1.089,
"rob_se": 0.2532,
"rcov_pct": 93.3,
"ratio_emp_rob": 1.058
}
},
"sep_flag_S": 4,
"sep_flag_P": 4,
"nonconverged": 8
}
},
"300": {
"null_obs": {
"R": 5000,
"events_mean": 181,
"epv_P": 18.1,
"rej_pct": {
"LR": 4.9,
"S_wald": 5.5,
"S_rwald": 5.5,
"S_score": 5.5,
"S_lrt": 5.5,
"S_CL": 5.3,
"S_CLkm": 4.9,
"P_wald": 5.9,
"P_rwald": 5.6,
"P_score": 6,
"P_lrt": 6,
"P_CL": 5.7,
"P_CLkm": 4.8,
"SLR": 5.4
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.32,
"S_rwald": 0.32,
"S_score": 0.32,
"S_lrt": 0.32,
"S_CL": 0.32,
"S_CLkm": 0.31,
"P_wald": 0.33,
"P_rwald": 0.33,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.33,
"P_CLkm": 0.3,
"SLR": 0.32
},
"mean_z": {
"LR": -0.008,
"S_wald": -0.002,
"S_rwald": -0.002,
"S_score": -0.002,
"S_lrt": -0.002,
"S_CL": -0.006,
"S_CLkm": -0.006,
"P_wald": -0.004,
"P_rwald": -0.004,
"P_score": -0.004,
"P_lrt": -0.004,
"P_CL": -0.01,
"P_CLkm": -0.009,
"SLR": -0.002
},
"var_z": {
"LR": 1.019,
"S_wald": 1.058,
"S_rwald": 1.044,
"S_score": 1.064,
"S_lrt": 1.062,
"S_CL": 1.032,
"S_CLkm": 0.998,
"P_wald": 1.084,
"P_rwald": 1.066,
"P_score": 1.09,
"P_lrt": 1.088,
"P_CL": 1.068,
"P_CLkm": 0.996,
"SLR": 1.023
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.021,
"P_rwald": 0.021,
"P_score": 0.021,
"P_lrt": 0.021,
"P_CL": 0.021,
"P_CLkm": 0.019,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 14.2,
"P": 16.9
},
"rej_pct_converged_only": {
"LR": 4.9,
"S_wald": 5.5,
"S_rwald": 5.5,
"S_score": 5.5,
"S_lrt": 5.5,
"S_CL": 5.3,
"S_CLkm": 4.9,
"P_wald": 5.9,
"P_rwald": 5.6,
"P_score": 6,
"P_lrt": 6,
"P_CL": 5.7,
"P_CLkm": 4.8,
"SLR": 5.4
},
"null_crit_abs_z": {
"LR": 1.954,
"S_wald": 2.009,
"S_rwald": 1.989,
"S_score": 2.016,
"S_lrt": 2.015,
"S_CL": 1.989,
"S_CLkm": 1.955,
"P_wald": 2.029,
"P_rwald": 2.006,
"P_score": 2.037,
"P_lrt": 2.035,
"P_CL": 2.006,
"P_CLkm": 1.937,
"SLR": 2.009
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.0011,
"emp_se": 0.15,
"mod_se": 0.1491,
"rmse": 0.15,
"cov_pct": 95.2,
"ratio_emp_mod": 1.006
},
"S": {
"truth": 0,
"mean": 0,
"bias": -0.0002,
"emp_se": 0.1555,
"mod_se": 0.1508,
"rmse": 0.1555,
"cov_pct": 94.5,
"ratio_emp_mod": 1.031,
"rob_se": 0.1518,
"rcov_pct": 94.5,
"ratio_emp_rob": 1.024
},
"P": {
"truth": 0,
"mean": -0.001,
"bias": -0.0006,
"emp_se": 0.1597,
"mod_se": 0.1529,
"rmse": 0.1597,
"cov_pct": 94.1,
"ratio_emp_mod": 1.044,
"rob_se": 0.1543,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.035
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"null_trt": {
"R": 5000,
"events_mean": 132.2,
"epv_P": 13.2,
"rej_pct": {
"LR": 4.4,
"S_wald": 5.9,
"S_rwald": 5.2,
"S_score": 6.1,
"S_lrt": 6.1,
"S_CL": 5,
"S_CLkm": 4.6,
"P_wald": 6.5,
"P_rwald": 5.6,
"P_score": 6.5,
"P_lrt": 6.5,
"P_CL": 5.5,
"P_CLkm": 4.8,
"SLR": 4.8
},
"rej_mcse_pct": {
"LR": 0.29,
"S_wald": 0.33,
"S_rwald": 0.31,
"S_score": 0.34,
"S_lrt": 0.34,
"S_CL": 0.31,
"S_CLkm": 0.3,
"P_wald": 0.35,
"P_rwald": 0.32,
"P_score": 0.35,
"P_lrt": 0.35,
"P_CL": 0.32,
"P_CLkm": 0.3,
"SLR": 0.3
},
"mean_z": {
"LR": -0.005,
"S_wald": -0.001,
"S_rwald": -0.001,
"S_score": -0.001,
"S_lrt": -0.001,
"S_CL": 0.001,
"S_CLkm": 0.001,
"P_wald": -0.004,
"P_rwald": -0.004,
"P_score": -0.004,
"P_lrt": -0.004,
"P_CL": 0.001,
"P_CLkm": 0.001,
"SLR": 0.001
},
"var_z": {
"LR": 0.958,
"S_wald": 1.07,
"S_rwald": 1.019,
"S_score": 1.078,
"S_lrt": 1.077,
"S_CL": 1.004,
"S_CLkm": 0.97,
"P_wald": 1.126,
"P_rwald": 1.056,
"P_score": 1.135,
"P_lrt": 1.134,
"P_CL": 1.05,
"P_CLkm": 0.98,
"SLR": 0.966
},
"var_z_mcse": {
"LR": 0.019,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.022,
"P_rwald": 0.021,
"P_score": 0.023,
"P_lrt": 0.023,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.019
},
"cl_var_reduction_pct": {
"S": 12.4,
"P": 16.8
},
"rej_pct_converged_only": {
"LR": 4.4,
"S_wald": 5.9,
"S_rwald": 5.2,
"S_score": 6.1,
"S_lrt": 6.1,
"S_CL": 5,
"S_CLkm": 4.6,
"P_wald": 6.5,
"P_rwald": 5.5,
"P_score": 6.5,
"P_lrt": 6.5,
"P_CL": 5.5,
"P_CLkm": 4.8,
"SLR": 4.8
},
"null_crit_abs_z": {
"LR": 1.917,
"S_wald": 2.015,
"S_rwald": 1.976,
"S_score": 2.025,
"S_lrt": 2.024,
"S_CL": 1.96,
"S_CLkm": 1.927,
"P_wald": 2.09,
"P_rwald": 2.024,
"P_score": 2.102,
"P_lrt": 2.098,
"P_CL": 2.014,
"P_CLkm": 1.945,
"SLR": 1.938
},
"est": {
"U": {
"truth": 0,
"mean": -0.001,
"bias": -0.0008,
"emp_se": 0.1706,
"mod_se": 0.1746,
"rmse": 0.1706,
"cov_pct": 95.8,
"ratio_emp_mod": 0.977
},
"S": {
"truth": 0,
"mean": 0,
"bias": -0.0002,
"emp_se": 0.184,
"mod_se": 0.1769,
"rmse": 0.184,
"cov_pct": 94.1,
"ratio_emp_mod": 1.04,
"rob_se": 0.1812,
"rcov_pct": 94.8,
"ratio_emp_rob": 1.015
},
"P": {
"truth": 0,
"mean": -0.001,
"bias": -0.0007,
"emp_se": 0.1921,
"mod_se": 0.1798,
"rmse": 0.1921,
"cov_pct": 93.5,
"ratio_emp_mod": 1.068,
"rob_se": 0.1858,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.034
}
},
"sep_flag_S": 7,
"sep_flag_P": 7,
"nonconverged": 14
},
"null": {
"R": 5000,
"events_mean": 157.2,
"epv_P": 15.7,
"rej_pct": {
"LR": 4.8,
"S_wald": 5.7,
"S_rwald": 5.3,
"S_score": 5.7,
"S_lrt": 5.7,
"S_CL": 5.4,
"S_CLkm": 4.8,
"P_wald": 6.1,
"P_rwald": 5.3,
"P_score": 6.2,
"P_lrt": 6.2,
"P_CL": 5.4,
"P_CLkm": 4.6,
"SLR": 4.8
},
"rej_mcse_pct": {
"LR": 0.3,
"S_wald": 0.33,
"S_rwald": 0.32,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.32,
"S_CLkm": 0.3,
"P_wald": 0.34,
"P_rwald": 0.32,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.32,
"P_CLkm": 0.3,
"SLR": 0.3
},
"mean_z": {
"LR": -0.019,
"S_wald": -0.012,
"S_rwald": -0.012,
"S_score": -0.012,
"S_lrt": -0.012,
"S_CL": -0.018,
"S_CLkm": -0.017,
"P_wald": -0.014,
"P_rwald": -0.014,
"P_score": -0.014,
"P_lrt": -0.014,
"P_CL": -0.019,
"P_CLkm": -0.019,
"SLR": -0.015
},
"var_z": {
"LR": 0.99,
"S_wald": 1.049,
"S_rwald": 1.017,
"S_score": 1.056,
"S_lrt": 1.054,
"S_CL": 1.013,
"S_CLkm": 0.979,
"P_wald": 1.094,
"P_rwald": 1.044,
"P_score": 1.101,
"P_lrt": 1.099,
"P_CL": 1.05,
"P_CLkm": 0.98,
"SLR": 0.975
},
"var_z_mcse": {
"LR": 0.019,
"S_wald": 0.02,
"S_rwald": 0.02,
"S_score": 0.02,
"S_lrt": 0.02,
"S_CL": 0.02,
"S_CLkm": 0.019,
"P_wald": 0.021,
"P_rwald": 0.02,
"P_score": 0.021,
"P_lrt": 0.021,
"P_CL": 0.021,
"P_CLkm": 0.019,
"SLR": 0.019
},
"cl_var_reduction_pct": {
"S": 13.2,
"P": 15.7
},
"rej_pct_converged_only": {
"LR": 4.8,
"S_wald": 5.7,
"S_rwald": 5.3,
"S_score": 5.7,
"S_lrt": 5.7,
"S_CL": 5.4,
"S_CLkm": 4.8,
"P_wald": 6.1,
"P_rwald": 5.3,
"P_score": 6.2,
"P_lrt": 6.2,
"P_CL": 5.4,
"P_CLkm": 4.6,
"SLR": 4.8
},
"null_crit_abs_z": {
"LR": 1.939,
"S_wald": 2.013,
"S_rwald": 1.992,
"S_score": 2.022,
"S_lrt": 2.021,
"S_CL": 1.983,
"S_CLkm": 1.95,
"P_wald": 2.022,
"P_rwald": 1.98,
"P_score": 2.029,
"P_lrt": 2.029,
"P_CL": 1.991,
"P_CLkm": 1.923,
"SLR": 1.939
},
"est": {
"U": {
"truth": 0,
"mean": -0.003,
"bias": -0.003,
"emp_se": 0.1591,
"mod_se": 0.16,
"rmse": 0.1591,
"cov_pct": 95.3,
"ratio_emp_mod": 0.994
},
"S": {
"truth": 0,
"mean": -0.002,
"bias": -0.0019,
"emp_se": 0.1666,
"mod_se": 0.162,
"rmse": 0.1666,
"cov_pct": 94.3,
"ratio_emp_mod": 1.029,
"rob_se": 0.1646,
"rcov_pct": 94.7,
"ratio_emp_rob": 1.012
},
"P": {
"truth": 0,
"mean": -0.002,
"bias": -0.0022,
"emp_se": 0.1727,
"mod_se": 0.1645,
"rmse": 0.1727,
"cov_pct": 93.9,
"ratio_emp_mod": 1.05,
"rob_se": 0.1685,
"rcov_pct": 94.7,
"ratio_emp_rob": 1.025
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"balanced": {
"R": 2000,
"events_mean": 155.9,
"epv_P": 15.6,
"rej_pct": {
"LR": 80,
"S_wald": 82.4,
"S_rwald": 81.7,
"S_score": 82.5,
"S_lrt": 82.5,
"S_CL": 84.2,
"S_CLkm": 83.8,
"P_wald": 82.8,
"P_rwald": 81.8,
"P_score": 83,
"P_lrt": 83,
"P_CL": 84.8,
"P_CLkm": 83.7,
"SLR": 78.2
},
"rej_mcse_pct": {
"LR": 0.9,
"S_wald": 0.85,
"S_rwald": 0.87,
"S_score": 0.85,
"S_lrt": 0.85,
"S_CL": 0.82,
"S_CLkm": 0.82,
"P_wald": 0.84,
"P_rwald": 0.86,
"P_score": 0.84,
"P_lrt": 0.84,
"P_CL": 0.8,
"P_CLkm": 0.83,
"SLR": 0.92
},
"mean_z": {
"LR": -2.792,
"S_wald": -2.917,
"S_rwald": -2.867,
"S_score": -2.952,
"S_lrt": -2.945,
"S_CL": -2.978,
"S_CLkm": -2.927,
"P_wald": -2.932,
"P_rwald": -2.862,
"P_score": -2.966,
"P_lrt": -2.96,
"P_CL": -3.013,
"P_CLkm": -2.911,
"SLR": -2.73
},
"var_z": {
"LR": 0.96,
"S_wald": 0.957,
"S_rwald": 0.929,
"S_score": 1.016,
"S_lrt": 1.004,
"S_CL": 0.961,
"S_CLkm": 0.929,
"P_wald": 0.99,
"P_rwald": 0.951,
"P_score": 1.049,
"P_lrt": 1.039,
"P_CL": 0.996,
"P_CLkm": 0.929,
"SLR": 0.978
},
"var_z_mcse": {
"LR": 0.03,
"S_wald": 0.03,
"S_rwald": 0.029,
"S_score": 0.032,
"S_lrt": 0.031,
"S_CL": 0.03,
"S_CLkm": 0.029,
"P_wald": 0.031,
"P_rwald": 0.029,
"P_score": 0.033,
"P_lrt": 0.032,
"P_CL": 0.031,
"P_CLkm": 0.029,
"SLR": 0.031
},
"cl_var_reduction_pct": {
"S": 12.8,
"P": 15
},
"rej_pct_converged_only": {
"LR": 80,
"S_wald": 82.4,
"S_rwald": 81.7,
"S_score": 82.5,
"S_lrt": 82.5,
"S_CL": 84.2,
"S_CLkm": 83.8,
"P_wald": 82.8,
"P_rwald": 81.8,
"P_score": 83,
"P_lrt": 83,
"P_CL": 84.8,
"P_CLkm": 83.7,
"SLR": 78.2
},
"diff_vs_LR_pct": {
"S_wald": [
2.5,
0.69
],
"S_rwald": [
1.7,
0.69
],
"S_score": [
2.5,
0.69
],
"S_lrt": [
2.6,
0.69
],
"S_CL": [
4.2,
0.66
],
"S_CLkm": [
3.9,
0.67
],
"P_wald": [
2.9,
0.73
],
"P_rwald": [
1.8,
0.72
],
"P_score": [
3.1,
0.72
],
"P_lrt": [
3,
0.72
],
"P_CL": [
4.8,
0.68
],
"P_CLkm": [
3.8,
0.68
],
"SLR": [
-1.8,
0.79
]
},
"size_adj_power_pct": {
"LR": 80.4,
"S_wald": 81.2,
"S_rwald": 80.9,
"S_score": 81.2,
"S_lrt": 81,
"S_CL": 83.9,
"S_CLkm": 83.9,
"P_wald": 81.3,
"P_rwald": 81.2,
"P_score": 81.3,
"P_lrt": 81.4,
"P_CL": 84.4,
"P_CLkm": 84.4,
"SLR": 78.8
},
"RE_cal": {
"S_wald": [
1.031,
1.004,
1.058
],
"S_rwald": [
1.027,
1,
1.055
],
"S_score": [
1.049,
1.022,
1.077
],
"S_lrt": [
1.046,
1.018,
1.074
],
"S_CL": [
1.112,
1.086,
1.138
],
"S_CLkm": [
1.112,
1.086,
1.138
],
"P_wald": [
0.998,
0.97,
1.028
],
"P_rwald": [
0.997,
0.969,
1.027
],
"P_score": [
1.016,
0.987,
1.046
],
"P_lrt": [
1.013,
0.984,
1.043
],
"P_CL": [
1.099,
1.071,
1.127
],
"P_CLkm": [
1.099,
1.071,
1.127
],
"SLR": [
0.971,
0.943,
1
]
},
"size_adj_power_pct_obs": {
"LR": 80.2,
"S_wald": 81.2,
"S_rwald": 81,
"S_score": 81.2,
"S_lrt": 81.2,
"S_CL": 83.8,
"S_CLkm": 83.8,
"P_wald": 81,
"P_rwald": 80.2,
"P_score": 81.2,
"P_lrt": 81.3,
"P_CL": 84.1,
"P_CLkm": 84.1,
"SLR": 76.8
},
"RE_cal_obs": {
"S_wald": [
1.051,
1.023,
1.08
],
"S_rwald": [
1.03,
1.002,
1.058
],
"S_score": [
1.071,
1.042,
1.1
],
"S_lrt": [
1.068,
1.039,
1.097
],
"S_CL": [
1.123,
1.096,
1.15
],
"S_CLkm": [
1.123,
1.096,
1.15
],
"P_wald": [
1.037,
1.006,
1.068
],
"P_rwald": [
1.005,
0.976,
1.036
],
"P_score": [
1.056,
1.025,
1.088
],
"P_lrt": [
1.053,
1.023,
1.085
],
"P_CL": [
1.111,
1.082,
1.141
],
"P_CLkm": [
1.111,
1.082,
1.141
],
"SLR": [
0.952,
0.924,
0.981
]
},
"size_adj_power_pct_trt": {
"LR": 80.8,
"S_wald": 81.2,
"S_rwald": 81.2,
"S_score": 81.1,
"S_lrt": 81,
"S_CL": 84.2,
"S_CLkm": 84.2,
"P_wald": 79.4,
"P_rwald": 79.7,
"P_score": 79.2,
"P_lrt": 79.3,
"P_CL": 84,
"P_CLkm": 84,
"SLR": 78.8
},
"RE_cal_trt": {
"S_wald": [
0.977,
0.951,
1.004
],
"S_rwald": [
0.991,
0.964,
1.018
],
"S_score": [
0.993,
0.966,
1.02
],
"S_lrt": [
0.989,
0.963,
1.017
],
"S_CL": [
1.085,
1.059,
1.111
],
"S_CLkm": [
1.085,
1.059,
1.111
],
"P_wald": [
0.937,
0.909,
0.966
],
"P_rwald": [
0.952,
0.923,
0.982
],
"P_score": [
0.952,
0.923,
0.982
],
"P_lrt": [
0.949,
0.92,
0.979
],
"P_CL": [
1.062,
1.033,
1.091
],
"P_CLkm": [
1.062,
1.033,
1.091
],
"SLR": [
0.948,
0.92,
0.977
]
},
"RE": {
"S_wald": [
1.091,
1.077,
1.106
],
"S_rwald": [
1.054,
1.04,
1.069
],
"S_score": [
1.118,
1.103,
1.133
],
"S_lrt": [
1.113,
1.098,
1.127
],
"S_CL": [
1.137,
1.124,
1.151
],
"S_CLkm": [
1.099,
1.086,
1.112
],
"P_wald": [
1.102,
1.086,
1.119
],
"P_rwald": [
1.051,
1.035,
1.067
],
"P_score": [
1.129,
1.112,
1.145
],
"P_lrt": [
1.124,
1.108,
1.141
],
"P_CL": [
1.164,
1.15,
1.179
],
"P_CLkm": [
1.087,
1.073,
1.1
],
"SLR": [
0.956,
0.941,
0.971
]
},
"decomp": {
"S": {
"est_ratio": 1.074,
"se_ratio": 1.014,
"empsd_ratio": 1.046
},
"P": {
"est_ratio": 1.096,
"se_ratio": 1.031,
"empsd_ratio": 1.08
}
},
"est": {
"U": {
"truth": -0.448,
"mean": -0.45,
"bias": -0.0015,
"emp_se": 0.1595,
"mod_se": 0.1627,
"rmse": 0.1595,
"cov_pct": 95.7,
"ratio_emp_mod": 0.981
},
"S": {
"truth": -0.483,
"mean": -0.483,
"bias": 0.0001,
"emp_se": 0.1668,
"mod_se": 0.165,
"rmse": 0.1668,
"cov_pct": 95.3,
"ratio_emp_mod": 1.011,
"rob_se": 0.168,
"rcov_pct": 95.6,
"ratio_emp_rob": 0.993
},
"P": {
"truth": -0.489,
"mean": -0.493,
"bias": -0.0037,
"emp_se": 0.1723,
"mod_se": 0.1677,
"rmse": 0.1723,
"cov_pct": 94.8,
"ratio_emp_mod": 1.027,
"rob_se": 0.1719,
"rcov_pct": 95.3,
"ratio_emp_rob": 1.002
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"trial": {
"R": 1000,
"events_mean": 156.1,
"epv_P": 15.6,
"rej_pct": {
"LR": 85.3,
"S_wald": 81.5,
"S_rwald": 80.2,
"S_score": 81.8,
"S_lrt": 81.7,
"S_CL": 83.9,
"S_CLkm": 82.5,
"P_wald": 83.3,
"P_rwald": 81.7,
"P_score": 83.3,
"P_lrt": 83.4,
"P_CL": 85.3,
"P_CLkm": 84,
"SLR": 76.8
},
"rej_mcse_pct": {
"LR": 1.12,
"S_wald": 1.23,
"S_rwald": 1.26,
"S_score": 1.22,
"S_lrt": 1.22,
"S_CL": 1.16,
"S_CLkm": 1.2,
"P_wald": 1.18,
"P_rwald": 1.22,
"P_score": 1.18,
"P_lrt": 1.18,
"P_CL": 1.12,
"P_CLkm": 1.16,
"SLR": 1.33
},
"mean_z": {
"LR": -2.996,
"S_wald": -2.849,
"S_rwald": -2.78,
"S_score": -2.882,
"S_lrt": -2.876,
"S_CL": -2.931,
"S_CLkm": -2.881,
"P_wald": -2.932,
"P_rwald": -2.84,
"P_score": -2.967,
"P_lrt": -2.96,
"P_CL": -3.028,
"P_CLkm": -2.925,
"SLR": -2.689
},
"var_z": {
"LR": 0.99,
"S_wald": 1.022,
"S_rwald": 0.982,
"S_score": 1.078,
"S_lrt": 1.068,
"S_CL": 1.014,
"S_CLkm": 0.98,
"P_wald": 1.081,
"P_rwald": 1.024,
"P_score": 1.141,
"P_lrt": 1.129,
"P_CL": 1.081,
"P_CLkm": 1.008,
"SLR": 1.037
},
"var_z_mcse": {
"LR": 0.046,
"S_wald": 0.05,
"S_rwald": 0.048,
"S_score": 0.053,
"S_lrt": 0.052,
"S_CL": 0.05,
"S_CLkm": 0.048,
"P_wald": 0.052,
"P_rwald": 0.049,
"P_score": 0.054,
"P_lrt": 0.054,
"P_CL": 0.052,
"P_CLkm": 0.048,
"SLR": 0.049
},
"cl_var_reduction_pct": {
"S": 13,
"P": 15.3
},
"rej_pct_converged_only": {
"LR": 85.3,
"S_wald": 81.5,
"S_rwald": 80.2,
"S_score": 81.8,
"S_lrt": 81.7,
"S_CL": 83.9,
"S_CLkm": 82.5,
"P_wald": 83.3,
"P_rwald": 81.7,
"P_score": 83.3,
"P_lrt": 83.4,
"P_CL": 85.3,
"P_CLkm": 84,
"SLR": 76.8
},
"diff_vs_LR_pct": {
"S_wald": [
-3.8,
0.97
],
"S_rwald": [
-5.1,
0.98
],
"S_score": [
-3.5,
0.98
],
"S_lrt": [
-3.6,
0.97
],
"S_CL": [
-1.4,
0.9
],
"S_CLkm": [
-2.8,
0.95
],
"P_wald": [
-2,
1.01
],
"P_rwald": [
-3.6,
1.03
],
"P_score": [
-2,
1.01
],
"P_lrt": [
-1.9,
1
],
"P_CL": [
0,
0.96
],
"P_CLkm": [
-1.3,
0.97
],
"SLR": [
-8.5,
1.09
]
},
"size_adj_power_pct": {
"LR": 85.5,
"S_wald": 80.3,
"S_rwald": 79.3,
"S_score": 80.3,
"S_lrt": 80.3,
"S_CL": 82.9,
"S_CLkm": 82.9,
"P_wald": 82,
"P_rwald": 81.2,
"P_score": 82,
"P_lrt": 82,
"P_CL": 84.6,
"P_CLkm": 84.6,
"SLR": 77.4
},
"RE_cal": {
"S_wald": [
0.854,
0.828,
0.88
],
"S_rwald": [
0.839,
0.814,
0.865
],
"S_score": [
0.868,
0.842,
0.895
],
"S_lrt": [
0.866,
0.84,
0.893
],
"S_CL": [
0.935,
0.911,
0.96
],
"S_CLkm": [
0.935,
0.911,
0.96
],
"P_wald": [
0.867,
0.839,
0.896
],
"P_rwald": [
0.853,
0.825,
0.882
],
"P_score": [
0.882,
0.854,
0.912
],
"P_lrt": [
0.88,
0.852,
0.909
],
"P_CL": [
0.964,
0.936,
0.992
],
"P_CLkm": [
0.964,
0.936,
0.992
],
"SLR": [
0.818,
0.791,
0.846
]
},
"size_adj_power_pct_obs": {
"LR": 85.4,
"S_wald": 80.3,
"S_rwald": 79.4,
"S_score": 80.3,
"S_lrt": 80.3,
"S_CL": 82.7,
"S_CLkm": 82.7,
"P_wald": 81.9,
"P_rwald": 80.7,
"P_score": 82,
"P_lrt": 82,
"P_CL": 84.4,
"P_CLkm": 84.4,
"SLR": 75.2
},
"RE_cal_obs": {
"S_wald": [
0.87,
0.844,
0.898
],
"S_rwald": [
0.841,
0.815,
0.867
],
"S_score": [
0.886,
0.859,
0.914
],
"S_lrt": [
0.884,
0.857,
0.912
],
"S_CL": [
0.945,
0.92,
0.971
],
"S_CLkm": [
0.945,
0.92,
0.971
],
"P_wald": [
0.901,
0.871,
0.931
],
"P_rwald": [
0.859,
0.831,
0.889
],
"P_score": [
0.917,
0.887,
0.949
],
"P_lrt": [
0.915,
0.885,
0.946
],
"P_CL": [
0.975,
0.946,
1.005
],
"P_CLkm": [
0.975,
0.946,
1.005
],
"SLR": [
0.802,
0.775,
0.83
]
},
"size_adj_power_pct_trt": {
"LR": 85.7,
"S_wald": 80.1,
"S_rwald": 79.8,
"S_score": 80.1,
"S_lrt": 80.1,
"S_CL": 83.9,
"S_CLkm": 83.9,
"P_wald": 80.4,
"P_rwald": 80.6,
"P_score": 80.3,
"P_lrt": 80.3,
"P_CL": 84.4,
"P_CLkm": 84.4,
"SLR": 77.6
},
"RE_cal_trt": {
"S_wald": [
0.809,
0.785,
0.834
],
"S_rwald": [
0.809,
0.785,
0.835
],
"S_score": [
0.822,
0.797,
0.847
],
"S_lrt": [
0.819,
0.795,
0.845
],
"S_CL": [
0.913,
0.889,
0.937
],
"S_CLkm": [
0.913,
0.889,
0.937
],
"P_wald": [
0.814,
0.787,
0.843
],
"P_rwald": [
0.814,
0.787,
0.843
],
"P_score": [
0.827,
0.799,
0.856
],
"P_lrt": [
0.824,
0.797,
0.853
],
"P_CL": [
0.931,
0.903,
0.96
],
"P_CLkm": [
0.931,
0.903,
0.96
],
"SLR": [
0.799,
0.772,
0.826
]
},
"RE": {
"S_wald": [
0.904,
0.886,
0.922
],
"S_rwald": [
0.861,
0.844,
0.878
],
"S_score": [
0.925,
0.907,
0.944
],
"S_lrt": [
0.922,
0.904,
0.94
],
"S_CL": [
0.957,
0.941,
0.973
],
"S_CLkm": [
0.925,
0.909,
0.941
],
"P_wald": [
0.958,
0.938,
0.978
],
"P_rwald": [
0.899,
0.879,
0.918
],
"P_score": [
0.981,
0.96,
1.002
],
"P_lrt": [
0.976,
0.956,
0.997
],
"P_CL": [
1.021,
1.002,
1.041
],
"P_CLkm": [
0.953,
0.935,
0.971
],
"SLR": [
0.806,
0.787,
0.824
]
},
"decomp": {
"S": {
"est_ratio": 0.981,
"se_ratio": 1.017,
"empsd_ratio": 1.06
},
"P": {
"est_ratio": 1.027,
"se_ratio": 1.035,
"empsd_ratio": 1.113
}
},
"est": {
"U": {
"truth": -0.477,
"mean": -0.482,
"bias": -0.0059,
"emp_se": 0.1618,
"mod_se": 0.1629,
"rmse": 0.1618,
"cov_pct": 95.1,
"ratio_emp_mod": 0.993
},
"S": {
"truth": -0.47,
"mean": -0.473,
"bias": -0.0033,
"emp_se": 0.1715,
"mod_se": 0.1657,
"rmse": 0.1714,
"cov_pct": 93.8,
"ratio_emp_mod": 1.035,
"rob_se": 0.1699,
"rcov_pct": 94.5,
"ratio_emp_rob": 1.009
},
"P": {
"truth": -0.488,
"mean": -0.495,
"bias": -0.0073,
"emp_se": 0.18,
"mod_se": 0.1686,
"rmse": 0.1801,
"cov_pct": 93.1,
"ratio_emp_mod": 1.068,
"rob_se": 0.1742,
"rcov_pct": 93.7,
"ratio_emp_rob": 1.033
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
},
"315": {
"perm_obs": {
"R": 2000,
"events_mean": 190,
"epv_P": 19,
"rej_pct": {
"LR": 5.3,
"S_wald": 6.5,
"S_rwald": 6,
"S_score": 6.6,
"S_lrt": 6.6,
"S_CL": 5.3,
"S_CLkm": 5.1,
"P_wald": 6.3,
"P_rwald": 6.2,
"P_score": 6.5,
"P_lrt": 6.3,
"P_CL": 5.5,
"P_CLkm": 5.1,
"SLR": 5
},
"rej_mcse_pct": {
"LR": 0.5,
"S_wald": 0.55,
"S_rwald": 0.53,
"S_score": 0.55,
"S_lrt": 0.55,
"S_CL": 0.5,
"S_CLkm": 0.49,
"P_wald": 0.55,
"P_rwald": 0.54,
"P_score": 0.55,
"P_lrt": 0.55,
"P_CL": 0.51,
"P_CLkm": 0.49,
"SLR": 0.49
},
"mean_z": {
"LR": 0.028,
"S_wald": 0.033,
"S_rwald": 0.032,
"S_score": 0.033,
"S_lrt": 0.033,
"S_CL": 0.034,
"S_CLkm": 0.034,
"P_wald": 0.031,
"P_rwald": 0.031,
"P_score": 0.031,
"P_lrt": 0.031,
"P_CL": 0.036,
"P_CLkm": 0.035,
"SLR": 0.039
},
"var_z": {
"LR": 1.037,
"S_wald": 1.085,
"S_rwald": 1.061,
"S_score": 1.091,
"S_lrt": 1.089,
"S_CL": 1.037,
"S_CLkm": 1.004,
"P_wald": 1.115,
"P_rwald": 1.082,
"P_score": 1.121,
"P_lrt": 1.119,
"P_CL": 1.075,
"P_CLkm": 1.007,
"SLR": 1.016
},
"var_z_mcse": {
"LR": 0.032,
"S_wald": 0.035,
"S_rwald": 0.034,
"S_score": 0.035,
"S_lrt": 0.035,
"S_CL": 0.033,
"S_CLkm": 0.032,
"P_wald": 0.036,
"P_rwald": 0.035,
"P_score": 0.036,
"P_lrt": 0.036,
"P_CL": 0.034,
"P_CLkm": 0.032,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 13.3,
"P": 14.5
},
"rej_pct_converged_only": {
"LR": 5.3,
"S_wald": 6.5,
"S_rwald": 6,
"S_score": 6.6,
"S_lrt": 6.6,
"S_CL": 5.3,
"S_CLkm": 5.1,
"P_wald": 6.3,
"P_rwald": 6.2,
"P_score": 6.5,
"P_lrt": 6.3,
"P_CL": 5.5,
"P_CLkm": 5.1,
"SLR": 5
},
"est": {
"U": {
"truth": 0,
"mean": 0.004,
"bias": 0.0041,
"emp_se": 0.148,
"mod_se": 0.1454,
"rmse": 0.148,
"cov_pct": 94.8,
"ratio_emp_mod": 1.018
},
"S": {
"truth": 0,
"mean": 0.005,
"bias": 0.0048,
"emp_se": 0.1536,
"mod_se": 0.147,
"rmse": 0.1536,
"cov_pct": 93.5,
"ratio_emp_mod": 1.045,
"rob_se": 0.1488,
"rcov_pct": 94,
"ratio_emp_rob": 1.032
},
"P": {
"truth": 0,
"mean": 0.005,
"bias": 0.0047,
"emp_se": 0.1578,
"mod_se": 0.149,
"rmse": 0.1579,
"cov_pct": 93.7,
"ratio_emp_mod": 1.059,
"rob_se": 0.1514,
"rcov_pct": 93.8,
"ratio_emp_rob": 1.042
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
},
"600": {
"null_obs": {
"R": 5000,
"events_mean": 361.9,
"epv_P": 36.2,
"rej_pct": {
"LR": 5.1,
"S_wald": 5.9,
"S_rwald": 5.6,
"S_score": 5.9,
"S_lrt": 5.9,
"S_CL": 5.1,
"S_CLkm": 4.8,
"P_wald": 6,
"P_rwald": 5.7,
"P_score": 6,
"P_lrt": 6,
"P_CL": 5.8,
"P_CLkm": 5.4,
"SLR": 5.3
},
"rej_mcse_pct": {
"LR": 0.31,
"S_wald": 0.33,
"S_rwald": 0.33,
"S_score": 0.33,
"S_lrt": 0.33,
"S_CL": 0.31,
"S_CLkm": 0.3,
"P_wald": 0.34,
"P_rwald": 0.33,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.33,
"P_CLkm": 0.32,
"SLR": 0.32
},
"mean_z": {
"LR": -0.004,
"S_wald": 0.017,
"S_rwald": 0.016,
"S_score": 0.017,
"S_lrt": 0.017,
"S_CL": 0.013,
"S_CLkm": 0.012,
"P_wald": 0.02,
"P_rwald": 0.02,
"P_score": 0.02,
"P_lrt": 0.02,
"P_CL": 0.017,
"P_CLkm": 0.016,
"SLR": 0.015
},
"var_z": {
"LR": 0.996,
"S_wald": 1.035,
"S_rwald": 1.015,
"S_score": 1.038,
"S_lrt": 1.037,
"S_CL": 1.003,
"S_CLkm": 0.986,
"P_wald": 1.053,
"P_rwald": 1.028,
"P_score": 1.055,
"P_lrt": 1.054,
"P_CL": 1.025,
"P_CLkm": 0.991,
"SLR": 1.011
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.02,
"P_wald": 0.021,
"P_rwald": 0.02,
"P_score": 0.021,
"P_lrt": 0.021,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 13.8,
"P": 15.6
},
"rej_pct_converged_only": {
"LR": 5.1,
"S_wald": 5.9,
"S_rwald": 5.6,
"S_score": 5.9,
"S_lrt": 5.9,
"S_CL": 5.1,
"S_CLkm": 4.8,
"P_wald": 6,
"P_rwald": 5.7,
"P_score": 6,
"P_lrt": 6,
"P_CL": 5.8,
"P_CLkm": 5.4,
"SLR": 5.3
},
"null_crit_abs_z": {
"LR": 1.963,
"S_wald": 2.031,
"S_rwald": 2.009,
"S_score": 2.035,
"S_lrt": 2.033,
"S_CL": 1.962,
"S_CLkm": 1.945,
"P_wald": 2.048,
"P_rwald": 2.028,
"P_score": 2.052,
"P_lrt": 2.052,
"P_CL": 2.023,
"P_CLkm": 1.989,
"SLR": 1.988
},
"est": {
"U": {
"truth": 0,
"mean": 0,
"bias": -0.0004,
"emp_se": 0.1047,
"mod_se": 0.1053,
"rmse": 0.1047,
"cov_pct": 95.1,
"ratio_emp_mod": 0.995
},
"S": {
"truth": 0,
"mean": 0.002,
"bias": 0.0018,
"emp_se": 0.1079,
"mod_se": 0.1059,
"rmse": 0.1079,
"cov_pct": 94.1,
"ratio_emp_mod": 1.019,
"rob_se": 0.1069,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.009
},
"P": {
"truth": 0,
"mean": 0.002,
"bias": 0.0022,
"emp_se": 0.1096,
"mod_se": 0.1066,
"rmse": 0.1096,
"cov_pct": 94,
"ratio_emp_mod": 1.028,
"rob_se": 0.1079,
"rcov_pct": 94.3,
"ratio_emp_rob": 1.016
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"null_trt": {
"R": 5000,
"events_mean": 264.5,
"epv_P": 26.5,
"rej_pct": {
"LR": 5.7,
"S_wald": 6.3,
"S_rwald": 5.6,
"S_score": 6.4,
"S_lrt": 6.3,
"S_CL": 5.6,
"S_CLkm": 5.4,
"P_wald": 7,
"P_rwald": 6,
"P_score": 7.1,
"P_lrt": 7.1,
"P_CL": 5.7,
"P_CLkm": 5.3,
"SLR": 5.7
},
"rej_mcse_pct": {
"LR": 0.33,
"S_wald": 0.34,
"S_rwald": 0.33,
"S_score": 0.35,
"S_lrt": 0.34,
"S_CL": 0.33,
"S_CLkm": 0.32,
"P_wald": 0.36,
"P_rwald": 0.33,
"P_score": 0.36,
"P_lrt": 0.36,
"P_CL": 0.33,
"P_CLkm": 0.32,
"SLR": 0.33
},
"mean_z": {
"LR": 0.009,
"S_wald": 0.003,
"S_rwald": 0.003,
"S_score": 0.003,
"S_lrt": 0.003,
"S_CL": 0.004,
"S_CLkm": 0.004,
"P_wald": -0.001,
"P_rwald": 0,
"P_score": -0.001,
"P_lrt": -0.001,
"P_CL": -0.002,
"P_CLkm": -0.002,
"SLR": 0.007
},
"var_z": {
"LR": 1.035,
"S_wald": 1.129,
"S_rwald": 1.065,
"S_score": 1.134,
"S_lrt": 1.133,
"S_CL": 1.057,
"S_CLkm": 1.039,
"P_wald": 1.177,
"P_rwald": 1.088,
"P_score": 1.182,
"P_lrt": 1.181,
"P_CL": 1.082,
"P_CLkm": 1.046,
"SLR": 1.048
},
"var_z_mcse": {
"LR": 0.021,
"S_wald": 0.022,
"S_rwald": 0.021,
"S_score": 0.022,
"S_lrt": 0.022,
"S_CL": 0.021,
"S_CLkm": 0.02,
"P_wald": 0.023,
"P_rwald": 0.021,
"P_score": 0.023,
"P_lrt": 0.023,
"P_CL": 0.021,
"P_CLkm": 0.021,
"SLR": 0.021
},
"cl_var_reduction_pct": {
"S": 11.7,
"P": 15.2
},
"rej_pct_converged_only": {
"LR": 5.7,
"S_wald": 6.3,
"S_rwald": 5.6,
"S_score": 6.4,
"S_lrt": 6.3,
"S_CL": 5.6,
"S_CLkm": 5.4,
"P_wald": 7,
"P_rwald": 6,
"P_score": 7.1,
"P_lrt": 7.1,
"P_CL": 5.7,
"P_CLkm": 5.3,
"SLR": 5.7
},
"null_crit_abs_z": {
"LR": 2.026,
"S_wald": 2.067,
"S_rwald": 2.014,
"S_score": 2.073,
"S_lrt": 2.072,
"S_CL": 2.011,
"S_CLkm": 1.994,
"P_wald": 2.12,
"P_rwald": 2.035,
"P_score": 2.125,
"P_lrt": 2.125,
"P_CL": 2.029,
"P_CLkm": 1.995,
"SLR": 2.013
},
"est": {
"U": {
"truth": 0,
"mean": 0.001,
"bias": 0.0012,
"emp_se": 0.1252,
"mod_se": 0.1232,
"rmse": 0.1252,
"cov_pct": 94.3,
"ratio_emp_mod": 1.016
},
"S": {
"truth": 0,
"mean": 0,
"bias": 0.0004,
"emp_se": 0.1322,
"mod_se": 0.124,
"rmse": 0.1322,
"cov_pct": 93.7,
"ratio_emp_mod": 1.066,
"rob_se": 0.1277,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.035
},
"P": {
"truth": 0,
"mean": 0,
"bias": 0,
"emp_se": 0.1361,
"mod_se": 0.125,
"rmse": 0.1361,
"cov_pct": 93,
"ratio_emp_mod": 1.088,
"rob_se": 0.13,
"rcov_pct": 94,
"ratio_emp_rob": 1.047
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"null": {
"R": 5000,
"events_mean": 314.3,
"epv_P": 31.4,
"rej_pct": {
"LR": 4.6,
"S_wald": 5.4,
"S_rwald": 5,
"S_score": 5.5,
"S_lrt": 5.5,
"S_CL": 4.9,
"S_CLkm": 4.6,
"P_wald": 6,
"P_rwald": 5.3,
"P_score": 6.1,
"P_lrt": 6,
"P_CL": 4.9,
"P_CLkm": 4.6,
"SLR": 4.2
},
"rej_mcse_pct": {
"LR": 0.3,
"S_wald": 0.32,
"S_rwald": 0.31,
"S_score": 0.32,
"S_lrt": 0.32,
"S_CL": 0.3,
"S_CLkm": 0.3,
"P_wald": 0.34,
"P_rwald": 0.32,
"P_score": 0.34,
"P_lrt": 0.34,
"P_CL": 0.31,
"P_CLkm": 0.3,
"SLR": 0.28
},
"mean_z": {
"LR": -0.003,
"S_wald": 0,
"S_rwald": 0,
"S_score": 0,
"S_lrt": 0,
"S_CL": 0,
"S_CLkm": 0,
"P_wald": 0.006,
"P_rwald": 0.006,
"P_score": 0.006,
"P_lrt": 0.006,
"P_CL": 0.004,
"P_CLkm": 0.004,
"SLR": 0.001
},
"var_z": {
"LR": 0.99,
"S_wald": 1.059,
"S_rwald": 1.014,
"S_score": 1.062,
"S_lrt": 1.061,
"S_CL": 1.002,
"S_CLkm": 0.986,
"P_wald": 1.099,
"P_rwald": 1.035,
"P_score": 1.103,
"P_lrt": 1.102,
"P_CL": 1.023,
"P_CLkm": 0.989,
"SLR": 0.982
},
"var_z_mcse": {
"LR": 0.02,
"S_wald": 0.021,
"S_rwald": 0.02,
"S_score": 0.021,
"S_lrt": 0.021,
"S_CL": 0.02,
"S_CLkm": 0.02,
"P_wald": 0.022,
"P_rwald": 0.021,
"P_score": 0.022,
"P_lrt": 0.022,
"P_CL": 0.021,
"P_CLkm": 0.02,
"SLR": 0.02
},
"cl_var_reduction_pct": {
"S": 12.6,
"P": 14.3
},
"rej_pct_converged_only": {
"LR": 4.6,
"S_wald": 5.4,
"S_rwald": 5,
"S_score": 5.5,
"S_lrt": 5.5,
"S_CL": 4.9,
"S_CLkm": 4.6,
"P_wald": 6,
"P_rwald": 5.3,
"P_score": 6.1,
"P_lrt": 6,
"P_CL": 4.9,
"P_CLkm": 4.6,
"SLR": 4.2
},
"null_crit_abs_z": {
"LR": 1.933,
"S_wald": 2.008,
"S_rwald": 1.961,
"S_score": 2.012,
"S_lrt": 2.01,
"S_CL": 1.953,
"S_CLkm": 1.936,
"P_wald": 2.041,
"P_rwald": 1.976,
"P_score": 2.045,
"P_lrt": 2.044,
"P_CL": 1.958,
"P_CLkm": 1.925,
"SLR": 1.9
},
"est": {
"U": {
"truth": 0,
"mean": 0,
"bias": -0.0004,
"emp_se": 0.1122,
"mod_se": 0.113,
"rmse": 0.1122,
"cov_pct": 95.4,
"ratio_emp_mod": 0.993
},
"S": {
"truth": 0,
"mean": 0,
"bias": 0,
"emp_se": 0.1171,
"mod_se": 0.1137,
"rmse": 0.1171,
"cov_pct": 94.6,
"ratio_emp_mod": 1.03,
"rob_se": 0.1161,
"rcov_pct": 95,
"ratio_emp_rob": 1.008
},
"P": {
"truth": 0,
"mean": 0.001,
"bias": 0.0006,
"emp_se": 0.1202,
"mod_se": 0.1145,
"rmse": 0.1202,
"cov_pct": 94,
"ratio_emp_mod": 1.05,
"rob_se": 0.1181,
"rcov_pct": 94.7,
"ratio_emp_rob": 1.019
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"balanced": {
"R": 2000,
"events_mean": 311.8,
"epv_P": 31.2,
"rej_pct": {
"LR": 97.8,
"S_wald": 98.4,
"S_rwald": 98.1,
"S_score": 98.4,
"S_lrt": 98.4,
"S_CL": 98.6,
"S_CLkm": 98.6,
"P_wald": 98.4,
"P_rwald": 98.2,
"P_score": 98.4,
"P_lrt": 98.4,
"P_CL": 98.8,
"P_CLkm": 98.6,
"SLR": 96.8
},
"rej_mcse_pct": {
"LR": 0.33,
"S_wald": 0.28,
"S_rwald": 0.31,
"S_score": 0.28,
"S_lrt": 0.28,
"S_CL": 0.27,
"S_CLkm": 0.27,
"P_wald": 0.28,
"P_rwald": 0.3,
"P_score": 0.28,
"P_lrt": 0.28,
"P_CL": 0.25,
"P_CLkm": 0.27,
"SLR": 0.4
},
"mean_z": {
"LR": -3.936,
"S_wald": -4.159,
"S_rwald": -4.08,
"S_score": -4.204,
"S_lrt": -4.194,
"S_CL": -4.195,
"S_CLkm": -4.159,
"P_wald": -4.194,
"P_rwald": -4.089,
"P_score": -4.239,
"P_lrt": -4.23,
"P_CL": -4.227,
"P_CLkm": -4.156,
"SLR": -3.888
},
"var_z": {
"LR": 0.98,
"S_wald": 0.998,
"S_rwald": 0.971,
"S_score": 1.057,
"S_lrt": 1.044,
"S_CL": 0.999,
"S_CLkm": 0.982,
"P_wald": 1.025,
"P_rwald": 0.989,
"P_score": 1.086,
"P_lrt": 1.073,
"P_CL": 1.019,
"P_CLkm": 0.985,
"SLR": 1.014
},
"var_z_mcse": {
"LR": 0.031,
"S_wald": 0.031,
"S_rwald": 0.03,
"S_score": 0.033,
"S_lrt": 0.033,
"S_CL": 0.031,
"S_CLkm": 0.03,
"P_wald": 0.032,
"P_rwald": 0.031,
"P_score": 0.034,
"P_lrt": 0.034,
"P_CL": 0.032,
"P_CLkm": 0.031,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 12.3,
"P": 13.6
},
"rej_pct_converged_only": {
"LR": 97.8,
"S_wald": 98.4,
"S_rwald": 98.1,
"S_score": 98.4,
"S_lrt": 98.4,
"S_CL": 98.6,
"S_CLkm": 98.6,
"P_wald": 98.4,
"P_rwald": 98.2,
"P_score": 98.4,
"P_lrt": 98.4,
"P_CL": 98.8,
"P_CLkm": 98.6,
"SLR": 96.8
},
"diff_vs_LR_pct": {
"S_wald": [
0.6,
0.29
],
"S_rwald": [
0.4,
0.3
],
"S_score": [
0.7,
0.29
],
"S_lrt": [
0.7,
0.29
],
"S_CL": [
0.8,
0.28
],
"S_CLkm": [
0.8,
0.28
],
"P_wald": [
0.6,
0.27
],
"P_rwald": [
0.4,
0.28
],
"P_score": [
0.6,
0.27
],
"P_lrt": [
0.6,
0.27
],
"P_CL": [
1,
0.27
],
"P_CLkm": [
0.8,
0.26
],
"SLR": [
-1,
0.37
]
},
"size_adj_power_pct": {
"LR": 98,
"S_wald": 98.1,
"S_rwald": 98.1,
"S_score": 98.1,
"S_lrt": 98.1,
"S_CL": 98.7,
"S_CLkm": 98.7,
"P_wald": 98.1,
"P_rwald": 98.2,
"P_score": 98.1,
"P_lrt": 98.1,
"P_CL": 98.8,
"P_CLkm": 98.8,
"SLR": 97.3
},
"RE_cal": {
"S_wald": [
1.045,
1.02,
1.071
],
"S_rwald": [
1.049,
1.024,
1.075
],
"S_score": [
1.064,
1.038,
1.091
],
"S_lrt": [
1.06,
1.034,
1.086
],
"S_CL": [
1.122,
1.098,
1.147
],
"S_CLkm": [
1.122,
1.098,
1.147
],
"P_wald": [
1.023,
0.996,
1.05
],
"P_rwald": [
1.033,
1.006,
1.06
],
"P_score": [
1.041,
1.015,
1.069
],
"P_lrt": [
1.038,
1.011,
1.065
],
"P_CL": [
1.117,
1.091,
1.143
],
"P_CLkm": [
1.117,
1.091,
1.143
],
"SLR": [
0.985,
0.96,
1.011
]
},
"size_adj_power_pct_obs": {
"LR": 97.8,
"S_wald": 98,
"S_rwald": 97.9,
"S_score": 98,
"S_lrt": 98,
"S_CL": 98.6,
"S_CLkm": 98.6,
"P_wald": 98,
"P_rwald": 97.9,
"P_score": 98,
"P_lrt": 98,
"P_CL": 98.2,
"P_CLkm": 98.2,
"SLR": 96.5
},
"RE_cal_obs": {
"S_wald": [
1.074,
1.046,
1.102
],
"S_rwald": [
1.054,
1.027,
1.082
],
"S_score": [
1.094,
1.066,
1.123
],
"S_lrt": [
1.09,
1.062,
1.119
],
"S_CL": [
1.127,
1.102,
1.153
],
"S_CLkm": [
1.127,
1.102,
1.153
],
"P_wald": [
1.074,
1.044,
1.104
],
"P_rwald": [
1.045,
1.016,
1.075
],
"P_score": [
1.094,
1.063,
1.125
],
"P_lrt": [
1.09,
1.06,
1.122
],
"P_CL": [
1.12,
1.093,
1.148
],
"P_CLkm": [
1.12,
1.093,
1.148
],
"SLR": [
0.961,
0.935,
0.988
]
},
"size_adj_power_pct_trt": {
"LR": 97.2,
"S_wald": 97.8,
"S_rwald": 97.9,
"S_score": 97.8,
"S_lrt": 97.8,
"S_CL": 98.2,
"S_CLkm": 98.2,
"P_wald": 97.8,
"P_rwald": 97.9,
"P_score": 97.8,
"P_lrt": 97.8,
"P_CL": 98.2,
"P_CLkm": 98.2,
"SLR": 96.4
},
"RE_cal_trt": {
"S_wald": [
1.024,
1,
1.048
],
"S_rwald": [
1.045,
1.021,
1.07
],
"S_score": [
1.042,
1.018,
1.067
],
"S_lrt": [
1.038,
1.014,
1.062
],
"S_CL": [
1.112,
1.09,
1.136
],
"S_CLkm": [
1.112,
1.09,
1.136
],
"P_wald": [
0.999,
0.973,
1.026
],
"P_rwald": [
1.027,
1,
1.054
],
"P_score": [
1.016,
0.99,
1.044
],
"P_lrt": [
1.013,
0.986,
1.04
],
"P_CL": [
1.103,
1.078,
1.129
],
"P_CLkm": [
1.103,
1.078,
1.129
],
"SLR": [
0.964,
0.939,
0.989
]
},
"RE": {
"S_wald": [
1.117,
1.107,
1.127
],
"S_rwald": [
1.075,
1.065,
1.085
],
"S_score": [
1.141,
1.131,
1.152
],
"S_lrt": [
1.136,
1.126,
1.146
],
"S_CL": [
1.136,
1.127,
1.145
],
"S_CLkm": [
1.117,
1.108,
1.126
],
"P_wald": [
1.135,
1.125,
1.146
],
"P_rwald": [
1.08,
1.069,
1.09
],
"P_score": [
1.16,
1.149,
1.171
],
"P_lrt": [
1.155,
1.144,
1.166
],
"P_CL": [
1.154,
1.144,
1.163
],
"P_CLkm": [
1.115,
1.106,
1.125
],
"SLR": [
0.976,
0.966,
0.986
]
},
"decomp": {
"S": {
"est_ratio": 1.079,
"se_ratio": 1.009,
"empsd_ratio": 1.051
},
"P": {
"est_ratio": 1.097,
"se_ratio": 1.018,
"empsd_ratio": 1.075
}
},
"est": {
"U": {
"truth": -0.448,
"mean": -0.448,
"bias": 0.0006,
"emp_se": 0.1141,
"mod_se": 0.1149,
"rmse": 0.1141,
"cov_pct": 94.8,
"ratio_emp_mod": 0.994
},
"S": {
"truth": -0.483,
"mean": -0.483,
"bias": 0,
"emp_se": 0.12,
"mod_se": 0.1159,
"rmse": 0.12,
"cov_pct": 93.8,
"ratio_emp_mod": 1.035,
"rob_se": 0.1182,
"rcov_pct": 94.4,
"ratio_emp_rob": 1.015
},
"P": {
"truth": -0.489,
"mean": -0.491,
"bias": -0.0019,
"emp_se": 0.1227,
"mod_se": 0.1169,
"rmse": 0.1227,
"cov_pct": 93.4,
"ratio_emp_mod": 1.05,
"rob_se": 0.12,
"rcov_pct": 94,
"ratio_emp_rob": 1.023
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
},
"trial": {
"R": 1000,
"events_mean": 312.4,
"epv_P": 31.2,
"rej_pct": {
"LR": 98.9,
"S_wald": 97.8,
"S_rwald": 97.3,
"S_score": 97.8,
"S_lrt": 97.8,
"S_CL": 98.9,
"S_CLkm": 98.8,
"P_wald": 98.3,
"P_rwald": 97.8,
"P_score": 98.3,
"P_lrt": 98.3,
"P_CL": 99,
"P_CLkm": 98.7,
"SLR": 96.7
},
"rej_mcse_pct": {
"LR": 0.33,
"S_wald": 0.46,
"S_rwald": 0.51,
"S_score": 0.46,
"S_lrt": 0.46,
"S_CL": 0.33,
"S_CLkm": 0.34,
"P_wald": 0.41,
"P_rwald": 0.46,
"P_score": 0.41,
"P_lrt": 0.41,
"P_CL": 0.31,
"P_CLkm": 0.36,
"SLR": 0.56
},
"mean_z": {
"LR": -4.167,
"S_wald": -3.994,
"S_rwald": -3.884,
"S_score": -4.035,
"S_lrt": -4.028,
"S_CL": -4.067,
"S_CLkm": -4.033,
"P_wald": -4.118,
"P_rwald": -3.976,
"P_score": -4.16,
"P_lrt": -4.153,
"P_CL": -4.173,
"P_CLkm": -4.103,
"SLR": -3.735
},
"var_z": {
"LR": 0.946,
"S_wald": 0.996,
"S_rwald": 0.951,
"S_score": 1.05,
"S_lrt": 1.04,
"S_CL": 0.98,
"S_CLkm": 0.964,
"P_wald": 1.049,
"P_rwald": 0.99,
"P_score": 1.109,
"P_lrt": 1.096,
"P_CL": 1.008,
"P_CLkm": 0.974,
"SLR": 0.984
},
"var_z_mcse": {
"LR": 0.042,
"S_wald": 0.044,
"S_rwald": 0.042,
"S_score": 0.046,
"S_lrt": 0.046,
"S_CL": 0.044,
"S_CLkm": 0.043,
"P_wald": 0.047,
"P_rwald": 0.044,
"P_score": 0.05,
"P_lrt": 0.049,
"P_CL": 0.045,
"P_CLkm": 0.044,
"SLR": 0.044
},
"cl_var_reduction_pct": {
"S": 12.3,
"P": 13.8
},
"rej_pct_converged_only": {
"LR": 98.9,
"S_wald": 97.8,
"S_rwald": 97.3,
"S_score": 97.8,
"S_lrt": 97.8,
"S_CL": 98.9,
"S_CLkm": 98.8,
"P_wald": 98.3,
"P_rwald": 97.8,
"P_score": 98.3,
"P_lrt": 98.3,
"P_CL": 99,
"P_CLkm": 98.7,
"SLR": 96.7
},
"diff_vs_LR_pct": {
"S_wald": [
-1.1,
0.41
],
"S_rwald": [
-1.6,
0.47
],
"S_score": [
-1.1,
0.41
],
"S_lrt": [
-1.1,
0.41
],
"S_CL": [
0,
0.35
],
"S_CLkm": [
-0.1,
0.33
],
"P_wald": [
-0.6,
0.4
],
"P_rwald": [
-1.1,
0.41
],
"P_score": [
-0.6,
0.4
],
"P_lrt": [
-0.6,
0.4
],
"P_CL": [
0.1,
0.3
],
"P_CLkm": [
-0.2,
0.35
],
"SLR": [
-2.2,
0.52
]
},
"size_adj_power_pct": {
"LR": 99,
"S_wald": 97.5,
"S_rwald": 97.3,
"S_score": 97.5,
"S_lrt": 97.6,
"S_CL": 98.9,
"S_CLkm": 98.9,
"P_wald": 97.9,
"P_rwald": 97.8,
"P_score": 97.9,
"P_lrt": 97.9,
"P_CL": 99.1,
"P_CLkm": 99.1,
"SLR": 97
},
"RE_cal": {
"S_wald": [
0.859,
0.837,
0.883
],
"S_rwald": [
0.848,
0.826,
0.871
],
"S_score": [
0.874,
0.851,
0.897
],
"S_lrt": [
0.872,
0.849,
0.895
],
"S_CL": [
0.941,
0.919,
0.963
],
"S_CLkm": [
0.941,
0.919,
0.963
],
"P_wald": [
0.879,
0.855,
0.905
],
"P_rwald": [
0.871,
0.847,
0.896
],
"P_score": [
0.895,
0.87,
0.92
],
"P_lrt": [
0.892,
0.867,
0.918
],
"P_CL": [
0.971,
0.947,
0.995
],
"P_CLkm": [
0.971,
0.947,
0.995
],
"SLR": [
0.811,
0.788,
0.834
]
},
"size_adj_power_pct_obs": {
"LR": 98.9,
"S_wald": 97.3,
"S_rwald": 97.1,
"S_score": 97.3,
"S_lrt": 97.3,
"S_CL": 98.9,
"S_CLkm": 98.9,
"P_wald": 97.9,
"P_rwald": 97.6,
"P_score": 97.9,
"P_lrt": 97.9,
"P_CL": 98.7,
"P_CLkm": 98.7,
"SLR": 96.6
},
"RE_cal_obs": {
"S_wald": [
0.883,
0.859,
0.908
],
"S_rwald": [
0.852,
0.829,
0.876
],
"S_score": [
0.899,
0.874,
0.924
],
"S_lrt": [
0.897,
0.872,
0.922
],
"S_CL": [
0.945,
0.922,
0.968
],
"S_CLkm": [
0.945,
0.922,
0.968
],
"P_wald": [
0.923,
0.896,
0.951
],
"P_rwald": [
0.881,
0.855,
0.908
],
"P_score": [
0.94,
0.912,
0.969
],
"P_lrt": [
0.937,
0.909,
0.966
],
"P_CL": [
0.974,
0.949,
0.999
],
"P_CLkm": [
0.974,
0.949,
0.999
],
"SLR": [
0.791,
0.768,
0.815
]
},
"size_adj_power_pct_trt": {
"LR": 98.9,
"S_wald": 97.2,
"S_rwald": 97.1,
"S_score": 97.2,
"S_lrt": 97.2,
"S_CL": 98.7,
"S_CLkm": 98.7,
"P_wald": 97.3,
"P_rwald": 97.6,
"P_score": 97.3,
"P_lrt": 97.3,
"P_CL": 98.7,
"P_CLkm": 98.7,
"SLR": 96.4
},
"RE_cal_trt": {
"S_wald": [
0.842,
0.821,
0.864
],
"S_rwald": [
0.845,
0.823,
0.867
],
"S_score": [
0.856,
0.834,
0.878
],
"S_lrt": [
0.854,
0.832,
0.876
],
"S_CL": [
0.933,
0.912,
0.954
],
"S_CLkm": [
0.933,
0.912,
0.954
],
"P_wald": [
0.859,
0.835,
0.884
],
"P_rwald": [
0.866,
0.841,
0.891
],
"P_score": [
0.873,
0.848,
0.899
],
"P_lrt": [
0.87,
0.846,
0.896
],
"P_CL": [
0.959,
0.935,
0.983
],
"P_CLkm": [
0.959,
0.935,
0.983
],
"SLR": [
0.793,
0.771,
0.816
]
},
"RE": {
"S_wald": [
0.919,
0.906,
0.931
],
"S_rwald": [
0.869,
0.857,
0.881
],
"S_score": [
0.937,
0.924,
0.95
],
"S_lrt": [
0.934,
0.922,
0.947
],
"S_CL": [
0.952,
0.941,
0.963
],
"S_CLkm": [
0.936,
0.926,
0.947
],
"P_wald": [
0.976,
0.962,
0.99
],
"P_rwald": [
0.91,
0.897,
0.924
],
"P_score": [
0.997,
0.982,
1.011
],
"P_lrt": [
0.993,
0.979,
1.007
],
"P_CL": [
1.003,
0.99,
1.015
],
"P_CLkm": [
0.969,
0.957,
0.981
],
"SLR": [
0.803,
0.791,
0.816
]
},
"decomp": {
"S": {
"est_ratio": 0.981,
"se_ratio": 1.011,
"empsd_ratio": 1.066
},
"P": {
"est_ratio": 1.021,
"se_ratio": 1.02,
"empsd_ratio": 1.106
}
},
"est": {
"U": {
"truth": -0.477,
"mean": -0.474,
"bias": 0.0029,
"emp_se": 0.112,
"mod_se": 0.1149,
"rmse": 0.1119,
"cov_pct": 95.6,
"ratio_emp_mod": 0.974
},
"S": {
"truth": -0.47,
"mean": -0.464,
"bias": 0.0053,
"emp_se": 0.1193,
"mod_se": 0.1161,
"rmse": 0.1194,
"cov_pct": 94.3,
"ratio_emp_mod": 1.028,
"rob_se": 0.1194,
"rcov_pct": 95.2,
"ratio_emp_rob": 0.999
},
"P": {
"truth": -0.488,
"mean": -0.483,
"bias": 0.0048,
"emp_se": 0.1238,
"mod_se": 0.1172,
"rmse": 0.1239,
"cov_pct": 94,
"ratio_emp_mod": 1.056,
"rob_se": 0.1214,
"rcov_pct": 95.3,
"ratio_emp_rob": 1.02
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
},
"619": {
"perm": {
"R": 2000,
"events_mean": 324,
"epv_P": 32.4,
"rej_pct": {
"LR": 4.5,
"S_wald": 5.3,
"S_rwald": 5.1,
"S_score": 5.4,
"S_lrt": 5.4,
"S_CL": 5,
"S_CLkm": 4.8,
"P_wald": 5.9,
"P_rwald": 5.5,
"P_score": 5.9,
"P_lrt": 5.9,
"P_CL": 5.1,
"P_CLkm": 4.8,
"SLR": 5.4
},
"rej_mcse_pct": {
"LR": 0.46,
"S_wald": 0.5,
"S_rwald": 0.49,
"S_score": 0.51,
"S_lrt": 0.51,
"S_CL": 0.49,
"S_CLkm": 0.48,
"P_wald": 0.52,
"P_rwald": 0.51,
"P_score": 0.53,
"P_lrt": 0.53,
"P_CL": 0.49,
"P_CLkm": 0.48,
"SLR": 0.51
},
"mean_z": {
"LR": -0.015,
"S_wald": -0.019,
"S_rwald": -0.019,
"S_score": -0.019,
"S_lrt": -0.019,
"S_CL": -0.018,
"S_CLkm": -0.017,
"P_wald": -0.02,
"P_rwald": -0.02,
"P_score": -0.02,
"P_lrt": -0.021,
"P_CL": -0.016,
"P_CLkm": -0.016,
"SLR": -0.018
},
"var_z": {
"LR": 0.988,
"S_wald": 1.08,
"S_rwald": 1.03,
"S_score": 1.083,
"S_lrt": 1.082,
"S_CL": 1.008,
"S_CLkm": 0.991,
"P_wald": 1.113,
"P_rwald": 1.041,
"P_score": 1.117,
"P_lrt": 1.116,
"P_CL": 1.025,
"P_CLkm": 0.992,
"SLR": 1.022
},
"var_z_mcse": {
"LR": 0.031,
"S_wald": 0.035,
"S_rwald": 0.033,
"S_score": 0.035,
"S_lrt": 0.035,
"S_CL": 0.032,
"S_CLkm": 0.031,
"P_wald": 0.036,
"P_rwald": 0.034,
"P_score": 0.036,
"P_lrt": 0.036,
"P_CL": 0.033,
"P_CLkm": 0.032,
"SLR": 0.033
},
"cl_var_reduction_pct": {
"S": 12,
"P": 13
},
"rej_pct_converged_only": {
"LR": 4.5,
"S_wald": 5.3,
"S_rwald": 5.1,
"S_score": 5.4,
"S_lrt": 5.4,
"S_CL": 5,
"S_CLkm": 4.8,
"P_wald": 5.9,
"P_rwald": 5.5,
"P_score": 5.9,
"P_lrt": 5.9,
"P_CL": 5.1,
"P_CLkm": 4.8,
"SLR": 5.4
},
"est": {
"U": {
"truth": 0,
"mean": -0.002,
"bias": -0.0017,
"emp_se": 0.1105,
"mod_se": 0.1112,
"rmse": 0.1105,
"cov_pct": 95.5,
"ratio_emp_mod": 0.994
},
"S": {
"truth": 0,
"mean": -0.002,
"bias": -0.0022,
"emp_se": 0.1165,
"mod_se": 0.1119,
"rmse": 0.1165,
"cov_pct": 94.7,
"ratio_emp_mod": 1.041,
"rob_se": 0.1146,
"rcov_pct": 95,
"ratio_emp_rob": 1.016
},
"P": {
"truth": 0,
"mean": -0.002,
"bias": -0.0024,
"emp_se": 0.1192,
"mod_se": 0.1127,
"rmse": 0.1192,
"cov_pct": 94.2,
"ratio_emp_mod": 1.057,
"rob_se": 0.1166,
"rcov_pct": 94.5,
"ratio_emp_rob": 1.022
}
},
"sep_flag_S": 0,
"sep_flag_P": 0,
"nonconverged": 0
}
}
}
},
"h1_RE_S_wald_balanced": [
1.055,
1.105,
1.151,
1.066,
1.091,
1.117
],
"h1_typeI_S_wald_null_pct": [
6.3,
5.7,
5.7,
6.6,
5.7,
5.4
],
"h1_supported": false,
"h2_supported": {
"S": true,
"P": true
},
"h3_status": "refuted",
"wald_inflated_cells": [
[
"OS",
"150",
"S"
],
[
"OS",
"150",
"P"
],
[
"RFS",
"150",
"S"
],
[
"RFS",
"150",
"P"
],
[
"RFS",
"300",
"P"
]
],
"single_arm_rules": {
"null_obs": {
"typeI_S_wald_pct": [
5.8,
5.7,
5.8,
5.6,
5.5,
5.9
],
"h1_supported": false,
"wald_inflated_cells": [
[
"OS",
"150",
"P"
],
[
"OS",
"600",
"P"
],
[
"RFS",
"150",
"P"
]
],
"h3_status": "refuted"
},
"null_trt": {
"typeI_S_wald_pct": [
6.1,
5.5,
6.3,
6.4,
5.9,
6.3
],
"h1_supported": false,
"wald_inflated_cells": [
[
"OS",
"150",
"S"
],
[
"OS",
"150",
"P"
],
[
"OS",
"600",
"S"
],
[
"OS",
"600",
"P"
],
[
"RFS",
"150",
"S"
],
[
"RFS",
"150",
"P"
],
[
"RFS",
"300",
"P"
],
[
"RFS",
"600",
"S"
],
[
"RFS",
"600",
"P"
]
],
"h3_status": "refuted"
}
},
"headline": {
"typeI_range_pct": {
"null": {
"LR": [
4.6,
5.3
],
"S_wald": [
5.4,
6.6
],
"S_rwald": [
5,
6.3
],
"S_score": [
5.5,
6.9
],
"S_lrt": [
5.5,
6.9
],
"S_CL": [
4.9,
5.8
],
"S_CLkm": [
4.6,
5.1
],
"P_wald": [
5.9,
7.3
],
"P_rwald": [
5.1,
6.7
],
"P_score": [
5.9,
7.6
],
"P_lrt": [
5.9,
7.6
],
"P_CL": [
4.9,
7.2
],
"P_CLkm": [
4.6,
5.5
],
"SLR": [
4.2,
5.6
]
},
"null_obs": {
"LR": [
4.7,
5.2
],
"S_wald": [
5.5,
5.9
],
"S_rwald": [
5.3,
5.6
],
"S_score": [
5.5,
6
],
"S_lrt": [
5.5,
6
],
"S_CL": [
5.1,
5.8
],
"S_CLkm": [
4.8,
5.3
],
"P_wald": [
5.6,
6.9
],
"P_rwald": [
5.4,
6.6
],
"P_score": [
5.7,
7.1
],
"P_lrt": [
5.7,
7
],
"P_CL": [
5.4,
6.8
],
"P_CLkm": [
4.6,
5.4
],
"SLR": [
4.8,
5.4
]
},
"null_trt": {
"LR": [
4.2,
5.7
],
"S_wald": [
5.5,
6.4
],
"S_rwald": [
5.2,
5.9
],
"S_score": [
5.6,
6.6
],
"S_lrt": [
5.6,
6.7
],
"S_CL": [
5,
6
],
"S_CLkm": [
4.5,
5.9
],
"P_wald": [
6,
7.3
],
"P_rwald": [
5.2,
6.7
],
"P_score": [
6.2,
7.4
],
"P_lrt": [
6.1,
7.5
],
"P_CL": [
5.4,
7.2
],
"P_CLkm": [
4.6,
5.5
],
"SLR": [
4.6,
5.9
]
}
},
"var_z_range": {
"null": {
"LR": [
0.99,
1.028
],
"S_wald": [
1.049,
1.114
],
"S_rwald": [
1.01,
1.093
],
"S_score": [
1.056,
1.129
],
"S_lrt": [
1.054,
1.126
],
"S_CL": [
1.002,
1.088
],
"S_CLkm": [
0.979,
1.015
],
"P_wald": [
1.094,
1.192
],
"P_rwald": [
1.034,
1.146
],
"P_score": [
1.101,
1.209
],
"P_lrt": [
1.099,
1.206
],
"P_CL": [
1.023,
1.197
],
"P_CLkm": [
0.98,
1.037
],
"SLR": [
0.975,
1.037
]
},
"null_obs": {
"LR": [
0.986,
1.019
],
"S_wald": [
1.035,
1.065
],
"S_rwald": [
1.009,
1.06
],
"S_score": [
1.038,
1.076
],
"S_lrt": [
1.037,
1.073
],
"S_CL": [
1.003,
1.055
],
"S_CLkm": [
0.979,
0.998
],
"P_wald": [
1.053,
1.141
],
"P_rwald": [
1.025,
1.134
],
"P_score": [
1.055,
1.153
],
"P_lrt": [
1.054,
1.149
],
"P_CL": [
1.024,
1.167
],
"P_CLkm": [
0.984,
1.011
],
"SLR": [
1.003,
1.023
]
},
"null_trt": {
"LR": [
0.958,
1.035
],
"S_wald": [
1.055,
1.129
],
"S_rwald": [
1.014,
1.065
],
"S_score": [
1.063,
1.134
],
"S_lrt": [
1.063,
1.133
],
"S_CL": [
1.004,
1.062
],
"S_CLkm": [
0.97,
1.04
],
"P_wald": [
1.104,
1.177
],
"P_rwald": [
1.045,
1.098
],
"P_score": [
1.113,
1.182
],
"P_lrt": [
1.113,
1.181
],
"P_CL": [
1.05,
1.161
],
"P_CLkm": [
0.98,
1.047
],
"SLR": [
0.966,
1.048
]
}
},
"RE_range": {
"S_wald": [
1.055,
1.151
],
"S_rwald": [
1.008,
1.093
],
"S_score": [
1.081,
1.168
],
"S_lrt": [
1.08,
1.166
],
"S_CL": [
1.125,
1.149
],
"S_CLkm": [
1.061,
1.117
],
"P_wald": [
1.017,
1.169
],
"P_rwald": [
0.954,
1.099
],
"P_score": [
1.041,
1.186
],
"P_lrt": [
1.041,
1.185
],
"P_CL": [
1.148,
1.205
],
"P_CLkm": [
1.039,
1.115
],
"SLR": [
0.808,
0.976
]
},
"RE_cal_range": {
"pooled": {
"S_wald": [
0.97,
1.083
],
"S_rwald": [
0.954,
1.078
],
"S_score": [
0.979,
1.094
],
"S_lrt": [
0.98,
1.094
],
"S_CL": [
1.075,
1.122
],
"S_CLkm": [
1.075,
1.122
],
"P_wald": [
0.874,
1.063
],
"P_rwald": [
0.862,
1.065
],
"P_score": [
0.882,
1.075
],
"P_lrt": [
0.883,
1.075
],
"P_CL": [
1.028,
1.117
],
"P_CLkm": [
1.028,
1.117
],
"SLR": [
0.801,
0.985
]
},
"obs": {
"S_wald": [
1.004,
1.082
],
"S_rwald": [
0.984,
1.069
],
"S_score": [
1.017,
1.094
],
"S_lrt": [
1.019,
1.094
],
"S_CL": [
1.095,
1.127
],
"S_CLkm": [
1.095,
1.127
],
"P_wald": [
0.914,
1.084
],
"P_rwald": [
0.882,
1.058
],
"P_score": [
0.925,
1.096
],
"P_lrt": [
0.928,
1.097
],
"P_CL": [
1.038,
1.12
],
"P_CLkm": [
1.038,
1.12
],
"SLR": [
0.816,
0.961
]
},
"trt": {
"S_wald": [
0.972,
1.075
],
"S_rwald": [
0.96,
1.067
],
"S_score": [
0.984,
1.086
],
"S_lrt": [
0.97,
1.085
],
"S_CL": [
1.07,
1.112
],
"S_CLkm": [
1.07,
1.112
],
"P_wald": [
0.891,
1.055
],
"P_rwald": [
0.88,
1.052
],
"P_score": [
0.896,
1.066
],
"P_lrt": [
0.893,
1.066
],
"P_CL": [
1.031,
1.103
],
"P_CLkm": [
1.031,
1.103
],
"SLR": [
0.807,
0.964
]
}
},
"event_saving_pct_obs": {
"S_wald": [
0.4,
4.9,
7.6,
1,
4.9,
6.9
],
"S_rwald": [
-1.6,
3.3,
6.5,
-1.3,
2.9,
5.1
],
"S_score": [
1.7,
5.9,
8.6,
2.9,
6.6,
8.6
],
"S_lrt": [
1.9,
6,
8.6,
2.7,
6.4,
8.3
],
"S_CL": [
8.8,
9.8,
9.7,
8.7,
11,
11.3
],
"S_CLkm": [
8.8,
9.8,
9.7,
8.7,
11,
11.3
],
"P_wald": [
-9.4,
3.3,
7.7,
-6.3,
3.6,
6.9
],
"P_rwald": [
-13.4,
0.2,
5.5,
-9.8,
0.5,
4.3
],
"P_score": [
-8.1,
4.4,
8.8,
-4.3,
5.3,
8.6
],
"P_lrt": [
-7.8,
4.5,
8.8,
-4.4,
5,
8.3
],
"P_CL": [
5.2,
9.8,
9.5,
3.7,
10,
10.7
],
"P_CLkm": [
5.2,
9.8,
9.5,
3.7,
10,
10.7
],
"SLR": [
-22.5,
-16.4,
-15.6,
-10.3,
-5,
-4.1
]
}
},
"runtime_s": 98.3,
"hypothesis_supported": false
}Experiment script · experiment_v1.py
"""Cox adjustment vs log-rank: real-data resampling of the NCCTG colon trial (Obs vs Lev+5FU, 619 patients).
Worlds: null (both arms drawn from the pooled 619), balanced (arms drawn with entropy-balancing weights that
equalise covariate means), trial (arms drawn uniformly, keeping the trial's chance imbalance), all with n/2 per arm;
perm (the actual 619 patients re-randomised 304:315, no resampling).
Revision 1 adds single-arm null worlds (null_obs / null_trt: both arms drawn from the 315 observation or the 304
Lev+5FU patients only, so no treatment-effect mixture), perm_obs (the 315 observation patients split at random
157:158, no resampling) and the stratified log-rank test (strata node4 x extent x surg) in every world. The new
worlds draw from their own random streams, so every draw of the original worlds is unchanged."""
import json, math, time
import numpy as np
from dataset import load
T0 = time.time()
SEED, SIZES, ZC, BLOCK = 20261024, (150, 300, 600), 1.959963985, 250
REPS = {'null': 5000, 'balanced': 2000, 'trial': 1000, 'perm': 2000, 'null_obs': 5000, 'null_trt': 5000, 'perm_obs': 2000}
rng = np.random.default_rng(SEED) # original stream, original order
RNG = {w: np.random.default_rng([SEED, k]) for k, w in enumerate(('null_obs', 'null_trt', 'perm_obs'), 1)}
NULLW, PERMW = ('null', 'null_obs', 'null_trt'), ('perm', 'perm_obs')
# ---------------- data: inspect, filter, endpoints, covariates ----------------
df = load()
print('loaded', df.shape, 'missing:', {k: int(v) for k, v in df.isna().sum().items() if v})
d = df[df.etype == 2].set_index('id') # death record, one per patient
r = df[df.etype == 1].set_index('id').loc[d.index] # recurrence record, same patients
assert d.index.is_unique and len(d) == 929
keep = d.rx.isin(['Obs', 'Lev+5FU']).to_numpy() # levamisole-alone arm excluded
d, r = d[keep], r[keep]
N = len(d); trt = (d.rx == 'Lev+5FU').to_numpy(int)
td, sd = d.time.to_numpy(float), d.status.to_numpy(int)
tr, sr = r.time.to_numpy(float), r.status.to_numpy(int)
t_rfs = np.minimum(tr, td); e_rfs = ((sr == 1) | ((sd == 1) & (td <= tr))).astype(int)
ENDP = {'OS': (td, sd), 'RFS': (t_rfs, e_rfs)}
NAMES = ['node4', 'extent_1_2', 'extent_4', 'surg', 'obstruct', 'adhere', 'poor_diff', 'age10', 'male']
COV = np.column_stack([d.node4, d.extent <= 2, d.extent == 4, d.surg, d.obstruct, d.adhere,
d.differ == 3, (d.age - d.age.mean()) / 10, d.sex]).astype(float) # differ NaN -> not poor
SETS = {'S': [0, 1, 2, 3], 'P': list(range(9))} # S = candidate stratification factors (node4, extent, surg)
data_info = {'N': N, 'n_trt': int(trt.sum()), 'n_ctl': int(N - trt.sum()), 'differ_missing_imputed': int(d.differ.isna().sum()),
'perfor_pct_excluded': round(100 * float(d.perfor.mean()), 1), 'median_age': float(d.age.median())}
for e, (t, s) in ENDP.items():
data_info[e] = {'events': int(s.sum()), 'events_trt': int(s[trt == 1].sum()), 'events_ctl': int(s[trt == 0].sum()),
'censored_pct': round(100 * float(1 - s.mean()), 1), 'median_fu_censored_days': float(np.median(t[s == 0]))}
data_info['prevalence_pct'] = {k: round(100 * float(COV[:, i].mean()), 1) for i, k in enumerate(NAMES) if k != 'age10'}
EXT3 = np.where(COV[:, 1] == 1, 0, np.where(COV[:, 2] == 1, 2, 1)) # submucosa/muscle, serosa, contiguous
STRAT = (6 * COV[:, 0] + 2 * EXT3 + COV[:, 3]).astype(int) # 12 strata: node4 x extent x surg
data_info['strata'] = {'n_strata': 12, 'sizes_sorted': sorted(np.bincount(STRAT, minlength=12).tolist()),
'occupied_ctl': int((np.bincount(STRAT[trt == 0], minlength=12) > 0).sum()),
'occupied_trt': int((np.bincount(STRAT[trt == 1], minlength=12) > 0).sum())}
print('analysis population', data_info)
# ---------------- entropy-balancing weights: each arm reweighted to the pooled covariate means ----------------
def ebal(X, target):
lam, Xc = np.zeros(X.shape[1]), X - target
for _ in range(100):
s = Xc @ lam; w = np.exp(s - s.max()); w /= w.sum()
g = w @ Xc
if np.abs(g).max() < 1e-12: break
lam -= np.linalg.solve((Xc * w[:, None]).T @ Xc - np.outer(g, g), g)
return w
W, IDX = np.zeros(N), {a: np.flatnonzero(trt == a) for a in (0, 1)}
for a in (0, 1):
W[IDX[a]] = ebal(COV[IDX[a]], COV.mean(0))
def smd(w1, w0):
m1, m0 = w1 @ COV[IDX[1]], w0 @ COV[IDX[0]]
sdp = np.sqrt((COV[IDX[1]].var(0) + COV[IDX[0]].var(0)) / 2)
return (m1 - m0) / sdp
u1, u0 = np.full(len(IDX[1]), 1 / len(IDX[1])), np.full(len(IDX[0]), 1 / len(IDX[0]))
balance = {'smd_before': {k: round(float(v), 3) for k, v in zip(NAMES, smd(u1, u0))},
'max_abs_smd_after': float('%.1e' % np.abs(smd(W[IDX[1]], W[IDX[0]])).max()),
'weight_ratio_range': [round(float(min((W[IDX[a]] * len(IDX[a])).min() for a in (0, 1))), 3),
round(float(max((W[IDX[a]] * len(IDX[a])).max() for a in (0, 1))), 3)],
'ess_trt': round(float(1 / (W[IDX[1]] ** 2).sum()), 1), 'ess_ctl': round(float(1 / (W[IDX[0]] ** 2).sum()), 1)}
print('balance', balance)
def draw(world, n, B):
h = n // 2
if world == 'perm': # re-randomisation of the real trial, arm sizes kept
z = np.zeros((B, N)); z[np.arange(B)[:, None], rng.random((B, N)).argsort(1)[:, :len(IDX[1])]] = 1
return np.broadcast_to(np.arange(N), (B, N)).copy(), z
if world == 'perm_obs': # observation arm alone split at random 157:158 (n = 315)
m = len(IDX[0]); z = np.zeros((B, m))
z[np.arange(B)[:, None], RNG[world].random((B, m)).argsort(1)[:, :m // 2]] = 1
return np.broadcast_to(IDX[0], (B, m)).copy(), z
if world == 'null':
idx = rng.integers(0, N, (B, n)) # exact null: arm labels independent of everything
elif world in ('null_obs', 'null_trt'): # single-arm null: both arms from one arm's patients
pool = IDX[0] if world == 'null_obs' else IDX[1]
idx = pool[RNG[world].integers(0, len(pool), (B, n))]
else:
p = (lambda a: W[IDX[a]]) if world == 'balanced' else (lambda a: None)
idx = np.concatenate([rng.choice(IDX[1], (B, h), p=p(1)), rng.choice(IDX[0], (B, h), p=p(0))], 1)
z = np.zeros((B, n)); z[:, :h] = 1
return idx, z
# ---------------- batched survival machinery (risk sets via sorted cumulative sums, Breslow ties) ----------------
def csum(a): # cumulative sum along axis 1, added in np.cumsum's order (a slice loop is faster in 3-D)
if a.ndim == 2: return np.cumsum(a, 1)
o = np.empty(a.shape); acc = a[:, 0].copy(); o[:, 0] = acc
for j in range(1, a.shape[1]): acc = acc + a[:, j]; o[:, j] = acc
return o
def at(a, i): # a[b, i[b, j]] along axis 1 (same values as take_along_axis, faster)
B, m = a.shape[:2]; flat = (i + (np.arange(B) * m)[:, None]).ravel()
if a.ndim == 2: return a.reshape(B * m, -1)[flat].reshape(i.shape)
return np.take(a.reshape(B * m, a.shape[2]), flat, axis=0).reshape(i.shape + a.shape[2:])
def rcs_at(a, i): return at(csum(a[:, ::-1]), a.shape[1] - 1 - i) # reverse cumulative sum along axis 1, read at i
def ties(T):
n = T.shape[1]; ar = np.broadcast_to(np.arange(n), T.shape)
new = np.ones(T.shape, bool); new[:, 1:] = T[:, 1:] != T[:, :-1]
end = np.ones(T.shape, bool); end[:, :-1] = new[:, 1:]
return (np.maximum.accumulate(np.where(new, ar, 0), 1),
np.flip(np.minimum.accumulate(np.flip(np.where(end, ar, n - 1), 1), 1), 1))
def cox_eval(X, D, c, f, l, beta, robust=False):
eta = np.einsum('bnp,bp->bn', X, beta); eta = eta - eta.max(1, keepdims=True)
w = c * np.exp(eta); Dc = c * D
S0 = rcs_at(w, f); xb = rcs_at(w[..., None] * X, f) / S0[..., None]
g = np.einsum('bn,bnp->bp', Dc, X - xb)
Lam = at(np.cumsum(Dc / S0, 1), l) # Breslow cumulative baseline hazard at own time
I = np.matmul((X * (w * Lam)[..., None]).transpose(0, 2, 1), X) - np.matmul((xb * Dc[..., None]).transpose(0, 2, 1), xb)
ll = (Dc * (eta - np.log(S0))).sum(1)
if not robust: return g, I, ll, None
G = at(csum((Dc / S0)[..., None] * xb), l)
L = D[..., None] * (X - xb) - (w / c)[..., None] * (X * Lam[..., None] - G) # Lin-Wei score residuals
return g, I, ll, np.matmul((L * c[..., None]).transpose(0, 2, 1), L)
def cox_fit(X, D, c, f, l, beta, iters):
eps = 1e-9 * np.eye(X.shape[2])
for _ in range(iters):
g, I, _, _ = cox_eval(X, D, c, f, l, beta)
step = np.clip(np.linalg.solve(I + eps, g[..., None])[..., 0], -5, 5); beta = beta + step
return beta, np.abs(step).max(1)
def slr(T, D, Zt, c, G):
"""Stratified log-rank z (hypergeometric; binomial) from time-sorted data: a stable sort by stratum keeps time order."""
o = np.argsort(G, 1, kind='stable'); T, D, Zt, c, G = (np.take_along_axis(a, o, 1) for a in (T, D, Zt, c, G))
n = T.shape[1]; ar = np.broadcast_to(np.arange(n), T.shape)
gnew = np.ones(T.shape, bool); gnew[:, 1:] = G[:, 1:] != G[:, :-1]
tnew = gnew.copy(); tnew[:, 1:] |= T[:, 1:] != T[:, :-1]
def last(new): # last index of own group
end = np.ones(T.shape, bool); end[:, :-1] = new[:, 1:]
return np.flip(np.minimum.accumulate(np.flip(np.where(end, ar, n - 1), 1), 1), 1)
f, l, e = np.maximum.accumulate(np.where(tnew, ar, 0), 1), last(tnew), last(gnew)
upto = lambda cs, i: np.where(i >= 0, at(cs, np.maximum(i, 0)), 0.0)
cc, cz, cd = np.cumsum(c, 1), np.cumsum(c * Zt, 1), np.cumsum(c * D, 1)
Y, Y1, dk = at(cc, e) - upto(cc, f - 1), at(cz, e) - upto(cz, f - 1), at(cd, l) - upto(cd, f - 1)
p = Y1 / Y; Dc = c * D; U = (Dc * (Zt - p)).sum(1) # within-stratum risk sets: own tie group to stratum end
V = (Dc * p * (1 - p) * np.where(Y > 1, (Y - dk) / np.maximum(Y - 1, 1e-12), 1)).sum(1)
return U / np.sqrt(V), U / np.sqrt((Dc * p * (1 - p)).sum(1))
def analyse(idx, Zt, t_all, e_all, c=None):
T, D, X = t_all[idx], e_all[idx].astype(float), COV[idx]
c = np.ones_like(T) if c is None else c
o = np.argsort(T, 1, kind='stable')
T, D, Zt, c = (np.take_along_axis(a, o, 1) for a in (T, D, Zt, c)); X = np.take_along_axis(X, o[..., None], 1)
f, l = ties(T); B, n = T.shape; Dc = c * D; nw = c.sum(1); pi = (c * Zt).sum(1) / nw; out = {}
# log-rank (hypergeometric variance VL for integer data; binomial VB is Ye et al.'s sigma_L^2)
Y = rcs_at(c, f); p1 = rcs_at(c * Zt, f) / Y; cD = np.cumsum(Dc, 1)
dk = at(cD, l) - np.where(f > 0, at(cD, np.maximum(f - 1, 0)), 0)
U = (Dc * (Zt - p1)).sum(1)
VL = (Dc * p1 * (1 - p1) * np.where(Y > 1, (Y - dk) / np.maximum(Y - 1, 1e-12), 1)).sum(1)
VB = (Dc * p1 * (1 - p1)).sum(1)
out['LR'] = U / np.sqrt(VL); out['LRb'] = U / np.sqrt(VB); out['events'] = Dc.sum(1)
out['SLR'], out['SLRb'] = slr(T, D, Zt, c, np.take_along_axis(STRAT[idx], o, 1)) # stratified log-rank (S strata)
# Ye-Shao-Yi derived outcomes O_ij
A0, A1 = at(np.cumsum(Dc * (1 - p1) / Y, 1), l), at(np.cumsum(Dc * p1 / Y, 1), l)
O = np.where(Zt == 1, D * (1 - p1) - A0, D * p1 - A1)
assert np.allclose((c * np.where(Zt == 1, O, -O)).sum(1), U)
bu, su = cox_fit(Zt[..., None], D, c, f, l, np.zeros((B, 1)), 6)
_, I, _, _ = cox_eval(Zt[..., None], D, c, f, l, bu)
out['U_b'], out['U_se'] = bu[:, 0], 1 / np.sqrt(I[:, 0, 0]); conv = su
for s, cols in SETS.items():
Xk, k = X[..., cols], len(cols); zk = np.zeros((B, 1))
gam, s0 = cox_fit(Xk, D, c, f, l, np.zeros((B, k)), 7)
ll0 = cox_eval(Xk, D, c, f, l, gam)[2]
Xf = np.concatenate([Zt[..., None], Xk], 2); b0 = np.concatenate([zk, gam], 1)
g, I, _, _ = cox_eval(Xf, D, c, f, l, b0)
out[s + '_score'] = g[:, 0] * np.sqrt(np.linalg.inv(I + 1e-9 * np.eye(k + 1))[:, 0, 0])
bf, s1 = cox_fit(Xf, D, c, f, l, b0, 5)
_, I, ll1, M = cox_eval(Xf, D, c, f, l, bf, robust=True)
Ii = np.linalg.inv(I + 1e-9 * np.eye(k + 1)); V = Ii @ M @ Ii
out[s + '_b'], out[s + '_se'], out[s + '_rse'] = bf[:, 0], np.sqrt(Ii[:, 0, 0]), np.sqrt(V[:, 0, 0])
out[s + '_wald'], out[s + '_rwald'] = bf[:, 0] / out[s + '_se'], bf[:, 0] / out[s + '_rse']
out[s + '_lrt'] = np.sign(bf[:, 0]) * np.sqrt(np.maximum(2 * (ll1 - ll0), 0))
out[s + '_sep'] = (np.abs(bf[:, 1:]).max(1) > 8).astype(float) # monotone-likelihood flag
conv = np.maximum(conv, np.maximum(s0, s1))
# covariate-adjusted log-rank (Ye, Shao and Yi 2024): arm-specific OLS of O on X
xbar = (c[..., None] * Xk).sum(1) / nw[:, None]; adj, bsum = 0.0, 0.0
for a, sign in ((1, 1.0), (0, -1.0)):
m = c * (Zt == a); xa = (m[..., None] * Xk).sum(1) / m.sum(1)[:, None]; Xd = Xk - xa[:, None, :]
ba = np.einsum('bkl,bl->bk', np.linalg.pinv(np.matmul((Xd * m[..., None]).transpose(0, 2, 1), Xd)),
np.einsum('bnk,bn->bk', Xd * m[..., None], O))
adj = adj + sign * (m * np.einsum('bnk,bk->bn', Xk - xbar[:, None, :], ba)).sum(1); bsum = bsum + ba
Xc = Xk - xbar[:, None, :]; Sx = np.matmul((Xc * c[..., None]).transpose(0, 2, 1), Xc) / nw[:, None, None]
Vcl = VB - nw * pi * (1 - pi) * np.einsum('bk,bkl,bl->b', bsum, Sx, bsum)
n1 = Zt.sum(1); n0 = n - n1; ra = n0 / n1 + n1 / n0 # Krotka-Magirr Gamma = C_df x E(VIF)
gam_km = n / (n - (k + 1) * ra + k) * (1 + k * ((n0 / n) ** 2 / (n1 - k - 2) + (n1 / n) ** 2 / (n0 - k - 2)))
out[s + '_CL'], out[s + '_CLkm'] = (U - adj) / np.sqrt(Vcl), (U - adj) / np.sqrt(gam_km * Vcl)
out[s + '_vr'] = 1 - Vcl / VB # CL variance reduction
out['conv'] = conv
return out
TESTS = ['LR', 'S_wald', 'S_rwald', 'S_score', 'S_lrt', 'S_CL', 'S_CLkm', 'P_wald', 'P_rwald', 'P_score', 'P_lrt', 'P_CL', 'P_CLkm', 'SLR']
R3 = lambda x: round(float(x), 3)
def mart_phi(units, t_all, e_all):
"""phi = sum of squared martingale residuals of the covariate-only Cox fit / events; 1 when the working model is right.
Under the null the adjusted treatment score is sum (Z - pi) r_A, so phi inflates its variance over the model information."""
T, D, X = t_all[units][None], e_all[units][None].astype(float), COV[units][None]
o = np.argsort(T, 1, kind='stable'); T, D, X = np.take_along_axis(T, o, 1), np.take_along_axis(D, o, 1), np.take_along_axis(X, o[..., None], 1)
f, l = ties(T); c = np.ones_like(T); out = {'none': float(((D - at(np.cumsum(D / rcs_at(c, f), 1), l)) ** 2).sum() / D.sum())}
for s, cols in SETS.items():
gam = cox_fit(X[..., cols], D, c, f, l, np.zeros((1, len(cols))), 10)[0]
w = np.exp(X[..., cols] @ gam[0]); out[s] = float(((D - w * at(np.cumsum(D / rcs_at(w, f), 1), l)) ** 2).sum() / D.sum())
g, r2 = np.take_along_axis(STRAT[units][None], o, 1), 0.0 # stratified Nelson-Aalen residuals (stratified log-rank)
for h in np.unique(g):
m = (g == h).astype(float); r2 += float((m * (D - at(np.cumsum(m * D / np.maximum(rcs_at(m, f), 1e-12), 1), l)) ** 2).sum())
out['strat'] = r2 / D.sum()
return out
size_from_phi = lambda ph: 100 * math.erfc(ZC / math.sqrt(ph) / math.sqrt(2)) # 2{1 - Phi(1.96 / sqrt(phi))}
phi = {}
for e, (t, s) in ENDP.items():
pp, pc, pt = mart_phi(np.arange(N), t, s), mart_phi(IDX[0], t, s), mart_phi(IDX[1], t, s)
phi[e] = {'pooled': {k: R3(v) for k, v in pp.items()}, 'controls_only': {k: R3(v) for k, v in pc.items()},
'implied_asymptotic_size_pct': {k: round(size_from_phi(v), 1) for k, v in pp.items()},
'implied_asymptotic_size_pct_controls': {k: round(size_from_phi(v), 1) for k, v in pc.items()},
'lev5fu_only': {k: R3(v) for k, v in pt.items()},
'implied_asymptotic_size_pct_lev5fu': {k: round(size_from_phi(v), 1) for k, v in pt.items()}}
print('phi', phi)
# ---------------- population values (estimands) of each world, and the observed trial ----------------
def pop_analysis(c):
res = {e: analyse(np.arange(N)[None], trt[None].astype(float), t, s, c[None]) for e, (t, s) in ENDP.items()}
return {e: {k: float(v[0]) for k, v in o.items()} for e, o in res.items()}
POP = {'observed': pop_analysis(np.ones(N)), 'trial': pop_analysis(np.where(trt == 1, N / 2 / len(IDX[1]), N / 2 / len(IDX[0]))),
'balanced': pop_analysis(W * N / 2)} # 1:1 normalised, as in the resampled trials
population = {}
for w_, P_ in POP.items():
for e, o in P_.items():
zl = o['LR'] if w_ == 'observed' else o['LRb'] # weighted worlds: binomial log-rank variance
q = {'z_LR': R3(zl)}
for s in ('U', 'S', 'P'):
q.update({s + '_logHR': R3(o[s + '_b']), s + '_HR': R3(np.exp(o[s + '_b'])),
s + '_HR_ci': [R3(np.exp(o[s + '_b'] - ZC * o[s + '_se'])), R3(np.exp(o[s + '_b'] + ZC * o[s + '_se']))]})
for s in ('S', 'P'):
q.update({s + '_z_wald': R3(o[s + '_wald']), s + '_z_CL': R3(o[s + '_CL']), s + '_CL_var_reduction_pct': round(100 * o[s + '_vr'], 1),
s + '_RE_pred_cox': R3((o[s + '_wald'] / zl) ** 2), s + '_RE_pred_CL': R3(1 / (1 - o[s + '_vr'])),
s + '_est_ratio': R3(o[s + '_b'] / o['U_b']), s + '_se_ratio': R3(o[s + '_se'] / o['U_se'])})
zs = o['SLR'] if w_ == 'observed' else o['SLRb']
q.update({'z_SLR': R3(zs), 'SLR_RE_pred': R3((zs / zl) ** 2)})
population.setdefault(w_, {})[e] = q
print('population', json.dumps(population))
# ---------------- Monte Carlo over resampled trials ----------------
NULLQ = {}
CAL = (('null', ''), ('null_obs', '_obs'), ('null_trt', '_trt')) # calibrating null world -> key suffix ('' = pooled, as before)
def vmcse(x): # Monte Carlo SE of a sample variance: sqrt{(m4 - m2^2) / R}
dv = x - x.mean(); return round(float(np.sqrt(max((dv ** 4).mean() - (dv ** 2).mean() ** 2, 0.0) / len(x))), 3)
def summarise(st, world, e, n):
R = len(st['LR']); ind = {t: (np.abs(st[t]) > ZC).astype(float) for t in TESTS}
pct = lambda v: round(100 * float(np.mean(v)), 1)
res = {'R': R, 'events_mean': round(float(st['events'].mean()), 1), 'epv_P': round(float(st['events'].mean() / 10), 1),
'rej_pct': {t: pct(v) for t, v in ind.items()},
'rej_mcse_pct': {t: round(100 * float(np.sqrt(v.mean() * (1 - v.mean()) / R)), 2) for t, v in ind.items()},
'mean_z': {t: R3(st[t].mean()) for t in TESTS}, 'var_z': {t: R3(st[t].var(ddof=1)) for t in TESTS},
'var_z_mcse': {t: vmcse(st[t]) for t in TESTS},
'cl_var_reduction_pct': {s: round(100 * float(st[s + '_vr'].mean()), 1) for s in ('S', 'P')},
'rej_pct_converged_only': {t: pct(v[st['conv'] <= 1e-6]) for t, v in ind.items()}}
if world in PERMW:
pass
elif world in NULLW: # null calibration for size-adjusted power
NULLQ[(world, e, n)] = {t: (float(np.quantile(np.abs(st[t]), 0.95)), st[t] ** 2) for t in TESTS}
res['null_crit_abs_z'] = {t: R3(v[0]) for t, v in NULLQ[(world, e, n)].items()}
else:
zl = st['LR']; ml = zl.mean()
res['diff_vs_LR_pct'] = {t: [pct(ind[t] - ind['LR']), round(100 * float((ind[t] - ind['LR']).std(ddof=1) / np.sqrt(R)), 2)] for t in TESTS[1:]}
for cw, sx in CAL:
res['size_adj_power_pct' + sx] = {t: pct(np.abs(st[t]) > NULLQ[(cw, e, n)][t][0]) for t in TESTS}
res['RE_cal' + sx] = {}
res['RE'] = {}
for t in TESTS[1:]:
zt = st[t]; rho = zt.mean() / ml
v = (zt.var() - 2 * rho * np.mean((zt - zt.mean()) * (zl - ml)) + rho ** 2 * zl.var()) / (R * ml ** 2)
res['RE'][t] = [R3(rho ** 2), R3((rho - ZC * np.sqrt(v)) ** 2), R3((rho + ZC * np.sqrt(v)) ** 2)]
for cw, sx in CAL:
q = NULLQ[(cw, e, n)]
aL, aT = q['LR'][1], q[t][1]; k = aL.mean() / aT.mean() # calibration: each z rescaled to unit null mean square
vk = (aL.var() / aL.mean() ** 2 + aT.var() / aT.mean() ** 2 - 2 * np.mean((aL - aL.mean()) * (aT - aT.mean())) / (aL.mean() * aT.mean())) / len(aL)
sd = np.sqrt(4 * v / rho ** 2 + vk) # log scale; includes null-calibration Monte Carlo error
res['RE_cal' + sx][t] = [R3(k * rho ** 2), R3(k * rho ** 2 * np.exp(-ZC * sd)), R3(k * rho ** 2 * np.exp(ZC * sd))]
res['decomp'] = {s: {'est_ratio': R3(st[s + '_b'].mean() / st['U_b'].mean()), 'se_ratio': R3(st[s + '_se'].mean() / st['U_se'].mean()),
'empsd_ratio': R3(st[s + '_b'].std() / st['U_b'].std())} for s in ('S', 'P')}
est = {}
for s in ('U', 'S', 'P'):
tru = 0.0 if world in NULLW + PERMW else POP[world][e][s + '_b']; b, se = st[s + '_b'], st[s + '_se']
est[s] = {'truth': R3(tru), 'mean': R3(b.mean()), 'bias': round(float(b.mean() - tru), 4), 'emp_se': round(float(b.std(ddof=1)), 4),
'mod_se': round(float(se.mean()), 4), 'rmse': round(float(np.sqrt(((b - tru) ** 2).mean())), 4),
'cov_pct': pct(np.abs(b - tru) <= ZC * se), 'ratio_emp_mod': R3(b.std(ddof=1) / se.mean())}
if s != 'U':
rse = st[s + '_rse']
est[s].update({'rob_se': round(float(rse.mean()), 4), 'rcov_pct': pct(np.abs(b - tru) <= ZC * rse), 'ratio_emp_rob': R3(b.std(ddof=1) / rse.mean())})
res.update({'est': est, 'sep_flag_S': int(st['S_sep'].sum()), 'sep_flag_P': int(st['P_sep'].sum()), 'nonconverged': int((st['conv'] > 1e-6).sum())})
return res
cells = {e: {} for e in ENDP}
for world in ('null_obs', 'null_trt', 'null', 'balanced', 'trial', 'perm', 'perm_obs'): # nulls first: they calibrate
R = REPS[world] # the others; the original stream keeps its order
for n in {'perm': (N,), 'perm_obs': (len(IDX[0]),)}.get(world, SIZES):
acc = {e: [] for e in ENDP}
for b0 in range(0, R, BLOCK):
idx, z = draw(world, n, min(BLOCK, R - b0)) # same resampled patients for both endpoints
for e, (t, s) in ENDP.items():
acc[e].append(analyse(idx, z, t, s))
for e in ENDP:
st = {k: np.concatenate([a[k] for a in acc[e]]) for k in acc[e][0]}
cells[e].setdefault(str(n), {})[world] = summarise(st, world, e, n)
print(world, n, 'done %.1fs' % (time.time() - T0), flush=True)
# ---------------- pre-specified decision rules ----------------
grid = [(e, str(n)) for e in ENDP for n in SIZES]
re_S = [cells[e][n]['balanced']['RE']['S_wald'][0] for e, n in grid]
size_S = [cells[e][n]['null']['rej_pct']['S_wald'] for e, n in grid]
h1 = all(x >= 1.10 for x in re_S) and all(x <= 6.0 for x in size_S)
h2 = {s: all(cells[e][n]['balanced']['decomp'][s]['est_ratio'] > 1 and cells[e][n]['balanced']['decomp'][s]['se_ratio'] >= 1 for e, n in grid) for s in ('S', 'P')}
infl = [[e, n, s] for e, n in grid for s in ('S', 'P') if cells[e][n]['null']['rej_pct'][s + '_wald'] > 6.0]
h3 = 'no_wald_inflation' if not infl else ('supported' if all(
cells[e][n]['null']['rej_pct'][s + '_score'] <= 6.0 and cells[e][n]['null']['rej_pct'][s + '_lrt'] <= 6.0
and cells[e][n]['null']['est'][s]['ratio_emp_mod'] > 1 for e, n, s in infl) else 'refuted')
single_arm_rules = {} # the same rules with a single-arm null world for size
for nw in ('null_obs', 'null_trt'):
sz = [cells[e][n][nw]['rej_pct']['S_wald'] for e, n in grid]
inf = [[e, n, s] for e, n in grid for s in ('S', 'P') if cells[e][n][nw]['rej_pct'][s + '_wald'] > 6.0]
single_arm_rules[nw] = {'typeI_S_wald_pct': sz, 'h1_supported': bool(all(x >= 1.10 for x in re_S) and all(x <= 6.0 for x in sz)),
'wald_inflated_cells': inf, 'h3_status': 'no_wald_inflation' if not inf else ('supported' if all(
cells[e][n][nw]['rej_pct'][s + '_score'] <= 6.0 and cells[e][n][nw]['rej_pct'][s + '_lrt'] <= 6.0
and cells[e][n][nw]['est'][s]['ratio_emp_mod'] > 1 for e, n, s in inf) else 'refuted')}
span = lambda v: [min(v), max(v)] # ranges over the six endpoint-by-size cells
headline = {'typeI_range_pct': {w: {t: span([cells[e][n][w]['rej_pct'][t] for e, n in grid]) for t in TESTS} for w in NULLW},
'var_z_range': {w: {t: span([cells[e][n][w]['var_z'][t] for e, n in grid]) for t in TESTS} for w in NULLW},
'RE_range': {t: span([cells[e][n]['balanced']['RE'][t][0] for e, n in grid]) for t in TESTS[1:]},
'RE_cal_range': {nm: {t: span([cells[e][n]['balanced']['RE_cal' + sx][t][0] for e, n in grid]) for t in TESTS[1:]}
for nm, sx in (('pooled', ''), ('obs', '_obs'), ('trt', '_trt'))},
'event_saving_pct_obs': {t: [round(100 * (1 - 1 / cells[e][n]['balanced']['RE_cal_obs'][t][0]), 1) for e, n in grid]
for t in TESTS[1:]}} # Schoenfeld: events needed scale with 1 / RE
result = {'design': {'seed': SEED, 'sizes': list(SIZES), 'reps': REPS, 'alpha': 0.05, 'tests': TESTS, 'covariates': NAMES,
'sets': {k: [NAMES[i] for i in v] for k, v in SETS.items()}, 'ties': 'Breslow', 'block': BLOCK,
'strata_SLR': 'node4 x extent (submucosa/muscle, serosa, contiguous) x surg',
'streams': 'original worlds: default_rng(seed); null_obs, null_trt, perm_obs: default_rng([seed, 1/2/3])',
'calibration_suffix': {'': 'pooled null', '_obs': 'observation-arm null', '_trt': 'Lev+5FU-arm null'}},
'data': data_info, 'balance': balance, 'phi': phi, 'population': population, 'cells': cells,
'h1_RE_S_wald_balanced': re_S, 'h1_typeI_S_wald_null_pct': size_S, 'h1_supported': bool(h1),
'h2_supported': h2, 'h3_status': h3, 'wald_inflated_cells': infl, 'single_arm_rules': single_arm_rules,
'headline': headline, 'runtime_s': round(time.time() - T0, 1), 'hypothesis_supported': bool(h1)}
def finite(x):
if isinstance(x, dict): return {k: finite(v) for k, v in x.items()}
if isinstance(x, list): return [finite(v) for v in x]
return None if isinstance(x, float) and not np.isfinite(x) else x
print('RESULT_JSON: ' + json.dumps(finite(result)))
4.Analysis
Analysis complete: experiment succeeded; hypothesis refuted; 53 key numbers.
Results and analysis
The revised analysis ran successfully on the real NCCTG data (619 patients; 291 deaths, 324 recurrence-free-survival events) in 98.3 s, and every number of the first run reproduced exactly. The pre-specified primary hypothesis is refuted. Cox adjustment for the three candidate stratification factors (S) gave nominal relative efficiencies (RE) of 1.055-1.151, below the 1.10 threshold in three of six cells. In the pre-specified pooled null its model-based Wald test also exceeded 6.0% at n = 150 (6.3%, 6.6%), but in the new primary null world, resampled from the 315 observation patients alone, the size criterion was met, so H1 fails on efficiency alone. In that world the Cox-S Wald test rejected 5.5%-5.9% of null trials at every size (MCSE 0.32-0.34) against 4.7%-5.2% for the log-rank test, and its null variance of z (1.035-1.065, MCSE 0.021-0.022) was close to the control arm's phi-hat (1.024-1.040); Cox-P tests reached up to 7.1% at n = 150. The pooled null overstated the excess at n = 150 (6.3% and 6.6%, against 5.8% and 5.6%), whereas a Lev+5FU-arm null gave 5.5%-6.4% (phi-hat 1.042-1.067), so working-model misspecification belongs to each arm's population, not only to pooling. Calibrated to the observation-arm null, Cox-S had RE 1.004-1.082 (intervals including 1 at n = 150): the conditional log hazard ratio lay 7%-9% further from zero than the marginal one but its sampling SD was 5%-7% larger, while the model SE grew only 1%-3% (H2 supported for S and P). The corrected covariate-adjusted log-rank test held its size (4.8%-5.3%) with calibrated RE 1.095-1.127, 8.7%-11.3% fewer events for equal power. The stratified log-rank test over 12 strata held its size (4.8%-5.4% in the resampled observation-arm null) but lost 1.0-5.6 points of power (calibrated RE 0.816-0.961), in line with its population prediction (0.893 OS, 0.977 RFS). H3 is refuted: wherever a Wald test exceeded 6.0%, the score and likelihood-ratio tests did too. Pooled and Lev+5FU-arm calibrations left the ranking unchanged. Keeping the trial's own imbalance made every adjusted test look worse (calibrated RE 0.683-0.975), a confounded contrast rather than lost efficiency.
Key numbers
| Metric | Value | Interpretation |
|---|---|---|
| run | succeeded in 98.3 s; seed 20261024; every value of the first run reproduced exactly | New worlds draw from their own random streams, so the earlier draws are unchanged. |
| analysis population | 619 patients (304 Lev+5FU, 315 observation) | Observation and levamisole-plus-fluorouracil arms; levamisole-alone arm excluded; one record per patient. |
| OS events | 291 (123 Lev+5FU, 168 observation); censored 53.0%; median follow-up of censored 2318.5 days | OS endpoint in the analysis population. |
| RFS events | 324 (134 Lev+5FU, 190 observation); censored 47.7%; median follow-up of censored 2309.0 days | RFS endpoint in the analysis population. |
| covariate handling | differentiation missing for 13 patients, imputed as not poor; perforation (2.7%) excluded; median age 61.0 | Deterministic imputation; rare covariate excluded to avoid degenerate resamples. |
| covariate prevalences (%) | node4 26.8, extent_1_2 14.2, extent_4 5.0, surg 27.0, obstruct 18.9, adhere 13.9, poor_diff 17.1, male 49.6 | Mostly binary, moderately prognostic factors. |
| chance imbalance (SMD, Lev+5FU minus observation) | node4 -0.037, extent_1_2 -0.023, extent_4 -0.126, surg -0.088, obstruct -0.057, adhere -0.061, poor_diff 0.033, age10 0.020, male -0.127 | The observation arm carried slightly more adverse factors. |
| entropy-balancing weights | max |SMD| after 1.2e-13; weight/uniform 0.69-1.683; ESS 299.3 (Lev+5FU), 311.1 (observation) | Balancing removes the observed imbalance at small cost. |
| strata of the stratified log-rank test | 12 (node4 x 3 extent categories x surg); sizes 1, 4, 6, 7, 12, 14, 19, 33, 56, 105, 111, 251; 11 occupied in the observation arm, 12 in Lev+5FU | Seven strata hold fewer than 20 of the 619 patients. |
| phi-hat, Cox-S (OS/RFS) | observation arm 1.040/1.024; Lev+5FU arm 1.042/1.067; pooled 1.054/1.056 | Mean squared martingale residual per event; 1 if the working model were correct. |
| phi-hat, Cox-P (OS/RFS) | observation arm 1.036/1.027; Lev+5FU arm 1.055/1.092; pooled 1.073/1.080 | Misspecification is present within each arm. |
| phi-hat, no covariates | observation 0.994, Lev+5FU 0.995, pooled 0.997 (both endpoints) | The log-rank working model is close to correct. |
| implied asymptotic size, Cox-S (OS/RFS) | observation arm 5.5%/5.3%; Lev+5FU arm 5.5%/5.8%; pooled 5.6%/5.6% | 2{1 - Phi(1.96/sqrt(phi))}. |
| type I error, observation-arm null (primary), range over 6 cells | log-rank 4.7-5.2%; stratified log-rank 4.8-5.4%; Cox-S Wald 5.5-5.9%; Cox-S robust 5.3-5.6%; Cox-S score 5.5-6.0%; CL-S corrected 4.8-5.3%; Cox-P Wald 5.6-6.9%; Cox-P robust 5.4-6.6%; Cox-P score 5.7-7.1%; CL-P 5.4-6.8%; CL-P corrected 4.6-5.4% | 5,000 replicates per cell; MCSE 0.30-0.36 points. |
| Cox-S Wald type I error by cell, observation-arm null | OS 5.8, 5.7, 5.8; RFS 5.6, 5.5, 5.9 (n = 150, 300, 600) | No trend with n; each cell 1.6-2.7 MCSE above 5%. |
| null var(z), observation-arm null | Cox-S Wald 1.063, 1.065, 1.049 (OS), 1.061, 1.058, 1.035 (RFS), MCSE 0.021-0.022; log-rank 0.986-1.019; Cox-P Wald 1.053-1.141 | Systematic evidence of the excess; at n = 600 Cox-S matches phi-hat 1.040/1.024, Cox-P exceeds phi-hat 1.036/1.027 (1.063/1.053). |
| type I error, Lev+5FU-arm null, range | log-rank 4.2-5.7%; stratified log-rank 4.6-5.9%; Cox-S Wald 5.5-6.4%; Cox-S score 5.6-6.6%; CL-S corrected 4.5-5.9%; Cox-P Wald 6.0-7.3%; CL-P corrected 4.6-5.5% | 5,000 replicates per cell; MCSE 0.28-0.37. |
| null var(z) at n = 600, Lev+5FU-arm null (OS/RFS) | Cox-S Wald 1.107/1.129; log-rank 1.034/1.035; Cox-P Wald 1.145/1.177 | Larger misspecification in the Lev+5FU population for RFS. The log-rank value (sizes 5.6%/5.7%), 1.6-1.7 MCSE above 1, is read as chance, not an effect of drawing 600 from 304 patients (600/304 = 1.97 appearances per patient per trial): the observation-arm null (600/315 = 1.90) gave 0.986/0.996, this world gave 0.969/0.958 at n = 300, and OS and RFS share replicates; Cox-S Wald exceeded log-rank at every size (1.055-1.129 against 0.958-1.035). |
| type I error, pooled null (pre-specified), range | log-rank 4.6-5.3%; stratified log-rank 4.2-5.6%; Cox-S Wald 5.4-6.6%; Cox-S score 5.5-6.9%; CL-S corrected 4.6-5.1%; Cox-P Wald 5.9-7.3%; Cox-P score 5.9-7.6%; CL-P corrected 4.6-5.5% | Pooling both arms overstated the n = 150 excess relative to the observation-arm null. |
| null var(z) at n = 600, pooled null (OS/RFS) | Cox-S Wald 1.065/1.059; Cox-P Wald 1.101/1.099; log-rank 1.002/0.990 | MCSE 0.020-0.023. |
| corrected CL and stratified log-rank versus log-rank, all null cells | within 0.5 point of the log-rank test in every cell of the three null worlds | Both held their size. |
| likelihood-ratio versus score tests | within 0.1 point in every resampled null cell; within 0.2 under re-randomisation | Likelihood-ratio rows omitted from the tables. |
| re-randomisation of the observation arm (157:158, n = 315, 2,000 splits) | log-rank 5.5/5.3%; stratified log-rank 5.9/5.0%; Cox-S Wald 6.2/6.5%; Cox-S robust 5.7/6.0%; CL-S corrected 5.2/5.1%; Cox-P Wald 6.4/6.3% (OS/RFS); MCSE 0.49-0.55 | Resampling-free single-arm check reproduces the pattern. |
| null var(z), re-randomised observation arm (OS/RFS) | log-rank 1.035/1.037 (MCSE 0.031/0.032); Cox-S Wald 1.097/1.085 (MCSE 0.035) | Relative to the log-rank statistic, the Cox-S Wald variance is inflated as in the resampled world. |
| re-randomisation of the trial (304:315, n = 619, 2,000) | log-rank 4.5/4.5%; stratified log-rank 5.0/5.4%; Cox-S Wald 5.5/5.3%; CL-S corrected 4.6/4.8%; Cox-P Wald 5.9/5.9% (OS/RFS); MCSE 0.46-0.53 | Labels permuted on data containing the real effect, so not independent of the pooled null. |
| non-convergence at n = 150 (replicates with a non-converged Cox fit; flagged for monotone likelihood) | observation-arm null 11/7 of 5,000 (6/0 flagged); pooled null 54/32 (50/14); Lev+5FU-arm null 236/227 (222/136), and 27/14 (15/7) at n = 300; balanced 27/18 of 2,000 (22/13), 1 at OS n = 300; trial-preserving 11/8 of 1,000 (10/4) | Excluding them changed rejection rates by at most 0.2 point in the null and balanced worlds and 0.3 in the trial-preserving world. |
| H1 (pre-specified, pooled null) | RE 1.055, 1.105, 1.151 (OS), 1.066, 1.091, 1.117 (RFS); type I error 6.3, 5.7, 5.7, 6.6, 5.7, 5.4% | Refuted on both criteria. |
| H1 with the observation-arm null | Cox-S Wald type I error 5.5-5.9% (all at most 6.0%) | Size criterion met; H1 refuted on efficiency alone. With the Lev+5FU-arm null the size criterion fails (6.1-6.4% in four cells). |
| H2 | estimate ratio 1.068-1.091 (S), 1.084-1.113 (P); mean model-SE ratio 1.009-1.026 (S), 1.018-1.061 (P) | Supported for S and P in all six cells. |
| H3 | pooled null: Wald above 6.0% in 5 cells (OS 150 S and P, RFS 150 S and P, RFS 300 P); observation-arm null: 3 cells (Cox-P OS 150, OS 600, RFS 150); Lev+5FU-arm null: 9 cells; score and likelihood-ratio tests also above 6.0% in every one; empirical/model SE ratio 1.033-1.103 there | Refuted in every null world; the SE-underestimation mechanism holds, but score and likelihood-ratio tests share the inflation and it is not confined to small trials. |
| power, balanced world (nominal), OS n = 150/300/600 | log-rank 30.7/52.7/81.9; stratified log-rank 25.4/47.1/76.4; Cox-S Wald 32.2/58.0/87.2; CL-S corrected 30.3/57.1/86.0; Cox-P Wald 31.2/58.1/86.7 | 2,000 replicates; MCSE 0.25-1.12. |
| power, balanced world (nominal), RFS n = 150/300/600 | log-rank 50.4/80.0/97.8; stratified log-rank 46.3/78.2/96.8; Cox-S Wald 53.5/82.4/98.4; CL-S corrected 53.8/83.8/98.6; Cox-P Wald 54.0/82.8/98.4 | RFS has more events and a larger effect. |
| size-adjusted power, observation-arm calibration, OS n = 150/300/600 | log-rank 30.3/53.6/82.0; stratified log-rank 24.8/47.5/75.9; Cox-S Wald 30.0/55.9/85.1; CL-S corrected 31.0/57.9/85.4; Cox-P Wald 28.2/56.0/84.2 | Cox-S loses its edge at n = 150. |
| size-adjusted power, observation-arm calibration, RFS n = 150/300/600 | log-rank 50.1/80.2/97.8; stratified log-rank 46.8/76.8/96.5; Cox-S Wald 51.7/81.2/98.0; CL-S corrected 54.1/83.8/98.6; Cox-P Wald 49.0/81.0/98.0 | Corrected CL highest at every size. |
| paired power difference from log-rank (points, MCSE) | Cox-S Wald +1.5 (0.78), +5.3 (0.85), +5.3 (0.65), +3.0 (0.88), +2.5 (0.69), +0.6 (0.29); stratified log-rank -5.3 (0.86), -5.6 (0.89), -5.5 (0.74), -4.2 (0.94), -1.8 (0.79), -1.0 (0.37) | OS then RFS at n = 150, 300, 600. |
| nominal RE, balanced world | Cox-S Wald 1.055-1.151; Cox-S robust 1.008-1.093; CL-S corrected 1.061-1.117; Cox-P Wald 1.017-1.169; stratified log-rank 0.808-0.976 | Ratio of squared mean z; equals a sample-size ratio only with unit null variance. |
| calibrated RE, observation-arm null (primary) | Cox-S Wald 1.004 (0.969-1.041), 1.051, 1.082 (OS); 1.010 (0.978-1.043), 1.051, 1.074 (RFS); range 1.004-1.082 | Intervals include 1 at n = 150. |
| calibrated RE, observation-arm null, corrected CL-S | 1.097 (1.064-1.131), 1.109, 1.108 (OS); 1.095, 1.123, 1.127 (1.102-1.153) (RFS); range 1.095-1.127 | Every interval above 1. |
| calibrated RE, observation-arm null, other tests | Cox-S robust 0.984-1.069; Cox-P Wald 0.914-1.084; CL-P corrected 1.038-1.120; stratified log-rank 0.816 (0.782-0.850) to 0.961 (0.935-0.988) | Stratified log-rank intervals all below 1. |
| calibrated RE, pooled null | Cox-S Wald 0.970-1.083; CL-S corrected 1.075-1.122; Cox-P Wald 0.874-1.063; CL-P corrected 1.028-1.117; stratified log-rank 0.801-0.985 | As in the first version for the original tests. |
| calibrated RE, Lev+5FU-arm null | Cox-S Wald 0.972-1.075; CL-S corrected 1.070-1.112; Cox-P Wald 0.891-1.055; CL-P corrected 1.031-1.103; stratified log-rank 0.807-0.964 | Ranking unchanged. |
| event savings, observation-arm calibration (Schoenfeld), 100(1 - 1/RE) | CL-S corrected 8.7-11.3%; Cox-S Wald 0.4-7.6%; CL-P corrected 3.7-10.7%; Cox-P Wald -9.4 to 7.7%; stratified log-rank -22.5 to -4.1% (4.1-22.5% more events) | Events needed for equal power relative to the log-rank test. |
| large-sample RE predictions, balanced population (OS/RFS) | Cox-S 1.171/1.133; Cox-P 1.211/1.157; CL-S 1.121/1.132; CL-P 1.132/1.138; stratified log-rank 0.893/0.977 | CL predictions approached by n = 600; Cox predictions far exceed calibrated gains; stratified prediction below 1. |
| CL variance reductions, balanced population | S 10.8% (OS), 11.7% (RFS); P 11.7%, 12.1% | Basis of the CL prediction. |
| population z, balanced world (OS/RFS) | log-rank -2.918/-4.003; stratified log-rank -2.758/-3.957 | The stratified prediction below 1 comes from the same 619 patients and 12 strata (seven under 20 patients), so it cannot separate sparse strata from weaker within-stratum contrasts. |
| estimand shift and precision (balanced world) | estimate ratio 1.068-1.091 (S), 1.084-1.113 (P); model-SE ratio 1.009-1.026 (S), 1.018-1.061 (P); empirical-SE ratio 1.046-1.066 (S), 1.075-1.140 (P) | Non-collapsibility shift largely cancelled by larger sampling variance. |
| true log hazard ratios, balanced population (OS/RFS) | marginal -0.345/-0.448 (HR 0.708/0.639); conditional S -0.376/-0.483; conditional P -0.384/-0.489 | Estimands of the balanced world. |
| bias and coverage (balanced world) | max |bias| 0.0139; marginal coverage 94.7-95.7%; conditional model-based 92.5-95.3%; robust 93.4-95.6%; RMSE within 0.0003 of empirical SE | Model-based conditional intervals undercover slightly. |
| trial-preserving world, calibrated RE (observation-arm null) | Cox-S Wald 0.787 (0.750-0.826) to 0.883 (0.859-0.908); CL-S corrected 0.898-0.945; stratified log-rank 0.683-0.802; Cox-P Wald 0.740-0.923; CL-P corrected 0.894-0.975 | Every adjusted test below 1 (0.683-0.975). |
| trial-preserving world, mean z at n = 600, OS | log-rank -3.077 (balanced -2.852); Cox-S Wald -2.934 (balanced -3.059) | The unadjusted statistic profits from a fixed imbalance favouring Lev+5FU: a confounded contrast, not efficiency. |
| observed trial, OS | HR 0.689 (0.546-0.869), log-rank z -3.157; Cox-S 0.693 (0.549-0.876), z -3.071; Cox-P 0.681 (0.539-0.862), z -3.203; CL-S/CL-P z -3.025/-3.094; stratified log-rank z -2.715 | Squared z ratios versus log-rank: Cox-S 0.946, Cox-P 1.029, stratified 0.739. |
| observed trial, RFS | HR 0.621 (0.498-0.775), log-rank z -4.258; Cox-S 0.625 (0.500-0.781), z -4.140; Cox-P 0.614 (0.491-0.767), z -4.290; CL-S/CL-P z -4.164/-4.257; stratified log-rank z -3.864 | Squared z ratios: Cox-S 0.945, Cox-P 1.015, stratified 0.823. |
| mean events per trial, null worlds (OS, n = 150/300/600) | observation arm 80.0/160.0/320.1; Lev+5FU arm 60.7/121.3/242.8; pooled 70.4/141.2/282.2 | Events per Cox-P parameter 8.0-32.0, 6.1-24.3 and 7.0-28.2; RFS 90.4-361.9, 66.1-264.5, 78.4-314.3 events. |
Caveats
- Single trial, two correlated endpoints analysed on the same resampled patients, one effect size per endpoint; magnitudes may not transfer to trials with stronger or continuous prognostic factors or lighter censoring.
- The size excess and the calibrated RE depend on which population is taken as the null: the observation arm (primary), the Lev+5FU arm and the pooled trial give Cox-S Wald sizes of 5.5-5.9%, 5.5-6.4% and 5.4-6.6%; all three are reported.
- The resampling population is the empirical distribution of the trial, including any chance subgroup structure; the stratified log-rank test is directly affected because its population prediction (0.893 OS, 0.977 RFS) is computed on the same 12 strata, seven under 20 patients, whose within-stratum contrasts may partly be chance, so it cannot separate sparse strata from weaker within-stratum contrasts.
- Bootstrap duplicates create tied times (Breslow); resampling-free re-randomisations of the trial and of the observation arm gave the same pattern but have larger MCSE (0.46-0.55 with 2,000 splits).
- Entropy balancing equalises the means of nine covariates, not their joint distribution, so stratum composition is not exactly balanced.
- Whether S equals the trial randomisation strata could not be confirmed: the abstracts of the primary and final reports do not name the stratification factors and they could not be confirmed from the full reports, so S is treated as candidate stratification factors; all worlds mimic simple randomisation, so stratified designs were not studied.
- phi-hat is a first-order heuristic: it matched the Cox-S null variance but under-predicted it for Cox-P (e.g. 1.036 against 1.063 at n = 600, OS, observation arm).
- Single null cells lie only 1.6-2.7 MCSE above 5% for Cox-S; the systematic evidence is the null variance of z (MCSE about 0.02) and its agreement with phi-hat.
- Calibrated RE and size-adjusted power borrow a null world for calibration; calibration error is included in the RE intervals.
- Non-convergence from monotone likelihood was common in the Lev+5FU-arm null at n = 150 (236 and 227 of 5,000), but excluding those replicates changed rejection rates by at most 0.2 point.
- Not studied: stratified log-rank tests on strata coarser than the fully crossed 12 (the only design examined), Efron ties, restricted mean survival time, other covariate codings, covariate-adaptive randomisation.
5.Paper writing
Manuscript drafted: “Does prespecified Cox adjustment really buy power over the log-rank test? A real-data resampling study of the NCCTG adjuvant colon cancer trial” (≈ 6726 words).
≈ 6726 words · AI-drafted manuscript · not peer-reviewed
Does prespecified Cox adjustment really buy power over the log-rank test? A real-data resampling study of the NCCTG adjuvant colon cancer trial
Abstract
Prespecified covariate adjustment is recommended to raise power in randomised survival trials, mainly on evidence from parametric simulations. We resampled the 619 observation and levamisole-plus-fluorouracil patients of the NCCTG adjuvant colon cancer trial into 1:1 trials of 150, 300 and 600 patients, removed the trial's chance covariate imbalance by entropy balancing, and compared the log-rank test with the stratified log-rank test, Cox models adjusted for three candidate stratification factors (Cox-S) or eight prognostic factors, and the covariate-adjusted log-rank test (CL) with a finite-sample correction. In null trials resampled from the observation arm alone, the model-based Cox-S Wald test rejected 5.5%-5.9% at every size (Monte Carlo SE about 0.3), against 4.7%-5.2% for the log-rank test; its null variance of z, 1.035-1.065 (Monte Carlo SE 0.02), was close to the control arm's mean squared martingale residual per event (1.024-1.040). A null world pooling both arms overstated the excess at n = 150 (6.3% and 6.6%). Cox-S estimated a conditional log hazard ratio 7%-9% further from zero than the marginal one, but with a 5%-7% larger sampling standard deviation, so its null-calibrated relative efficiency was only 1.004-1.082. Adjusting for the same three factors, the corrected CL test held its size (4.8%-5.3%) and had calibrated relative efficiency 1.095-1.127, 8.7%-11.3% fewer events for equal power, whereas the 12-stratum stratified log-rank test lost efficiency (0.816-0.961). In trials like this one, the corrected CL test is a better route to power than Cox adjustment or stratification.
Keywords: covariate adjustment; Cox proportional hazards model; log-rank test; stratified log-rank test; type I error; resampling
1. Introduction
Randomised survival trials are routinely analysed with the log-rank test, and reviews of regulatory and statistical guidance encourage prespecified adjustment for prognostic baseline covariates [1,2], including factors balanced at randomisation [3], usually through a Cox model [4] or a stratified log-rank test [5]. The quantitative case rests on Monte Carlo studies: adjusting for one binary covariate cut the sample size needed for equal power by 4%-44% [6]; across 12 outcomes (two time-to-event) simulated from parameters of eight studies, adjustment raised median power from 80% to 92.6% [7]; simulations derived from real patient data quantified precision gains for time-to-event outcomes [8]; and adjusted nonparametric log-rank tests have been compared with Cox models by simulation [9]. Drawing on a cancer trial of more than 6,000 patients, Akazawa and colleagues argued that the simple log-rank test can mislead in heterogeneous samples and that stratified log-rank tests lose power with many strata [10].
Two features of the Cox model make that evidence incomplete. The hazard ratio is non-collapsible: omitting a balanced prognostic covariate attenuates the treatment coefficient even under randomisation [11-14], so adjusted and unadjusted models estimate different quantities [15-17] and power can rise without any gain in precision [18]. And the working model is rarely correct: tests of no effect keep a mean-zero numerator [19,20], but the model-based variance is then inconsistent and Wald tests can be supranominal [21]; Cox Z statistics degrade with few events per variable [22,23], and covariates cost variance inflation and degrees of freedom in small trials [24]. Adjusted log-rank statistics keep the marginal estimand [25-28]; the covariate-adjusted log-rank (CL) test of Ye, Shao and Yi has a guaranteed asymptotic efficiency gain [29], is advocated for oncology [30,31] and has a finite-sample correction [32]. The stratified log-rank test gains asymptotically when a common treatment effect holds within strata [5].
None of the real-data-based studies [7,8,10] asks, on one trial's real event and censoring times, how much of the nominal Cox gain survives calibration to an appropriate null, or sets it against the CL test; and a single reanalysis cannot, because its realised gain mixes the expected gain with the trial's chance imbalance [33,34]. We resample the real patients of the NCCTG adjuvant colon cancer trial [35,36], which followed an earlier NCCTG study [37], remove its imbalance by entropy balancing [38], calibrate every test in null trials drawn from a single arm, and ask whether Cox adjustment raises efficiency by at least 10%, where any gain comes from, which tests hold their size, and whether large-sample predictions anticipate the realised gains.
2. Methods
2.1 Data
The trial randomised 1,296 patients with resected stage B2 or C colon cancer: stage B2 patients to observation or levamisole plus fluorouracil (Lev+5FU), stage C patients also to levamisole alone [35]. The final report covers all 929 eligible stage III (Dukes C) patients, in whom Lev+5FU reduced the recurrence rate by 40% and the death rate by 33% [36]. We analysed the trial's public data, the colon dataset of the R package survival [39] mirrored by the Rdatasets archive (retrieved 24 October 2026; SHA-256 6f3472a64f696e31...). The file is titled "stage B/C colon cancer" after the whole trial, but its 929 patients in three arms match the final report's 929 eligible stage III patients [36], so we treat it as that cohort. Its 1,858 rows hold a recurrence and a death record per patient, with times in days. We kept the 619 patients randomised to observation (315) or Lev+5FU (304). Overall survival (OS) is death from any cause; recurrence-free survival (RFS), a standard adjuvant endpoint [40], is the first of recurrence or death (Table 1).
Two covariate sets were fixed before analysis. Set S contains three baseline factors coded as four indicators: more than four positive lymph nodes (node4), extent of local spread (submucosa or muscle; contiguous structures; reference serosa) and a long interval from surgery to registration (surg). Our study plan called these the trial's randomisation stratification factors, but the abstracts of the trial reports [35,36] do not name the stratification factors and we could not confirm them from the full reports, so we treat S as candidate stratification factors. Set P adds obstruction, adherence, poor differentiation, age (per 10 years, centred) and male sex. Differentiation, missing for 13 patients, was imputed deterministically as not poor [41]; perforation (2.7% of patients) was excluded because it is absent from many small resamples. Crossing node4, the three extent categories and surg gives 12 strata.
Table 1. Analysis population: 619 patients of the NCCTG trial randomised to observation or Lev+5FU.
| Quantity | Value | Standardised difference, Lev+5FU minus observation |
|---|---|---|
| OS events (Lev+5FU, observation); censored | 291 (123, 168); 53.0% | |
| RFS events (Lev+5FU, observation); censored | 324 (134, 190); 47.7% | |
| Median follow-up of censored patients, days (OS; RFS) | 2318.5; 2309.0 | |
| More than four positive nodes | 26.8% | -0.037 |
| Extent: submucosa or muscle / contiguous structures | 14.2% / 5.0% | -0.023 / -0.126 |
| Long interval from surgery | 27.0% | -0.088 |
| Obstruction / adherence | 18.9% / 13.9% | -0.057 / -0.061 |
| Poor differentiation | 17.1% | 0.033 |
| Age, median years | 61.0 | 0.020 (per 10 years) |
| Male | 49.6% | -0.127 |
Note: standardised difference = difference in means divided by the pooled standard deviation. After entropy balancing the largest absolute standardised difference was 1.2e-13. The 12 S strata held 1, 4, 6, 7, 12, 14, 19, 33, 56, 105, 111 and 251 patients; 11 were occupied in the observation arm and 12 in the Lev+5FU arm.
2.2 Tests compared
Let indicate Lev+5FU, the covariates, and at event time let () be the number at risk (on Lev+5FU) and () the events. The log-rank test [42] is with and hypergeometric variance . The stratified log-rank test (SLR) computes and within each of the 12 S strata and divides the summed by the square root of the summed [5]; it is the score test of a Cox model with stratum-specific baseline hazards.
The Cox model was fitted by partial likelihood with Breslow ties [43], adjusting for S (Cox-S) or P (Cox-P). was tested four ways: Wald with the model-based standard error ; Wald with the Lin-Wei robust standard error from , where are score residuals [21]; the score test at from the covariate-only fit; and the partial likelihood-ratio test. An unadjusted Cox model gave the marginal log hazard ratio. Fitting details and the convergence and monotone-likelihood rules are in Supplementary Note S1.
The CL test [29] forms derived outcomes , regresses them on separately in each arm (slopes ) and uses
where , is the binomial log-rank variance per patient, the allocation fraction and the variance reduction. The corrected CL test multiplies by Krotka and Magirr's [32]. With arm sizes and , corrects for the residual degrees of freedom spent on the slopes, and is the expected variance inflation factor from estimating them, derived for Gaussian covariates [24,32]. For 1:1 allocation ; the general form served the 304:315 and 157:158 re-randomisations. depends only on , and the allocation, so it is constant within a world and null calibration cancels it.
2.3 Why model-based adjusted tests can be miscalibrated: a first-order heuristic
Under the adjusted score is approximately , with the martingale residuals of the covariate-only model and the event indicator. With randomised , , whereas the model-based information is approximately for events, so is, to first order, the expected ratio of the Lin-Wei robust to the model-based variance of the treatment score under randomisation. It is near 1 when the working model is correct; otherwise score, Wald and likelihood-ratio statistics have null variance inflated by about and asymptotic size , while robust variances estimate the score's variance directly, which is why robust Wald tests approach 5% in large samples [20,21]. This is a heuristic, not a derivation: it ignores the profile correction for , the imbalance of within risk sets and finite-sample effects. For the covariate-free model behind the log-rank test, Nelson-Aalen residuals give near 1.
3. Simulation Study
3.1 Aims
To estimate the type I error, power and relative efficiency of 14 tests in trials built from NCCTG patients, to decompose the Cox gain into estimand shift and precision, and to compare realised with predicted gains, following ADEMP [44].
3.2 Data-generating mechanisms
No outcome was generated: every replicate consists of real patients with their real times, events and covariates (Table 2). Under both arms of a trial come from one population, so the primary null world draws both arms from the 315 observation patients, the population of a trial in which treatment does nothing; a second uses the 304 Lev+5FU patients. Our first analysis drew both arms from the pooled 619, a mixture carrying the real treatment effect (hazard ratio about 0.69) as heterogeneity no null trial contains; it remains as a sensitivity world. In the balanced world, the primary alternative, each arm is resampled with entropy-balancing weights chosen so that the weighted means of the nine P covariates in each arm equal the pooled means [38], so arms differ only by the real treatment effect and chance. Two resampling-free checks re-randomise the 619 trial patients, whose outcomes contain the treatment effect, and the 315 observation patients, whose outcomes do not. All worlds mimic simple randomisation, and both endpoints are analysed on the same patients.
Table 2. Data-generating mechanisms (seed 20261024).
| World | Construction | Marginal HR, OS / RFS | Total n | Replicates per endpoint and n |
|---|---|---|---|---|
| Null, observation arm (primary null) | Both arms drawn with replacement from the 315 observation patients | 1 / 1 | 150, 300, 600 (1:1) | 5,000 |
| Null, Lev+5FU arm | Both arms drawn with replacement from the 304 Lev+5FU patients | 1 / 1 | 150, 300, 600 (1:1) | 5,000 |
| Null, pooled | Both arms drawn with replacement from the pooled 619 | 1 / 1 | 150, 300, 600 (1:1) | 5,000 |
| Balanced (primary alternative) | Each arm drawn from its own patients with entropy-balancing weights | 0.708 / 0.639 | 150, 300, 600 (1:1) | 2,000 |
| Trial-preserving | Each arm drawn uniformly from its own patients, keeping the trial's imbalance | 0.689 / 0.621 | 150, 300, 600 (1:1) | 1,000 |
| Re-randomisation, trial | The 619 patients re-randomised 304:315, no resampling | 1 / 1 | 619 | 2,000 |
| Re-randomisation, observation arm | The 315 observation patients split at random 157:158, no resampling | 1 / 1 | 315 | 2,000 |
Note: marginal hazard ratios are unadjusted Cox fits to the world's population (weighted for the balanced world). Mean OS events at n = 150, 300 and 600 were 80.0, 160.0 and 320.1 in the observation-arm null, 60.7, 121.3 and 242.8 in the Lev+5FU-arm null and 70.4, 141.2 and 282.2 in the pooled null, that is 8.0-32.0, 6.1-24.3 and 7.0-28.2 events per Cox-P parameter; mean RFS events were 90.4-361.9, 66.1-264.5 and 78.4-314.3. Entropy-balancing weights ranged from 0.69 to 1.683 times uniform; effective sample sizes were 299.3 (Lev+5FU) and 311.1 (observation). The two single-arm null worlds and the observation-arm re-randomisation draw from their own random streams, so the other worlds reproduce our first analysis exactly. With 5,000 null replicates the Monte Carlo standard error (MCSE) of a 5% rejection rate is 0.31 percentage points and that of a null variance of about 0.02; 2,000 alternative replicates give MCSEs of power of at most 1.12 points.
3.3 Estimands
The tests address : no treatment effect. The estimation targets are the marginal log hazard ratio and the conditional log hazard ratios of Cox-S and Cox-P, defined as each world's population values (weighted full-data fits in the balanced world; 0 under the null).
3.4 Methods
The 14 tests of Section 2.2: log-rank; stratified log-rank; Cox-S and Cox-P with model-based Wald, robust Wald, score and likelihood-ratio tests; CL-S and CL-P with and without correction.
3.5 Performance measures
Type I error and power are rejection rates at two-sided α = 0.05, with MCSE for replicates; a null variance of has MCSE from the sample central moments. Relative efficiency (RE) versus the log-rank test is . It equals the ratio of trial sizes, or by Schoenfeld's formula of events [45], needed for equal power under local alternatives only if both statistics have unit null variance; a test with RE then needs fewer events. Because anti-conservative tests also gain nominal power, calibrated RE first divides each statistic by its root mean square in a null world (same endpoint and size), and size-adjusted power uses the null 95th percentile of as critical value; the observation-arm null is the primary calibration. The 95% Monte Carlo interval for nominal RE squares the endpoints of a delta-method interval for the ratio of mean over paired replicates; that for calibrated RE is formed on the log scale and adds the Monte Carlo variance of the calibration constant. Estimators are summarised by bias, empirical SE, mean model-based and robust SE, coverage of 95% Wald intervals, and the ratios of mean estimate, mean model SE and empirical SE to those of the unadjusted model. Large-sample RE predictions are squared ratios in the balanced population (Cox, stratified log-rank) and for CL.
3.6 Results
Type I error. In the observation-arm null world (Table 3) the log-rank, stratified log-rank and corrected CL tests rejected 4.6%-5.4% of trials in the six resampled cells and, when the observation arm was re-randomised, up to 5.5% (log-rank) and 5.9% (stratified log-rank), MCSE about 0.5. Every model-based Cox test exceeded 5%, by 0.5-1.0 points for Cox-S (Wald, score and likelihood ratio) and 0.6-2.1 points for Cox-P; the Cox-S Wald test showed no trend in n (5.5%-5.9%). Each Cox-S Wald cell lies only 1.6-2.7 MCSE above 5%, so the systematic evidence is the null variance of : 1.035-1.065 for the Cox-S Wald statistic (MCSE 0.021-0.022) against 0.986-1.019 for the log-rank statistic, and at n = 600 1.049 (OS) and 1.035 (RFS) against of 1.040 and 1.024 (Table 4). For Cox-P, (1.036, 1.027) under-predicted the null variance (1.063, 1.053 at n = 600), as in the pooled world. Robust standard errors gave 5.3%-5.6% for Cox-S but up to 6.6% for Cox-P at n = 150. The re-randomised split reproduced the Cox-S excess (Wald 6.2% and 6.5%).
Table 3. Type I error (%) of nominal 5% two-sided tests in the observation-arm null world (primary).
| Test | OS 150 | OS 300 | OS 600 | RFS 150 | RFS 300 | RFS 600 | Re-rand. obs. arm, OS | Re-rand. obs. arm, RFS |
|---|---|---|---|---|---|---|---|---|
| Log-rank | 5.1 | 4.7 | 5.0 | 5.2 | 4.9 | 5.1 | 5.5 | 5.3 |
| Stratified log-rank | 5.3 | 4.8 | 5.2 | 4.9 | 5.4 | 5.3 | 5.9 | 5.0 |
| Cox-S, Wald (model SE) | 5.8 | 5.7 | 5.8 | 5.6 | 5.5 | 5.9 | 6.2 | 6.5 |
| Cox-S, Wald (robust SE) | 5.6 | 5.3 | 5.4 | 5.6 | 5.5 | 5.6 | 5.7 | 6.0 |
| Cox-S, score | 6.0 | 5.8 | 5.9 | 5.8 | 5.5 | 5.9 | 6.3 | 6.6 |
| CL-S | 5.8 | 5.1 | 5.4 | 5.7 | 5.3 | 5.1 | 5.5 | 5.3 |
| CL-S, corrected | 4.9 | 4.8 | 5.3 | 4.9 | 4.9 | 4.8 | 5.2 | 5.1 |
| Cox-P, Wald (model SE) | 6.4 | 5.6 | 6.3 | 6.9 | 5.9 | 6.0 | 6.4 | 6.3 |
| Cox-P, Wald (robust SE) | 5.9 | 5.4 | 5.8 | 6.6 | 5.6 | 5.7 | 5.8 | 6.2 |
| Cox-P, score | 6.6 | 5.7 | 6.3 | 7.1 | 6.0 | 6.0 | 6.5 | 6.5 |
| CL-P | 6.6 | 5.4 | 5.7 | 6.8 | 5.7 | 5.8 | 5.8 | 5.5 |
| CL-P, corrected | 4.8 | 4.6 | 5.3 | 5.0 | 4.8 | 5.4 | 5.1 | 5.1 |
Note: 5,000 resampled trials per cell (MCSE 0.30-0.36) and 2,000 random 157:158 splits of the 315 observation patients (MCSE 0.49-0.55). Likelihood-ratio tests were within 0.1 point of the score tests in every resampled null cell and within 0.2 under re-randomisation. Null variance of for the Cox-S Wald statistic: 1.063, 1.065 and 1.049 (OS) and 1.061, 1.058 and 1.035 (RFS), MCSE 0.021-0.022.
Choice of null world. Pooling both arms overstated the excess at n = 150 (Cox-S Wald 6.3% and 6.6%, against 5.8% and 5.6%), consistent with its larger (1.054 and 1.056, against 1.040 and 1.024), but the Lev+5FU arm alone gave as large an excess (5.5%-6.4%; 1.042 and 1.067): misspecification belongs to each arm's population, not only to pooling (Table 4; Supplementary Tables S1 and S2). In every resampled null cell the corrected CL and stratified log-rank tests stayed within 0.5 point of the log-rank test. The re-randomised trial, whose outcomes contain the treatment effect, does not check the pooled world independently. Non-converged fits, mostly from monotone likelihood [46], are counted under Table 4; excluding them changed rejection rates by at most 0.2 point (0.3 in the trial-preserving world).
Table 4. The three null worlds compared: range of type I error (%) over the six endpoint-by-size cells, null variance of z at n = 600 (OS / RFS) and .
| Test | Observation arm: type I error | Observation arm: var(z) | Lev+5FU arm: type I error | Lev+5FU arm: var(z) | Pooled: type I error | Pooled: var(z) |
|---|---|---|---|---|---|---|
| Log-rank | 4.7-5.2 | 0.986 / 0.996 | 4.2-5.7 | 1.034 / 1.035 | 4.6-5.3 | 1.002 / 0.990 |
| Stratified log-rank | 4.8-5.4 | 1.006 / 1.011 | 4.6-5.9 | 1.042 / 1.048 | 4.2-5.6 | 0.994 / 0.982 |
| Cox-S, Wald (model SE) | 5.5-5.9 | 1.049 / 1.035 | 5.5-6.4 | 1.107 / 1.129 | 5.4-6.6 | 1.065 / 1.059 |
| Cox-S, Wald (robust SE) | 5.3-5.6 | 1.009 / 1.015 | 5.2-5.9 | 1.060 / 1.065 | 5.0-6.3 | 1.017 / 1.014 |
| Cox-S, score | 5.5-6.0 | 1.053 / 1.038 | 5.6-6.6 | 1.112 / 1.134 | 5.5-6.9 | 1.069 / 1.062 |
| CL-S, corrected | 4.8-5.3 | 0.986 / 0.986 | 4.5-5.9 | 1.040 / 1.039 | 4.6-5.1 | 0.996 / 0.986 |
| Cox-P, Wald (model SE) | 5.6-6.9 | 1.063 / 1.053 | 6.0-7.3 | 1.145 / 1.177 | 5.9-7.3 | 1.101 / 1.099 |
| Cox-P, Wald (robust SE) | 5.4-6.6 | 1.025 / 1.028 | 5.2-6.7 | 1.080 / 1.088 | 5.1-6.7 | 1.034 / 1.035 |
| CL-P, corrected | 4.6-5.4 | 0.990 / 0.991 | 4.6-5.5 | 1.047 / 1.046 | 4.6-5.5 | 1.000 / 0.989 |
| , Cox-S (OS / RFS) | 1.040 / 1.024 | 1.042 / 1.067 | 1.054 / 1.056 | |||
| , Cox-P (OS / RFS) | 1.036 / 1.027 | 1.055 / 1.092 | 1.073 / 1.080 |
Note: 5,000 replicates per cell; MCSE of type I error 0.28-0.37 points, of var(z) 0.020-0.023 at n = 600. is the mean squared martingale residual per event of the covariate-only Cox model in the world's population (Section 2.3); for the covariate-free model it was 0.994-0.997. Implied asymptotic sizes for Cox-S were 5.5% and 5.3% (observation arm), 5.5% and 5.8% (Lev+5FU arm) and 5.6% (pooled). The Lev+5FU-arm log-rank variance at n = 600 (1.034 and 1.035; sizes 5.6% and 5.7%) is 1.6-1.7 MCSE above 1. We read it as chance, not an effect of drawing 600 patients from 304 (on average 600/304 = 1.97 appearances per patient per trial): the observation-arm null, at 600/315 = 1.90, gave 0.986 and 0.996, the same world gave 0.969 and 0.958 at n = 300, and OS and RFS share replicates. As all tests share those replicates, the evidence for misspecification within this arm is its , computed without resampling, and Cox-S Wald variances above the log-rank ones at every size (1.055-1.129 against 0.958-1.035), not the n = 600 sizes alone. Replicates with a non-converged Cox fit at n = 150, OS and RFS (of which flagged for monotone likelihood): observation-arm null 11 (6) and 7 (0) of 5,000; pooled null 54 (50) and 32 (14); Lev+5FU-arm null 236 (222) and 227 (136), plus 27 (15) and 14 (7) at n = 300; balanced world 27 (22) and 18 (13) of 2,000, plus 1 (1) for OS at n = 300; trial-preserving world 11 (10) and 8 (4) of 1,000.
Prespecified hypotheses. H1 (Cox-S RE at least 1.10 in all six cells, Wald type I error at most 6.0%) is refuted: RE fell below 1.10 in three cells (1.055, 1.066 and 1.091; Table 6), and in the prespecified pooled null the size criterion also failed at n = 150; with the observation-arm null it held, so H1 fails on efficiency alone. H2 (gain from estimand shift: estimate ratio above 1 with mean model-SE ratio at least 1) is supported for S and P in every cell (Table 7). H3 (inflation confined to small trials with many covariates, explained by model-SE underestimation, absent from score and likelihood-ratio tests) is refuted in every null world: in all five pooled-null cells with Wald inflation above 6.0% (OS and RFS at n = 150 for S and P; RFS at n = 300 for P) and the three observation-arm cells (Cox-P: OS at n = 150 and 600, RFS at n = 150), score and likelihood-ratio tests exceeded 6.0% too, although the empirical SE did exceed the model SE (ratios 1.033-1.103), as H3 required.
Power and efficiency. Nominally, Cox-S added 0.6-5.3 points of power over the log-rank test (paired MCSE 0.29-0.88) and reached RE 1.151 at 600 patients (Tables 5 and 6). Calibrated to the observation-arm null, its RE was 1.004-1.082, with intervals including 1 at n = 150; Cox-P had 0.914-1.084. The corrected CL tests reached 1.095-1.127 (S) and 1.038-1.120 (P), every interval above 1, approaching their predictions of 1.121-1.138 by n = 600; for S this means 8.7%-11.3% fewer events for equal power, against 0.4%-7.6% for Cox-S. The stratified log-rank test lost 1.0-5.6 points of power (paired MCSE 0.37-0.94), with calibrated RE 0.816-0.961 (4.1%-22.5% more events), every interval below 1; its large-sample prediction was also below 1 (0.893 for OS, 0.977 for RFS) but, from the same 619 patients and 12 strata (seven under 20 patients), cannot separate sparse strata from weaker within-stratum contrasts. In every cell and under all three calibrations the corrected CL test beat Cox-S, and Cox-S the stratified log-rank test (Table 6), and the model-based Cox predictions (1.133-1.211) far exceeded every calibrated Cox gain.
Table 5. Power (%) in the balanced world: nominal (size-adjusted to the observation-arm null).
| Test | OS 150 | OS 300 | OS 600 | RFS 150 | RFS 300 | RFS 600 |
|---|---|---|---|---|---|---|
| Log-rank | 30.7 (30.3) | 52.7 (53.6) | 81.9 (82.0) | 50.4 (50.1) | 80.0 (80.2) | 97.8 (97.8) |
| Stratified log-rank | 25.4 (24.8) | 47.1 (47.5) | 76.4 (75.9) | 46.3 (46.8) | 78.2 (76.8) | 96.8 (96.5) |
| Cox-S, Wald (model SE) | 32.2 (30.0) | 58.0 (55.9) | 87.2 (85.1) | 53.5 (51.7) | 82.4 (81.2) | 98.4 (98.0) |
| Cox-S, Wald (robust SE) | 30.7 (28.8) | 56.0 (55.3) | 85.7 (84.9) | 53.2 (50.5) | 81.7 (81.0) | 98.1 (97.9) |
| CL-S, corrected | 30.3 (31.0) | 57.1 (57.9) | 86.0 (85.4) | 53.8 (54.1) | 83.8 (83.8) | 98.6 (98.6) |
| Cox-P, Wald (model SE) | 31.2 (28.2) | 58.1 (56.0) | 86.7 (84.2) | 54.0 (49.0) | 82.8 (81.0) | 98.4 (98.0) |
| CL-P, corrected | 30.1 (30.6) | 57.0 (58.1) | 86.0 (85.2) | 52.4 (52.4) | 83.7 (84.1) | 98.6 (98.2) |
Note: 2,000 replicates; MCSE 0.25-1.12 points. Paired differences from the log-rank test (MCSE): Cox-S Wald 1.5 (0.78), 5.3 (0.85) and 5.3 (0.65) for OS and 3.0 (0.88), 2.5 (0.69) and 0.6 (0.29) for RFS; stratified log-rank -5.3 (0.86), -5.6 (0.89), -5.5 (0.74), -4.2 (0.94), -1.8 (0.79) and -1.0 (0.37).
Table 6. Relative efficiency versus the log-rank test (95% Monte Carlo interval).
| Test | OS 150 | OS 300 | OS 600 | RFS 150 | RFS 300 | RFS 600 |
|---|---|---|---|---|---|---|
| Nominal, balanced world | ||||||
| Cox-S, Wald (model SE) | 1.055 (1.027-1.083) | 1.105 (1.086-1.124) | 1.151 (1.137-1.164) | 1.066 (1.045-1.087) | 1.091 (1.077-1.106) | 1.117 (1.107-1.127) |
| Stratified log-rank | 0.808 (0.782-0.834) | 0.859 (0.841-0.878) | 0.882 (0.869-0.895) | 0.916 (0.895-0.938) | 0.956 (0.941-0.971) | 0.976 (0.966-0.986) |
| Calibrated to the observation-arm null, balanced world (primary) | ||||||
| Cox-S, Wald (model SE) | 1.004 (0.969-1.041) | 1.051 (1.021-1.082) | 1.082 (1.054-1.110) | 1.010 (0.978-1.043) | 1.051 (1.023-1.080) | 1.074 (1.046-1.102) |
| Cox-S, Wald (robust SE) | 0.984 (0.949-1.020) | 1.034 (1.004-1.064) | 1.069 (1.042-1.097) | 0.987 (0.956-1.020) | 1.030 (1.002-1.058) | 1.054 (1.027-1.082) |
| CL-S, corrected | 1.097 (1.064-1.131) | 1.109 (1.082-1.138) | 1.108 (1.083-1.133) | 1.095 (1.066-1.126) | 1.123 (1.096-1.150) | 1.127 (1.102-1.153) |
| Stratified log-rank | 0.816 (0.782-0.850) | 0.859 (0.832-0.888) | 0.865 (0.841-0.890) | 0.907 (0.874-0.940) | 0.952 (0.924-0.981) | 0.961 (0.935-0.988) |
| Cox-P, Wald (model SE) | 0.914 (0.877-0.953) | 1.034 (1.001-1.068) | 1.084 (1.054-1.116) | 0.941 (0.908-0.976) | 1.037 (1.006-1.068) | 1.074 (1.044-1.104) |
| CL-P, corrected | 1.055 (1.018-1.094) | 1.109 (1.078-1.142) | 1.105 (1.078-1.133) | 1.038 (1.006-1.071) | 1.111 (1.082-1.141) | 1.120 (1.093-1.148) |
| Calibrated to the pooled null, balanced world | ||||||
| Cox-S, Wald (model SE) | 0.970 (0.936-1.005) | 1.049 (1.020-1.080) | 1.083 (1.056-1.110) | 0.985 (0.954-1.016) | 1.031 (1.004-1.058) | 1.045 (1.020-1.071) |
| CL-S, corrected | 1.075 (1.042-1.108) | 1.101 (1.074-1.129) | 1.115 (1.090-1.140) | 1.086 (1.057-1.116) | 1.112 (1.086-1.138) | 1.122 (1.098-1.147) |
| Stratified log-rank | 0.801 (0.769-0.836) | 0.872 (0.844-0.901) | 0.890 (0.866-0.914) | 0.909 (0.877-0.942) | 0.971 (0.943-1.000) | 0.985 (0.960-1.011) |
| Calibrated to the observation-arm null, trial-preserving world | ||||||
| Cox-S, Wald (model SE) | 0.787 (0.750-0.826) | 0.839 (0.811-0.869) | 0.855 (0.830-0.880) | 0.815 (0.783-0.848) | 0.870 (0.844-0.898) | 0.883 (0.859-0.908) |
| CL-S, corrected | 0.900 (0.863-0.939) | 0.902 (0.875-0.930) | 0.898 (0.876-0.921) | 0.923 (0.892-0.955) | 0.945 (0.920-0.971) | 0.945 (0.922-0.968) |
| Stratified log-rank | 0.683 (0.645-0.723) | 0.694 (0.666-0.722) | 0.687 (0.665-0.711) | 0.764 (0.730-0.799) | 0.802 (0.775-0.830) | 0.791 (0.768-0.815) |
| Cox-P, Wald (model SE) | 0.740 (0.699-0.783) | 0.868 (0.835-0.901) | 0.898 (0.869-0.927) | 0.796 (0.761-0.833) | 0.901 (0.871-0.931) | 0.923 (0.896-0.951) |
| CL-P, corrected | 0.894 (0.851-0.938) | 0.938 (0.906-0.972) | 0.926 (0.900-0.952) | 0.909 (0.874-0.946) | 0.975 (0.946-1.005) | 0.974 (0.949-0.999) |
Note: intervals as in Section 3.5. Large-sample predictions from the balanced population: Cox-S 1.171 (OS) and 1.133 (RFS), Cox-P 1.211 and 1.157; CL-S 1.121 and 1.132, CL-P 1.132 and 1.138 (variance reductions 10.8%, 11.7%, 11.7% and 12.1%); stratified log-rank 0.893 and 0.977. The correction rescales the CL statistic, so its calibrated RE equals that of uncorrected CL. Lev+5FU-arm calibration: Cox-S 0.972-1.075, corrected CL-S 1.070-1.112, stratified log-rank 0.807-0.964.
The trial-preserving world. With the trial's imbalance kept, every adjusted test had calibrated RE below 1 (0.683-0.975; Table 6). This is not lost efficiency. The observation arm carries slightly more adverse factors (Table 1), so the unadjusted statistic profits from a fixed imbalance favouring Lev+5FU: at n = 600 the OS log-rank mean was -3.077, against -2.852 in the balanced world, while that of Cox-S was -2.934, against -3.059. The larger log-rank is a confounded contrast, which the balanced world removes.
Where the gain comes from. The conditional log hazard ratios were 7%-11% further from zero than the marginal one, the non-collapsibility shift, while the model-based SE grew only 1%-6% (Table 7); hence the nominal gain. But the empirical SE grew 5%-7% (S) and 8%-14% (P), almost cancelling the shift. Bias was negligible (at most 0.0139 in absolute value; Supplementary Table S3), and model-based intervals for the conditional log hazard ratios undercovered slightly (92.5%-95.3%; robust 93.4%-95.6%).
Table 7. Estimand shift and precision of the adjusted Cox estimators in the balanced world.
| Endpoint, adjustment set | n | Estimate ratio | Model-SE ratio | Empirical-SE ratio | Coverage, model (%) | Coverage, robust (%) |
|---|---|---|---|---|---|---|
| OS, S | 150 | 1.068 | 1.026 | 1.064 | 93.5 | 94.3 |
| OS, S | 300 | 1.076 | 1.014 | 1.047 | 94.5 | 95.2 |
| OS, S | 600 | 1.091 | 1.009 | 1.052 | 93.7 | 94.2 |
| OS, P | 150 | 1.084 | 1.061 | 1.140 | 93.0 | 93.7 |
| OS, P | 300 | 1.093 | 1.031 | 1.090 | 94.3 | 94.9 |
| OS, P | 600 | 1.110 | 1.018 | 1.080 | 93.5 | 94.0 |
| RFS, S | 150 | 1.077 | 1.026 | 1.066 | 93.2 | 93.8 |
| RFS, S | 300 | 1.074 | 1.014 | 1.046 | 95.3 | 95.6 |
| RFS, S | 600 | 1.079 | 1.009 | 1.051 | 93.8 | 94.4 |
| RFS, P | 150 | 1.113 | 1.059 | 1.140 | 92.5 | 93.4 |
| RFS, P | 300 | 1.096 | 1.031 | 1.080 | 94.8 | 95.3 |
| RFS, P | 600 | 1.097 | 1.018 | 1.075 | 93.4 | 94.0 |
Note: 2,000 replicates; MCSE of coverage about 0.5 points; marginal intervals covered 94.7%-95.7%. Ratios compare the conditional estimator with the unadjusted (marginal) one in the same replicates; true conditional log hazard ratios -0.376 (S) and -0.384 (P) for OS, -0.483 and -0.489 for RFS, against marginal -0.345 and -0.448. Bias, SEs and coverage of all three estimators: Supplementary Table S3.
4. Illustrative Example: The Trial Itself
The trial itself shows why one dataset cannot settle the question (Table 8). Given the imbalance in Table 1, adjusting for S moved the OS hazard ratio from 0.689 to 0.693 and lowered from 3.157 to 3.071, a realised squared ratio of 0.946 against the 1.171 the balanced population predicts; Cox-P raised to 3.203, and the stratified log-rank test gave 2.715 (OS) and 3.864 (RFS). All analyses agree on a substantial benefit, with hazard ratios between 0.614 and 0.693. If the randomisation was stratified on S, both the analysis and a re-randomisation check should have been stratified [3]: the stratified rows of Table 8 are then the design-based ones, while the unstratified rows and our simple re-randomisation treat the trial as simply randomised.
Table 8. Analysis of the actual trial (619 patients).
| Endpoint | Analysis | Hazard ratio (95% CI) | z |
|---|---|---|---|
| OS | Log-rank / unadjusted Cox | 0.689 (0.546-0.869) | -3.157 |
| OS | Stratified log-rank | - | -2.715 |
| OS | Cox-S, Wald | 0.693 (0.549-0.876) | -3.071 |
| OS | Cox-P, Wald | 0.681 (0.539-0.862) | -3.203 |
| OS | CL-S / CL-P (uncorrected) | - | -3.025 / -3.094 |
| RFS | Log-rank / unadjusted Cox | 0.621 (0.498-0.775) | -4.258 |
| RFS | Stratified log-rank | - | -3.864 |
| RFS | Cox-S, Wald | 0.625 (0.500-0.781) | -4.140 |
| RFS | Cox-P, Wald | 0.614 (0.491-0.767) | -4.290 |
| RFS | CL-S / CL-P (uncorrected) | - | -4.164 / -4.257 |
Note: for the first row of each endpoint is the log-rank statistic; the hazard ratio is from the unadjusted Cox model. Realised squared ratios versus the log-rank test: 0.739 (stratified log-rank), 0.946 (Cox-S) and 1.029 (Cox-P) for OS; 0.823, 0.945 and 1.015 for RFS. CL variance reductions: 11.0%/12.0% (OS), 11.8%/12.3% (RFS).
5. Discussion
In trials built from real NCCTG patients, prespecified Cox adjustment did not deliver the power its nominal output suggests. In null trials drawn from the observation arm its model-based tests rejected 5.5%-6.0% (S) and 5.6%-7.1% (P): less than our pooled null world suggested at n = 150, but at every size, of the size predicts for S, and shared by score and likelihood-ratio tests. Calibrated, Cox-S had RE 1.004-1.082, because the non-collapsibility shift behind the nominal gain [15-17] is matched by a larger sampling variance. The corrected CL test held its size and reached calibrated RE 1.095-1.127 with three factors, close to its prediction [29,32]. The stratified log-rank test lost efficiency instead of showing its asymptotic gain [5], as Akazawa and colleagues warned it might with many strata [10]; our design cannot tell whether sparse strata or within-stratum contrasts weaker than the marginal one are responsible.
The contrast with earlier simulations [6,7] has a simple reading: when outcomes come from a proportional hazards model containing the covariates, by construction and nominal and real gains coincide. Real outcomes do not; each arm alone gave of 1.024-1.067 for S. The choice of null population shifts calibrated RE appreciably, enough to decide whether Cox-S beats the log-rank test at 150 patients, but not the ordering of corrected CL, Cox-S and stratified log-rank tests.
Recommendations. For a randomised survival trial with moderately prognostic covariates and heavy censoring, as here: (i) to test the null of no effect, prefer the corrected covariate-adjusted log-rank test, which stayed within 0.5 point of the log-rank test's size in every null cell and was more efficient than both Cox-S and the stratified log-rank test; it targets the marginal estimand and yields no conditional hazard ratio [2,29]; (ii) if the prespecified estimand is the conditional hazard ratio, use the Cox model with robust standard errors, which were still anti-conservative with eight covariates and 150 patients (up to 6.6% in the observation-arm null) [1,47]; (iii) do not expect a stratified log-rank test with many sparse strata to add power (only the fully crossed 12 strata were studied, not coarser stratifications); (iv) predict the gain from the CL variance reduction in prior data [30], not from model-based Cox output; and (v) treat from prior data as a screening heuristic, not a correction, since it under-predicted the null variance for the larger covariate set.
Limitations. One trial, two correlated endpoints and one effect size per endpoint were studied, with mostly binary covariates and about half the patients censored; stronger continuous prognostic factors would change magnitudes. The resampling population is the trial's empirical distribution, chance subgroup structure included, which bears directly on the stratified test; duplicates create ties, though the resampling-free re-randomisations gave similar sizes. Sizes and calibrated RE depend on the population taken as null; we report three. Entropy balancing equalises covariate means only. The stratified test used only the fully crossed 12 strata; coarser stratifications were not examined. We could not confirm that S equals the randomisation strata, and all worlds mimic simple randomisation, so stratified designs, to which the argument of Kahan and Morris applies [3], were not studied; nor were Efron ties, restricted mean survival or other covariate codings.
Extensions. The design applies to any trial with patient-level data and would show how , the CL variance reduction and the cost of stratification vary across diseases and endpoints.
Data Availability and Software
The data are the public colon dataset of the R package survival (LGPL licence), from the Rdatasets mirror, https://vincentarelbundock.github.io/Rdatasets/csv/survival/colon.csv, retrieved on 24 October 2026. The analysis is a single deterministic Python 3 script (numpy, pandas; seed 20261024, with separate streams for the worlds added in revision) implementing the Cox fits and the log-rank, stratified log-rank and CL statistics in vectorised form; it ran in 98.3 seconds. The script and its complete output are archived with this article.
References
- Morris TP, Walker AS, Williamson EJ, White IR. Planning a method for covariate adjustment in individually randomised trials: a practical guide. Trials. 2022;23:328. https://doi.org/10.1186/s13063-022-06097-z
- Van Lancker K, Bretz F, Dukes O. Covariate adjustment in randomized controlled trials: general concepts and practical considerations. Clin Trials. 2024;21(4):399-411. https://doi.org/10.1177/17407745241251568
- Kahan BC, Morris TP. Improper analysis of trials randomised using stratified blocks or minimisation. Stat Med. 2012;31(4):328-340. https://doi.org/10.1002/sim.4431
- Cox DR. Regression models and life-tables. J R Stat Soc Series B Stat Methodol. 1972;34(2):187-202. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
- Ye T, Shao J, Yi Y. Log-rank and stratified log-rank tests. Stat Theory Relat Fields. 2023;7(4):309-317. https://doi.org/10.1080/24754269.2023.2263720
- Hernández AV, Eijkemans MJ, Steyerberg EW. Randomized controlled trials with time-to-event outcomes: how much does prespecified covariate adjustment increase power? Ann Epidemiol. 2006;16(1):41-48. https://doi.org/10.1016/j.annepidem.2005.09.007
- Kahan BC, Jairath V, Doré CJ, Morris TP. The risks and rewards of covariate adjustment in randomized trials: an assessment of 12 outcomes from 8 studies. Trials. 2014;15:139. https://doi.org/10.1186/1745-6215-15-139
- Benkeser D, Díaz I, Luedtke A, Segal J, Scharfstein D, Rosenblum M. Improving precision and power in randomized trials for COVID-19 treatments using covariate adjustment, for binary, ordinal, and time-to-event outcomes. Biometrics. 2021;77(4):1467-1481. https://doi.org/10.1111/biom.13377
- Jiang H, Symanowski J, Paul S, Qu Y, Zagar A, Hong S. The type I error and power of non-parametric logrank and Wilcoxon tests with adjustment for covariates: a simulation study. Stat Med. 2008;27(28):5850-5860. https://doi.org/10.1002/sim.3406
- Akazawa K, Nakamura T, Palesch Y. Power of logrank test and Cox regression model in clinical trials with heterogeneous samples. Stat Med. 1997;16(5):583-597. https://pubmed.ncbi.nlm.nih.gov/9089965/
- Gail MH, Wieand S, Piantadosi S. Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates. Biometrika. 1984;71(3):431-444. https://doi.org/10.1093/biomet/71.3.431
- Struthers CA, Kalbfleisch JD. Misspecified proportional hazard models. Biometrika. 1986;73(2):363-369. https://doi.org/10.1093/biomet/73.2.363
- Chastang C, Byar D, Piantadosi S. A quantitative study of the bias in estimating the treatment effect caused by omitting a balanced covariate in survival models. Stat Med. 1988;7(12):1243-1255. https://doi.org/10.1002/sim.4780071205
- Ford I, Norrie J, Ahmadi S. Model inconsistency, illustrated by the Cox proportional hazards model. Stat Med. 1995;14(8):735-746. https://doi.org/10.1002/sim.4780140804
- Martinussen T, Vansteelandt S. On collapsibility and confounding bias in Cox and Aalen regression models. Lifetime Data Anal. 2013;19(3):279-296. https://doi.org/10.1007/s10985-013-9242-z
- Aalen OO, Cook RJ, Røysland K. Does Cox analysis of a randomized survival study yield a causal treatment effect? Lifetime Data Anal. 2015;21(4):579-593. https://doi.org/10.1007/s10985-015-9335-y
- Daniel R, Zhang J, Farewell D. Making apples from oranges: comparing noncollapsible effect estimators and their standard errors after adjustment for different covariate sets. Biom J. 2021;63(3):528-557. https://doi.org/10.1002/bimj.201900297
- Robinson LD, Jewell NP. Some surprising results about covariate adjustment in logistic regression models. Int Stat Rev. 1991;59(2):227-240. https://doi.org/10.2307/1403444
- Lagakos SW, Schoenfeld DA. Properties of proportional-hazards score tests under misspecified regression models. Biometrics. 1984;40(4):1037-1048. https://doi.org/10.2307/2531154
- DiRienzo AG, Lagakos SW. Effects of model misspecification on tests of no randomized treatment effect arising from Cox's proportional hazards model. J R Stat Soc Series B Stat Methodol. 2001;63(4):745-757. https://doi.org/10.1111/1467-9868.00310
- Lin DY, Wei LJ. The robust inference for the Cox proportional hazards model. J Am Stat Assoc. 1989;84(408):1074-1078. https://doi.org/10.1080/01621459.1989.10478874
- Peduzzi P, Concato J, Feinstein AR, Holford TR. Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol. 1995;48(12):1503-1510. https://doi.org/10.1016/0895-4356(95)00048-8
- Vittinghoff E, McCulloch CE. Relaxing the rule of ten events per variable in logistic and Cox regression. Am J Epidemiol. 2007;165(6):710-718. https://doi.org/10.1093/aje/kwk052
- Senn S, König F, Posch M. Stratification in small randomised clinical trials and analysis of covariance: some simple theory and recommendations. Stat Methods Med Res. 2026;35(9):1838-1856. https://doi.org/10.1177/09622802261454833
- Tangen CM, Koch GG. Nonparametric analysis of covariance for hypothesis testing with logrank and Wilcoxon scores and survival-rate estimation in a randomized clinical trial. J Biopharm Stat. 1999;9(2):307-338. https://doi.org/10.1081/BIP-100101179
- Kong FH, Slud E. Robust covariate-adjusted logrank tests. Biometrika. 1997;84(4):847-862. https://doi.org/10.1093/biomet/84.4.847
- Lu X, Tsiatis AA. Improving the efficiency of the log-rank test using auxiliary covariates. Biometrika. 2008;95(3):679-694. https://doi.org/10.1093/biomet/asn003
- Tsiatis AA, Davidian M, Zhang M, Lu X. Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: a principled yet flexible approach. Stat Med. 2008;27(23):4658-4677. https://doi.org/10.1002/sim.3113
- Ye T, Shao J, Yi Y. Covariate-adjusted log-rank test: guaranteed efficiency gain and universal applicability. Biometrika. 2024;111(2):691-705. https://doi.org/10.1093/biomet/asad045
- Backenroth D, Roels S, Dibaj S, Ye T, Öhrn F, Van Lancker K. Practical considerations when using the covariate-adjusted log-rank test for the analysis of time-to-event endpoints in oncology trials. Biometrics. 2026;82(2):ujag109. https://doi.org/10.1093/biomtc/ujag109
- Bannick MS, Yi Y, Ye T. Covariate adjustment in randomized clinical trials: from general theory to practical insights. Clin Trials. 2026;23(3):272-281. https://doi.org/10.1177/17407745261442586
- Krotka P, Magirr D. Finite-sample correction for the covariate-adjusted log-rank test. arXiv:2608.10923 [stat.ME]. 2026. https://arxiv.org/abs/2608.10923
- Senn S. Covariate imbalance and random allocation in clinical trials. Stat Med. 1989;8(4):467-475. https://doi.org/10.1002/sim.4780080410
- Kinukawa N, Nakamura T, Akazawa K, Nose Y. The impact of covariate imbalance on the size of the logrank test in randomized clinical trials. Stat Med. 2000;19(15):1955-1967. https://pubmed.ncbi.nlm.nih.gov/10900445/
- Moertel CG, Fleming TR, Macdonald JS, Haller DG, Laurie JA, Goodman PJ, et al. Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma. N Engl J Med. 1990;322(6):352-358. https://doi.org/10.1056/NEJM199002083220602
- Moertel CG, Fleming TR, Macdonald JS, Haller DG, Laurie JA, Tangen CM, et al. Fluorouracil plus levamisole as effective adjuvant therapy after resection of stage III colon carcinoma: a final report. Ann Intern Med. 1995;122(5):321-326. https://doi.org/10.7326/0003-4819-122-5-199503010-00001
- Laurie JA, Moertel CG, Fleming TR, Wieand HS, Leigh JE, Rubin J, et al. Surgical adjuvant therapy of large-bowel carcinoma: an evaluation of levamisole and the combination of levamisole and fluorouracil. J Clin Oncol. 1989;7(10):1447-1456. https://doi.org/10.1200/JCO.1989.7.10.1447
- Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Polit Anal. 2012;20(1):25-46. https://doi.org/10.1093/pan/mpr025
- Therneau TM, Grambsch PM. Modeling Survival Data: Extending the Cox Model. New York: Springer; 2000. https://doi.org/10.1007/978-1-4757-3294-8
- Sargent DJ, Wieand HS, Haller DG, Gray R, Benedetti JK, Buyse M, et al. Disease-free survival versus overall survival as a primary end point for adjuvant colon cancer studies: individual patient data from 20,898 patients on 18 randomized trials. J Clin Oncol. 2005;23(34):8664-8670. https://doi.org/10.1200/JCO.2005.01.6071
- White IR, Thompson SG. Adjusting for partially missing baseline measurements in randomized trials. Stat Med. 2005;24(7):993-1007. https://doi.org/10.1002/sim.1981
- Peto R, Peto J. Asymptotically efficient rank invariant test procedures. J R Stat Soc Series A. 1972;135(2):185-207. https://doi.org/10.2307/2344317
- Breslow N. Covariance analysis of censored survival data. Biometrics. 1974;30(1):89-99. https://doi.org/10.2307/2529620
- Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Stat Med. 2019;38(11):2074-2102. https://doi.org/10.1002/sim.8086
- Schoenfeld D. Sample-size formula for the proportional-hazards regression model. Biometrics. 1983;39(2):499-503. https://doi.org/10.2307/2531021
- Heinze G, Schemper M. A solution to the problem of monotone likelihood in Cox regression. Biometrics. 2001;57(1):114-119. https://doi.org/10.1111/j.0006-341X.2001.00114.x
- Tackney MS, Morris T, White I, Leyrat C, Diaz-Ordaz K, Williamson E. A comparison of covariate adjustment approaches under model misspecification in individually randomized trials. Trials. 2023;24:14. https://doi.org/10.1186/s13063-022-06967-6
Supplementary Material
Supplementary Note S1. Computational details. All Cox models were fitted by Newton-Raphson with fixed iteration counts (six for the unadjusted model, seven for the covariate-only model and five more from it for the full model), steps capped at 5 in absolute value and a ridge of 1e-9 on the information. A fit counted as non-converged when its final step exceeded 1e-6, and monotone likelihood was flagged when a covariate coefficient exceeded 8 in absolute value. All statistics were computed in vectorised form over blocks of 250 replicates.
Table S1. Type I error (%) in the pooled null world and under re-randomisation of the trial.
| Test | OS 150 | OS 300 | OS 600 | RFS 150 | RFS 300 | RFS 600 | Re-rand. trial, OS | Re-rand. trial, RFS |
|---|---|---|---|---|---|---|---|---|
| Log-rank | 5.3 | 4.9 | 4.7 | 5.2 | 4.8 | 4.6 | 4.5 | 4.5 |
| Stratified log-rank | 5.6 | 4.9 | 4.6 | 5.4 | 4.8 | 4.2 | 5.0 | 5.4 |
| Cox-S, Wald (model SE) | 6.3 | 5.7 | 5.7 | 6.6 | 5.7 | 5.4 | 5.5 | 5.3 |
| Cox-S, Wald (robust SE) | 5.8 | 5.2 | 5.0 | 6.3 | 5.3 | 5.0 | 5.3 | 5.1 |
| Cox-S, score | 6.6 | 5.7 | 5.8 | 6.9 | 5.7 | 5.5 | 5.5 | 5.4 |
| CL-S | 5.6 | 5.1 | 4.9 | 5.8 | 5.4 | 4.9 | 4.7 | 5.0 |
| CL-S, corrected | 4.9 | 4.7 | 4.6 | 5.1 | 4.8 | 4.6 | 4.6 | 4.8 |
| Cox-P, Wald (model SE) | 7.1 | 5.9 | 5.9 | 7.3 | 6.1 | 6.0 | 5.9 | 5.9 |
| Cox-P, Wald (robust SE) | 6.5 | 5.3 | 5.1 | 6.7 | 5.3 | 5.3 | 5.5 | 5.5 |
| Cox-P, score | 7.3 | 6.1 | 5.9 | 7.6 | 6.2 | 6.1 | 5.9 | 5.9 |
| CL-P | 7.0 | 5.7 | 5.1 | 7.2 | 5.4 | 4.9 | 4.9 | 5.1 |
| CL-P, corrected | 5.1 | 4.7 | 4.6 | 5.5 | 4.6 | 4.6 | 4.5 | 4.8 |
Note: 5,000 replicates per cell (MCSE 0.28-0.37) and 2,000 re-randomisations of the 619 patients, 304:315 (MCSE 0.46-0.53). Null variance of for the Cox-S Wald statistic: 1.108, 1.052 and 1.065 (OS) and 1.114, 1.049 and 1.059 (RFS), MCSE 0.020-0.023; under re-randomisation 1.084 and 1.080 (MCSE 0.035), against 1.007 and 0.988 for the log-rank test (MCSE 0.031).
Table S2. Type I error (%) in the Lev+5FU-arm null world.
| Test | OS 150 | OS 300 | OS 600 | RFS 150 | RFS 300 | RFS 600 |
|---|---|---|---|---|---|---|
| Log-rank | 5.0 | 4.2 | 5.6 | 5.0 | 4.4 | 5.7 |
| Stratified log-rank | 4.7 | 4.6 | 5.9 | 5.1 | 4.8 | 5.7 |
| Cox-S, Wald (model SE) | 6.1 | 5.5 | 6.3 | 6.4 | 5.9 | 6.3 |
| Cox-S, Wald (robust SE) | 5.9 | 5.2 | 5.8 | 5.9 | 5.2 | 5.6 |
| Cox-S, score | 6.4 | 5.6 | 6.3 | 6.6 | 6.1 | 6.4 |
| CL-S | 5.9 | 5.1 | 6.0 | 6.0 | 5.0 | 5.6 |
| CL-S, corrected | 5.3 | 4.5 | 5.9 | 5.1 | 4.6 | 5.4 |
| Cox-P, Wald (model SE) | 7.0 | 6.0 | 6.6 | 7.3 | 6.5 | 7.0 |
| Cox-P, Wald (robust SE) | 6.3 | 5.2 | 5.8 | 6.7 | 5.6 | 6.0 |
| Cox-P, score | 7.1 | 6.2 | 6.7 | 7.4 | 6.5 | 7.1 |
| CL-P | 7.2 | 5.4 | 5.8 | 7.2 | 5.5 | 5.7 |
| CL-P, corrected | 5.3 | 4.6 | 5.5 | 5.3 | 4.8 | 5.3 |
Note: 5,000 replicates per cell (MCSE 0.28-0.37). Null variance of for the Cox-S Wald statistic: 1.071, 1.055 and 1.107 (OS) and 1.095, 1.070 and 1.129 (RFS), MCSE 0.021-0.022. Non-converged replicates are reported in Section 3.6.
Table S3. Estimation of the marginal and conditional log hazard ratios in the balanced world.
| Endpoint, estimand (true value) | n | Bias | Empirical SE | Model SE | Robust SE | Coverage, model (%) | Coverage, robust (%) |
|---|---|---|---|---|---|---|---|
| OS, marginal (-0.345) | 150 | -0.0055 | 0.2422 | 0.2436 | - | 95.0 | - |
| OS, marginal (-0.345) | 300 | -0.0047 | 0.1691 | 0.1713 | - | 95.6 | - |
| OS, marginal (-0.345) | 600 | 0.0022 | 0.1197 | 0.1209 | - | 95.2 | - |
| OS, conditional S (-0.376) | 150 | 0.0021 | 0.2577 | 0.2500 | 0.2557 | 93.5 | 94.3 |
| OS, conditional S (-0.376) | 300 | 0.0001 | 0.1770 | 0.1737 | 0.1782 | 94.5 | 95.2 |
| OS, conditional S (-0.376) | 600 | 0.0025 | 0.1259 | 0.1219 | 0.1251 | 93.7 | 94.2 |
| OS, conditional P (-0.384) | 150 | 0.0041 | 0.2761 | 0.2584 | 0.2673 | 93.0 | 93.7 |
| OS, conditional P (-0.384) | 300 | 0.0021 | 0.1842 | 0.1767 | 0.1826 | 94.3 | 94.9 |
| OS, conditional P (-0.384) | 600 | 0.0037 | 0.1292 | 0.1231 | 0.1270 | 93.5 | 94.0 |
| RFS, marginal (-0.448) | 150 | -0.0038 | 0.2333 | 0.2313 | - | 94.7 | - |
| RFS, marginal (-0.448) | 300 | -0.0015 | 0.1595 | 0.1627 | - | 95.7 | - |
| RFS, marginal (-0.448) | 600 | 0.0006 | 0.1141 | 0.1149 | - | 94.8 | - |
| RFS, conditional S (-0.483) | 150 | -0.0042 | 0.2488 | 0.2374 | 0.2406 | 93.2 | 93.8 |
| RFS, conditional S (-0.483) | 300 | 0.0001 | 0.1668 | 0.1650 | 0.1680 | 95.3 | 95.6 |
| RFS, conditional S (-0.483) | 600 | 0.0000 | 0.1200 | 0.1159 | 0.1182 | 93.8 | 94.4 |
| RFS, conditional P (-0.489) | 150 | -0.0139 | 0.2660 | 0.2450 | 0.2505 | 92.5 | 93.4 |
| RFS, conditional P (-0.489) | 300 | -0.0037 | 0.1723 | 0.1677 | 0.1719 | 94.8 | 95.3 |
| RFS, conditional P (-0.489) | 600 | -0.0019 | 0.1227 | 0.1169 | 0.1200 | 93.4 | 94.0 |
Note: 2,000 replicates; MCSE of coverage about 0.5 points. Root mean squared error differed from the empirical SE by at most 0.0003. True values are the weighted full-data fits of the balanced population.
6.Review
Automated review: Accept (novelty 4, rigor 4, clarity 4, reproducibility 4).
| Novelty 4/5 | Rigor 4/5 | Clarity 4/5 | Reproducibility 4/5 |
Refereed by Claude Fable 5.1, a different model from the one that wrote the paper (Claude Opus 5.5).
This third revision is text-only and makes every change my second report required, each one checked against the manuscript and the unchanged executed record. The abstract no longer overgeneralises the pooled-null comparison; the size ranges in Section 3.6 are scoped to the cells they describe, with the re-randomised observation arm reported separately and the 'within 0.5 point' claim correctly narrowed to the resampled null cells; the stratified log-rank result is interpreted without claiming a separation the design cannot make, and the recommendation about stratification is confined to the 12-stratum design studied; the Lev+5FU-arm log-rank variance at n = 600 is read as chance with supporting figures that are all in the record; and Section 2.1 states what the trial reports do and do not say. The paper's findings stand as verified in my earlier reports: in null trials drawn from the observation arm the model-based Cox-S Wald test rejects 5.5%-5.9% with null var(z) 1.035-1.065 matching the control-arm phi-hat, its null-calibrated relative efficiency is 1.004-1.082 against 1.095-1.127 for the corrected covariate-adjusted log-rank test, and the 12-stratum stratified log-rank test holds its size but loses efficiency; the ordering holds in all cells under all three calibrations. I have no further required changes and recommend acceptance.
Strengths
- All five text changes from my second report are made exactly and accurately. The abstract and Discussion now say the pooled null overstated the excess 'at n = 150 (6.3% and 6.6%)'; the opening of Section 3.6 is scoped to the six resampled cells with the re-randomised observation-arm sizes (5.5% log-rank, 5.9% stratified log-rank, MCSE about 0.5) added; the stratified log-rank interpretation now says the population prediction, computed from the same 619 patients and 12 strata, cannot separate sparse strata from weaker within-stratum contrasts, and recommendation (iii) and the Limitations are scoped to the fully crossed 12-stratum design; the Table 4 note gives a reasoned reading of the Lev+5FU-arm log-rank variance at n = 600; Section 2.1 reads as a statement of record.
- The author also narrowed 'within 0.5 point of the log-rank test in every null cell' to 'every resampled null cell', which is correct: under whole-trial re-randomisation the RFS stratified log-rank test is 0.9 points above the log-rank test (5.4% against 4.5%). For the corrected CL test alone the claim holds in every null cell including both re-randomisations (largest gap 0.4 points), so recommendation (i) is right as written.
- Every new number in the revised text is in the unchanged record: 600/304 = 1.97 and 600/315 = 1.90 appearances per patient, observation-arm log-rank var(z) 0.986 and 0.996 at n = 600, Lev+5FU-arm 0.969 and 0.958 at n = 300, and Cox-S Wald var(z) 1.055-1.129 above log-rank 0.958-1.035 in each of the six Lev+5FU cells. RESULT_JSON is unchanged (same script, same runtime 98.3 s), and all tables still match it.
- The manuscript now meets the venue's conventions throughout: ADEMP structure, nominal levels, MCSEs for rejection rates, power, coverage, relative efficiency and null variances, every performance measure defined, notation before use, pre-specified hypotheses reported with their outcomes, and recommendations confined to the scenarios studied.
Weaknesses
- The study remains one trial with two correlated endpoints and one effect size per endpoint; the magnitudes (phi-hat 1.02-1.09, CL variance reduction 11%-12%, the 12-stratum loss) are specific to this population, as the paper states.
- The trial's stratification factors remain unconfirmed; the paper treats S as candidate stratification factors and mimics simple randomisation, which is the honest handling, but it means stratified designs are outside the scenarios studied.
- The main text is somewhat above the journal's length guide; the supplement already carries the secondary tables, so this is for the editor rather than a scientific concern.
Required changes
—
Record
§Event timeline
- Run createdQuestion: “Does prespecified covariate adjustment in the Cox model really increase power over the unadjusted log-rank test in a randomised survival trial? A real-data resampling study of the NCCTG adjuvant colon cancer trial' language=en discipline=biostatistics venue='Statistics in Medicine” · output language: English · models: agent:claude-opus-5-5
- Stage started · Literature reviewAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Literature reviewsearch response for literature_review: literature_review-search-1.response.json
- Resume requested
- Stage started · Literature reviewAttempt 2 · agent:claude-opus-5-5
- Search complete · Literature review55 sources · 16,840 characters of notes
- awaiting_agent · Literature reviewcomplete_json response for literature_review: literature_review-complete_json-1.response.json
- Resume requested
- Stage started · Literature reviewAttempt 3 · agent:claude-opus-5-5
- Search complete · Literature review55 sources · 16,840 characters of notes
- Stage completed · Literature review
- Stage started · IdeationAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Ideationcomplete_json response for ideation: ideation-complete_json-1.response.json
- Resume requested
- Stage started · IdeationAttempt 2 · agent:claude-opus-5-5
- Awaiting approval · Ideation
- Approved · IdeationIdea 1 selected
- Stage started · ExperimentationAttempt 1 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationcolon_cancer_trial · 1,858 observations · 16 variables · sha256 6f3472a64f69
- awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
- Resume requested
- Stage started · ExperimentationAttempt 2 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationcolon_cancer_trial · 1,858 observations · 16 variables · sha256 6f3472a64f69
- Experiment round · ExperimentationRound 1: exit 0; result parsed (55.86s)
- Stage completed · Experimentation
- Stage started · AnalysisAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 2 · agent:claude-opus-5-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 2 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-opus-5-5
- Awaiting approval · Review
- Sent back for revision · ReviewRe-running from Paper writing: “The referee recommended minor revision. Make these required changes, quoted from the report, using only results the executed script already produced (no new analyses or numbers):\n1. Reword the phi cla”
- Stage started · Paper writingAttempt 3 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 4 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-opus-5-5
- Awaiting approval · Review
- Approved · Review
- Run completereport=/Users/haodingresearch/Desktop/auto-research/runs/studies-opus55/biostatistics-en/runs/418fc309f9c1/REPORT.md
- Reopened after sign-off · ReviewReopened after sign-off; re-running from Review: “Independent review: the paper is re-refereed by an independent referee (Claude Fable 5.1) that took no part in writing it.”
- Stage started · ReviewAttempt 3 · agent:claude-fable-5-1
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 4 · agent:claude-fable-5-1
- Awaiting approval · Review
- Sent back for revision · ReviewRe-running from Experimentation: “Independent referee (Claude Fable 5.1): major_revision. Required changes: (1) [analysis] Add a null world resampled from a single arm (the 315 observation patients; preferably also the 304 Lev+5FU pat”
- Stage started · ExperimentationAttempt 3 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationcolon_cancer_trial · 1,858 observations · 16 variables · sha256 6f3472a64f69
- awaiting_agent · Experimentationcomplete response for experimentation: experimentation-complete-1.response.txt
- Resume requested
- Stage started · ExperimentationAttempt 4 · agent:claude-opus-5-5
- Dataset provisioned · Experimentationcolon_cancer_trial · 1,858 observations · 16 variables · sha256 6f3472a64f69
- Experiment round · ExperimentationRound 1: exit 0; result parsed (98.375s)
- Stage completed · Experimentation
- Stage started · AnalysisAttempt 1 · agent:claude-opus-5-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 2 · agent:claude-opus-5-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 5 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 6 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-fable-5-1
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-fable-5-1
- Awaiting approval · Review
- Sent back for revision · ReviewRe-running from Analysis: “Independent referee (Claude Fable 5.1), round 2: minor_revision. Required changes: (1) [text] Abstract: replace 'A null world pooling both arms overstated the excess (5.4%-6.6%)' with a statement limi”
- Stage started · AnalysisAttempt 3 · agent:claude-opus-5-5
- awaiting_agent · Analysiscomplete_json response for analysis: analysis-complete_json-1.response.json
- Resume requested
- Stage started · AnalysisAttempt 4 · agent:claude-opus-5-5
- Stage completed · Analysis
- Stage started · Paper writingAttempt 7 · agent:claude-opus-5-5
- awaiting_agent · Paper writingcomplete response for paper_writing: paper_writing-complete-1.response.txt
- Resume requested
- Stage started · Paper writingAttempt 8 · agent:claude-opus-5-5
- Stage completed · Paper writing
- Stage started · ReviewAttempt 1 · agent:claude-fable-5-1
- awaiting_agent · Reviewcomplete_json response for review: review-complete_json-1.response.json
- Resume requested
- Stage started · ReviewAttempt 2 · agent:claude-fable-5-1
- Awaiting approval · Review
- Approved · Review
- Run completereport=/Users/haodingresearch/Desktop/auto-research/runs/studies-opus55/biostatistics-en/runs/418fc309f9c1/REPORT.md