ALL-IN meta-analysis stands for Anytime Live and Leading INterim meta-analysis.
This statistical methodology provides meta-analysis of interim data that can be updated at Anytime – so after each new observed data point – Live – without the need to prespecify when to look; we can spot live trends in the evidence – and Leading – because the evidence in the meta-analysis can inform whether individual trials should be stopped or expanded.
ALL-IN-META-BCG-CORONA intends to ALL-IN meta-analyze the time-to-event data from clinical trials studying the Bacillus Calmette-Guérin vaccination in health care workers during the SARS-CoV-2 pandemic and uses the Safe logrank test and Safe hazard ratio confidence sequences.
ALL-IN-META-BCG-ELDERLY intends to ALL-IN meta-analyze the time-to-event data from clinical trials studying the Bacillus Calmette-Guérin vaccination in elderly with or without chronic comorbidities during the SARS-CoV-2 pandemic and uses the Safe logrank test and Safe hazard ratio confidence sequences.
Both collaborations aim (1) to prevent the detrimental effects of a putative false-positive interim trial result by immediately confirming or negating the result through pooling of all interim trial results, and (2) to increase the chance of identifying the putative beneficial effects of BCG by continuously pooling all interim trial results.
The analysis is literally ALL-IN: any new trial result can be included, even if the decision to start or expand a trial was based on the data already available.
F1000 preprint paper: J. ter Schure & P.D. Grünwald (2022) ALL-IN meta-analysis: breathing life into living systematic reviews.
R package: safestats on CRAN.
ArXiv preprint paper: P.D. Grünwald, R. de Heide & W. Koolen (2019) Safe Testing.
Please find a recording of a talk by Peter Grünwald at the International Selective Inference Seminar here: E is the new P.
Please find a link to Peter Grünwald's research website Safe Statistics here.
ArXiv preprint paper: R. Turner, A. Ly & P.D. Grünwald (2022) Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping.
Chapter 2 of Ph.D. thesis: J. ter Schure, M.F. Pérez-Ortiz, A. Ly & P.D. Grünwald (2020) The Safe Logrank Test.
Please find a tutorial on the use of the Safe Logrank test for ALL-IN meta-analysis here.
Please find a tutorial on left-truncation here.
Please find a tutorial on staggered entry here.
Please find the Newsletters sent around with updates below:
2020-10_Newsletter_1_ALL-IN-META-BCG-CORONA.
2020-11_Newsletter_2_ALL-IN-META-BCG-CORONA.
2021-05_Newsletter_3_ALL-IN-META-BCG-CORONA.
2022-09_Newsletter_4_ALL-IN-META-BCG-CORONA.
Please find the recorded webinars on ALL-IN-META-BCG-CORONA in a Youtube playlist here,
Webinar Part 1 requires only basic statistical knowledge, Webinar Part 2 contains more technical details.
Please find the first version (June 17, 2020) of the Statistical Analysis Plan of ALL-IN-META-BCG-CORONA here.
Please find the second version (September 19, 2022) of the Statistical Analysis Plan of ALL-IN-META-BCG-CORONA here.
Please find the first version (June 16, 2020) of the working instructions for ALL-IN-META-BCG-CORONA data extraction and upload here.
Please find the second version (October 31, 2022) of the working instructions for ALL-IN-META-BCG-CORONA data extraction and upload here.
Please find the ALL-IN-META-BCG-CORONA-dashboard here.
Please find example code on how to process your BCG-CORONA data set into an e-value sequence by calendar date here.
Please find the Statistical Analysis Plan of ALL-IN-META-BCG-ELDERLY here.
Please find the working instructions for ALL-IN-META-BCG-ELDERLY data extraction and upload here.
Please find the ALL-IN-META-BCG-ELDERLY-dashboard here (currently FAKE data only, will soon be replaced by real link).
Please find example code on how to process your BCG-ELDERLY data set into an e-value sequence by calendar date here.
Suppose we have a medical question that urgently needs a reliable answer. Suppose further that this question is considered by multiple clinical trials running at the same time or at least overlapping in time. To provide a reliable answer as fast as possible, we want to combine the results, use interim analyses and/or encourage trials that finish early to inform whether others should be stopped. In principle, this can provide more power, and in case of time-to-event analysis, more early events to support sufficient evidence at interim. Furthermore, early stopping might prevent many events that otherwise occur if all trials are continued to their prespecified completion. However, in such an interim meta-analysis scenario, standard analysis is not able to smoothly retain type-I error guarantees.
Take the example in which trial results are only evaluated in isolation and a response follows the first positive result of a single trial. Such multiple testing inflates the type-I error dramatically if individual trials are evaluated with a standard methods. Part of the solution to this problem lies in pooling the trials: always evaluate (interim) trial results in unison and regain type-I error control over the meta-analysis. But new problems arise, because ideally, we would meta-analyze as soon as an interim result looks encouraging, but would like to test again if the first meta-analysis does not yet conclude the research effort. So, multiple testing issues arise also in meta-analysis, and standard analysis of isolated trials as well as meta-analysis end up with an unreliable inference and, subsequently, poor decisions.
The solution lies in a new statistical method that allows to flexibly pool trials, increase power and efficiency by unlimited interim analyses, while always retaining type-I error guarantees. The theory underlying such an interim meta-analysis is that of Safe Testing (Grünwald, Heide, and Koolen (2019)) with its basic tool, the e-value. These e-values are analogous to p-values, or rather their inverse: a large e-value indicates evidence against the null hypothesis. But, completely unlike p-values, e-values of several trials can be combined effortlessly – by multiplication – without compromising type-I error. Such combination does not have to be prespecified and can be done even if the trials are dependent, e.g if the decision to start or stop an extra trial depends on (interim) results of earlier trials.
The Safe Testing methodology is described in detail by Grünwald, Heide, and Koolen (2019). Its relation to dependency problems in meta-analysis, called accumulation bias, is outlined in Ter Schure and Grünwald (2019). Safe Tests are an extension of earlier work by Vovk, Shafer and others on so-called martingale tests, and Robbins and Lai on always-valid tests, e.g. Shafer et al. (2011), Robbins (1970). This latter work was a precursor to both Safe Testing and group sequential testing with alpha-spending approaches.
It could, in fact, be possible to combine interim test results via an alpha-spending or a group-sequential approach. But in contexts of multiple trials with no top-down planning these have considerable disadvantages compared to Safe Testing. Most importantly, they are much less flexible. Suppose, for example, that a new study is started halfway through our analysis; a study we did not know about when designing our meta-analyses procedure. Because alpha-spending requires us to know and fix the number of trials beforehand, it would not allow us to include this additional study. Fortunately, with Safe Testing accounting for the new study is no problem at all.
Here is a link to a blogpost about the use of Safe testing in BCG Corona research.
Grünwald, Peter, Rianne de Heide, and Wouter Koolen. 2019. “Safe Testing.” arXiv Preprint arXiv:1906.07801.
Robbins, Herbert. 1970. “Statistical Methods Related to the Law of the Iterated Logarithm.” Annals of Mathematical Statistics 41: 1397–1409.
Schure, Judith ter, and Peter Grünwald. 2019. “Accumulation Bias in Meta-Analysis: The Need to Consider Time in Error Control [Version 1; Peer Review: 2 Approved].” F1000Research 8 (June): 962. https://doi.org/10.12688/f1000research.19375.1.
Shafer, Glenn, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. 2011. “Test Martingales, Bayes Factors and P-Values.” Statistical Science 26 (1): 84–101.