Anytime-valid inference

E-values, mixture likelihood ratios and confidence sequences: monitor continuously, with no fixed looks or horizon, and still control the type I error.

Monitoring without a plan

Group sequential designs fix the maximum sample size and the spending of \(\alpha\) in advance. Some settings do not allow that:

  • online experiments that run until a decision is needed;
  • a study that may be extended when new funding arrives;
  • a living systematic review that is updated whenever a new trial is published.

Anytime-valid methods control the type I error at any stopping time, however it is chosen (Ramdas et al. 2023).

E-values and e-processes

An e-value is a non-negative statistic \(E\) whose expectation under \(H_0\) is at most 1. By Markov’s inequality, \(P_{H_0}(E \ge 1/\alpha) \le \alpha\). A sequence of e-values that forms a non-negative supermartingale under \(H_0\) — an e-process — satisfies Ville’s inequality:

\[ P_{H_0}\!\left(\exists\, n:\ E_n \ge 1/\alpha\right) \le \alpha . \]

The data can be monitored after every observation and the test rejects the first time \(E_n \ge 1/\alpha\) (20 for \(\alpha = 0.05\)). The type I error stays controlled even if the study continues beyond any planned size. E-values can also be multiplied across independent studies (Vovk and Wang 2021; Grünwald et al. 2024).

Any likelihood ratio against a fixed alternative is an e-process — Wald’s SPRT statistic is one. Averaging the likelihood ratio over a distribution of alternatives keeps the property. For normal data with variance \(\sigma^2\) and a normal mixing distribution with variance \(\tau^2\), this gives Robbins’ mixture likelihood ratio (Robbins 1970):

\[ E_n = \sqrt{\frac{\sigma^2}{\sigma^2 + n\tau^2}}\; \exp\!\left(\frac{n^2\tau^2(\bar x_n - \theta_0)^2}{2\sigma^2(\sigma^2 + n\tau^2)}\right). \]

Its reciprocal gives the always-valid p-value used by several industry A/B-testing platforms (the mixture SPRT, mSPRT) (Johari et al. 2022).

Confidence sequences

Inverting an e-process gives a confidence sequence: a sequence of intervals that contains \(\theta\) at all sample sizes simultaneously with probability \(1-\alpha\). For the normal mixture above, the interval at sample size \(n\) is

\[ \bar x_n \pm \sqrt{\frac{2\sigma^2(\sigma^2 + n\tau^2)}{n^2\tau^2}\, \ln\!\left(\frac{\sqrt{1 + n\tau^2/\sigma^2}}{\alpha}\right)} . \]

Nonparametric and asymptotic versions exist (Howard et al. 2021; Waudby-Smith et al. 2024).

The price of this flexibility is width. A fixed-sample confidence interval is valid only at the one sample size chosen in advance; a confidence sequence is valid at all of them.

Figure 1: Half-width of a 95 % confidence sequence (normal mixture tuned for n = 100) and of the fixed-sample 95 % confidence interval, sigma = 1.

At the tuning point \(n = 100\) the confidence sequence is 55 % wider than the fixed interval.

Safe tests

Grünwald et al. (2024) construct safe tests: e-values that grow as fast as possible under the alternative (GROW optimality), for the \(t\)-test, 2×2 tables and other standard settings. An anytime-valid log-rank test exists for survival data (Schure et al. 2024). ALL-IN meta-analysis applies the same ideas to living systematic reviews (Schure and Grünwald 2022).

Compared with a well-planned group sequential design, anytime-valid methods need a somewhat larger sample at a given horizon. In exchange, the analysis plan does not have to be fixed in advance.

Example: planning and running an anytime-valid test

The price of anytime validity. Take a one-sample test with known SD, a standardised effect of 0.5, two-sided \(\alpha = 0.05\) and power 80 %. The mixture e-value uses \(g = \delta^2 = 0.25\). The Anytime-Valid Monitoring analysis computes its operating characteristics exactly: with a known SD, “reject when \(E_n \ge 20\)” is a boundary on \(|z_n|\), handled by the same numerical integration as group sequential designs.

Design N for 80 % power
Fixed-sample z-test, one analysis 32
The e-value, one analysis 61
The e-value, tested after every observation 54

Testing after every observation needs 54 patients to reach 80 % power, against 32 for the fixed design. Under the design effect the test usually stops much earlier: the expected sample size is 36.6 and the median 32. The R package safestats estimates the same design by simulation (55 ± 3 patients). For an unknown SD the analysis simulates the safe \(t\)-test. Two samples with \(d = 0.5\) need 218 patients in total for 80 % power with continuous monitoring, against 130 for the fixed-sample \(t\)-test.

Monitoring. On the dataset Sequential outcomes (80 patients in arrival order, true difference 0.6 SD), the two-sample safe \(t\)-test first reaches \(E \ge 20\) after 73 patients. From then on the 95 % confidence sequence excludes 0. The sequential t-test on the same data stops after 41 patients. It is one-sided, bets on the single alternative \(d = 0.5\), and rejects at a likelihood ratio of \((1-\beta)/\alpha = 16\). The mixture e-value is two-sided and spreads its bet over all effect sizes. That costs speed when the true effect is close to the planned one, and gains robustness when it is not.

Figure 2: E-process of the two-sample safe \(t\)-test on the example data; H0 is rejected when it first reaches 1/α = 20.
TipIn jmsequential

Sequential → Anytime-Valid Monitoring (version 0.5.0).

  • Tests: mean (one sample) or difference in means (two samples). With a known SD the test is the normal-mixture mSPRT; with an unknown SD it is the safe \(t\)-test. Both are two-sided, with a Gaussian mixing variance \(g\) (default \(\delta^2\)).
  • Design: the N for the target power when testing after every observation, exact for a known SD and simulated otherwise. It also gives the expected, median and 90th-percentile stopping N, the type I error by the horizon, and the N for one analysis with the e-value and for the fixed-sample test. A plot shows the probability of rejecting by each N.
  • Data: an outcome in arrival order (and a group variable for two samples). At each shown observation the analysis reports the e-value, the always-valid p-value, the confidence sequence and the decision, with plots of the e-process and of the confidence sequence.

The example in jamovi

  1. Open the data set Sequential outcomes - patients in arrival order (☰ → Open → Data Library → jmsequential).
  2. Sequential → Anytime-Valid Monitoring: the defaults are the two-sample test of the example (unknown SD, hence the safe t-test; α = 0.05, d = 0.5, power 0.8).
  3. In the Analysis of observed data box, move score to Outcome and arm to Group.

The Monitoring table shows every fourth patient and the decision: E first reaches 1/α = 20 after 73 patients (E = 24.3), and from then on the confidence sequence excludes 0. The Design table above it gives the planned N; the E-process plot further down is Figure 2.

Back to top

References

Grünwald, Peter, Rianne de Heide, and Wouter Koolen. 2024. “Safe Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology 86 (5): 1091–128. https://doi.org/10.1093/jrsssb/qkae011.
Howard, Steven R., Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. 2021. “Time-Uniform, Nonparametric, Nonasymptotic Confidence Sequences.” The Annals of Statistics 49 (2). https://doi.org/10.1214/20-aos1991.
Johari, Ramesh, Pete Koomen, Leonid Pekelis, and David Walsh. 2022. “Always Valid Inference: Continuous Monitoring of a/b Tests.” Operations Research 70 (3): 1806–21. https://doi.org/10.1287/opre.2021.2135.
Ramdas, Aaditya, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. 2023. “Game-Theoretic Statistics and Safe Anytime-Valid Inference.” Statistical Science 38 (4). https://doi.org/10.1214/23-sts894.
Robbins, Herbert. 1970. “Statistical Methods Related to the Law of the Iterated Logarithm.” The Annals of Mathematical Statistics 41 (5): 1397–409. https://doi.org/10.1214/aoms/1177696786.
Schure, Judith ter, and Peter Grünwald. 2022. “ALL-IN Meta-Analysis: Breathing Life into Living Systematic Reviews.” F1000Research 11 (May): 549. https://doi.org/10.12688/f1000research.74223.1.
Schure, Judith ter, Muriel F. Pérez-Ortiz, Alexander Ly, and Peter D. Grünwald. 2024. “The Anytime-Valid Logrank Test: Error Control Under Continuous Monitoring with Unlimited Horizon.” The New England Journal of Statistics in Data Science, 190–214. https://doi.org/10.51387/24-nejsds65.
Vovk, Vladimir, and Ruodu Wang. 2021. “E-Values: Calibration, Combination and Applications.” The Annals of Statistics 49 (3). https://doi.org/10.1214/20-aos2020.
Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. 2024. “Time-Uniform Central Limit Theory and Asymptotic Confidence Sequences.” The Annals of Statistics 52 (6). https://doi.org/10.1214/24-aos2408.