Fragility Index for Time-to-Event Endpoints in Single-Arm Clinical Trials

Abstract.

The reliability of clinical trial outcomes is crucial, especially in guiding medical decisions. In this paper, we introduce the Fragility Index (FI) for time-to-event endpoints in single-arm clinical trials—a novel metric designed to quantify the robustness of study conclusions. The FI represents the smallest number of censored observations that, when reclassified as uncensored events, causes the posterior probability of the median survival time exceeding a specified threshold to fall below a predefined confidence level. While drug effectiveness is typically assessed by determining whether the posterior probability exceeds a specified confidence level, the FI offers a complementary measure, indicating how robust these conclusions are to potential shifts in the data. Using a Bayesian approach, we develop a practical framework for computing the FI based on the exponential survival model. To facilitate the application of our method, we developed an R package fi, which provides a tool to compute the Fragility Index. Through real world case studies involving time to event data from single arms clinical trials, we demonstrate the utility of this index. Our findings highlight how the FI can be a valuable tool for assessing the robustness of survival analyses in single-arm studies, aiding researchers and clinicians in making more informed decisions.

Keywords: Bayesian analysis, Exponential survival model, Fragility Index, Single-arm clinical trials, Time-to-event data

1. Introduction

The Fragility Index (FI) is a widely used metric in randomized controlled trials (RCTs) to assess the robustness of statistically significant findings, particularly in studies with small sample sizes or limited outcome events. Traditional significance testing, which relies on p-values, can sometimes create a false sense of confidence, as minor changes in the data may shift results from significant to non-significant, casting doubt on the reliability of conclusions (Editorial, 2019). For binary outcomes, the FI, introduced by Walsh et al. (2014), quantifies the vulnerability of a study’s results by determining the minimum number of events (e.g., successes or failures) that would need to change to reverse statistical significance. This is calculated by systematically altering the outcomes—changing non-events to events or vice versa—until the p-value exceeds the significance threshold, typically $p>0.05$. For example, in a trial testing a new treatment, if statistical significance is observed, a low FI, meaning that just one or two more events would change the result, indicates that the findings are fragile and may lack robustness. Although there is no universal threshold for robustness, a higher FI generally indicates more reliable results (Andrade, 2020). To adjust for sample size, the Fragility Quotient (FQ), calculated as FI/sample size, has been proposed to provide a normalized measure of fragility (Heston, 2023). Understanding fragility allows researchers and clinicians to interpret trial results with more caution, particularly when clinical decisions are based on fragile evidence (Garcia et al., 2023).

Several studies have highlighted the practical implications of the FI. Meyer et al. (2014) demonstrated that in a pulmonary embolism trial, altering the outcomes of only three patients rendered the results non-significant, exposing the limited robustness of the findings. Similarly, Tignanelli and Napolitano (2019) emphasized the value of the FI in identifying fragile findings and improving patient care by fostering a deeper understanding of trial robustness. In critical care research, Ridgeon et al. (2016) revealed that many trials reporting statistically significant effects on mortality had low Fragility Index scores, suggesting that their results relied heavily on a small number of events, raising concerns about their robustness. With the development of tools like the fragility R package (Lin and Chu, 2022), calculating and visualizing the FI for clinical trials with binary outcomes has become more accessible. However, the FI has some limitations, particularly its reliance on binary outcomes, small sample sizes, and the use of Fisher’s exact test, which can affect its accurate calculation, especially in studies involving time-to-event data (Baer et al., 2021; Potter, 2020).

To overcome these limitations in survival analysis, the concept of fragility has been extended to time-to-event data in two arm trials with the introduction of the Survival-Inferred Fragility Index (SIFI) (Bomze et al., 2020). The SIFI measures the robustness of statistically significant findings in clinical trials with survival endpoints by identifying the minimum number of patients with the longest survival times in treatment group who must be reclassified to the control group to reverse statistical significance, typically assessed using a two-sided log-rank test ($P>0.05$). This provides a more precise measure of how sensitive survival outcomes are to changes in the data, offering deeper insights into trial stability. Liu et al. (2024) assessed the robustness of 332 phase III oncology trials using the SIFI and found that trials involving targeted therapies, progression-free survival endpoints, and positive outcomes tend to be the most robust. Similarly, Olsen et al. (2024) analyzed 113 pediatric oncology phase 3 trials and found that pediatric trials were similarly or less fragile compared to adult oncology trials.

Historically, the FI has been used primarily in two-arm or multi-arm trials. In this paper, we extend the concept of the Fragility Index to time-to-event data in single-arm trials and explore its practical applications. Section 2 provides a detailed definition of the FI for time-to-event data in single arm trials, explaining how it measures the robustness of trial outcomes. Section 3 presents case studies using real-world survival datasets, illustrating the usefulness of the FI in practice. Finally, Section 4 discusses the broader implications of using the FI for survival analysis, emphasizing its value as a supplementary metric for interpreting trial results and aiding clinical decision-making.

2. Methodology

In clinical trials, particularly in fields like oncology and chronic diseases, outcomes are frequently associated with the time until a significant event occurs, such as disease progression or death. This leads to the analysis of time-to-event data, also known as survival data. Survival analysis not only captures whether the event occurs but also when it happens, offering richer insights into the treatment’s efficacy and impact. These time-based outcomes are critical in understanding treatment effects over the course of a study and are a common focus in clinical research involving chronic or progressive conditions. A common challenge in survival analysis is the presence of censoring—a scenario where the exact time of the event is not fully observed. Right-censoring, the most common type, occurs when a patient either has not experienced the event by the end of the study or is lost to follow-up before the event occurs. In these cases, the patient’s data is incomplete, as we only know that the event did not occur up to a certain point in time, but the exact time (if or when it will occur) remains unknown. Analyzing such data poses complexities, particularly in accounting for censored observations.

To model time-to-event data in survival analysis, one of the widely used distributions is the exponential distribution. Its popularity can be traced back to the seminal works of Epstein (Epstein and Sobel, 1953, 1954; Epstein, 1954), who demonstrated its practicality and versatility in modeling life data and survival times. The exponential distribution has the memoryless property, meaning the hazard rate remains constant over time. This implies that the likelihood of the event occurring at any moment is independent of how much time has already passed, making it a suitable choice in situations where the event risk is expected to stay stable throughout the study.

The probability density function (pdf) of the exponential distribution is:

$$f(t)=\lambda e^{-\lambda t},\quad t\geq 0$$

where $\lambda$ is the rate parameter. The expected time to the event is the inverse of the rate parameter, $1/\lambda$, providing an estimate of the average time until the event occurs. The survival function, which gives the probability that a patient survives beyond a given time $t$, is:

$$S(t)=e^{-\lambda t}$$

and the hazard function, which describes the instantaneous rate of the event occurrence given that the patient has survived up to time $t$, is constant and given by:

$$h(t)=\lambda$$

2.1. Likelihood Function with Censoring

In the presence of censoring, the likelihood function must account for both observed events and censored data. Suppose there be $n$ independent observations and for $i\leq i\leq n$, let $(T_{i},\delta_{i})$ represent the observed time and censoring indicator for patient $i$, where $\delta_{i}=1$ if the event is observed and $\delta_{i}=0$ if the observation is censored. The likelihood function for $n$ independent observations is given by:

(2.1) $$L(\lambda)=\prod_{i=1}^{n}\left[\lambda e^{-\lambda T_{i}}\right]^{\delta_{i}}\left[e^{-\lambda T_{i}}\right]^{1-\delta_{i}}$$

2.2. Choice of Prior

In Bayesian analysis, the choice of prior distribution reflects initial beliefs about the parameter before observing the data. This could be based on prior studies, expert knowledge, or other relevant information available before the trial. A natural choice for the rate parameter $\lambda$ is the Gamma prior, as it is conjugate to the exponential likelihood, simplifying posterior inference. The Gamma distribution is defined by the shape parameter $\alpha$ and the rate parameter $\beta$, with the probability density function:

$$\pi(\lambda)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}\lambda^{\alpha-1}e^{-\beta\lambda},\quad\lambda>0$$

Using Bayes’ theorem, the posterior distribution of $\lambda$ is derived by combining the prior and the likelihood from the observed data. The posterior distribution remains a Gamma distribution with updated parameters:

$$\alpha^{\prime}=\alpha+\sum_{i=1}^{n}\delta_{i},\quad\beta^{\prime}=\beta+\sum_{i=1}^{n}T_{i}$$

Theorem 1 (Posterior Distribution of Rate Parameter $\lambda$).

Given $n$ independent observations $(T_{i},\delta_{i})$ for $i=1,\dots,n$, where $T_{i}$ represents the observed time and $\delta_{i}$ is the censoring indicator, and assuming a Gamma prior $\text{Gamma}(\alpha,\beta)$ for $\lambda$, the posterior distribution of $\lambda$ is:

$$\lambda\,|\,\left(T_{i},\delta_{i}\right)_{i=1}^{n}\sim\text{Gamma}\left(\alpha+\sum_{i=1}^{n}\delta_{i},\,\beta+\sum_{i=1}^{n}T_{i}\right)$$

Proof.

See Appendix A ∎

This posterior distribution provides an updated estimate of $\lambda$, incorporating both the prior belief and the observed survival data.

2.3. Fragility Index

In time-to-event data, drug effectiveness is often assessed by determining whether the posterior probability that the median survival time exceeds a specified threshold is above a certain confidence level. The Fragility Index (FI) provides an additional measure of robustness by quantifying how many events must change to bring this probability below that confidence level. Specifically, for survival data from single-arm trials, where the posterior probability that the median survival time exceeds a threshold $t_{0}$ initially surpasses a predefined confidence level $p_{0}$, the Fragility Index (FI) is defined as the smallest number $k$ of censored observations with the shortest censoring times that, when reclassified as uncensored events, reduce the posterior probability below the specified confidence level $p_{0}$.

This index provides a precise measure of the trial’s robustness, indicating how sensitive the results are to changes in the data. In essence, it indicates the minimum number of censored observations that must be reclassified to uncensored events to reduce confidence in the treatment effect. Mathematically, the median survival time $t_{\text{med}}$ for an exponential distribution is given by:

$$t_{\text{med}}=\frac{\ln 2}{\lambda}$$

Theorem 2 (Median Survival Time for Exponential Distribution).

For a random variable $T$ following an exponential distribution with rate parameter $\lambda$, the median survival time is:

$$t_{\text{med}}=\frac{\ln 2}{\lambda}$$

Proof.

See Appendix A ∎

The posterior probability that the median survival time exceeds a threshold $t_{0}$ is computed by integrating over the posterior distribution of $\lambda$.

Theorem 3 (Posterior Probability of Median Survival Time).

Given the posterior distribution of the rate parameter $\lambda$ for an exponential survival model, with data $\left(T_{i},\delta_{i}\right)_{i=1}^{n}$, as $\text{Gamma}\left(\alpha+\sum_{i=1}^{n}\delta_{i},\,\beta+\sum_{i=1}^{n}T_{i}\right)$, the posterior probability that the median survival time $t_{\text{med}}$ exceeds a threshold $t_{0}$ is:

$$P\left(t_{\text{med}}>t_{0}\,|\,\left(T_{i},\delta_{i}\right)_{i=1}^{n}\right)=P\left(\lambda<\frac{\ln 2}{t_{0}}\,|\,\left(T_{i},\delta_{i}\right)_{i=1}^{n}\right).$$

This probability is computed by integrating the posterior Gamma distribution:

$$P\left(t_{\text{med}}>t_{0}\right)=\int_{0}^{\frac{\ln 2}{t_{0}}}\frac{\beta^{\prime\alpha^{\prime}}}{\Gamma(\alpha^{\prime})}\lambda^{\alpha^{\prime}-1}e^{-\beta^{\prime}\lambda}\,d\lambda,$$

where $\alpha^{\prime}=\alpha+\sum_{i=1}^{n}\delta_{i}$ is the updated shape parameter, and $\beta^{\prime}=\beta+\sum_{i=1}^{n}T_{i}$ is the updated rate parameter.

The proof of the Theorem 3 follows directly from Theorem 1 and 2. The Fragility Index is determined by sequentially reclassifying censored observations with the smallest censoring times to uncensored events and recalculating the posterior probability until it falls below the specified confidence level. A higher Fragility Index indicates greater robustness of the study results, implying that the treatment effect remains consistent even when multiple censoring statuses are changed. Conversely, a lower Fragility Index suggests that the results are sensitive to small changes in the data, potentially undermining confidence in the conclusions. It is important to note that there is no universally defined threshold for the FI; it serves as a relative indicator of robustness rather than an absolute measure of significance. Therefore, the FI should be used alongside other statistical measures to provide a more comprehensive evaluation of a study’s findings.

3. Numerical Study

In this section, we apply the Fragility Index (FI) methodology to real-world clinical trial data to evaluate the robustness of study conclusions. By utilizing our R package fi, we efficiently calculate the FI, demonstrating its effectiveness and ease of use in survival analysis.

3.1. Case Study 1: Lung Cancer

We begin with the North Central Cancer Treatment Group data, available in the lung dataset from the survival package in R. This dataset comprises observations from patients with advanced lung cancer, providing detailed information on survival times and various clinical variables.

For this analysis, we randomly selected 30 patients from the dataset, of whom 22 had experienced the event (death), while the remaining 8 were censored. Using the observed survival times and censoring data, we calculated the posterior probability that the median survival time exceeded 7 months. A Gamma prior with shape parameter $\alpha=0.5$ and rate parameter $\beta=0.5$ was chosen for the survival rate parameter $\lambda$, as it is weakly informative and conjugate to the exponential likelihood, making it an appropriate choice for survival analysis. The resulting posterior probability was 0.935, indicating a high likelihood that the median survival time exceeded 7 months.

The Kaplan-Meier curve for this dataset is shown in Figure 1.

Figure 1. Kaplan-Meier curve for the lung cancer dataset

The Fragility Index was determined by sequentially reclassifying censored observations with the shortest censoring times as events and recalculating the posterior probability until it fell below the predefined confidence threshold of 0.7, a standard choice balancing statistical confidence and flexibility. For this dataset, the Fragility Index was found to be 5. This means that reclassifying five censored patients as having experienced the event (death) would reduce the posterior probability of the median survival time exceeding 7 months to below 0.7. Given a sample size of 30, an FI of 5 suggests that the study’s conclusions are robust. This level of FI suggests that the findings of the study are stable and reliable, even with moderate changes in the data.

3.2. Case Study 2: Pembrolizumab in hepatocellular carcinoma (HCC)

For the third case study, we analyzed progression free survival time for Pembrolizumab, an immune checkpoint inhibitor widely used in the treatment of various cancers, including advanced hepatocellular carcinoma (HCC) and other malignancies. Pembrolizumab functions by blocking the programmed cell death protein 1 (PD-1) receptor, thereby enhancing the immune system’s ability to detect and destroy cancer cells. The study (Feun et al., 2019) focused on its application in advanced hepatocellular carcinoma (HCC), a disease that has shown modest response rates to checkpoint inhibitors. The Individual Patient Data (IPD) from the treatment arm was extracted from the Kaplan-Meier curve presented in Feun et al. (2019), using the MD Anderson Cancer Center’s IPD extraction tool.

The dataset, constructed using the above extraction tool, comprises 28 patients. Among them, 20 experienced disease progression, while the remaining 8 were censored. Following the same methodology as in the previous analyses, we calculated the posterior probability that the median progression-free survival time exceeded 3.5 months. A Gamma prior with shape parameter $\alpha=0.5$ and rate parameter $\beta=0.5$ was applied, consistent with our prior studies, and the cutoff for the posterior probability was set at 0.7. The resulting posterior probability was 0.958, indicating a high likelihood that the median progression-free survival time exceeded 3.5 months.

The Kaplan-Meier curve for this dataset is shown in Figure 2.

Figure 2. Kaplan-Meier curve for the breast cancer dataset treated with Pembrolizumab

The Fragility Index for this dataset was determined to be 6, meaning that reclassifying six censored patients as having experienced the event (disease progression) would reduce the posterior probability of the median progression-free survival time exceeding 3.5 months to below 0.7. Given the sample size of 28, an FI of 6 indicates that the study’s conclusions are robust. This level of FI suggests that the findings regarding Pembrolizumab’s efficacy are stable and reliable, even with moderate changes in the data.

3.3. Case Study 3: Palbociclib in Breast Cancer

For this case study, we analyzed progression free survival time from a single-arm phase II study investigating Palbociclib, a CDK4/6 inhibitor commonly used for the treatment of hormone receptor (HR)-positive, HER2-negative advanced or metastatic breast cancer (MBC). The study protocols are outlined in detail in Krishnamurthy et al. (2022) and, as with Case Study 2, Individual Patient Data (IPD) for the treatment arm was obtained from the Kaplan-Meier curve in Krishnamurthy et al. (2022).

The dataset, constructed using the MD Anderson Cancer Center’s IPD extraction tool, includes 51 patients. Among them, 31 experienced disease progression, while the remaining patients were censored.

Using the same methodology as in the previous analysis, we calculated the posterior probability that the median survival time exceeded 15 months. A Gamma prior with shape parameter $\alpha=0.5$ and rate parameter $\beta=0.5$ was applied, with the cutoff for the posterior probability set at 0.7. The resulting posterior probability was 0.948, indicating a high likelihood that the median survival time exceeded 15 months.

The Kaplan-Meier curve for this dataset is displayed in Figure 3.

Figure 3. Kaplan-Meier curve for the breast cancer dataset treated with Palbociclib

The Fragility Index (FI) for this dataset was calculated to be 6, indicating that if six censored patients were reclassified as having experienced the event (disease progression), the posterior probability of the median survival time exceeding 15 months would drop below 0.7. With a sample size of 51, an FI of 6 suggests that the study’s conclusions are moderately robust and not entirely resistant to changes in the data. This implies that the treatment effect shows some sensitivity to data alterations.

4. Concluding Remarks

In this paper, we defined the Fragility Index (FI) for time-to-event endpoints in single-arm clinical trials, developed a Bayesian methodology using the exponential distribution, and demonstrated its application through three real-world case studies. We also developed an R package fi, to facilitate the calculation of the FI, making it accessible for researchers to apply this in similar studies. The FI quantifies the robustness of study conclusions by identifying the minimum number of censored observations that, when reclassified as events, reduce the posterior probability of the median survival time exceeding a specified threshold below a confidence level. Our case studies yielded FI values of 5 and 6, indicating moderate robustness; while the findings are relatively stable, they remain somewhat sensitive to data changes. This underscores the FI’s value as a complementary tool to traditional statistical methods, enhancing the interpretability and reliability of clinical trial outcomes, especially in the absence of control groups. However, the FI lacks a universal threshold and is influenced by the choice of prior distribution, necessitating careful interpretation and integration with other clinical impact measures. In conclusion, the Fragility Index represents a significant advancement in assessing the robustness of single-arm clinical trials with time-to-event data, supporting more informed and reliable clinical decision-making.

APPENDIX

The calculation of the Fragility Index and related numerical studies are implemented in R and made accessible through the GitHub repository (https://github.com/arnabkrmaity/fi).

A. Proof of the Theorems

Proof of Theorem 1.

The likelihood function for the censored data is:

$$L(\lambda)=\prod_{i=1}^{n}\left(\lambda e^{-\lambda T_{i}}\right)^{\delta_{i}}\left(e^{-\lambda T_{i}}\right)^{1-\delta_{i}}=\lambda^{\sum_{i=1}^{n}\delta_{i}}e^{-\lambda\sum_{i=1}^{n}T_{i}}$$

By applying Bayes’ Theorem, the posterior distribution is proportional to the product of the likelihood and the prior:

$$p(\lambda\,|\,T_{1},\dots,T_{n},\delta_{1},\dots,\delta_{n})\propto L(\lambda)\pi(\lambda)$$

Substituting the likelihood and prior expressions:

$$p(\lambda\,|\,T_{1},\dots,T_{n},\delta_{1},\dots,\delta_{n})\propto\lambda^{\alpha+\sum_{i=1}^{n}\delta_{i}-1}e^{-\lambda\left(\beta+\sum_{i=1}^{n}T_{i}\right)}$$

This is a Gamma distribution with updated parameters $\alpha^{\prime}=\alpha+\sum_{i=1}^{n}\delta_{i}$ and $\beta^{\prime}=\beta+\sum_{i=1}^{n}T_{i}$, thus:

$$\lambda\,|\,\left(T_{i},\delta_{i}\right)_{i=1}^{n}\sim\text{Gamma}\left(\alpha+\sum_{i=1}^{n}\delta_{i},\,\beta+\sum_{i=1}^{n}T_{i}\right)$$

∎

Proof of Theorem 2.

The survival function $S(t)$ for the exponential distribution is:

$$S(t)=P(T>t)=e^{-\lambda t}$$

Setting $S(t_{\text{med}})=0.5$, we solve:

$$e^{-\lambda t_{\text{med}}}=0.5\quad\Rightarrow\quad t_{\text{med}}=\frac{\ln 0.5}{-\lambda}=\frac{\ln 2}{\lambda}$$

∎