Pneumonia Severity Index And CURB-65 Scores Are Poor Predictors For Long-Term Mortality In Hospitalized Patients With Community-Acquired Pneumonia

Introduction

Historically, data from Centers for Disease Control and Prevention indicate that pneumonia and influenza ranked as the eighth cause of death in the United States, being the primary cause of mortality due to infection.¹ With the COVID-19 pandemic, SARS-CoV-2 has overtaken influenza as a cause of mortality, however the primary disease state resulting in death from SARS-CoV-2 is community-acquired pneumonia (CAP). Thus, the impact of pneumonia-related death has not decreased during the pandemic.

Mortality statistics are based on information collected from death certificates in the United States, and thus deaths due to CAP are only captured if the death occurs during hospitalization. Patients hospitalized with CAP who are discharged alive are at increased risk for mortality months or even years after hospital discharge.² This suggests that the actual impact of pneumonia on mortality in the general population is most likely undestimated.

Several studies have documented the risk for long-term mortality in hospitalized patients with CAP.^3–9 In some of these studies, investigators evaluated patients hospitalized with CAP in comparison to control groups of patients hospitalized without CAP.^4,9 The increased mortality in patients with CAP when compared to controls indicate an association of CAP with long-term mortlity.

Due to the documented decreased survival in patients discharged alive after an episode of CAP, there is increased interest in investigating how to properly identifiy the population of patients at risk for decreased survival. The identification of patients hospitalized with CAP who are at risk for short-term mortality is currently performed using CAP severity scores. The two primary severity scores use to predict short term mortality are the Pneumonia Severity Index (PSI)¹⁰ and CURB-65.¹¹ These two scores that can identify patients with low, intermediate, and high risk for short term mortality have helped physicians define the need for hospitalization of patients with CAP. These scores have also supplemented clinical judgment in the need for hospitalization to a medical ward or intensive care unit.

Although these CAP severity scores are well-defined as tools to predict short-term mortality, the role of these scores in predicting long-term mortality is not well defined. Considering the benefits in patient care offered by the PSI and CURB-65 for the acute management of patients with CAP, and the paucity of data in predicting long-term survival, we designed an study with the goal to evaluate the role of PSI and CURB-65 as predictors of long term mortality in patients discharged after an episode of CAP. In this study we calculated the area under the curve (AUC) to evaluate the overall performance of the PSI and CURB-65 and to compare this two different severity scores in their ability to predict long-term mortality in hospitalized patients with CAP.

Methods

Study Design

This was a secondary analysis of the University of Louisville Pneumonia Study database. The ULP study took place from June 1, 2014 – May 31, 2016 at all nine adult acute-care hospitals in Louisville, KY and enrolled every consecutive patient hospitalized with CAP into the study. Patients were followed for one year, and death certificates were used to verify vital status at one year if patients were lost to follow-up. Primary results from the ULP study were previously published.¹²

Study population

Patients met inclusion into the parent study if they were hospitalized with CAP as defined by:

1) presence of a new pulmonary infiltrate on chest radiograph and/or chest computed tomography scan at the time of hospitalization, defined by a board-certified radiologist’s reading; 2) at least 1 of the following: (a) new cough or increased cough or sputum production, (b) fever >37.8°C (100.0°F) or hypothermia <35.6°C (96.0°F), (c) changes in leukocytecount (leukocytosis: >11 000 cells/μL; left shift: >10% band forms/mL; or leukopenia: <4000 cells/μL); and (3) no alternative diagnosis at the time of hospital discharge that justified the presence of criteria 1 and 2. To be included in this secondary analysis, patients must have been discharged alive from the hospital.

Severity of disease indices

The pneumonia severity index (PSI) risk classes and the CURB-65^10,11 were calculated at the time of hospital admission for the full study population.

Mortality groups

The study population was characterized in two groups. Group 1: patients with long-term mortality post-discharge, comprised of all patients discharged alive but dead at 1 year. Group 2: Patients alive post-discharge, comprised of all patients discharged alive and alive at 1 year.

Descriptive statistics

Descriptive statistics were produced for mortality groups. Categorical data was summarized as frequency and percent with differences between groups tested using Chi-squared tests. Continuous data was summarized as median and interquartile range with differences tested using Mann-Whitney U-tests. Histograms were used to visualize the distribution of PSI Risk Classes and CURB-65 scores among mortality groups.

ROC Analysis and Diagnostic Statistics

Receiver operating characteristic (ROC) analysis was performed to determine the diagnostic performance of the PSI and CURB-65 in predicting long-term mortality. The following statistics were produced for each ROC analysis: 1) Threshold; 2) Sensitivity; 3) Specificity; 4) Area under the curve (AUC); 5) Positive predictive value (PPV); and 6) Negative predictive values (NPV).

95% confidence intervals were produced using the bootstrap method with 1000 replicates for each diagnostic statistic. ROC curves were produced to plot the diagnostic sensitivity against 1-specificity. The integrated discrimination improvement (IDI) was calculated between the PSI and CURB-65 to describe the net reclassification between the scores. P-values <0.05 were considered statistically significant.

Results

A total of 6,870 patients hospitalized with CAP who were alive at time of hospital discharged and in whom data on 1 year mortality was ascertained, constituted the study population. From this 6,870 patients, 1,744 (25%) patients died within one-year post discharge, and 5,126 (75%) patients were alive one-year post-discharge. Baseline characteristics between groups are summarized in Table 1.

The distribution of mortality by PSI Risk Class and CURB-65 score among groups are depicted in Figures 1 and 2. ROC curves depicting the diagnostic accuracy of the PSI and CURB-65 in predicting mortality one year post discharge are depicted in Figure 3. Diagnostic statistics for each ROC analysis are depicted in Table 2. When compared to the CURB-65, a patient’s PSI risk classification was more sensitive at ruling out mortality at one-year. Additionally, the AUC and NPV for PSI outperformed the CURB-65. Prediction of long-term mortality improved by 5.7% when PSI risk classification was compared to the CURB-65 (IDI = 0.057, 95% CI: 0.051-0.063).

Figure 1.Distribution of PSI Risk Classes among mortality groups. The proportion and frequency represented above each bar corresponds to the risk classification for patients among their respective mortality group.

Figure 2.Distribution of CURB-65 scores among mortality groups. The proportion and frequency represented above each bar corresponds to the CURB-65 score for patients among their respective mortality group.

Figure 3.ROC curves depicting the diagnostic accuracy of the PSI (solid line) and CURB-65 (dotted line) in predicting long-term mortality.

Discussion

In this study we used the AUC to compare the ability of PSI and CURB-65 to distinguish patients that would be alive or dead one year after discharge alive from a hospitalization due to CAP. The study indicates that the PSI may better predict long-term mortality compared to the CURB-65, with areas under the curve of 0.72 for PSI and 0.66 for CURB-65. When evaluating the AUC results, finding an AUC of 0.5 would suggests that the score have no no discriminative ability to define long-term mortality (equivalent to random guessing), while finding an AUC of 1.0 would indicate a perfect discrimination of patients alive or dead, indicating the score can perfectly distinguish between all patients alive and dead after one year of being discharged from the hospital. Even though there is no standard for interpreting for AUC values ranging from >0.5 and <1.0, as a general guideline, a test with an AUC ranging from >0.5 to 0.7 is consider to have poor accuracy, AUC of 0.7 to 0.9, moderate accuracy, and AUC of 0.9 to <1.0, good accuracy. Our data indicates that CURB-65 has a poor accuracy (AUC 0.66) and PSI has moderate accuracy on the low end of the continuum (AUC 0.72). The PSI has diminished predictive ability due to it’s low specificity, meaning it is better at ruling out long-term mortality than identifying which patients may be at risk.

Long-term mortality in CAP is more common in patients with comorbidities, and comorbidity stacking is associated with long-term mortality.¹³ The PSI incorporates many comorbidities into its score, which may help explain the improved performance relative to the CURB-65.

The performance of the PSI and CURB-65 for prediction of short-term mortality was evaluated in multiple studies. In a systematic review and meta-analysis of the articles published from 1980 until 2009, the investigators identified 40 sudies fulfilling selection criteria that investigated PSI and CURB-65 as tools for prediction of short-term mortality.¹⁴ The PSI had an AUC of 0.81, compared with CURB-65, 0.80, with a difference that was not statistically significant. These data showing an AUC of approximately 0.8 for PSI and CURB-65 indicates that these test have a moderate ability to discriminate between patients that will be dead or alive during the acute episode of CAP. But even with a moderate accuracy, the PSI and CURB-65 are considered useful tests for supporting clinical decision at the time that a physician needs to decide site of care for a patient with CAP. Extending to long term mortality, in our study we found that the accuracy of PSI and CURB-65 to predict one-year mortality decreased to an AUC of approximately 0.7. This value is in the limit of defining a test as poor moderate accuracy to predict long-term mortality. This translates in a higher rate of misclassifications for survival, including false positives (identifying patients at risk for death when they will be alive) and false negatives (identifying patients at low risk for death, when they will died).

The PSI is well-regarded as a predictor for short term mortality. Previous studies have also documented that patients with higher PSI scores at time of hospital admission have significantly increased risk for long-term mortality compared to patients with lower PSI scores.^3,15 The authors suggest that PSI can be used to predict long-term mortality. Variability of the role of PSI in predicting long-term mortality from these studies compared to our current study may be due in part to different study definition of long-term mortality. Most studies use mortality from time of hospitalization until 1 year. We decided to use mortality from time of discharge alive until one year to better describe this population for two reasons. First, is that by including mortality from time of hospitalization to one year, you are including both short-term and long-term mortality in the same cohort. Second, any future studies assessing long-term mortality in CAP must consent patients that have already survived hospitalization.

One important limitation of this study is that the ULPS database does not contain information required to calculate the Charlson Comorbidity Index or any other severity scores for use in hospitalized patients with CAP and sepsis. Another limitation is that we have data only up to one year, although the impact of CAP on mortality extends beyond one year.

A working group of pneumonia investigators asambled by the National Institutes of Heath to prioritize areas of pneumonia research, reported “the need for methods that discriminate those patients likely to decline over time, and in which specific ways, compared with others with a more favorable course.”¹⁶ At the present time we see two approaches moving forward in an attempt to generate optimal tools to discriminate patients likely to decline over time or suffer long-term sequela of CAP. One approach is to perform research with the goal to improve the PSI and other severity scores by eliminating parameters identified as poor predictors and adding some clinical or laboratory markers that better predict long-term sequelae. Another approach may be to develop new prediction models using artifitial intelligence (AI). With this approach, the severity scores will be one aspect of a large comprehensive dataset including patients demographics, clinical history, laboratory results, imaging findings, microbiological findings, treatments, and clinical outomes during hospitalization. Development of an optimal model to predict post-acute sequela of CAP and long-term mortality will identify special populations to investigate the pathogenesis leading to poor outcomes and potential interventions to improve long-term clinical outcomes.

In conclusion, this study indicates that the PSI is a better tool to predict long-term mortality when compared to CURB-65, but the predictive ability of PSI is limited. Future research is necessary to identify research tools to predict long-term mortality in hospitalized patients with CAP.