Received date: August 01, 2016; Accepted date: August 26, 2016; Published date: August 31, 2016
Citation: Zuick S, Asch DA, Graustein A, et al. Can a Computerized Sepsis Screening and Alert System Accurately Diagnose Sepsis in Hospitalized Floor Patients and Potentially Provide Opportunities for Early Intervention? A Pilot Study. J Intensive& Crit Care 2016, 2:3.
Title: Can a Computerized Sepsis Screening and Alert System Accurately Diagnose Sepsis in Hospitalized Floor Patients and Potentially Provide Opportunities for Early Intervention? A Pilot Study.
Background: Sepsis Syndromes are major causes of morbidity and mortality for hospitalized patients. Several evidence-based interventions have been shown to improve outcomes but they must be instituted early to achieve better outcomes. However, early recognition of sepsis syndromes in hospitalized patients is challenging. Regrettably, SIRS is overly sensitive and not specific in the patient populations at greatest risk for developing sepsis. In addition, many hospitalized patients have baseline end-organ dysfunction, wherein subtle trends in laboratory abnormalities may escape detection. Electronic detection and alert systems offer a more focused and efficient methodology. This strategy has been deployed successfully in other aspects of critical care, e.g. in the early recognition of acute respiratory distress syndrome. We developed an electronic recognition and alert system to identify floor patients with sepsis syndromes. The objective of this study was to test the feasibility, accuracy, and potential value of a computerized sepsis screening and alert system in a large university hospital.
Methods and findings: The sepsis alert used an abnormal white blood cell count coupled with a blood culture order to define sepsis. Cases were categorized as severe sepsis by meeting specified changes in laboratory tests for organ dysfunction in accordance with consensus conference criteria. Using a retrospective cohort study design, we evaluated 97 consecutive, non-intensive care unit patients who triggered a sepsis alert at a large, urban, academic medical center. The charts of the patients were reviewed and abstracted manually to determine whether sepsis was present. For confirmed sepsis cases, we determined adherence with sepsis care bundle measures, including the recognition of sepsis (using physician documentation as a proxy measure), measurement of serum lactate, administration of antibiotics, and intravenous fluid resuscitation for severe sepsis cases. Within the 97 patient cohort, 72 were confirmed to have sepsis or severe sepsis (positive predictive value of 74%). Sepsis or severe sepsis was not documented in 79%, serum lactate was not measured in 57%, antibiotics were not administered in 14% of patients with severe sepsis, and fluid boluses were not administered in 17% of patients with severe sepsis who had lactic acidosis, hypotension, and/or acute kidney injury. In patients with sepsis or severe sepsis, adherence to the complete sepsis bundle did not occur in 65%. Opportunities to improve sepsis care were more common when sepsis was not documented.
Conclusion: A computerized sepsis screening and alert system designed to identify sepsis in hospitalized medical ward patients without the use of vital signs was feasible to implement and predictive for sepsis. This suggests that implementation of this system may improve the quality of sepsis care in hospitalized ward patients.
Sepsis; Documentation; Clinical decision support systems; Patient monitoring; Heathcare quality indicators; Outcome and process assessment
Sepsis Syndromes (sepsis, severe sepsis, cryptic septic shock, and septic shock) are major causes of morbidity and mortality for hospitalized patients. Approximately, 750,000 cases of severe sepsis occur each year with a mortality rate estimated at 30 to 50% [1,2]. Multiple evidence-based interventions have been developed to help improve sepsis outcomes. To facilitate their implementation, evidence-based practice guidelines from the Severe Sepsis Campaign recommend several interventions in the first 6 h, the sepsis resuscitation bundle, and other interventions within 24 h, the sepsis management bundle . The sepsis resuscitation bundle includes serum lactate measurement to detect shock and risk-stratify patients [4-7], drawing blood cultures followed by administering appropriate antibiotics [5,8-10] and administering early and adequate intravenous fluids for signs of end-organ hypoperfusion [3,4,6,11-13]. These interventions are time-sensitive and must be instituted early to achieve better outcomes [3,9,11,12].
Early recognition of sepsis syndromes in hospitalized patients is challenging. The diagnosis relies on the detection of the systemic inflammatory response syndrome (SIRS) in patients with a suspected or proven infection . In addition, severe sepsis requires recognition of new organ dysfunction. Regrettably, SIRS is overly sensitive, and not specific in the patient populations at greatest risk for developing sepsis. In addition, many hospitalized patients have baseline end-organ dysfunction, wherein subtle trends in laboratory abnormalities may escape detection. Last, detection may be compromised in current clinical care settings with increasing reliance on shift work [15,16]. Routine screening of all hospitalized patients for sepsis represents one strategy for trying to capture sepsis cases early. However, this is time consuming and requires extra effort to screen all patients to benefit only a few. In contrast, electronic detection and alert systems offer a more focused and efficient methodology. This strategy has been deployed successfully in other aspects of critical care, e.g. in the early recognition of acute lung injury (ALI) .
Given the concerns about the difficulty in recognizing sepsis patients and the time-sensitive nature of early sepsis care, we developed an electronic recognition and alert system to identify floor patients with sepsis syndromes. The goal of this study was to evaluate the positive predictive value of the new automated system in identifying sepsis cases. We also sought to determine the potential efficacy of the system by examining clinician compliance, when blinded to the alerts, with the evidence-based bundle for early sepsis management at the time the system identified the patient.
This study was performed at the Hospital of the University of Pennsylvania. This study protocol was reviewed and approved by the Institutional Review Board of the University of Pennsylvania with a waiver for the requirement of written informed consent from the participating subjects or their legally authorized representatives.
Sepsis detection and alert algorithm
An automated computer algorithm was designed to detect hospitalized patients with sepsis and provide additional information about associated end-organ dysfunction to identify when sepsis is severe. Since vital signs were not available electronically at the time of this study, to identify SIRS we included the WBC as one SIRS criterion. A blood culture order was included as a surrogate criterion for both fever (or hypothermia) and the suspicion for infection. In effect, the combination of the blood culture order and the WBC criteria was used to define sepsis in accordance with the established definition; i.e. suspected infection in the presence of two or more SIRS criteria [14,18].
Specifically, the system first scans the hospitals information system (Medview) for the presence of a blood culture order. It then looks for either leukocytosis (WBC>12,000/mm3) or leukopenia (WBC<4) in the 24 h before or after the time of the blood culture order. Then, to classify the severity of sepsis, the system screens for organ dysfunction by looking for abnormalities in lactic acid, partial pressure of arterial oxygen (PO2 ), glucose, bilirubin, INR, creatinine, platelets, and PTT using established values for organ dysfunction criteria [14,18]. Of note, severe sepsis is defined by the onset of new organ dysfunction. Since baseline comorbidities are relatively common in hospitalized patients, the system compares the current values to the most recent ones, looking for a predefined change to indicate that the organ dysfunction is new. The software algorithm then automatically e-mails an alert to the study coordinator for patients who meet these criteria. Each of these data elements is included in the e-mailed sepsisalert. The algorithm is setup to create an alert only on patients who are not in the intensive care unit. It generates an alert at the first evidence of sepsis during a hospitalization, but not more than once. Clinicians were blinded to the alert. As a result, their management decisions were unaffected by whether the sepsis alert had been triggered.
Study design and data collection
This was a retrospective cohort study of 97 consecutive patients identified by the sepsis detection and alert computer algorithm from October 14 through October 27, 2011. For each patient for whom an alert was generated, reviewers manually evaluated each patient’s chart and recorded the outcome and exposure variable along with demographic information. Two independent data collectors determined if a sepsis syndrome was present for each patient. Study data were collected and managed using REDCap electronic data capture tools hosted at the University of Pennsylvania .
The primary objective of the study was to determine the positive predictive value of the computer generated sepsis alert. The secondary objective was to evaluate the potential value of the alert by first assessing, at the time of the alert, clinician adherence to the early sepsis treatment bundle that included drawing of a lactic acid, early administration of new broadspectrum antibiotics, and aggressive (≥ 500 cc IVF bolus) fluid administration when end-organ hypoperfusion was present. Second, we assessed for the occurrence of a serious adverse outcome that occurred following the alert, to assess whether earlier sepsis recognition could potentially improve patient care.
Data and definitions
Baseline characteristics were recorded from each patient and included race, hospital service, presence of SIRS criteria, and comorbidities. Sepsis was defined as suspected or proven infection in the presence of two or more SIRS criteria. Suspected infection was defined by the notation in the patient’s chart that either the provider had suspected infection or had ordered an intervention for the management of an infection (e.g. antibiotics) in the absence of an alternative diagnosis. Severe sepsis was defined as sepsis associated with organ dysfunction, hypoperfusion or hypotension [14,18]. Hypoperfusion was primarily defined as a serum lactate of ≥ 2 mmol/L. Definitions of hypotension and end organ dysfunction was based on the 2001 International Sepsis Definition Conference criteria  with the exception of hepatic dysfunction which was defined as bilirubin >2.0 mg/dL and an elevation above the patients baseline . Septic shock was defined as hypotension (systolic blood pressure <90 mmHg) despite adequate fluid resuscitation (≥ 1500 mL) or the use of vasoactive agents . To determine whether sepsis was recognized, we reviewed clinician documentation. Appropriate documentation of sepsis was defined by the presence of a term defining the proper syndrome (sepsis, severe sepsis, cryptic septic shock, or septic shock) documented in a clinician’s progress note at the time of or in the next progress note that followed the sepsis alert. Severe sepsis was also considered documented appropriately if sepsis was documented in the progress notes with the additional documentation of a new organ dysfunction within the same time frame .
In order for the sepsis alert system to have clinical value, it must fire prior to the initiation of evidenced-based sepsis interventions and occur prior to a serious adverse outcome. An intervention was considered performed if it occurred within the previous 24 h before (unless otherwise stated), and up to 6 h after, the time of the sepsis alert [3,21]. The interventions included drawing a serum lactate, administering a new broad-spectrum antibiotic (up to 48 h before, and 6 h after the alert, and administering a fluid bolus of ≥ 500 mL of fluid over a 30 min time period. For all patients with a sepsis syndrome, serum lactic was considered indicated based on the sepsis campaign practice guidelines . For all patients with severe sepsis or septic shock, new broadspectrum antibiotics were considered indicated. For all patients with circulatory dysfunction, as evidenced by lactic acidosis, acute kidney injury, or hypotension, a fluid challenge (>500 cc IVF bolus) was considered indicated.
We assessed for serious adverse outcomes that occurred within the subsequent 48 h after the alert, including any rapid responses, codes, ICU transfers or deaths. In addition, we included in our list of important outcomes, mortality occurring any time following the alert.
Concordance between the two independent reviewers was evaluated using the kappa statistic. Positive predictive value was calculated as the proportion of actual cases of sepsis within those patients for whom an alert was generated. No other values of alert accuracy could be determined due the manner in which the cohort of patients was derived. A complete assessment of accuracy would have required chart review of all hospital discharges to identify all cases of sepsis, which was beyond the scope of this investigation. Chi squared statistic or Fisher’s exact test were used to compare categorical variables. Student’s t test was performed for continuous variables. All tests were two-tailed and a p value of <0.05 was considered statistically significant. Statistical analysis was performed using Stata 11.2 software package.
In ninety-seven patients a sepsis alert was generated by the computer algorithm during the study period. Of these 97 patients, 48 had sepsis only, 24 had severe sepsis (including 1 patient with septic shock) and 25 failed to meet criteria for any sepsis syndrome. The positive predictive value of the computer algorithm for determining the presence of a sepsis syndrome was 74% (95% CI=64% to 83%). Of all the patients with a sepsis syndrome, 21% (25% with sepsis and 13% with severe sepsis) were documented appropriately. Of the 25 patients that did not meet criteria for a sepsis syndrome (false +), 52% had less than two SIRS criteria and in 68% there was evidence of an alternative clinical diagnosis.
The baseline characteristics of the cohort demonstrated a mean age of 53, with 63% of patients being white and 24% African American (Table 1). The majority (76%) of the patients were on the medicine service with 9/20 (45%) of those patients being oncology patients. The sources of infection included pneumonia (19%), urogenital (17%), skin/soft tissue/wound (2%), intraabdominal (9%), catheter/device-related (8%), bacteremia (10%), and unknown (25%). The baseline characteristics of the patients identified with a confirmed sepsis syndrome, including both sepsis and severe sepsis, are shown in Table 1.
The proportion of cases where there were missed clinical opportunities identified at the time of the sepsis alert are shown in Table 2, categorized by whether or not they were documented as sepsis appropriately in the chart. In 56% of cases with a sepsis syndrome a serum lactate was not measured and some aspect of the care bundle was not initiated in 65% of cases. The indicated interventions were performed significantly more likely in sepsis cases that were documented appropriately (67% vs. 28% for lactate; p=0.004, and 78% vs. 28% for the whole bundle, p<0.001). The data presented in Tables 3 and 4 suggest that adverse outcomes occurred more commonly in septic patients that were not documented appropriately.
This study demonstrated that an automated sepsis detection and alert system can identify sepsis in hospitalized, non-ICU patients with a positive predictive value of 74%. We found that this system identified the presence of a sepsis syndrome often before treating clinicians documented it. Since we found a signal that adherence with the sepsis bundle at the time of the alert was low, our data suggests that the alert may have value to improve sepsis care. In support of this hypothesis is the finding that serious adverse outcomes occurring after the alert were more likely in patients where sepsis was not documented.
|Sepsis Not Present (n=25)||Sepsis or Severe Sepsis* (n=72)||p value||Total Cohort (n=97)|
|Age, years [mean (SD)]||56 ||52 ||0.202||53 |
|White||15 (60%)||46 (64%)||0.729||61 (63%)|
|African American||6 (24%)||17 (24%)||0.969||23 (24%)|
|Hispanic||1 (4%)||0 (0%)||0.258||1 (1%)|
|Asian||1 (4%)||3 (4%)||1.000||4 (4%)|
|Other||2 (8%)||5 (7%)||1.000||7 (7%)|
|Medical Service||20 (80%)||54 (75%)||0.613||74 (76%)|
|Oncology Service||9 (36%)||29 (40%)||0.706||38 (39%)|
|Other Services||5 (20%)||18 (25%)||0.613||23 (24%)|
|Temperature Max/Min>38ÃâÃÂ°C or <36ÃâÃÂ°C||3 (12%)||49 (68%)||<0.001||52(54%)|
|Heart Rate Max>90||12 (48%)||68 (94%)||<0.001||81 (84%)|
|Respiratory Rate Max>20||1 (4%)||35 (49%)||<0.001||36 (37%)|
|Hypertension||10 (40%)||30 (42%)||0.884||40 (41%)|
|Cardiovascular Disease||6 (24%)||12 (17%)||0.416||18 (19%)|
|Diabetes||1 (4%)||21 (29%)||0.011||22 (23%)|
|Cancer||11 (44%)||31 (43%)||0.935||42 (43%)|
|ESRD||0 (0%)||6 (8%)||0.334||6 (6%)|
|Cirrhosis||5 (20%)||6 (8%)||0.113||11 (11%)|
|Immunocompromised||12 (48%)||30 (42%)||0.582||42 (43%)|
|Source of Infection|
|Pneumonia||2 (8%)||16 (33%)||0.144||18 (19%)|
|Urogenital||2 (8%)||14 (19%)||0.227||16 (17%)|
|Skin/Soft Tissue/Wound||1 (4%)||1 (1%)||0.451||2 (2%)|
|Intra-abdominal||3 (12%)||6 (8%)||0.691||9 (9%)|
|Catheter/Device Related||0 (0%)||8 (11%)||0.108||8 (8%)|
|Bacteremia||1 (4%)||9 (13%)||0.445||10 (10%)|
|Unknown||3 (12%)||21 (29%)||0.110||24 (25%)|
Table 1 Baseline demographics.
All Patients with Sepsis (including severe sepsis)
|Documented Appropriately (n=18)||Not Documented Appropriately (n=54)||p value||Total Patient (n=72)|
|Lactate not Drawn||5 (28%)||36 (67%)||0.004||41 (57%)|
|Incomplete Early Sepsis Bundle*||5 (28%)||42 (78%)||<0.001||47 (65%)|
*Complete sepsis bundle includes drawing a lactic acid if you have sepsis and drawing a lactic acid, receiving appropriate antibiotics, and administration of IV fluid bolus if lactic acidosis, acute kidney injury or hypotension Patients with Severe Sepsis
|Documented Appropriately (n=7)||Not Documented Appropriately (n=22)||p value||Total Patients (n=29)|
|Antibiotics not Administered or Changed||0 (0%)||4 (18%)||0.546||4 (14%)|
|IV Fluid Bolus (=500 mL) not Administered^||0 (0%)||5 (23%)||0.296||5 (17%)|
^Including only those patients with acute kidney injury, lactic acidosis or hypotension
Table 2: Missed clinical opportunities identified at the time of the sepsis alert stratified by whether sepsis was documented appropriately.
|Adverse Outcome||Documented Appropriately (n=18)||Not Documented Appropriately (n=54)||p value||Total Patients with Sepsis Syndrome (n=72)|
|Transfer to ICU within 48 h of the alert||0 (0%)||5 (9%)||0.322||5 (7%)|
|Rapid Response within 48 h of the alert||0 (0%)||1 (2%)||1.000||1 (1%)|
|Code Call within 48 h of the alert||0 (0%)||1 (2%)||1.000||1 (1%)|
|Death within 48 h of the alert||0 (0%)||2 (4%)||1.000||2 (3%)|
|Hospital Mortality||0 (0%)||5 (9%)||0.332||5 (7%)|
|Any Adverse Outcome||0 (0%)||8 (15%)||0.188||8 (11%)|
Table 3 Adverse outcomes of patients identified by the sepsis alert system, stratified by whether sepsis was documented appropriately
There have been several prior studies that attempted to use electronic surveillance to diagnosis sepsis in a variety of venues. These studies have reported a positive predictive value that ranged from 19.5% to 54%. Nelson et al. used a real-time electronic surveillance algorithm in the emergency room that screened for SIRS and hypotension and found a 54% positive predictive value. The authors noted that their system usually failed to identify the patient before the clinician . Meurer et al. reported on an electronic sepsis screen of geriatric patients in the emergency room that screened for SIRS and sent an automatic page to clinicians. It had a sensitivity of 36% and a specificity of 78%. In this study, they did not report a positive predictive value, however, based on the data presented it would be approximately 48% . The positive predictive value of the screening systems in both of these studies was lower than ours. The screening systems in these studies differed from our algorithm in two major respects. First, both screened SIRS criteria, whereas our system was created to function without the use of vital signs. The most likely explanation for the higher PPV in our study is that the requirement for a blood culture order provides a more specific context than vital signs. Other than sepsis, few conditions are associated with a blood culture order, whereas many conditions cause vital sign abnormalities. In addition, in the study by Nelson et al they found that clinicians often identified sepsis before the alert system. This difference is most likely due to the ED setting where every patient in this study was undergoing rapid initial evaluation, while the ward patients in our study were not.
Computerized sepsis screening algorithms have also been utilized in the intensive care unit setting. Herasevich et al developed a computerized septic shock sniffer for the ICU, which employed a complex algorithm screening for SIRS criteria, microbiology data, and hypoperfusion. The positive predictive value reported was 34%, much lower than our study . Although the specificity would be expected to be higher using microbiological data, the shock screening criteria likely negated any favorable effect on specificity, given the many diagnoses-other than sepsis-that cause hypoperfusion in an ICU population. Furthermore, this ICU sepsis screening system was designed as a research study screening method, not as a clinical tool to improve the care process or patient outcomes.
Kollef et al. was the first to report on the development of an electronic sepsis detection system for floor patients [25,26]. Our algorithm performs well compared to theirs. Their sepsis detection algorithm was designed using a statistical model that used laboratory values taken retrospectively from a cohort of known sepsis patients. When their system was tested in a validation cohort the positive predictive value was between 19.5 and 21.4% depending on which year they used to validate the cohort, which is much lower than ours . Another advantage of our system is that it distinguishes sepsis severity by evaluating trends in laboratory values to identify acute organ dysfunction. Furthermore, our system may be more generalizable to other health care institutions since the algorithm was designed based on standardized sepsis criteria, in contrast to their unique criteria specific to the ward population of a single institution.
|Adverse Outcome||Documented Appropriately (n=7)||Not Documented Appropriately (n=22)||p value||Total (n=29)|
|Transfer to ICU within 48 h of the alert||0 (0%)||5 (23%)||0.296||5 (17%)|
|Rapid Response within 48 h of the alert||0 (0%)||1 (5%)||1.000||1 (3%)|
|Code Call within 48 h of the alert||0 (0%)||1 (5%)||1.000||1 (3%)|
|Death within 48 h of the alert||0 (0%)||2 (9%)||1.000||2 (7%)|
|Hospital Mortality||0 (0%)||5 (23%)||0.296||5 (17%)|
|Any Adverse Outcome||0 (0%)||7 (32%)||0.147||7 (24%)|
Table 4: Adverse outcomes of the severe sepsis patients identified by the sepsis alert system stratified by whether sepsis was documented appropriately.
We found that in the majority (80%) of the sepsis cases detected, sepsis was not documented by providers at the time the alert fired. Clearly, documentation is not a reliable perfect substitute for physician recognition; however, trends in the data-though not statistically significant-suggest that the lack of documentation was associated with non-compliance with the sepsis care bundle. This does provide support for the notion that a sepsis alert system that can improve patient recognition will improve sepsis management. By that standard, a sepsis alert may improve patient management.
We have also shown that the sepsis detection system provides alerts before early sepsis management occurs. Sepsis management is time-sensitive. Most evidence-based recommendations require rapid implementation to influence most significantly patients’ outcomes [3,9,12]. Management delays due to lack of recognition can be detrimental. We have shown that in 56% of the sepsis patients identified by the system, serum lactate had not been obtained to help stratify sepsis severity . In addition, serum lactate was not measured in a significantly higher percentage of patients who did not have their sepsis syndrome documented appropriately.
Although these results suggest that the sepsis detection and alert system has the potential to provide for earlier, appropriate recognition and treatment of sepsis, for the system to truly make a difference it must ultimately improve patient outcomes. This study was not designed to evaluate whether or not the system could improve outcomes; however, it provided insight into adverse events in septic patients and how documentation and recognition might influence outcomes. Many would assume that those recognized would be more severely ill as they would be easier to identify. Our findings suggest that recognition and intervention might be more important than perceived acuity. This relationship needs to be studied in a larger cohort to see if this association becomes significant. The trend toward significance, however, supports the notion that this detection system could ultimately improve patient outcomes.
This study has multiple limitations. First, the algorithm is limited because of its lack of vital signs and its reliance on a blood culture order as a surrogate for fever. If informed by better electronic record systems including vital signs, the effectiveness of the algorithm is likely to improve. Of note, however, the inclusion of the blood culture order makes the SIRS designation more specific to sepsis, a feature absent from other early warning systems, which use vital signs to detect clinical deterioration. In addition, this more simplified system may be used now by institutions that do not yet have vital signs available electronically. Second, the study took place in a single center. While lack of recognition is likely a general problem, the detection system needs to be evaluated in different clinical environments. In addition, while documentation is not a true surrogate for clinician recognition, it is the best proxy we have, short of prospective survey, to assess physician’s thinking. Third, the study design did not allow us to evaluate what cases of sepsis were missed by the algorithm. Nevertheless, this study did suggest that the algorithm could increase detection. Fourth, we did not test the outcomes of the intervention in clinical practice-an evaluation that might reveal the incremental benefits and risks of deploying the system in usual settings.
In conclusion, despite several limitations, this study demonstrated that an automated sepsis detection and alert system was easily implemented and reasonably accurate in the real-time identification of septic patients. It provided these alerts before interventions of the sepsis bundle could be implemented leading to adverse outcomes. This system should improve patient care by identifying under-recognized septic patients and by alerting clinicians to provide aspects of sepsis management that are timesensitive and may otherwise not be instituted.