Why use an expert ECG interpretation service when most ECG machines have on-board ECG interpretation algorithms?

In 2018, ECG On-Demand published a white paper to identify and quantify the difference in ECG interpretation performance between ECG machine algorithm and human expert.

ECG machines equipped with built-in ECG interpretation algorithms are frequently used by non-cardiologists as an aid to determine the most appropriate patient care pathway following a 12-lead ECG. Although ECG machine algorithms reduce the probability of missing important ECG abnormalities, reliance on them is responsible for significant unnecessary secondary care referral and patient anxiety. The precision of diagnosis of commonly encountered ECG abnormalities, claimed by ECG machine manufacturers, are rarely reproduced in real life clinical practice.

 

Methods

12-lead ECGs were recorded and transmitted for expert interpretation from multiple primary care sites over a 1 year period. Digital ECGs sampled at 1000 Hz were recorded using a Mortara ELi-10 ECG machine with on-board Veritas algorithm. The digital ECGs, together with the corresponding computer generated interpretations, were transmitted to Technomed ECG On-Demand service for interpretation by expert cardiac physiologists using Technomed’s ECG Cloud platform. To ensure the cardiac physiologists were categorising the ECGs appropriately, the cardiac physiologists work was sampled and audited by a panel of consultant cardiologists to estimate the concurrence of agreement with consultant cardiologist opinion. The risk classification of the Veritas ECG algorithm was compared to the risk classifications of the cardiac physiologist using the same ECG data. To simplify the comparison between on-board ECG machine algorithm and ECG expert, only the “condition” statement generated by ECG machine algorithm was considered. 

The full paper can be read here however we present a summary of the results:

 

Key Findings

KEY FINDING 1: THERE WAS GOOD AGREEMENT (97%) BETWEEN THE ECG ALGORITHM AND THE ECG EXPERT WHEN THE ECG MACHINE CONDITION STATEMENT WAS “NORMAL ECG”. 

Significant abnormal findings were found in approximately 2% of the “normal” ECGs checked by the ECG expert. The author believes this is due to the ECG algorithm not having the benefit of knowing why the patient was undergoing an ECG test. (e.g. Veritas classified the ECG of a 45 year old female with a heart rate of 45 bpm as “normal” when the patient’s symptoms were “dizziness”.) 

 

KEY FINDING 2: THERE WAS POOR AGREEMENT (31%) BETWEEN THE ECG MACHINE ALGORITHM AND THE ECG EXPERT WHEN THE ECG CONDITION STATEMENT WAS “ABNORMAL“ ECG.

This would suggest that 2 out of every 3 patients referred to secondary care on reliance of the on-board ECG algorithm would be sent inappropriately, resulting in unnecessary expense and patient anxiety. 2017/18 tariff (excluding market forces factor) for a 1st attendance cardiology outpatients appointment is £174. The author estimates that reliance on the on-board ECG machine algorithm may result in up to 29 unnecessary outpatient appointments per 100 ECGs performed in a GP practice environment. Averaged out this would be an unnecessary spend of £50.46 for every ECG recorded.

 

KEY FINDING 3: 6% OF THE ECGs WERE CATEGORISED BY THE ECG EXPERT AS BEING UNACCEPTABLE QUALITY FOR RELIABLE ECG INTERPRETATION. DISTRIBUTION WAS NOT EQUAL BETWEEN THE ALGORITHM DERIVED CONDITIONS

The prevalence of poor quality ECGs was highest in the “Atypical” algorithm derived condition. Unhelpfully, there was also a significant percentage of abnormal ECGs within the “Atypical” group also. This means an ECG with an “Atypical” condition statement is probably poorly recorded but may be abnormal.

 

KEY FINDING 4: 1% OF ECGs WITHIN THE ABNORMAL ECG CONDITION GROUP REQUIRED URGENT SECONDARY CARE REFERRAL

The on-board ECG algorithm was unable to differentiate between those abnormal ECGs requiring standard and urgent attention. 

 

KEY FINDING 5: AGREEMENT BETWEEN THE ECG EXPERT AND CONSENSUS OPINION OF THE AUDITING CONSULTANT CARDIOLOGIST PANEL WAS 96% WHICH IS WITHIN THE EXPECTED RANGE. AGREEMENT WAS NOT EQUAL BETWEEN THE ALGORITHM DERIVED CONDITIONS

Divergence of opinion was highest in the “Atypical” and “Borderline” ECG conditions suggesting that ECG recording quality has a negative impact on ECG interpretation reproducibility. Unsurprisingly, divergence of opinion was also relatively high in the “Borderline” ECG condition.