Cardiac data processing by an expert is ECG On-Demand’s core business yet I have come to the conclusion that total reliance on an expert’s skill for ECG analysis may not be the best approach to overcome the challenge of delivering a consistently accurate ECG interpretation service with the lowest probability of error.

Expert ECG interpretation is promoted by the cardiac data analysis industry as a method of reducing unnecessary secondary care referral or as a means of avoiding costly and embarrassing medical negligence claims. Although an expert is more likely to correctly interpret an ECG than a non-expert, even an expert will deviate from the “gold standard” result (If a consensus opinion of 3 consultant cardiologists can be regarded as a gold standard?) with a statistically significant frequency. In my opinion, unrealistic expectation levels are widely promoted by industry with some providers claiming a zero or close to zero error rate.

Those who believe they have procured an “error-free” ECG interpretation service are advised to read the paper Competency in Interpretation of 12-Lead Electrocardiograms: A Summary and Appraisal of Published Evidence by Stephen M. Salerno et al. The gist of the paper is well made by the following choice passage;

One common feature of most ECG interpretation studies is the use of an expert electrocardiographer gold standard, typically a consensus panel of cardiologists. This may be problematic because interpretations by several cardiologists reading the same ECG often vary substantially. Even one cardiologist reading the same ECG on separate occasions may have substantially different interpretations

The concurrence of opinion between expert services

Differences of opinion between expert ECG interpretation providers were demonstrated in a 2015 High Court counterclaim for breach of contract involving 2 commercial competitors. 25,545 patients with a mean age of 60 were screened for AF, using a limb-lead ECG test as part of a health screening service.

B claimed A’s service was sub-standard and that its own service, in comparison, was almost error-free with its internal QA processes showing only 1 erroneous ECG report per 10,000.

  • A was of the opinion that 312 of the patients exhibited AF.
  • B was of the opinion that 18 of the 312 (5.6%) were false positives. [The 18 alleged false-positive results were then reviewed by an independent third party, appointed by B, who was of the opinion that only 11 were false positives.]
  • B then alleged that based on its own work in a similar population, A’s AF detection rate was too low by 167. [After a full review of all 25,545 ECGs, B amended the 167 to 12.]

Following comment from the High Court Judge, B abandoned its case.

This unusual case demonstrates that complete agreement between the 3 expert ECG interpretation providers could not be achieved and some divergence of opinion between expert electrocardiologists will inevitably exist.

However, the accuracy of AF detection by experts compares favourably with non-experts. The 2005 SAFE Study where general practitioners and nurses were trained to detect AF using a limb-lead ECG, serves as a reasonable comparison.

No alt text provided for this image

The reader should be aware that the AF prevalence in the SAFE Study was estimated to be 7.2% vs 1.2% in A’s work. This means the task of AF detection in the 25,545 is statistically harder.

The Current State-of -Art

At ECG On-Demand we employ continuous audit techniques backed by scheduled statistical analysis for quality assurance. I can therefore categorically confirm that all cardiac physiologists make mistakes and even consultant cardiologists are not immune from error. To complicate matters further, there will always be a proportion of ECG recordings that are open to a divergence in professional opinion. Although training and rigorous quality assurance can deliver a diagnostic result with a greater probability of being correct and a level of consistency, this is far from a perfect system.

The excessive false-positive results from ECG machine onboard algorithms provide the financial incentive for employing my own company’s services with most clinicians believing that expert ECG analysis is more reliable than computer-based ECG analysis. Whilst this conclusion may have had good justification in the past, new computer ECG analysis techniques challenge this convention.

Deep Learning – New Horizons

Machine Learning (ML) and in particular, deep learning, is a computer programming technique that aims to train a computer in pattern recognition

You probably use ML algorithms without knowing it. Facial recognition software used in Apple, Google and Facebook photo applications, Siri on the iPhone, Spotify and Netflix suggestions, all rely on ML.

Machine Learning can be used to recognise complex patterns in medical imaging with more accuracy and reproducibility than the human eye. Deep learning uses computing power far larger than that available from a personal computer or medical device processor.

Conventional on-board ECG machine algorithms are rule-based:

IF QRS duration ≥ 120 ms
and
S duration ≥ 40 ms in any 2 leads of I/aVL/V4/V5/V6
or
R duration > 60 ms and R amplitude > 500 μV in V1
and
R duration < 100 ms in any 4 leads of I/aVL/V4/V5/V6 and QRS area > 0 in V1
and V1 does not terminate in S or S'
or
QRS duration > 105 ms
and
S duration ≥ 60 ms in any 3 leads of I/aVL/V4/V5/V6
and
R duration > 60 ms in V1
and
QRS area > 0 in V1

PRINT “Right bundle branch block”

(Mortara Instrument REF 9515-001-51-ENG Rev A1)

Machine Learning works on the principle of pattern recognition i.e. if it looks like Right Bundle Branch Block PRINT “Right Bundle Branch Block”.
A standard 12-lead ECG sampled at 1000Hz will contain 120,000 data points, each with a time and voltage domain. Each data point and its spatial relationship with its other co-existing data points are mapped. Complex pattern recognition of ECG wave morphology is achieved by comparison of the ECG signal to an extensive database of categorised and classified ECG signals

We are currently developing and testing an ECG processing system that uses Machine Learning developed by Cardiologs. At the time of writing the algorithm had been trained using a data set of over 500,000 ECGs, each having been individually annotated by a consultant cardiologist.

If an individual were to review 25 ECGs per working day, it would take them over 90 years to build up the same experience as the AI system.

The computer learns to recognise patterns it was taught with the annotated database. It’s much like the way an actual physician learns. In doing so, the computer-generated algorithm replicates the consensus opinion of a population of experts rather than an individual and thus provides a result with a greater statistical probability of being correct whilst removing individual bias. Machine greater accuracy and consistency together with a quantitative measure of confidence in the test result.

Machine Learning – Human feedback loop

No alt text provided for this image

At ECG On-Demand, we do not plan to replace our clinicians with Machine Learning but to use ML to assist them in their decision-making processes. ML increases the speed and accuracy of ECG data analysis. Each ECG will be post-processed by a ECG On-Demand clinician not only to confirm or edit the ML-derived output but also act as a feedback mechanism into the machine learning process.

Barriers to Adoption

A 2017 PwC survey revealed a general mistrust of Artificial Intelligence (AI) “67% of CEOs think that AI and automation will have a negative impact on stakeholder trust in their industry over the next 5 years”. The healthcare industry if anything will be even more conservative with current negative assessments of computer-based ECG interpretation algorithms.

AI requires access to Cloud-based processing power. Because AI “learns”, its performance improves significantly when fed with large numbers of annotated ECGs. Effective implementation of AI requires both access and contribution to big data sets. Some healthcare organisation will need to change their approach to outsourced data processing and sharing if they wish to benefit from newer computer technologies.

Widespread adoption will require evidenced studies supported by rigorous data analysis.

Conclusion

Differences in professional opinion when interpreting ECG data are inevitable and where they occur they carry both patient and litigation risk. We believe Machine Learning acting as the second opinion to a human expert, will reduce this risk.

The future of electrocardiography will likely involve the migration from onboard signal processing to “dumb” digitally connected data collection devices feeding into AI-powered pattern recognition platforms.

If implemented successfully, ML will improve quality, reduce the burden of cardiac data analysis and increase the capacity of the healthcare system to deliver more care.

Even experts can benefit from a little help.

Mark

BBB ECG image courtesy of https://commons.wikimedia.org/wiki/User:Madhero88