WFDB Programmer's Guide: Evaluation of ECG Analyzers
Evaluation of ECG Analyzers
| bxb -r
record
-a
reference-annotator test-annotator [ options ... ]
rxr -r
record
-a
reference-annotator test-annotator [ options ... ]
mxm -r
record
-a
reference-annotator test-annotator [ options ... ]
epic -r
record
-a
reference-annotator test-annotator [ options ... ]
sumstats
file
plotstm
file
ecgeval
nst
[ options ... ]
|
The motivation for developing the MIT and AHA databases was to provide
material for evaluating the accuracy of arrhythmia detectors,
particularly with respect to ventricular arrhythmias. Between 1984
and 1987, the Association for the Advancement of Medical
Instrumentation (AAMI) sponsored the development of a recommended
practice (designated ECAR) for using the databases for this purpose.
The aim of ECAR was to specify the evaluation methodology in
sufficient detail to permit reproducible testing, and to encourage
informed comparisons of the performance of ventricular arrhythmia
detectors in the analysis of these standard test recordings. More
recently, the AAMI has developed, and ANSI has adopted as American
National Standards, a standard (ANSI/AAMI EC38:1998) for ambulatory
electrocardiographs, and a companion standard (ANSI/AAMI EC57:1998)
for testing and reporting performance results of cardiac rhythm and ST
segment measurement algorithms. EC38 and EC57 specify standard
protocols for evaluating the automated analysis algorithms that are
included in many such devices. These protocols include those
developed for the earlier recommended practice, and extend them to
evaluation of supraventricular arrhythmia and ischemia detection.
EC38 and EC57 specify the use of ‘
bxb
’, ‘
rxr
’, ‘
mxm
’,
and ‘
epic
’ to perform evaluations, and further specifies the use
of the MIT DB (as well as two other databases formerly available on
the MIT-BIH Arrhythmia Database CDROM and now freely available on
PhysioNet), the AHA DB, and (for devices that perform analysis of the
ST segment) the ESC DB. If you are interested in this subject, obtain
copies of the American National Standards for
Ambulatory
Electrocardiographs
(ANSI/AAMI EC38:1998) and for
Testing and
Reporting Performance Results of Cardiac Rhythm and ST Segment
Measurements Algorithms
(ANSI/AAMI EC57:1998; see section
Sources
).
To evaluate an arrhythmia detector using this software, obtain for each
WFDB record to be used in the test an annotation file containing the
detector’s analysis of each beat. These are referred to as the ‘test’
annotation files (or the ‘algorithm’ annotation files, in EC38 and EC57).
The placement of the beat annotations must match those in the reference
annotations within 150 msec; thus it is not necessary to place
annotations precisely at the PQ junction (as in the AHA DB reference
annotations) or on the major local extremum (as in the MIT DB reference
annotations). If the detector is capable of shut-down (i.e., if it
inhibits its QRS detection function during periods that it judges are
unreadable), the test annotation files should include a
NOISE
annotation with
subtyp = -1
at the beginning of each period of
shut-down, and a
NOISE
annotation with any other
subtyp
at the end of each such period. (If the record ends while the detector
is shut down, the annotation file should include a final ‘end of shut-down’
annotation as above to permit correct shut-down accounting.) If the
detector is capable of ventricular fibrillation detection, the test
annotation files should also include
VFON
and
VFOFF
annotations; it is not necessary to mark flutter waves (use
FLWAV
annotations to do so if desired). See the
man
page for ‘
epic
’,
in the
WFDB Applications Guide
, for information on marking
atrial fibrillation, ischemic ST episodes, and ST deviation measurements
in test annotation files. Any annotations that appear in the
first five minutes of an annotation file are treated as belonging to the
detector’s learning period, and are not used in the evaluation. The
evaluation software examines such annotations only to determine the
detector’s state (normal, shut down, or in VF) at the beginning of the
test period.
Program ‘
bxb
’ implements the beat-by-beat comparison algorithm described
in EC38 (section 4.2.14.2.2) and EC57 (section 4.3.2). By default, the output
is in a self-explanatory matrix format. The ‘
-L
’ option, which must be
followed by two file names, specifies that the output of ‘
bxb
’ should be
written in line format, for further processing by ‘
sumstats
’. The
line-format output includes column headings only if the output file must be
created from scratch. In this way, ‘
bxb
’ can be used repeatedly to build
up a line-format tables for multiple records. Among the other options is
‘
-o
’, which causes ‘
bxb
’ to generate an output annotation file (with
annotator name ‘
bxb
’) indicating agreements and discrepancies between the
input annotators.
‘
rxr
’ can be used to performed the run-by-run comparison described in EC38
(section 5.2.14) and in EC57 (sections 4.4.3 and 4.4.4). ‘
mxm
’ compares
heart rate, HRV, or other measurements, as described in EC38 (section
4.2.14.2.3). ‘
epic
’ evaluates VF and AF detection, and ST analysis, as
described in EC38 (sections 5.2.14), and EC57 (sections 4.5 and 4.6). These
programs also accept a ‘
-L
’ option to produce line-format output as for
‘
bxb
’.
‘
sumstats
’ derives the record-by-record, episode-by-episode, and
aggregate performance statistics described in EC38 (section 4.2.14.3) and in
EC57 (sections 3.5.2 and 3.5.3) from line-format output files produced
by ‘
bxb
’, ‘
rxr
’, ‘
mxm
’, and ‘
epic
’. The input file
must include the column headings so that ‘
sumstats
’ can recognize
the file type. The output includes a copy of the input, with aggregate
statistics appended at the end. ‘
plotstm
’ generates a PostScript
scatter plot of ST measurement comparisons gathered by ‘
epic
’, as
described in EC57 (section 4.6.2).
The easiest way to use these programs is to run ‘
ecgeval
’, which
generates a script (batch) file to run ‘
bxb
’, ‘
rxr
’, etc., for
each record in a database. See
Evaluating ECG Analyzers
(in the
WFDB Applications Guide
) for details.
By adding noise to annotated ECG records, the noise tolerance of an
arrhythmia detector can be measured. This idea was described by the
author, along with W.K. Muldrow and R.G. Mark, in “A noise stress test
for arrhythmia detectors”,
Computers in Cardiology
11
:381-384 (1984). Program ‘
nst
’ adds calibrated amounts
of noise to ECGs (or other signals), generating an output record in WFDB
format. ‘
nst
’ was used to generate the graded series of noisy ECG
records in the MIT-BIH Noise Stress Test Database (see
http://physionet.org/physiobank/database/nstdb/
).
These records are among those specified as standard test
material by EC38 (section 4.2.14.2) and EC57 (section 3.2).
PhysioNet
(
wfdb@physionet.org
)