Next: Introduction
Evaluation of a New Method for Cardiovascular
Reasoning
William J. Long, PhD
MIT Laboratory for Computer Science, Cambridge, MA, USA
Shapur Naimi, MD and M. G. Criscitiello, MD
New England Medical Center and
Tufts University School of Medicine, Boston, MA, USA
Reprinted from Journal of the American Medical Informatics Association,1:
127-141, 1994
Abstract:
- Objective:
- Evaluate the accuracy of the detailed diagnostic
reasoning of the Heart Failure Program incorporating a new mechanism to
handle temporal relationships and severity constraints.
- Design:
- Tools were developed to summarize diagnoses and
automatically generate evaluation forms. Five expert cardiologists were
asked to review the reasoning of the program, with two analyzing each
case. Cases were gathered retrospectively for diversity and difficulty
and 26 randomly selected cases were evaluated. The underlying issues
were identified and classified.
- Results:
- Both reviewers rated the first diagnosis correct in 25%of the cases and at least one rated it wrong in 10%. Analyzing the
detailed reasoning, 137 issues were raised, about 5.3 per case. 53%of
these were possible concerns raised by one reviewer. Of the 5.3 issues
per case, 2.5 were attributable to controversies, misunderstandings, or
mistakes; 1 was due to the overly simplistic representation of the
summaries; and 1.8 were issues related to the program.
- Conclusion:
- Overall, the program is capable of providing high
quality detailed diagnostic hypotheses for complex cardiovascular cases.
The results highlight several issues: 1) the difficulty of effectively
summarizing hypotheses, 2) the nature of a physician's causal
explanation, and 3) some problems in evaluating detailed diagnostic
reasoning. The mistakes the program made imply that some additional
refinement is needed but that the reasoning mechanisms developed can
support the appropriate reasoning. The appropriate next step is a
prospective evaluation addressing the program's usefulness.