Using the 242 cardiac cases collected from discharge summaries to conduct a formative evaluation of the Heart Failure Program has shown that with some adjustments to the knowledge base the mechanism for generating differential diagnoses using a probability network and a heuristic hypothesis generator is effective enough to produce appropriate diagnoses about 90%of the time in this training set. This seems quite respectable for several reasons: 1) The cases included the more complicated ones in a tertiary care hospital and excluded more simple ones. 2) The criteria for appropriate diagnosis was demanding, requiring that the first hypothesis exactly match one of the interpretations of the cardiologists' diagnosis. 3) In many of the failing cases the second hypothesis was satisfactory. 4) Almost all of the errors involved single aspects of complicated diagnoses.
However, the errors made by the program are ones that are obvious to the cardiologists using the same data. All the errors have been reviewed, eliminating any due to oversight. Thus, if the program can not produce an acceptable diagnosis, there is some weakness in the representation or use of the knowledge. It is probably possible to make further modifications to the KB until all of the top hypotheses are correct, but from the changes that have been made it is apparent that some of the diagnoses are becoming sensitive to small changes in causal probabilities. This indicates that the changes may not improve the performance of the program on future cases and that we are now placing too much reliance on a mechanism that is not using all of the information available about the diagnostic problem. Therefore, it is more appropriate to enhance the reasoning mechanisms to deal appropriately with the problems identified in the previous section. These include relationships of chronicity, severity, disease coexistence, more complicated dependencies among pathophysiologic states, and context sensitive interpretation of findings.
The second objective of the evaluation was to determine the applicability of the program to the types of cases that appear in a tertiary care hospital. The sample of cases in the study included all of those for whom discharge summaries were available and by DRG fit in the general categories of complicated cardiovascular disease. There were no cases that the program was unable to handle because the relevant diseases were not included in the KB. There were eight primary diseases covered by the KB that did not appear in the sample, because of their rarity. This is no guarantee that all future cases will be covered, because there are rare diseases that we know are not included, but it is a good indication that these situations will be rare.