The remaining 35%of the issues were related to the functioning of the
program. These can be further classified as shown in table 4.
The largest source of these stem from a difference between the nature of hypotheses from a Bayesian network and hypotheses that a human expert would generate. The single criterion for a good hypothesis from a Bayesian network is high probability. Our heuristic approach also tends to find single causal pathways unless there is separate evidence for more than one. The criteria implicitly used by experts in generating hypotheses require adequate evidence for diseases, inclusion of all important diagnoses, and inclusion of all plausible explanations for findings. They handle side-effects of therapies in a special way, liberally considering the possibility but being conservative about definite attribution. These differences come in conflict in a number of ways. First, the program does not require any particular evidence to invoke a disease. In several instances the program included diseases with inadequate evidence (from the reviewer's perspective) because that yielded a higher probability hypothesis, eg, concluding low renal perfusion on the basis of the renal function tests without evidence of pedal edema, suggesting anemia from findings without having a hematocrit, or using functional tricuspid regurgitation to account for a murmur when there is cardiomegaly. Second, the program will eliminate important diseases if the probability is too low. The program left out pulmonary embolism because the pleuritic chest pain has a high false positive rate and the shortness of breath could be otherwise explained, not taking into account the importance of diagnosing pulmonary embolism, which is potentially serious and very treatable. Third, the program also left out possible causal pathways for findings such as cough, LVH on electrocardiogram, and pulmonary hypertension because one reasonable explanation had been found. In one case, poor ventricular function was attributed to existing hypertensive heart disease and not to an acute MI because all of the findings were consistent with chronic disease. The reviewers disagreed because the acute MI will certainly worsen the ventricular function even if it did not cause it and even though there is no actual evidence of acute worsening. Finally, in a few instances the program attributed findings to drug side-effects when there were other possibilities, eg, steroids to account for pedal edema when there were possible cardiac causes.
These differences do not imply that the Bayesian network is incompatible with expert-like hypotheses, but it does imply that some additional processing is needed. First, a utility model is needed to avoid pruning out important hypotheses with probabilities below the threshold. This could be implemented by extending the reasoning mechanism to include value nodes as in an influence diagram or by doing some post-processing of the hypotheses to allow the appropriate reasoning about utilities. Second, additional pathways to findings need to be added to reflect alternative causes within the hypothesis. Third, diseases that are not well justified usually represent the most likely of several possible causes for a set of findings. These should be presented to the user in a more general form, using names such as ``poor ventricular function'' that leave the actual cause unspecified. Finally, therapy side-effects should be handled in a way that considers the additional means the physician has of determining the causes by changing or adjusting the therapy.
A few of the alternative diagnoses suggested by the reviewers involved
concepts that we have not included in the knowledge base, although we
have considered adding them. These were thyroid therapy causing
hyperthyroidism, high pCO from inappropriate ventilator setting, and
left ventricular aneurysm. These are concepts that can be added to the
knowledge base. There were other suggested alternatives that still seem
inappropriate, such as using diabetes as an explicit cause (beyond
changing some of the probabilities, which it currently does) and
cerebral embolism.
Another source of criticisms were inappropriate severity constraints. These included the level of hematocrit needed to account for high cardiac output, the severity of renal insufficiency that causes nausea, and the degree of pulmonary hypertension that causes RV hypertrophy. These can all be corrected in the knowledge base.
The rest of the issues are attributable to the probabilities in the knowledge base. There were diseases or pathophysiological states in hypotheses deemed too unlikely to be considered (but not ruled out) and diseases and states that the reviewers felt were should have been included. One problem that occurred twice was determining the source of murmurs from their location. The worst example was concluding the rare disease, isolated tricuspid stenosis, because the murmur description did not fit anything else. We have done considerable work on murmur description, but the variability of descriptions encountered in cases continues to be a challenge. One solution is to give the locations a high error rate. In two situations unlikely causal mechanisms were proposed by the program. One was essentially overcompensation: high vagal stimulation (evidenced by diaphoresis) causing low vascular resistance (with low blood pressure) causing high cardiac output (with a flow murmur). Another was letting the wrong causal pathway dominate in determining the effects of septic shock. The program concluded that the decreased filling pressure would cause low cardiac output rather than the decreased systemic resistance causing increased (or normal) cardiac output. Both of these problems can be corrected by adding causal constraints. Still, they provide interesting examples of how the underlying assumption of conditional independence in Bayesian networks can misrepresent causal models if secondary dependencies are overlooked.
One very general problem is the use of pertinent negatives to justify conclusions. The program uses negatives to find appropriate causal pathways in building hypotheses and to determine their probabilities, but only the abnormal findings are included on the displays and used to justify the nodes. It was clear both from written and oral comments that in many cases the negatives were crucial to making the statements convincing. In one case, the program had coronary spasm as the primary cause for an MI because a catheterization done six months prior did not mention the coronaries, which the program interpreted to mean that they were clear. Thus, coronary spasm was the most likely cause. Without that assumption, coronary artery disease would be a more likely cause. In another case, one reviewer judged the lack of aortic stenosis in the hypothesis to be a serious error and the other reviewer judged the same statement to be correct and commented that it should be justified by the lack of LVH on the electrocardiogram.
Thus, we have determined that the first hypotheses were rated correct by both reviewers in 25%of the cases and wrong by at least one in 10%. Analysis of the detailed reasoning identified 137 issues, about 5.3 per case. Of these 53%were possible concerns raised by one reviewer. Of the 5.3 issues per case, 2.5 were attributable to controversies, misunderstandings, or mistakes; 1 was due to the overly simplistic representation of the summaries; and 1.8 were issues related to the program. All of the program issues are ones that can be handled with refinement of the knowledge base and some additional processing to incorporate concerns of utility and evidence.