The process of revising the KB, expert diagnoses, and diagnosis interpretation was done iteratively. That is, after each run we reviewed a number of the cases in which the program did not have an acceptable hypothesis as the first hypothesis, determined the types of errors involved and made revisions. Because of the time involved in this process, we did not review all of the unacceptable cases until the numbers were manageable. However, one approximate way to measure the effect of the changes made in the KB separately from changes in the expert diagnoses and interpretation is to rerun the cases with the original KB. Doing this yields 141 of 242 cases correct or 58.3%. This is likely to be somewhat low since many of the failed cases have not been reviewed and may have acceptable top hypotheses. In the final run of the cases through the program, the program produced good first hypotheses for 216 of the 242 cases or 89.3%. The cases that failed are listed in table 3. The refinement process leading to this state has not been completely monotonic. Indeed, three of the cases that failed in the final run are done correctly using the original KB. Thus, at this point handling the existing problems in a more general way is more likely to produce a robust diagnostic program than further adjustments to the probabilities in the KB.
We analyzed the errors in the failed diagnoses and attempted to categorize the problems. Because of the interactions among different parts of the KB and the diagnosis process, the classification is somewhat subjective and for many of the cases multiple explanations are possible. The most illuminating classifications (those used in table 3) included the varying manifestations with different chronicities and severities of the causes, the independence assumption for diseases, incomplete or imprecise data, and the generation of causal pathways especially when there were conditional probabilities involved. In the following paragraphs we will discuss the nature of these problems in the final run plus illustrative examples from the previous runs.