CDM Seminar Series 2004-05
The inconsistency of expert assignments to variables in expert systems can
be a major source of performance limitation. I will present the results of an
experiment designed to assess the inter-rater and intra-rater consistency of
assignments within a commercially successful Medical Diagnostic Expert System.
The expert-assigned variables represent the strength of relationships between
diseases and findings (which include patient-reported symptoms, signs found
on examination, lab values, etc.), and are intended to allow the system to infer
the likelihood of disease given the presence or absence of a particular set
The experiment necessitated mapping the system's variables to a probabilistic framework, and inferring the experts' mental models in terms of that framework. Inter-rater consistency was measured by direct comparison of variable assignments. Intra-rater consistency was measured by inferring a likely mental model in probabilistic terms for each expert, then noting that each set of variables that reference a common finding implies a specific Bayesian leak probability that should be constant across all variables in the set. The consistency of assignments involving a common finding can then be estimated by the consistency of the Bayesian leak probabilities implied by those assignments. This also suggests a method for correcting expert assignments by forcing "leak invariance", and I investigated the hypothesis that correcting for intra-rater consistency in this way would improve inter-rater consistency.