CDM Seminar Series 2004-05


On the Consistency of Experts in Assigning Variables within Medical Expert Systems
Thomas Lasko, MD
Graduate Student

Friday, 11/19/04
1:00pm, 32-250

The inconsistency of expert assignments to variables in expert systems can be a major source of performance limitation. I will present the results of an experiment designed to assess the inter-rater and intra-rater consistency of assignments within a commercially successful Medical Diagnostic Expert System. The expert-assigned variables represent the strength of relationships between diseases and findings (which include patient-reported symptoms, signs found on examination, lab values, etc.), and are intended to allow the system to infer the likelihood of disease given the presence or absence of a particular set of findings.
The experiment necessitated mapping the system's variables to a probabilistic framework, and inferring the experts' mental models in terms of that framework. Inter-rater consistency was measured by direct comparison of variable assignments. Intra-rater consistency was measured by inferring a likely mental model in probabilistic terms for each expert, then noting that each set of variables that reference a common finding implies a specific Bayesian leak probability that should be constant across all variables in the set. The consistency of assignments involving a common finding can then be estimated by the consistency of the Bayesian leak probabilities implied by those assignments. This also suggests a method for correcting expert assignments by forcing "leak invariance", and I investigated the hypothesis that correcting for intra-rater consistency in this way would improve inter-rater consistency.


CDM Seminar Series