Automated Knowledge Acquisition

Automated Knowledge Acquisition and Diagnosis of Level of Competence

The readiness of teams to perform complex tasks hinges critically on cognitive abilities that are not well measured in current team evaluations. CTI has developed a highly automated approach to assessing mental models. Specifically, the approach enables us to
  1. assess the global similarity of individuals and teams to domain experts and journeymen
  2. specify mental models used by experts and journeymen
  3. diagnose specific strengths and deficiencies in the use of those models by individuals and team

The arguments with which team members defend and critique their assessments of a problem are taken to be a window onto mental models of tactical situations. The content of arguments is analyzed using Latent Semantic Analysis. LSA essentially defines a highly dimensioned space in which an argument or corpus of arguments made by an individual or a team has a measurable distance from previously indexed arguments.

1.Global similarity measurement In a recent study, 377 arguments by 31 Navy officers with high and low experience in anti-air warfare were used to generate an LSA index. A jackknife classification analysis was then conducted (in which each officer was classified using a discriminant function developed from proximity data concerning all other officers, but not the officer being classified). Officers were classified as members of the high-experience group or low-experience group with 87% accuracy. However, using a sample that was more homogeneous with respect to AAW experience produced near-chance categorization accuracy.
2.Specifying mental models Distinctive arguments cluster on different dimensions of the LSA space. The elements of mental models can be rapidly defined by examining the distribution of arguments on these dimensions. In the present study, officers considering a suspect track in a high-fidelity anti-air warfare scenario were found to use a mental model consisting of five elements: track route, track response, track kinematics, track localization capability, and an unclassifiable element. These results are not dependent upon the homogeneity of the sample of officers.
3.Diagnosing deficiencies in mental models We can test for systematic differences in the instantiation of a mental models between more and less experienced officers. In the present study, arguments related to the localization element were found to be used reliably more often by officers with greater experience than those with less experience. This finding is consistent with prior research. Arguments generated by the officers studied here were classified with 80% average accuracy into appropriate model elements. The accuracy for classifying localization arguments (which discriminated officers by experience) was 95%. These findings demonstrate the feasibility of automated diagnosis of the mental models of individuals or teams.

The current research advances the application of LSA by demonstrating that indexing brief arguments, rather than much longer documents such as essays or article abstracts, supports both accurate global assessment of individuals and detailed diagnosis of the conceptual content of their arguments. The techniques were tested using the arguments of individuals, but they are directly extensible to team assessment by treating the team as an individual represented by the many arguments of team members.

We have developed extensions of the method to identify and test relationships between the mental models of team members (e.g., overlapping models, contiguous models). Several promising methods of improving upon the results reported here, by using non-linear SVD, factor rotation, word order data, and other approaches are also being reviewed.


Copyright © 2000-2011 Cognitive Technologies, Inc.
Questions? Comments? Contact