Automated Knowledge Acquisition and Diagnosis of Level
of Competence
The readiness of teams to perform complex
tasks hinges critically on cognitive abilities that are
not well measured in current team evaluations. CTI has
developed a highly automated approach to assessing mental
models. Specifically, the approach enables us to
assess the global similarity of individuals and
teams to domain experts and journeymen
specify mental models used by experts and journeymen
diagnose specific strengths and deficiencies in
the use of those models by individuals and team
The arguments with which team members
defend and critique their assessments of a problem are
taken to be a window onto mental models of tactical situations.
The content of arguments is analyzed using Latent Semantic
Analysis. LSA essentially defines a highly dimensioned
space in which an argument or corpus of arguments made
by an individual or a team has a measurable distance from
previously indexed arguments.
1.Global similarity
measurement
In a recent study,
377 arguments by 31 Navy officers with high and low experience
in anti-air warfare were used to generate an LSA index.
A jackknife classification analysis was then conducted
(in which each officer was classified using a discriminant
function developed from proximity data concerning all
other officers, but not the officer being classified).
Officers were classified as members of the high-experience
group or low-experience group with 87% accuracy. However,
using a sample that was more homogeneous with respect
to AAW experience produced near-chance categorization
accuracy.
2.Specifying
mental models
Distinctive arguments
cluster on different dimensions of the LSA space. The
elements of mental models can be rapidly defined by examining
the distribution of arguments on these dimensions. In
the present study, officers considering a suspect track
in a high-fidelity anti-air warfare scenario were found
to use a mental model consisting of five elements: track
route, track response, track kinematics, track localization
capability, and an unclassifiable element. These results
are not dependent upon the homogeneity of the sample of
officers.
3.Diagnosing
deficiencies in mental models
We can test for
systematic differences in the instantiation of a mental
models between more and less experienced officers. In
the present study, arguments related to the localization
element were found to be used reliably more often by officers
with greater experience than those with less experience.
This finding is consistent with prior research. Arguments
generated by the officers studied here were classified
with 80% average accuracy into appropriate model elements.
The accuracy for classifying localization arguments (which
discriminated officers by experience) was 95%. These findings
demonstrate the feasibility of automated diagnosis of
the mental models of individuals or teams.
The current research advances the application
of LSA by demonstrating that indexing brief arguments,
rather than much longer documents such as essays or article
abstracts, supports both accurate global assessment of
individuals and detailed diagnosis of the conceptual content
of their arguments. The techniques were tested using the
arguments of individuals, but they are directly extensible
to team assessment by treating the team as an individual
represented by the many arguments of team members.
We have developed extensions of the method
to identify and test relationships between the mental
models of team members (e.g., overlapping models, contiguous
models). Several promising methods of improving upon the
results reported here, by using non-linear SVD, factor
rotation, word order data, and other approaches are also
being reviewed.
