Hybrid Architecture for Metacognitive Learning
- Year 1
Overview of scientific progress
Metacognitive learning experiments.

We have developed three training and two test scenarios for the Decision-Making Evaluation Facility For Tactical Teams (DEFTT) simulator. These scenarios have been used in studies for training metacognitive skills for Naval officers (Cohen, Freemen, Wolf, Militello, 1995; Freemen, Cohen, 1996). The scenarios were derived from existing DEFTT simulations, and were enhanced to include tracks whose behaviors were modeled after tracks discussed at length in the critical incident interviews conducted as part of the TADMUS program. We are currently translating these scenarios from DEFTT to the CIC simulator. We have copies of scenarios developed by Sandra Marshall, and will work with her and the other team to develop a common set of scenarios for testing the learning characteristics and performance of the various hybrid architectures, and for comparison with human subjects.

Working with human subjects (16 officers with CIC or equivalent experience) and the scenarios identified above, we have collected data that includes assessments of likely intent, judgments of confidence in those assessments, explanations and criticisms of the most likely intent, and actions that officers consider appropriate for the current situation. (The findings of this study are summarized below.) These data are being used to develop the training program for the hybrid architecture, and will be used to compare its performance to that of the human subjects in experiments this fall.

Related data from another subject pool (Surface Warfare Officers School; 60 Naval officers, 92% with shipboard CIC experience) is available, and may be analyzed if required to form a more extensive basis for training. Further data regarding the domain or metacognitive performance of human subjects will be collected as required using Towne's CIC simulator.

Long-Term Knowledge Base

Using critical incident interviews collected as part of the TADMUS program we have encoded a Long Term Knowledge Base (LTKB). This encodes domain knowledge required to combine evidence concerning the behavior of tracks in support of hypotheses, e.g., 'search-and-rescue,' 'commercial air,' or 'hostile-intent', and associates situation models structures with plan structures, e.g., 'issue warning,' 'VID track,' or 'illuminate track.' Working with the SHRUTI simulator, we are now verifying that the LTKB can draw the inferences that support reflexive recognitional responses and metacognitive reasoning.

Hybrid architecture design & implementation

We have constructed adaptive critic memories and performed various tests concerning improved exploration methods, the use of measures of confidence in estimates of expected value, and the relative merit of various associative memory structures and algorithms. We are currently integrating the various components of the hybrid architecture (the CIC simulator, Shruti and the LTKB, and the adaptive critic architectures) for machine learning experiments this fall.

Year two: Metacognition organizes domain learning

In the second year, we will explore how metacognitive behavior organizes and influences domain learning. This occurs at two time scales. (1) Evidence is received and reflexively combined into stories (causal structures that organize evidence within frames that, e.g., explain intent). Metacognitive skills critique these reflexive results, and set the goals for adapting the results (correcting). As the stories are adapted, new situation models and plan structures are instantiated. These are then stored back into the domain model (a la Case Based Reasoning), where they are available for future recognitional pattern matching. (2) At a slower time scale, metacognition provides for the review of prior episodes. Here metacognition sets knowledge acquisition goals that focus learning behavior. At this level, metacognition can be sensitive to a number of factors involved in developing a rich and diverse domain understanding, including the abilities to explain conflicting evidence, to identify unreliable evidence, and to complete gaps in understanding. Incorrect, or inappropriate, explanations are also identified.


Recent accomplishments
Two important findings have emerged over the past 12 months in relationship to this work. In studies with human subjects in Naval combat simulations, we have validated (a) measures of metacognition and (b) methods of manipulating metacognition via training, (c) we have collected data concerning human metacognitive performance. Based on this work, we can proceed to develop parallel (a) measure, (b) training programs, and (c) performance criteria for the hybrid system. (2) An extension to the SHRUTI architecture has been developed that supports encoding and efficient reasoning over inconsistent knowledge. This advance in SHRUTI's functionality will enable the hybrid system to recognize inconsistant knowledge and use it, as human decision makers do, to improve situation models. These findings are summarized below.

Learning metacognitive skills
A study of training based on the Recognition / Metacognition model was conducted at the Naval Postgraduate School (NPS) at Monterey, CA. Thirty-five officers with an average of ten years of military experience participated in the study. CTI's training was designed to help officers formulate situation assessments in a highly realistic simulation of CIC AAW operations, critique those assessments, correct them, and take action (that is, ceasing critical thinking) in a timely manner. Subjects received familiarization with the domain and the simulator, performed a pretest, received training (involving lectures, discussion, reading a training text, and performing exercises using paper and the simulator), and then completed a posttest. Results of the study indicated that training influenced metacognitive skills, as it was design to do. Training enabled officers to:
  • devise more arguments in defense of their assessments (an increase of 25%, p=.001; An assessment of argument quality is in progress. In an earlier study for ARI, CTI found that quality rose (ns) as the number of arguments rose)
  • identify more of the assumptions underlying their assessments (an increase of 41%, p<.001)
  • recognize more of the evidence conflicting with a given assessment (an increase of 58%, p<.001)
  • specify more of the exception conditions under which seemingly conflicting evidence was consistent with the given assessment (an increase of 28%, p<.001)
Training also had a positive effect on decision outcomes, this is, on the bottom line. It enabled officers to:
  • make more accurate assessments of highly ambiguous, high-stakes situations (an increase of 30%,p=.001 for one of the two test scenarios, ns for the second scenario)
  • take actions that were more appropriate to the situation (p=.039)
  • maintain their confidence (i.e., their ability to act) even as they conducted more through critiques of their assessments (confidence rose 12.5%, ns).
Officers rated the training positively, and were more likely to do so the greater their tactical experience.

These results are significant for the hybrid architecture project in several ways. (1) They demonstrate that we have developed effective measures of several metacognitive skills used by Naval officers in complex CIC scenarios. On this basis, we can implement similar measures to monitor and test metacognitive operations in the hybrid system. (2) We have developed training (based on the Recognition / Metacognition model) that influences human use of metacognitive skills and domain outcomes. This training will be used to develop the training program for metacognitive machine learning in the hybrid architecture. (3) The results of the human training and the machine learning can be directly compared in terms of the elicited metacognitive measures.


Modeling inconsistent knowledge
Where traditional analytical processes simply aggregate concordant and conflicting data, we have observed that officers treat the two types of information quite differently: Conflicting evidence (e.g., regarding intent) is used as a symptom of erroneous assumptions and spurs efforts to find alternative interpretations of cues or alternative hypotheses. (Cohen, 1986, 1989). Metacognitive correcting actions attempt to resolve the three classes of problems identified by critiquing (incompleteness, conflict, and unreliability). They do so by selecting operations that will transform some part of the situation model, or cause it to be abandoned in favor of a different one. Experienced officers use several tools in their efforts to correct the deficiencies of models. To represent and reason efficiently over inconsistent knowledge it is crucial to model these human behaviors.

SHRUTI is a connectionist model of reflexive reasoning. It demonstrates how connectionist networks can represent relational structures and perform certain types of computations over such structures in an efficient manner. The SHRUTI architecture is being utilized in this project to encode the domain model and plan structures, and to support recognitional elaboration of situation models and plans in response to evidence. The results instantiated in the SHRUTI network are reviewed by metacognition (criticized and corrected), subject to available time, stakes, uncertainty and novelty.

Recent work has extended SHRUTI such that it can now deal with positive knowledge as well as negated facts and systematic knowledge (rules) involving negated antecedents and consequents. The extension only requires local inhibitory connections. The extended model explains how an agent can hold inconsistent knowledge in its long-term memory without being 'aware' that its beliefs are inconsistent, but detect a contradiction when two contradictory beliefs that are within a small inferential distance of each other become co-active during an episode of reasoning. Thus the model is not logically omniscient, but detects contradictions whenever it tries to make use of inconsistent knowledge in particular situations. The extended model also explains how limited attentional focus or action under time pressure can lead an agent to produce an erroneous response. The extended SHRUTI model is therefore capable of modeling a wider range of reflexive reasoning phenomena. Shastri and Grannes (1995).

This work has implications both for training and for the design of the hybrid architecture. Training may be improved by focusing on bringing together logically inconsistent aspects of the knowledge base and providing officers with improved skills for detecting, attending to, and resolving conflict. By explicitly modeling inconsistent knowledge, the hybrid architecture is able to maintain alternative explanations, to identify conflicting conclusions, and to activate metacognitive processes for improving situation models.


References
Cohen, M; Freemen, J; Wolf, S; Militello, L (1995) Training Metacognitive Skills in Naval Combat Decision Making. Arlington, VA. CTI, Inc.
Freeman, J, Cohen, M. (1996). Training for Complex Decision-Making: A Test of Instruction Based on the Recognition / Metacognition Model. Proceedings of the 1996 Command and Control Research and Technology Symposium, Monterey, CA.
Cohen, M, Freeman, J, Wolf, S. (In press). Metarecognition in Time-Stress Decision Making: Recognizing, critiquing, and correcting. Journal of the Human Factors and Ergonomics Society.





 

 

Copyright © 2000-2011 Cognitive Technologies, Inc.
Questions? Comments? Contact webmaster@cog-tech.com