Citation
The effects of cognitive styles and orientation procedures on learning performance in a complex computer-based learning environment

Material Information

Title:
The effects of cognitive styles and orientation procedures on learning performance in a complex computer-based learning environment
Creator:
Wang, Sherwood R
Publication Date:
Language:
English
Physical Description:
xv, 219 leaves : illustrations ; 29 cm

Subjects

Subjects / Keywords:
Medicine -- Data processing ( lcsh )
Medicine -- Study and teaching ( lcsh )
Cognitive learning theory ( lcsh )
Cognitive learning theory ( fast )
Medicine -- Data processing ( fast )
Medicine -- Study and teaching ( fast )
Genre:
bibliography ( marcgt )
theses ( marcgt )
non-fiction ( marcgt )

Notes

Bibliography:
Includes bibliographical references.
General Note:
Submitted in partial fulfillment of the requirements for the degree, Doctor of Philosophy, School of Education and Human Development, Administration, Curriculum, and Supervision.
Statement of Responsibility:
by Sherwood R. Wang.

Record Information

Source Institution:
|University of Colorado Denver
Holding Location:
|Auraria Library
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
28478183 ( OCLC )
ocm28478183
Classification:
LD1190.E3 1992d .W36 ( lcc )

Full Text
THE EFFECTS OF COGNITIVE STYLES AND
ORIENTATION PROCEDURES ON LEARNER PERFORMANCE
IN A COMPLEX COMPUTER-BASED LEARNING ENVIRONMENT
by
Sherwood R. Wang
B.A., University of Chicago, 1978
M.S.T., University of Chicago, 1983
A thesis submitted to the
Faculty of the Graduate School of the
University of Colorado at Denver
in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
Administration, Curriculum, and Supervision
1992


1992 by Sherwood R. Wang
All rights reserved.


This thesis for the Doctor of Philosophy
degree by
Sherwood R. Wang
has been approved for the
School of Education
by
Brent G. Wilson


Wang,Sherwood R. (PhD., Administration, Curriculum, and Supervision)
The Effects of Cognitive Styles and Orientation Procedures on Learner Performance in
a Complex Computer-Based Learning Environment
Thesis directed by Dr. David H. Jonassen, Chair, Division of Instructional Technology
ABSTRACT
In medical education, there has been considerable interest in fostering the
development of clinical expertise in medical students, residents and other health
professionals. Developing the ability to apply/construct knowledge appropriate to a
clinical problem solving environment has been considered an advanced knowledge
acquisition task. In order to facilitate the processes involved in advanced knowledge
acquisition, a computer-based learning system was created in the area of transfusion
medicine in accordance with cognitive flexibility theory which prescribed specific
principles of learning and instruction for advanced knowledge acquisition.
The present study investigated the interaction of cognitive abilities/styles and
orientation procedures on the effectiveness of this computer-based module and on
student usage patterns during information access and problem solving activities.
ANOVA and ANCOVA techniques were used to explore main effects and interactions
between individual difference factors (field dependence/independence and cognitive
constriction/flexibility) and orientation procedures on test case scores after solving
practice cases. Pearson product-moment correlation analysis was also used to relate
usage variables in the practice cases to factors listed above.
IV


The results of the ANOVA and ANCOVA analyses consisted of a significant
main effect (p < .1) for the orientation treatment variable and for cognitive
constriction/flexibility as well as a significant interaction between these two factors.
The correlational analysis did not yield significant results between time spent in
different parts of the program and test case scores; however, the factors used in the
ANOVA analysis did influence usage patterns within the computer-based learning
environment.
This research suggested that learners did not benefit equally from the use of
complex computer-based instruction, even when the learner population was relatively
homogeneous in background knowledge and ability. Cognitive processing styles
which facilitated the integration of information from a variety of sources, allowed
learners to function more effectively in this complex computer-based learning
environment. To some extent, orientation procedures supplanted the information
processing skills necessary for successful knowledge integration from complex
computer-based instruction. Recommendations for future research based on these
findings are also presented.
This abstract accurately represents the content of the candidate's thesis. I recommend
its publication.
Signed
David H. Jonassen
v


DEDICATION
To my mother, who told me at age ten, that she thought I would be a great (if
eccentric) college professor someday, to my father, who supported me both
emotionally and financially throughout my doctoral studies, and to my sister Sharon,
who suffered through my most manic stages while proof reading of this manuscript
vi


ACKNOWLEDGEMENTS
First, I wish to acknowledge the National Institutes of Health for its support of
this research and Dr Ritchard Cable and Sarah Thai for their help in facilitating the data
collection phase of this project. This research could not have been completed without
the programming expertise of Julie Olesen Heidt, Patti Skari, and Joni Dunlap who
invested their time and effort to operationalize the module and tracking system used in
the research. I would like to recognize Minaruth Galey, Martin Tessmer, and Dian
Walster for their endless patience, encouragement, and good humor in listening the
many setbacks and problems associated with the completion of this project
I would like to thank Daniel Ambruso, Scott Grabinger, and Brent Wilson for
their participation on my dissertation committee and their insightful comments on the
paper. Scott also provided some much needed perspective on the dissertation process
which helped me through many rough times. Alan Davis deserves special recognition
for his help and guidance in the data analysis section of this paper. Laura Goodwin
also contributed to the project by giving me the confidence and statistical background to
attempt this project through coursework and in numerous unscheduled consultations.
Her generous donation of time to this paper significantly improved the analysis.
David Jonassen as an advisor, mentor, collaborator, and friend supported and
encouraged me throughout my doctoral program and deserves as much credit for the
successful completion of this manuscript as I do. Without him, this paper would never
have been. A graduate student could have no finer role model of scholarship,
creativity, and personal integrity to emulate.
vii


CONTENTS
CHAPTER
1. INTRODUCTION.................................................. 1
Problem Statement.......................................... 2
Proposed Solution.......................................... 4
Research Issues............................................ 4
Hypertext Learning Environments..................... 6
Research on Computer-Based Instruction and Hypertext.... 7
Defining the Inquiry....................................... 8
Research Methodology................................ 9
2. REVIEW OF RELATED LITERATURE.................................. II
Problem Definition......................................... 12
Advanced Knowledge Acquisition in Biomedicine.............. 13
Modeling Content and Problem Solving in Biomedicine........ 18
Proposed Solution: Facilitating Schema Assembly
Using Cognitive Flexibility Theory and Hypertext........... 22
Integrating Knowledge Structures: Schema Theory............ 23
Cognitive Flexibility Theory........................ 28
Hypertext, Cognitive Flexibility Theory,
and the Network of Ideas (Organizational Structure). 33
Defining The Inquiry..................................... 38
Individual Cognitive Styles/Abilities and
Flexible Schema Assembly............................ 39
Process instruction and evaluation methodology............. 44
Summary...................................................I... 41
viii


3. RESEARCH METHODOLOGY...........................................53
Background and Context of the Study.........................54
Research Design Overview................................... 54
Subject Population....................................... 56
Computer-Based Experimental Materials.......................57
Content and Task Parameters................................ 58
Content Level and Structure.......................... 60
Information Access Structures.............................. 61
Perspectives..........................................61
Similar Cases........................................ 63
Textbook............................................. 65
Summary.............................................. 66
Knowledge Application Tasks.................................66
Ordering Laboratory Tests............................ 66
Assess and Manage the Case........................... 67
Cross-Case Comparisons............................... 67
Evaluation of Learning: Test Cases......................... 68
Assessment................................................. 69
Individual Difference Testing........................ 69
Usage Questionnaire.................................. 71
On-line Data Collection of User-Computer interactions...... 71
Experimental design....................................... 72
Independent variables............................... 73
ix


Dependent variables.......................................74
Procedures...................................................... 75
Control Group............................................. 76
Experimental Group........................................ 74
General Research Hypothesis..................................... 78
4. RESULTS AND DISCUSSION ............................................. 81
Introduction.................................................... 81
Pre-treatment Group Differences............*.................... 82
Main Effects: Analysis of Variance and Covariance............... 85
Main Effects: Analysis of Variance........................ 85
Main Effects: Analysis of Covariance...................... 87
Summary: Main Effects..................................... 88
Interaction Effects: Analysis of Variance and Covariance........ 89
Summary......................................................... 94
Treatment on Test Case Scores............................. 95
Field Dependence/Independence on Test Case Scores.........96
Cognitive Constriction/Flexibility on Test Case Scores.... 97
Interaction Effect: Treatment/Control x
Field Dependence/Independence on Test Case Scores.........97
Interaction Effect:
Treatment/Control x Cognitive Constriction/Flexibility.... 98
Assumptions of ANOVA and ANCOVA.................................98
Correlational Data and Exploratory Hypothesis Testing........... 103
Correlational Analysis.......................................... 103
x


Exploratory Hypotheses: Usage Variables........................ 108
Main Effects and Interactions On the
Total Time Spent in the Practice Cases................... 110
Main Effects and Interactions on Perspectives Data....... 107
Main Effects and Interactions on Lab Test Data........... 117
Main Effects and Interactions on Textbook Data........... 121
Main Effects and Interactions on Information Access Sections. 129
Navigational Time in the Practice Cases.................. 130
Summary........................................................ 130
5. CONCLUSIONS AND RECOMMENDATIONS.................................... 134
Implications................................................... 134
Results summary.......................................... 135
Review of the inquiry.................................... 142
Implications for Theory........................................ 145
Cognitive Flexibility Theory......:.......................... 145
Cognitive Styles/Abilities and Practice Case Performance..... 150
Computer-Based Learning Environments
and Medical Problem Solving.............................. 153
Recommendations for Future Research............................ 154
Limitations.................................................... 156
References............................................................ 159
Appendix A......................................................... 167
Appendix A......................................................... 167
Appendix A......................................................... 167
xi


Appendix A................................................. 167
Appendix A................................................. 167
xu


FIGURES
Figure
3.1. Cell design for independent variables.................................55
3.2. Sample perspectives screen from Practice Case Five.................... 63
3.3. Sample similar case screen............................................ 64
3.4. Textbook index screen.................................................66
3.5. Predicted and possible main effects of independent variables.......... 74
4.1. Cell sizes for the three-way ANOVA: Trt/Ctl x FD/I x CC/F............. 83
4.2. Trt/Ctl x CC/F interaction with adjusted means........................ 93
4.3. FD/I x Trt/Ctl interaction graph with adjusted means.................. 94
4.4. Treatment x field dependence/independence on perspectives time data... 113
4.5. Treatment x field dependence/independence on perspectives access data.... 113
4.6. Interaction graph:
Cognitive constriction/flexibility x treatment on access data........ 123
4.7. Interaction graph:
Cognitive constriction/flexibility x treatment on textbook time data. 125
5.1. Summary of effects from hypothesis testing and exploratory analyses..... 141
xin


TABLES ,
Table
3.1. Demographic Data on Subject Population................................ 55
3.2. Perspectives Available in Practice Cases 1-7........................ 64
4.1. Independent Samples t Test Summary for
Group Assignment and Questionnaire Data............................. 83
4.2. Independent Samples t Test Summary for
Group Assignment and Individual Difference Scores................... 84
4.3. Summary Statistics from ANOVA Analysis.............................. 85
4.4. Three-Way ANOVA Summary Treatment x
Cognitive Constriction/Flexibility x Field Dependence/Independence.. 86
4.5. One-Way ANCOVA for Orientation Treatment............................ 88
4.6. Unadjusted and Adjusted Means for Orientation Treatment Variable.... 88
4.7. Two-Way ANOVA Treatment x Field Dependence/Independence............. 89
4.8. Two-Way ANOVA Treatment x Cognitive Constriction/Flexibility........ 89
4.9. Two-Way ANCOVA Trt/Ctl x Cognitive Constriction/Flexibility......... 90
4.10. Two-Way ANCOVA Trt/Ctl x Field Dependence/Independence.............. 91
4.11. Unadjusted and Adjusted Means for
Orientation Treatment x Cognitive Constriction/Flexibility.......... 91
4.12. Effect Sizes for Treatment/Control Groups............................ 95
4.13. Bartletts Test for Homogeneity of Variance.......................... 100
4.14. Three-Way ANOVA with Macintosh Experience as the Dependent Variable 101
4.15. Sample Descriptive Statistics From the Pearson Correlation Matrix... 105
4.16. Representative Correlation Coefficients for
Time and Numbers of Access in Different Program Segments............ 106
xiv


Tables
4.17. Dependent Variables List and Definition .............................. 109
4.18. One-Way ANOVA on Total Time
in Practice Cases Using Field Dependence/Independence................. 110
4.19. ANOVA Table: Orientation Treatment x
Field Dependence/Independence on Perspectives Time Data............... 112
4.20. ANOVA Table: Orientation Treatment x
Field Dependence/Independence on Perspectives Access Data............. 114
4.21. ANOVA Significance Data for
Treatment/Control x Field Dependence/Independence..................... 114
4.22. One-Way ANOVA: Cognitive
Constriction/Flexibility on Perspective Average ...................... 116
4.23. One-Way ANOVA Significance
Data for Cognitive Constriction/Flexibility and
Field Dependence/Independence on Lab Test Data....................... 117
4.24. Mean on Lab Test Data Using Cognitive
Constriction/Flexibility and Field Dependence/Independence............ 118
4.25. One-Way ANOVA: Orientation Treatment on Lab Tests Average............. 120
4.26. Significance Data for Orientation Treatment x
and Cognitive Constriction/Flexibility on Textbook Data............... 121
4.27. ANOVA Table: Orientation Treatment x
Cognitive Constriction/Flexibility on Textbook Count Data............. 122
4.28. ANOVA Table: Orientation Treatment x
Cognitive Constriction/Flexibility on Textbook Time Data.............. 124
4.29. Significance Data on One-Way ANOVAs for Textbook...................... 127
4.30. One-Way ANOVA: Field Dependence/Independence on Textbook Average 128
xv


CHAPTER 1
INTRODUCTION
In the study of acquiring competence and expertise in complex fields, many
authors have noted the qualitatively different skills which novices, competent
performers, and experts bring to problem solving tasks in different knowledge domains
(Chi, Glaser, & Farr, 1988). In fact, Patel and Groen (1991) insisted that the notion of
expert was only meaningful within a specific domain of expertise. In the area of
biomedicine, non-cognitive factors such as the structure of the knowledge domain,
interacted with instructional designs to play a significant role in how students acquire
and apply this complex knowledge (Patel & Groen, 1991; Feltovich, Spiro, &
Coulson, 1989). Managing the transition from novice level knowledge application to
competent problem solver has been an area of interest to many fields, including medical
education. The present study investigated the acquisition of advanced knowledge
structures by novices in a complex ill-structured environment by examining the
performance of medical students in a case-based, computer learning environment. This
learning environment was specifically designed to communicate the structural
characteristics of a medical knowledge domain to students while they interact with
information resources and solve simulated problems. This chapter provides a problem
statement, proposes a solution, define the inquiry, and provides an overview of the
research procedures.


Problem Statement
The field of biomedicine contains many areas where competent clinical problem
solvers are able to recognize and integrate domain specific contextual information and
basic science information and apply this knowledge to clinical tasks (Patel, Evans, &
Kaufman, 1990; Hobus, Schmidt, Boshuizen, & Patel, 1987; Groen & Patel, 1985).
Transfusion medicine is an excellent example of a biomedical area which includes
components from many disciplines. Hematology, biochemistry, immunology,
pediatrics, internal medicine, and pathology contribute to the knowledge base of
transfusion medicine. Getting students and medical professionals to apply their
academic knowledge of these disciplines to clinical situations where assessing risks to
donors and recipients of blood products is a goal of educators involved in transfusion
medicine.
The structure of knowledge used in transfusion medicine, combining
perspectives from several disciplines to solve specific clinical problems, has created
new problems for learners. These problems were also experienced by learners and
practitioners in other areas of biomedicine. Feltovich, Spiro, and Coulson (1989) cited
several causes of faulty understandings in medical students, including the problem of
transferring previously learned knowledge acquired in one context (e.g., academic
coursework in medical school) to new situations (e.g., clinical problem solving).
The shift from introductory learning, in this case, basic sciences approach,
towards learning to apply knowledge in clinical situations has been described as an
advanced knowledge acquisition task. Spiro, Feltovich, Coulson, and Anderson
2


(1988) defined the goals of advanced knowledge acquisition as follows: "the learner
must attain a deeper understanding of content material, reason with it, and apply it
flexibly in diverse contexts" (p. 376). Spiro and his colleagues indicated that ill-
structured domains such as biomedicine had content structures which were context
dependent, defied explanation by the use of single, simple analogies, could not be
reduced to a concise set of "top down" processing rules, could not be adequately
represented using prototypical examples, and/or resisted rigid compartmentalization into
knowledge components.
As medical students began to apply the knowledge acquired throughout college
and medical school, many tended to apply the learning strategies of memorization and
recall which were detrimental in advanced knowledge acquisition (Spiro et al., 1988).
This leads to serious misconceptions in the students understanding of complex
biomedical knowledge (Feltovich et al., 1989). There were several mutually
reinforcing processes which contributed to this kind of misunderstanding in
biomedicine or other complex and ill-structured domains (Spiro et al., 1988).
Feltovich, Coulson, Spiro, and Dawson-Saunders (in press) suggested that many areas
of biomedicine were both ill-structured and contained difficult, complex problems;
these problems characterized advanced (i.e., post introductory) learning. The
combination of ill-structured content domains (e.g., biomedicine) with the need for
instruction in advanced knowledge acquisition created particularly difficult and complex
problems for both students and designers of instruction. How would different kinds of
learners respond to complexity? If there were differences in performance on complex
tasks based on cognitive styles/abilities, how would these manifest in computer-based
3


instruction (CBI)? Also, is it possible to improve learner performance on complex
learning tasks by providing orientation to the tasks? These questions are addressed in
the next two sections.
Proposed Solution
The solution to the problems of learning complex application tasks in ill-
structured domains involved providing practice which integrated the complex structure
of the knowledge domain and combined the practice with information resources which
help to manage the complex material. Constructing/applying knowledge in multiple
examples or cases encouraged the development of more flexible representations of
content. Working in a modified case-based format, facilitated the representation of
medical knowledge. In order to implement these solutions, a computer-based learning
system was created in the area of risk assessment and other blood banking issues in
accordance with cognitive flexibility theory (Spiro et al., 1988) which prescribed
specific principles of learning and instruction for advanced knowledge acquisition.
This section will outline cognitive flexibility theory, provide an overview of the
implementation environment (computer-based hypertext), and conclude with a brief
discussion of this module compared to other computer-based learning programs.
Research Issues
In response to the problems of advanced knowledge acquisition in ill-structured
domains, Spiro et al., (1988) proposed cognitive flexibility theory. This theory sought
to avoid the many problems associated with the ill-structuredness of content and
advanced learning through including the following features in instruction:
4


1. avoidance of oversimplification by demonstrating and highlighting
complexities, irregularities, and component interactions in instruction;
2. creating multiple representations and analogies of the content area so that
learners have a diversified repertoire of ways of thinking about a
conceptual topic;
3. using cases as the conceptual unit of instruction to avoid "top down"
processing of information and overgeneralizing from a single set of
rules;
4. tailoring concepts to their application contexts to facilitate appropriate
use of abstract knowledge structures which do apply across some cases;
5. encouraging schema assembly, instead of retrieval of intact,
precompiled, or prototypical schemata, as a process of knowledge
construction; and
6. encouraging multiple interconnectedness of concepts and cases to foster
conceptions across cases and concepts, (pp. 3-6)
Utilizing the components of cognitive flexibility theory created learning environments
which retain the complexity, multidimensionality, and context specificity of ill-
structured knowledge domains. At the same time, learners were able to manage the
complexity of the information through access to structural or thematic knowledge
embedded in the instruction.
This study proposes that communicating the structure of areas of biomedical
knowledge to medical students while they engage in problem solving activities will aid
them in constructing flexible knowledge representations of those specific areas. This
flexibility is needed to adapt to the complex and ill-structured nature of biomedical
concepts and cases.
5


In other words, access to relevant structural information about the knowledge domain
while students are involved in problem solving activities will help them build flexible
interconnected knowledge structures appropriate for application in specific areas of
biomedicine.
Hypertext Learning Environments
Computer-based hypertext systems have provided the design flexibility needed
to instantiate the features in cognitive flexibility theory in other domains (Spiro et al.,
1988). Hypertext has been defined by various authors (Bush, 1945; Englebart, 1963)
as non-linear text consisting of nodes of information connected by links which can be
structured in a variety of ways. Nodes could be created of any size from a single word
or concept to an entire document. Links could have been organized to reflect many
different kinds of information access. These features made a hypertext format
particularly well suited to creating computer-based learning environments by
incorporating the features listed above (Spiro & Jehng, 1990).
The program investigated in this study used the node and link structure of
hypertext to simulate aspects of seven practice cases or problem scenarios, four
assessment cases, and information resources available to the learner during the practice
segment of the program. The case-based problem solving format of the practice and
test cases required the learners to order laboratory tests and choose appropriate
assessment and management options. The nature of the tasks required by the program
encouraged purposeful searching through the information resources provided in the
practice cases. The information resources, in turn, provided thematic access to
6


information organized around relevant perspectives of health professionals, a set of
abbreviated comparison cases, and a textbook of biomedical information concerning
blood donation, administration, and screening. Learners had continuous access to the
information resources throughout the practice cases. During the assessment portion,
only the textbook was available for consultation.
Some hypertext learning environments required a great deal of responsibility
from learners to use effective and efficient strategies to process information. The
cognitive overhead (Dede, 1988) associated with using a hypertext designed using
cognitive flexibility theory could have been extremely high. Very little has been
discovered about how students with little knowledge of hypertext or cognitive flexibility
theory would respond to the demands of creating flexible knowledge representations in a
complex learning environment when working independently. Several authors have
discussed the need to understand what patterns of use are the most effective in achieving
the higher level learning objectives of application and transfer of knowledge (Spiro, et
al., 1988; Gay & Mazur, 1991; Homey & Anderson-Inman, 1992).
Research on Computer-Based Instruction and Hypertext
With the growing popularity of CBI and its potential application to advanced
knowledge acquisition, there has been a need to investigate the way different learners
have attained and applied the complex information presented through this medium.
The issue of how learners used computer-based instruction has become more important
with of the growing popularity of hypertext based instruction. A greater degree of
7


learner control, needed to actively extract and/or construct various kinds of
information, was often a consequence of complex computer-based instruction.
Over the last 20 years, many studies have been conducted to investigate the
effectiveness of computer-based instruction. Recent meta-analyses conducted by Kulik
and Kulik (1987) and Roblyer, Castine, & King, 1988) suggested that many studies
have shown learning gains from the use of CBI, but the focus of the research has often
been on the scores the learners achieved on outcome measures and not on the behaviors
the learners engaged in while using the system. The ways in which learners used CBI,
especially systems utilizing hypertext design features, may have been influenced by the
way learners differ in their processing of information. Individual differences in
cognition and processing styles may have played an important role in how learners
interact with CBI.
As the content being taught using CBI shifted from well-structured to ill-
structured domains and the goals of CBI shifted from introductory to advanced
knowledge acquisition, the evaluation of learning outcomes also became more complex.
It became important to know both how well students performed on outcome measures
and how students performed when using complex CBI environments with hypertext
features. At the advanced knowledge acquisition level, applying appropriate
knowledge in context became the benchmark for successful learning, instead of simply
stating concepts or performing simple procedures.
8


Defining the Inquiry
This study examined the effectiveness of a hypertext-based system designed
using the principles of cognitive flexibility theory in the domain of transfusion
medicine. In this instructional context, this study explored two main research areas:
1. Do learners need instructional support to effectively use cognitive
flexibility hypertexts?
2. How do individual differences in processing styles affect:
a. the patterns of use of cognitive flexibility hypertexts?
b. the differential performance of learners on a complex problem
solving task?
c. the differential effect of instructional support on performance?
This study used a combination of qualitative and quantitative methodologies to
investigate these issues. Each of these are described below.
Research Methodology
The nonequivalent control group design model (Campbell & Stanley, 1963;
Cook & Campbell, 1979) provided a model to address questions relating to the
interaction of cognitive styles/abilities and orientation procedures on test case scores .
The sample was divided into two groups and a treatment was administered to one group
(an orientation to the programs features). This treatment/control difference was an
active independent variable in the study. Two instruments were administered to
measure cognitive ability/style differences: field dependence/independence and
cognitive constriction/flexibility. These passive independent variables have been linked
to traits which may facilitate the successful use of computer-based learning
environments, designed using cognitive flexibility theory. An interaction between the
cognitive style variables and the treatment/control variable was also expected.
9


The second part of the analysis used correlational data to explore relationships
between the independent variables listed above and usage variables extracted from a
computer record of subjects usage variables (audit trails) collected while they worked
through the practice cases. The correlations were used to discover associations
between cognitive styles/abilities, orientation procedures and the amount of time spent
on instructional screens and the number of visits to those screens. Once suitable
representations of time and visit numbers were found, the effects of the passive and
active independent variables on these measures of module use were investigated.
10


CHAPTER 2
REVIEW OF RELATED LITERATURE
This chapter presents a selective review of research and theory related to the
development and evaluation of the computer-based learning environment investigated in
this study. The chapter's structure parallels the argument put forth in Chapter One and
has four parts:
1. The problem statement:
An instructional need has been identified in medical education
relating to the acquisition of knowledge application skills in the
clinical training of medical school students, residents, and
practicing physicians in the area of transfusion medicine.
2. The proposed solution:
Create a computer-based learning environment specifically
designed to facilitate advanced knowledge acquisition using
cognitive flexibility theory with hypertext features.
3. The definition of inquiry:
Research on hypertext based systems designed for advanced
knowledge acquisition is limited at this time. Therefore a
combination of quantitative and qualitative methodologies are
employed to gain a greater understanding of learner-computer
interactions and their relationship to performance and cognitive
ability/style differences.
4. The summary.
Research and theory related to each of these topics are presented below.
11


Problem Definition
This section contains two parts: literature related to advanced knowledge
acquisition in biomedicine and a review of modeling content and problem solving
(diagnosis) using computer-based applications. The goal of the computer-based
module investigated in this study was to foster advanced knowledge acquisition in the
area of risk assessment to donors and recipients of blood products. Because this was a
clinical skill involving knowledge application, it could be classified as an advanced
knowledge acquisition task (Spiro et al., 1988). Transfusion medicine has included
several disciplines, such as, biochemistry, immunology, and physiology, and clinical
areas, such as, hematology, surgery, internal medicine, and pediatrics (Jonassen,
Ambruso, & Olesen, 1992). Hence, the knowledge domain was ill-structured, that is,
it did not have a clear set of principles to apply in diverse clinical situations. The first
part of this section deals with defining problems encountered in ill-structured domains
while acquiring advanced knowledge.
The second part of this section reviews projects undertaken to both understand
and train medical problem solving using computer-based reasoning systems. These
projects represented an effort to analyze the medical problem solving process and model
appropriate strategies for learners. This part introduces the role of computers as a
potential tool for improving medical education and highlights both the promises and
drawbacks different approaches have for representing and solving problems in the field
of biomedicine. Of particular note was the difficulty of representing medical expertise
in a content area with either production rule formalisms or methods of modeling
decision-making.
12


Advanced Knowledge Acquisition in Biomedicine
Education for health professionals has encompassed years of basic academic
sciences and practice-based clinical knowledge. Making connections between these
two bodies of knowledge has proven to be problematic in a number of areas in
biomedicine because of the unique structure of biomedical knowledge (Balia, 1980;
Patel, Groen, & Scott, 1988; Feltovich et al., 1989). Clinical problem solving was not
considered solely dependent on recall of facts pertinent to a particular case or disease
(Patel et al., 1990; Norman, Tugwell, Feightner, Muzzin, & Jacoby, 1985). Instead,
students organization of knowledge proved to be more highly correlated with proper
diagnoses than measures of basic sciences knowledge (Balia et al., 1990). Patel et al.
(1990) stated that students with more clinical experience developed clinical models for
diagnosis which were confirmed or amended by relevant basic science knowledge.
Before basic science information could be utilized effectively in a clinical situations, a
classification structure of disease categories had to be present. Creating an environment
which facilitated the development of knowledge structures in biomedicine could have
enabled medical students to connect more basic sciences information to their clinical
problem solving tasks.
One kind of knowledge structure particular to biomedicine has been described
as the illness script. Schmidt, Norman, and Boshuizen (1990) suggested a stage
theory of clinical reasoning which involves the creation of illness scripts, a qualitatively
different representation of how diseases manifest. The clinical knowledge in an illness
script is had three parts: the enabling conditions which include contextual information
and patient characteristics; the faults or type of problem in the body; and consequences,
13


which include complications resulting from the fault Contextual information,
including risk factors associated with gender and age, have played an integral role in
clinical problem solving tasks (Hobus et al., 1987).
Schmidt and Boshuizen (in press) suggested that a process called encapsulation,
which included the formation of higher-order concepts (i.e., new knowledge
structures), is facilitated by experience in practical settings, such as, clinical rotations or
clerkships. At the same time as the encapsulation of illness scripts was occurring, there
was a relative decline in the use of overt biomedical knowledge learned during the first
two years in medical school (Custers, Boshuizen, & Schmidt, 1992). It was the use of
contextual case specific information combined with the encapsulation process which
characterized increasing medical expertise; this was reflected in more refined illness
scripts. In designing instructional/leaming environments for medical education,
making contextual information readily available facilitated the creation of appropriate
knowledge structures (e.g., illness scripts).
Because the transition in information processing (as well as knowledge
structuring) from basic science information to clinical problem solving involved the
complex processes described above, students had problems developing correct and
useful knowledge representations (e.g., illness scripts) in many areas of medical
education. Feltovich et al., (1989) discussed three caused of faulty understandings in
medical students and practitioners: multiplicity, interdependency, and
oversimplification. Multiplicity involved the source of misconceptions or faulty
understandings. These sources can include any mistaken preconceptions the learner
might have, the educational process which did not always prepare students for
14


complexity, or the ill-structuredness of the content area. Interdependency was defined
as the tendency for systems or networks of faulty reasoning to reinforce each other and
support an overall misconception. Oversimplification was seen as the tendency to
reduce complex information to simpler concepts in order to facilitate the
teaching/leaming process. Reductive biases (Coulson, Feltovich, & Spiro, 1989)
resulted from the interaction of the factors listed above and often appear in the way new
complex material was presented to learners. The ability to teach and learn content
specificity was significantly hampered when reductive biases are present in instruction.
Reductive biases tended to occur when the intended learning outcomes focused
more on advanced knowledge acquisition and/or when the knowledge domains being
taught were ill-structured. Advanced knowledge acquisition has been contrasted with
introductory learning in a number of ways. Introductory instruction in a subject area
tended to stress learning goals which were more superficial in nature and oriented
toward reproductive memory tasks and imitative rule following (Spiro & Jehng, 1990).
However, as learning moved toward more advanced knowledge acquisition, mastery of
the complexity of the material was required and learning outcomes shifted to higher
levels, such as application, evaluation, and transfer.
Unfortunately, teaching methods which were effective at an introductory level
(basic sciences approach) often led to oversimplification of the material and were
unsuitable and even detrimental to success at advanced knowledge acquisition (Spiro et
al., 1989). Learners sometimes regressed to habits of thought which were effective for
simplified introductory instruction, but hampered their ability to learn more advanced
complex material. The same was true for assessment or evaluation of learning.
15


As learning moved from introductory to advanced learning, appropriate assessment
tasks shifted towards instruments which reflected the higher level outcomes of
application and transfer, instead of recall or simple rule application (Feltovich et al.,
1989). Hence, reductive biases occurred as a result of both inappropriate,
oversimplified presentation and assessment of advanced material as well as
inappropriate processing strategies used by learners.
Teaching and learning complex concepts in ill-structured knowledge domains
also created an environment where reductive biases can occur. The ill-structured
nature of biomedical knowledge required a change in the way new experiences are
indexed in terms of prior knowledge (Clancey, 1989). This change in processing or
encoding new experiences created special problems for medical education. Feltovich et
al., (1989) listed unusual demands which ill-structured domains place on teaching and
learning. First, demands on working memory were greater in complex learning. Often
several sets of nested steps and/or variables needed to be managed simultaneously in
order to fully understand complex processes. Second, the ill-structuredness of the
content made formal representations more complex and abstract, which moved them
further from their real world referents. Third, surface similarities between prior
knowledge or intuitive analogies and the current knowledge domain could have
obscured the underlying complexity of the domain and made acquisition of the
complexity more difficult. Finally, contextual meaning, exceptions to rules, grey areas,
and counterintuitive associations became more common in knowledge domains that
were ill-structured. The sensitivity to context of concept and rule application across the
knowledge domain made ill-structured knowledge localized.
16


When these concepts interacted and reciprocated with other localized concepts these
became interactions between families of related concepts (Feltovich et al., 1989, p.
117). Contextual meaning was a parallel phenomenon to content specificity. All of
these demands interacted and created an understanding of the knowledge domain.
When learners successfully balanced these demands and avoid the
counterproductive elements, they created flexible, interactive schemata to represent the
knowledge domain. However, when one or more of the demands of ill-structured
domains were not met, often the result was ...a misconception [with] a set of
component misconceptions which interact in mutually supporting ways (Coulson et
al., 1989, p. 109). The problems associated with advanced knowledge acquisition and
ill-structured domains interacted to increase the difficulties learners experienced when
tackling this kind of material. Biomedicine has contained numerous examples of
complex and ill-structured content domains (Feltovich et al., 1989).
In summary, the way knowledge was encoded influenced the ways it was
accessed when solving problems in clinical medicine. The creation of appropriate
knowledge structures has been described as facilitating the ability to recall and apply
relevant basic science information. When complex and ill-structured knowledge was
oversimplified, overgeneralized, and/or misrepresented as regular rule-based domains,
the resulting reductive biases inhibited students from linking their basic sciences
information to clinical problem solving tasks.
17


In order to design instructional and learning environments to address the need for
acquiring advanced knowledge in clinical areas, it was necessary to investigate the
ways that medical knowledge can be represented using computer-based methods. The
following section addresses this issue.
Modeling Content and Problem Solving in Biomedicine
Many attempts have been made to represent problems and problem solving in
biomedicine using a variety of methodologies. Elstein, Holzman, and Ravitch (1986)
demonstrated the difficulty of applying rule-based formalisms to biomedical problems.
In their study, each of fifty physicians were sent a questionnaire which asked her/him
to recommend treatment for twelve cases involving estrogen replacement therapy.
Additional information was collected through interviews to create a normative decision-
making model based on each physician's assessment of probabilities, risk preferences,
and importance weights concerning this problem area. The decision-making model
recommended a clear indication of treatment in 40% of the cases, the rest (60%) were
seen as ambiguous. The practicing physicians recommended withholding treatment
more than 50% of the time. The discrepancy between the two judgements
demonstrated the difficulty of applying axiomatic solutions to ill-structured complex
knowledge domains. Hershey and Baron (1987) reflected the same concerns in their
study of clinical reasoning and decision theory. They examined expected utility theory,
based on Bayesian probability theory, in terms of its utility as a descriptive, normative,
and prescriptive theory. They found that the formalisms and axioms of the decision
analytic model did not adequately describe physician behavior.
18


They suggested that new normative models be developed which can accommodate
inconsistencies in physicians' choices. These inconsistencies were examples of ill-
structuredness and the difficulty of developing or imitating expertise in biomedicine.
Other researchers have modeled aspects of biomedicine using computer-based
materials and techniques. Chin and Cooper (1989) created an instructional tool in the
area of cardiovascular diseases which generated simulated patient cases for students to
diagnose. They converted a knowledge base of cardiovascular diseases into an
intelligent tutoring system using a probabilistic format. The focus of the project was to
teach students to associate manifestations with diseases. The knowledge base
generated cases where the patient had one and only one disease and provided
information sufficient to solve the case. The student could have asked for more
information (e.g., more lab work, additional symptoms, etc.), ask for a hint about
which symptom was the strongest indicator, or go to the next case. Based on a student
responses to previously completed cases, the system created a student model based on
the diseases the student had difficulty identifying. This model was successful in
helping students associate cardiovascular diseases with symptoms, however, this
limited domain did not include strategies for implementing diagnoses. The design of
the program also eliminated a substantial amount of the complexity concerning multiple
diseases and ambiguous symptoms. This regularization of the knowledge domain
made the program more efficient, but limited its transferability to actual cases with more
ill-structured symptoms and causes.
19


Another way to model medical expertise was described by Koton (1988).
Using artificial intelligence techniques, CASEY, a medical reasoning program, was
developed which used previously solved cases and causal modeling to solve new cases.
In this way, parts of the reasoning process which practicing physicians use (recalling
similar cases) were combined with principles for reasoning about evidence in causal
explanations and incorporated into the computer model. This reasoning model
demonstrated both experience-based knowledge representations and rule based
reasoning structures to more accurately portray the knowledge domain. CASEY was
limited by the fact that some cases tended to generate numerous solutions which limited
its usefulness as a learning tool.
The GUIDON/NEOMYCIN projects (Clancey, 1985) used rule based
formalisms to teach diagnosis and therapy recommendations. Among other things, the
system diagnosed simulated or actual patients and tutored students about the rules or
hueristics used by the system to make the diagnoses. The heuristics used by the system
were strictly rule based and did not include information about the structure or strategy
of medical diagnosis, which limited its teaching potential (Chin & Cooper, 1989).
An alternative to using rale-based formalisms or Bayesian probability theory
was investigated by Schwartz (1989) who used expert performance in real or simulated
cases as the basis for evaluating clinical practice. He documented faculty and student
responses to four practice and two assessment cases in pediatrics using a computer-
based simulation. In the project, five faculty members independently created diagnoses
for the cases using the CAMPS software. This set of responses was organized into
seven subscores. Student responses to the same cases were recorded, summarized,
20


and compared to class and faculty averages. The students were also evaluated on their
problem solving ability by their instructors. Discrepancies between the faculty and
student simulation scores guided instructors to help students in specific areas of
diagnostic skills.
In an earlier study using CAMPS software, Diserens, Schwartz, Guenin, and
Taylor (1986) compared the scores of three groups of users: residents, third year
medical students, and faculty members. Differences between the faculty group and the
other groups appeared only on the most difficult case. The more advanced knowledge
and problem solving capabilities of faculty discriminated them from the less
experienced groups. Norman, Trott, and Brooks (1992) produced similar results,
using think aloud protocols. Specifically, they found that most experienced
practitioners selected and attended to a smaller number of more critical data than medical
students or residents. They attributed this superior performance to the chunking of
data around physiological principles. These studies also demonstrated the usefulness
of process data as a way to measure specific aspects of problem solving.
In summary, formal rule-based systems and other decision-making models have
advanced the understanding of the clinical reasoning process. Case-based
representation of the knowledge domain, employed by many projects, facilitated the
development of knowledge structures in students which resemble the simulated aspects
of the clinical environment they will be working in. The ability of some systems to
model problem solving and evaluate student models has made these programs more
effective. Yet, there have been drawbacks to this approach. In order to generate
meaningful decisions from many computer-based reasoning systems, the knowledge
21


domain may have been either limited in scope or oversimplified. The problems
associated with reductive biases which infiltrate this approach have been outlined in the
previous section. While many of the programs have modeled competent or expert
problem solving performance, the knowledge structures used for reasoning must be
programmed into the systems. Hence, in order to assist students in integrating basic
science information into clinical problem solving tasks, learning environments needed
to be more effective in facilitating the acquisition of clinical models (Patel et al., 1990).
Improvement has been noted when the learning environment has retained the
complexity of the knowledge domain and included contextual information, such as,
enabling conditions, in a case-based format (Schmidt, Norman, & Boshuizen, 1990).
The following sections outline the theoretical context and software environment of a
learning environment designed to meet the needs of complex knowledge application
tasks in ill-structured environments.
Proposed Solution: Facilitating Schema Assembly
Using Cognitive Flexibility Theory and Hypertext
This section addresses the problems involved in linking and applying basic
science information in clinical problem solving tasks and contains two parts. The first
part discusses schema theory and knowledge acquisition integrating examples from the
previous sections. It is argued that learning flexible schema assembly was best
facilitated by providing a case-based practice environment which integrated thematic
information resources. The second part outlines cognitive flexibility theory (Spiro et
al., 1988) providing design guidelines for thematic access instruction (which facilitates
22


flexible schema assembly) and discusses the implementation of cognitive flexibility
theory in hypertext software environments. When cognitive flexibility theory was
instantiated in a computer-based hypertext, the resulting learning environment provided
opportunities to acquire and apply flexible schemata.
Integrating Knowledge Structures: Schema Theory
Throughout the preceding discussion, researchers described the combination of
context specific content knowledge and related problem solving strategies variously as,
hueristics (Clancey, 1985), data chunking (Norman et al., 1992), illness scripts
(Schmidt, Norman, & Boshuizen, 1990), case-based reasoning (Koton, 1988),
knowledge taxonomies (Clancey, 1989), and clinical models (Patel et al., 1990). All of
these terms described knowledge representations specific to biomedicine. Schemata
(Rumelhart, 1980; Rumelhart & Ortony, 1977) incorporated many aspects of these
descriptions. The major features of schemata were listed by Rumelhart and Ortony
(1977) as:
1. Schemata have variables.
2. Schemata can be imbedded in one another.
3. Schemata represent knowledge at all levels of abstraction.
4. Schemata represent knowledge rather than definitions.
To which Rumelhart (1980) added:
5. Schemata are active processes.
6. Schemata are recognition devices whose processing is aimed at the evaluation
of their goodness of fit to the data being processed.
23


Schemata could have been activated by conceptual-driven or data-driven control
structures. Conceptual-driven processing could be seen as analogous to top-down,
or whole-to-part processing, where a more abstract schema activated their subschemata.
Data-driven processing reflected bottom-up, or part-to-whole processing, where
contextual information activated more abstract schemata. These processes could have
happened simultaneously throughout most thinking and reasoning processes, including
the medical reasoning processes described above. Learning, according to this
perspective, consisted of accretion of information into existing schemata, tuning or
modifying schemata, and restructuring, which includes pattern-generated as well as
newly generated schemata.
Many parallels have existed between aspects of schema theory and the work of
researchers creating and evaluating medical CBI programs. The research described by
Koton (1988) using the CASEY program illustrated both the use of schemata, in the
form of cases, and data-driven control structures. In this model, the data-driven control
structures were represented by the case-based reasoning, where the program searched
through case inventory for similar cases. Upon finding one and matching up important
features, the program made simple adaptations to tailor the case recalled to the new
information (referred to by Rumelhart, 1980, as tuning), retrieved the causal model
which explained that case, and applied it to the new case. When there was not a close
enough match between the present case and the pre-defined cases, the program
activated the causal model to produce a solution based on the case with the most
24


similarity (patterned generation restructuring or tuning) and stored the case for future
use. The GUIDON/ NEOMYCIN projects (Clancey, 1985) modeled conceptual-driven
control structures in the hueristics they used and taught to students.
The most direct application of schema theory using computer modeling in
biomedicine was the MEDIC program described by Turner (1989,1988). Using the
domain of pulmonology, Turner developed a diagnostic reasoner, which was always
under the control of a schema-like reasoning system. The program contained global
schemata which managed significant portions of the overall consultation (i.e., gathering
information, performing a consultation, etc.). Local schemata were activated by the
global schemata to solve more specific goals, which may not have been directly
involved in the consultation schema. The specialization hierarchy of schemata, where
global schemata organize or index other more local schemata, used a conceptual-driven
control structure. Another way to organize schemata described by Turner was by
context sensitivity. Here, diagnostic memory organization packets (dxMOPs)
contained contextual information about a condition (e.g., expectations about the
problem, typical patient characteristics, common findings, etc.) and information linking
goals which may in turn activate other schemata using a data-driven control structure.
Another key part of this program was its flexibility of applying schemata. Schemata
did not have to be completed in order and they could also have invoked other schemata
based on information and conclusions gained from previous schemata used in the
consultation.
25


Although the MEDIC system accurately modeled the application of schemata to
medical diagnosis, the program did not address how schemata are formed. All of the
initial biomedical knowledge in the MEDIC system came from the human expert who
assembled the schemata for the knowledge domain. The system could have reproduced
and modified these schemata, what Rumelhart (1980) called patterned generation
restructuring, but creating new, completely different schemata not already in the system
(schema induction restructuring) was not a part of the model. Schema induction
involved ability to note the recurrence of configurations of schemata that do not, at the
time they occur, match any existing schemata (Rumelhart, 1980, p. 54) and generate
new schemata based on the new knowledge configurations. Rumelhart acknowledged
that schema induction was not a natural part of a schema based system (1980, p. 54).
Although many computer simulations could have successfully modeled schema
application to complex performance, they often lacked an appropriate model for
generating new schemata in response to unfamiliar complex problems. Even though
computer programs have modeled some complex performances well, the extent to
which computer schema restructuring has meaningful parallels in human learning
processes has remained questionable.
Restructuring, or the acquisition of new schemata, represented a qualitatively
different kind of learning from accretion or tuning. Vosniadou and Brewer (1987)
suggested that restructuring may be domain specific, with different types of
restructuring occurring in different contexts. The authors suggested differentiating two
kinds of restructuring: weak and radical. Weak restructuring was often characterized
26


by an increase in the numbers of abstract relational schemata and more elaborated
conceptual networks. Radical restructuring emphasized a reconceptualization of the
domain in a broader, more comprehensive theoretical structure. Both weak and radical
restructuring could have taken place during the development of expertise.
Schmidt et al. (1990) addressed the restructuring process in the field of
biomedicine in the context of illness scripts. They hypothesized a four stage process in
the development of illness scripts. The first two stages of this process resembled weak
restructuring. In these stages causal networks of factual medical knowledge became
more elaborated and compiled into higher level causal models (which explained signs
and symptoms under diagnostic labels). The final two stages involved radical
restructuring with the emergence of illness scripts and the construction of instance
scripts which represented individual patient encounters. Custers et al. (1992) also
addressed the issue of different kinds of restructuring in their discussion of the
encapsulation process. During the pre-clinical phase of learning, medical students
depended and elaborated on overt medical knowledge using weak restructuring. The
encapsulation process marked a shift towards radical restructuring with the emergence
of new qualitatively different categories (e.g., enabling conditions, faults, and
consequences, discussed above) which reshaped the understanding of patients and
illness. The illness script was an induced schema, a new category of meaning. One of
the key processes which occurred when encapsulated scripts were activated was the
integration of contextual information into the analysis of biomedical problems.
27


Probably the most systemic example of weak versus radical restructuring
occurred in physician training. Typically, preclinical training in the first two years of
medical school was usually an extension of the basic sciences approach to biomedicine.
Much of this learning was made up of accretion of more factual information into
existing schemata, tuning, and weak restructuring. In the clinical training phase of
medical education, students were called on to apply their basic sciences knowledge to
much more complex and ill-structured problems, including clinical cases. Integrating
information from different instructional contexts to create useful problem
representations and solutions required the students' knowledge structures to undergo
radical restructuring. However, many medical students did not develop the global or
strategic schemata (Turner, 1989) necessary for radical restructuring to take place.
Instead they sometimes applied strategies used during their basic sciences training
which introduced many problems including reductive biases (Spiro et al., 1988) into
their thinking.
In summary, both conceptual-driven and data-driven control structures needed
be engaged in order to encourage radical restructuring. Schemata formed through
radical restructuring included illness scripts (Schmidt et al., 1990; Custers et al., 1992),
clinical models (Patel et al., 1990), and indexing taxonomies (Clancey, 1989). The ill-
structured nature of clinical problem solving made this restructuring more difficult,
especially when students engaged in learning strategies from basic science acquisition.
28


An intended outcome of cognitive flexibility theory was facilitating radical restructuring
which enhances flexible schema assembly and adaptability to the ill-structuredness of
biomedicine. The next section outlines cognitive flexibility theory and explains the
principles of thematic access instruction.
Cognitive Flexibility Theory
Many of the computer-based models of expert performance relied on retrieval of
intact schemata, whereas human experts have tended to adapt or construct schemata in
response to contextual information provided by the situation (data-driven control
structures) as well as by accessing the deep structure (Feltovich et al., 1989) of the
domain for information (using conceptual-driven control structures). One strategy for
increasing the effectiveness of instruction in biomedicine was to base the instruction
using a design informed by a particular theory of cognition and learning. Cognitive
flexibility theory (Spiro et al., 1988) was one approach which provided a framework
for representing complex and ill-structured knowledge domains and focused on
advanced knowledge acquisition. Cognitive flexibility theory addressed the difficulties
inherent in biomedicine (and other complex domains) through a variety of strategies.
Avoidance of oversimplification and overregularization. This major theme was
integrated into cognitive flexibility theory and colored other related strategies.
Special measures were taken to highlight exceptions and demonstrate complexities,
regularities, and intricate patterns of conceptual combination in ill-structured knowledge
domains. Introductory learning strategies, such as, decomposition and additive
reassembly, have created inadequate models for component interactions.
29


In biomedicine, content specificity (Schmidt et al., 1990), the intertwining of
biomedical knowledge with context specific problem solving hueristics, was an
example of an knowledge structure which required diverse, holistic presentation in
order to be acquired.
Multiple representations of content. In ill-structured domains, it stood to reason
that no single schema, conception, example, or strategy could have adequately
represented the unique facets of the domain. Multiple representations of phenomena
more accurately reflected the multifaceted nature of ill-structured domains. Working
with dissonant or conflicting models forced the learner or performer to construct a
representation which fit the constraints of the problem, case, or example. Another way
to create a multiple representations was to view the same problem from a number of
different perspectives. Hence, viewing a concept over multiple contexts or cases and
examining a case or problem from a number of conceptual viewpoints were both
instances of multiple representation (Spiro & Jehng, 1990).
Schema assembly: From rigidity to flexibility. Related to the previous point,
Spiro and his colleagues stated that in ill-structured domains, knowledge must be
constructed, not merely retrieved. The notion of rigid pre-compiled schema, needed to
be replaced with flexible, knowledge structures which could recombine parts of various
schemata to suit individual contexts (Spiro et al., 1988). There were still many smaller
portions of pre-compiled knowledge which could be applied intact, but as domains
exhibited more ill-structuredness, the number and size of pre-compiled knowledge
structures which could have been applied intact, decreased.
30


Centrality of cases. As Rumelhart (1980) stated, control structures could have
been either conceptual-driven or data-driven. In well-structured domains, conceptually
driven top down strategies were more likely to be effective. However, in ill-
structured domains, abstract general principles provided poor guidance for knowledge
application. Using data-driven control structures, such as, case-based reasoning could
have increased the sensitivity to interactions between contextual variables within and
between cases.
Conceptual knowledge as knowledge in use. One of the distinguishing features
of advanced knowledge acquisition was the ability to integrate prior knowledge and
context specific information into a newly generated schemata. In biomedical education,
the basic science concepts, often acquired through decontextualized classroom
instruction, were often difficult to apply in a clinical context where the presenting
features (the instantiation of basic science concepts) varied greatly within and between
clinical cases. By giving students access to the full range of uses of basic science
concepts across a set of cases with dissimilar surface features, the concepts were
intimately connected with their patterns of use. Additionally, the similarity between the
case-based instruction and the problem solving environment the medical students
worked in, may have improved the acquisition of context sensitive application of
knowledge (Brown, Collins, & Duguid, 1989).
Noncompartmentalization of concepts and cases: Multiple interconnectedness.
As a corollary to multiple representations of content, this theme of cognitive flexibility
stressed the interconnectedness of seemingly disparate concepts by providing multiple
links or perspectives which revealed more of the deep structure of the knowledge
31


domain (Feltovich et al., 1989) and the content specific nature of problem solving
strategies. The ability to construct alternate paths to navigate the knowledge domain
may have fostered the development of flexible situation adaptive schema assembly
(Spiro, et al., 1988).
Meaningful learner interactions and support for the management of complexity.
As the theme of knowledge in use implied, it was necessary to have active learner
participation and knowledge construction, including exploration of the complexity of
the knowledge domain and development of rich, practice-based associative networks
(Onorato, 1990) in order to create adaptive content specific schemata. To facilitate this
development, the domain's complexity needed to be managed so that the complexity
was preserved, while the cognitive support was available to scaffold learners when
encoding the knowledge structures. Examples of cognitive support were strategic
guidance, thematic access to cases, and appropriate feedback integrated into the learning
environment.
Thematic access instruction/leaming. The main metaphor used by Spiro and
Jehng (1990) for learning and instruction in ill-structured domains was criss-crossing
the conceptual landscape (p. 169). This metaphor, borrowed from Wittgenstein, was
extended to include a general theory of learning which stresses supplying the learners
with materials which facilitated the development of flexible multidimensional
knowledge structures based on integrating themes. Instruction, according to this
theory, included providing guidance and commentary to the learner which encouraged
multidimensional exploration. The result of thematic criss-crossing was to further
understand the content domain and to demonstrate to the students the complex nature
32


of literary comprehension and to help students to begin to build a more adequate
repertoire of cognitive skills for the processing of complexity and for the
application/transfer of complex knowledge to new situations (p. 173). With flexible
knowledge structures acquired through use of cognitive flexibility hypertexts, learners
had the proper experience to shift from intact schema retrieval to flexible schema
assembly.
In summary, cognitive flexibility theory integrated solutions to many problems
in the acquisition of advanced knowledge in ill-structured domains. By utilizing a
modified case-based approach and providing thematic structures for information access,
a learning environment was created which encouraged the development and application
of flexible schema. Embedding thematic information resources, in a diverse set of
simulated cases, facilitated purposeful information searching and relevant problem
solving activities. The process of examining multiple cases with similar constellations
of themes activated data-driven control structures. Understanding complex concepts or
themes by their instantiations in cases promoted conceptual-driven control structures.
Schemata appropriate for clinical problem solving, such as, illness scripts (Schmidt et
al., 1990), clinical models (Patel et al., 1990), or knowledge indices (Clancey, 1989),
accessed both types of control structures flexibly to construct a representation of the
problem and solution. Spiro, et al.(1988) hypothesized that thematically organized
information resources in a practice-based environment facilitated the development of
flexible schema. The next section discusses hypertext, a software environment flexible
enough to instantiate many of the design principles of cognitive flexibility theory.
33


Hypertext. Cognitive Flexibility Theory.
and the Network of Ideas (Organizational Structure)
cognitive flexibility. The firs
discusses the advantages and
This section discusses hypertext, a software environment based on a node and
link structure, as a platform for creating instruction modeled after principles of
: part provides a more complete definition of hypertext and
drawbacks of hypertext for learning. The second part
explains the relationship between the principles of cognitive flexibility theory and the
network of ideas present in hypertexts. It is argued that hypertext-like access to
information can support many of the principles of cognitive flexibility.
|
Hypertext: A definition. Hypertext has been defined as a generic term for
organizing and accessing knowledge using a node and link or network structure
(Jonassen & Wang, 1990). Nodes, chunks of text, graphics, video, or other
information were connected by a set of links, which defined an associative connection
between nodes. Theoretically, there could have been no limit to the number of links or
nodes a hypertext system could have. Hypertext authors used nodes and links to create
a network of ideas or organizational structure: a knowledge representation. The
network of ideas has been organized to facilitate many different kinds of interactions in
CBI. Users have interacted with the knowledge representation by traversing through
a set of nodes and links. Authors have also predetermined a path of nodes and links
(e.g., a guided tour) arranged to expose the user to certain features or ideas in the
knowledge representation.


One of the great strengths of using hypertext for knowledge representation was
its flexibility. Jonassen and Grabinger (1990) listed five kinds of information
structures which can be modelled in hypertext:
Semantic structures reflect the knowledge of the author or expert.
Conceptual structures include predetermined content relationships,
such as, taxonomies.
Task-related structures are those that resemble or facilitate the
completion of a task.
Knowledge-related structures are those that are based upon the
knowledge of the expert of the learner.
Problem-related structures simulate problems of decision
making, (p. 12)
Many hypertexts contained a mixture of these elements. These structures supported a
range of learning from recall and replication to generative learning strategies (Jonassen,
1986). Depending on the functionality of the hypertext system, users were empowered
to go beyond the authors conception of the knowledge domain. When users created
their own paths, they had the opportunity to construct and/or convey their own
knowledge or problem representation based on their experience of the hypertext
(Jonassen & Wang, 1990, p. 158).
Some of the advantages of hypertext also implied potential problems for its use.
In general, users were not accustomed to navigating through a hypertext to obtain
information. In complex hypertexts, users sometimes became disoriented or frustrated,
particularly if the user interface was poorly designed. To facilitate certain kinds of
learning when creating paths in hypertext, the user had to continually process
information received and use that information, in combination with prior knowledge, to
35


choose the next link and node to address. Integrating and synthesizing information, a
challenging task in any learning environment, could have been especially difficult when
hypertexts had not provided suitable support for learners. The cognitive overhead
(Jonassen & Grabinger, 1990) associated with these problems could have drawn away
resources needed for other cognitive activities required for learning. The potential
difficulty of relating knowledge acquired through hypertext to prior knowledge
structures was another issue which needed to be addressed by hypertext designers.
Hypertext and cognitive flexibility theory: Structuring the network of ideas.
Hypertext has had the potential of supporting many of the themes of cognitive
flexibility theory. Spiro and Jehng (1990) suggested that specialized hypertexts can be
developed with a network of ideas which reflect the structural characteristics of the
knowledge representations. The KANE program instantiated many aspects of cognitive
flexibility theory using computer-based interactive video instruction. After viewing a
film segment, the user had the option of accessing several kinds of information about
the film's semantic structure. The program included access to as many as ten literary
themes and symbolic perspectives. Users could also have selected contextual support,
thematic commentary, and guidance. Students could be led through the program
sequentially, taken to specific film segments to demonstrate an instance of a theme, or
allowed to operate it in self-directed mode. When students used the self-directed mode,
they had the opportunity to generate a personal combination of film segments (a
knowledge representation) which expressed their understanding of an aspect of the
film. Other tools for learner generated exploration of the program were also provided.
36


Cognitive flexibility theory, in turn, provided a cohesive theory of advanced
knowledge acquisition in ill-structured domains which added unique and powerful
access structures to the many networks of ideas contained in a hypertext environment
(Spiro & Jehng, 1990). Representing structural knowledge and dynamic
interrelationships (stressed in cognitive flexibility theory), channeled the functionality
of hypertext towards meeting the demands of advanced knowledge acquisition. The
contextual support, thematic commentary, and guidance features included in the KANE
scaffolded potential navigation and orientation problems. The multi-dimensional and
web-like structure created by thematic access to case segments made knowledge
construction, internal integration, and synthesis more likely to occur.
Adapting hypertext designs to model aspects of cognitive flexibility theory
allowed for unique access to multiple representations of knowledge. Thematic access
structures and reliance on a modified case-based approach allowed users to engage in
situation specific schema assembly; this was essential for advanced knowledge
acquisition. Spiro, Feltovich, Jacobson, and Coulson (1991) noted that the learner
interaction encouraged by the program design is a process of learning, not a single
event. Cognitive flexibility hypertexts provided practice in situation specific
knowledge assembly and in recognition of common patterns; both were crucial to
advanced knowledge acquisition.
In summary, a proposed solution to facilitating the acquisition and application
of advanced knowledge was using hypertext as a platform to instantiate cognitive
flexibility. Jacobson (1991) described a cognitive flexibility hypertext organized
around six social impact of technology cases. As part of the experimental design of
37


the study, subjects were guided through case segments designed to highlight the
semantic and structural relationships between the themes and cases. Subjects exposed
to this treatment outperformed two drill and practice treatments on a problem solving
essay, even though the other groups outperformed the thematic group in recall tasks.
Creating in biomedicine the kind of learning environment Jacobson described could
facilitate the transition from basic sciences approaches to knowledge representation to
more clinically based, context-sensitive knowledge structures in medical students and
residents. Because flexible schema assembly was seen as a process, and not a discrete
chunk of information, evaluating learning outcomes involved understanding how
learners interacted with the information access structures while performing problem
solving tasks in simulated clinical cases. The next section outlines the approach taken
to capture aspects the complex program-learner interaction and its relationship to
performance in a specific medical knowledge domain, transfusion medicine.
Defining the Inquiry
The research reported in this study was supported by a Transfusion Medicine
Grant from the National Institutes of Health. One of the goals of the grant was the
development and production of computer-based instructional materials to be used in
medical school curricula across the country. The design team's first completed module
was a computer-based learning environment (in a modified hypertext format) utilizing
principles of cognitive flexibility theory in the area of donor/recipient risk assessment
38


and other blood banking issues. The module's intended population included advanced
medical students, interns, residents, and other health professionals involved in handling
blood and blood products.
This investigation focused on one population, medical students finishing their
second year and preparing for clinical rotations at a large New England medical center
and their performance in a series of practice and test cases. In particular, it was
hypothesized that the cognitive abilities/styles of these learners would influence their
behavior in the practice cases and their scores on the test cases. This section discusses
the cognitive ability/styles selected for this study and the methodological approach taken
towards process data in this inquiry.
Individual Cognitive Stvles/Abilities and Flexible Schema Assembly
The knowledge restructuring required by flexible schema assembly was an
ability partially influenced by an individual's cognitive processing styles/abilities.
Cognitive flexibility theory sought to promote the ability to integrate information from a
number of perspectives and assemble a flexible schema of the problem and solution; an
information structuring task. Ofter when schema assembly occurred, the learner
recognized the underlying or deep structure (Feltovich et al., 1989) of the content and
applied it to previously learned knowledge. The ability to recognize underlying
structure within a larger information environment could have significantly affected a
learners ability to comprehend the deep structure of a knowledge domain. Some of the
ability to actively assemble a unique schema customized for the ill-structuredness of the
knowledge domain was also dependent on the ability to focus on the information
39


relevant to the problem and solution, and screen out extraneous information. As a part
of the research design, two cognitive style/ability tests were administered to investigate
the relationship between processing styles and performance on schema assembly tasks
while solving simulated clinical problems. The following sections introduce a class of
individual differences, describe the measures used, and relate them to the abilities/skills
which facilitate flexible schema assembly.
Individual differences. Jonassen and Grabowski (in press) in their extensive
review of individual differences in education, delineated seven different types of
intellectual (and affective) styles or propensities which influenced learning in different
situations. Of these, cognitive controls, which influenced and/or control perception of
environmental stimuli (p. 60), were of particular interest to designers of learning
environments. Many researchers (Spiro et al., 1991; Jonassen, 1991; Cognition and
Technology Group, 1991; and others) have integrated complex content and structural
knowledge components into computer learning environments. The learner had the
option (and the responsibility) of accessing, analyzing, and integrating a wide range of
cognitive resources into his/her individual structure. Differences which influenced and
directed the organization and analysis of complex material have acted as intellectual
executive controls (Guilford, 1980, p. 717) which affected the effectiveness of
computer-based learning environments. Therefore, the ability to assimilate, represent,
and/or construct knowledge was partially a function of the interaction of individual
cognitive controls and the design of the computer environment.
40


Field dependence/independence. Field dependence/independence (FD/I) has
been one of the most studied cognitive controls (Claxton & Murrel, 1987: Green, 1985;
Guilford, 1980). It was described as an ability/style (Jonassen & Grabowski, in press)
which may prove to be integral to using complex computer based instruction. FD/I
originated from a spatial/kinesthetic test of visual orientation and was correlated with
the Embedded Figures Test, which required learners to find given simple figures within
larger relatively complex figures. The underlying task in both the original tests and the
Embedded Figures Test was the ability to arrive at a correct perception [of a figure] by
ignoring interfering contexts (Guilford, 1980, p. 719). Although FD/I measures
relied primarily on spatial tasks, scores from these measures have been related to
general problem solving and other organizational and analytical tasks (Jonassen &
Grabowski, 1992). Although Witkin, Faterson, Goodenough, and Larp (1962)
proposed four dimensions of FD/I, the specific characteristics of field independence
involved in learning described by Jonassen and Grabowski (in press) were more
relevant to assembling flexible schema:
select information sources;
search for and validate information;
transfer knowledge (predict outcomes, infer causes, or evaluate
implications);
generate metaphors and analogies; and
evaluate knowledge (test personal skills or judge utility or value
of knowledge), (p. 68)
Learners who have acquired and utilized these strategies easily and efficiently had the
tools to benefit fully from complex computer-based learning environments.
41


Likewise, students who lacked these strategies or used them reluctantly and
inefficiently could have been significantly hampered in their use of such complex
information.
Several studies indicate that FD/I influenced cognitive processes similar to those
needed for using cognitive flexibility hypertexts and other complex computer-based
learning environments. Recognizing the structure of information was a key component
of flexible schema assembly. In a related structural identification task, Thompson
(1987) found that adding external structure (headings) to text passages aided field
dependent (FD) individuals and hindered field independent (FIs). Likewise, FI
individuals outperformed FDs using plain text passages. Tannenbaum (1982) and
Lambert (1981) also found that low structure materials favored FIs and high structure
materials helped FDs. The module investigated in this study contains a set of highly
interconnected, thematically structured information resources. FI individuals exhibited
abilities and preferences which facilitated identifying themes and relating them to
specific problems or cases because this process required learners to create or impose a
structure on the information presented. Furthermore it was largely the learner's
responsibility to access information relevant to the problem or case. Studies by
Wilbom (1981) and Adams and McLeod (1979) suggested that FI individuals are more
self-directed and do better in discovery learning than FDs. Providing structure or
explaining the structure of information resources (Frank, 1984; Satterly & Telfer,
1979) could have helped to supplant the ability which FIs exhibited to impose structure
42


on information. Hence, FI traits predisposed an individual to gain maximum benefit
from interacting with cognitive flexibility hypertexts. Likewise, FD individuals could
have needed extra support to use the instructional environment effectively.
Cognitive rigidity/flexibilitv. Another cognitive control described by Jonassen
and Grabowski (in press) was cognitive flexibility/constriction (CF/C). Like FD/I, this
cognitive dimension included the ability to identify and judge relevant information
flexibly and screen out other irrelevant information. There were several tests for this
cognitive control; all of them were based on either completion of an ambiguous figure
or screening out irrelevant context cues (i.e., color) while performing perceptual tasks.
The kinds of strategies which cognitive flexibility (CF) learners tended to acquire and
use were similar to those of FI learners. CF learners in addition exhibited the following
three skills:
compare new knowledge with existing ideas, beliefs,
predict outcomes and infer causes, and
judge utility or value of knowledge, (p. 77)
These skills have been integrated into the design of complex hypertext learning
environments. The implications for instruction proposed by Jonassen and Grabowski
(in press) included:
high learner control (more learner self-regulation),
self-instructional learning environments, and
use of inductive teaching methods (requiring the student to
organize, hypothesize, manipulate symbolic meaning), (p. 78).
43


Cognitive flexibility hypertexts often required students to interact with instruction
which stressed the skills listed above. Constricted/rigid individuals without the
propensity or ability to construct or restructure their cognitive models may have been
hampered in their use of cognitive flexibility hypertexts.
In summary, the two cognitive style/ability measures used in this study
correlated with the ability to attend to relevant information, identify information
structures, engage in self directed learning, and actively assemble information into new
structures. All of these skills (or abilities) were integral to the kind of learning defined
by Spiro and Jehng (1990). If FI and CF individuals were predisposed to interact
effectively with the cognitive flexibility hypertext used in this study, then their scores
on the test cases would have been improved. Furthermore, identifying learner-
computer interactions which produced high scores on the test cases may provide
guidelines for designing remediation or supplantation strategies for learners who
require more structured learning. The next section discusses the identification of
meaningful learner-computer interactions in a hypertext software environment.
Process Instruction and Evaluation Methodology
This section discusses ways of evaluating the learner-computer interaction
through the analysis of computer audit trails. The first part outlines approaches to audit
trail analysis from library and information science. The second part presents recent
research in the analysis of hypertext audit trails. The final part reviews methods for
interpreting data collected from the hypertext audit trails.
44


Student models in audit trail analysis. One of the requirements for the effective
use of a computer-based cognitive flexibility hypertext was having an active learner
engaged in a schema assembly process as s/he interacts with the information. The
human computer interactions have been system directed, learner directed, or a
combination of the two. When the learner engaged in more self-directed exploration
and construction, the record of their interactions with the system could have provided
insight into the knowledge construction process. Several authors have addressed the
use of audit trails in interpreting different types of knowledge acquisition. In a
thorough literature review of information science, Rice and Borgman (1983) focused
on two kinds of computer-based systems which have used computer-monitored data:
on-line information retrieval systems (e.g., bibliographic databases) and
communication systems (e.g., electronic mail systems, computer conferencing, etc). In
addition to various system specific functions, the data collected by recording
interactions during searching procedures in a database or electronic mail session was
used to provide group or individual user analysis of performance. They suggested
three kinds of analysis be used in interpreting a range of data points: pattern, error, and
time (p. 248). Pattern analysis consisted of looking at sessions (from log-in to log-
out) created by individual transactions (user commands and system responses) to study
user behavior. Transactions were looked at across sessions of one or several users to
evaluate system characteristics. Error analysis included focusing on the frequency,
type, and context of what any individual system codes as an error.
45


Timing and duration described when certain actions were performed and how long they
took. The authors suggested that future research should focus on a combination of
these factors depending on the nature of the task and the kind of information desired
from the audit trail.
Some authors focused on constructing a user model from audit trail data. In a
review of cognitive modelling in computer based information retrieval and tutoring
systems, Daniels (1986) suggested that systems should create user models which help
the system work more efficiently, rather than attempting to model every possible
property of the user. The information retrieval systems that created user models from
audit trails described users in terms of the relatively small set of transactions involved in
querying databases and other on-line information sources. Many of the other tutoring
systems mentioned by Daniels used portions of audit trail information to create a user
model to aid the system in guidance, repudiation, and future case selection. The user
models usually reflected what content the user has been exposed to, but were not
intended to describe the user's thought processes.
In more recent work, Marchionini (1989) used audit trails and observation to
infer student models of information searching using cd-rom based encyclopedias. His
methodology included relating aggregate patterns of behavior abstracted from audit
trails with cognitive information seeking strategies. Using a delimited set of moves
within the system, patterns of use characterizing younger versus older users, and less
experienced versus more experienced users were found. The connection between a
particular strategy and the user's cognitive state or process was also established.
46


Hypertext audit trails. The research from information science has created the
groundwork for analyzing computer monitored audit trails in on-line data and tutoring
systems. However, there were many important differences between database searching
and information retrieval environments and hypertext systems designed for learning.
Misanchuk and Schwier (1992) suggested three program structures which were
represented by different kinds of audit trails: linear, branching, and
hypermedia/multimedia. Linear structures constrained all learners to the same path,
although the time spent on each node may differ between individuals. Branching
structures varied in complexity from simple branch and return patterns to learner
controlled parallel paths (p. 358), where the learners choice of nodes (screens,
program segments, etc.) added complexity to the audit trail. Dependency, where the
preceding node influenced the meaning of the next one, and looping or revisiting,
where learners go over the same node more than once, both made the audit trail more
difficult to analyze. Many information retrieval systems contained branching or parallel
path structures and analysis of the audit trails produced by theses structures was
quantitative in nature (p. 361). The unique paths that learners created through
hypertexts with a high degree of generativity and learner control could contain both
dependency and looping as well as a virtually unlimited branching capacity.
Misanchuk and Schwier (1992) listed several kinds of audit trail representations
for hypertexts, both numeric and graphical, but found these methods inadequate for
representing the complexity of hypertext audit trails. Analyzing the audit trails from
complex hypertexts necessarily required a shift toward more qualitative research
methodologies. In moving toward this goal, the authors pointed out two crucial
47


concepts which needed to be addressed when analyzing complex audit trails: the
integrity of the user's path and the reasoning behind choosing a particular path.
Integrity of the audit trail implied that the unique path taken by a user is the result of
conscious choices, not the result of random browsing through the system.
Understanding the user's reasoning was difficult because users may have different
reasoning processes for the same path taken. This made attributing particular segments
of an audit trail to a particular line of reasoning difficult.
Several preliminary studies used information from hypertext audit trails.
Gillingham (1992) examined depth-first versus breadth-first searching strategies while
answering recall and inferential questions using hypertext. The program used a tree
organizational structure with two levels of specificity. Each node was at least one
paragraph (1 screen) in length and links were indicated by bold face words in the text.
The total number of nodes is thirty-five. By analyzing efficiency (time and nodes
access ratios) and patterns of node traversal, Gillingham categorized subjects by the
number of accesses each made to level-1 (general) or level-2 (specific) nodes. Level-1
accesses correlated positively with question scores. By contrast, subjects who
accessed few pertinent level-2 nodes had scores which correlated positively with up to
four level-1 accesses. Five or more level-1 accesses correlated with poorer scores on
questions. Overall, the depth-first searchers (many level-2 accesses) were generally
more successful than the breadth-first (fewer level-2 accesses) in answering questions
related to the hypertext.
48


These findings suggested that the organizational structure of a hypertext influenced
what kind of strategies were effective for certain kinds of tasks and that hypertext audit
trails could be analyzed to produce models of user behavior and cognitive structure.
Trumbull, Gay, and Mazur (1991) created four access structures (index,
browse, guide, or mixture) for a hypertext on cultural entomology. Each access
structure represented a different perspective on the network of ideas inherent in the
hypertext. The index structure provided alphabetical and thematic (keyword) text
searching and access of the nodes in the program similar to a standard book index.
Browsing structures allowed users to traverse the nodes using the metaphor of a
Victorian country house and grounds. Users moved through the rooms of the house
and the garden with the support of on-line maps which showed their location in the
hypertext house. The guide feature suggested paths based on the user's previous
searches (if any) or on the basis of preset associational patterns. Two weeks after an
introductory session, users were asked to explore the program related to one of three
interpretative questions. Based on their audit trails, the authors looked for patterns of
use which reflected the four access strategies: browse, index, guide, or mixture.
Subjects were categorized as belonging to one of the four groups by using one strategy
significantly (one standard deviation) more than the others. Performance was judged
by how many nodes, relevant to a focus question, a subject viewed. Among other
findings, the indexers accessed less relevant information than the other groups. The
authors attributed this to the narrow, less intuitive nature of traditional indices which
translated into hypertext through the index access structure.
49


Homey and Anderson-Imman (1992) used audit trails from two hypertext based
stories to identify six patterns of hypertext use in middle school students. The program
featured vocabulary support, comprehension and self monitoring questions, and
prompted writing assignments augmenting a linear text with a structured overview
(graphical browser). Each audit trail was analyzed and segmented into clusters of
events indicating a common behavior. Six patterns (skimming, checking, reading,
responding, studying, and reviewing) recurred in the data analysis and some students
showed an evolution of strategy selection from less to more interactive over the course
of the study.
Jonassen (1991) investigated many facets of semantically structured hypertext
and their effects on different kinds of performance using differing access structures. In
a series of three studies, the effects of different kinds of semantic structuring of
hypertext on learners structural knowledge development were examined through
analysis of audit trail data and performance on a variety of measures designed to elicit
factual and structural knowledge from learners. The content of the hypertext was
hypertext itself, including, historical events, characteristics, information access, etc.
Among other findings, the author suggested that the number of structural cards
accessed significantly predicted scores on relationship proximity judgements, a measure
of structural knowledge acquisition.
50


Summary
Throughout the preceding discussion, several themes related to the present
study. First, clinical medical knowledge domain was seen as ill-structured and
complex. In addition to being ill-structured, biomedical knowledge was not stable and
changed with experience and time. Making the transition from the basic sciences
approach (to biomedical knowledge) to a more clinically based approach created
problems for students. Reductive biases and faulty understandings created by failing to
represent and communicate the complex structure of biomedicine (and other domains),
resulted in faulty reasoning and misapplied generalizations. In order to successfully
integrate previously learned information into clinical problem solving tasks, a radical
restructuring of knowledge was necessary which resulted in qualitatively different
schemata.
The medical reasoning and problem solving processes were studied through
various kinds of computer modeling and instruction with varying degrees of success.
Modeling competent performance proved to be an easier task than modeling acquisition
of knowledge. Cognitive flexibility theory provided a coherent knowledge acquisition
model for learning complex and ill-structured domains like biomedicine. Using
hypertext systems to interrelate and connect complex concepts and procedures had the
potential to produce meaningful instructional resources.
While many knowledge acquisition models were based on schema theory,
cognitive flexibility theory focused on fostering situation specific and flexible schema
assembly requiring radical restructuring. Cognitive flexibility hypertexts contained
51


I
links specifically designed to engender and support viewing concepts and principles
from multiple perspectives. The multiple representations were united and made
comprehensible through other links provided to highlight multiple themes. This unique
structure was well suited to represent content specificity and gave students the kind of
practice needed to radically restructure their schemata and create new knowledge
structures, such as, illness taxonomies (Clancey, 1989), illness scripts (Custers et al.,
1992), or clinical models (Patel et al., 1990).
Many of the skills important for learning from cognitive flexibility hypertexts
have been found to be unevenly distributed throughout the population. Individual
differences, specifically, field dependence/independence and cognitive
constriction/flexibility, could have influenced how effectively students utilized the
complex resources provided by cognitive flexibility hypertexts. Supplantive or
remedial interventions could be necessary to empower some learners in the use of this
new and robust technology of instruction.
52


CHAPTER 3
RESEARCH METHODOLOGY
This study intended to examine issues related to the use of cognitive flexibility
hypertexts in complex ill-structured knowledge domains to foster advanced knowledge
acquisition in biomedicine. This chapter is composed of eight parts:
background and context,
research design overview,
subjects,
materials,
assessment,
experimental design,
procedures, and
research hypotheses.
Each of theses areas are discussed in the following sections.


Background and Context of the Study
I
The research reported in this study was supported by a Transfusion Medicine
Grant from the National Institutes of Health. One of the goals of the grant was the
development and production of computer-based instructional materials to be used in
medical school curricula across the country. The first module produced by the grant's
research team, a cognitive flexibility hypertext in the field of transfusion medicine, was
the instructional environment investigated in this study. The fact that the module was
designed as a learning environment for a particular audience (medical students and
professionals) and not as a research tool influenced the selection of appropriate
evaluation methodologies. Also, the problems associated with conducting medical
education research, especially finding time in students' already hectic and overbooked
schedules to participate in research, constrained the data collection options available.
Each of these influences will be discussed below.
The module used in this study was designed to be completed independently by
medical students, residents, and other health professionals. Over a period of almost
two years instructional designers and content experts collaborated to build a computer-
based learning environment which would serve the needs of its intended audience.
Because this study was designed to be an unobtrusive part of the field testing phase of
the larger design effort, it was deemed inappropriate to artificially alter the structure of
the module in order to create experimental and control versions. Instead, the study has
focused on how different individuals actually use the practice portion of the module and
how these patterns of use are correlated with instructional support and with scores on
the assessment portion of the module.
54


I
I
Another factor which influenced the research methodologies selected was the
scheduling of testing sessions. Because the research was conducted with medical
school students functioning under very tight schedules and time lines, several time slots
were made available over a four day period. Students signed up for the sessions which
were convenient for them. The treatment variable (the presence or absence of a
researcher-led orientation) had to be administered in some of the time slots and not
others. While it was impossible to use true random assignment with this independent
variable, the only factor influencing the assignment to treatment versus no treatment
was balancing the numbers in the two groups. Figure 3.1 illustrates the different cells
of the research design.
Figure 3.1. Cell design for independent variables.
field dependence/ independence cognitive con- striction/flexibility
treatment group (orientation) FD-Trt FI-Trt CC-Trt CF-Trt
control group (no orientation) FD-Ctl H-Ctl CC-Ctl CF-Ctl
Research Design Overview
The research methodologies used in this study were based on testing initial
hypotheses and exploratory hypothesis development and testing. Initial hypothesis
testing procedures were used to investigate the influences of individual differences and
orientation procedures on scores derived from test cases. The specific questions and
hypotheses are listed at the end of this chapter. Many of the methodological
55


assumptions for these questions conformed to a quasi-experimental research design
(Cook & Campbell, 1979; Campbell & Stanley, 1963). The exploration and
development of new hypotheses were achieved by using correlational analysis to
identify usage patterns which have the potential of interacting with the independent
variables identified above. These usage patterns were then treated as dependent
variables for ANOVA analyses. The results from this portion of the analysis were
designed to stimulate new hypotheses and research about hypertext usage patterns.
Subject Population
Subjects in the present study were drawn from a class of second year, eighth
month medical students who were finishing their hematology rotation at a large New
England medical center. The subjects volunteered to participate in the study. As an
inducement to participate, each student who volunteered was entered in a drawing to
win one of two seventy five dollar gift certificates at the university book store. Twenty
six subjects successfully completed the program and were included in the statistical
analysis. Table 3.1 shows the demographic data on the subjects.
56


Table 3.1
Demographic Data on Subject Population
Response item Mean SD Unit
Age 25.0 0.4 years
DOS experience 3.0 1.6 1-6*
Macintosh experience 4.3 1.7 1-6*
Computer use data 2.5 2.7 # of programs**
Gender Male 13 Female 13
N = 26
Responses to the question, How familiar are you with using Macintosh (or
DOS) computers? (not familiar = 1, very familiar = 6)
**indicates the number of CBI programs used in medical school
Experimental Materials
The experimental materials used in this study consisted of the following items:
computer-based instructional materials, including practice and
assessment modes;
scores on standardized cognitive control/individual difference scale's;
experimenter developed usage questionnaire; and
on-line data collection of user computer interactions.
This section will describe the different experimental materials.
Computer-Based Experimental Materials
The computer-based instructional/leaming materials used in this study were
developed for use in the medical school curriculum at the University of Colorado
Health Sciences Center as a part of the clinical clerkships in several disciplines (internal
medicine, pediatrics, surgery, etc.), for medical residents and other health
professionals. The development of this module was influenced by a number of factors
57


including the unique structure and content represented in transfusion medicine, the
features of hypertext based computer instructional/leaming environments, and the
model of advanced knowledge acquisition included in cognitive flexibility theory.
These areas are discussed in the following pages.
Content and Task Parameters
The cognitive flexibility hypertext used in this study focused on risk assessment
to donors and recipients of blood products. This topic has been considered part of the
larger field of transfusion medicine which includes: the collection, processing, and
storing of cellular and acellular components of blood; the administration of blood
components and products; and the survival and function of the elements in
transfusions. The field has been viewed as very broad, entailing a number of basic
sciences, including, biochemistry, immunology, and physiology as well as clinical
disciplines, such as, hematology, surgery, internal medicine, and pediatrics (Jonassen,
Ambruso, & Olesen, 1992). Only recently has transfusion medicine emerged as a
distinct area of medicine because of the rapid advances in the fields of biomedicine
listed above.
At the University of Colorado Health Sciences Center Medical School, third
year students have begun clinical clerkships in internal medicine, surgery, and
pediatrics among other topics. This period represented a shift in emphasis from a basic
science approach to a more practice-based clinical curriculum where students were
expected to apply previously learned knowledge in new and varying circumstances.
Many researchers commented on the difficulties involved in successfully applying
58


previously learned knowledge to clinical problem solving tasks (e.g., Patel & Groen,
1991; Lesgold, Rubinson, Feltovich, Glaser, Klopfer, & Wang, 1988; Patel, Evans, &
Kaufman, 1990; Feltovich, Spiro, & Coulson, 1989). In an effort to improve the
effectiveness of instruction in transfusion medicine and avoid the problems endemic to
advanced knowledge acquisition, a computer-based module was designed using
components of cognitive flexibility theory and implemented in a hypertext software
environment.
The specific topic of risk assessment was selected after an analysis of the
relevant outcomes of instruction for third year medical students in transfusion medicine.
From the results of this analysis a set of goals was created to be addressed by the
program:
analyze the significant issues in donor recruitment;
recognize the potential complications of blood donations;
describe the prevention and management of the complications of blood
donation;
identify the most common and immunologic risks of blood transfusion
to the recipient;
outline the rationale and procedures for testing blood prior to
transfusion;
apply basic immunologic principles of compatibility; and
use relevant sources of information to generate hypotheses on clinical
problems involving transfusion medicine.
For a number of reasons related to both the structure and content of the task as well as
the guidelines provided by cognitive flexibility theory, a modified case-based format
was used to achieve these goals. These reasons are outlined below.
59


Content Level and Structure
Most of the goals listed above involved application or transfer of previously
learned information. The new context for acquiring and applying this information was
solving medical problems encountered when dealing with issues related to transfusion
medicine. Therefore, the instructional environment created by the hypertext used in this
study reflected many elements of the context where the problem solving activity would
occur. Also, knowledge application and transfer reflected an advanced (post-
introductory) stage of the learning. Applying knowledge in a modified case-based
environment could have facilitated advanced knowledge acquisition.
The present study contained modified cases or scenarios which required that
learners order laboratory tests (if appropriate) and choose appropriate assessment and
management options for each case or problem situation. Each scenario represented a
specific combination of factors related to risk assessment in transfusion medicine. In
keeping with the content's unique structure, each scenario also had a unique set of
surface features represented by a short history/physical section. Throughout the
practice cases, the learners had access to an information resource rich in interconnected
structural knowledge components as well as relevant factual data. Additional contextual
information was provided through computer responses to the learner's selections
during the application task section. The information access structures and the modified
case-based knowledge application tasks are described below.
60


Information Access Structures
Working in a case-based environment has not resulted in efficient advanced
knowledge acquisition unless the cases were presented in a way which included some
aspects of the structural characteristics of the knowledge domain. One of the key
organizing principles of cognitive flexibility theory was the attention to the structural
characteristics of the knowledge domain. In the hypertext used in this study, the cases
contained thematic access structures to information resources which were available at
any point in the practice cases. These themes were represented by the perspectives
(containing sub-themes), similar cases, and the textbook. Each of these parts of the
program is described below.
Perspectives
The perspectives represented information structured around a set of prototypical
medical professionals as well as the patient's point of view. Each case contained four
or five of the professional opinions depending on the content of the case. The rotating
cast of information sources illustrated the ill-structuredness of the problems faced in the
field of transfusion medicine. As learners interacted with this knowledge base, they
were made aware of the themes which contained potentially relevant information by the
selection of perspectives provided in each case. Figure 3.2 presents a sample
perspective screen.
61


Figure 3.2. Sample perspectives screen from Practice Case Five.
PRACTICE CASES: Perspective of Attending Physician onjOase 51
i think the resident is right.
My differential diagnosis would begin with post transfusion
hepatitis and a delayed transfusion reaction, however it
would probably be early for post-transfusion hepatitis.
I think he has developed alloantibodies and is destroying
the transfused cells.
When this is complete, the hemolysis will stop since the
antibodies will not destroy his own cells.
We don't need admission but close follow up and
supportive care as an outpatient would be appropriate.
Research:
[ Patient )
C
Resident
L (Attending Physician^
[Blood Bank Director^
[ [ Tertbook ^
Actions:
[ Return
Help:___________
[ Program Help ^
Appendix A contains representative screens of this and other sections of the program.
Table 3.2 shows which perspectives are associated with each practice case.
62


Table 3.2
Perspectives Available in Practice Cases 1-7
Perspective Cl C 2 C 3 C4 C 5 C 6 C 7
blood bank director X X X X X X X
pediatrician X
surgeon X X
internist X
fellow X
patient X X X X
donor X
recipient follow-up X
attending physician X X X
gastroenterologist X
phlebotomist X X
resident X X
Similar Cases
The 12 similar cases were designed to simulate a clinician's memory of cases
which share themes with the case the learner is currently working on. Adapting the
mini-case construct from Spiro & Jehng (1990), each of the similar cases was a
condensed version of a practice case. Instead of including perspectives, the similar
cases combined relevant information and the "solution" to the case and presented it as a
commentary on the case. The laboratory test section of the similar cases reported on
relevant tests rather than having the learner request separate tests. The changes made
the similar cases faster and less detailed than the practice cases, while still maintaining
the case-based, problem solving structure of the knowledge domain. Figure 3.3 shows
an example of a similar case screen.
63


Figure 3.3. Sample similar case screen.
Similar Oases: Yellow Eyes II Comment ________________|--------
I - ' ^
With a normal hepatocellular enzyme study and................
documentation of Keli antibody the diagnosis of a delayed....
hemoiytic transfusion reaction has been confirmed. The.......
exposure to the keli antigen probably occurred with the......
transfusions at the time of surgery. The primary immune........
response took i iii days to two weeks to develop significant...
enough antibody to bind to the transfused ceils. Subsequent....
destruction resulted in symptoms consistent with hemoiysis.____
By the time the patient was seen in the ciinic. signs and....
symptoms of hemolysis had resolved as all the transfused cells
were destroyed. Because only recipient ceils remain, the DAT
was negative but the alioantibody couid be detected in the.....
patient's serum. S upportive care is the halimarik of managing.
patients with deiayed transfusion reactions. Documentation of
this ailoantibody (anti-Keil) should be part of his medicai....
record.
v ------------------------------------------------------------------- J
Research:
( Case Introduction ^
[ Case History ^
[ Physical Exam ^
f Lab Data ^
( Textbook )
Actions:
c
c
Next Case
Return
3
3
Help:___________
( Program Help ^
Each practice case had between three and six similar cases assigned to it. The
assignment procedure was based on the content analysis performed in the design phase
of the project. This analysis produced themes important to solving cases in transfusion
medicine. Some of these themes included symptoms, pathophysiology, screening
tests, and risk assessment. If a similar case shared three or more of the nine themes
with the current practice case, that practice case would be considered closely related. A
match on one or two of the themes classified a similar case as simply related. When
learners accessed similar cases, they could have examined either kind of similar case or
see all similar cases.
64


Textbook
The textbook included basic science information arranged in terms of donor and
recipient issues. Selecting an indented topic took the learner to a summary page of
information on that topic. Figure 3.4 shows the organization of specific topics in the
textbook. A glossary was also accessible by clicking on words in text fields or in the
text-book. The textbook and glossary functions were the only information sources
which were not context sensitive. In other words, the textbook for Practice Case 1 was
the same for Practice Case 6.
Figure 3.4. Textbook index screen.
TEXTBOOK: Index
S/
Donor Issues :: Recipient Issues
DONOR PROCESSING : RECIPIENT PROCESSING
Assessment ;; "Look-back"
Registration ;; I! RECIPIENT RISKS
Requirements for donation ;; Infectious
Donor protection I: Acute viral hepatitis
Recipient protection ;; : Hepatitis A
Other restrictions :: Hepatitis B
Mandatory tests ;; Hepatitis C
Post-phlebotomy care - Othercauses of hepatitis
DONOR RISKS :: Retroviruses
General reactions to donation - HIV-1
Mild reactions " :l HIV-2
Moderate reactions ;; HTLV-I
Severe reactions " HTLV-II
Other reactions i: Syphilis
Nature of volunteer H em olytic Transf usi on R eacti ons
Autologous donations ;; Acute HTR
Directed donations ;; Delayed HTR :: Immunologic risks
|l GLOSSARYOFTERMS

Click right
on the index
topic that
you wish to
see
Actions:
[ Return j
Help:
[ ProgramHelp
65


Summary
The information access structures represented a knowledge web supported by
the themes described above. Because the knowledge base were case-sensitive,
different themes played greater or lesser roles in each individual case. By purposefully
exploring and interacting with this flexible, yet structured knowledge base, the learner
engaged the cognitive processes which were necessary for the development of the
flexible schema assembly process.
Knowledge Application Tasks
The knowledge application tasks provided additional contextual information and
also contained evaluative feedback. The laboratory tests, in particular, often provided
crucial information which aids in assessing and managing the case as well as providing
guidance in the form of feedback. Assessment and management sections also provided
meaningful feedback. Each of these program features will be described below.
Ordering Laboratory Tests
The test ordering section of the program was a combination information source
and evaluation instrument. When learners selected the lab test portion of the program,
they were given a choice of six different kinds of data to investigate: chemistry,
hematology/coagulation, blood bank/serology, microbiology, urinalysis, and diagnostic
procedures/radiology. There was a group of tests and procedures (between 4 and 20)
associated with each of these topics. When a learner selected a test, a value was given
and a box appeared which provided feedback on the appropriateness of his/her choice
66


and why the choice was judged good or poor. The learners could have also opted to
see a suggested list of tests either before or after they requested test data on their own.
Assess and Manage the Case
Because these two sections functioned identically, they will be addressed
together. These two sections used a multiple-choice format (3-10 options per section)
with the possibility of more than one answer being correct. A feedback box appeared
whenever a possible assessment/management option is chosen. A learner could have
viewed the correct choices and feedback at any time in this section.
Cross-Case Comparisons
In addition to the information access structures, learners could have also
traversed cases. No order of completion was mandated by the program. The presumed
order, given the structure of transfusion medicine, was to order lab tests, assess, and
manage the case. Learners could have also accessed the "cases menu" from any screen
in a practice case, allowing them the opportunity to compare perspectives across
different cases. Students could have referred back to previously solved cases or
jumped ahead without finishing a case.
In summary, the practice cases were an environment rich in structural and
thematic information sources. The foundation of the program was a modified case-
based learning environment where the structure of the program reflected the structure of
the knowledge domain, transfusion medicine. The test ordering, assessment, and
management sections simulated aspects of the problem situations which could have
67


been encountered in clinical practice and provided a purpose for investigating the
information resources available. The information resources were structured to provide
coherence and manageability to the knowledge base without removing the complexity
of the subject matter.
Evaluation of Learning: Test Cases
Assessment of the kind of learning encouraged by this environment was
achieved through a series four of test cases similar to the practice cases in structure
except the information resources (e.g., perspectives, similar cases) removed. The
textbook was available in the test cases and was identical to the practice case version.
The lab test, assessment, and management sections included a point based system for
scoring the learner's performance.. The scoring system is outlined below.
Order lab tests: Each test ordered receives a score of -1,0,+1, or +2
depending on whether or not it is essential for establishing a diagnosis
for the given case, its effect on the patient, and cost considerations.
Assess case: The actions selected for assessing the case will receive a
score of -1 or +1 depending on whether the selections are valid or not.
Manage case: The actions selected for managing the case will receive a
score of -1 or +1 depending on how appropriate the selections are for
the case.
Overall scores are determined by the following method:
points from correct choices incorrect choices
total possible
68


Learners received their scores in the evaluative feedback box associated with each
choice. Access to correct answers was also provided, however, once this was selected
learners could not score any more points in that section of the case.
Learners accessed their total scores at the end of each test case. The version of the
program used in this study required that the test cases be completed in one session.
Assessment
The assessment portion of this study included outcome scores from the test
cases discussed above. Other assessment tools included instruments to measure
cognitive ability/style differences in learners, a usage questionnaire, and audit trails
from the learners' practice sessions. These parts of the assessment are discussed
below.
Individual Difference Testing
The two individual difference/cognitive control scales used in this study were
field dependence/independence and cognitive constriction/flexibility. Both of these test
instruments came from the Kit of Factor Referenced Cognitive Tests (Ekstrom,
French, & Harman, 1976). The tests were administered according to the instructions
included in the manual which accompanies them with one exception: subjects were
allowed to read the instructions silently instead of reading them aloud. The researcher
solicited questions and outlined the major tasks and time restraints associated with the
tests before administering them. See Appendix B for specific testing instructions and
copies of the test instruments.
69


Field dependence/independence. This style/ability was measured using the
Hidden Figures Test (Ekstrom et al., 1976). Internal consistency reliabilities for this
instrument clustered around .8 (Goldstein & Blackman, 1978). This measure was
chosen because of the kind of traits it correlated with and the ease of administration.
The interaction of a subject's acquisition of (or lack of) this information structuring
ability (or preference) and performance in a complex thematically structured hypertext
was a key area of interest in this study. The ability to structure information could have
potentially affected the processes involved in flexible schema assembly, one of the main
learning outcomes hypothesized by cognitive flexibility theory.
Cognitive constriction/flexibility. This style/ability was measured using the
Gestalt Completion Test (Ekstrom et al., 1976). While Ekstrom and her colleagues
reported internal consistency reliabilities of .85, Jonassen and Grabowski (1992) cited
studies which indicated lower values (.67 -.7). This measure is chosen because one of
the traits identified by Gardner, Holzman, Klein, Linton, & Spence (1959) was the
ability to concentrate on a task and avoid being distracted by competing stimuli. The
ability to search meaningfully and screen out less relevant information could have
significantly affected the kinds of information-seeking interactions engaged in by
learners in cognitive flexibility hypertexts. The ability to criss-cross the conceptual
landscape could have been influenced by the ability or preference for focusing on
information relating to task completion.
70


Although cognitive constriction/flexibility correlated fairly highly with the other
ascribed independent variable, field dependence/independence, it did not measure the
same dimension of cognitive style. Rather, these two factors may in combination
represent the broader construct, field articulation (Gardner et al., 1959), which could
have interacted with the treatment variable.
Usage Questionnaire
A 21 item questionnaire, designed for formative evaluation purposes, was
adapted for use in this study as a self-report summary of the kinds of information
seeking strategies employed by learners. The questionnaire addressed two areas,
information access behavior and comparisons to other instructional modes, and used
Likert scales and open response items. (See Appendix B for a complete
questionnaire). Demographic and previous computer use data were also collected on a
consent form distributed and signed by all students.
On-line Data Collection of User-Computer Interactions
An audit trail of computer-user interaction was collected unobtrusively for each
user in the study. When subjects begin the program, they were asked to enter their
name and student identification number. When this occurred, a file was created which
records all interactions during the practice and test cases. Since the program was
completely mouse-driven, interactions consist of mouseclicks on buttons or in text
fields for glossary items.
71


In the information access portion of the program, mouseclicks entailed moving to other
navigational or content related screens. During the knowledge application tasks,
subjects could have accessed the information resources, selected laboratory tests, and
accessed the assessment or management options.
Experimental Design
The major experimental design procedure investigating between and within
group factors was a hybridized form of a non-equivalent control group design (Cook &
Campbell, 1979). This quasi-experimental design model assumed a pre-test post-test
difference in the dependent variable as a factor potentially interacting with the treatment-
control factor. The post-test measure, outcome scores on four test cases, took between
20 and 60 minutes. A suitable pre-test for this kind of learning outcome would have
taken more of the participants learning time and would have been cumbersome to
administer. The similarity of background knowledge of the subjects (all from the same
class of one medical school on the same rotation) also factored into the decision to not
administer a pre-test. Instead, measures of cognitive style/ability were used as factors
to partially account for within group differences. Instruments correlating with the
individual difference measures, field dependence/independence and cognitive
constriction/flexibility, were administered, and those scores (interval data) were used in
the analysis as a factor in the place of a pre-test post-test difference.
Because of logistical considerations and the time constraints imposed by the
participants' schedule, true random assignment was not possible. Participants signed
up for the time slots which worked best with their schedules. The decision of which of
the four sessions received the treatment (orientation) and which did not was made based
72


on attempting to balance the numbers in each group. The lack of true random
assignment and the resulting non-equivalence of the control and treatment groups and
the lack of a true pre-test was compensated for in part by investigating possible sources
of bias (age, gender, computer experience) which might have created within-group
differences. The following sections outline the characteristics of the variables
employed in the study.
Independent Variables
This study employed three independent variables, one active and two passive.
The active independent variable was the presence or absence of a 20 minute
introduction to the program which included cognitive strategy instruction. The
complete description of the orientation follows. The instruction focused on
demonstrating the ability to use the information resources to view a case from several
perspectives and to build a model of the case. Subjects were also prompted to develop
a sense for the kinds of problems each of the viewpoints (represented by the
perspectives) express. The control group received a minimal introduction to the
program and was strongly prompted to go through Practice Case 1 because it was
designed as an basic tour of the module. The two passive independent variables were
field dependence/independence and cognitive constriction/flexibility. These cognitive
styles/abilities were chosen because they could have enhanced (or impeded) a subjects
ability to acquire and/or construct knowledge in the module used in this study.
73


Dependent Variables
The main dependent variable was the set of scores generated by participants
while ordering laboratory tests, assessing, and managing the test cases. For the initial
data analysis, the scores from all four cases were combined to yield an overall score.
Sub-scores on individual cases and on specific application tasks (ordering lab tests,
etc.) will be broken out in Chapter Four. Another set of dependent variables were
learner behaviors as recorded by the audit trail. The amount of time spent and the
number of screens accessed in the various sections of the information resources section
(perspectives, similar cases, and textbook) could have been affected by the treatment
and the cognitive style variables of the subjects described above.
From the available research, it was possible to make predictions about
performance on the test cases based on the independent variables. These predictions
(listed below) were directional. For example, an improvement on outcome scores was
predicted by the presence of the orientation treatment for field dependent learners. The
effects of the independent variables on the use variables was less certain. It is logical to
predict that time spent in a particular section of the program would have influenced test
case scores, but to predict the nature of that influence would have been premature. Not
enough was known about how the independent variables would affect performance to
form firm hypotheses. Hence, non-directional hypotheses were used in stating the null
and alternate hypotheses relating to use variables. Figure 3.5 shows the relationship
between the independent and dependent variables in this study.
74


Figure 3.5 Predicted and possible main effects of independent variables.
field
dependence/
independence
Independent Variables
cognitive rigidity/
flexibility
treatment
vs control
amount of time
in different parts of the program number of Dependent Variables test case
screens accessed in diffrerent parts of the program scores
Key:
predicted effect
unknown/possible effect
Procedures
Both groups, the experimental (given a 20 minute orientation) and control
(without a 20 minute orientation), received a basic mousing tutorial to eliminate
technical problems operating the software. Because of the participants familiarity with
Macintosh computing environments, interference due to incorrect usage of the software
75


was minimal. Both groups also received a booklet detailing the different sections of the
program and a job aid to help them begin. (See Appendix C for copies of the
materials.) The participants completed a consent form which guaranteed their
confidentiality and collected some baseline demographic information. All participants
completed the individual difference measures as a group before beginning the session.
Both groups are guided to the opening menu screen. It was from this point that the
treatment group differs from the control group.
Control Group
Participants were assigned to either the orientation treatment or the control based
on which of the four time slots they wanted to attend. The control group was prompted
to go through an overview of the program and Practice Case 1, which served as an
introduction. After all participants had successfully arrived at the main menu screen,
the group began the program with no additional help from the researcher, unless a
participant had a particular problem or question. The researcher announced when the
time is half over as well as when the participants had a half-hour to go. The total work
time allowed was approximately two and one half hours.
Experimental Group
The experimental group received an orientation which took approximately 20
minutes. During this time, the subjects went through the main features of the program
as a group, using the first practice case as an example. The orientation began by having
the subjects open the module and proceed to the introduction menu of Practice Case 1.
76


At this point the researcher had the subjects follow along while he demonstrated the
functions of the different buttons shown. First, the history/physical section of the
program was demonstrated. Then the perspectives section was shown.
The researcher pointed out that many of the perspectives appeared in many cases and
that a main goal of the program was for them to learn to look at the cases from multiple
perspectives and build a model or understanding of the perspectives. Next, the subjects
were taken to the introduction screen of the first similar case, but not all of the
informational screens were accessed. Next, the procedure for ordering tests was
demonstrated. After ordering two tests, the researcher pointed out the suggested tests
button which showed all of the tests recommended by the program. Here, the
researcher demonstrated clicking on a test which showed the feedback window related to
that test. The researcher pointed out that feedback was available to them at any time.
While in the test ordering section of the program, the subjects received a strategy for
using the similar cases section of the program. From the test ordering screen, the
researcher went to the similar cases and selected only the case history and lab data
sections of the two similar cases. The researcher explained that it is not necessary to
read all of the parts of the similar cases (or any other part of the program); he suggested
the option of accessing only the parts of the program which were relevant to the task the
subjects are working on. From the test ordering section, the researcher also accessed
the textbook section of the program. Again, the strategy of searching out specific
information without necessarily reading the entire section was explained. Next, the
assessment and management sections were demonstrated. The ability to receive
feedback before or after the correct answers button was used, was demonstrated
77


again. From the management section of the program, the researcher went back to the
perspectives section to demonstrate the ability to revisit information from a different
part of the program. In concluding the orientation, the researcher stressed the
importance of creating a model for each case based on the information given in that case
as well as developing an understanding of the themes represented by the perspectives.
The subjects then began the program and worked through the practice cases and test
cases. The researcher announced when the time is half over and when the participants
had a half-hour to go. The participants had approximately 2 hours and 10 minutes to
finish the practice and test cases. After the time expired, the subjects in both groups
filled out a short questionnaire evaluating the program and how they used it.
General Research Hypotheses
The major research hypotheses for this study involved the effects and
interactions of two individual difference characteristics and a treatment variable (the
orientation session) on performance in a case-based problem solving module (practice
cases) and scores generated from similarly designed assessment module (test cases).
There were two areas of investigation:
the effects and interactions of field dependence/independence (FD/I),
cognitive constriction/flexibility (CC/F), and treatment vs control
(Trt/Ctl) on test case scores; and
the effects/interactions of field dependence/independence (FD/I),
cognitive rigidity/flexibility (CC/F), and treatment vs control (Trt/Ctl) on
use variables related to time spent in different sections of the program.
Specific hypotheses related to these areas are listed below.
78


Treatment on Test Case Scores
The mean scores on the test cases for the treatment group will be significantly
higher than for the control group.
H(0): |I(TG) = |l(CG)
H(l): p, (TG) > p,(CG)
Field Dependence/Independence on Test Case Scores
The mean scores on the test cases for the field independent learners will be
significantly higher than for the field dependent learners.
H(0): (I (FI) = fl(FD)
H(l): [l (FI) > |l(FD)
Cognitive Constriction/Flexibility on Test Case Scores
The mean scores on the test cases for the cognitively flexible learners will be
significantly higher than for the constricted learners.
H(0): p(CF) = |i(CR)
H(l): |l(CF) > p(CR)
79


Interaction Effects
There will be an interaction between field dependence/independence and
exposure to the treatment variable as measured by scores in the test cases.
There will be an interaction between cognitive constriction/flexibility and
exposure to the treatment variable as measured by scores in the test cases.
Field Dependence/independence on Learner Behavior in the Practice Cases
The mean times spent in the various parts of the program by field independent
learners will be significantly different from the mean times spent in the various
parts of the program by field dependent learners.
H(0): p.(FI) = |I(FD)
H(l): M-(FI) * ji(FD)
Cognitive Constriction/Flexibilitv on Learner Behavior in the Practice Cases
The mean times spent in the various parts of the program by cognitively flexible
learners will be significantly different from the mean times spent in the various
parts of the program by constricted learners.
H(0): |I(CF) = |ll(CR)
H(l): |i(CF) * |i(CR)
Interaction Effects
There will be an interaction between field dependence/independence and
exposure to the treatment variable on the mean times spent in the various parts
of the program.
There will be an interaction between cognitive constriction/flexibility and
exposure to the treatment variable on the mean times spent in the various parts
of the program.
80


CHAPTER 4
RESULTS AND DISCUSSION
This chapter reports the results obtained from this investigation. The following
introduction contains data which addresses the hypotheses listed in Chapter Three.
These deal with the effects and interactions of cognitive style/abilities and orientation
procedures on problem solving performance in a computer-based learning environment.
The second section addresses the correlational hypotheses which relate time spent in
various parts of the practice cases and the factors listed above.
Introduction
This section reports the findings relevant to the experimental hypotheses
presented in Chapter Three. The active independent variable was the presence (Trt) or
absence (Ctl) of an orientation session which demonstrated the information access
structures included in the program. (This orientation is described in Chapter Three.)
The two within group independent variables were the cognitive style/ability measures of
field dependence (FD) or independence (FI) and cognitive constriction (CC) or
flexibility (CF). The dependent variable was the total of the scores on the four test
cases. Both analysis of variance and analysis of covariance techniques were employed
to investigate these main effects and interactions.


The absolute values of the t test were less than the critical values (CV) which implied
that there were no significant differences between the means of the orientation aid the
control groups which could have been attributed to the population characteristics listed
above.
Table 4.1
Independent Samples t Test Summary for
Group Assignment and Questionnaire Data
Factor Mean SD L CV
Age Trt 25.07 3.3
Ctl 24.93 4.4 0.10 2.06
Macintosh
experience*(l-6) Trt 4.0 2.0
Ctl 4.6 1.4 -.091 -2.06
Number of CBI
programs used* Trt 2.46 3.1
Cd 2.69 2.6 -0.20 -2.06
N = 26, n(Trt) = n(Ctl) = 13, df = 24, a = .05
*see Table 3.1 for descriptions of these measures
Additional t tests were conducted on the cognitive style/ability measures to
investigate possible initial differences between the orientation group and the control
group. The raw scores from the cognitive style ability tests were divided into groups
based on a mean split. That is, subjects with scores below the mean were classified as
field dependent or cognitively constricted. Subjects who scored above the mean were
classified as field independent or cognitively flexible. This method of group
83


assignment was chosen because norming data was not available for the specialized
population (medical students) involved in this study. These results are summarized in
Table 4.2. The results of the t tests on individual difference measures indicated that the
groups (Trt and Ctl) were not significantly different on the field
dependence/independence and cognitive constriction/flexibility variables.
Table 4.2
Independent Samples t Test Summary for
Group Assignment and Individual Difference Scores
Factor Mean SD t CV
Field dependence/ independence (FD/I) Trt Cd 6.54 7.54 4.4 3.6 -0.54 -2.07
Cognitive constriction/ flexibility (CC/F) Trt Cd 11.30 13.69 4.09 3.09 -1.52 -2.07
N = 26, n(Trt) = n(Ctl) = 13, d£ = 24, a = .05
84


Main Effects: Analysis of Variance and Covariance
This section reports the results from the the analyses of variance and covariance
to test for main effects and interactions. First, the results of a three-way ANOVA,
conducted on the three independent variables (Trt/Ctl, FD/I, CC/F), are reported. (See
Appendix E for complete ANOVA tables.) Next, a summary of the findings
concerning main and interactions effects will be given, including a review of the
study's adherence to ANOVA and ANCOVA assumptions.
Main Effects: Analysis of Variance
Summary statistics (means and standard deviations) for the three-way ANOVAs
appear in Table 4.3.
Table 4.3
Summary Statistics from ANOVA Analysis
1
Trt Ctl Total
Factor n X SD n X SD n X SD
Main 13 39.1 4.5 13 34.7 11.5 - - -
FD/I FD 7 38.9 5.9 5 29.6 9.5 12 35.0 8.6
FI 6 39.5 2.4 8 37.8 10.5 14 38.5 9.0
CC/F CC 6 38.3 4.9 5 28.6 9.7 11 : 33.9 8.7
CF 7 39.9 4.3 8 38.5 11.5 15 39.1 8.6
N = 26
85


The three-way ANOVA, conducted using the three independent variables in
combination, produced a significant (g < .1) main effect for treatment and a main effect
for CC/F (g = .10). There was no significant effect for FD/I. Figure 4.1 gives the
means and cell sizes for the three-way ANOVA. Table 4.4 summarizes the results of
the three-way ANOVA.
Figure 4.1. Cell sizes for the three-way ANOVA: Trt/Ctl x FD/I x CC/F.
FD/I: CC/F FD FI - Totals:
CC CF CC CF
Trt/Ctl Trt 3 37.7 4 39.8 3 39.0 3 40.0 13 39.2
Cd 2 22.0 3 34.7 3 33.0 5 40.8 13 34.7
Totals: 5 31.4 7 37.6 6 36 8 40.5 26 36.9
Table 4.4
Three-Wav ANOVA Summary Treatment x
Cognitive Constriction/Flexibilitv x Field Dependence/Independence
Factor df SS MS F e<
Trt/Ctl- 1 257.35 257.35 3.55 .08
CC/F* 1 211.95 211.95 2.92 .10
FD/I 1 133.88 133.88 1.88 .19
Trt/Ctl x CC/F 1 115.48 115.48 1.59 .22
Trt/Ctl x FD/I 1 92.40 92.40 1.28 .27
CC/F x FD/I 1 13.53 13.53 0.19 .67
Trt/Ctl x CC/F x
FD/I 1 5.47 5.47 0.08 .79
Error 18 1304.89 72.50
-significant gc.l
*marginally significant g=.l
86