Citation
The effect of a supportive organizational environment on transfer of training in child welfare organizations

Material Information

Title:
The effect of a supportive organizational environment on transfer of training in child welfare organizations
Creator:
Brittain, Charmaine Renée
Publication Date:
Language:
English
Physical Description:
282 leaves : ; 28 cm

Subjects

Subjects / Keywords:
Child welfare workers -- Training of ( lcsh )
Transfer of training ( lcsh )
Child welfare workers -- Training of ( fast )
Transfer of training ( fast )
Genre:
bibliography ( marcgt )
theses ( marcgt )
non-fiction ( marcgt )

Notes

Bibliography:
Includes bibliographical references (leaves 269-282).
Statement of Responsibility:
by Charmaine Renée Brittain.

Record Information

Source Institution:
University of Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
45141427 ( OCLC )
ocm45141427
Classification:
LD1190.P86 2000d .B75 ( lcc )

Downloads

This item has the following downloads:


Full Text
THE EFFECT OF A SUPPORTIVE
ORGANIZATIONAL ENVIRONMENT ON
TRANSFER OF TRAINING IN
CHILD WELFARE ORGANIZATIONS
by
Charmaine Renee Brittain
B.S., Indiana University, 1984
M S W., Denver University, 1991
A thesis submitted to the
University of Colorado at Denver
in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
Public Affairs
2000


This thesis for the Doctor of Philosophy
degree by
Charmaine Renee Brittain
has been approved
by
Linda deLeon
Peter deLeon
Date


Brittain, Charmaine Renee (Ph D., Public Affairs)
The Effect of a Supportive Organizational Environment on Transfer of Training
in Child Welfare Organizations
Thesis directed by Associate Professor Linda deLeon
ABSTRACT
Training is a tool public managers use to more efficiently and effectively manage
child welfare organizations. Training is not always transferred to on-the-job
performance however. Transfer is thought to be contingent upon three sets of
factors: trainee characteristics, training design, and organizational environment
characteristics. In this study, three types of factors in the organizational
environmentorganizational climate, supervisor support, and co-worker
supportwere explored to understand their impact on retention, a measure of
transfer. The research project is a higher-level evaluation of the Specialized
Interviewing Skills (SIS) training for new child welfare workers
Training does indeed improve interview skills performance as evidenced by the
results on the pretest interview, the training interview, and the posttest interview.
It was hypothesized that different types of organizational environments
(supportive and unsupportive) towards training, would result in differences in the
retention of interview skills at the posttest. Using multiple regression analysis,
the results did not support the hypotheses. In fact, the results suggest that there is
little variability in employees and supervisors perceptions of their organizational
support for training that this variable seems to have no explanatory power. An
attitude of supportiveness toward training may instead be part of the
organizations culture. While the hypotheses were not supported, there were
important benefits to trainees for attending the training. Other data gathered
indicated that some variance in the self-reported transfer composite variable could
be explained by the organizational environment variables. These results lead to


the conclusion that trainees' motivation to use the training, perceived ability at
applying the skills, and the opportunity to use the training back at the job were the
most salient variables for achieving perceived transfer of training for this group of
caseworkers. Because the opportunity to use the training is an organizational
environment characteristic, the implication is that if transfer of training is to
occur, training must be relevant to caseworkers job responsibilities and they must
be able to use the training once they return to their jobs. The research project also
concludes with some recommendations for training managers and for future
researchers.
This abstract accurately represents the content of the candidates thesis. I
recommend its publication.
Signec
.mda deLeon
IV


DEDICATION
To my two families:
To the family I came from: my parents, Charles and Judy Brittain who inspired
me to go for the pot of gold at the end of the rainbow.
and
To the family we made: Jeff, Devyn Rose and Colton Brittain Frasier who showed
me the way over the rainbow.
Both have their place.


ACKNOWLEDGEMENTS
Thanks to Linda deLeon, for talking me through the inevitable bumps in the
dissertation road and providing insightful and helpful comments throughout the
draft iterations. Special thanks to Cindy Parry for perceptive guidance of all sorts
throughout the project and Jody Fitzpatrick for statistical assistance. Also, a
heartfelt thank-you to Jane Berdie, for raising the bar by her own example and
being a thoughtful and sagacious mentor.
Special thanks to Art Atwell and the Colorado Department of Human Services for
funding the project on which this thesis is based. Last, but certainly not least, a
heartfelt thank-you to everyone at American Humane Association and University
of Denver, Graduate School of School Work who worked on this project. It was a
pleasure to work with such an esteemed group of individuals.


CONTENTS
Figures...................................................................xi
Tables...................................................................xii
Chapter
1. introduction........................................................... 1
1.1 Importance of Public Management....................................... 1
1.2 Background Information.................................................3
1 2.1 Description of Training.............................................6
1.3 Public Management and Organizational Development.......................8
1 .4 Research Questions..................................................17
2 Literature Review......................................................19
2 1 State of Training....................................................19
2.2 Training Evaluation..................................................24
2.2.1 Training Evaluation Models..........................................32
2.3 Transfer of Learning.................................................38
2.3.1 Conducting Transfer Evaluations.....................................42
2.3 2 The Baldwin and Ford Model of Transfer of Learning.................45
2.4 Organizational Culture...............................................62
2.5 Statement of the Problem.............................................66
2.5.1 Sample Size.........................................................71
2 5 2 Type of Subjects...................................................7!
2.5.3 Equivalency of Training Programs....................................72
2.5.4 Criterion Measures of Performance..................................74
2.5.5 Experimental Rigor.................................................76
2.5.6 Rationale for Current Research Study...............................76
3 Methodology............................................................76
3.1 Research Project....................................................76
3 2 Research Design .....................................................76
3.3 Research Subjects....................................................76
3.3.1 Child Welfare Caseworkers Characteristics...........................76
3.3.2 Supervisor Characteristics..........................................76
3.4 Instruments..........................................................76
3.4 1 Interview Skill Assessment Instrument..............................76
3.4.2 Satisfaction Survey.................................................76
3 4.3 Follow-up Questionnaire............................................76
3 4.4 Supervisor Survey..................................................76
3.4.5 Qualitative Interviews..............................................76
3 5 Procedures...........................................................76
3 5 1 Recruitment........................................................76
3.5.2 Pretest of Interview Skills........................................76


3.5.3 Training Interview Skills Assessment................................76
3.5.4 Posttest Interviewing Skills Assessment............................76
3.5 5 Follow-up Questionnaire............................................76
3.5 6 Supervisor Questionnaire...........................................76
3.5.7 Qualitative Interviews.............................................76
3 6 Data Analysis.........................................................76
3 6.1 Composite Variables.................................................76
3.6.2 Confirmatory Factor Analyses........................................76
3 6 3 Statistical Analyses...............................................76
4. Results................................................................76
4.1 Results from Satisfaction Survey.....................................76
4.2 Results from Follow-up Questionnaire.................................76
4.2.1 Intervening Factors on Follow-up Questionnaire......................76
4.3 Results from Supervisor Questionnaire................................76
4.4 Comparison of Supervisor and Trainee Questionnaire...................76
4.5 Results from Composite Variables.....................................76
4.5.1 Relationships between Trainee and Supervisor
Questionnaire Composite Variables.........................................76
4.6 Agreement Regarding Video-tape Viewing...............................76
4.7 Test Scores Results..................................................76
4.7. i Raw Score Results..................................................76
4.7.2 Rasc'n Score Results...............................................76
4 7.3 Pretest and Training Test Scores as Predictors of
Posttest Score............................................................76
4 8 Primary Analysis of Major Hypotheses
(Retention Predicted by Organizational Environment)...................... 76
4.8.1 Relationships between Retention Measures,
Test Scores and Key Variables.............................................76
4.8.2 Multiple Regression Analysis of Retention with
Hypothesized Predictor Variables..........................................76
4.9 Secondary Analysis of Other Transfer Measures
(Test Scores Predicted by Organizational Environment).....................76
4 9.1 Relationships between Retention Measures,
Test Scores and Self-Reported Transfer....................................76
4.9.2 Relationships between Adjusted Test Scores and
Organizational Environment Variables Adjusted
Training Score............................................................76
4.9.3 Multiple Regression Analysis of Self-Reported
Transfer Predicted by Organizational Environment..........................76
4.10 Results from Telephone Interviews with
Research Subjects.........................................................76


4.11 Concluding Observations............................................76
5 Discussion.............................................................76
5 1 Overview of Significant Findings.....................................76
5.2 Alternative Explanations for Transfer of Training...................76
5.2.1 Relationship between Adjusted Retention and
Predictor Variables.......................................................76
5.2.2 Relationship between Adjusted Training Test
Score and Predictor Variables.............................................76
5.2.3 Relationship between Adjusted Posttest Score and
Predictor Variables.......................................................76
5.2.4 Relationship between Trainee Self-Reported
Transfer and Predictor Variables..........................................76
5.2.5 Differences in Opportunity to Use..................................76
5.2.6 Subjects Emotional Reactions.......................................76
5.2.7 Other Types of Transfer are Occurring..............................76
5.2.8 A Consideration of Culture..........................................76
5.2.9 Summary of Alternative Explanations.................................76
5.3 Consideration of the Relevant Literature.............................76
5.4 Study Limitations....................................................76
5 4.1 Selection Bias.....................................................76
5.4.2 Time Frame..........................................................76
5.4.3 Additional Training or Research.....................................76
5 4.4 Videotape Raters....................................................76
5 4 5 Test Conditions.....................................................76
5 4 6 Questionnaire Design................................................76
5.4.7 Comparison of Problem Statement Criteria and this Research.........76
5.4.8 Summary of Study Limitations........................................76
5.5 Implications of the Study for Professional Practice.................76
5 5.1 Reconsideration of Training Curriculum..............................76
5 5.2 Reconsideration of the Training Participants.......................76
5.5.3 Reconsideration of the Interview Instrument and Videotape Rating...76
5.5.4 Reconsideration of the Transfer Instrument..........................76
5.5.5 General Recommendations for Training Managers.......................76
5.6 Recommendations for Further Research.................................76
5.7 Conclusion...........................................................76
Appendix
A. Description of Coordination between Research Projects.................76
B. Specialized Interviewing Skills Interview Evaluation..................76
C. Satisfaction Survey...................................................76
D Trainee Follow-up Questionnaire.........................................76
E. Supervisor Follow up Questionnaire....................................76
IX


F. Transfer of Learning Qualitative Interview Questions...................76
G. Recruitment Letter and Consent Form....................................76
H SIS Data Collection Plan.................................................76
1 Task Letter.............................................................76
References.................................................................76
x


FIGURES
Figure
2.1 American Humane Associations Model for Strategic
Evaluation................................................................36
2.2 Baldwin and Fords Model for Transfer of Training...................47
4.1 Comparison of Rasch Adjusted Scores for Pretest,
Training, and Posnest.....................................................76
5.1 Graphical Representation Comparing Opportunity to Use................76
xi


TABLES
Table
2 1 Comparison of Training Evaluation Models...........................33
2.2 Summary of Transfer and Organizational Environment
Research Studies........................................................67
3.1 Research Subjects and Current Job Titles............................76
3.2 Research Subjects and Unit Assignments..............................76
3.3 Educational Levels of Research Subjects (Trainees).................76
3 .4 Ethnicity of Research Subjects (Trainees).........................76
3.5 Supervisors Current Unit...........................................76
3 6 Data Collection Points and Instruments.............................76
3.7 Ratings Adjustment Calculation.....................................76
3.8 Definitions of Composite Variables.................................76
3.9 Factor Loading of Items Contained within Factor 1:
Transfer/Used training and Previously Designated
Composite Variable......................................................76
3.10 Factor Loading of Items Contained within Factor 2:
Organization Support and Previously Designated
Composite Variable......................................................76
3.11 Factor Loading of Items Contained within Factor 3:
Supervisor Support and Previously Designated
Composite Variable......................................................76
3.12 Factor Loading of Items Contained within Factor 4:
Trainee Characteristics and Previously Designated
Composite Variable......................................................76
3.13 Factor Loading of Items Contained within Factor 5
Co-Worker Support and Previously Designated
Composite Variable......................................................76
3.14 Research Questions, Hypotheses, Variables, and
Statistical Tests.......................................................76
4 1 Responses from Questions Related to Workshop Content...............76
4 2 Trainees Perceived Achievement of Training
Competencies............................................................76
4.3 Trainee Perceptions of Trainers Ability...........................76
xii


4.4 Descriptive Statistics from the Trainee Follow-up
Questionnaire.............................................................76
4.5 Descriptive Statistics of Intervening Factors from the
Trainee Follow-up Questionnaire...........................................76
4.6 Descriptive Statistics of Supervisor Survey..........................76
4.7 Comparison of Supervisor and Trainee Means for
Questionnaire Items.......................................................76
4 8 Descriptive Statistics of Composite Variables......................76
4.9 Correlations between Trainee and Supervisor
Reported Trainee Variables................................................76
4.10 Correlations between Trainee and Supervisor
Reported Organizational Variables.........................................76
4.11 Individual Items Composing the Organizational
Support Variable..........................................................76
4.12 Comparison of Pretest, Training and Posttest Raw
Interview Scores Adjusted for Mean Differences in Raters................76
4 13 Comparison of Pretest, Training and Posttest Rasch
Interview Scores Adjusted for Mean Differences in Raters................76
4.14 Paired Samples T-Test for Pretest, Training, and Posttest..........76
4.15 Multiple Regression Results with Adjusted Posttest Score
on Adjusted Pretest Score and Adjusted Training Score.....................76
4 16 Correlations of Key Variables and Adjusted Retention from
Training to Follow-up.....................................................76
4.17 Multiple Regression Results of Retention on Trainee-
Reported Organization Support, Supervisor Support,
and Co-Worker Support.....................................................76
4.18 Relationships between Retention Measures,...........................76
Test Scores and Self-Reported Transfer....................................76
4.19 Relationship Between Training and Posttest Scores
for Trainee Reported Organizational Environment Variables.................76
4.20 Multiple Regression Results with Training Test Score on
Trainee-Reported Organization Support, Supervisor
Support, and Co-Worker Support............................................76
4.21 Multiple Regression Results with Adjusted Posttest
Score on Trainee-Reported Organization Support,
Supervisor Support, and Co-Worker Support.................................76
4.22 Relationships between Trainee Reported Transfer and
Key Variables.............................................................76
4.23 Multiple Regression Resuits with Trainee Self-
Reported Transfer on Trainee-Reported Organization
Support, Supervisor Support, and Co-Worker Support........................76
xiii


I
5.1 Correlations between Adjusted Retention and
Predictor Variables.........................................................76
5.2 Correlations between Adjusted Training Test Score
and Predictor Variables.....................................................76
5.3 Multiple Regression Results with Adjusted Training
Score and Predictor Variables...............................................76
5.4 Correlations between Adjusted Posttest Score and
Predictor Variables.........................................................76
5.5 Correlations between Trainee Self-Reported
Transfer and Predictor Variables............................................76
5.6 Multiple Regression Results with Trainee Self-
Reported Transfer and Predictor Variables...................................76
5 7 Comparison of Mean Adjusted Training Test Scores
and Retention by Opportunity to Use.........................................76
5.8 Potential Transfer Interventions by Person and Time Period.............76
xiv


1 Introduction
1 1 Importance of Public Management
Governments exist, amongst many other mandates, to protect their most
vulnerable citizens, especially children, from harm. Child welfare organizations,
as an arm of government, represent one set of institutions charged with that
important obligation of protecting children from abuse and neglect. While
carrying out their duties, child welfare organizations are subject to the same
scrutiny and oversight as other public organizations and are accountable to the
public for how tax dollars are spent and duties performed Public management
strives to make public organizations more effective and efficient for the common
good; public management and more specifically, organizational development, has
long been concerned with whether management (things managers do) really
matters in improving the efficiency and effectiveness of public organizations.
Public child welfare organizations prepare child welfare caseworkers for their
important jobs of protecting children through organizational development
techniques, one of which is training child welfare caseworkers on the skills
necessary to perform their jobs. Training, however, is only effective if it is
transferred to on-the-job performance, otherwise it is waste of the publics
1


investment and the organizations attention. Public agencies are always under
pressure to improve efficiency and therefore the more that can be understood
about the problem of training transfer, the better for public organizations.
Wilson made the call to study more effective techniques for administration
(Wilson, 1887) and many scholars and practitioners have answered that charge
over the last hundred-plus years. In that spirit, this research project seeks to
explore the contextual factors affecting the transfer of training by evaluating a
course for child welfare caseworkers to teach them specialized interviewing skills
for investigating and assessing situations in which children may have sexually
abused. Specifically, the research project studies the transfer of training from the
classroom to the job and the factors that facilitate or inhibit that transfer.
This research project was coordinated with the Colorado Department of
Human Services (CDHS), American Humane Association (AHA), and University
of Denvers Graduate School of Social Work (DU-GSSW). A description of the
coordination between the three organizations and the roles for each can be found
in Appendix A. This author has been involved in some aspect of the project for
approximately ten years, working at various times for all three organizations. At
this time, she is currently employed by AHA as part of their independent team of
third-party evaluators. While she has had a long history with the project at the
state level, this research was primarily conducted to determine whether there is a
relationship between county-based organizational support for training and
o


retention of interviewing skillsan area in which this author has not worked nor
had any involvement and thus is not subject to undue bias. To avoid even the
appearance of a conflict of interest, measures were taken by AHA and DU-GSSW
at the beginning of the project to ensure balanced results. For instance,
videotapes were rated by DU-GSSW employees (to be explained later) and the
test scores entered into AHAs database by other AHA employees. A more
detailed explanation of the authors history with the project can be found in
Appendix A.
1.2 Background Information
Interviewing, and specifically sexual abuse interviewing, is an important
and necessary skill that child welfare caseworkers must possess in order to do
their jobs effectively. The sexual abuse interview is different from other child
welfare interviews for several reasons. By its nature, the topic is uncomfortable
for both the interviewer and the child since typically children do not want to talk
about this taboo subject and interviewers may not want to hear about it. Secrecy
is one of the dynamics of sexual abuse that perpetrators use to keep themselves
from being exposed, and any kind of sexual contactespecially with children
between family members is extremely uncomfortable to discuss and hear about.
Specialized interview techniques should be used and practiced to make the
interview more comfortable for each person and to ensure the legal integrity of
3


the interview for potential subsequent legal proceedings to protect the child and
prosecute the perpetrator(s). Sexual abuse cases are more likely to go to both
civii and criminal court than other types of abuse and neglect cases and therefore
are more vulnerable to dismissal or other legal compromises if there is insufficient
attention to professional standards and protocols. For instance, a criminal court
judge may dismiss a case if a caseworker used leading questions to obtain the
information about the sexual abuse incident. Lastly, the initial sexual abuse
interview sets a context for future treatment. The child must not be frightened or
traumatized by the interview and/or subsequent interaction with Social Services,
or psychological treatment to help the child deal with the abuse may be
compromised.
Before this type of training was offered, caseworkers had little formal
opportunity to develop their interviewing skills at a formal training, especially the
specialized skills necessary to conduct comfortable and productive sexual abuse
interviews. These interviewing skills can be generalized to assist with other types
of interviews, such as those to assess for physical abuse or neglect and can be
used to facilitate ongoing case management interactions with all family members.
CDHS recognized the importance of good interviewing skills and
particularly the specialized skills necessary to perform sexual abuse interviews; in
1985, CDHS partnered with DU-GSSW under the auspices of a federal three-year
grant to develop and implement the Specialized Interviewing Skills (SIS) training
4


to teach caseworkers the skills necessary to interview children who may have
been sexually abused. An interview protocol based upon good practice research
and legal requirements was developed and became the foundation for the SIS
training curriculum. Also, the project developed a novel approach to training at
that time by videotaping the caseworker as he or she conducted a simulated
interview with an actor who role-played a child sexual abuse victim. The
videotaped interview assessed skill performance and was an integral part of the
curriculum by providing an opportunity to evaluate the interview and give
extensive feedback on interview performance to the trainee. Also, the project
developed a formal Interview Skills Assessment Instrument (henceforth called
Interview Instrument) based upon the interview protocol to measure skill
performance during the videotaped interview. Live videotaped interviews with
children were not used because of ethical issues, lack of equivalent situations,
potential discomfort, and even trauma to a child Therefore, it was decided to use
simulated interviews with an adult role-playing a latency-age child
After the federal grant expired, CDHS remained committed to the SIS
training and integrated it into their schedule of classes for new child welfare
caseworkers. CDHS currently contract with DU-GSSW to provide the SIS
training, coordinate the actors to role-play the victim, and rate the videotapes
using the Interview Instrument. The trainer who originally developed the SIS
training during the 1980s remains the primary trainer, with another trainer
5


occasionally substituting. This author also worked as a graduate research
assistant on the original project from 1989 through 1992, role-playing the child
sexual abuse victim and rating the videotapes. Currently, the author is a staff
person on AHAs third-party evaluation team (for a more complete description of
her involvement in the SIS training over the last ten years, see Appendix A).
I 2.1 Description of Training
The SIS training is a three-day course to teach caseworkers how to
conduct a sexual abuse interview using the formal interview protocol. The
interview protocol is designed to increase accurate disclosure, minimize
discomfort to the child, and help to raise the legal integrity of the interview by
providing specific guidelines for the interviewers conduct during the interview.
During the first day and a half of training, the interview protocol is
presented and practiced in various situations in the classroom. On the afternoon
of the second day, the trainees perform a simulated interview with an adult who is
role-playing a iatency-age child who has been sexually abused as dictated by a
standardized scenario. During the original project development phase at DU-
GSSW, graduate students role-played the child sexual abuse victim as part of the
research project. Currently, professional actors role-play the sexual abuse victim.
Eight scenarios of approximately equal difficulty and sexual abuse severity were
developed and are rotated during the trainings to give trainees exposure to
6


different situations and types of children (e g., shy, reluctant children versus
verbal, forthcoming children) Prior to the videotaped simulated interview, the
trainees are given an intake form containing basic information about the child, his
or her family, and the presenting situation. The trainees task is to establish
rapport with the child, determine w'hether sexual abuse occurred, and if so, obtain
the details of the incident(s). The trainer reviews each videotaped interview
during the evening of the second day and selects strengths and weakness of each
interview On the third day of training, the trainer presents clips from each
videotape to illustrate important interviewing concepts, reinforce positive
interview' skills, and help correct skill deficiencies. Next, the videotapes are sent
to DU-GSSW for formal rating by a trained graduate research assistant using the
Interview Instrument Once rated, the completed Interview Instrument rating and
additional written feedback are sent back to the trainees for review with their
supervisor to help them further enhance and refine their sexual abuse interviewing
techniques. While the feedback is sometimes excruciatingly painful for trainees
to hear, these multiple sources of feedback are an effective method for developing
skills so caseworkers can perform better interviews w'ith children. Better
interviews mean that situations are assessed more accurately and children are
better protected from harmthe public child welfare organizations main
responsibility.
7


Public management is the tool used to manage child welfare organizations
and has a long tradition striving to more efficiently and effectively administer
organizations for the public good. The following section contains highlights of
the literature related to the development of public administration and specifically
public management, with particular emphasis on organizational development.
1.3 Public Management and Organizational
Development
The early public management literature was concerned with improving the
concrete aspects of management and understanding the nature of bureaucracy and
administration. Scientific management was developed by mechanical engineers
to improve employee motivation (Martin, 1989). The scientific management
method minutely analyzed every task performed by the worker with time-motion
studies in order to increase efficiency. One of the founders of the scientific
management movement, Frederick Taylor (1992), said it was managements
responsibility to develop the systems and heuristics that would save the worker
and the company time and money. Managers also were responsible for personnel
duties such as selection and training. According to March and Simon (1958),
Taylor analyzed the interaction between the characteristics of humans and the
social and task environments created by organizations and thus laid the
foundation for contemporary' organizational development.
8


In contrast to scientific managements mechanistic efficiency, the human
relations school attempted to thaw managements icy grip on the workers. Gantt
(1916) was, to some, the first humanist and utilized the field of psychology to
improve employee motivation. Workers and management are each mutually
dependent upon the other, he suggested, therefore work morale is imperative to
organizational success. Training of all workers is one such way of attaining
mutual goals (Gantt, 1916). Gilhreth (1914) linked the scientific management and
psychology fields by applying psychological principles to management. The
Hawthorne experiments created the humanist school within the study of
administration, promoting the idea that social leadership skills as important as
technical knowledge (Martin, 1989). Foilett (1926) further tried to humanize the
relationship between management and worker. She dispensed with the giving of
orders and instead argued for uniting around a particular situation and letting the
situation dictate the solution.
The nature of organizational boundaries was defined by theorists including
Barnard and Simon. Barnard (1938) defined the symbiotic relationship between
informal and formal organizations. Informal organizations set the cultural context
within a formal organization by establishing customs, norms, folklore,
institutions, and ideals. Inattention to the informal organization weakens the
formal organization. Simon (1945) said that constructing an efficient
administration meant hiring an operative staff, then superimposing on that staff a
9


supervisory staff to influence the operative staff to coordinate behavior.
Employee decisions and behaviors are influenced within and by the organization.
Employees can be influenced two ways in order to carry out decisions made by
the organization. The first process indoctrinated the employee into the ways of
the company or trained him, while the second process used authority. Simon
(1945), one of the preeminent theorists of public administration, explicitly stated
the importance of training to the entire administrative process.
In the mid-twentieth century, public administration, and specifically public
management, scholarship went down several different paths including leadership
theory, personnel management, decision-making theory', fiscal management, and
organizational development. These paths often intertwined, as public
management scholars tried to understand the process of management in the
continuing effort to make it more efficient and effective.
Drucker (1954) was among the most significant contributors to theories of
organizational development. His contribution was far-ranging and had a profound
effect on both public and private management practice. Melding the precision of
the mechanical engineers with the compassion of the management humanists, he
developed a goal-driven management theory He minimized the scientific
management approach as simplistic and even unworkable and took a much more
holistic approach to management that set high standards in order to achieve goals.
Managements ultimate goal was business performance and there are three jobs of
10


management: managing a business, managing managers, and managing workers
and work Drucker wittily reminded us that you cannot hire a hand, as its
owner always comes with it (1954, p. 262) Therefore, in order for managers to
motivate an employee, they must attend to the whole person.
Many theorists explored the nature of the relationship between the
organization and the individual (for instance, Argyris, 1957, Golembiewski &
Eddy, 1978, March & Simon, 1958; McGregor, 1960). Organizational
development theorists emphasized the growth of the individual as a way of
achieving high business performance (Argyris, 1971, Golembiewski & Kiepper,
1988, Senge, 1990) Systems theory helped to explain how individuals reacted to
their environment and also offered approaches to improving management within
the context of an organizational system (Katz & Kahn, 1966; Thompson, 1967).
Argyris (1957) linked Barnard and McGregor (discussed below) in his
groundbreaking book on individuals and formal organizations, which tried to
explain why people behave as they do in organizations. Many of his contentions
are more clearly enunciated in McGregors Theory X and Theory Y management
styles. In McGregors Theory X, workers are essentially indolent Argyris,
however, believed workers were not lazy but rather are constrained to childlike
dependence; management practices designed to allow7 full expression of their
dignity and creativity will benefit both the worker and organization. Management
must be cognizant of both formal and informal organizations. Inherently, there is


an incongruity between the requisites of formal organizations and the self-
actualization of individuals (Argyris, 1957)
McGregor (1960) differentiated between Theory X and Theory Y
management. Most companies operated under Theory X management, believing
that workers were basically lazy, passive, resistant to change, and indifferent to
the organization. Theory Y, on the other hand, emphasized the competence and
self-direction of the worker, so management should encourage workers assuming
responsibility for the organization through such techniques as decentralization and
management by objective (McGregor, 1960).
March and Simons (1958) seminal book postulated the nature of
organizations and the relationship between individuals and organizations. It
argued that formal organizations do have a significant effect on the behavior of
individuals within organizations In particular, influence theory modeled the
behavior of individuals within organizations as functions of various inputs that
determine whether an employee will join and stay with an organization and, once
in it, whether he or she will or will not be productive. Perceived consequences of
an individuals actions are a function of the environment, individual
characteristics, sub-group pressures, and organizational rewards. All behavior
within organizations is bounded by the limits of individual rationality, which
results in satisficing (March & Simon, 1958).
12


Building on his earlier work, Argyris in Personality and Organization
(1971) further refined his theory of organization development. He borrowed
McGregors Theory X and Y management styles and added Patterns A and B.
The patterns are the personal behavior, group dynamics, and organizational norms
associated with Theory X or Y (Argyris, 1971, p. xi). McGregors theories were
about the worker and his or her motivations, while Argyris patterns were about
the organization. Argyris advocated a Theory Y, Pattern B approach, a much
more humanistic and holistic approach to management Organizational
development is the process by which this is achieved. He strongly believed in the
importance of organizational development for developing organizational health
and human potential and dignity.
Organizational development was touted as helping the organizations
efficiency and increasing the growth and satisfaction of employees
(Golembiewski & Eddy, 1978). The field of organizational development is
complicated due to its varied philosophical foundations of rationalism,
pragmatism, and existentialism and faces unique problems and challenges in
public agencies that have largely been ignored (Golembiewski, 1978). Four
properties specific to public institutions interfere with achieving organizational
goals: multiple access, greater variety of groups and individuals; command
linkages; and weak linkages between political and career levels (Golembiewski,
1978).
13


Golembiewski and Kiepper (1988) developed a more mature strategy for
organizational development reminiscent of the popular Peters and Waterman-
book, In Search of Excellence (1982). High performing units were characterized
by specific traits that emphasized high energy, commitment, team spirit and
action, among others Organizational development programs should attend to
both the individual and the organization, and the growth of each will result in
satisfaction, enhanced skill, and competence within interpersonal and intergroup
situations (Golembiewski & Kiepper, 1988). They described ten characteristics of
high performance settings from organization structure to physical lay-out and
included training as one of these characteristics (Golembiewski & Kiepper, 1988).
Many authors have presented prescriptions for improving management
techniques, but their effect often seems negligible as the same issues, often called
different names (e g., morale, motivation), arise over and over again. The
question plaguing public management in the late 1990s is, does public
management matter9 Theorists trying to answer this question want to know
whether the practice of management, such as different motivation techniques, or
training, can make a difference in terms of outcomes. The interaction between the
organizational culture, management practices, and institutional context predicts
the behavior of organizations (Behn, 1991; OToole, 1997; Khademian, 1997).
Managerial actions (or inactions) influence the structure, policy, and even culture
of organizations. Questions arise as to whether culture is a variable to be
14


manipulatedmuch like the childhood toy silly putty in order to achieve
organizational purposes, or whether it is a factor that pervades organizational life
with managers having little ability to control or change it (Khademian, 1997).
Some researchers say that the former interpretation of organizational culture is
valid and therefore justifies the role of management. Success lies in the extent of
the managers leadership efforts to create an organizational culture oriented
towards program outcomes (Behn, 1991). Practical how-to strategy books
preaching a principle-centered message dominated the popular business literature
during the last decade (for instance, Behn, 1991; Covey, 1991; Spencer, 1998,
Ziglar. 1994).
Senge (1990) moved the management literature into the 1990s with his
book on the learning organization. He tried to close Argyris gap between the
organization and the individual. Truly excellent organizations are those that tap
into individuals commitment and are oriented to learning at all levels, thereby-
creating an organizational culture dedicated to growth He labeled these work
environments learning organizations and argued that they are achieved by a
commitment to the fifth discipline, which is systems thinking, or the ability to
see interrelationships and patterns. Management is then oriented to the goal of
becoming and remaining a learning organization.
In another paper in the does management matter7 vein, OToole (1997)
stated that network contexts offer variables that are Khademians silly putty in
15


that they can be manipulated to achieved desired goals. Networks will replace the
old hierarchical system and are more realistic of todays management
environment.
Further research on organizational culture within an organizations
institutional context is necessary. In contrast to the methods of best practices
research, organizational cultures of well-performing and poorly-performing units
should be studied to determine the unique characteristics of each to inform
management decision-making (Khademian, 1997). Understanding organizational
culture is central to the future of public management:
Indeed, a reliance upon culture may be the only viable way to
manage some organizations. Discovering how deeply embedded
cultures might be managed (that of the IRS and the CIA come to
mind), or how to manage around such cultures, is a key challenge
for public management research (Khademian, 1997, p. 16).
The utility of managing culture, however, is dampened by the difficulty of
understanding its role within organizations and the connections between abstract
cultural concepts and the reality of the organizational environment (Khademian,
1997).
The public management literature traces a theoretical thread concerned
with developing an efficient and effective organization while attending to the
needs of individuals within organizations This body of literature is concerned
with determining the right management techniques in order to achieve
organizational performance. The does management matter17 public
16


management literature sets a managerial context for creating an organizational
culture to influence the transfer of training to on-the-job performance. Training is
but one method of many managerial techniques designed to increase the
efficiency and effectiveness of public organizations while recognizing the
importance and needs of individuals within organizations. It remains unclear,
however, whether training realiy does improve performance, which leads to the
questions this study seeks to answer.
14 Research Questions
This authors interest in understanding the role of training for improving
organizational effectiveness and the factors that might affect that, led to the
following research questions for this study:
First, does training improve interviewing skills performance ? Before
understanding the factors that influence the transfer of training, it is important to
establish that training does indeed improve interview skills performance by
examining scores on the pre, training, and posttest. It would be expected that
scores would be at their lowest at the pretest, improve significantly after skills are
learned at the training and tested at the training test, then scores would decrease
slightly over time for the posttest (the effect of forgetting), but still remain greater
than the pretest level People who learn more in training also perform better on
the job. If this should prove to be untrue, there would be no reason to test for the
17


effects of the transfer climate (Rouiller & Goldstein, 1993), the next area for
research
Second, which organizational factors are most influential in
promoting the transfer of training? The goal of the research project is to
understand the organizational factors while controlling for other factors
that may contribute to learning retention and thus transfer Three types of
factors characterize the organizational environment: organizational
climate, supervisor support, and co-worker support and will be explored to
understand their impact on transfer.
The reasons why these particular factors are the focus of this research will
be explained in the next chapter. The next chapter sets a context for these
research questions and discusses how and why the topic of the transfer of training
is important to the field of public management.
18


2 Literature Review
The literature related to training, transfer of learning, and culture is
explored in this chapter to establish a context for this dissertation Multiple
bodies of literature pertinent to the transfer of training are explored The
theoretical thread starts by discussing training, a common organizational
development intervention strategy. Next, the focus moves to a detailed discussion
of training evaluation and then assesses the literature specifically related to
transfer of training. The last section focuses on the literature related to research
studies of transfer of training and the organizational environment to validate the
reason for this research project and understand the rationale for its structure. Such
a broad academic exploration is necessary to understand the context of transfer of
training within contemporary public management research and theory'.
2.1 State of Training
It is estimated that American industries spend $100 billion annually on
staff development, $30 billion on formal training, and approximately $180 billion
on informal on-the-job training (Broad & Newstrom. 1992; Carnevale, Gainer, &
Villet, 1990, Georgenson, 1982, Gist, Stevens, & Bavetta, 1991). Training
expenditures continue to increase significantly each year but generally there is a
19


lag between training and performance (Bassi & Van Buren, 1999), thus making it
difficult to measure a return on investment (Abernathy, 1999). Given the
changing American workforce and strong economy, training will continue to play
a significant role in recruiting and retaining qualified employees (Facteau,
Dobbins, Russell, Ladd, & Kudisch, 1995) It might be expected that government
organizations would spend more on training due to two contrasting reasons: either
government is wasteful and spends too much money on training, or more
optimistically, that government invests more in training because workers need it
to serve the people more effectively Government organizations, however, rank
near the bottom in terms of training expenditures as a percentage of payroll (Bassi
& Van Buren, 1999).
Training and education can be used as a strategy to improve organizational
effectiveness and competitiveness (Saari, Johnson, McLauglin, & Zimmerle,
1988) by changing workers attitudes, improving their knowledge, and developing
their skills. After training, worker attitudes were shown to improve in several
studies (Chaffin, Kelleher, Harder, Harper, & Crone, 1994, Gregoire, 1984, OSG,
1995, Pecora, Dodson, Teather, & Whittaker, 1985) Other studies showed an
increase in knowledge before and after training (Chaffin et al., 1994; Cheung,
Stevenson, & Leung, 1991, Cronin & Drake, 1995; Schinke, Smith, Gilchrist, &
W'ong, 1981; Sullivan & Clancy, 1990). Others were able to determine that skills
20


developed as a result of training (Cheung et al., 1991, Cronin & Drake, 1995,
Osgood, 1995).
Training can improve the utilization of services such as substance abuse
treatment, because learning about the system and available treatment increases the
knowledge and comfort level of caseworkers seeking to use those services for
their clients (Chaffin et al., 1994, Sullivan & Clancy, 1990). A study about the
impact of training on child welfare caseworkers in Pennsylvania found that nine
fewer children died from child abuse than the previous year (Osgood, 1995), and
while it is difficult to associate those results with training, it certainly helps to
makes the case for training as an important management tool to help reduce child
abuse
Despite the positive benefits of training, seldom is it developed or
evaluated systematically so that training can be used to address the most
important questions facing an organization (Saari et al., 1988). Only one-third of
U S companies conducted some type of training needs assessment and 42 percent
of companies failed to conduct evaluations of training effectiveness (Saari et a!.,
1988). In contrast, educational institutions are noted for their emphasis on
accountability of learning through tests, grades, and evaluation while training in
industry (including government and non-profit sectors) is much less so occupied
(Baldwin & Magjuka, 1991). Kaufman & Keller (1994) blame less-than-effective
evaluation on three factors: evaluation models are too restrictive, evaluations do
21


not ask the right questions, and the relationship between ends and means is
unclear.
According to some researchers, only about 10 percent of training
expenditures actually result in transfer to on-the-job performance or, conversely,
90 percent of the information learned in training was not used by training
participants (Broad & Newstrom, 1992; Carnevale et al., 1990; Garavaglia, 1996,
Georgenson, 1987, Gist et al., 1991, Marx, 1986, Seaburg, 1982). In a study of
child welfare caseworkers, 28 percent of participants did not transfer training to
on-the-job performance (Curry, 1997). Training managers more optimistically
estimate that 40 percent of training was transferred to on-the-job performance
(Newstrom. 1985, Seaburg, 1982).
While companies may believe in the value of training as evidenced by the
enormous amount of money being spent on it, they do little to attend to important
training principles such as conducting needs assessments and rigorous
evaluations. In the literature specific to child welfare training, Vinokur-Kaplan
(1986) conducted a landmark national study in 1981 to evaluate child welfare in-
service training Results glaringly point to the weaknesses of child welfare
training programs during the late seventies and early eighties. Approximately
three-quarters of participants surveyed stated that they had a formal in-service
training program while the balance had either an informal program or none at all.
Factors most influencing the effectiveness of training from the Vinokur-Kaplan
22


study were: training relevance, a needs assessment to determine training needs,
incentives for worker participation, adequate agency support for training; on the
job follow-up of training content and sufficient and appropriate training materials
at the training. There was significant evidence that participants were only
moderately satisfied with the training program offered by their agency, indicating
that training programs could be much improved. The research also suggested that
it is important to consider the total environment when developing training
programs and encourage effective follow-up efforts that assess whether workers
implement what they have learned in the training on their jobs and the factors
influencing this transfer. In another perspective, Seaburg (1982) said the problem
with child welfare training is its current focus on process instead of systems issues
that are much larger than training, such as raising the level of education,
increasing salaries, assigning reasonable caseloads, developing structures to
prevent burnout, and utilizing specialized training in service delivery practice.
Despite positive findings on the benefits of training, generalizability of
training results to other populations is limited given various methodological issues
with the training evaluation (Cronin & Drake, 1995, Chaffin et ah, 1994, Cheung
et ah, 1991). Before delving into why training is effective or ineffective and
whether it encourages transfer of training, it is important to understand how it is
evaluated. The next section examines the training evaluation literature in detail,
with particular emphasis on child welfare training evaluation.
23


2.2 Training Evaluation
The training evaluation literature is divided first into the theory or model
literature that describes how evaluation should be constructed and, second,
literature that explores how those models or theories are applied. As already
discussed, it is clear that training has potential to have a positive effect on work
performance Controversy arises regarding the model or theory upon which the
evaluation is based, which in turn raises debate about the information collected,
testing protocols, (i.e., for knowledge, skill, or attitudes), and evaluation target
levels (e g., trainee satisfaction versus community impact). In both bodies of
literature, researchers vary widely in their preferred evaluation techniques. Some
advocate for knowledge testing throughout a training course, while others call for
transfer of training evaluations (such as the development of an action plan that is
later checked to see if trainees implemented what they learned in training). The
following section sets a context for training evaluation with particular attention
paid to child welfare training evaluation.
Almost thirty years ago, Campbell (1971) pronounced the training and
development literature to be voluminous, nonempirical, nontheoretical, poorly
written, and dull (p. 565). Unfortunately, while the voluminous label could be
contested, it does not appear to have changed greatly in this time period. Dionne
(1996) took a historical look at evaluation of training activities, saying that the
24


lack of consensus may be due to the lack of a unifying model and theory for
research. The literature as reviewed by Dionne, has achieved consensus regarding
three factors: 1) information learned in training may or may not be applied, 2) a
favorable organizational climate and probably more complex interventions than
training are necessary to ensure learning retention and transfer of knowledge, 3)
researchers, trainers, and managers use different standards and judge training
activities against their own criteria, thus the information they seek is not used for
the same purposes. Training evaluation is a component of an organizational
subsystem that is influenced by its organizational context (Dionne, 1996).
More attention is now given to child welfare training evaluation due to
several factors, including increasing scrutiny by media, legislators, child welfare
lawsuits, and worker certification requirements. Training evaluation can serve
many purposes, such as showing increased employee knowledge and skills,
achieving buy-in for agency values and mission, and providing feedback about
training structure and content (Parry & Berdie, 1999).
Factors to consider when designing a training evaluation include (Berdie
& England, 1994, p. 5):
* Level of evaluation (e.g., satisfaction, knowledge, skills)
* Cognitive domain (e.g., knowledge, comprehension, application)
* Rigor of test development (e.g., content validity, reliability, bias)


* Implementation (e g., test confidentiality, accuracy of scoring)
Impact of evaluation (e g., remedial work, curricula and trainer deficits).
Planning for evaluation should start at the beginning of the training initiative and
continue through each step in the training process (Parry' & Berdie, 1999). Good
evaluation is contingent upon a thorough, well-developed curriculum, clearly
defined learning objectives, and communication between the trainers and trainees
about the training content (Jones, Stevenson, Leung, & Cheung, 1995).
Levels of evaluation were originally proposed by Kirkpatrick and soon
became adopted as the standard model across disciplines for training evaluation
(Campbell, 1971, Phillips, 1991). Kirkpatrick (1959) proposed a training
evaluation schema with four levels for evaluating the quality of training:
1 Reaction Assess participants reactions to the training.
Measurements tools include happiness surveys and verbal feedback.
2 Learning Measure knowledge and skills gained at the training.
Measurement tools include paper and pencil knowledge tests, essay
exams, and simulations,
3 Behavior Measure how learning is applied to on-the-job
performance. Measurement tools include participant action plans,
record reviews, observation, and self-report surveys.
4. Results Measure results and/or outcomes from training.
Measurement may use empirical measures to determine organizational
26


outcomes such as determining whether fewer children are in out-of-
home placement due to training.
Forty years later, Kirkpatricks model still remains as popular as ever with
many training evaluation texts based on it (for instance, Basarab, 1992, Holcomb,
1994). This model has been the basis for most training evaluation designs and is
widely regarded and recommended in both the public and private sectors to
evaluate training (Abernathy, 1999; Alliger & Janak, 1989, Birnbrauer, 1987,
Krein & Weldon, 1994) Even when Kirkpatrick is not used as a reference, many
evaluation approaches inherently seem to adopt the levels approach. For instance,
Schinke et al (1981) presented a new technique for child welfare training
evaluation to determine workshop impact empirically with tests before, during,
and after training (i.e., level-two evaluation) Rae (1991) also discussed different
levels of evaluation, and while he did not arrange them in a specific typology, his
treatment bears a striking similarity to Kirkpatricks model. In a testament to the
conventionality of Kirkpatricks model, when researchers introduce new
evaluation models, they often first denigrate Kirkpatricks model for various
deficiencies as a way of justifying their own models existence (for instance, see
Holton, 1996; Kaufman & Keller, 1994).
In Kirkpatricks level-one evaluation, trainees fill out surveys assessing
their satisfaction with the training and trainers, hence, the term happiness
survey has become part of the training evaluation lexicon. Studies have shown,
27


however, that there is little correlation between how trainees feel about a program
as evidenced by responses on participant reaction forms and what they have
learnedor more importantly, what they will do on the job because of training
(Birnbrauer, 1987, Broad, 1997, Dixon, 1990; Russ-Efi & Zenger, 1985). These
findings support Kirkpatricks contention that a favorable reaction to a program
does not assure learning (Dixon, 1990. p. 137). It should also be noted that
trainees often underestimate their pre-training skills and exaggerate their post-
training performance because of a strong need to justify their attendance at
training (Yancey & Kelly, 1990).
Some researchers would dispense with the reaction level (level-one)
altogether as an evaluation focus (Holton, 1996, Kaufman & Keller, 1994,
Phillips, 1991, Russ-Eft & Zenger, 1985). Holton (1996) instead calls it an
intervening variable to consider when evaluating training program results. More
rigorous evaluation is necessary to enhance the training functions perceived
credibility and value (Yancey & Kelly, 1990).
In the 1980s, Kirkpatrick (1983) encouraged rigor in level-two evaluations
by advocating for pretest, posttest, control group designs to assess knowledge
Other researchers concerned about the integrity of evaluation research also called
for a more rigorous approach, warning against self-report of behavior change as it
may be unreliable and invalid (Campbell & Stanley, 1980, Jones et al., 1995,
Parry & Berdie, 1999, Phillips, 1991; Russ-Eft & Zenger, 1985) Campbell &
28


Stanley (1980) said that educational improvements can only be verified through
experimental design. Though the directive to conduct rigorous evaluations is an
admirable admonition, this advice has largely gone unheeded by training
evaluators due to cost and time factors (Jones et al, 1995).
Despite the depth of knowledge that can be gained by utilizing all four
levels of evaluation, training is typically evaluated at level-one of Kirkpatricks
model or the trainee satisfaction level (Birnbrauer, 1987, Brinkerhoff, 1987, Russ-
Eft & Zenger, 1985, Saari et al., 1988; Yancey & Kelly, 1990). A lack of
rigorous evaluation beyond the satisfaction level also was found to be the case in
child welfare training evaluation (Berdie & England, 1994, Jones et al, 1995).
Evaluation at Kirkpatricks levels three and four is rarely used because the
multiple influences on results related to transfer and organizational effects make
causal inferences difficult to isolate (Kaufman & Keller, 1994). Lack of rigorous
evaluation is attributed to attitudina! biases against training and a lack of
commitment or resources for designing valid and reliable evaluation instruments
(Jones et al., 1995). Testing for retention of knowledge poses significant
challenges due to a variety of factors including cost and the confounding effects
of intervening variables, such as experience and additional supervision (see
Cheung et al., 1991).
Testing for skills can be accomplished through the use of videotaping.
Video-based testing is closer to real job performance and thus is a valid predictor
29


of actual behavior on the job (Weekley & Jones, 1997). The use of simulations
for evaluation has several advantages: reproducibility, cost-effectiveness, and
safety considerations (Phillips, 1991), and they also can effectively evaluate
training (Holcomb, 1994).
More recently, Kirkpatricks model has undergone much criticism as
being unrealistic and unworkable (Alliger & Janak, 1989, Holton, 1996, Kaufman
& Keller, 1994) Holton (1996) said that Kirkpatricks four-level system of
training evaluation is more a taxonomy of outcomes and is flawed as an
evaluation model. A more appropriate model needs to specify outcomes
correctly, account for intervening variables, and indicate causal relationships
(Holton, 1996). In another criticism of Kirkpatricks model, Alliger and Janak
(1989) analyzed Kirkpatricks taxonomy and found three assumptions:
1 The levels are arranged in ascending value of information. A measure of
impact at the fourth level provides more information than that of trainee
reaction
2. The levels of evaluation are causally linked and there is a domino effect, if
you have positive reaction, then you get positive learning, then behavior
change, and so forth.
3 The levels are positively intercorrelated.
According to Alliger & Janak (1989), Kirkpatricks model would be more
appropriate as a global heuristic for training evaluation In fact, Kirkpatrick
30


(1996) claims this was the original intention of the model and said he is
ambivalent about whether his four levels of evaluation meets the criteria for a
model. Rather, it is important that the model be used to help clarify the meaning
of evaluation and offer guidelines on how to conduct an evaluation The model
has practical application rather than scholarly roots (Kirkpatrick, 1996).
Recently, there has been a move to elevate the evaluation literature and to
move beyond Kirkpatrick and into more sophisticated evaluation designs. A host
of researchers recommend other models of evaluation which encompass larger
goals (American Humane Association, 1996, Brinkerhoff, 1987, Holton, 1996,
Kaufman & Keller, 1994, Ottoson, 1997, Russ-Eft & Zenger, 1985; Stokking,
1996). One of these goals is behavior change and it requires the following
components: 1) persons must have a desire to change, 2) they must have the
necessary knowledge and skills to try the new behavior, 3) they must work for a
manager who allows or encourages change, 4) they must have help; and 5) they
must be rewarded for change. It is necessary to have a systematic appraisal of the
organizational environment before and after training by the person receiving the
training, their superiors, subordinates, and peers (Kirkpatrick, 1983).
In contrast (and perhaps from the corporate sector), other models advocate
a return to simplicity in order to spend more money on training and less on
evaluation (Abernathy, 1999). Some of these models, both simple and complex,


are explored in the next section to illustrate the breadth of evaluation model
designs available in the literature
2 2 1 Training Evaluation Models
Stokking (1996) analyzed several popular evaluation models that prescribe
how training should be evaluated and found them all to be lacking but argued that
a dialogue about the utility of various evaluation models would more likely lead
to results. In that spirit, the following section explores different training
evaluation models. A sampling of the more popular evaluation models are
presented in Table 2.1 to illustrate differences in approaches and strengths and
weaknesses of each. Phillips (1991) described seven evaluation models in his
book that are all variations of the same theme: a four-level approach with an
emphasis on calculating the value of training and the potential return on
investment. The model developed by Gordon (1987), also labeled by him as the
IBM approach, (designated as such as that is where it was first applied) was
chosen to represent all models in this genre. Also represented is Kirkpatricks
classic evaluation model, a model designed for human services agencies
(American Humane Association, 1996), a model that is less a model and more of
task list (Rae, 1991), and three models that evolved from Kirkpatricks model
(Brinkerhoff, 1987, Holton, 1996, Kaufman & Keller, 1994). Each model was
assessed for the potential rigor of the evaluation efforts. More rigorous
32


evaluations are those that at least have the potential for experimental design and
control for various factors that might influence outcomes
Table 2.1 Comparison of Training Evaluation Models
Author Components Most appropriate organization Focus Strengths Weaknesses Poten- tial Rigor
American Humane .Assoc.. 1996 1. Formative evaluation 2. Satisfaction 3. Opinion 4. Knowledge acquisition 5. Knowledge comprehension 6. Skill demonstration 7. Skill transfer 8. Agency impact 9. Client outcomes 10. Community impact Human Services agency Built from levels of learning. Levels revolve around distance from training, emphasis is on measurement of these factors. .All dimensions of training impact can be captured. Levels of learning are broken down. Often neglected formative evaluation is included. Difficult to operationalize higher levels. May be overwhelming to try to address all levels. Also, may not be economically feasible. High
Brink er- hoff. 198"7 1) evaluate needs and goals: 2) evaluate training design; 3) evaluate operation; 4) evaluate learning: 5) evaluate usage and endurance of learning; and 6) evaluate payoff Oriented towards business First part of model focuses on the training itself and then results. Focuses on areas within the training designer's control, simple to implement. Lack of specificity in higher levels of evaluation. Trainee learning is accreted. Mediu m
Gordon. 1987 (IBM approach) 1. Reaction 2. Testing 3. Application 4. Business Results Business organization Return on investment. Stresses value of training and return on investment. Lack of specificity in model, levels are broad. Low
Holton. 1996 Primary and secondary influences Outcomes: 1. Learning 2. Individual performance 3. Organizational results Any organization (because of its complexity, may only be viable for large organizations) Outcomes from organization perspective, contextual focus. Model is comprehensive and accounts for most factors. All or nothing, so may not be economically feasible to implement. Levels too broad to suggest an implementation strategy. High
33


Table 2.1 (Corn.)
Author Components Most appropriate organization Focus Strengths Weaknesses Potenti al Rigor
Kirkpatrick. 1959 1. Reaction 2. Learning 3. Behavior 4. Results Any organization of any size. Outcomes from trainees' point of view. Model is simple. Levels are too broad. Difficult to operationalize fourth level. Higher levels are not appropriate for "soft skills". Mediu m
Rae. 1991 No levels, sequential process from identifying a problem to long- term evaluation follow-up (laundry list of tasks). Any organization cf any size. The evaluator's tasks to effectively evaluate. Offers step by step instructions. Approaches evaluation as one size fits all.'' Xo flexibility in evaluation. Prescriptive. Low (goes to Kirkpat rick's 3d level).
Evaluation models vary in their complexity from simple (for instance,
Rae, 1991) to complex (for instance, Holton 1996). Their complexity is a
function of the considerations the training evaluator must attend to when
designing evaluations and the amount and type of information obtained when the
evaluation is complete. More complex models consider a wider degree of
stakeholders and assess for the mega-effect of training courses. Kaufman and
Keller (1994) said a more effective approach to evaluation should encompass an
expanded concept of evaluation, one that includes result-related questions that
contribute to continuous improvement by comparing intentions with results. The
evaluation design should also incorporate other interventions such as strategic
planning, organization, and development (Kaufman & Keller, 1994).
The degree of complexity directly influences the potential rigor of data
collection and analysis from the training evaluations. More rigorous training
34


evaluations assess the training programs at multiple levels with increasing
complexity of training designs For instance, the Holton (1996) model requires
that intervening variables, such as ability, motivation, and environment, be
measured as well as the more conventional, yet fairly sophisticated, measurements
of trainee performance and organizational results, thus were rated high for
potential evaluation rigor. While admirable in theory, the cost of conducting such
an evaluation may be prohibitive for many organizations AHAs (1996) model,
in contrast, encourages a complete evaluation but recognizes the difficulty of
measuring outcomes such as community impact, so it acknowledges that training
evaluation design must be flexible depending upon information needed and
resources allowed .AHAs mode! (see Figure 2.1 for the conceptual rendering of
this model) incorporated Blooms taxonomy of educational objectives (i.e.,
cognitive, affective, and psychomotor) and significantly expanded Kirkpatricks
levels of evaluation (American Humane Association, 1996, Parry & Berdie,
1999)
35


Figure 2.1 American Humane Associations Model for Strategic Evaluation
Note. From A strategic plan for training evaluation in Colorado (Working Paper), (p. 9) by
American Humane Association. Englewood. CO: Author. Copyright 1997 by the American
Humane Association. Reprinted with permission.
Raes (1991) model also uses a menu approach but evaluation only goes so far as
skills testing over time.
In 1989, Brinkerhoff (19S9) augmented his model by delineating three
zones within the training processbefore, during, and after trainingthat can
provide an opportunity for interventions that would increase training effectiveness
(also advocated by Curry et al., 1994).
36


The evaluation models can also be categorized by the type of organization
for which the model is most appropriate given its components. AHAs model was
specifically designed for human service agencies that are most concerned about
the effectiveness of training and its impact More business-oriented models, eg.,
the IBM approach, are simple to implement and focus on the bottom line.
Other models, (for instance, Kaufman & Keller, 1994, Kirkpatrick, 1959) are
appropriate for any type of organization, whether public or private and of any
size.
Selecting the most appropriate evaluation model is contingent upon the
organizations capacity to implement the training evaluation and information
needed to make decisions. Unfortunately, few organizations deliberately choose
an evaluation model but instead may succumb to tradition and familiarity.
Purposeful evaluation, especially to assess transfer of training is not a
predominant feature of current evaluation practice CDHS, however, has bucked
this trend and has committed resources to ensuring that training does indeed work
They have adopted the AHA model to guide training development,
implementation, and evaluation.
The next section reviews the literature related to transfer of training, also
known as Kaufman and Keller and Kirkpatricks level-three, AHAs level-seven,
and so forth.
37


2.3 Transfer of Learning
Most of the literature regarding factors that contribute to the transfer of
learning or training comes from the business, psychology, or education fields.
Broad and Newstrom (1992) offered this definition of transfer of training (or
learning)1.
[T]he effective and continuing application, by trainees to their
jobs, of the knowledge and skills gained in trainingboth on and
off the job (p 6).
Kozlowski said; Transfer is the core issue with respect to linking individual
change to the requirements of the organizational system (1997, p. 255). Human
services, and more specifically the field of child welfare, has borrowed the
transfer of training literature from these other fields in an effort to increase the
effectiveness of training.
Transfer of learning has a long research history. Much of the early
transfer literature relates to its cognitive processes. Schema theory is often used
as the theoretical framework for understanding transfer, it says that transfer
depends on the activation of relevant anticipatory schemata (Macaulay & Cree,
1999; Reed, 1993, Salomon & Perkins, 1989) In 1903, Thorndyke discussed the
theory of identical elements, stating that information about how to do one task
1 The terms transfer of learning and transfer of training are equivalent and both are used
throughout the thesis.
38


can be applied to another (Clark & Voogel, 1985, Detterman & Sternberg, 1993,
Gick & Holyoak, 1987)
The Bruce-Wylie laws were developed in the early part of this century and
assert: 1) transfer depends on the degree of similarity between two situations, and
2) the direction of the transfer depends on the similarity of the two responses
(Gick & Holyoak, 1987, p. 18). Four factors determine transfer performance
(Gick & Holyoak, 1987, p. 20):
1. The structural relationship of the training task to the transfer task.
2. The conditions at the time of the learning experience (encoding) for
fostering the learning of the material (e g., classroom experience,
trainee characteristics).
3. Conditions when the transfer task is performed.
4 The background knowledge of the subject.
It should be noted that the first factor is concerned with transfers cognitive
process, the second with the training environment, the third with the
organizational environment, and the fourth with trainee characteristics
Detailed theories on cognitive models of performance have been
developed by researchers to explain the phenomenon of transfer (Ellis, 1965).
From a cognitive perspective, the basic rule of transfer is that the original
encoding of one task can be applied to the performance of another task. Rule-
based models of knowledge representation make it possible to see how what is
39


learned will influence transfer (Gick & Holyoak, 1987). For instance, if I see
something that fits in the palm of my hand, is red and shiny, and has a small dark
stem emerging from its core, its probably an apple.
Positive transfer means performance on one task aids or facilitates
performance on a second task. Negative transfer is when the performance on one
task inhibits or disrupts performance on a second task. Transfer is positive when
later acquisition or performance is facilitated and negative when later acquisition
or performance is hampered (Ellis, 1965, Cormier & Hagman, 1987) Zero
transfer occurs when there is no effect of one task on another (Ellis, 1965). Task
similarity is necessary in order for transfer to occur and the greater the similarity,
the greater the positive transfer (see factor 1 above). Other factors influencing
transfer are: the time interval between tasks, degree of original-task learning,
variety of previous tasks; and task difficulty. Prior experience with a task to be
learned is called stimulus pre-differentiation (Ellis, 1965).
The transfer continuum anchors are labeled near and far with the
desired goal of far transfer to generalize training content outside the context of the
training environment so that it is decontextualized (Clark & Voogel, 1985)
Similarly, Salomon and Perkins (1989) distinguished between high and low-road
transfer with the former being more abstract. Low road transfer is the result of
extensive, varied practice and occurs by the automatic triggering of well-learned
behavior in a new context. High road transfer occurs by abstracting from one
40


context and application to a new context Instructional methods should be
matched to the type of transfer desired (Clark & Voogel, 1985).
Ellis (1965) said much early research failed to consider the factors that
influenced training. Most recently, researchers say that positive transfer of
training is dependent on a host of factors. Some relate these factors to a timeline,
specifically factors and events prior to training, those at training, and another set
of factors and events that occur after training (Beudin, 1987, Cannon-Bowers,
Rhodenizer, Salas, & Bowers, 1998; Curry et al., 1994, Holcomb, 1994, Johnson
& Johnson, 1998, Newstrom, 1985; Ohio Child Welfare Training Program
Evaluation Subcommittee, 1993; Rae, 1991).
Another perspective says transfer of training is dependent on three groups
of key playerstrainees, trainers, and managersand thus transfer interventions
are directed at one or all of these groups (Holcomb, 1994, Marx, 1986; Rae,
1991). Others say that transfer is contingent upon a combination of strategies that
account for the timeline and are divided by the person who has responsibility for
the intervention strategy (Baldwin & Ford, 1988, Baldwin & Magjuka, 1991,
Beudin, 1987, Broad & Newstrom, 1992; Newstrom, 1985). Huczynski and
Lewis (1980) likened the transfer process to the links in a chain: For transfer to
occur, all the links [i.e., trainee, trainer, supervisor, and organization] must hold
when the chain is pulled. Should even one of them fail, the whole chain will
break (p. 240).
41


Other researchers focus on solving the transfer problem through
development of integrative models that consider multiple factors. Garavaglia
(1996) developed a model to incorporate all the factors to describe a sequential
process for increasing transfer potential at every step. Transfer is contingent upon
a systematic process for proactively stimulating it rather than a function of context
(Garavaglia, 1996). Broads (1997) moaei is similar in that it recognizes the
integration of strategies to facilitate transfer, but emphasizes stakeholder
involvement as the catalyst at every phase of the training development process,
rather than the strategies themselves.
Because of the complex nature of transfer of training, the transfer
evaluation must be planned and carefully implemented. The next section
examines the literature related to transfer evaluations.
2 3 1 Conducting Transfer Evaluations
A transfer of training evaluation should be conducted to demonstrate the
value of training to an organization, justify training costs, and determine the
effectiveness of training curriculum (Garavaglia, 1993). Ottoson (1997),
however, disagrees with the whole notion of a transfer evaluation. She says that
other effects of the education program that are not specific to those learned in
training are lost, because they are not sought after. So while, the transfer
evaluation may look for retention of predefined knowledge and skills, it may miss
42


other effects from the training program. It is misguided to expect that transfer
will occur since it assumes that training participants return to contexts that can
support or accommodate new skills. Also, a transfer evaluation may lose
information, because training participants may be using the information learned in
training in a different manner than that which the transfer evaluation is designed
to capture. This argument assumes that the trainee is willing and eager to apply
the new skills, but it does not consider the organizational context or environment
in which the new skills are being applied Basarab (1992) suggested performing
transfer evaluations only when the training program is critical to the success of
the organization.
When a transfer evaluation is conducted, it should:
1. Use a pre/post test design (Basarab, 1992, Sullivan & Clancy, 1990).
. t
2. Question key players who can inform whether transfer occurred (i.e.,
participant, supervisor, participant subordinates, and participant peers),
because self-report data can be biased or contaminated (Basarab, 1992).
3. Organize data to determine skills that have and have not been transferred
(Basarab, 1992).
4 Measure transfer after some period of time has elapsed after training, there
is a fine line to knowing exactly when to measure so it is accurate
(Basarab, 1992).
5 Employ a control group (Basarab, 1992; Garavaglia, 1993).
43


Clark and Voogel (1985) said that transfer evaluation should determine not only
whether transfer has been achieved but also the "level and amount.
Transfer evaluations are challenging to implement, in part because transfer
is difficult to measure due to intervening variables such as supervisory support
(Holton, 1996) Recommended data collection instruments for transfer
evaluations include: questionnaires, participant action plans, work logs,
observation, accounting records, interviews, focus groups, supervisor reports, and
performance appraisals (Basarab, 1992, Garavaglia, 1993, Delewski et al., 1986).
The development of valid and reliable transfer instruments, however, is still in its
infancy (Holton, 1996).
One instrument for measuring transfer is the Participant Action Plan
Approach (PAPA), developed by the U S Office of Personnel Management in
1980 to assist training evaluators in determining whether participants changed
their job behavior as the result of training (Delewski, Pecora, Smith, & Smith,
1986). Trainees were asked to set goals (i.e., an action plan) for how they would
implement what they learned in training. Several months after the training,
evaluators contact trainees to determine if they achieved their goals. This was
determined to be an effective method of operationalizing what was learned in
training since almost all of the trainees had tried to implement at least one of their
action items (Delewski et al., 1986). The PAPA method empowered trainees to
take control of their own learning and thereby to generate a higher probability of
44


actual implementation. This method, though, relied on self-reports, which may
have reduced the evaluations validity. Holton (1996) developed another
comprehensive transfer instrument he called upon other researchers to validate,
noting that this was a daunting task given its scope.
Universally, the transfer literature states that transfer of training is
complex and is dependent upon multiple factors. Not only is it in complex in
nature, but it also is notoriously difficult to measure An assessment of pure
transfer would require extensive documentation and observation of before and
after training-related behaviors. Therefore, some researchers have responded to
the measurement challenge by operationalizing transfer in ways that represent
transfer rather than observation of actual transfer. One measure of transfer is
retention of training content as a pre-condition for the transfer of learning
(Baldwin & Ford, 1988; Curry, 1997, Ellis, 1965). It is assumed that training
content must be remembered before it can be applied.
The next section introduces the model that captures the factors affecting
the transfer of learning. It is the primary model upon which this dissertation is
based and guides the various components of the research design.
2.3.2 The Baldwin and Ford Mode! of
Transfer of Learning
The most complete and yet parsimonious model of transfer of learning
was developed by Baldwin and Ford in 1988. They analyzed the literature related
45


to transfer of learning and developed the model in Figure 2.2 to understand the
dynamics of transfer. Their transfer model incorporates the key players as well as
time perspective to predict transfer.
46


Figure 2.2 Baldwin and Fords Model for Transfer of Training
Training Inputs Training Outputs Conditions of
Transfer
! T rainee ..... ..................................
j Characteristics
| Ability __
| Personality
| Motivation
Training Design Principles of Learning Sequencing Training Content
Learning & Retention Generalization & Maintenance

j Work
Environment
| Support
* Opportunity to Use
Note From "Transfer of training: A review and directions for future research. by T. T.
Baldwin & J. K. Ford. 1988. Personnel Psychology, 41, p. 65.
Training inputstraining characteristics, training design, and work
environmentare posited to have an indirect effect on transfer by affecting
training outputs (or outcomes) of learning and retention, thus determining training
effectiveness. Positive transfer depends both on generalization of training content
to on-the-job performance and the maintenance of trained skills over time
(Baldwin & Ford, 1988) According to Baldwin and Ford, transfer is measured by
47


self-report, if it is measured at all. Of course, self-report is not a reliable measure
of transfer, so criterion measures are necessary for determining which
interventions are more likely to lead to transfer (Baldwin & Ford, 1988). Baldwin
and Fords model has been adopted by many researchers to help understand
transfer of learning and develop intervention strategies to encourage transfer
(Broad & Newstrom, 1992, Curry, 1997, Johnson & Johnson, 1998, Lim, 1999).
Broad and Newstrom (1992) used the Baldwin and Ford model to develop
strategies, called Relapse Prevention Training, for increasing transfer. Relapse
prevention strategies emphasize the importance of the interaction between trainee,
training, and organizational environment characteristics Lim (1999) added a
cultural component to make the model relevant for international training. Other
researchers have developed similar theories for understanding transfer. For
instance, Parrys, S model (1990) delineated three sets of factors that affect
transfer: personal factors, instructional factors, and organizational factors.
Using the Baldwin and Ford model as the framework, the transfer
literature was reviewed to determine research that supported contentions within
each domain of the model: trainee characteristics, training design, and work
(organization) environment. It should be noted that some studies researched
multiple influences (e g., trainee characteristics and training design) at the same
time, so some research overlaps. Each domain is examined in detail below
48


2.3.2.1 Trainee Characteristics
Some researchers focus on transfer as predicted by characteristics that
trainees possess. One such trait is the trainees general ability (Burke, 1997;
Clark & Voogel, 1985; Marx, 1986). Relapse Prevention Training, which
involves information and strategies given to trainees during the training session to
help them retrieve or transfer the training content back at the work setting (Broad
& Newstrom, 1992, Burke, 1997, Gist et al., 1991; Marx, 1986, Newstrom,
1997), can enhance general ability. Inherently, it places the onus of responsibility
on trainees but helps them by providing techniques and strategies to prevent
relapse to pre-training behaviors (Burke, 1997, Burke & Baldwin, 1999).
Newstrom said trainees should be responsible for their own application of
information learned in training (1985). Similar to the character trait of general
ability, Davey and Hill (1995) said that variation in their study was accounted for
by the respondents professional background, followed by the experience of
workers. Trainees v/ith greater metacognitive skills (i e., the knowledge of and
control over ones cognitions) also were found to have better outcomes (Ford,
Weissbein, Smith, Gully, & Salas, 1998) as were trainees who are more confident
(Gist et al., 1991)
Another trainee characteristic influencing transfer is whether trainees are
mastery- or performance-oriented. Mastery-oriented trainees are defined as
trainees who are focused on enhancing task competence, are ego-involved and
49


have an external locus of control. Performance-oriented trainees are focused on
demonstrating competence to oneself or others, have an internal locus of control
and see feedback as constructive, in studies analyzing the performance
differences between these two groups, mastery-oriented employees engaged in
more interim skill-maintenance activities, planned to exert more effort, and
showed more positive affect than did performance-oriented employees (Fisher &
Ford, 1998; Ford et al., 1998, Stevens & Gist, 1997).
Motivation is another factor affecting transfer. If trainees are motivated,
they will remember and apply course content (Burke, 1997; Hicks & Klimoski,
1987). Motivation to transfer, along with self-efficacy, were found by Machin
and Fogarty (1997) to be the main influences for the development of transfer
intentions. According to Facteau et al. (1995), pre-training motivation is more
than a trainee characteristic, but a function of trainee personality variables, such
as intrinsic and extrinsic incentives, as well as perceived environmental support.
In one study, voluntary attendance at the training improved performance (Hicks &
Klimoski, 1987). Baldwin and Magjuka (1997) disagreed that voluntary
attendance resulted in better transfer and instead said that voluntary training may
convey a signal of relative unimportance to trainees. To a lesser extent than other
trainee characteristics, prior information also had an effect on transfer (Hicks &
Klimoski, 1987). This suggests that training outcomes and the overall success of
50


training interventions are contingent upon understanding the point of view of
trainees (Hicks & Klimoski, 1987).
In the Miles (1965) and Xiao (1996) studies, trainee characteristics were
found to have no effect on transfer. Personality variables (i.e., ego strength,
flexibility, and need affiliation) did not affect outcomes (Miles, 1965). However,
learning in training as perceived by trainees did contribute to better performance
and transfer in the workplace (Xiao, 1996).
To summarize, studies on trainee characteristics suggest that trainers
should focus on trainees intrinsic motivation, build on employee strengths that
encourage transfer, and employ strategies to compensate for trainee weaknesses.
The next section discusses the training design domain in the Baldwin and Ford
model.
2 3 2 2 Training Design
Many researchers focus on the relationship between transfer and training
design factors related to the instructor, curriculum, and training techniques. This
body of literature has its roots in the early research conducted by Thorndyke and
others who were most concerned about the equivalency of training content and
job tasks (recall the theory of identical elements discussed on page 37).
This perspective suggests that transfer often fails due to inappropriate
training design models that hinder the application of training concepts beyond the
51


classroom. To be effective, training must be relevant to the workers job (Clark &
Voogel, 1985; Garavaglia, 1993). Training can be made more relevant by
conducting needs assessments to ensure that training content is reflective of
workers training needs and is based on the reality of job tasks and necessary
performance (Darou, 1990, Pecora et al., 1983, Rae, 1991, Taylor, 1992).
Training needs assessments attempt to match training content to job tasks by
conducting a thorough job analysis that includes distributing staff questionnaires,
engaging in structured interviews of front-line supervisors about their
subordinates, and defining a critical incident method to ensure that training is
more relevant (Darou, 1990).
Implicitly, training methods chosen during training affect transfer
(Baumgartel & Jeanapierre, 1972; Gist et al., 1991; Rae, 1991, Vinokur-Kaplan,
1986). Transfer is successful when levels of transfer, types of learning objectives,
and student aptitude levels are matched to instructional methods (Clark & Voogel,
1985). Cognitive modeling is one such training method to improve training
retention, it trains students to attend to their thoughts as they perform an activity
and then uses self-instructional thoughts to guide performance on the training task
(Gist, 1989). Also important are the quality of the trainer and the use of
appropriate materials to which workers can refer during and after training
(Vinokur-Kaplan, 1986) Yorks, ONeil, Marsick, Lamm, Kolodny, & Nilson
(1998) recommend a curriculum based on the principles of Action Reflection
52


Learning, a program that integrates real-world challenges. Ford et al. (1998)
recommended incorporating curriculum content related to building self-efficacy
and other motivational outcomes to facilitate transfer rather than relying on
trainees intrinsic ability and motivation, which may or may not be sufficient.
Other researchers say transfer is dependent on how trainees are taught to
retrieve the information when back at the job, not whether training matches job
tasks. Relapse prevention techniques are relevant to two domains of the Baldwin
& Ford model as successful relapse prevention training depends on what is taught
at the training and the trainees characteristics. Using the language of learning
researchers, retrieval at the transfer stage depends on the encoding done at
training. Intervention strategies taught during the training are the most critical
aspect of transfer for this group of researchers (Broad & Newstrom, 1992; Burke,
1997, Cannon-Bowers et ai., 1998, Gist et al., 1991; Marx, 1986, Newstrom,
1997, Phye & Sanders, 1992). Many of these researchers are proponents of
relapse prevention training discussed earlier in the trainee characteristics section
(Broad & Newstrom, 1992, Burke, 1997, Burke & Baldwin, 1999, Gist et al.,
1991; Marx, 1986; Newstrom, 1997). In a variation of relapse prevention
training, Foxon (1997) posited that action planning will lead to greater transfer,
but, an empirical investigation of this technique resulted in ambiguous results.
Olivero, Bane, and Kopelman (1997) recommend one-to-one executive coaching
once trainees return to the job to more expediently facilitate transfer.
53


Curry (1997) found the most salient factors leading to transfer were
training relevance, trainer use of adult learning strategies, transfer strategies, and
perceived learning. To a lesser extent, but also relevant, were: planning for
application, pre-training preparation, post-training participant motivation, time
and caseload demands, and congruence of the training content with the transfer
environment.
To summarize, these training design studies suggest that efforts for
improving transfer should focus on the relevancy of training, trainers proficiency,
curriculum, and training techniques.
2.3.2 3 Work Environment Characteristics
Studies of the work or organizational environment characteristics and their
effect on training are predicated on the belief that training develops potential
capacity in trainees while organizational environment factors are key to
facilitating the transfer of knowledge, skills, and attitudes learned in training
(Kozlowski, 1997, Xiao, 1996). In, 1953, Fleishman conducted a study on
attitudes taught in training and the extent to which they were continued back at
the job Almost fifty years ago, he astutely noted, It is difficult to produce in an
individual a behavior-change that violates the culture in which this behavior is
imbedded (Fleishman, 1953, p. 221). Similarly, but forty-five years later,
54


Baldwin and Magjuka said: Organizational contextual factors can easily
overwhelm the effects of the best planned and delivered training (1997, p. 122).
Attention to the work environment may hold the greatest potential for
ensuring transfer of training, but it also receives the least amount of attention
(Baldwin & Ford, 1988, Beudin, 1987, Kozlowski, 1997). Transfer is affected by
work context because it facilitates or inhibits an individuals opportunity to
perform tasks or skills trained (Ford, Quinones, Sego, & Sorra, 1992). The work
environments complexity is compounded by the difficulty of predicting how
employees will attach meaning to management behaviors and attitudes. Also,
training is never conducted in a pristine environment since outside influences
affect the training situation (Baldwin & Magjuka, 1997).
Researchers do not use a uniform definition of the organizational
environment, rather, the variables that cannot be attributed to either trainee
characteristics or training design are categorized into a generic catchall called the
work environment Baldwin & Ford (1988) created two categories for the work
environment, supervisory support and favorable organizational climate The
organizational climate variable is defined more broadly and includes: supportive
management style, peer support, and the opportunity to use information learned in
training on the job. On the other hand, the supervisory support variable is more
narrowly defined as the behaviors and conditions associated with the supervisor
(or manager), such as the support the supervisor gives the employee to attend
55


training, to apply what is learned in training, and his or her efforts to reinforce
new behaviors. The work environment literature related to transfer can be divided
into studies focusing on one or both of these variables. It should be noted that
research often overlaps, for researchers may study several aspects of the
organizational environment at one time as well as factors related to trainee
characteristics and training design.
2 3 2 3 1 Supervisory Support
Many studies focus on the different dimensions of the supervisors
transfer-encouraging role within the organizational environment (Baldwin &
Magjuka, 1991, Brinkerhoff & Montesino, 1995, Broad & Newstrom, 1992;
Curry, 1997; Gregoire, Propp, & Poertner, 1998; Holton, Bates, Seller, &
Carvalho, 1997, Huczynski & Lewis, 1980, Ohio Child Welfare Training
Program Evaluation Subcommittee, 1993, Taylor, 1992, Warren, Woodall,
Nunno, Keeney, Larson, & Stadfeld, 1998, Xiao, 1996). According to some
research, the supervisor is perhaps the most important variable within the work
environment, even more significant than top-management support (Taylor, 1992,
Xiao, 1996;). For others, it is no less important than other variables but remains
an essential component for transfer to occur (Facteau et al., 1995).
Supervisors influence transfer in different ways. Supervisors have the
most influence for improving transfer by providing reinforcement of training
56


concepts (Broad & Newstrom, 1992; Phillips, 1991). Supportive supervisors can
help stimulate transfer through their behaviors and attitudes (Broad & Newstrom,
1992; Gregoire, et al., 1998). Management support includes both tangible and
perceived support and can be significantly and positively influenced by the
immediate supervisor before and after training (Brinkerhoff & Montesino, 1995,
Foxon, 1997, Holton, et al., 1997, Phillips, 1991). The extent of a trainees
accountability to his or her supervisor also affects transfer (Baldwin & Magjuka,
1991).
In Huczynski and Lewiss 1980 study, a consultative management style
rather than an autocratic management style was found to influence transfer most
positively. Transfer improved when trainees discussed the content before the
course with their immediate supervisor (Huczynski & Lewis, 1980). Supervisors
may often fail in their efforts to provide organizational support, however, by not
spending enough time in their role as mentor (Zunz, 1995) or by not attending to
other supportive behaviors, such as one-to-one coaching, which have been shown
to increase transfer (Gregoire, et al., 1998).
2 3 2 3 2 Organizational Climate
Many researchers have concluded that the organizational climate in w'hich
trainees work influence the degree of positive transfer (Baldwin & Magjuka,
1997; Baumgartel & Jeanapierre, 1972, Beudin, 1987, Burke & Baldwin, 1999,
57


Carnevaie et al., 1990, Curry, 1997, Dionne, 1996, Facteau et al., 1995;
Fleishman, 1953; Ford et al., 1992, Hand, Richards, & Slocum, 1973, Holton et
al., 1997, Huczynski & Lewis, 1980, Kozlowski, 1997, Lim, 1999, Miles, 1965,
Newstrom, 1985, Phillips, 1991, Tesluk, Tesluk, Farr, Mathieu, & Vance, 1995;
Vinokur-Kaplan, 1986, Warren et al., 1998) Factors falling in the organizational
climate domain are more diverse and extend beyond the supervisors sphere of
influence within the organization. The organizational climate includes supervisor,
administrative, organizational, and co-worker support (Curry, 1997; Holton,
1997). Behaviors within the organizational climate send a message that learning
is or is not valued and thus influence whether newly learned behaviors are applied
(Tracey, Tannenbaum, & Kavanagh. 1995) Different transfer climates may exist
simultaneously in one organization (Newstrom, 1997, Tesluk et al., 1995).
Trainers thought the most significant barrier to transfer was the lack of on-
the-job reinforcement designed to support trainees in applying training to their
jobs (Broad & Newstrom, 1992). Other organizational factors acting as barriers
to transfer were: interference from the immediate environment, non-supportive
organizational climate, and negative peer pressure (Broad & Newstrom, 1992,
Newstrom, 1985)
Conversely, organizational factors that increased the likelihood of
positive transfer were: sensitivity of higher management, entrepreneurial
attitude of organization, expectations of higher management to use
58


training, interaction within the work environment, and compatibility of
goals of the organization and training (Baumgarte! & Jeanapierre, 1972,
Beudin, 1987, Dionne, 1996, Garavaglia, 1993).
In stark contrast, (and contrary to what was hypothesized), Tesluk
et a! (1995), found that a participative climate for decision making was
negatively related to adoption of employee involvement training. Post-
hoc, the researchers speculated that employees would have more
opportunity to transfer since the horizon for employee involvement was so
much greater in a non-participative decision making environments (Tesluk
et al., 1995).
More effective training occurred when the organization integrated and
focused on training and fostered a climate conducive to training (Carnevale et al.,
1990, Newstrom, 1985) The effectiveness of training techniquessuch as
relapse prevention strategiesare moderated by the nature of the organizational
climate for encouraging transfer. Transfer climate should be assessed during the
training needs analysis, and then the w'ork environment should be manipulated to
support transfer. Positive transfer can be improved if organizational members
receive training or other assistance to create supportive transfer climates (Rouiller
& Goldstein, 1993, Tracey et al., 1995). Bennett, Lehman and Forst (1999) take a
broader view of transfer climate, stating that it is one component of the contextual
factors, along with structural factors and change climate, that influence the degree
59


of transfer. Interventions then are aimed at all of these areas to increase transfer
potential within organizations (Bennett et al., 1999).
Another vein of research looked at the opportunity to perform what was
learned in the classroom to the job (Ford et al., 1992, Holton et al., 1997,
Newstrom, 1997). New job-related behaviors can appear only if trainees are
given the opportunity to perform their newly acquired skills (Newstrom, 1997).
This variable is placed within the organizational climate domain since it is
contingent upon many factors within the organization, not just the supervisor.
Training must be relevant to the trainees job and once training is completed,
forces within the organization must encourage and provide opportunities for the
employee to practice what was learned in training. Breadth, activity level, and
type of tasks performed should all be examined as three dimensions of the
opportunity to perform (Ford et al., 1992).
When the work environment is believed to the primary factor for
increasing transfer of training, then interventions would focus on changing the
organizational context to encourage transfer of training. Little evidence of
organizational environment interventions, however, has been documented in the
literature Baldwin and Ford (1988) found seven studies related to the influence
of the organizational environment. But they criticized these studies as not being
experimentally rigorous enough by actually applying interventions to change the
organizational environment, thus they were unable to study the effect of
60


organizational differences. Brinkerhoff and Montesino (1995) responded to
Baldwin and Fords gauntlet and conducted a study of an organizational
environment intea^ention to determine the effect on transfer. While the
intervention was admirable, the study contained methodological flaws. Because
the intervention was good practice, some subjects in the control group
inadvertently provided the same intervention as the experimental group. Olivero,
Bane, and Kopelman (1997) conducted a study to provide coaching by fellow
participants to help improve transfer, but it too had methodological problems.
Given the importance of empirically testing for organizational climate, it is
curious that so little research has been conducted in this area.
A lack of theory has prevented application of organizational context
factors affecting the implementation and transfer of training (Kozlowski, 1997).
Kozlowski (1997) suggested remedying this theoretical dearth by applying
systems-oriented theories of organizational behavior to help draw inferences
regarding the effects of contextual factors on behavior found in the work
environment and in training transfer situations. Such a relationship has been
theorized for many years. Simon (1945) said employee decisions and behaviors
are influenced within and by the organization In 1954, Drucker noted the
existence of a symbiotic relationship between group relationships and tasks:
Group relationships influence the task, the task in turn includes personal
relationships with the group... Group and individuals must therefore be brought
61


into harmony in the organization of work (p. 265) Exploration of the transfer
literature leads one to question why and how environmental factors influence
employee performance. The answer to this lies, in part, in the organizations
culture The next section explores this relevant literature.
2 4 Organizational Culture
Organizational culture dictates the employees work environment and thus
has a profound effect on whether positive transfer occurs. The notion of
organizations having culture was borrowed from the fields of sociology and
anthropology. Organizational culture was studied because of a need to explain
variability in patterns of group and organizational behavior and a need to
understand levels of stability in order to answer the question of why some
organizations did well and others did not (Schein, 1990). Organizations are social
systems with pasts, presents, and futures (Pettigrew, 1979).
Scholars agree that there is no consensus definition of culture (Gordon &
DiTomaso, 1992; Schein, 1990, Vestal et al., 1997). For the purpose of this
research, culture is defined as:
[A] patterned system of perceptions, meanings, and beliefs about
the organization which facilitates sense-making amongst a group
of people sharing common experiences and guides individual
behaviour at work. (Bloor & Dawson, 1994, p. 276).
62


Organizational climate is differentiated from the concept of organizational culture
and is a surface manifestation of culture (Schein, 1990). Culture is embedded in
the schemas that help people make sense of organizational life Schemas are
cognitive maps that use past experiences and understandings to make predictions
about events so that individuals can select appropriate responses to anticipated
events (Bloor & Dawson, 1994). Culture acts as a filter for interpreting activities
and events.
Many researchers use the trait approach to understanding the content of
culture (Gordon & DiTomaso, 1992). According to Vestal, Fralicx, and Spreier
(1997), there are 56 attributes (or traits) that organizations may possess that can
then be categorized into four dominant organizational cultures: functional,
process; time-based; and network. These attributes range from the organizations
behaviors around adaptation to those about authority and chain of command.
From these attributes, cultural profiles are developed that differentiate
organizations. Peters and Waterman (1982) outlined eight characteristics of
excellent, or well performing, cultures: 1) a bias for action, 2) close to the
customer, 3) autonomy and entrepreneurship, 4) productivity through people, 5)
hands-on, value-driven; 6) stick to the knitting (staying close to the business they
know), 7) simple form, lean staff; 8) simultaneous loose-tight properties (qualities
of both decentralization and centralization when appropriate).
63


Organizations that have internal cultures supportive of their strategies are
more likely to be successful (Gordon & DiTomaso, 1992, Lahiry, 1994, Smircich,
1983, Vestal, 1997). Cultural values can, and perhaps should, vary significantly
across organizations (Golden, 1992; Schein, 1990, Sheridan, 1992; Vestal, 1997,
Brown & Starkey, 1994). In fact, there may be great divergence between a
collective cultural identity and sub-cultures within the work environment resulting
in tension and dissension (Brown & Starkey, 1994). But individuals within the
organization are never culture-less. If they reject the dominant organizational
culture, they would then adhere to a sub-cultures values and cultural guidelines
(Golden, 1992) A weak organizational culture will give rise to more sub-cultures
and vice versa (Golden, 1992).
Organizational effectiveness can be improved by attending to the
organizational culture (Dunn, Norburn, & Birley, 1994, Sheridan, 1992).
Members of an organization exert pressure on each other to conform to shared
codes (Lahiry, 1994). Organizational cultures emphasizing interpersonal
relationship values tend to retain workers longer than organizations who have
work task values (Sheridan, 1992). Organizations with strong cultures, as defined
by consistency of perceptions of company values, tend to have better performance
(Dunn et al., 1994, Gordon & DiTomaso, 1992).
There is, however, some disagreement about whether culture can be
manipulated. Meek (1988) says culture is something that an organization is, not
64


something that an organization has, therefore it is not an independent variable
(Meek, 1988, p. 469). Culture is not static and while it is shaped by an
organizations historical context, it changes gradually but continuously to reflect
the organizational environment and sub-cultures operating within it (Bloor &
Dawson, 1994). In another perspective on the influence of culture, Golden (1992)
stated that too much emphasis on organizational culture may minimize the actions
of individuals who may act by challenging cultural guidelines.
By understanding an organizations culture and its effect on organizational
members, one can start to make sense of the processes and outcomes, thus
influencing them toward a desired end (Brown & Starkey, 1994; Carnevaie et al.,
1990; Dunn et al., 1994). Smircich (1983) points out the value of studying culture
as an effort to improve management:
A cultural mode of analysis encourages us to recognize that both
the practice of organizational inquiry and the practice of corporate
management are cultural forms, products of a particular
sociohistorica! context and embodying particular value
commitments. In our present day these values are efficiency,
orderliness, and even organization itself (p. 355).
Since culture conditions attitudes about all types of organizational activities and
events, opportunities for improving transfer of training may lie in researching the
work environment and its organizational culture with respect to training. A wish
to understand the influence of the organizational environment on transfer of
training led to the design of this research project and its methodology. A
65


thorough analysis of the literature specifically related to empirical studies of
transfer of training and the organizational environment is described in the next
section.
2.5 Statement of the Problem
While many acknowledge that transfer is contingent on the post-training
organizational environment, insufficient empirical research has been conducted to
guide practice, and the studies that have been conducted are often
methodologically deficient to adequately assess transfer (Baldwin & Ford, 1988,
Curry, 1997; Macaulay & Cree, 1999; Tannenbaum & Yukl, 1992). Some of the
authors prescribing transfer approaches do not even offer empirical studies to
support their contentions (for instance, Garavaglia, 1993; Johnson & Johnson,
1998; Marx, 1986), but rely instead upon the opinions of their authors.
Methodological problems with the studies on transfer of training and the
organizational environment can be categorized into the following areas: sample
size, type of subjects; equivalency of training programs, criterion measures of
performance, and experimental rigor. Using these criteria, the research studies
which focused on the effects of the organizational environment on transfer were
analyzed and are presented in Table 2.2.
66


Table 2.2 Summary of Transfer and Organizational Environment
Research Studies
Research Sample Size Type of Equivalency of C riterion Experimental
Study Subjects training measures of rigor
programs performance
Baldwin & Magjuka, 1991 N= 180 Good (engineers) Problematic, trainees attended multiple training programs Inadequate, self- report of intentions Low, survey, self- report, lack of equivalent training programs
Baumgartel & Jeanapierre. 1972 N = 240 Good (managers) Problematic, participants attended 17 different training programs Inadequate, self-report, matched responses to supervisors for 33 respondents Low, lack of equivalent programs, self- report
Bennett, Lehman, & Forsi, 1999 N = 909 May be problematic (city employees at different levels & departments) Good Inadequate, self-report Low, Survey
BrinkerholY & N = 91, 37 in Good Problematic, Inadequate, Medium,
Montesino, 1955 experimental group (corporate employees) participants attended 5 different training courses self-report random sample, control group was tainted
Burke, 1997 N = 90; 65 in experimental group Problem (college students) Good Inadequate, self-report Medium, random sample, 2 experimental groups
Burke & Baldwin, 1999 N= 78 Good (research scientists) Good Inadequate, self-report, matched responses to subordinates Medium, quasi- experimental, 3 treaunent conditions, self-selection, self-report, multiple measures
Curry, 1997 N = 598 Good (child welfare caseworkers) Good Inadequate, self-report Low, Survey
67


Table 2.2 (Cont.)
Research Study Sample Size Type of Subjects Equivalency of training programs C riterion measures of performance Experimental rigor
Facteau, Dobbins, Russeli, Ladd, Kudisch, 1995 N = 967 May be problematic (state government managers at various levels) Problematic, no specific training courses defined, study was a needs assessment) Inadequate, self- report Lov/, survey, no defined training course, discussed transfer in general, analyzed many influences w ith scales
Fleishman, 1953 N = 122 Good (plant foremen) Good Excellent self-report, survey of supervisors and co-workers, validated leadership assessment instruments High, random sample, control group
Ford, Quinones, Sego & Sorra. 1992 N = 180 Good (Air Force personnel) Good Somewhat adequate, self-report matched responses to supervisors Low, Survey
Foxon, 1997 N = 104 May be problematic (corporate employees at all levels and positions) Mas be problematic, participants attended 3 different training programs Somewhat adequate, self- report, matched responses to some managers Medium, random sample, control group
Gregoire, Propp, & Poertner, 1998 N = 210 Good (child welfare caseworkers) Problematic, participants attended from 1 to 6 different training programs Inadequate, self-report Low, Survey
FI and, Richards, & Slocum. 1973 N = 42, 21 in experimental group Good (corporate managers) Good Adequate, Subordinates rated, validated performance Medium, random sample, control group, other
68


Table 2.2 (Cont.)
Research Sample Size Type of Equivalency of Criterion Experimental
Study Subjects training measures of rigor
programs performance
instruments explanations for effect
Huczynski & N = 49 Good May be Inadequate, Low,
Lewis, 1980 (corporate problematic, self-report no random
employees) participants assignment, no
attended 2 control group,
different no criterion
training programs measures
Lim, 1999 N= 10 May be Good Inadequate, self- Low,
problematic report case study
(corporate design, no
employees at criterion
different performance
levels) measure.
descriptive study only
Machm & N = 40 Good Good Inadequate, Low,
Fogartv, (police self-report survey
1997 officers)
Miles. 1965 N = 148, 34 Good Good Excellent, High,
m (elemer.tarv validated modified
experimental school instruments and Solomon four-
group principals) ratings by 5 associates group design
Ohio Child N = 499 May be Problematic, Inadequate, Low,
Welfare problematic participants self-report survey
Training (child welfare attended
Program caseworkers, different
Evaluation supervisors, training
Subcommit- directors, programs
tee, 1993 administrativ offered by same
0 agency
Olivero, N = 31 (8 Good Good Excellent, Low,
Bane & participants (managers) used an index of no random
Kopeiman, became part productivity as sample, no
1997 of the outcome formal
intervention. variable intervention, no
reducing control group,
sample size did use criterion
to 23) measures
I
69


Table 2.2 (Cont.)
Research Sample Size Type of Equivalency of Criterion Experimental
Stuck Subjects training measures of rigor
programs performance
Rouiller & Goldstein. 1993 N = 102 Good (assistant fast food store managers) Good Adequate, multiple measures, transfer behavior rated by managers and crew members High, no control group was used, did have multiple measures
Taylor, 1992 N = 350 Good (corporate employees) Problematic, surveyed 350 corporations, no equivalency Inadequate, training directors provided opinion for training participants Very Low, survey, training directors reported on trainee behavior, no equivalency
Tesluk. Fany Mathieu. Vance, 1995 N = 252 May be problematic (employees of transportation dept, at different levels Problematic, subjects could have attended any of 12 courses Inadequate, self- report data Medium, multiple measures to establish climate, comparison groups, no criterion performance measure
Tracey. T annenbaunt Kavanash 1995 N = 104 Good (supermarket managers) Good Adequate, multiple measures for climate and transfer Medium, multiple measures to establish climate, comparison groups, no criterion performance measure
Xiao, 1996 N = 106 Good (factory workers) Problematic, subjects attended training at four different factories Somewhat adequate Self-report and group efficiency measures Low, survey, no control group, used group efficiency rates
70


2 5 1 Sample Size
An inadequate sample size affected the generalizability of several studies.
Lims (1999) research used a case study design and had only 10 subjects.
Olivero, Bane and Kopelmans (1997) study had an original sample size of 31 that
was reduced to 23 when eight subjects became part of the intervention. Machin
and Fogarty (1997) had only 40 subjects, while Huczynski and Lewis did better
with a total sample size of 47 When surveys were used, the sample size
increased substantially For instance, Facteau et al. (1995) had a sample of 967,
Currys (1997) study used a total of 598 respondents, Baumgartel & Jeanapierre
(1972) were able to obtain 240 surveys, and Bennett, Lehman, and Forst (1999)
surveyed 909 employees. Taylors survey generated 350 completed
questionnaires, but the response rate was only 26 percent. Otherwise, most of the
studies had sample sizes between 90 and 200 subjects.
2 5 2 Type of Subjects
Research studies can be generalized to the larger population when subjects
are similar to the population that the study intends to understand (Rubin &
Babbie, 1989). While the Burke (1997) study had many positive attributes and
attempted to be experimentally rigorous, it was fundamentally flawed for several
reasons, one of which was the choice of college students as subjects. Another
problem occurs when subjects at different job levels are mixed together without
71


controlling for employee level For instance, the Foxon (1997) study used
subjects at several different job levels from clerical staff to plant operators. The
Ohio Child Welfare Training Program Evaluation Subcommittee analyzed data
from caseworkers, supervisors, administrators, and directors. Bennett, Lehman,
and Forst (1999) surveyed 909 subjects from different city departments and with
widely disparate job responsibilities and titles. Subjects at diverse job levels and
functions may be dissimilar enough to seriously jeopardize the study findings.
W'hen research subjects are similar enough in job level and responsibilities, they
react more similarly to the training intervention and any differences in reaction
could be attributed to any of the studys independent variables.
2 5 3 Equivalency of Training Programs
A lack of equivalent training programs also may affect the ability to
predict results as the findings could be from differences between training
programs rather than some other factor such as the organizational environment.
Baldwin and Fords mode! states that transfer is contingent upon trainee
characteristics, training design, and work environment, so differences in training
programs could explain variances in transfer. For instance, the Baumgartel and
Jeanapierre (1972) study assessed 17 different training programs to determine the
kinds of managers who are most likely to apply new knowledge, and the kinds of
organizational environments that are most likely to facilitate managerial effort to
72


transfer, but not enough attention was paid to how the training programs may
have affected results. Tesluk, Farr, Mathieu, and Vance (1995) drew subjects
who had attended any of 12 training programs. Facteau et al. (1995) was even
more ambiguous as survey subjects had attended the state management training
in the past. No information was provided on whether the training included
different courses and the period of time covered by the past Baldwin and
Magjuka (1991) drew their sample from engineers attending multiple training
programs, the type and number were unspecified. Brinkerhoff and Montesinos
(1995) study used subjects enrolled in five different training courses and Xiao
(1996) assessed Chinese workers who received in-house training at four different
factories but did not provide evidence whether the training was the same at all
four locations. The Ohio Child Welfare Training Program Evaluation
Subcommittee (1993) also analyzed results from employees who attended
different training courses In another study, Gregoire, Propp, and Poertner (1998)
looked at child welfare caseworkers who attended anywhere from one to six
training programs. Participants who attended more training courses could have
improved transfer, but this was not accounted for in the analysis. Taylors (1992)
study appeared to be the most egregious as 350 corporations were surveyed, but
these corporations could have had widely different training programs and
requirements When a disparity between training programs exists, it is


methodologically brash to claim results without controlling for the training
program differences.
2 5 4 Criterion Measures of Performance
Many of the studies that presumed to predict transfer had no criterion
measures of performance and instead relied upon trainees self-report to assess
transfer (Baldwin & Magjuka, 1991, Baumgartel & Jeanapierre, 1972, Bennett,
Lehman, & Forst, 1999; Brinkerhoff & Montesino, 1995; Burke, 1997; Curry,
1997; Facteau et ah, 1995, Ford et al., 1992; Foxon, 1997, Gregoire, et ah, 1998;
Huczynski & Lewis, 1980, Lim, 1999; Machin & Fogarty, 1997, Ohio Child
Welfare Training Program Evaluation Subcommittee, 1993; Tesluk et ah, 1995;
Xiao, 1996). Self-report distorts an accurate measure of transfer (Baldwin &
Ford, 1988, Cruz, 1997) Cruz (1997) recommended measuring transfer using
high instructional alignment conditions rather than low or moderate conditions
Assessment (or instructional) alignment occurs when the assessment most closely
matches the conditions at the training. Self-report is considered low to moderate
alignment while criterion measures of job performance using tools such as
observation by trained raters results in high alignment (Cruz, 1997). Also,
questionnaires administered shortly after training may have a halo effect thus
providing inaccurate information about transfer (Huczynski & Lewis, 1980).
74


Some studies that used self-report of transfer attempted to obviate the
inherent bias by cross-checking the participants self-report against supervisor
and/or co-workers opinions For instance, Ford et al. (1992) surveyed
supervisors, Fleishman (1953) surveyed supervisors and co-workers, and Burke
and Baldwin (1999) cross-validated results with subordinates. Tracey,
Tannenbaum, and Kavanagh (1995) measured organizational climate and transfer
behaviors with questionnaires sent to supervisors, co-workers, and subordinates.
Baumgartel and Jeanapierres study attempted to correlate responses between
workers and supervisors for 33 subjects (out of a sample size of 240) and found a
correlation of .43. Taylors study (1992) sent questionnaires to the trainees
supervisors about their reactions, degree of learning and changes in supervisor
behaviors.
Interestingly, Rae (1991) found little correlation between the managers
and trainees responses, in fact, the similarity was so weak that it sometimes
seemed as if they were talking about different people However, Rae does
recommend sending a questionnaire to the supervisor to compare the similarities
and differences to develop a more complete picture of the situation.
Some studies did attempt to use criterion measures of performance. For
instance, the Miles (1965) study accounted for trainee, training, and
organizational variables in relation to performance on three criterion variables
using a performance test, evaluations by the trainers, and a self-perceived learning
75


measure. Also an average of five co-workers were surveyed using the same
instrument to arrive at a verified change score (Miles, 1965). Hand et ai.s
(1973) study also used several measures including subordinate surveys and
validated instruments Olivero, Bane, and Kopelman (1997) looked at an
outcome variable using an index of productivity. Xiao (1996) also tried to look at
an outcome variable by using work efficiency data, but its utility was weakened
because it measured group rather than individual efficiency rates.
2 5 5 Experimental Rigor
Many studies lacked experimental rigor, thus reducing the ability to
generalize findings. For purposes of this analysis, experimental rigor is inclusive
of all the previous criteria discussed, as well as additional factors looking at the
experimental methodology to arrive at a generalized rating for each study.
Only three studies were rated high for experimental rigor. Qualities that
made Fleishmans (1953) study rate high were the pre/post test design, random
assignment, control group, and criterion measures of performance. Miles (1965)
used a modified Solomon four-group design with two control groups, a pre-
measure, and two post measures to assess performance. Rouiller and Goldsteins
(1993) study lacked an intervention other than the training and there was no
comparison group; deficiencies, however, were made up by the strength of the
measures used to assess organizational climate and transfer, and thus the
76


experimental rigor was rated high Current managers established the
organizational climate of various work environments and multiple measures
assessed transfer behavior and job performance. Rouiller and Goldstein also used
control variables to isolate the effects of training from other explanations.
Some studies were rated medium due to positive characteristics that would
have resulted in a high rating except that they were diminished due to a variety of
methodological weaknesses. For instance, Brinkerhoff and Montesinos (1995)
study used random selection and a control group, but intervention effects were not
isolated to the experimental group. Thus, it would be difficult to generalize
findings on tainted results. Hand et al.s study (1973) used a pre/post test design,
experimental/control group, and criterion measures of performance. This study,
however, had a low sample size and because the outcome indicators were so
broad (e g., job performance and promotions), the results may or may not have
been the result of training.
Many studies were rated low because of flagrant errors that compromise
the validity of their research results. For instance, the Xiao (1996) study looked at
factory workers in four organizations. There was no control group and transfer
was measured by looking at a group measure of efficiency and self-report
questionnaires. It would be difficult to generalize the findings when a group
measure of performance was used rather than individual. Also, Xiao personally
observed the organizational variables at each of the sites rather than allowing the
77


subjects to rate them and attach their own meaning to the climate. While the
subsequent statistical analysis was sophisticated, the Xiao study was rated low
because of the flawed experimental conditions. The Baumgartel and Jeanapierre
(1972) study also was rated low despite their attempt to make it scientifically
rigorous by using a stratified random sample and controlling for various factors.
Problems included the number of different training programs (17 total over a four
year period), the use of open-ended questions, and the weak correlations
presented in the findings (the highest was 27, most ranged around 18).
Other studies were rated low because they were not by nature
experimental, but rather were surveys to report behavior (for instance, Curry,
1997, Facteau et a!., 1995, Gregoire, et al., 1998). Baldwin and Magjukas study
reported on training participants intention to transfer, rather than actual transfer
behavior. The Taylor (1992) study was rated even lower because it asked for
training directors to generalize and report on all trainees transfer behavior. These
individuals may or may not know about the transfer experiences of all trainees
and aggregating all experiences dilutes the strength of personal experiences for
making comparisons. The Lim (1999) study was rated low (at least for purposes
of generalizability) because it was a descriptive case study analysis and by nature
it was not experimental.
78


2 5 6 Rationale for Current Research Study
In, 1988, Baldwin and Ford found only seven articles dealing with transfer
and the organizational environment. This literature review found a total of 25
articles, more than a three-fold increase in the quantity of studies. Nonetheless,
while it is encouraging that such a quantity of legitimate studies could be found,
they are still inadequate to conclusively understand the influence of the
organizational environment on the transfer of training, especially since so many of
research studies were methodologically deficient. Of particularly concern is the
lack of studies with criterion measures of performance to assess transfer
empirically. More methodologically rigorous studies are needed to build a solid
foundation for understanding the role of the organization in transfer of training
from the classroom to the job.
This research project seeks to add another block to the transfer literature
foundation and address some of these deficiencies of previous studies through the
methodology described in the next chapter Also, a review of the literature leads
to the hypotheses presented in the next section.
2 5 6 1 Research Hypotheses
Baldwin and Fords (1988) framework states that transfer is contingent
upon three groups of factors, trainee characteristics, training design, and the work
environment. While public managers may have influence over any of these
79


inputs, the work environment factor may hold the greatest potential for
intervention to improve transfer. The work environment consists of three groups
1. The organization defined as the broader agency including
administrators, managers, and other units.
2. The caseworkers supervisor who provides direct supervision
including case consultation, performance review, and day-to-day
oversight.
3 The caseworkers co-workers within his or her assigned unit.
Typically, there are four to eight caseworkers assigned to a unit and
most are at the same employment level
Based upon the literature review, especially Baldwin and Fords
framework, the following research hypotheses were tested:
Hi Participants from organizational environments they perceive as
supportive will retain more information in the post-training
interviews.
H2 Participants with supervisors who are perceived as supportive will
retain more information in the post-training interviews.
H3 Participants with co-workers who are perceived as supportive will
retain more information in the post-training interviews.
80


Organizations perceived as supportive should produce trainees who perform
better on the skill demonstration interviews while non-supportive organizations
are hypothesized to generate trainees who do not perform as well in the skill
demonstration interviews. Likewise, supervisors and co-workers who are
supportive are hypothesized to positively influence employees who will then
perform better on the skill demonstration interviews. The organization,
supervisor, and co-workers combine to influence the organizational culture
These hypotheses will inform whether there is an organizational culture that is
supportive of training results in improved interview performance by using a
criterion measure of performance for testing training retention. The next section
discusses the research methodology used in the project.
81


3 Methodology
3.1 Research Project
The dissertation represents a higher levei evaluation of the Specialized
Interviewing Skills (SIS) training for new child welfare workers to assess whether
training improved performance and to understand the conditions that might have
facilitated or inhibited transfer of training from the classroom to on-the-job
performance The thesis design utilizes the existing training format and adds
several components to obtain the needed data. As part of the usual training,
trainees attend a three-day training, and as an activity of that training, interview
an actor who is role-playing a latency-child who may have been the victim of
sexual abuse. The interview between the trainee and actor is videotaped for
review by the trainer during the evening of the second day and subsequent rating
by a trained videotape rater using a 50-item instrument that reflects the
components of an effective sexual abuse interview. The research design adds a
pretest to establish pre-training skills, a posttest to determine retention of
interview skills after a period of time, and survey questionnaires to understand the
factors that may have facilitated or inhibited transfer of training.
82


3.2 Research Design
The dissertation is a descriptive study using a descriptive design.
Primarily, quantitative data are used for the analyses with some qualitative
information providing context to understand the factors that contribute to transfer
of training Data on interviewing skills were collected at three points in time:
1. Pre-training interview skill assessment (pretest)
2. Training interview skill assessment (training test)
3. Post-training interview skill assessment (posttest)
The intent of a pre-training interview skill assessment was to establish the skills
that trainees brought to the training. The training interview skill assessment
demonstrated how much the subject was able to do immediately after the training.
Retention and thus transfer were assessed at the posttest interview skill
assessment and through questionnaires completed by the subjects and their
supervisors. Other factors were assessed through the collection of questionnaires
from the research subjects and their supervisors (as explained in the Section 3.4
below).
3 3 Research Subjects
3.3.1 Child Welfare Caseworkers Characteristics
The total sample of 61 research subjects were child welfare caseworkers
from any of the 63 counties within the State of Colorado. Their job positions and
83


functions varied and included Intake (conducting child abuse and neglect
investigations), Ongoing (providing ongoing case management services), and
specialized units (e g., Youth-in-Confiict, Adoptions, Intensive Family Therapy).
While their job functions varied somewhat, all trainees were at approximately the
same employment level and all had direct contact with clients of Social Services
(see below for a more thorough discussion of characteristics). Typically, the SIS
training course is taken during the first year of employment and is optional for all
caseworkers, although it is mandatory in some counties for caseworkers from
Intake or Sexual Abuse Specialized units.
A total of 70 subjects were recruited and completed at least the pretest and
training test interviews. Nine subjects were later dropped from the project
because either they did not complete the posttest or they interviewed one actor
who had to be dropped from the study due to measurement issues (thereby
disqualifying those tests). Ultimately, 61 subjects completed the three videotaped
interview' skill assessments and the required questionnaires.
New caseworkers may be at levels one through four depending on county
policy, salary requirements, and education and experience level. See Table 3.1
for a breakdown of current job titles for the research sample.
84


Table 3.1 Research Subjects and Current Job Titles
(N = 61)
Job Title Frequency Percent
Caseworker I 9 14.8
Caseworker II 13 21.3
Caseworker III 32 52.5
Caseworker IV 7 11.5
Caseworker levels imply the amount of experience and education
caseworkers bring to their job rather than a differentiation in the types of
assignments given to each level. A caseworker at Caseworker I could have the
same job responsibilities as a caseworker employed at Caseworker III, although a
caseworker assigned to a specialized unit, such as the sexual abuse team, would
likely be at a higher level because of the experience necessary to perform those
responsibilities. Thus, this group of trainees, with 65 percent at the III or IV
level, is already quite experienced (note: a less experienced group might get more
benefit from training).
85


Table 3.2 presents the break down of their current unit assignments
Table 3.2 Research Subjects and Unit Assignments
(N = 61)
Unit Frequency Percent
Intake 14 23.0
Ongoing 18 29.5
General 10 16.4
Youth-in-Conflict 2 *> ->
Sexual Abuse Team 4 6.6
Adoption 5 8.2
Core Sendees 1 1.6
Other 7 11.5
Caseworkers who are assigned to the Intake and Sexual Abuse Team units are
more likely to conduct investigative interviews with children while caseworkers
assigned to units such as Adoption and Ongoing would not generally be
conducting investigative interviews unless a child discloses sexual abuse during
the course of regular casework with the child. Caseworkers who are assigned to
General units work in more rural counties and typically engage in the full range of
casework responsibilities including conducting investigations and providing case
management to clients. From this information, it would appear that only about 46
86


Full Text

PAGE 1

THE EFFECT OF A SUPPORTIVE ORGANIZATIO" ALE VIRONMENT 0 TRANSFER OF TRAINING IN CHILD WELFARE ORGANIZATIONS by Charmaine Renee Brittain B.S., Indiana University 1984 M S .W., Denver U niversity 1991 A thesi s submitted to the University of Colorado at Den ver in partia l fulfillment of the requirements for the degree of Doctor of Ph i losophy Public Affairs 2000

PAGE 2

This thesis for the Doctor of Philosophy degree by Charm aine Renee Brittain has been approved by Linda deLeon Peter deLeon Date

PAGE 3

Brittain Charmaine Renee (Ph .D., Public Affairs) T he Effect of a Supportive Organizational Environ ment on Trans fer ofTr aini ng in Child Welfare Organization s Thesis directed b y A ss ociate Professor Linda deLeon ABSTRACT Training i s a tool public mana gers use t o more efficie n t l y and effectively manage child we lfare org an i zation s Train i ng is not always tra nsfe rred to on-the-job p erformance howe ver. Transfer is thoug ht to be contingent upon three sets of factors : t rainee characteristics traini ng design and organizational environment characteristics In this st udy three t y pes of factors in the organizationa l environment-organizational c limate supervisor support a nd co w orker support-were expl o red to understand ther impact on retention a measure of transfer. The research proje c t is a higher-level evaluation of the Speci a l iz ed Interviewing Skills (SIS) trainin g for ne w child welfare workers T rai ning does indeed impro ve intervi ew skills p erformance as evidenced by the results on the p rete st i ntervie w, the trai nin g interview, and the posttest interview. It was hypothe siz ed that different types of organizational environments (supportive and unsupportive ) towar ds training, would result i n difference s in the re tention of interview sk ills at the posttest. U sing multiple regression anal ysis, the resu lt s did not support the hypotheses In f a ct the results suggest that there is little va riability i n emplo y ees and supervisors' perceptions of their organizational support for training that this varia ble seems to have no explanat ory power. An att itud e of supponiveness toward training ma y instead be part of the organization's culture Whil e the hypotheses were not supported there were i mportanT benefit' t o t rainees for attend ing the tr aini ng O th er data gathered indicat e d th at some var ian ce in t he selfreported transfer composite va ria ble co uld be explained by the organizational environment variables. These results lead to 111

PAGE 4

the conclusion that trainees motivation to use the training perceived ability at applying the skills and the opportunity to use the training back at the job were the most salient variables for achieving perceived transfer of training for this group of caseworkers Because the opportunity to use the training is an orga n izational environment characteristic, the implication is that if transfer of training is to occur training must be relevant to caseworkers' job responsibilities and they must be able to use the training once they return to their jobs. The research project also concludes with some recommendations for training managers and for future researchers This abstract accurately represents the content of the recommend its publication Si IV

PAGE 5

DEDICATION To my two families : To the family I came from : my parents Charles and Judy Brittain who inspired me to go for the pot of gold at the end of the rainbow. and To the family we made : J eff, Devyn Rose and Colton Brittain Frasier who showed me the way over the rainbow Both have their place

PAGE 6

ACKNOWLEDGEMENTS Thanks to Linda deLeon for talking me through the inevitable bumps in the dissertation road and providing in sightful and helpful comments throughout the draft iterations Special tha nks to Cindy Parr y for perceptive guida n ce of all sorts through out the project and Jody Fitzpatrick for statistica l assistance Also a heartfelt thank -you to Jane Berdie, for raising the bar by her own example and being a thoughtful and sagacous mentor. Special thanks to Art Atwell and the Colorado Department of Human Services for funding the project on which this thesis is based Last, but certainly not le ast a heartfelt thank-you to everyone at American Humane Association and University of Denver Graduate School of School Work who worked on this project. It was a pleasure to work with such an esteemed g roup of individuals.

PAGE 7

CONTENTS Figu r es ......... ........ ..... .... .......... .......... .... ............................. ............... XI Tables ..................................................... .............. XII Chapte r 1 Intro duction .... .. .. .. .. .. .. .. .. .. .. .. .. .. . .. .. .. .. 1 1.1 Importanc e of Public M anagement . . . .... 1 1.2 Background Information . . .. .. .. . .. 3 1.2.1 Descr i ption ofTraining ............................................ ....................... 6 1.3 Public Management and Organizational Develop m ent ........................ .... 8 14 Rese arch Q uest io ns . .. .. .. .. .. . .. .. .. .. .. .. .. .. .. 17 2 Li te ra ture Review . .. .. . .. . .. . .. .... 19 2.1 State of Training... ... ...... ...... ....... ... ...... ................... 19 2.2 Training Evaluation ... . . . .... ............ ........ ...... ..... 24 2.2.1 Training Evaluation Models ......... ........ ....... ... ...................................... 32 2.3 Transfer of Lea rning ......... .............. ........ ... .. ... 38 2 3.1 Co n d ucting T ransfer Evaluations.. .. ..... ....... ... .42 2.3 2 The Baldwin and Ford Model o f Transfer ofLearning ........... .. .45 2.4 Organizatio nal Cultu re . ... .. . .. . . . .. ............ ........ 62 2 5 St atement of the Problem . . . .. .. .. .. .. .. . .. .. 66 2.5.1 Sample Size ................ .... .......... .. ......... .... .......... ... .... ............... 71 2 5.2 Typ e of Sub j ects ...... ....... ... .... ... .... ........... .. .... .. 71 2.5.3 Equval e n c y of Training Programs.. ..... ............................ 72 2.54 Cr ite r ion Measures of Performance .. .. .. ...... ....... ...... ....... 74 2. 5 5 Experimental Rigor ................................................... 7 6 2.5.6 Rationale fo r Current Research Study .................. ....... .. 76 3 Methodology. .. .. .. . . .. . .. .. . ......... 76 3.1 Research Project.. ........ .... ............... ..... ... ...... 76 3 2 Research Design .... ...... ... ... ................ 76 3.3 Research Sub j ects ........ .......... ....... ........ ..... ...... ................. 76 3.3 .1 Ch ild Welfare Caseworkers C har act eristics ..... ........... 76 3 3 2 S upervisor C haracteristic s.. .. ......................... ......... 76 3 .4 Instruments ... ....... .................................................. 7 6 3.4 1 I n t erview Skill Assessment Im:trument . ........... ... ..... .... ....... ... ..... 76 3 4.2 Satisfaction Survey .... .... .... ... .. .. .... ........ ... .... 76 3 .4.3 Follow up Questionnaire .... .... .. ... ...... .................... .............. 76 3.4.4 Supervisor Survey.. .... .. ................ ................. 76 3 .4.5 Qualitative Interviews ............... ................... .... .... .................. 76 3 5 Procedures . .. . . . . . .. .. . .. .. .. .. .. .. .. .. .. 7 6 3 5 1 Recruitment . . . . . . . . . . . . . . . ..... 7 6 3.5.2 Pretest of Interview Skills ............................ .......... ............... 76 VII

PAGE 8

3 5.3 Training Interview Skills Assessment ... ooo o oooo oooo o ooOooooooooooo o o ..... 76 3 5.4 Posttest Interviewing Skills Assessment ........... 0............... .. 0 ........ 7 6 3 5 5 Follow-up Questionnaire .......................... 0000000000 ............... ... .... 76 3.5 6 Supervisor Questionnaire.. ............. .. 0 ...... 0 .... 0 ......... 76 3 5.7 Qualitative Interviews .............. oo .... ................. 0....... ... .. ........ .... ..76 3.6 Data Analysis ......... 0 .......... oooo oooo ........................ .... 0 76 3.6.1 Composite Variables ...... 0..................... ...... 0 ........ 0 ............ 76 3 6 2 Confirmatory Factor Analyses ..... .... .......................... ....... ...... .......... 76 3 .6.3 Statistical Analyses ............ .... 0 76 4 Results................... ................ ... 76 4 1 Results from Satisfaction Survey .... .. .. .. ... .... ........... .... 00 ..... 00 .. 76 4 2 Results from Follow-up Questionnaire ............... o ................ 76 4.2 1 Interven ing Factors on Follow-up Questionnaire .......... .... .... ............ 76 4 3 Results from Supervisor Questionnaire .... .. ....... ..... 0 0 ... 0000 .. 0 76 4.4 Comparison of Supervisor and Trainee Questionnaire ........ ....... .... ........ 0 76 4 5 Results from Composite Variables.. .. ...... .. .......... 76 4.5.1 Relationships between Trainee and Supervisor Questionnaire Composite Variables .... .. ........ ..... .............. o ...... 76 4 .6 Agreem ent Regarding Video-tape V iewing ...... ............. o .......... o ... o ........... 76 4. 7 Test Scores Results.. .. .. . . .. .. 76 4.7.1 Raw Score Results ..... 0 ........................... o ................. o ........... 76 4. 7 2 Rase Score Results ................. .. 76 4 0 7 3 Pretest and Training Test Scores as Predictors of Posttest Score ......... .......... ............ .......... ......... 0 0 ............. o 76 4 8 Primary Analysis of Major Hypotheses ( Retention Predicted by Organ!zational Environment) ....... .......... ............ 76 4 8 1 Relationships between Retenti on Measures Test Scores and Key Variables ..... o...... .......... .............................. 00 ......... 76 4.8.2 Multipl e Regression Analysis ofRetention with Hypothesized Predictor Variables.... 0 0 .................... 76 4 9 Secondary Analysis of Other Transfer Measures (Test Scores Predicted by O rganizational Environment) ................... ..... 76 4 91 Relationships between Retention Measures, Test Scores and Se l f-Reported Transfer....... .. ...................................... 76 409.2 Relationships between Adjusted Test Scores and Organizational Environment Variables A djusted Training Score...... .. .. .................. ...... ................ ........ o 0 .............. 0 76 4.9 3 Multiple Regression Analysis of Self-Reported Transfer Predicted by Organizational Environment ........................... oo .. 0 00 ... 76 4 .10 Results from Telephone Interviews with Research Subjects .. . .. .. 0.. .. .. .. 0 .. .. ....................... 00" 0 .... ... 76 Vlll

PAGE 9

4 .11 C oncluding Observat i ons . .... ............... .... 76 5 D is cussi o n . . . . . .... .... .... ......... .... .... ... ...... .... 7 6 5 1 O v e rv iew of Significant Findings....... ...... ..... . ... .... ...... ... 76 5 2 A lternat iv e Explanations for Transfer of Training ........ ... .... ......... 76 5 2 1 Relationship between A d j usted Retention and Predictor Variables . . ... .... . . . ............ . ..... 76 5 2.2 Relationship between Ad j usted Training Test Score a nd Predictor V ariable s . . . . . . . . . .. . . . . 76 5 .2.3 Relationship bet w een Adjusted Posttest Score and P r edi c tor V ariable ... ..... . ...... .... . . ............... ... ..... .... ......... 76 5 .24 Rel a ti o n s h i p bet w een T rainee S elf-Reported T ransf er an d P redict o r V a r i ables 5 2.5 Differences in Opportuni t y to U se 7 6 ..76 5 2 6 Subjects Emotional Reac tions... .. ........ ... .... ... ...... . . ... .. ... ..... 76 5 2 7 Othe r Types ofT r a nsfe r ar e O c curring .... ... .......... ............... ... 76 5 2 8 A Consideration o f Culture.. .... ... .. .. ............... ... . ........... 76 5 2 9 Summary of Alte rn ative Explanations . . . . . . . . . . . 76 5.3 C onsiderati o n ofthe Relev a n t L iterature ... . ...... .... ... .... .... . .... .... ... 76 5 4 S t u d y L imitatio n s ..... ... ........... .... ........ .......... .... 76 54.1 S e l ect i on Bias . ... ...... . ... . .... ..... ... . . ... ............. ... ... 76 5.4 2 Time Frame ..... ....... .... ..... ...... ...... ..... ..... . ........ ... ...... 76 5.4.3 Additional Trainin g or Research ... .... ... ... ..... ............ .... ....... 7 6 5.4 4 V ideotape Rat ers .......... ... ... ... ....... 7 6 5.4. 5 Te. t Condit i ons ... ...... ... ... ... . ... .... .................. .... ... ... 76 546 Qu e stionnaire Desi g n ....... ........ ............... . ... ....... 76 54.7 C omparison ofProblem State m ent Crite ria and this Research .............. 76 548 S ummary of Study Limitat i ons ... ..... ........ ..... .... .... ..... ... 76 5 5 I mpl i cations o f the St u dy for Professional Practice ...... ... ... 76 5 51 Reconsideration ofTr ain i ng Curriculum ...... ..... ..... 76 5 5.2 R eco n sider a t io n o f the Training Participant s.. ... ....... 7 6 5 5 3 Rec n sideratio n o f the I n t ervie w I ns trument a nd V ideotape Rating ... ... 76 5 5.4 Reconsid e rati n of the Transfer Instmmen L ...... ....... ... ... 76 5.5. R e commendat i o n s for T r a i ning Man agers .... ....... ....... .... .... 76 5 6 Recommendations for Further Research .. .... .... ....... 76 5 7 C onclusion ............. . ....... . ... ...... 76 Append i x A Description of Coordination betwee n Research Projects.. ...... ... ... 76 B Specialized Interviewing Skills Interv iew Evaluation ........ . .... ... ... ...... 76 C Sat i sfaction Survey .... .... ..... ... ... ... . .... ... ... 76 D Tr a inee F ollo w -up Questionnaire ...... ... .... ........... . 76 E S up ervisor Follo w u p Questionnaire . . . 76 lX

PAGE 10

F. Transfer ofLearning Qualitative Interview Questions G. Recruitment Letter and Consent Form ........... H. SIS Data Collection Plan .......... ........... I. Task Letter References. ....... ..... .. ... ... ...... .... ............. ......... ..... ... .... X .... 76 ......... ......... 76 .... 76 ...... 7 6 ..... 76

PAGE 11

FIGURES Figure 2 1 American Humane Association s Model for Stra t egic Evaluation. ....... . . . . . . . . . . . . . . 36 2.2 Baldwi n an d Ford s Model for Transfer ofTraini ng ............................... 47 4.1 Comparison of Rasch Adjusted Scores for Pretest Training, and Poshest ............... ... .. ............ ........... .... ............ .. 76 5.1 G raphi cal Representation Comparing Opportunity to Use .............. .. 76 X I

PAGE 12

TABLES Table 2 1 C o mpar i so n of Training E v aluat i o n Modeis .... 33 2 2 Summary of Transfe r and Organizational Environment Research Studies ............. ....... ....... ...... .... ...... .... . ..... 67 3.1 Research Subjects and Curr en t Job Titles ...... 76 3.2 Research Subjects and Unit A ssi g nments ........... . ... ....... ... .... ..... 76 3.3 Educat io na l Levels ofResearch Subjects (Trainees) ...................... 7 6 3.4 Ethnicity of Research Subject s (Trainees) ...................... .... ... ... ... ....... 76 3 5 Superviso r s Current U nit ...... ..... ..... . ............... ..... 76 3 6 Data Coll ec ti o n Points and Instruments .................. .................. ... ..... .. 76 3 7 Rating Adjustment Calculation. . . ....... ........ 76 3.8 D efi n ition s of C o mp os ite ariab le s ..... ..... ...... ....... ........ ......... 76 3 9 F a cto r L oading ofltems Containe d within Factor 1 Transfer/Used train i n g and Previously Designated Composi t e Variable . .... .. ....... .... .......... .......... 76 3 .10 F actor Loading ofltems Contai n ed wit h i n Factor 2 : Org anization Sup port and Previously Designated ompos ite V aria bl e ........................... ......... ........ ... ......... ...... ...... . . ...... 76 3 .11 Factor Loadi n g of Items Contained within Factor 3 : Supervisor Support and Prev i ous l y D esig nat ed Composite V aria ble ......... .... ..................... .... ...... .... ... ....... ... ..... 76 3.12 Factor Loadi ng ofltems Contained within F acto r 4 : Trai n e e Ch ar acterist ic s and P r eviousl y Designated Composi t e V ar ia ble .. ..... .......... .... ......... ................ ... . ...... 76 3 .13 Facto Loading ofltems Contained within Factor 5 Co-Worker Support and Prev i ously Designated Composite Variable ...... ... ...... .... ............ .... ... ... ....... 7 6 3 .14 Research Questions Hypotheses Variables and Sta i stical Tests .... ... . . . . . . . . . . . . . . . . . . . ... 76 4.1 Responses from Questions R el ated to Works hop Content .... 7 6 4 2 Trainees Perce iv ed Achievem ent of Trainin g Compet en cies . . . . . . . ...... ...... ............... 76 4 3 T ra inee Perceptions of Trainer's Abili t y ...... .......... ... ... .. ......... ... ... 76 XII

PAGE 13

4.4 Descriptive Stat'stics from the Trainee Follow-up Quest i onnaire . . . . . . ...... .......... .. 76 4 5 Descrip t ive Sta istics oflntervening Factors from the Trainee Follow-up Questionnaire . .... .................. . ....... 76 4 6 Descript i ve t atistics of Superv i sor Survey ...... ....... . . .... ... 76 4 7 C omparison of Supervisor and Trainee Means for Ques t onnaire Items ........... ... ... . .............. .......... ... 76 4 8 D e scriptive Statist i cs ofComposite Variables .. . ...... ...... 76 4 9 Correlations between Trainee and Supervisor Reported Trainee Va r iables .. ....... . . 76 4 .10 Cor r elations between Trainee and Supervisor Reported Organizat i onal Variables .......... ......... ..... .... ............ .... ..... 76 4.11 Indivi dua l Items Composing the Organizationa l Supp o rt Variabl e .... ... . . ......... .. .. ... .... ....... ........ ... 76 4 1 2 C ompar i son ofPr ete st, Trainin g and Posttest Raw Interview Scores Adjusted for Mean Differences in Raters ...... ... ... . ......... 76 413 Comparison ofPretest, Training and Posttest Rasch Interview Scores Adjusted for Mean Differences in Raters. .... . ...... . 76 4.14 Paired Samples TTest for Pretest T raining and Posttest. 76 4.15 Multiple Regression Results with Adjusted Posttest Score on Adju s t ed Pretest Score and Adjusted Training Score ... ........ ........ .... 76 4 .16 Cor r e l ations ofKey Var ia bles and Adjus t ed Retention from Training to Follow-up .... .......... ..... .... ..... ........... ........ .............. 76 4.17 M ultiple Regression Results ofRetention on TraineeReported Organization Support Supervisor Support and Co-Worker Support. .... .. ........ ... .... ........ . ... ......... 76 4 i 8 Relat i onships between Ret ention Measures ...... ....... . 76 T est Scor e s and Self-Reported Transfer .. .. . .. . .. .. . ....... ... ........ 76 419 Relationship Between Training and Posttest Scores for Trainee Reported Organizational Environment Variables ..... ..... .... ........ 76 4 2 0 M ultiple Regress i on Results with Training Test Score on Trainee-Reported Organization Support, Supervisor Support and Co-Worker Support 4 .21 Multiple Regression Results with Adjusted Posttest Sco r e on T r ainee-Reported Organization Support ....... . 76 Supervi or Support and Co-Worker Support ....... .... .................. ... .... ..... 76 4 22 Rela t ionships between Trainee Reported Transfer and Ke y Variables.. ....... ... . . .... . .... .... ....... . 4 .23 Multiple Regression Resuits with Trainee Self Reported Transfer on Trainee-Reported Organization ..... ..... ...... .... 76 Support Supervisor Support and Co-Worker Support .... ..... . .......... 76 X Ill

PAGE 14

5.1 Correlations between Adjusted Retention and Pre d ictor Variables ........ .............................. ....... ........... ... 76 5 2 Correlations between Ad j u st ed Training Test Score and Predictor Variables.. .................................... .............. .. 76 5.3 Multipie Regression Results with Adjusted Training Sco r e and Predictor Variables. .............. ............. ..... 76 54 Corre l ations between Adjusted Posttest Score and Predictor Va riables ...... .... ...... ............ 76 5 5 Corre l ations between Trainee Self-Rep orted T ransfer and Predictor Variables . . ... .. ... ..... ......... ..... ... 76 5 .6 M ultiple Regression Results with Trainee Self-Reponed Transfer and Predict o r Variables ........................ ..... .... ....... . ..... 76 5.7 Comparison ofMean Adjusted Training Test Scores and Retent i on by Opportunity to Use...... ......... .... ..... .... ........ 76 5 8 Potential Transfer Interventions by Person and Time Period .. 76 XIV

PAGE 15

1 Introduction 1 Importance of Public Management Governments exist amongst many other mandates to protect their most vulnerable citizens, especially chi ldren from harm. Child welfare organizat i ons as an arm of government represent one set of insti tutions charged with that important obligat ion of protecting children from abuse and neglect. While carrying out their duties, child welfare organizations are subject to the same scrutiny and oversight as other public organizations and are accountable to the public for how tax dollars are spent and dut ies performed Public managem ent s triv es to make public organ i zations more effective and efficient or the common good; pu b lic management and more specifically, organizational development, has long been concerned with whether management (things managers do) really matters in i mpro ving the efficiency and effectiveness of public organizations Publ i c child wei fare organizations prepare c hild welfare c aseworkers for their important jobs of protecting childre n through organizational development techniques one of which is t rai ning child welfare caseworkers on the skills necessary to perform their jobs Training, however is only effective if it is transferred t o on-the-job performance ; otherwise it is waste of the public's

PAGE 16

investment and the organization's attention Public agencies are always under p r essure tO improve efficiency and therefor e the more that can be understood about the problem of training transfer, the better for public organizations Wilson made the call to st u dy more effective techniques for administration ( Wilson, 1887) and many scholars and practitioners have answered that charge over the la s t hundred plus years In that spiri t t h is research project seeks to explore the contextual factors affecting the transfer of training by evaluating a course for child welfa r e caseworkers to teach them specialized interviewing skills for investigating and assessing situations in which children may have sexually abused Specifically the research project studies the transfer of training from the classroom to the j ob and the factors that facilitate or inhibit that transfer Th i s research project was coordinated with the Colorado Depanment of Human Services (CDHS), American Humane Association (AHA) and Univers ity ofDenver' s Graduate School of Social Work (DU-GSSW) A description ofthe coordinati o n between the three organizations and the roles for each can be found in Appendix A. This a t hor has been in v ol v ed in some aspect of the project for approximately ten y ears wor king at va r ious times for all three organizations At this time she is current y employed by AHA as part of their independent team of third-party evaluators While she has had a long history with the project at the state level this research was primarily conducted to determine whether there is a relationsh i p between county-based organizat iona l support for training and 2

PAGE 17

retenti o n o f i nterviewing skills-an area in which this author has not worked nor had any invol v ement and thus is not subject to undue bias To avoid even the ap p earance of a conflict of interest measures were taken by AHA and DU-GSSW at the beg i nning ofthe project .o ensure balanced results For instance video t apes were rated by DU-GSSW employees (to be explained later) and the t es t s c o r es entered into AHA s database by other AHA employees A more detailed explanation of the author's history with the project can be found in Appendi x A 1 .2 Background Information Int e rv i ewing, and specificall y sexual abuse interviewing is an important and necessary skill that child we l fare caseworkers must possess in order to do their jobs effectively. The sexual abuse inte r view is different from other child wel f are i n terview s for several r easons By i t s nature, the topic is uncomfortable f or b o th t h e int er v iewer and the child s i nce t ypically children do not want t o talk a bout this taboo subject and inte rv iewers may not want to hear about it Secrecy i s o n e of the dynamics of sexual abuse that perpetrators use to keep themselves from b ein g exposed and an y kind of sexual contact-especially with children between family members is extremel y uncomfortable to discuss and hear about. Specialized interview techniques should be used and practiced to make the inter v iew more comfortable for each person and to ensure the legal integrit y of 3

PAGE 18

the interview for p ot en tial su bsequent legal proceed i ngs to protect the child and prosecute the perpetrator(s). Sexual abuse cases are more likel y to go to both civil and criminal court than other types of abuse and neglect cases and therefore are more vu lne r able to dism issal or other legal compromises if there is insufficient atten t ion to professional standards and protocols For instance a criminal court judge may dism iss a c a se if a caseworker used leading questions to obtain the information about the se x ua l abuse incident Lastly, the initial sexual abuse interview sets a context for future treatment The child must not be frightened or t ra umatized b y the interview and / or sub sequent interaction with Social Services or psychological tr eatment to help the child dea l with the abuse may be comp ro mised Before this type of training was offered caseworkers had l ittle formal opportunity to devel p their interviewing skills at a formal train i ng, especially the specialized skills n ecessary to conduct comfortable a d p roducti v e sexual abuse int erv iews. These interviewing sk i lls can be generalize d to assist with other types of interviews such as those to assess for physical abuse or neglect and can be u sed to fac ilitate ongoing case management interactions with all family members CDHS recognized t he importance of goo d inte rvi ewing sk ills and particularly the speci a lized skills necessary to perform sexual abuse intervi ews; in 1985 C DHS partnered with DU-GSSW under the auspices of a federal th ree y ear grant to develop and implement t h e Specialized Interviewing Skills ( SIS ) training 4

PAGE 19

to teach caseworkers the skills necessary to interview children who may have been sexually abused. An interview protocol based upon good practice research and legal requirements w as developed and became the foundation for the SIS training curriculum. Also, the project developed a novel approach to training at that time b y videotaping the caseworker as he or she conducted a simulated interview with an actor who role-played a child sexual abuse v ictim. The videotaped interview assessed skill performance and was an integral part ofthe curriculum by providing an opportunity to evaluate t he interview and give xtensiv e feedback on interview performance to the trainee Also, the project developed a formal Interview Skills Assessment Instrument (henceforth called "Interview Instrument") based u pon the interview protocol to measure skill per fo rmance during the videotaped interview. Live videotaped interviews with ch i ldren were not u ed because of e t hical issues lack of equivalent situations poten t ia l dis omfort, and even trauma to a child Therefore, it was decided to use simulated interviews with an adult role -playing a latency-age child. A..fter the federal grant expired CDH remained committed to the SIS training and integrated it into their schedule of classes for new child welfare caseworkers CDHS currently contract with DU-GSSW to provide the SIS traini g, coordinate the actors to role-play the victim, and rate the videotapes using the Interview Instrument. The trainer who originally developed the SIS training during the 980s remains the primary trainer with another trainer 5

PAGE 20

occasionally substituting This author also worked as a graduate research assistan t on the original project from 1989 through 1992 role-playing the child sexual ab u se victim a nd rating the videotapes Currently the author is a staff pe r son on AHA s third party evaluation team (for a more complete description of her inv olvement in the SIS training over the last ten years see Appendix A). I 2.1 Description of Training T h e SIS tra i ning is a three-da y course to teach caseworkers how to conduc t a exual abuse interview us i ng the formal interview protocol. The intervie w protocol is designed to increase accurate disclosure minimize discomfort to the child and help to raise the legal integrity of the interview by prov i ding s pecific guidelines for the interviewer's conduct during the interview During the first day and a half of training the interview protocol is presented and practiced i n v arious situations i n the classroom On the afternoon of t he s eco nd day, the trainees pe r form a simulated interview with an adult who is r o l ep laying a latency-age child who has been sexually abused as dictated by a standardized scenario During t he original project development phase at DU GSSW, graduate students role played the child sexual abuse victim as part ofthe research p roject Currently p ro fessional actors role-play the sexual abuse victim Ei g ht s cenarios of approximately equal difficulty and sexual abuse severity were devel o ped and are rotated dur i ng the trainings to give trainees exposure to 6

PAGE 21

different situations and types o f children (e.g., shy, reluctant children versus v e r bal, forthcoming children) Pri o r to the videotaped simulated interview the trainees are given an intake form contain ng basic information about the child his or her family, and the presenting situation The trainee's task is to establish rapport with the ch i ld, determine whether sexual abuse occurred and if so obta i n the details of the incident(s). The trainer re v iews each videotaped interv i e w dur i ng the evening of the seco n d da y and selects strengths and weakness of each interview On th e thi rd day of training the trainer presents clips from each video tape to illustrate important int e rviewing concepts reinforce positive interview skills, and help correct skill deficiencies Next the videotapes are sent to DU-GSSW for formal rating b y a tra ined g raduate research assistant using the Interview Instrument Once rated the completed Interview Instrument rating and add itio nal written feedback are sent back to the trainees for review with their superviso r to help them furthe r enha c e and refine their sexual abuse interviewing techniques Wh i le the feedback is sometimes excruciatingly painful for trainees to hear these mul t ipl e sources of feedback are an effective method for developing s ills so caseworkers can perform bett e r interviews w ith children Better inte rv iews mean that situations are asses sed more accurately and children are better protected from harm-the p ubl ic child welfare organization s main responsibil i ty 7

PAGE 22

Public management is the tool used to manage child welfare organizations and has a long tradition striving to more efficiently an d effectively administer organizations for the public good The following section contains highlights of the literature rela ted to the development of pub! ic administration and spec ific ally public management, with particular emphasis on organizational development 1.3 Public Management and Organizational Development The early public management literature was concerned with improving the concrete aspects of management and understanding the nature of bureaucracy and administration. Scientific management was developed by mechanical engineers to improve employee motivation (Martin 1989) The sc ientific management method min utel y anal yzed e ery ta k performed by the worker with time -motion stud ies in order to increase efficiency. One of the founders of the scientific mana gement movement Frederick Taylor ( 1992) said it was management's responsibility to develop the sys tems and heuristics that would save the worker and the company time and money Managers also were responsible for personnel duties such as selection and training According to Marc h and Simon ( 1958), Taylor anal y zed the interaction between the characteristics of humans an d the social and task environments crea ted by organizations and thus laid the foundation fo; contemporary organizational development 8

PAGE 23

In contrast to scientific management's mechanistic efficiency the human rela ions school at empted to thaw management's icy grip on the workers Gantt ( 1916) was, to some, the first humanist and utilized the field of psychology to improve employee motivation Workers and management are each mutuall y dependent upo n the other he suggested; therefore work morale is imperati ve to organizational success Trai nin g of all w orkers is one such way of attaining mutual goals ( Gantt, 1916). Gilbreth (1914) inked the scientific management and psycho log y fields by applying psychological principles to management The Hawthorne experiments created the humanist school within the study of administration, promoting the idea that social leadership skills as important as technical knowledge (Manin, 1989) Follett ( 1926) further tried to humanize the relationship between management and worker. She dispensed with the giving of orders and instead argued for uniting around a particular situation and letting the situation dictate the solution The nature of organizafonal boundaries was defined by theorists including Barnard and Simon Barnard (1938) defined t he symbiotic relationship between informal and fo rmal organizati ns. Informal organizations set the cultural context w i thin a formal organization by establishing customs, norms, folklore, institutions and ideals. Inattention to the informal organization weakens the formal organization. Simon ( 1945) said that constructing an efficient administrati on meant hiring an operative staff then superimposing on that staff a 9

PAGE 24

supervisory staff to influence the operative staff to coordinate beha vior. Employee decisions and behaviors are influenced within and by the organization Employees can be influenced two ways in order to carry out decisions made by the organization The fir t process indoctrinated the employee into the ways o f the company or trained him while t he second process used authority. Simon (1945) one ofthe preeminent theorists ofpublic administration explicitly stated the impo rtan ce of training to the entire administrat i ve process In the mid-twentieth century, public administr ation and specificall y public management, scholarship we nt down several different paths including leadership th e ory personnel management decision-making theory fiscal management, and organizational development. Thes e paths often intertw ined as public management scholars tried to understand the process of management in the continuing effort to make it more efficient and effective. Drucker ( 1 954) was among the most significa nt contributors to the o ries of or anizational development His contribution was far ranging and had a profound effect on bo th public and private management practice Melding the precision of the mechanical engineers with the compassion of the management humanists, he developed a goal-driven management theory He minimized the scientific man agemen t approach as simplistic and even unworkable and took a much more holistic approach to management that set high standards in order to achieve goals. Man a gement's ultimate goal was business performance and there are three jobs of 10

PAGE 25

management: managing a business, m anag i ng manag ers and managing and wo r k Drucker wittily rem i nd ed us tha t you cannot hire a hand," as i ts owne r always comes with it (1954, p 262). Therefore in order for managers to mot iva t e an e mp lo ye e they must attend to the whole pe rs on Many theor ists explored t he nature of the relationship between the organization and the indivi dual (fo r inst ance Argyris, 1957 ; Golembie wski & Eddy 1978 ; Marc h & Simon 1958 ; McGrego r 1960) Organiz at ional devel opm ent theorists emphas'zed the growth ofthe individual as a way of achieving h i gh business pe r formance (Ar gyris 1971; Golembiewski & Kiepper 1988; Senge, 1990) Systems theory helped to exp l ain how i ndi v iduals reacted to thei r environment and also offered approaches to improving management w ithin the context of an organizational sys te m (Katz & Kahn 1 966; T h ompson, 1967). Argyris (1957) l i n k ed Barnard and McGregor (d is cussed below ) in his groundbreaking book on indi viduals an d formal o rga nizations, which tried t o explain why people behave as they do in organizat i ons Many of his contentions are more clearl y enunciated in McGrego r s Theory X and Theory Y management sty l es In McGregor' s Theory X workers are essentially indolen t Argyris however believed w ork ers were not lazy bu t rather are constrained to childlike dependence ; man agem ent practices desig ned to allow full expressio n of their dignity and creativit y will benefi t b oth the worker and organization Management be cogn izant ofboth formal and informal organizations Inherently, there is ll

PAGE 26

an incongruity between the requisites offormal organizations and the self actualization of individuals (Argyris, 195 7) McGregor ( 1960) differentiated between Theory X and Theory Y manageme t. Most companies operate d under Theory X management believing that worker were basically lazy passive, resistant to change, and indifferent to the organization. Theory Y, on the other hand emphasized the competence and self-direction of the worker, so manage ment should encourage wo rk ers' assuming responsibility for the organization through such techniques as decentralization and management b y objective (McGregor 1960) March and Simon's (1958) seminal book postulated the nature of organizations and the relationship between individuals and organizations. It argued that formal organizations do have a significant effect on the behavior of individuals within organizations In particular, influen ce theory modeled the behavior of individuals within organiza tions as functions of various inputs that determine whe ther an employee will join and stay with an organization and once in it, whether he or she will or will not be productive. Perceived consequences of an individua l's actions are a fl .. mction of the environment individual characteristics, sub-group pressures, and organizational rewards All behavior within organizations is bounded by the limits of individual rationality, which results in satisficing (March & Simon, 1958) 12

PAGE 27

Building o n his earlier wo rk Argyris in Perso nali ty and Organization ( 1971) further refined his theory of organization development. He borr o wed McGregor's Theory X andY management sty les and added Patterns A and B. The patterns are the per onal be havior, group dynamics and organiza t ional norms a ssoc i ated wi th T heory X or Y ( Argyris 1 9 71, p xi). McGregor s theories were about the w o rke r and his or her motivations while Argyris patterns were about the organizat ion Argyris advocated a Theory Y Pattern B approach a much more huma nis tic and holistic approach to management Organizational de v elopment i s the process by w hich this is achieved He stro ngly be l ieved in the importance o f organizational develop ment for de ve lop ing organizational health and human potent ial and dign:t y Organ izationa l deve lopmen t was touted a s helping the organization's efficiency and increasing the growth and sat isfa ct ion of employees (Go lembiewsk i & Eddy, 1978) The field of organjzational development i s complica ed due to it s varied phi loso phical fo undation s of rationalism pragmatism, and existentialism and faces unique prob lem s and challenges in public agenc ies that have l argely been ignored (Go lemb iewski 197 8 ) F ou r properties specific to pu l ie institutions inter fere w ith achieving organizational goals m u lti ple access ; greater vari ety of groups and indi v iduals ; command linkages; and weak linkage s between political and career levels ( Golembiewski 197 ). 13

PAGE 28

Golembiewski and Kiepper (1988) developed a more mature strategy for orga niz ational development reminiscent of the popular Peters and Waterma n book, in Search of Excelle nce (198 2). High performing units were characterized by specific traits tha t emphasiz e d high energy co mmitment te am spirit and action amo n g others Organiz ational development programs should attend to both t he in dividual a nd the organization. and the g rowth of each will resul t in sa t is f act i on enhan c ed s k ill and com pe te nce wi t hin interpersonal and inter g roup situat i ons ( Golemb i ewsk i & K i epper 1988) They described ten characteristics of h i gh perform ance se ttin gs fr om organ ization structure to physical lay-out and i ncluded tra ining as one of these ch arac ter istics (Golembiewski & Kiepper 1988). Many authors ha v e p re-ented p r escr iptions for imp rov i n g mana ge m e nt techniq ues but their effect often seems ne gl igibl e a s the same issues often called different nam e s (e.g. mo rale, mo t ivation), a r ise over and over again The qu estion plaguing public man age men t i n the late 1990 s is, does public managemen t matter? Theor is ts trying to answer this questi o n want t o know whe t her t e practice of management s uc h as diffe r ent m otiv ation techniques or t raining, ca n make a differe n ce in terms of outcomes The interaction between the o rganizationa l culture, mana gement practices, an d institutional context predicts the behavior of organizations (B ehn 1991; O Toole 1997; Khademian 1 99 7) Mana g eria l actions (or inactions ) influe nc e the structure, policy and e ve n cu lture of organizatio n s Q uestions ari se as to whether cultu r e i s a variable to be 14

PAGE 29

manipulated-much like the childhood toy silly putty in order to achieve o r ganizational purposes or w h ethe r it is a factor tha t pe rvad es organizational life with managers having little abi lity to control o r change i t (K hade mian 1 997). S o me researchers say that the former interpretat ion o f organizational culture i s v alid and therefore justifies t h e ro l e of management Success lies in the extent of the manager's leadership effort s to create an organizational culture oriented towards program outcomes (Behn 1991 ). Pra c tical how-to strategy boo s preaching a princip l e-centered message dominated the popular bu s iness litera ture during the l a st decade (for in tance Behn 1991; Covey 1991; Spencer 1998 ; Zi g lar. 1 9 94). Se n ge ( 1990) mo v ed t he management literatur e into the 1990s with his book o n the learning organizat i on He tried to close Argyris gap between the organization and the individual Truly ex cell ent organizations are t hose tha t tap into indiv i duals comm i tment and are oriented to learnin g at all l e ve l s thereby creating an organizational culture ded icated o growt h He lab eled these work environm ents learning organizations and argued that they are achieved by a commitme nt t o the fifth discipline," which is systems thinking or the abi lity to see i nterrelat ion ships and patterns. Man agem ent is then orien te d to the goal of becoming a n d remaining a lea rning organization. In an other paper in the does management matter ?" v ein O'Toole (199 7 ) stated that net w or k contexts offer varia bles that are Khademian's silly putty in 15

PAGE 30

that they can be manipulated t o achieved desired goals 1 etworks will r eplace the old h i erarchica l system and are more realistic of today s management environment Funher research on organizational culture within an organization s insti t utional c ont ext is nec es ary In contrast to the methods of best practices research, org anizat ional c u ltures ofwell-p e rforming and poorly-performing units should be studi ed to determine the ur ique character istics of each to i nform management de c i sion m a king (Khademian 1 997) Understanding organizational cul ture is central to th e future of public management : Indeed, a reliance upon culture ma y be the only viable way to manage some organizations Di covering how deep ly embedded culture s m i gh t be m anaged ( that of the IRS an d the CIA come to mi n d), or ho w to manage arou nd uch cultures is a k ey challenge for public manageme n t res earc h (Kha d emi an 1997 p 16). The uti l i t y o f managing culture, however is dampened b y the difficult y of under st a ndi n g i ts rol e w ithin organizations and the connections between abstract cultural c o ncept s and the realit y ofthe o rgani zational environment (Khademian 1997). The public management literature traces a t heoretical thread concerned w i th devel oping an efficient and effective organization while attending to t he needs of indi v iduals within organizations This body of lit erature is concerned with determining t he r ight mana ge ment techn iq ues in order to achieve organizational performance The does management matter ?" public 16

PAGE 31

management literature sets a managerial context for creating an organizational culture to influence the transfer of training to on-the-job performance Training is but one method of many managerial techniques designed to increase the efficiency and effectiveness of public organizations w hile reco gnizing the importance and needs of individuals within organizations It remains unclear however, w hether training realiy does improve performance which leads to the quest i ons this study seeks to answer. 1 4 Research Questions This author's interest in understanding the role oftraining for improving organizational effectiveness and the factors that mig ht affect that, led to the follo w ing research quest ion s for this study : First, does training improve interviewing skills performance? Before understanding the factors that influence the transfer of training it is important to establish th at training does indeed improve interview skills performance by examining scores on the pre, training, and posttest. It would be expected tha t scores would be at their lowest at the pretest improve significantly after skills are learned at the training and test e d at the training test th e n scores would decrease slightly over time for the posttest (the effect of forgetting), but still remain greater than the pretest level. People who Jearn more in training also perform better on the job If this should prove to be untrue there would be no r eason to test for the 17

PAGE 32

effects of t he transfer climate (Rouiller & Goldstein 1993) the next area for resear ch Second which organizational factors are most influential in promoti n g t h e 1ransjer of tra ining.? The goal ofthe research project is to understand the organizational factors while controlling for other factors th at may con tribute to learning retention and thus transfer. Three types of factors charac teri ze the organizat i onal environment: organizational cli mate sup ervisor support and co w orker support and will be e plor ed to understand their impact on tra n sfer. The reasons why t hese particular factors are the focus of this research w ill be explaine d in the next chapter. T he next chapter s ets a context for these res e arch qu estions and discuss es how and why the top i c of the t ransfer of training s importan t to the field of p ublic manageme n t 18

PAGE 33

2 Litera t ure Review The literature related to training, t r ansfer of learning and culture is exp lor d in this chapter to establish a context for this dissertation Multiple bo1ie s of literature pertinent to the transfer of training a r e explored The theor tical thread starts by discu sing training, a common organizational de v elopment intervention strategy Next the focus moves to a detailed discussion o f tra i ning evaluation and then assesses the literatu r e specifically r elated to trans f er oftraining The last sec t ion focuses on the literature related to research studies oftransfer of training and the organizational envi r onment to validate the reason for this research project and understand the rationale for its structure Such a broad academic exploration is necessary to understand the context of transfer of tra inin g w i thin contemporary public management research and theory 2 1 State of Training It is estimated that American industries spend $100 billion annually on staff development $30 billion on formal training, and approxima t ely $180 billion on informal on-the-job training ( Broad & Newstrom 1992; Carnevale, Gai ner & Villet 1990 ; Georgenson 1982; Gist Stevens & Bavetta, 1991 ) Training expe n ditures continue to increase significantly each year but generally there is a 19

PAGE 34

lag between training and performance ( Bassi & Van Buren 1 999), thus making it d i ffic u lt to measure a return on investmen t (Abe rnathy 1 999) Given the changing American workforce and strong economy t raining will continue to play a significant role in recruiting and retaining quartied employees (F acteau Dobbins, Russell Ladd, & Kudisch, 1 995) It might be expected that government organizations wo ld spend more on t raining due t o two contrasting reasons : either go v ernment is wasteful and spends too much money on training or more optimistically that government inv ests mo r e in trai nin g because workers need i t to serve the people more effectively Government organizations howe ver rank near t h e bottom in terms of training expenditure s as a percentage of payroll (Bassi & Van Buren 1999) Training and educa ion can be used as a strategy to improve organizational effec t iveness and competitiveness (Saari Johnson McLa uglin & Zimmerle 1 988) by changing worke rs attitudes, improving t heir knowledge, and developing their kiils After training worker attitudes were shown to improve in several studies (Ch affin, Kelleher Harber Harper & Crone, 1994 ; Gregoire 1 984; OSG, 1995 ; Pecora Dodson Teather & Whittaker 1 985) Other studies showed an increase in knowledge before and after training (Chaffin et al, 1994 ; Cheung Stevenson, & Leung 1991; Cronin & Drake 1995; Schinke Smith, Gi l christ & Wong, 1981; S ulli van & Clancy, 1 990) Others were able to determine that sk ills 20

PAGE 35

dev loped as a result oftraining (Cheung et al., 1991; Cronin & Drake, 1995; Osgood 1995) Training can improve the utilization of services such as substance abuse ueatment because learning about the system and available treatment increases the knowledge and comfort level of caseworkers seeking to use those services for th eir clients (Chaffin et al, 1994; Sullivan & Clancy 1990 ) A study about the impac t oftraining on c hild welfare casewor kers in Pennsylvania found that nine fewer children died from child abuse than the previous year (Osgood, 1995), and while i t is difficult to associate those results with training, it certainly helps to rna es the case for train ing as an important management tool to help reduce child abuse Despite the positive benefits of training seldom is it developed or evaluated systematically so that training can be used to address the most imponant questions fa cin g an o rganization (Saari eta!. 1988). Only one -t hird of U S compa nies conducted some type of training needs assessment and 42 percent of companies failed to conduct evaluations of training effectiveness (Saari et al, 1988 ). In contrast educational institutions are noted for their emphasis on accountability of learning through tests grades and evaluation while trainin g in industry (including government and non-profit sectors) is much less so occupied (Baldwin & Magjuka 1991 ). Kaufman & Keller (I 994) blame less-than-effective evaluation on three factors: evaluation models are too restrictive; evaluations do 21

PAGE 36

not ask the right questions ; and the relationship between ends and means is unclear. Acco rdin g to some researchers, only about 10 percent of t rainin g expenditures actually result in transfer to on-the-job performance or, conversely 90 percent of the information learned in training was not used by training participants (Broad & Newstrom, 1992; Carnevale et a!., 1990; Garavaglia, 1996; Georgenson 1987 ; Gist eta!., 1991 ; Marx, 1986 ; Seaburg, 1982) Jn a study of chiid welfare caseworkers 28 percent of participants did not transfer training to onthe-job performance (Curry, 1 Q97) Training managers more optimistically est i mate that 40 percent of training w as transferred to on-the-job performance ( ewstrom 1985 ; Seaburg, 1982). While companies may believe in the value oftraining as eviden ced by the enormous amount of money being spent on it they do little to attend to important training principles such as conducting needs assessments and rigorous evaluat ions. In the literature specific to child welfare training, Vinokur-Kaplan (1986) conducted a landmark national study in 1981 to evaluate child welfare in servi c e training Res Its glaringly point to the weaknesses of child welfare traini t g programs during the late seventies and early eighties. Approximately three-quarters of participants surveyed stated that they had a formal in-service train in g program while the balance had either an informal program or none at all. Factors most influencing the effectiveness oftraining from the Vinokur-Kaplan 22

PAGE 37

study were : tra ning relevance ; a needs assessment to determine training needs ; ince n ti v es for worker participation ; adequate agency support for training ; on the job follow-up oftraining content ; and sufficient and appropriate training materials at the training There was significant evidence that participants were only moderate l y satisfied with the training program offered by their agency indicating that training programs could be much improved The research also suggested that it i s important to co n sider the total e nv ironment when developing training p rogr ams a nd encourage effecti v follow-up efforts that assess whether workers implement what they have learned in the training on their jobs and the factors nfluencing this transfer. In another perspective Seaburg (1982) said the problem with child w e l fare t r ainin g is its current focus on process instead of systems issues that a r e much l arger than train i ng s u c h a s r aisi ng the level of education increasin g salaries assign i ng reasonable caseloads, developing structures to pre v ent burnout, and utilizing specialized training in service delivery practice Despite positi v e findings on t he benefits of training generalizability v f traini n g resul s to other populations is l i mited given various methodological issues with the t rain i ng e v alua t ion (Cr o n i n & Drake 1995 ; Chaffin et aL, 1994 ; Cheung t a!, 1991 ) Before delving into why training is effective or ineffective and whether it encourages transfer oftraining, it is important to understand how it is evaluated The next section examines the training evaluation literature in detail with part i cular emphasis on child welfare training evaluation

PAGE 38

2 2 Training Evaluation The training evaluation literature is divided first into t he theory or model l iterature that describes how evaluation should be constructed and second lite r ature that explores how those models or theories are applied As already d scussed it is clear th at t rainin g has po tential to have a positive effect on work performance Controversy arises regarding the model or theory upon which the evaluation is based which in turn raises debate about the information collected testing protocols, (i.e, for knowiedge skill, or attitu des) and evaluation target levels (e g., t rainee satisfaction versus communit y impact) In both bodies of lite rat u r e researchers vary widely in their preferred evaluation techniques Some advocate for knowledge testing throughout a training course while others call for transfer of training evaluations (s uch as the development of an action plan that is later checked to see if trainees im plemente d what they learned in training) The following s ection set s a context for training evaluation with particular attention paid to child welfare training e v aluation Almost thirty years ago Camp bell ( 1971) pronounced the training and development literature to be "voluminous nonempirical, nontheoretical, poorly written, and dull (p. 565) Unfo rtunately while the voluminous label could be contested it does not appear to have changed greatly in this time period Dionne (1996) took a his t orical look at evaluat i on of training activities, saying that the 24

PAGE 39

lack of consensus may be due to t he l ac k of a unify in g model and t heo ry for research T he literature as reviewed by D ionne has achieved c o nsensus reg ar ding thr ee factors 1 ) informat io n learn ed in tr ainin g may or ma y not be appl i ed ; 2) a favorable organizational climate and probably more complex interventions than training are nec e ssar y to ensure learning retention and trans fer of knowledge ; 3) research e r t rainers, and managers use d iffe rent standa rds and judge training activities against their own crit eria, thus the i nfo rma tion the y seek is not used for the sam e purposes Training e va l u ation is a component of an organizational subsystem t hat is i nflue nced b y its orga nizational context (Dionne, 1996) More attention is now given to child welfare training evaluation due to seve ra l factors including i ncreasing scrutiny b y media le gislators, c hild we lfare law s uits and worker certification requirements Training e v alu at ion can serve many purposes such as showing in re a sed emplo y ee knowledge and s k ills, achie v ing buy-in for agency vaiues and miss ion and providing feedback about training structure and content (Parry & Berdie 1999) Facto r s to c o n s i d er wh en designing a training e v aluation i n clude (Berd e & England, 1994, p. 5) Level of eva luat ion (e.g., satisfaction kno wledge skills) o Cogn itiv e doma i n (e.g., knowledge comprehens i on application) o Rigor oftest de ve lopment (e g., content vali dit y reliability bia s) 25

PAGE 40

Implementation (e.g test confidentiality, accuracy of scoring) Impact of evaluation (e.g., remedial work, curricula and trainer deficits) Planning for evaluation should start at the beginning of the training initiative and continue through each step in the training process (P arry & Berdie, 1999). Good evaluation is contingent upon a thorough well developed curriculum, clearly defined learning objectives and communication between the trainers and train ees about the training content (Jones Stevenson Leung, & Cheung 1995) Levels of evaluation were originally proposed by Kirkpatrick and soon became adopted as the standard model across disciplines for training evaluation (Campbell 197! ; Phillips, 1991 ) Kirkpatrick ( 1959) proposed a training evaluation schema w i th four level for evaiuating the quality oftraining: 1 Reaction-Assess panicipants' reactions to the traini ng Measurements tools include happiness surveys and verbal feedback 2 Learning-M easure knowledge and skills gained at the training Measurement tools include paper and penci l knowledge tests, essay exams, and simulations 3 Behavior-Measure how learning is applied to on the-job performance. Measurement tools include participant action plans record reviews observation and selfre port surveys 4 Results Measure results and /or outcomes from training. Measurement may use em pirical measures to determine organizational 26

PAGE 41

outcomes such as determining whe ther fewer children are in out-of home placemen t du e to training Forty ye ars later Kirkpatrick s model still remains as popular as ever with many uaini g evaluation texts based on it (for instance Basarab 1992 ; Holcomb 1994) This model has been the basis for most training evaluation designs and is widel y regarded and recommended in both the public and private sectors to evaluate training (Abernathy 1999; Allig er & Janak 1989; Birnbrauer 1987 ; Krein & W eld on 1994) Even when Kirkpatrick is not used as a reference many evalua t ion approaches inherentl y see m to adopt the levels approach For insta nce Schinke et al. ( 1981) presen ted a new technique for child welfare training evaluation to determine work hop impact emp i rically with tests before, during and after rr aining (i.e., leveltwo evaluation) Rae (1991) also d i scussed different levels of evaluation and while he did not arrange them in a specific typology his treatmen t bears a striking similarity to Kirkpatrick's model. In a testament to the convent i onality of Kirkpatrick s model when researc her s introduce new evaluation models they often first denigrate Kirkpatrick's model for various deficiencies as a way of justifying their own models existence (for instance see Holton 1996; Kaufman & Keller 1994). In Kirkpatrick s level one evaluation, trainees fill out surveys assessing their atisfaction with the training and trainers ; hence the term happiness survey has become part of the training evaluation lexicon Studies have shown 27

PAGE 42

ho w ever that there is little correlat i on between how trainees feel about a program a s evidenced by responses on participant reaction forms and what they have learned -or more importantly, what they will do on the job because of training ( Birnbrauer 1987 ; Broad 1997 ; Dixon 1 990 ; Russ-Eft & Zenger 1985) These sup ort Kirkpatrick s contention t hat a favorable reaction to a program does n ot ass ure le a rn ing" (Dixon 1 990, p 13 7) It should also be noted that trai n ee s o ften underestimate their pre-training skills a nd exaggerate their post trainin g perfo r mance because of a s t rong need to justify their attendance at rain in g ( Yancey & Kell y, 1 9 9 0). S ome researchers would d i spense wit h the react i on level (level-one ) altoge t her as an evaluation focus ( Holton 1996 ; Kaufman & Keller 19 9 4 ; Phillips, 1 99 1 ; Rus s -Eft & Zenger 1 9 85). Holton (1996 ) instead calls it an inter v ening variable to consider when evaluating training program results More rig o ro u s e v alua t ion i n e cessar y to enhance the training function s perce i ved credibility and value (Yance y & Kelly, 1990) I n the 1 9 80s Kirkpatrick (198 3) e ncouraged rigor in level-two evaluations by a dvocating for pretest posttest c ontrol group designs to assess knowledge Othe r researchers concerned about the integrity of evaluation research also called for a more rigorous approach warning against self-report of behavior change as it ma y be unreliable and invalid ( Campbell & Stanley 1980 ; Jones eta!., 1995 ; Parry & Berd i e 1999; Phillips 1 9 9 1 ; Russ-Eft & Zenger 1985) Campbell & 28

PAGE 43

Stanley (1980) said that educational improvements can only be verified through experimental design Though the directive to conduct rigorous evaluations is an admirable admonition this advice has largely gone unheeded by training evaluators due to cost and time factors (Jones et al, 1995) Despite the depth of knowledge that can be gained by utilizing all four levels of evaluation training is typically evaluated at level-one ofKirkpatrick' s model or the trainee satisfaction level (Birnbrauer, 1987; Brinkerhoff 1987; Russ Eft & Zen0er 1985 ; Saari eta!., 1988 ; Yancey & Kelly, 1990) A lack of rigorous evaluation beyond the satisfaction leve l also was found t o be the case in chi d welfare t ra i ning evaluation (Berdie & England, 1994 ; Jones et al, 1995). Evaluation at Kirkpatrick's levels thr ee and four is rarely used because the mul t iple i n fluences on results related to transfer and organizational effects make causal i nferences difficult to isolate (Kaufman & Keller 1994). Lack of rigorous evaluation is a tributed to attitudinal biases against training and a lack of commitment or re sourc es for designing valid and reliable evaluation instruments (Jones et al, 1995). Testing for retention ofknowledge poses significant challenges due to a variety of factors including cost and the confounding effects of intervening variables, such as experience and addit i onal supervision ( see C heung eta!., 1991). Testing for skills can be accomplished through the use of videotaping. Video-based testing is closer to real job performance and thus is a valid predictor 29

PAGE 44

of actual behavior on t he job (Wee kle y & Jones 1997). The use of simulations for evaluation has se veral advantages : reproducibility cost-effectiveness a nd safety consi de ra tion s ( Philli ps 1991 ), and t hey also can effectively evaluate traini n g (Holcomb, 1994) More recently, Kirkpatrick s model ha s undergone much criticism as being unrealis t ic and unwo rkab le (Allige r & Janak, 1989; Holton 1 996, Kaufman & K eller !994) Holton (1996) sa d th a t Kirkpatrick's four-level sys tem of training eva lua tion is more a taxonomy o f outcomes and is flawed as an evaluation model. A more appropriate model needs to specify outcomes correctly acco unt for int ervening variab le s and in dicate causal r elationships (Holton 1996) In another criticism ofKirkpatr ick's model Alliger and J a na k (1989) analyzed Kirkpatrick's t axonomy a n d fo und three assumptions The levels are ar ra nged in ascen ding value of info r mation A measure of im pact a t t he fourth level provides more information than that o f trainee r action 2. The levels of eva luation are causally linked a n d there is a domino effect, if you have pos i t ive react i on, then y ou get positive learning, then behavior change, an d so forth 3 The levels are positively i ntercorrelated Accor din g to Alliger & J anak (19 89) Kirkpatrick's model would be more appropriate as a global h eu rist ic for traini n g eval uation In fact Kirkpa t rick 30

PAGE 45

( 1996) claims this was the original intention of the model and said he is amb iva lent about whether his four levels of evaluation meets the criteria for a model. Rather i t is important that the model be used to help clarify the meaning of evaluation and offer guidelines on how to conduct an evaluation T he model h a s practical applicat ion rather than scholarly roots (Kirkpatrick 1996) Recently there has been a move to elevate the evaluation literature and to mo ve be yo nd Kirkpatrick and i nto more sophisticated evaluation designs A hos t of re earchers recommend other models of evaluation which encompass goal (Ame ric an Humane Assoc ia tion, 1996 ; Bri nker hoff, 1987 ; Holton 1996; Kaufman & Keller, 1994 Otto so n 1997, Russ Eft & Zenger, 1 985; Stokking 199 6) O ne of these goa l s i s beh avior change and it requires the following components : 1) persons must ha v e a desire to change; 2) they mus t have the necessary kn owledge and skills to try th e new behav ior; 3 ) they must work for a manager who allows or e nc ourages change ; 4 ) they must have help ; and 5) the y mu st be rewarded for change I t is necessa r y t o have a s y stemati c appraisal of the organizatio nal enviro nment befo re and after tra i nin g by the person receiving the training th eir superiors subordinates and peers (Kirkpatrick, 1983) I n contras t (a nd perhaps from the corporate sector) other models ad v ocate a return to simplicit y in order to spend more money on training and less on evaluation ( Abernathy, 1999) Some oft hese models, both simple and complex 3 1

PAGE 46

are explored in the next section to illustrate the breadth of evaluation model designs available in the literature 2 2 1 Traini ng Evaluation Models Stokking ( 1996) analyzed several popular evaluat i on models that prescribe how training should be evaluated and found them all to be lacking but a rgued that a dialo g ue about the utili t y of v arious evaluation mode l s would more likely lead to re s u lts In t h at spi rit, the fl llow ing section explores different train i ng e va luatio n models A sampling of the more popular evaluation models are presented in Table 2.1 to illustra t e differences in approaches and strengths and w eakn sses of each Phillips ( 1991) described seven evaluation models in his bo o k that are all v ariations of the same theme : a four -le vel approach wi t h an emphasis on calculating the value of t raining and the potential return on inves t me nt. The model deve l oped by Gordon (1987) also labeled by him as the "IBM approach ," ( designated as such as that is where it was first applied) was ch os en to r epresent all m o dels in th i s genre A lso represented is Kirkpatrick s classic evaluation model a model des ign e d for hum an services agencies ( American Humane Association 1 9 96 ), a model that i s les s a model and more of ta k l is t (Rae, 1991 ), and three m o dels t hat evolved from Kirkpatrick's model (Brinkerhoff, 1987 ; Holton 1 9 96 ; Kaufman & Keller 1994) Each model was assessed for the potential rigor of the evaluation efforts More rigorous 32

PAGE 47

eva l uations a re t h ose that a t l east ha v e th e potential for experimental design and c on t rol for v ar i ou s fa ctors that m i ght in fluence outcomes Table 2. 1 C ompar i son of Train i ng E v aluation Mode l s rlut hor Co mpon e n t s Most Focus S t r e ng t hs Wealmesses Pot enappro p riare rial organization Rigo r American I. Forma t ive Hu m an Huilt fror.1 All d i m ens i ons Diffic ult to Hig h Humane evaluation Services ds cf of tr a i n ing opera i o nali z e Assoc .. 2. Sati sfac ti on agency I learning. Levels imp a c t c an be higher level s. 1996 3. O p inion revolv e aro u n d capt u red. May be 4. Kno w ledge dis t ance f r om Levels o f overwhelming acquisition I training. lear nin g are to try to address 5. Knowledge ts on broken down all leve ls. Also, comprehension measurement of Ofte n neg l ected ma y not be 6 Sk ill these facto rs. form a ti ve eco n omically dem o nstrati o n I evaluation is feasible. 7. Skill transfe r included. 8. Agency impact 9. Client outco me s I 10. Commun i!! impact I Brinker1) evaluate Oriented F irst pa rt of Foc u ses on Lack of Mediu hoff. 1987 net!d s a n d goals: t o ward s m o del focuses areas with i n th e specificity in m 2 ) ev al uate b u siness on t he training tra inin g higher leveb o f t ra i n in g d e s ign ; itself a n d th e n des i g n er s evaluat i o n 3) evalua t e results. co ntr ol. sim p l e T rai n ee l ea rn i n g opera ti on; 4) t o i mpleme nt. is accre t ed. I ev.tlua t e learning: 5 ) I eva l ua t e u sage a n d en du rance of learning: and I I 6) evaluate I payoff Go r don. 1 Reaction Business Return o n Stresses value of Lack of Low 198 7 2. Tes t i ng organizati o n invest m ent tra i n in g and spec i fic ity i n ( 18 \ l 3. Applicat i on re turn on m o d el, leve l s a ppr oach) 4. B u s ines s I i n ves t me nt. a r e broad. Results I Holton. Pri m ary and An_ O utco mes from Mo d e l i s All o r no thing. Hig h 19 96 secondary organ i za t ion o rganiza t ion com p rehensive so m ay n o t be influence s ( b ecau se of its perspecti an d acco u n t s for eco n o m icall y Omcomes: c o m p l exity. conlell.1ua l most fac t ors. f eas i b l e t o 1. Learn i ng ma y focus. 2 Ind i vidu a l I via ble f o r I Leve l s too b r oad performa n ce i la rg e to s u ggest an 3. o r gan i zatio n s) imple m enta t i o n Organ i zational st r ategy. results I I I 3 3

PAGE 48

Table 2.1 (Co nt. ) Azahor Componenc s Mos1 I Focu., S1reng1hs Weaknesses I POlen/! appropna/e a/ RIJ::Or Kirkpatri K. 1 R eact i on Any I Outcomes fro m Model is s imple Levels are t oo 1959 2 Learning org. mi zation trainees p o int br o ad m 3 Behavior of a n y s i ze ofvie,v. Difficult to 4 Results operat i onalize fourth level. Higher levels are not appr o priate for 'soft skills' Rae. 1991 :-.lo l eveis. Any The eva lu ato r s Offe rs s tep b y Approache!> Low s equential orga mz ation tasks t o step evaluation a s pro c es s fro m of :my s i ze. I effectively instructi ons. o ne size fits to I zdenti fying a I eYaluate. a11. Kirk pat I I pr ob le m t o lon g I I flexibilitv in rick's evaluation. 3'd follow-up Pr escri ptive l eve l). J ( laundr y I ist of I i Evaluation models vary in their complexity from simple (for instance Rae 1991) to complex (for instance Holton 19 96) Their complexity is a function of the considerations the training evaluator must attend to when designing evaluations and the amount and type of information obtained when the evaluc. tio n is complete More complex models consider a wider degree of stakeholders and ass ess for the mega-effect of training courses Kaufman and Keller ( 1 994) said a more effective approach to evaluation should encompass an expanded concept of evaluation one that includes result related questions that contribute to continuous improvement by comparing intentions w ith results The evaluation design should also incorporate other interventions such as strategic plann ng organization and development (Kaufman & Keller 1994) The degree of complexity direc tly influences the potential rigor of data collection and analys i s from the traini ng e va luations More rigorous training 34

PAGE 49

evaluations assess the training programs at multiple levels with increasing complexity of training design s F or instance, the Holton (1996 ) model requires that intervening va r iabl es, s uch as ability, motivation and environment be measured as well as the mo re conventional yet fairl y sophistica te d measurements of trainee performance and organ izational results thu s were rated high for potential evaluation rigor While admirable in theory, the cost of conductin g such an evaluatio n may be prohibitive for many organizations AHA s (1996) model, in contrast encourages a compl ete evaluation but recognizes the difficulty of measuring outcomes such as community impact so it acknowledges that training eva l uation design must be flexible depending upon information needed and resour ces allowed AHA s model (see Figure 2 1 for the conceptual rendering of this mo d el) i nc o rpo rated Bloom s taxo nom y of educati ona l objectives (i e., cogni tive, affective and psychomotor) and significantly expanded Kirkpat r ick s levels of evaluation (American Humane Association 1996, Parry & Berdie 1999) 35

PAGE 50

Figure 2 1 A merican Humane Association's Model for Strategic Evaluation : CLIE;NT OUTCOMES -o t e Fro m A srrateg i c plan for tra i n ina e valua t ion in Co lorado ( Working P a per ) ( p 9) b y A meric an Human e Ass o ci a tion. E ng l ew o o d CO : A uthor C op y right 1997 by the American Hum a ne Assoc iat io n R e print e d wi th penni s s i o n R ae s ( 199 1 ) model also u s es a m enu approach but evaluation only goes so far as kill s t esti n g ov e r t i me I n 19 8 9 B r inkerhoff (19 8 9 ) augmented his model b y delineating thre e z ones within the t r aining process before during and after training that can p r ovide an opportunity for inte rv ent i ons t hat would increase training effectiveness (a lso ad \' ocated b y Cur r y eta!. 1994) 36

PAGE 51

The e valua tion models can also be categorized by the type of organization for which the model is most appropriate given its components AHA's model was specifically designed for human service agencies that are most concerned about the effectiveness oftraining and i ts impact. More business-oriented models, e.g., the "IBM approach," are simple to implement and focus on the bottom line ." Other models (for instance, Kaufman & Keller, 1 994; Kirkpatrick 1959) are appropriate for any type of organization, whether public or private and of a ny SIZe. Selecting the most appropriate evaluation model is contingent upon the organization s capac.ty to implement the training evaluation and information needed to make decisions Unfortunately, few organizations deliberately choose an evaluat ion model but instead may succumb to tradition and familiarity. Purpo eful e valuation, especially to assess transfer of training is not a pre domin ant feature of current evaluation practice CDHS, however has bucked this trend and has committed resources to ensuring that training do es indeed work They have adopted the AHA model to guide training de vel opment implementat ion and evaluation. The next sect" on reviews the literature related to transfer of training also known as Kaufman and Keller and Kirkpatrick's level-three AHA's level-seven and so forth. 37

PAGE 52

2.3 Transfer of Learning Most ofthe iiterature reg ardin g factors th at contribute to the transfer of learning or traini ng comes from the business, psychology or education fields Bro a d and ewstrom (1992) offered this defin i tion of tramfer (or 1 I .ea rnmg) [T]he effect iv e and continuing application by trainees to their jobs, of the knowledge a nd ski lls gained in training-bo t h on and offthejob (p 6). Kozlowski said ; "Transfer is the core issue with respect to linkin g i ndividua l chang e to the r equ ire ments of the organ i z a t i o al syst em (1997, p 255). Hum an services, a 1 d more spec i ficall y the field of child welfare has borrowed the transfer of training literature from these other fields in an effort to increase the effectiveness oftraining. Transfe r of learning has a long research history Much of the early transfer literature relates to its cognitive processes Schema theory is often used as t he theoret ical ramework for u n der standing transfer; it says that transfe r depends on the ac tiva t io n of re l e v ant anticipatory schemata (Macaulay & Cree, 1999 ; Reed, 1993; Salomon & Perkins 1989) In 1903 Thorndyke discussed the "theor y of identical elements ," sta ting th a t information about ho w to do one task i The tenns trans fer of learning and transfer of trainin g are equivalent a nd both are used throughout the thesis 38

PAGE 53

can be applie d to another (Clark & Voogel, 1985; Detterman & St ernber g 1993; Gick & Holyoak 1987) The Bruce-Wylie laws were developed in the early part of this century and ass en 1) transfer depends on the degree of similarity between two situations ; and 2) the direction ofthe transfer depends on the similarity ofthe two responses (Gick & Holyoak, 1987, p. 18). Four factors determine transfer performance (Gick & Holyoak 1987, p. 20) : 1 The structural relation s hip of the training task to the transfer task. 2 The conditions at th e time ofthe learning experience (encoding) for fostering the learning of the material (e g., classroom experience trainee characteristics) 3. Conditions when the transfer task is performed. 4 The backg r ound knowledge ofthe subject It should be noted that the first factor is concerned with transfer's cognitive process the second wi th the training environment, the third with the organizational environment and the fourth with tra inee character ist ics Detailed theories on cognitive models of perfo r m an ce ha ve been developed by researchers to explain the phenomenon oftransfer (Ellis, 1965) From a cognitive perspective the basic rule of transfer is that the original encoding of o n e task can be applied to the performance of another task Rule based models of knowledge repres entation make it possible to see how what is 39

PAGE 54

learned will influence transfer (G ick & Holyoak 1987) For instance ifl see something that fits in the palm of my hand, is red and shiny and has a small dark stem emerging from its core, it's pr obab l y an apple Positi v e transfe r m eans performance on one task aids or facilitates performance on a secon d task. Negative transfer is whe n the performan ce on one task in hibits or disrupts performance on a second t a sk Transfer is positive when later acquisi tion or performance is facilitated and negative when later acquisition or per ormance is hamp ered (Ellis, 1965 ; Cormier & Hagman 1 987). Zero transfer occurs when there is no effect of one task o n another (Ellis, 1965) Task similarity is necessary in order for transfe r to occur an d t he g r eater the similarity, the greater the positive transfer (s ee facto r 1 above). Other factors influencing tra nsfer are : the time interval b e twee n tasks ; degree of original-task learning ; variety of previous tasks; and task difficulty Prior experience with a task to be learned is called stimulus pre d iffer entiation (Ellis, 1 9 65). The transfer continuum "anchors" are lab eled near and far with the de sire d goal of far transfer to generalize training content outside the conte x t of the t ra ining environment so that it is decontextualized" ( Clark & Voogel 1985) S i milarly Salomon and Perkin s (1989) dist ing uished bet we en high and low road transfer with the former b e ing more abst ract. Low road transfer is the result of extensive varied practice and occurs by th e automatic triggering of well le arned behavior in a new context. Hi gh road t ra nsfer occurs by abstracting from one 40

PAGE 55

context and application to a new con te x t Instructional methods should be matched to the type of transfer desired (Clark & V oogel 1 985). Ellis (196 5) said much early research failed to consider the factors that influenced t raining. Most r ec ently researchers say that positive transfe r of training is dependen t on a host o f factors. Some relate these factors to a timeline specifically factors and even ts prio r to training, those at training and another set of facrors and events that occur after training (Beudin, 1 9 8 7 ; Cannon -Bowers, Rhoden izer, Salas & Bowers 1998; Curry eta!, 1994; Holcomb 1994 ; John son & Johnson 199 8 ; Newstrom 1985; Ohio Child Welfare Training Program Evaluation Subcommittee 1993 ; Rae 1991 ). Anoth er perspective sa ys transfer of training is dependent on three groups of key players-trainees, tr a i n ers, and managers-and thus transfer interventions are directed at one or all of the se gro ups ( Holcomb 1994 ; Marx 1986 ; Rae, 199 1 ). Other s say that t ransfer is contingen t upon a combination of strategies that account for the ti m e line and are divided by t he person who has responsibility for the intervention strategy ( Ba ldwi n & Ford 1988 ; Baldw i n & Magjuka 1991 ; Beudin, 1987 ; Broad & Newst rom, 1992; N ews tro m 19 85) Huczynski and Lewis ( 1980) l ik ene d the transfe r process to the links i n a chain : "For transfer to occur all the links [i e., trainee t r aine r supervisor and organization] must hold w hen the chain is pulled. Should e v en one of them fail the whole chain will b re ak" (p. 240) 41

PAGE 56

Other researchers focus on solving the transfer problem through development of integrative models that consider multiple factors Garavaglia ( 1996 ) developed a model to incorporate all the factors to describe a sequential process for increasing transfer potential at every step Transfer is contingent upon a systematic process for proactively stimulating it rather than a function of context ( Garavaglia, 1996). Broad's (1997) modei is similar in that it recognizes t he i ntegration of strategies to facilitate transfer but emphasizes stakeholder involvement as the catalyst at every pha se of the training development process rather than t he strategies themselves Because of the complex nature of transfer of training the transfer evalua tion must be planned and carefi.illy implemented The nex t section e xa m i nes the literature related to transfer evaluations 2 3 .1 Condu cting Tr ansfer Evaluations A transfer of training e val uation should be conducted to demonstrate the value of training to an o rganization justify tra i ning costs and determine the effect" ven ess oftraining curriculum (Gar avagl i a 1993 ) Ottoson (1997) however, disagrees wi th the whole notion of a transfer evaluation She says that other effects of the education program that are not specific to those learned in train i ng are lost, because they are not sought after So while the transfer evalu a tion may look for reten tion of predefined knowledge and s kills it ma y miss 42

PAGE 57

other effects from the training program. I t is misguided to expect that transfer will occur since i t assumes that traini n g participants return to contexts that can support or accommodate new ski lls A l so a transfer evaluation ma y lose information, because train i ng participants ma y be usi n g t he inform at ion learne d in t raining in a d iffe rent manner th an that which the transfer evaluation is designed to capture T h is argument assumes that the trainee is willing and eager to appl y the new skills, but it does not c o nsider the organizational contex t or e n v ironm ent in which the new skills are being a p pli ed Basarab ( 1992) suggested performing transfer e alua tion s on l y when the tra ining p rogr am is cri tical to the success of the org an iz ation When a transfer evaluafon i s cond ucted it should : 1 Use a pre / post tes t design (Basarab 199 2 ; Sullivan & Clanc y 1990) 2 Question key p la yers who c an in o r m whet her transfer occurred (i. e., partic ip ant supervisor participant subor dinates and participant peers) because self-report data can be bi a ed or contaminated (Basarab 1992) 3. Organize data to dete rmi n e skills that have and have not been transferred (Basarab 1992) 4 Measure transfer after s ome period of time has elapsed after training ; there is a fine line to knowing exactly when to measure so it is accurate (Basarab 1992) 5 Emp loy a control group ( Bas a rab 1992; Garavaglia 1993) 43

PAGE 58

Clark and Voogel (1985) said that transfer evaluation should determine not only whether transfer has been achieved but also the level and amount. Tra nsfer evaluations are challenging to implement, in part because transfer is difficult to measure due to intervening var iables such as supervisory support (Holton, 1996) Recommended data collection ins truments for transfer evaluations include : q u est ionna ires participant action plans work logs observation accounting records interviews focus groups supervisor reports and performance appraisals (Basarab 199 2 ; Garavaglia 1993; Delewski et al, 1986) The development ofvalid and reliable transfer instruments, however, is still in its infancy (Holton 1996). On instrument for measuring t ransfer is the "Part icipant Action Plan Approach" (PAP A), developed by the US. Office of Personnel Management in 1980 to assist training evaluators in determining whether participants changed their j ob behavior a the result of training ( Dele wski Pecora, Smith & Smith, 1986). Trainees were a ked to set goals ( i e an action plan) for how they would implement w ha t they learned in training. Several months after the train ing evaluators contact trainees to determine if t hey achieved their goals This was determined to be an effective method of operationalizing w ha t was learned in training since almost all of the trainees had tried to implement at least one of their action items (Delewski et al., 1986). The PAP A method empowered trainees to take control of their own learn ing and thereby to generate a higher probability of 44

PAGE 59

actual implementation. This method though, relied on selfreports, which may have re duced the evaluation s v alidity Holton (1996) developed another comprehensive transfer instrument he called upon other researchers to validate noting that this was a daunting task given its scope unv ersally, the transfer li erature states that transfer of training is complex and is dependent upon multipie factors Not only is it in complex in na ture, but it also is notoriously difficult to measure An assessment of pure transfer would require extensive documentation and observation of before and after ra ning-related behavior Therefore some researchers have responded to the measurement challenge by operationalizing transfer in ways that represent tran fer rather than observation of actual transfer. One measure of transfer is retention of training content as a pre condition for the transfer of learning (B aldwin & Ford, 1988 ; Curry 1 997; Ellis 1965) It is assumed that training content must be remembered before it can be applied The next section introduces the model that captures the factors affecting the transfer of learning It is the primary model upon which this dissertation is based and guides the various components of the research design 2 3.2 The Baldwin and Ford Model of Transfer of Learning The most complete and ye t parsimonious model of transfer of learning was developed by Baldwin and Ford in 1988 They analyzed the literature related 45

PAGE 60

t o transfer of learning and developed the model in Figure 2 2 to understand the dynamics of transfer. Their transfer model incorporates the key players as Wt'll as time perspective to predict transfer. 46

PAGE 61

Figure 2.2 Ba l dw i n and Ford' s M odel for Tr ansfer of Tr a ini n g Training Inputs Train e. O t a r ac teristics Ability
PAGE 62

self-report if it is measured at all. Of course, self-r eport is not a reliable measure of transfer, so criterion measures are necessary for determining which interventions are more likely to lead to transfer (Baldwin & Ford, 1988) Baldwi n and Ford's model has been adopted by many researchers to help understand transfer of lea rni ng and develop intervention strategies to encourage transfer (Broad & ewstrom, 1 99 2 ; Curry, 1997 ; Johnson & Johnson 1998 ; Lim, 1999) Broad and T ewstrom ( 1992) used the Baldwin and Ford model to develop strategies called Relapse Prevention Training for increasing transfer. Relapse pr venti n strategies emphasize the importan ce of the interaction between trainee, training and organizational environment characteris tics Lim (1999) added a cultural component to make the model relevant for international training Other re earchers have de v eloped similar theories for understanding transfer For inst a nce Parry s S model (1990) delineated three sets offactors that affect transfer personal factors ; instructional factors; and o rganizational factors Using the Baldwin and Ford model as the framework, the transf er literature was reviewed to determine research that s up ported contentions within each domain of the model: trainee characteristics training des i gn, and wor (organization) environment It should be noted that some studies researched multiple influences (e.g., trainee characteristics and training design ) at the same time so some research overlaps Each domain is examined in detail below. 48

PAGE 63

2.3 .2.1 Train ee Characteristics Some researchers focus on transfer as predicted by characteristics that t r ainees p os sess. One such trait i s the trainee s general abil i ty (Burke, 1997 ; Clark & Voogel, 1985 ; Marx, 19 86). Relapse Prevention Training which involves informat ion and strategie s giv en to trainees during the training ses s i o n to help them retr ieve o r transfer the tra i ning content back at the work setting ( Broad & Newstrom 1992 ; Burke 1997; Gist e t al., 1991 ; Marx 1986 ; News trom 1 997) can enhance general abi i ty. Inherently it places the onus of responsibility on trainees but helps them b y providing techniques and strategie s to prevent relapse to pre-training beha v iors (Burke, 1997 ; Burke & Baldwin 1999) 1 ews trom said trainees should be res ponsible for their own application of inform a t ion learned in training ( 1985) Similar to the character trait of geneml abili ty, Dav ey and Hill (1995 ) said that v ariation in their study was accounted for b y t he respo ndent s professional bac kg round followed by the experience of worke rs. Trainees wi th grea t e r metacognitive skills (i e., the knowledge of and c ontrol over one's cognitio ns ) a l so were found to have better outcomes (Ford, We i s sb ein, Smith Gully & Salas 199 8) as were trainees who are more confident ( Gist et al., 1991) Another trainee characteristic influ e ncing transfer is whether trainees are masteryor performance oriented Mastery -orie nted trainees are defined as t rainees who are focused on enhan cing task competence are ego-involved a nd 49

PAGE 64

have an external locus of control. Performance-oriented trainees are focused on demonstrating competence to oneself or others have an internal locus of control and see feedback as constructive In stud ies analyzing the performance differences between these two group mastery-oriented employees engaged in more interim skill -m aintenance activities planned to exert more effort, and showed more positive affect than did performance -ori ented employees (Fisher & Ford, 1998; Ford et al., 1998 ; Stevens & Gist, 1997) Moti v ation is another factor affecting transfer If trainees are motivated they will rememb er and apply c ou rse content (Burke, 1997 ; Hicks & Klimoski 1 987). Motivation to transfer along with self-efficacy were found by Machin and Fogarty ( 1997 ) to be the main influences for the development of transfer i ntention s According to Facteau et al. (1995), pre-training motivation is more than a trainee character is tic, but a funct.ion of trainee personality var iables such as i ntrinsic and extrinsic incentives as well as perceived environmental support In one study voluntary attendance at the training improved performance (Hicks & Klimoski 1987). Baldwin and Magjuka ( 1997) disagreed that voluntary attendance resulted in better transfer and instead said that voluntary training may convey a signal of relative unimportance to trainees To a lesse r extent than other tranee characteristics prior information also had an effect on transfer (Hicks & Klimoski 1987) This suggests that training outcomes and the overall success of 50

PAGE 65

training interventions are con tingent upon understanding the point of view of trainees (Hicks & Klimo ski 1 98 7) In the Miles ( 1965) and Xiao ( 1996) studies trainee characteristics were found to have no effect on transfer. Personality varia bles (i.e. ego strength flexibility and need affiliation ) did no affect o u tcom es (Miles, 1965). However learning i n training as perceived by t rainees did contribute to better performance and transfer in the workpl ace (Xiao, 1996). To summarize, stud i es on trainee characteristics sug g est that trainer s should focus on trainees intrinsic motivation bui l d on employee strengths that encourage tran sfer and employ st r ategies to compens ate fo r t rainee weaknesses. The next s ection discusses th e training design domain in th e Baldwin and Ford model. 2.3.2 2 Training De sign Many researchers focus on t he relatio n ship betwee n transfer a nd training design factors re la t e d to the i nstmctor, curriculum and training techniques This body of l i terature has its roots in the early resear ch conducted b y Thorndyke and others who were most concern d about t he equivalency of training conten t and job tasks (recall the theory of identical elements discussed on page 37) This perspective sugg e sts that transfer often fails due to inappropriate training d es ign mo dels that hin de r t he application of training concepts beyond t he 51

PAGE 66

classroom. To be effective t raining must be relevant to the worker's job (Clark & Voogel 1985; Garavaglia, 1993) Training can be made more relevant by conducting needs assessments to ensure that training content is reflective of worker s t r aining needs and is based on the reality of job tasks and necessary performance (Darou 1990; Pecora et al, 1983 ; Rae 1991; Taylor 1992). Training needs assessments attempt to match training content to job tasks by conducting a thorough job analysis that includes distributing staff questionnaires, engaging in structured int erviews of fro nt-line supervisors about their subordinates and defining a critical incident method to ensure tha t training is mo r e r elevan t ( Darou, 1990). Implicitly training methods chosen during t raining affect transfer ( B aumgartel & Jeanapierre, 1972 ; Gist et al, 1991; Rae, 1991; Vinok.'Ur-Kaplan 1986). Transfer is successful when levels of transfer, types of learnin g objectives and t u dent aptitude levels are matched to ins tructional methods (Clark & Voogel, 1985) Cognitive m o deling is one such t raining method to improve trainin g retention ; it trains students to attend to t heir thoughts as they perform an activity and t h en uses self-instructional thoughts to guide performance on the training task ( Gist, !98 9 ) Also important are the quality of the tra i ner and the use of appropriate materials to which workers can refer dur ing and after training (Vinokur-Kaplan 1986) Yorks O'Neil, Marsick Lamm, Kolodny & Nilson ( 1998) recommend a curriculum based on the principles of Action Reflection 52

PAGE 67

Learning a program that int e g r ates real-world" challenges Ford et al. ( 1 9 98) rec o mmende d i ncorporating curriculum cont e nt related t o building self-efficacy and other motivation a l outcomes to f acilitate transfer rather than relying on trainees i n t rinsic abilit y and mo t ivation which may or may not be sufficient. Oth e r re earcher s s ay t ransfer is dependen t on how train e es a r e ta ught to r et riev e the infor m a t ion when back at the job not wheth e r training matches job t ask Re l a p se pre v ention techniques are relevan to two domain s of the B a ldwin & Ford model a s u cc essful re l ap s e p r e vention training depends on what is taught at t h e t raining and the trainees char a c t eristi c s U sing the language oflearning re e archer s retrie va l at t he tra n sfer s t a ge depends on the encoding done at traini g. I nterventi o n strategies t aught dur i ng the training are the most cr i t i cal as p ec t of tran s f e r for thi s grou of r se arc h er (Broad & New strom 1 9 9 2 ; Bur ke, 1997 ; C a n no n -Bo we rs e t al., 1 99 8 ; Gis t et al., 1991; Marx 1986 ; Newstrom 1 99 7 ; Ph ye & Sanders 1992 ) Man y of these researchers are proponents of r elaps e pr e v ention t raining dis cussed earlier in the trainee characteristics section (Bro ad & N e w strom 1 992 ; Burke 1 997; Burke & Baldwin 1 99 9 ; Gist et al, 1991; Marx, 1 9 86; e wst rom 1 997) I n a v ariation o f relapse pre v ention t r aining Foxon (19 9 7 ) posited that action planning will lead to greater transfer but a n emp i rical investigation ofthis technique resulted in ambiguous results Oli v e o B ane a n d Kopelman (1997) r e commend one-to-one executive coaching o n ce tr a i n e e s re t urn to the job to mo re exp e dientl y facilitate transfer. 53

PAGE 68

Curry (1997) found the most salient factors leading to transfer were t raining reievance trainer use of adult learning strategies transfer strategies, and perce i ved learning To a lesser exten but also relevant were : planning for applica t ion pre-tra i ning preparation post-training participant motivation time a nd caseload demands and co n gruence of the training content with the transfer environment. To summarize these training design studies suggest that efforts for tmpro ing transfer s hould focus on the relevancy oftraining, trainers proficiency curriculum and training techniques 2 3 .2 3 Work Environment Characteristics Studies of the work or organizational environment characteristics and their effect on training are predicated on the beliefthat training develops potential capacity in trainees while organizational en v ironment factors are key to faci i t ating the transfer of knowledge sk i lls and attitudes learned in training (Kozlowski 1997 ; Xiao 1996) In 1 953 Fleishman conducted a study on attitudes taught in training and the extent to which they were continued back at the job Almost fifty y ars ago he astutely noted "It is difficult to produce in an individual a behav i or-change that v i olates the culture in which this behavior is imbedded (Fleishman 1953 p. 221 ) Similarly but forty-five years later 54

PAGE 69

Baldwin and Magjuka said: Organizational contextual factors can easily overwhelm the effects ofthe best planned and delivered training'' (1997 p 122) Attention to the work environment may hold the greatest potential for ensuring transfer of training, but it also receives the least amount of attention (Baldwin & Ford 1988 ; Beudin, 1987 ; Kozlowski 1997) Transfer is affected by work context because it facilitates or inhibits an individual's opportunity to perform tasks or skills trained (Ford Quinones, Sego & Sorra 1992). The work environment s complexity is compounded by the difficulty of predicting how employees will attach meaning to management behaviors and attitudes Also training is never conducted in a pristine e nvi ro nment since ou tside influences affect the training situation (Baldwin & Magjuka 1997) Researchers do not use a uniform definition ofthe organizational env i ron ment ; rather, the variables that cannot be attributed to either trainee characteristics r training des ign are categorized into a generic catchall called the "work environment ." Baldwin & Ford (1988) created two categories for the work environment ; supervisory support and favorable organizational climate The org anizational climate va riable is defined more broadly and includes : supportive management style, peer support and the opportunity to use information learned in training on the job. On the other hand the supervisory support va riable is more narrow l y defined as the behaviors and conditions assoc iated with the supervisor (or manager) such as the support the supervisor gives the employee to attend 55

PAGE 70

training to apply what is learned in training and his or her efforts to reinforce new behaviors. The work environment literature related to transfer can be divided into studies focusing on one or both of these var i ables It should be noted that research o ften overlaps, for researchers may study several aspects of the organ i zational environment at one tim e as well as factors related to trainee characteristics and training design 2 3 2 3 1 Su perv isory Supp ort Many studies focus on the different dimensions ofthe supervisor s transfer-encouraging role within the organizational environment (Baldwin & Magjuka 1 991; Brinkerhoff & Montesino 1995 ; Broad & Newstrom 1992 ; Curr y 199 7 ; Gregoire Propp & Poertner 1998; Holton, Bates, Seller & Ca rv alho 1997 ; Huczynski & Lewis 1980 ; Ohio Child Welfare Training P r ogram Evaluation Subcommittee 1993 ; Taylor 1992; Warren Woodall Nunno Keeney, Larson & Stadfeld 1998 ; Xiao 1996) According to some research t he supervisor is perhaps the most important v ariable within the work e n vironme n t even more significant t han top-management support (Taylor 1992 ; Xiao 1996 ; ) For others it is no less important than other variables but remains an essential component for transfer to occur (Facteau et al, 1995). Supervisors influence t ransfer in different ways Supervisors have the most influence for improving transfer by providing reinforcement of training 56

PAGE 71

concepts (Broad & Newstrom, 1992 ; Phillips, 1991 ). Supponive supervisors can help stimulate transfer through the i r behaviors and att itudes (Broad & Newstrom, 1992 ; Gregoire, et a!., 1998) Management support includes both tangible and percei v ed support and can be s i gnificantly an d positively influenced by the imm diate supervisor before and after traini n g (Brinkerhoff & Montesino, 1995 ; Foxon, 1997; Holton, e ta!, 1997 ; Phillips, 1991). The extent ofa t rainee's accountability to his o r her supervisor also affects transfer (Baldwin & Magjuka, 199 1). I n Buczynski and Lewis s 1980 study a consultative management style rather than an autocratic management style was found to influence transfer mos t posit iv ely Transfer improved when trainees discussed the co tent before the course with their immediate supervisor (Huczynski & Lewis, 1980) Supervisors may ofte n fail i n their efforts to provide organiza tiona l support however, by not spendin g enough time in thei r role as mentor (Zunz 1995) or by not attending to other supporti v e behaviors such as one-to one coachin g which have been shown to increase transfer (Gregoire eta!, 1998) 2 3 2 3 2 Or ganiz ational Cli ma te Many researchers have concluded that the organizational climate in which trai ees work influence the degree of positive transfer (Baldwin & Magjuka, 1997 ; Baumgartel & Jeanapierre 1972 ; Beudin, 19 87 ; Burke & Baldwin 1999; 57

PAGE 72

Carnevale et al., 1990; Curry, 1997; Dionne, 1996 ; Facteau et al., 1995; F leis h man 1953 ; Ford et al., 1992 ; Hand Richards & Slocum 1973 ; Holton et al., 1997 ; Huczynski & Lewis 1980; Kozlowski 1997 ; Lim, 1999 ; Miles 1965; Newstrom 1985 ; Phillips 1991; Tesluk Tesluk Farr Mathieu & Vance, 1995 ; Vinokur-Kaplan, 1986; Warren et al., 1998 ) Factors falling in the organizational climate d o main are more div erse and ex tend beyond the supervisor's sphere of influence within the organizat ion. The organizational climate includes supervisor, administrative, organ i zational and co worker support (Curry, 1997; Holton l 997). Behaviors within the organizational climate send a message that learning is or i s not valued and thus influence w hether newly learned behaviors are applied ( Tra cey Ta nnenbaum & Kavanagh_ l 995) D iff erent tra nsfer climates may exist simultaneously in one organization (Newstr om 1997 Tesluk et al, 1995) T rainers th ought the most significant barrier to transfer was the lack of on the job re info rcement designed to support trainees in applying training to their jobs (Bro ad & ewstrom, 1992). Other organizational factors acting as barriers to tran s fer were : interference from the immediate environment, non-supportive organ i zati onal climate and negative pee r pressure (Broad & Newstrom 1992; Newstrom 1985) Conversely, organizational factors that i ncreased the likelihood of positive transfe r were sensitivity of higher management entrepreneurial attitude of organization, expectations of higher management to use 58

PAGE 73

train ing, i nteraction within the work environ ment and compatibility of goals ofthe organization and t r a i ning (Baumgartel & Jeanapierre 1972 ; Beudin, 1987; Dionne 1996; Garavaglia, 1993) In s tar k contrast (and contrary to what was hypothesized), Tesluk et al. ( 199 5), found that a participa t: ve climate for decision making was negatively related to adoption of employee involvement training Post hoc the researchers speculated that employees would have more opportunity to transfer since the horizon for empl o yee involvement was so much greater in a non -particip ative decision making environments (Tesluk et al., 1995) More effective training occurred when the organization integrated and focused on t raini ng and fostered a cli mate conducive to training (Carneva l e et al., 1990;! 1ewstrom 19 5) The effectivenes of training techniquessuch as r l apse p re vention strategies are moderated by the nature of the organizational climate for encouraging transfer Transfer climate should be assessed during the training needs analys is, and then the work environment should be manipulated to support transfer. Positive transfer can be improved if o rganizationa l members receive training or other assistance to create supportive transfer climates (Rouiller & Goldstein, 1993 ; Tracey et al., 1995) Bennett Lehman and Forst (1999) take a broader v iew of transfer clima te, stating that it is one component of the contextual f actors along with struc tural factors and change climate that influence the de g ree 59

PAGE 74

of transfer Interventions th e n are aimed at all of these ar e as t o increase transfer po t ent ial within organizations (Bennett et al, 1999) An other vein of research loo ed at the oppo rt uni t y to perfo r m w h at was learned in the classroom to the job (Ford et al, 1992 ; Holton et al, 1997 ; Newstrom, 1997). 1 ew jobrelated behaviors can appear only iftrainees are given the opportunity to perform their new ly acquired skills (Newstrom, 1997). Th i s var iable is placed within the organiza t io nal climate domain since it is continge t upon man y factor s within t he o rg anization not just the supervisor. Training must be relevant to the trainee s job and once training is completed forces within the organization must encourage and provid e opport u n i ties for the emplo y ee t o practice what was learned in training. Breadth activity level, and type of tasks performed should all be e x amined as three d i mensions of the opportunity to perform (Ford e t al, 1 992) When the work environment is believed to the pr i mary factor for increasing transfer of training then interventions woul d focus on changing the organizational con e x to encourage tran sfer of training Little evidence of organizational environment interventions, however, has been documented in the literature Baldwin and Ford (J 988) fou n d seven st ud ies related t o th e i nflue nc e of the or g an i zational environment. But t h ey c r iticize d these studies as not being experimentally rigorous enough by actually applying i nterventions to change the organizational environment, th u s t hey were unable to st ud y t he effect o f 60

PAGE 75

organizational differences. Brinkerhoff and Montesino (1995) respo nded to Baldwin and Ford's gauntlet and conducted a study of an organiza t ional environment intervention to determine the effect on transfer. While he inte r vention was admirable, the study contained methodological flaws. Because the i ntervention was good practice, some subjects in the control group inadvertently provided the same intervention as the experimental group Olivero, Bane and Kopelman (1997) conducted a study to provide coaching by fellow participant to help improve transfer but it too had methodological problems Given the importance of empirically testing for organizational climate it is cur i ous that so little research has been conducted in this area. A lack of theory has prevented application of organizational context factors affecting the implementation and transfer of training (Kozlowski 1997). Koz low ki ( 1997) suggested remedying this theoretical dearth by applying systems oriented theories of organizational behavior to help draw inferences regarding the effects of contextual factors on behavior found in the work environment and in tra i ning transfer situations Such a relationship has been theorized for many years Simon (1945) said employee decisions and behaviors are influenced within and by the organization In 1954 Drucker noted the existence of a symbiotic relationship between group relationships and tasks : 'Group relationships influence the task; the task in turn includes personal relation hips with the group Group and individuals must therefore be brought 61

PAGE 76

into harmony in the organizatio n of work (p. 265) Exploration of the transfer literature eads one to quest i on why and how environmental factors influence employee performance. The answer to this lies, in part in the organization's culture T he next section explores this relevant literature. 2.4 Organizational Cult ure Organizational culture dictates the e mployee's work environment and t hus a profound effect on whether positive transfer occurs. T he notion of organizat ions hav ing cuiture was borrowed from the fields of sociology and anthropology Organizational culture was studied because of a need to explain var ia bility in patterns of group and organizational behavior and a need to understand levels of stability in order to answer the question of why some organ i zations did well and others did not (Schein 1990) Organ i zations are social sys t ems with pasts, presents, and futures (Pettigrew 1979). Scholars agree that there is no consensus definition of culture (Gordon & DiTomaso 1992 ; Schein 1990 ; Vestal et al., 1997) For the purpose of this research culture is defined as [A] patterned system of perceptions, meanings and beliefs about the organization which facilitates sense-making amongst a group of people sharing common experiences and guides individual behaviour at work. (Bloor & Dawson 1994 p. 276). 62

PAGE 77

Org nizational c limate is diffe rentiate d from the concept of organizational culture and is a surface manifestation of culture (Schein, 1990) Culture is embedded in the schemas that help people make sense of organizational life Schemas are cognitive maps that use pa t experiences and understandings to make predictions about events so that individuals can select appropriate responses to anticipated events (Bloor & Dawson, 1994). Culture acts as a filter for interpreting activ i ties and events Many researchers use the t r ait approach to understanding the content of culture (Gordon & DiTomaso, 1992 ). According to Vestal Fralicx and Spreier (1997) here are 56 attributes (or traits) that organizations may posses s that can then be catego r ized into four dominant organizational cultures: functional, process ; time-based; and network. These attributes range from the organization's beha vio rs ar ound adaptation to those about authority and chain of command. From these attributes cultural profiles are d veioped that differentiat organizations Peters and Waterman (1982) outline d eight characteristics of excellent or well pe r forming cultures: 1) a bias for action ; 2) close to the customer ; 3 ) autonomy and entrepreneurship ; 4) productivity through people ; 5) hand -on v alue-dri ven; 6) stick to the knitting (staying close to the business they know) ; 7) simp l e form lean staff; 8) simultaneo us loose-tight propertie s (qualities ofbot h decentralization and centralization when appropriate) 63

PAGE 78

Organizations that have internal cultures supportive oft heir strategies are more likely to be successful (Gordon & DiTomaso 1992; Lahiry 1994 ; Smircich, 1983 ; Vestal, 1997). Cultural values can, and perhaps should, vary significantly across organ izations (Gol den 1992; Schein 1990; Sheridan 1992 ; Vestal 1997 ; Bro wn & Starkey, 1994). In fact, there may be great divergence between a collective cultural identity and sub-cultures within the work environment resulting in tension and dissension (Brown & Starkey 1 994). But individuals within the organization are never culture-less. If they reject the dominant organizational culture, they would then adher e to a sub -culture's val ues and cultural guidelines (Golden, 1992) A weak organizational culture will give rise to more sub-cultures and vice versa (Golden, 1992) Organizational effectiveness can be improved by attending to the orga .izational c u lture (Dunn o burn & Birley 1994; Sheridan 1992). Members of an organization exert press ure on each other to conform to shared codes (Lahiry, 1994). Organizational cultures emphasizing interpersonal relationsh ip values tend to retain workers longer than organizations who have work task values (Sheridan 1992) Organizations with strong cultures, as defined by con istency of perceptions of company val ues tend to have better performance (Dunn eta!, 1994; Gordon & DiTomaso, 199 2) There is, however, some disagreement about whether culture can be manipulated. Meek (1988) says culture is something that an organization "is," not 64

PAGE 79

something that an organization "has ," therefore it is not an independent variable (Meek 1988, p 469). Culture is not s tatic and while it is shaped by an organization's histo r ical conte x t it changes gradually but continuously to reflect the organ i zational environment and sub-cultures operating within it (Bloor & Dawsvn 1994) In another perspective on t he influence of culture, Golden (1992) state d that too much emphasis on organizational c ulture may minimize the actions of individuals who may act by challenging cultural guidelines By understanding an organization's culture and its effect on organizational memb ers one can start to ma ke sense of the processes and outcomes thus infl uencin g the m to war d a desired end (Brown & S t arke y 1994 ; Carnevale et a!., 1990 ; Dunn et al., 1994). Smi rcich ( 1 9 83) po in ts out the value of studying culture as an effort to i mpro v e management: A cultural mode of anal y sis encourages us to recognize that both the practice of organizational inquiry and the practice of corporate management are cultural forms products of a particular sociohi sto rical context and embodying particular value commitments In our pre se n t day these values are efficiency orderliness and even organization itself (p 355). Since cultur e con ditions attitudes about all types of organizational activities and events opp ortunit ies for i mpro ving transfer of training may lie in researching the work environment and its organizational culture with respect to training A wish to understand the influence ofthe organizational environment on trwsfer of tra ining led to t he design of this res earch p r oject and its methodolog y A 6 5

PAGE 80

thorough analysis of the literature specifically related to empirical studies of transfe r of training and the organizational environment is described in the next section. 2.5 Statement of the Problem While many acknowledge that tra nsfer is contingent on the post-training organizat i onal environment, insufficient empirical research has been conducted to guide practice and the studies that have been conducted are often methodologically deficient to adequately assess transfer (Baldwin & Ford 1988 ; Curry 1997 ; Macaulay & Cree 1999 ; Tannen baum & Yuki, 1992). Some of the authors prescribing transfer approaches do not even offer empirical studies to support their contentions (for instance Garavaglia 1993; Johnson & Johnson, 1998 ; Marx, 1 986), but rely in stead upon the opinions of their authors. Methodological problems with the studies on transfer of training and the organizational environment can be categorized into the following areas : sample size type of subjects ; equivalency of training programs, criterion measures of performance and experimental rigor. Using these criteria the research st dies which focused on the effects of t he organizational environment on transfer were analyzed and are presented in Table 2 2 66

PAGE 81

Table 2.2 Summary o f Transfer and Organizational Environment Research Studies Resear ch Sample Size Type of E qui v al e n cy of Cri r er ion Exp eri m e ntal S r udy Trainin g measures of rigor programs perfomwnce Baldwin& N= 180 Good P r oblematic In adeq u ate self-Low, Magjuka ( engi r.eers) trainees a tten ded report of survey sdf1991 mu ltiple training i n tenti ons repo rt lack of programs equivalent training prog r ams Baumg a rtel & N = 240 Good Problema ti c Inadequate Low, Jeanapierre. (ma n agers) part i cipan t s self-report lack of 1972 atte nded 17 m a tched equivalent different responses t o programs selftrai ning supervisors for report progran1s 3 3 respondents Bennett N = 909 Maybe Good Inadequate Low, Lehman & probl malic selfreport Survey Forsr. 1 99 9 (city emplo yees at d i fferelll levels & departments ) B rinke rhoff & N = 9 1 37 in Good P r obl em at i c Inadequate Medium, Mon t e ino experimenta l (corpo r a t e participants self-report random group employees) att ended 5 sample, control d ifferent gro up was traini..ng cou r ses t ai nted Burke 1997 N = 90:65 in Problem Good Inadequate Medium, experimental (co lle ge sel f-report r a nd om group students) sampl e 2 experimental gro u ps Burke & = 78 Good Good Inadequate Medi m Baldwin 1999 (research selfreport quas1scient i sts) matched expe rimental 3 responses to treatment subordinat es con dit io n s self-selection, sel f .. report multiple measures Curry, 199 7 N = 598 Good Good Inadequate Low, (chi ld welfare self-report Surve y caseworkers) 67

PAGE 82

Table 2.2 (Cont) Research Sampl e Size T y pe of Equivalency of Criterion Experimen tal Swdy Subject s training measures of rigor programs perfomwnce Facteau N= 967 Maybe Problemat ic no Inadequate selfLow Dobbins problematic spec i fic training report survey no Rus s eli (state courses defined defmed tra i ning L ad ::I, g over.'.ment s tudy was a course K udis c h manager s a t needs discussed 1 99 5 variou s assessment) transfer in levels) ge n eral. analyzed man y influences v.1th sca l es Flei s lunan N= 122 Good Good Excelle nt High random 1953 (plant selfr eport sample control foremen ) s urvey of group supe r visors and co-workers valida t ed leadership assessme nt instruments Ford N = !8 0 Good (Air Good Somewhat Low Qui non es, Force adequate Survey Sego& personne l ) se l f report Sorra 1 992 matc h ed respo n ses to supervisors Foxon 1 99 7 N= i 04 Maybe Maybe So m ewhat Medium problematic problematic, a d equa t e selfr andom samp l e (corpora t e participan t s r eport matched control gro u p emplo y ees at attended 3 re:;ponses to all levels and different some managers posnions ) training programs Gregoir e N= 210 Good Problematic Inadequa t e Low Propp, & (child welfare participants self-report Survey Poertner case orkers) a ttended from I 99 8 to 6 different train i ng programs Hand N = 42 ,21 in Good Good Adequate Medium, Richards & experimental ( corporate Su b ordinates rando m samp l e Slocum 1973 grou p manager s) rated validated co n trol group pe rform a n ce other 6 8

PAGE 83

Table 2.2 (Cont.) Research Sample Size Type of E quival e ncy of Cn' terion Exper im e ntal Swdy Subjects training m easures of rigor programs perj'om1ance instruments explanations for effect Huc zynski & N = -19 Good Maybe Inadequate Low Lewis 1 980 (co rpor a te problematic se lf-report no random employees) participants assigmnent no attended 2 control group diffe rent no cri terion training measures programs Lim, 1999 N=W Maybe Good Inadequate Lov., problematic report case study ( corporate desig11, no employees at criterion dif!erem performance levels) measure descriptive study only Machin & N =40 Good Good Inadequate, Low, Fogart\', ( polic se lf-report survey 1997 officers) Miles 1965 N = 148,34 Gocd Good Exceilent, High ll1 ( eleme tary v a lidated modified expe rim enta l sc hool instruments and Solomon fourgroup principals) rating s by 5 group design associates Ohio Child N = 499 Maybe Problema t ic Inadequate Low, Welfa re problematic participants self-report survey Training ( child we! fare attended Program ..:aseworkers ditTerent Evaluation supervisors, training Subconunitdirectors programs tee 1993 admin ist rativ offered by same e) agencv Oliver o, N = 31 ( 8 Good Good Excellent Low Bane& participants (managers) used an index of no random Kopelman became part productivity as sample, no 1997 of the outcome fonnal intervention variable intervention no reducing control group sample size did use criterion to 23) measures 69

PAGE 84

Table 2.2 (Cont.) Research Sample Size Type of Equivalency of Criterion Experimental Study Subjects training measures of rigor programs perfomwnce Rouiller N= 102 Good Good Adequate High Goldste i n ( ::1ssistant fast multiple no control group 1993 food store measures was used did managers ) tran sfe r have multiple behavior rated measures by managers and crew members Tav l or 1 992 N = 350 Good Problematic Inadequate Very Lo\\, (::orporate surveyed 350 training survey training emp l o y e e s) corporations, no directors directors equivalency provided reported on opinion for trainee behavior training no equivalency participants Tesluk. F arr, N= 252 Maybe Problematic Ina d equate. selfMedium Mathieu, problematic subjects could report data multiple Vance 1 995 ( employe es have attended measures to of d!lY of 12 establish rran ponatio n course s climate dept at companson different groups no levels criterion performance measure Tracev N= 104 Good Good Adequate Medium Tanne n baum (supermarket multiple multiple Kavanagh :nanagers ) measures for measures to 1 995 climate and estahlish transfer ;;Jimate comparison groups, no criterion perfonnance measure Xiao 1 9 9 6 = 106 Good Problematic Somewhat Low ( factory subjects adequate survey no workers ) attended training Self-report and control group, at four different group efficiency used group factories measures efficiency ra tes 70

PAGE 85

2.5 1 Sample Size An inadequate sample size affected the generalizability of several studies Lim's (I 999) research used a case study design and had only I 0 subjects Oli v ero Bane and Kopelman (1997) study had an original sample size of 31 that was reduced to 23 when eight subjects became part ofthe intervention. Machin and Fogarty (1997) had only 40 subjects, while Huczynski and Lewis did better with a total sample size of 47 When surveys were used the sample size increased substantially For instanc Facteau et al. (1995) had a sample of 967, Curry s (1997) study used a tota l of598 respondents, Baumgartel & Jeanapierre ( 1972) were able to obtain 240 surveys, and Bennett Lehman and Forst ( 1999) s u rveyed 909 employees T aylor's surv ey generated 350 completed question n a i res but the response rate was only 26 percent. Otherwise most of the studie s had sample sizes between 90 and 200 subjects 2 5 2 Ty pe of Subj e cts Research studies can be generalized to the larger population when subjects are similar to the population that the study intends to understand ( Rubin & Babb ie, 1989). W hile the Burke (1997 ) study had many positive attributes and attempted to be experimentally rigorous it was fundamentally flawed for several reasons, one of which was the choice of college students as subjects Another problem occurs when subjects at different job levels are mixed together without 71

PAGE 86

controlling for employee level. For instance, the Faxon ( 1997) study used subjects at several different job levels from clerical staff to plant operators The Ohio Child Welfare Training Program Evaluation Subcommittee analyzed data from caseworkers supervisors, administrators, and directors Bennett Lehman, and Forst ( 1999) surveyed 909 subjects from different city departments and with widely disparate job responsibilities and titles. Subjects at diver e job levels and functions may be dissimilar enough to seriously jeopardize the study findings When research subjects are similar enough in job level and responsibilities they react more similarly to the training interve ntion and any differences in reaction could be attributed to any of the study's i ndependent variables 2 5 3 Equivalency of Training Programs A lack of equivalent training p rograms also may affect the ability to predict results as the findings could be from differences between training programs rather than some other factor such as the organizational environment Baldwin an d Ford s model states that transfer is contingent upon trainee chara teristivs training design a nd work environment, so differences in training programs could explain variances in transfer. For instance, the Baumgartel and Jeanapierre (1972) study assessed 17 different training programs to determine the kinds of managers who are most likely to apply new knowledge and the kinds of organizational environments that are most likel y to facilitate managerial effort to 72

PAGE 87

transfer, but not enough attention was paid to how the training programs may have affected results. Tesluk, Farr Math ieu and Vance (1995) drew subjects who had attended any of 12 train i ng programs. Facteau et aL ( 1995) was even more ambiguous as survey subjects had attended the "state management t raining" in the past No information was pro v ided on whether the training included different courses and the period of time covered b y the "past." Bald win and Magjuka ( 1991) drew their sample from engineers attending multiple training pr ograms; the type an d number were unspecified Brinkerhoff and Montesino s ( 1995) s t udy used subjects enrolled in five different training course s and Xiao (1996) assessed Chine se workers who received in-house training at four different factories but did not pro vide ev ide nce whether the training was the same at all four l ocations The Ohio Child Welfare Training Program Evaluation Subcommittee (1993) also ana lyz ed results from employees who attended differ e nt t raining courses In a n oth e r tudy Gregoire Propp, and Poertner ( 1998) l ooked at c h i ld welfare caseworkers who atte ded a nywher e from one to six training programs Participan t s wh o a tt ended more tra i ning courses could have i mproved transfe r but this was not accou nted for in the analysis Taylor s (1992) s t udy appeared to be the most egregious as 350 corporations were surveyed, but these corpora tion s cou l d have had widely different training programs and reqwremems. When a disparity between t rainin g programs exist s it is 73

PAGE 88

methodologically brash to claim results without controlling for the training program differences. 2.5.4 Criterion Measures of Performance \1any of the studies that presumed to predict transfer had no criterion measures of performance and instead relied upon trai nees self-report to assess trans er (Baldwin & Magjuka 1991; Baumgartel & Jeanapierre, 1972; Bennett Lehman & Forst, 1999; Brinkerhoff & Montesino 1995 ; Burke 1997 ; Curry, 1997 ; Facteau et al, 1995 ; Ford et al., 1992; Foxon, 1997; Gregoire et al., 1998; Huczynski & Lewis 1980 ; Lim 1999; Machin & Fogarty 1997 ; Ohio Child Welfare Training Program Evaluation Subcommittee, i 993; Tesluk et al, 1995; Xiao i 996). Self-report distorts an accurate measure of transfer (Baldwin & Ford 1988; Cruz, 1997) Cruz (1997) recommended measurin g transfer using high instructional alignment conditions ra ther than low or moderate conditions Assessment (or instructional) alignment occurs when the assessment most closely matches the conditions at t he training Self-report is considered low to moderate alignment while criterion measures of job performance using tools such as observation by trained raters results in high alignment (Cruz 1997). Also questionnaires administered shortl y after training may have a "halo effect thu s providing inaccurate information about transfer (Buczynski & Lewis, 1980). 74

PAGE 89

Some studies that used self-report oftransfer attempted to obviate the inherent bias by vrOSS-checking the participant s selfre port against supervisor and / or co workers opinions For instance Ford et al. (1992) surveyed supervisors, Fleishman (1953) surveyed supervisors and co-workers and Burke and Baldwin ( 1999) cross va l idated results with subordinates. Tracey, Tannenbaum and Kavanagh (1995) measured organizational climate and transfer behaviors with uest i onnaires sent t o sup rv i sors co-workers and subordinates Baumganel and Jeanapierre s study attempted to correlate responses between workers and supervisors for 33 subjects (out of a sample size of240) and found a correlation of .43 Taylor s study (1992) sent questionnaires to the trainees' supervisors about their r eactions degree of learning and changes in supervisor behaviors Interestingly Rae (1991) found littie correlation between the managers' and trainees responses ; in fact, the similarity was so weak that it sometimes seemed as if they were talking about differ ent people However Rae does recommend sending a questionnaire to the upervisor to compare the similarities and differences to develop a more comp l e te picture of the situation. Some studies did attempt to use criterion measures of performance. For instance the Miles (1965) study accounted for trainee training, and organizational variables in relation to performance on three criterion variables using a performance test evaluations by the tra iners; and a self-perceived learning 75

PAGE 90

measure Also an average of fiv e co-workers were surveyed using the same instrument to arrive at a "verified change score (Miles 1 965). Hand et al 's (I 973) study also used several measures including subordinate surveys and validated instruments Olivero Bane, and Kopelman (1997) looked at an outcome variable using an index of productivity. Xiao (1996) also tried to look at an outcome variable by using work efficiency data but its utility was weakened because it measured group rather than individual efficiency rates 2 5 5 Expe rimen tal Rigor Many studies lacked experimental rigor thus reducing the ability to generalize findings For purposes of this analysis, experimental rigor is inclusive of all the previous criteria discussed as well as additional fact rs looking at the experimental methodology to arrive at a generalized rating for each study Only three studies were ra t ed high for experimental rigor Qualities that made Fleishman s (1953) study rate high were the pre / post test design, random assignment control group and criterion measures of performance Miles ( 1965) used a modified Solomon four-group design with two control groups a pre measure and two post measures to assess performance Rouiller and Goldstein s ( 1993) study lacked an intervention other than the training and there was no comparison group; deficiencies however were made up by the strength of the measures used to assess organizational climate and transfer and thus the 76

PAGE 91

experimental rigo r w as rated high Current managers established the organizational climate ofvarious work envir onments and multiple measures assessed transfer behavior and job performance Rouiller and Goldstein also used co n trol varia b les to i so late the effe cts of training from other explanations Some studie were rate d medium due to positive characteristics that w ould ha e resulte d in a high rating except that they were dimini s hed due to a va riet y of methodological weak ness es For instance Brinkerhoff and M ontesino s (1995) study used ra ndom selection and a c ontrol g roup but intervention effects were not isolated to the experimental group Thus it would be difficult to generalize findings on tainted results. Hand et aL' s study (19 73) used a pre / post test de sign experimental/contro l group, and crite r ion measures of performance T his study however, had a lo w sa m ple size and because the outcome indicators were so broad (e g job performance and promotions) the results may or may not have been the result oftraining. Many studies were rated low be cau se o f flagran t errors that compromise the valid i ty oftheir research re s ult s For instance the Xiao (1996) study lo oked at factory workers in four organizations. There was no control group and tra n sfer was measured b y loo k in g at a gro up measure of efficiency and selfreport q ues tion naires It w ould be difficult t o generalize the findings when a group measure of performance was u ed rather than individual. Also Xiao personally observed the organizational v ar i ab les at each of the sites rather than allowin g t he 77

PAGE 92

subjects to rate them and attach their own meaning to the climate While the subsequent statistical anal y s i s was sophisticated the Xiao study was rated low because of the flawe d experimental conditions The Baumgartel and Jea napierre (1972) study also was ra t ed low d espit e th e ir attempt to make it sc i entificall y rigorous by u sing a stratified random sample and controlling for various factors Problems included the number of different training programs (17 total over a four year p eriod), the use of open-ended questions and the weak c o rrelations presented in the findings (t h e highest was 2 7 most ranged around .18) O th er studies wer e rated low because t hey were not by nature experimental, but r ath er were s urveys to report behavior ( for instance Curry 1997 F ac t eau eta!, 1995 ; Gregoire, et al, 199 8) Baldwin and Magjuka s study reponed o n trainin g participants intention to transfer rather than actual t ransfer behavior. The Tay lor (1992) study was r a te d even I wer because it asked for t raining d i rectors to g en eralize and report on all t ra inee s transfer behavior These indivi duals ma y or may not know about t he t r ansfer experie n ces of all trainees and aggregating all e x perienc es d i l ute the strength of personal experiences for making comparisons The Lim (1999) study was rated low (at least for purposes of generalizabili t y) becaus e it w as a descriptive case study analysis and by nature it was not e xperimental 78

PAGE 93

2 5 6 Rationale for Current Research Study ln 1 988 Baldwin and Ford f ound only seveP. articles dealing with transfer an d the organizational environment This literature review found a total of 25 articles, more than a three-fold increase in the quantity of studies Nonetheless while it is encouraging that such a quantity of legitimate studies could be found, the y ar e s t ill inadequate to conclusively understand the influence of the organiza t ional environment on t e transfer oftraining, especially since so many of research studies were methodologically deficient Of particularly concern is the lack of studies with criterion measures of performance to assess transfer empir i cally More methodologically rigorous studies are needed to build a solid foundation for understanding the role of the organzation in transfer of training from the classroom to the j ob This research pro j ect seeks to add another block to the transfer literature foundation and address some of these deficiencies of previous studies through the methodology described in the next chapter. Also, a review ofthe literature leads to the hy p otheses presented in the next section 2 5 6 1 Research Hypotheses Baldwin and Ford s (1988) framework states that transfer is contingent upon three groups of factors ; t rainee characteristics training design and the work environme t While public managers may have influence over any of these 79

PAGE 94

inputs the work environment factor may hold the greatest potential for intervent i on to improve transfer The work environment consists of three groups 1 The organization defined as the broader agency including administrators, managers and other units 2 The caseworker's supervisor who provides direct supervis i on including case consultation, performance review and day to day oversight 3 The caseworker' co-workers withi n his or her assigned unit Typically there are four t o eight caseworkers assigned to a unit and most are at the same employment le vel Based upon the literature re view, especially Baldwin and Fo rd's fram ewo rk the following research hypotheses were tested : H1 Participants from organizational envi ronments they pe r ceive as su pportive will retain mor information in the post-training interviews H2 Participants with supervisors who are percei ved as supportive will retain more information in the post-tra ining interviews H3 Participants with co-workers who are perceived as supponive will retain more information in the post-training i n terviews 80

PAGE 95

Organizations perceived as supportive should produce trainees who perform better on the skill demonstration interviews while non supportive organizations are hypo th esized to generate trainees who do not perform as well in the skill demonstration interviews Likewise supervisor s and co-workers who are supportive are hypothesized to positively influence employees who will then perform better on the skill demonstration interviews The organization, super v isor and co workers combine to influence the organizational culture T hese hypotheses will inform whether the r e i an organizational cult ure that is support i ve of training results in improved interview performance by using a criterion measure of performance for testing trai ning retention The next section discusses the research methodology used in the project 81

PAGE 96

3. Met hodology 3 1 Research Project The dissertation represents a higher levei evaluation ofthe Specialized Int erv iewing Skills ( SIS) training for new child welfar e workers to assess whether t raini ng impro ve d performance and t o understand the conditions that might have facilitated or inhibited transfer of training from the classroom to on-the-job p rformance The thesis desi gn utilizes the existing training format and adds e ve ra i components to obtain the needed d at a As part ofthe usual training trainees attend a three-day tra i ning and as an activit y fthat training interview an actor w ho is role-playing a late nc y -child who may have been the victim of sexual abuse The interview between the trainee and actor is v ideotaped for review b y th e trainer during the evening of the second day and subsequent rating b y a trained vide otape rater using a 50 -it em i n stru ment that reflects the com pon ems of a n effective sexual abuse interview The research design adds a pretest to establish pre-traini ng ski Is, a posttest to determine retention of interview skills after a pe r iod oftime, and survey questionnaires to understand the factors that may h av e fac ilitated or inhibited t ra nsfer of training. 82

PAGE 97

3 2 Research Design The dissertation is a descriptive stud y using a descriptive design Primarily. quantitative data are used for t he analyses with some qualitative inform a t i on prov i ding context t o understand the factors that contribute to transfer of train i ng Data on interviewing skills were collected at three points in time : 1 Pre-training interview skill assessment ( pretest) 2 Training intervie w sk ill assessment (training test) 3 Post -tr a i n ing i nterview skill as sessment ( posttest) Th e in t en t o f a pre-trainin g inte rv ie w skill assessment was t o establish the skills hat t r ainees brought to the training The training interview skill assessment demons t rated how much the subject was able to do immediately after the training Retentio n a nd thus transfer we r e ass e ssed at the posttest intervie w skill asses s men t and thro u gh que s tio n na i res c ompleted by the subjects and their su p erv isors Other factors w ere assessed through the collection of questionnaires from t h e research subjects and thei r supervisors (as explained in the Section 3.4 bel ow) 3.3 Research Subjects 3 3 1 Child Welfare Caseworkers Characteristics T he total sample of 61 research subjects were child welfare caseworkers from any of the 63 counties within the State of Colorado. Their job positions and 83

PAGE 98

functions varied and included Intake (conducti ng child abuse and neglect investig ations ), Ongoing (providing ongoing case management services), and specialized units (e.g. Yout h-in-Conflict Adoptions, Intensive Family Therapy) While their job funcfons var ied somewhat all trainees were at approximately the same employment level and all had d i rect contact with clients of Social Services (see below for a more thorough discussion of characteristics). Typically the SIS training course is taken during the first year of employment and is optional for all caseworkers although it is mandatory in some counties for caseworkers from Inta e or Sexual Abuse Specialized units A total of70 subjec s were recruited and completed at least the pretest and training test interviews. Nine subjects were later dropped from the project because either they did not complete the posttest or they interviewed one actor who had to be dropped from the study due to measurement issues (thereby disqualifying those tests) Ultimately 61 subjects completed the three videotaped in:erview kill assessments and the required questionnaires. ew caseworke r s may be at levels o ne through four dependin g on county policy salary requirements, and education and experience level. See Table 3.1 for a breakdown of curre nt j b titles for the research sample 84

PAGE 99

Table 3.1 Rese a r c h Subjects and Current Job Titles (N = 61) Joh Title Frequency Percent Caseworker I 9 14.8 Caseworke r II 13 21. 3 Caseworker IIJ 32 52.5 Casewo r ker IV 7 11.5 Caseworker levels imply the amount of exper i e n ce and education caseworkers bring to thei r job rather than a differentiation in the types of assignments given t o each level. A cas eworker at C a s ewor ker I could have th e same job responsibilities as a caseworker employed at Caseworker Til although a caseworker assigned to a specialized unit, such as the sexual abuse team, would likely be at a higher level because ofthe experience necessary to perform those respons i bilities Thus, th i s group of trainees with 65 percent at the Ill or IV level, is already quite expe rienced (note: a less experienced gro up might get more benefit from tra i ning) 85

PAGE 100

Table 3 2 p r esen ts the break down of thei r current unit assignments Table 3 2 Research S ubjects and Unit Assignments (N=61) U nit F r e quency Percent Intak e 14 23. 0 Ongoing 18 29 5 Gene r al 10 16.4 Youth-in-Conflict 2 3 3 Sexual Abuse Team 4 6 .6 Adoption 5 8 2 Core Services 1 1 6 Other 7 11.5 Caseworkers w ho are as sig ne d to the Intake and Sexual Abuse Team units are m o re likel y to conduct investigative interviews with children whi l e caseworkers assigned to units such as Adoption and Ongoing would not generally be c ond uct ing invest igative interv e w s unless a child discloses sexual abuse d uri ng the co urse of regular casework w ith the child Caseworkers who are assigned to G eneral units work in more rural counties and typically engage in the full range of casework responsibil i ties including conducting investigations and providing case managemen t to clients From this information, it would appear that only about 4 6 86

PAGE 101

percent ofthe sample would have sufficient opportunity to conduct investigative interviews and more specifically sexual abuse i nterviews with children The balance of 54 percent would likel y not have the opportunity to conduct any investigative i nterviews much less sexual abuse interviews with children to apply what they learned in training This group would still be engaging clients but m o re as a case manager or in a therapeutic manner (for insta n ce, as a family therapist) Caseworkers had been assigned to their current positions for an average of 10 8 months with a range from one month t o just over three years They had been at the i r county departments of Socia l Services for just over 19 months but had been working in the area of chi d and family services at this and other agencies for an average of 5.4 years Research subjects generally had little experience with this population but had been helping clients in some capacity pr.or to their current positi o n Table 3 3 presents the educat i onal levels of research subjects Table 3.3 Educational Levels of Research Subjects (Trainees) (N = 61) Education Level Fr e quenc y Percent Some college 1.6 College degree 22 36 1 Some graduate work 12 19 7 Graduate d e gree 26 42 6 87

PAGE 102

Study part i cipants were similar in educational level to all caseworker staff t hrough Colorado s 63 county departments In comparison 42.6 percent of research subjects 41. 1 percent of all casew orkers had g rad uate degrees, 50 7 percent had a college degree in areas rel ated to Human Behavioral Sciences, and 8 2 percent were categorized as Other, meaning they could have a four y ear degr e e i n a non-related field (e g scie nce ) or no degree The research sub j ects were not differen iated by the type o f college degree but ail but one person had a four-year college degree The research subjects were 91.8 percent female and 8 2 percent male and their av erage ag e was 35.6 yea r s Table 3.4 presents the i r ethnicity Table 3.4 Ethn i city o f Research Subjects (Trainees) (N = 61) E t hnicity Frequency Percent African-Ameri can 2 3 3 Hispanic 4 6 6 ative American 1.6 White 53 86 9 Oth e r 1.6 88

PAGE 103

Most of there earch subjects classified themselves as white, with few other ethnic groups represented in the sample (only seven individuals classified themselves as a member of an ethnic group other than white). No statewide comparison data on ethnicit y of child welfare caseworkers wer e available. 3 3. 2 upervisor Characteristics Of the 61 research subjects, 49 of their supervisors responded to the survey for a return percentage of 803 percent Supervisors were asked a few demog r aphic questions mainly to es t ablish their type of units and the experience they bring to their interactions with employees. T able 3 5 presents the breakdown of the supervisors' units Note that it is possible that supervisors are in different units than their employees if th e y are temporarily filling in or for some other managerial reason 89

PAGE 104

Unit Intak e Ongoing General You hi n -Conflict Sexual Abuse Team Adoption Core Services Other Table 3.5 Supervisors' Current Unit (N = 49) frequency Percent 8 16.3 18 22.4 15 30.6 3 6 1 2 4.1 5 0 2 2 4 1 3 6 1 All the ongoing and adoption superv isors responded to the survey Six of the fo u rt e en Intake supervisors five of the fifteen general supervisors two of the four sexual abuse team supervisor one of the three youth-in-conflict supervisors, and one oftwo core services supervisors did not respond to the survey The supervisors who did not respond to the survey would be more likely to supervisor workers who had the opportunity to use the training. As with caseworkers, supervisor' are categoriz ed at different levels dep e nding on their experience and job responsibi lities. Of the supervisors respondi ng to the survey 8 2 percent were at the Caseworker IV level 6. 1 were 90

PAGE 105

Supervisor I, 38 8 percent were Supervisor II 40.8 percent were Supervisor III and 6 1 percent classified themselves as Other." Supervisors had worked in their current position for an average of 4 7 9 years, been with Social Services for an average of 14.43 years and been in the field of child and family services for an av er a g e of 17.3 years This information i ndicates that supervisors were fairly experienced and wou ld presumably be able to judge accurately their trainees abilities actual transfer and organizational environment 3. Instrume nts Data w ere collected at s vera! pain s in time and with multiple instruments See Table 3.6 for a visual r e presentation of the data collection instruments and t im e frames Table 3.6 Data Collection Points and Instruments Before the Training During t h e train in g After the t raining Interview Skill Intervie w Skill In terv iew Skill Assess ment Assessm e nt Instrument Assessment Instrument Instrument Satisfaction Survey Trainee Follow-up Questionnaire Su pervisor Follow -u p Questionnaire Phone interviews with trainees Each ins trument is descr i bed below. 9 1

PAGE 106

3.4 1 Interview Skill Ass e ssme n t Instr u me n t The Interview Instrument was previously validated during the original project period (Wright, personal communication, 1999) and has been administe r ed to o ver 1000 chi l d wel f a re caseworkers since its creation lt is used to r a t e the v ideotaped interv iew bet w een a child welfare caseworker and an actor role-playing a latency -age child who may have been sexually abused The Inte rview Instrument measures the performance of the interviewer on 50 key items from the sexual abuse interview prot ocol t aught during the SIS t r a ining Use ofthe i n st rumen t fulfills Baldwi n & Ford s (1988) call for use of criterion measur es to assess training transfer Together, the 50 items on the In t erview Instrument reflect the important compo nents of an inter v iew wi t h a child to determine whether abuse, and specifically sexual abuse, has occurred and if it has to describe the incident (or incidents). Acc o rding to the exp e rts who developed the instrumen t, the items co ve r a comp l ete and efLctive interview. All items on the Interview Instrument are d iscussed and details for addressing the items are taught during the initial l 1;2 day s oftraining as part ofthe comprehensive sexua l abuse interview protoc I taught before trainees conduct the videotaped interview during training Prot ocol topics cover each stage of the interv iew from the initial greeting of the child to concluding the interview Spec i fically the Interview Instrument assesses the 92

PAGE 107

interviewer's competence in the following topic areas (each of which has multiple items) Introductions to the child (including exchange of names) Establishment of truth/lie spatial co n cepts and body parts vocabulary o Establishment of when where who what and how the abuse occurred Use of in t erviewing techniques (universalization, bui ldin g rapport non leading questions empathy) Pacing and c mfort level of i nterview Conclusion of interview (including next steps) See A ppendix B for a copy ofthe Specialized Interviewing Skill Interview Evaluation (the Interview Instrument) Trainees are g i ven a copy of the Interview Instrument prior to the videotaping and they may use it as a reference during the Interview Skills Assessment i nterview Each item i s rated on a scale of 0 to 2 Detailed i n structions for scoring each i tem and anchors for indi v idual scores are prese n ted in the rating guide used by the videotape raters at DU-GSSW. Typically a ra t ing of 0 means the interviewer did not attempt the i tem or it was completely inadequate or even de t rimen tal to the interview or child A rating of 1 means the item was partially addressed by the interviewer but was not adequate on th i s particular dimension, and 2 means the interviewer adequately addressed the item Training of videotape rate r s is discussed in the next section 93

PAGE 108

3.4 1 1 Videotape Raters Training Results from the Interview Instrument provide the criterion measure of perfo r mance necessary to more accurately assess transfer of training DU-GSSW tra in s the graduate assistant videotape raters using an instruction guide that specifies how to score individual items and the differential within each item Additionally the trainer meets with th videotape rater(s) and provides one-on one instruction on how to rate individual items Usi ng the instrument rating guide videotapes were rated by two graduate research assistants at DU-GSSW over the course of the project It was hoped that the same rater would rate all video videotapes for the pre trainin g, and posttest But this DU-GSSW graduate resea r h assistant had other commitments so was unable to rate video t apes for the duration of the project In August 1997 another rater was contracted and trained to rate th e interview videotapes To train the new rater and improve rating consistency, a sample o 14 interview videotapes was jointly ra t ed using the instrument by Videota pe Rater 1 Video tape Rater 2 and this author Differences in r ating assig nments were compared and discussed Two more interview videotapes were then rated and compared. At that point it was concluded by staff at AHA that the individual item ratings were sufficien t ly consistent between the raters and the transition to Videotape Rater 2 could be made The comparison of ratings was primarily done as a train i ng tool rather than to establish inter-rater r eliability as the intent was for the Rasch ana l ysis to account and adj u st for 94

PAGE 109

variances in rater (dis cussed below) Of the 122 interv iew videotapes from the training and posttest, 72 interview videotape s (59 percent) were rated by Videotape Rater 1 and 50 interview vid eotapes ( 41 percen t) were rated by ideotape Rater 2. Subjects could have had interview videota pes (i.e., pretest training and posttest) rated by one or both of the raters For instance Subject A could have had his pre test and t raining test rated by V ideotape Rater 1 while Video tape Rater 2 could have rated the posttest while Subject B could have had all her training tests rated by Videotape Rater 2 Some of the variance in scoring was controlled for by using Rasch modeling and a means analysis explained in the next sections 34.1.2 Rasch Adjustme nts to Test Scores The Rasch measure me nt model an item response theory model was used to calibrate the i nstrument for inheren t defects Test analysis is c ompromised when simple a rithmet i c operations are applied to tes t scores when s cores have not been a djusted for t est effects such as the r a tings of individual items Typically linearity is assumed for tes t items, but the reality is that test scores are not linear in the variable they imply To accurately analyze differences tes t scores must be transformed into measures that approximate linearity It is assumed that linearit y applies for the interviewing skill performance and in fact the Rasch modeling did fit adequately so this assumption is justified Rasch modeling does this by 95

PAGE 110

transforming a person s abilit y and item difficulty into logits which uses an equal interval scale and the natural log odds for calculating success and failure (Wright & Stone 197 9 p 17) The use ofthe Rasch model solves most of the me a surement problems inherent in test construction According to Linacre : T e primary intent ofRasch analysis is to construct generalizable lin ear measures each o f kno w n standard error ( reliability ) and v alid i t y (fit) Wi t hout the benefit of the Rasch analysis the approximation to linear i ty would have been an assumption now it is an es t ablished prope rt y o f the data ( 1 9 94 p 22) AH A staff d e cided to use the Ras c h analysis to 1) check the reliability and item function for the i nstrument in this sample and 2) to assess the assumption tha t scores are a representat i on of an u n derl y ing linear cont i nuum of ab ility and to t r a n s f orm the scale into equal intervals to better represent where people fall along the c on inuum of greate r or l esser interviewing ability (C. Parry personal communication March 9 2000) For the rating scale to function correctly the t h r ee l e v els ofthe scale must be clearly and reliab l y d i ffe re ntiated and cor r elate posi ti v ely with t he i nstrument as a whole A Rasch a nal ysis using the FACETS p r ogram indic at ed a problem with Interv iew I nstrument as the raters were not adequate ly differentiating between the three points of the rating sca les Next the Rasch statistical program BIGSTEPS was used to in effect clean up the data for u e a s the dependent variable This analysis identified and corrected ins t rument items tha t functioned poorly to arrive at abili t y measures with more structural integri y for cleaner analysis during the subsequent statist i cal tes t ing 96

PAGE 111

These ability measures form an equal interval scale independent ofthe particular irem difficulties or rating scale struc t ures in the original data The Rasch analys is was conducted by AHA as part oftheir evaluation contract with CDHS to test empirically the effectiveness of training for improving caseworker skills This dissertation is conducting a secondary data anal y sis with the pretest, training and posttest data from the AHA project to determine how factor s influence the prima ry dependent var iable which, for this research is the amount of retention from the training to th posttest as well as on other dependent var"ables (e.g., training tes t scores posttest scores) 3.4.1.3 Analysis of Means and Adjustments to Tes t Scores Post -hoc analysis showed a significant difference between the scores of inte rviews rated b y the two raters and the Rasch FACETS analysis could not be used to correct for the variance as it was found that there were not enough linkages between th e variables to make the adjustme nt. To correct for this a constant was calculated by taking the mean difference between raters at each test period and then adding it to t he test scores rated by the more difficult rater. Table 3 7 presents the Rasch means for each test period and the subsequent adjustments mad e to sco res rated by Rater l 97

PAGE 112

Table 3. 7 Ratings Ad jus tment Calculation Time Period N Mean Standard Rater Deviation Pretest Rater 1 40 1265 2678 Rater 2 2i 5200 3605 Adjustment .3935 Training Test Rater l 44 14998 5269 Rater 2 17 2 1665 .4951 Adjustment 6667 Posttest Rater 1 28 92 82 4916 R ater 2 33 1.5682 7400 Adjustment 6400 To ensure that the adjus ted scores still represented the original a correlation betwee n t he adjusted raw score and adjusted Rasch scores was performed There was a positive and strong relationship between the adjusted r aw pretest training and posttest score and the Rasch adjusted measures (pretests 98

PAGE 113

. 985 p .01; training : .963,p 01; posttests : 919 p 01) This anal ysis indicates that the corrected dat a are still sim i lar enough to the original data fo r continued data analysis 3.4 1.4 Retention as a M easure of Transfer Because of t he limitations imposed by the nature of the sexual abu se interview wi th childre l i v e inter v iews with children could not be revi ewed for tran s fer. The Interv iew I nstrum e nt measures the application of the interview protoco l during t he simulated interview w i h an actor. T he pretest provides baselin e i. formation o n the research subj ects interview s kills prior to training and le arning from the training course was e stablished by t he training t es t co n ducted on the second day of training A ppro x imatel y thre e months after traini n g a posttest was adminis tered to measure the interview skills that wer e re t ained s i nce training Learning r etentio n is an essential comp o nent of skill tran sfer to on -the-jo b pe r formance a s withou t it skill trans fe r pr es umabl y will not occur Retention has been used as a meas ure to indicate transfer (Sullivan & Clancy, 1990) and is a pre conditi on for t h e transfer of l e arnin g ( Curry 1 99 7 ; Ellis 1965 If skills are retained they can be used on the job and therefore learn i ng can be t ransferred to on-thej ob p e r for mance however t should be noted that whi l e it is a precondition for use it do es not mean the skills a r e actually used 99

PAGE 114

3.4.1.5 Time Period between Measures Longitudinal testing is necessary to as ess transfer of learning to on-thejob performance (Jones et al., 1995). The posttesting interval helps to estimate the ex tent to which training effects are sustained or retained (Sullivan & Clancy 1990) Researchers have used a wide range of intervals from one week to more than a year to assess transfer. Hand et al. (1973) and Fleishman (1953) made final ass essme nts 18 months after training F ord et al. (1992) found that four months was the critical time period fl r assessing transfer Brinkerhoff and Montesino (1995) assessed transfer at 45 days after training Similarly, Tracey Tannenbaum, and Kavanagh (1995) assessed transfer beha v iors between 6 and 8 weeks after training Delewski et al. (1986) used a twoto six-month interval. Other researchers apply a three-month time period to determine whether behavior change has occurred (Garavaglia 1993 ; Hand et al., 1973; Kirkpatrick, 1983; Rae 1991; Sullivan & Clancy, 1990) As evidenced by the previous discussion a wide range of intervals has been used to assess transfer b t according to some studies it does not matier how much time elapses Whether it is one week or 90 days, transfer will be the same (Ellis, 1965) While the literature lacks uniform guidelines other researchers have set a precedent of a minimum of six weeks between training (for instance Brinkerhoff & Montesino 1995; Delewski et al, 1986; Ford et al., 1992 ; Tracey 100

PAGE 115

et al., 1995) and the transfe r measure For this project a three-month interval between the skills assessments at training and posttest was attempted. This timeframe however varied for each person as the inducement to conduct the posttest, the advanced training, was offered six times throughout the project period and subjects were drawn fro m 18 different SIS training classes Thus the time interval between the training and posttest interview varied from two to four months The average length of time between the training test and posttest was 1 3.49 w eeks with a range of 9 to 19 w eeks Time-between tests was used as a covar i ate to determ ine if lapse of time re s ulted in significant differences i n re tenti on. 3 .4. 2 Satisfaction Survey At the conclusion of each SIS training a training satisfaction survey was administered to all trainees (including non-research project participants). The satisfaction survey was designed by AHA and is giv en at the conclusion of all trainings spo nsored by CDHS. The survey assessed trainees' reasons for attending the training class degree of perceived learning of training course competencies, satisfaction with various dimensions ofthe trainer's performance intention to use tr ai ning and p e rceived agency support of training It consists of 27 i tems categorized into the following sections : r easons for attend i ng ; trainer feedback; workshop content; and competencies ; these were measured with a 101

PAGE 116

Likert scale See Appendix C for a blank copy ofthe satisfaction survey Responses are confidential but bar coded to match demographic and performance infor mati on t o trainees Differences in satisfaction, intention to use, and perception of su pport cou l d have been used during the data analysis phase to deter m ine if perceived satisfaction influenced outcomes, but were not since so little variance was present 3 4 3 Follow -up Questionnaire Empirical measurement of organiza tional behavior to assess for organizational culture towards training is difficult due to the lack of measures to determine whether an organization is training friendly, (e g., percent of dollars spen t on training activities) It was decid ed to measure subjects' organizational environment by asking for their opinions about their organizational env ironment on self-scored follow-up questionnaire, becau e ultimately, their beliefs about the organizational culture help to determine their behavior. The follow-up questionnaire focused on the factors t hat affected the transfer of training to determine whether the organizational culture as defined by the trainee's perceptions of agency, supervisor and co-workers' attitudes and beha v ior regarding training influenced their ability to retain and thus transfer the skills l earned in training It was designed in part using the factors from Baldwin and Ford's t y pology as well as questions borrowed from Curry's 1997 survey to 102

PAGE 117

assess transfer of train in g with child welfare caseworkers See Appendix D for a cop y of the que t ionnaire The 59-item questionnaire assessed the following factor s Support oftraining b y agenc y Support of training b y superv i so r Support of training by co-worker 8 Supervisor s familiar i ty w ith training Subject's p l an for imp lementin g training S ubject's opportunit y to us e trainin g Subject s per ceiv ed implemen at i on oftraining Pre training informat i on received by the subject o Subject s attitudes towar d s training and motivation level Consistency of training with agency's mission philosophy and goals C ons i s t ency of t r a ining with agency policies and individual responsibilities Subject s use of interview protocol segments A s ix -item Likert scale ( i e., strongly agree moderately agree mildly agree mild l y disagree moderately disagree and strongly disagree) was used and when a p propriate questions asked for specific quantities of a behavior performed (e g number of intervi ews conducted). N o neutral category was offered so trainees 103

PAGE 118

had to make a decision regarding their opinion Occasionally questions were p re sented negativel y to ensure that subjects were payi n g close attentio n to the questionnaire and did not j ust circle numbers arbitra r i l y (e g., instead of" I was able to implement. .. it read I was not able to imp l ement") The questionnaire was pretested with a group of casew o rkers who did not participate in the study Minor adjustments were made to the questionnaire following their feedback Multiple questions were asked for each topic to improve reliabilit y and to de v elop composite v ariables to use as independent v ar iables for the data analysis Demographic questions also were incl uded to establish years of experience level of education position unit gen d er ag e marital sta tus number of children and ethnicity These variables were captured for use during the data analysis as co variates and independe n t variables 3 44 Supervisor Survey A surve y was mailed to each subject s supervisor to dete r mine his or her perception ofthe organ i za t ional climate r egard i ng training and the subject's abili ty to transfer The 45-item questionna i re assessed for the same factor s as those delineated in the previous sec t ion but from the supervisor s point of-view e g instead of; "My supervisor ..vas familiar with the content of this training ," it read ; I was familiar with the content of this training This was done to match resea r ch subjects opinions to their s uper v isors and has been utilized by other 104

PAGE 119

researchers to crossv alidate opinions ( Baumgartel & Jeanapierre 1972 ; Burke & Baldwin 1999; Fleishman 1953 ; Ford et al., 1992 ; Hand et al., 1973; Rouiller & Goldstein, 1993) As with the t rainee follow-up questionnaire the same Likert scale was used a n d multiple questions were asked for each topic to build composite var iables and impro v e rel iabili t y Demographic variables were reques t ed t o help categorize supervis ors ( e g., n u m ber of years as a supervisor) Se e Appendix E for a copy of thi s questionnai re 3 A 5 Qualitative Interviews Additionally qualitative interviews were conducted with trainees by telephone t o illuminate th e quantitative findings Standardized q uestions were used as a gu ide to gather more de a i led informati on about the organizational culture r e garding training but participan s were encouraged to digress if approp riate See Append i x F for a copy of the telephone interview guide 3. 5 P!roced u res Proce dure s for collec t ing t he data are described in this section 3 5. 1 Recruitment The rese arc h projec commenced in October 1997 but recru itm ent efforts to enlist subjects i nto the research project had limited success during the initial stage As an incentive to participate trainees were offered the opportunity to att end an Adv anced Training provided by t he same trainer who teache the SIS 105

PAGE 120

training course The trainer has an excellent reputation throughout Colorado and his train i ngs are generally tho ugh t of as worthw hile and enjoyable experiences The Advan ced T rainin g also was used as an opportunity for brin g ing ba c k the resea rch subj ects for the posttest interview. Only trainees who agreed to part i cipate in the research pro ject were able to attend the Advanced Training This initial incentive howe ver, was not attractive enough to entice a u fficient number of people into participating in the research study It became apparent t hat additional inc entives and intensi v e recru itin g would be necessa ry to enl i st the necessary number o f subjec ts Shonly aft er the re search proje ct commenced a more intensive effort as made to recruit re s earch subjects CDHS agreed to purchase $30 gift cert ificat es from a local department store as an incent ive for all research subjects All t r ainees signed up for t h e SIS training wer e sent two introduction l e tte r s The first l etter came f r om the t ra i n er and introduce d the research project and aske d for thei r p articipation About a w eek afte r the first letter was sent, AHA mailed a second letter to exp lain more fully t he research project and again asking cand id ates t o signal their agreeme nt to participate by sending back the info rmed consent form in a self-addre sed st a mped envelope. AHA's recruitment let te r specified t he follo wi ng tasks a s required of participants in the research project: 106

PAGE 121

l. Conduct a pre-training videotaped interview skills assessment with an actor prior to training. 2 Attend training and complete a videotaped interview skills assessment immediately following training 3. Com plet a satisfaction survey at t he conclusion of the training 4. Conduct a post-training interview skills assessment a pproximate l y three months after t r aining 5 Complete a follow-up questionnaire The letter also included a consent form specifying the benefits and risks ofthe research project complying with the University of Co lorado s Human Research Committee policy (see Appendix G for a cop y of the recruitment letter and consent form) After about a week this a u thor personally called all trainees who had n o t responded to the recruitment letters, a more detailed explanation about the project was given and another request for their participation made. Through these efforts especially the personal t elephone calls to trainees about one-sixth to onehalf of each SIS training class (two-six research subjects out of a total class of approximately 12 trainees) were r ecruited to participate in the research project. Training c mmenced as usual and was cond ucted with the same curriculum that has been used for more than 1 0 years The primary trainer conducted all but two ofthe eighteen SIS training classes held during the fifteen month project period 107

PAGE 122

A substitute trainer used the same curricu l u m and format for t h e rem aini ng two sessions 3 5.2 Pretest of Interview Skills Recruited trainee s were asked to conduct a pre-training videotaped interview skills assessment the day prior to the training at AHA's facility Four case scenarios (two male two female) judged by the trainer to be equal in terms of difficult y and sexual abuse severity were selected f r om the original pool of eight scenarios and used exclusivel y during the project period A rigorous data collection plan designated the case scenarios and ac t ors assigned to each research sub j ect (see Appendix H) to ensure that trainees did not interview the same actor or h e ar the same scena rio more than once This data collection plan also was important in order to apply the Rasch FACETS model analysis for calibrating the Interview Instrument (see data analysis section 3 9 1 2 for further discussi o n on this statis tical tech n ique). Subsequent to their performance, all actors were trained on the purpose of the interview key characteristics of r ole playing a chil d (e g., latency age chi ld r e n are concrete in their vocabulary and behavior) and t he d etails of each scenario ( e .g., the child's personality family situation and the sexual abuse incident) Further o hel p gu id e their ad-Jibbed performances the actors were given ideas 108

PAGE 123

for responding as the children would in reaction to questions posed by the interviewer. Research subjects were presen t ed an intake form containing a brief summary of the presenting situation that inspired a report to the Department of Social Services and demographic data about the child and his or her family For the pretest they also were given a lette r (see Appendix I) stating the nature of t heir ta k and important points to cover in the interview Each interview was conducted in a pr iva te room with on ly the a ctor and trainee present and lasted no more than 30 minute Props such as ana tomical drawings anatomical dolls Lego blocks, play-dough markers and blank paper were provided in the taping area T he actor operated the camera and gave the research subject instruct ions about timek eeping and use of the props. During the vide otapin g, the actor assumed the role of the la tenc y ag child as dictated by the case scenario The acror stayed in c haracter during the entire interview and ad-libbed in response to the research subject's conduc t and questions during the interview After the intervie / s conclusion, actors wer e instructed to pr ovide no feedback to subjects lest they re ceive an u nfair ad v antage duri ng the next interview skills test to be held two days later (the training test). Each completed pretest videotape was lab eled with the research subject's name county, date, social security number and supervisor name and immediately sent to DU-GSSW for rating by the graduate st udent videotape ra ter. 109

PAGE 124

3 .5.3 Training Interview Skills Assessment All trainees received a day and a half of training using the formal curricu l um prior to the tra i ning test. The curriculum covers the use of the interview protocol with latenc y -a ge children who may ha v e been sexually abused Training ends early on t he second day to give trainees time to prepare for their v i d eotaped interviews with the actors later that afternoon at the hotel where the t ra ining is held Trainees we re r andomly assigned to either of two actors but research subjects were not allowed to see th e same actor twice If an actor were scheduled for both t he pretest and training in tervie w s the re sear ch subjects were directed t o see the other actor who h d not been seen during the pretest All t ra inee were encouraged to bring in notes t he Interview Instrument or other prop s to the v ideotaped i nterv iew to help guide them during the i nterview and lessen the ir anxiety Durin g t he v ideotaped i ntervie w, the actors role-played the vict i m and cond uc te d the in t e rview i n the same manner as dis c u ssed in th e p re test interview This time howe v er the a ctors did pro v ide all trainees with feedback ab ut t heir performances at the conclusion of the interview Videotapes were r e view ed that evening by the trainer then f e edback was provided to all trainees on the final day of train i ng Research subjects were given instructions regarding the posttest interview and Advanced Trai ning to occur approximately two to four months later. 110

PAGE 125

At t he conclusion of the training, all v ideotapes were labeled and sent to DU-GSSW for formal rating with the Interview Instrument. Research subjects were clearly labeled to differentiate them f rom non -p articipants Onl y AHA and DU-GSSW project s taff had access to the v ideotapes t o ensure confidentiality Approximate l y one month after raining, all trainees received their videotaped i nte rviews conducted at the training and copie s of the Interview Instrument with more extensive written feedback from DU-GSSW personnel. Pretest forms we re not returned Trainees were enc ou r ag ed to share the I n terview Instrument wi th their supervisors and view the video tape together to further assess their pe rforma nce and develop stra t egies for improving their interviewing skills this step was encouraged bu t it was n ot done in most c ases for which information is kno w n 3. 5 4 Posttest Interviewing Skills Assessment The Advanced T ra ining curriculu m was de v eloped by the SIS trainer as an i ncenti ve to indu ce subjects bac k for t he posttest interview at AHA Onl y trainees who particip at ed in the research p r oject were eligible for the Advanced Training This wa an o ptional training t hat enhanced performance but was not necessary for caseworkers to perform t hei r jobs Six adv anced trainings were offe red throughout the proje c t period Subjects were asked to conduct the post-training interview the day before th e Advanced Training at AHA As with the previous I 1 1

PAGE 126

two t est the same cond it ions w ithin the taping room (e g a v ailability of prop s 30 minute time limit) applied for th e posttest. Subjec ts were assigned an actor and scenario based upo n the data collection plan (see Appendix H) As before they were allowed to bring in notes to the int erview and utilize the props in the t aping room as t he y saw fit At the conclus i o n of the videotapin g, subjects were thanked and given th eir gift certificates far participating in all phases of the st udy The posttest interview videotapes were sent to D U -GSSW for rating. All posttest data w ere ente r ed into the research project database at AHA Posttest videos we re not returned but copies ofthe com pleted Interview Instruments were sen t to trainees 3.5 5 Follow up Q uestionnair e At t he posttest intervie w held before the A dvanced Training, a questionnaire was administered to all research subjects about their percept ions of thei r organiza t iona l envir o nment and other factors that ma y have affected th e transfer of train ing Questionnaires w ere given to subjects before the y co nducted their pos tt ests which were comple t ed priv atel y (s ee Appendix D ) A survey response r ate of 100 percent w a s attaine d since it was administered at the posttest interv iew. 112

PAGE 127

3.5.6 Supervisor Questionnaire After the SIS training, all supervisors of research subjects were contacted y letter to inform them that their employee(s) had participated in the research study and alert them that they would be receiving a questionnaire about their empioyees' training experience and training environment Supervisors were mailed questio nnai res immediately after their employees attended the Advanced training (two to four months after the SIS training) A self-addressed stamped envelope was included in the survey If supervisors did not return a survey within 30 days another urvey was mailed Followup telephone calls also were made to improve the response rate. Through these efforts a response rate of 80 3 percent was att ained ( 49 completed surveys) 3. 5. 7 Qualitative Interviews Approximately five months after the last set of posttests research subjects were called for telephone interviews. Tel ephone interview participants volunteered at the Advanced Training by signing up on a sheet that was sent around to all participants Twenty of sixty -one research subjects volunteered to participate in the te leph one interviews Each subject was called and an interview conducted at that time or arrangements were made at a later date Telephone interviews were guided by a set of standard ized questions and notes were typed into a computer file during the discussion Before starting the interview, subjects 113

PAGE 128

were asked p er mission for t he au tho r to t y pe comments and confirmed th at this note-taking method would not b e d istr acting This documentation method allowed the author to capture most of th e conversation Ultimately, only fourteen inte rviews (23 p ercent of total sample) were complete d due to subjects lack of int erest or because the y were no lon ger emplo y ed at their agencies 3 6 Data Analysis 3 6 1 Composite Variables Factors p o tentially affect ing transfer were operationalized in the trainee and sup e rv sor quest ionnaire. Several q uestions regard i ng each of t he factors were pu rpo se l y asked in o r de r to develop composite v ariables for use during the data analysis phase Items th at were presen ted negatively on the questionnaires were revers ed so the scales were in the same direction ( e g 6 became 1 ) All the composite va riable s were constructed during the post-hoc analysis Questionnaire items were studied to de te rmin e w ith w h i ch of the Baldwin an d F ord ( 1988 ) framework fac t or they w ere most closely associat ed to establish face validity A subsequent reliability ana l ysis calculating Cronbach' s alpha to ensure an internally consistent subscale revealed that t his initial face v alidity was i nadequate as some of t he scales had poor Cronbach's alpha coefficients Items that did not correlate w ith the ot h er items in the scale we r e dropped from the original 114

PAGE 129

composite variables to increase Cronbach's alpha and new items were added that seemed to fit better with the other items within the composite variable Another reliability anal ysi s was conducted and any composi te variable with a Cronbach's alpha greater than 50 indicating acceptable internal consistency was retained for analysis A list of the composite variables the items composing them, and their Cron ach's alpha are included in Table 3.8. Trainee self-reported variables consist of items from the trainee follow-up questionnaire and supervisor-reported var iables were developed using items from the supervisor questionnaire The tra inee characteristics composite variables from the trainee and supervisor questionnaires are mac o variables and are composed of trainee characteristics ofperceived ability and motivation Similarly, the organizational en viron ment composite va riables also are macro variables and are constructed from t he organization, supervisor and co-worker composite variables Variabl e Trainee selfr eported transfer \/aximum value: 30 Table 3.8 Definitions of Composite Variables Quest i onnaire items 5 I i m plem ented on the job at least one conce p t/skill that l learned. 10. J was able to implement the training on the job. 20 I have had sufficient opportunities to practice the new ideas/skills/techniques on the job. 2-1-. I used the training on the job. 37. Since the training. I have retained what I learned at the training. 115 Cro nhach 's A I ha 8146

PAGE 130

T'ariabl e Trainee reported motivation Alaximu m value: 24 Trainee reported percei ve d ability \Jaxim um val u e : 24 Trainee reported se l f characte ri stics .\laximum value : -18 T r ai nee rep orte d organi za tio n support 'vfaximum val ue: 3 0 Traine e reported su p ervisor support A l axunum val ue: 36 Table 3 8 ( Com ) Q uestionnaire I tems 4 I was motivated to put this training into practice on the job. 9 I had a plan a s to how I woul d implement the tr a ining 15. Even if no one not iced or notices I w ill use k no wledge l earne d from this trainin g on the job 3 I. It is i mportant for me to do well in m y job I As a result of the training I s ub stantia lly increas e d my know l edge on this topi c 2. I did lea rn new skills at this training. 27. As a res Ilt of thi s train ing. I am a more effective m terviewer. 34 After the training. r felt more confident to interview children. 4 I was moti va ted to put this training into practice on the job 9 I had a plan as to how I would implement the training 15. Even i f no one noticed or notices I w ill use knowledge le a rned from thi s t rai nin g on the job 31. It is important for me to do well in my job I A s a result of the trai nin g I substantially increased m y knowledge on this topic 2 r did learn new skills at this training 27 As a resu lt of thi s training. I a m a more effective int erviewer. 34. After th e training I felt more confident to i ntervie w child r e n Cro n bac h 's A I ha 5844 .6541 .76 45 13. In my agency. top managem e nt v alues staff training .8411 1 4 In m y agency top management views th is training a s a high prior ity 21. My organization values training. 22. My organization supported m y attem pts to use the training on the job 2 6 The uaining con tent is consistent with my agency's mission. philosophy a nd goals 6. My s up erv isor supported me in impl e mentin g the 7836 training on the job 7. My supervisor was familiar with the content of this 116

PAGE 131

f' a ri ab!e Train ee reported c o-wo r ker support \ Jaximu m val ue: 2-1 Traine e r eported environment support A;Ja ximum value: 90 Supervisor report e d transfer \Ja x im u m val ue: 2-1 Table 3 8 (Cont. ) Que s tionn a ir e ite m s 8 I met with my supervisor to discuss applica t ion of this tr a ining on the j ob 11. My supe r visor thinks training is important. 12. M y supervisor e xpected me to use this training on the job 3 0 My supervisor helped to prepare me for the training by discussing m y leaming needs C ron ba c h s A I ha 1 7 M y coor k ers value training 5969 18. My cow orkers supported my attempts to use the training on the job 19. 1 made a plan with a co-worker to use this tr a ining 23. W hen I had trouble implementing the training on the job I could count on others in my agency to help me 13. In my agen y top managemem values staff training. 8-l!S 1-l. In m y a gency, top management v iews thi s training a s a high p ri ori t y 21. My organization values training. 22 My organization supported m y attempts to use the tra i ning on the job 26 The training content is consistent with my agency s mission philosophy. and goals 6. My supervisor supported me in implementing the t rainin g on the job 7 M y superviso r was familiar with the content of this t r a ining 8 I m e t with my supervisor to discuss application of thi s training on the job 11. M y supervisor thinks training is important. 12. M y supervisor expected me to use this training on the j ob 30 My supervisor helped to prepare me for the training by discussing my leaming needs 17. M y co-workers value training 18 M y co-workers supported my attempts to use the tr a ining on the job 1 9 I made a plan with a co worker to use this training 23. When I had trouble implementing the training on t he j ob I could count on othe r s in my agency to help me 5. M y employee implemented on the job at least one 8 013 concept/skill that he/she leamed 10. M y employee w a s able to implement the training on 117

PAGE 132

SuperY is o r r eported tra ine e mot i a i on \l!axi m u m val ue: ]4 Supenisor Reported T rain e e Ability \laximu m val ue: ]4 Superv i s or r eported traine e ch a rac t erist i c s \1ax i mum v a lue : 66 S u pervisor r e ported o r g anizati on s upport H aximum v alue: 3 0 Table 3 8 (Cont) Q u es ti o n n a ire items C ronthe j o b 20. M y employee had sufficient opportunities to prac ti ce the n ew ideas/skills / techniques on the job 24. M y emplo y ee used the training on the job 37 S i nce the training m y employee has retained what he / she learned at the training bac h 's A I h a 4. My employee was mot ivated to put this training into 7783 prac tice on the job 9 My emplo y ee had a plan as to how he or she would imp l ement th e training 15. M y emplo y ee is self-motivated 3 1 M y emp l oyee is self-motivated l. As a re sult of the training, m y employee substantiall y 812 7 inc r eased hi s/ her knowledge on this topic 2 As a res u lt of the training. m y employee has developed n e w skills 2 7 As a result of this training, m y employee is a more effect i ve interviewer. 3 4 After the training my employee felt more confident to interview children -1.. My emplo y ee was motivated to put this training into 8506 practice on the job 9 My emplo y ee had a p l an as to how he or she would implement training 1 5 M y e m pl o y e e is s e lfm otivated 3 1 My e m ploye e is sel f -mo t iva ted 1. As a resu l t of the trainin g m y emplo y ee substantially increased his / her knowledge on this topic 2 As a result of the training m y employee has developed n ew skills. 2 7 As a result of this training my employee is a more effect i ve interviewer. 3 4 After the training m y employee felt more confident t o interview children 13. In m y agency. top management v alues staff traini1 g. .82-+l 1 4 In my a g en c y top management views tllis training as a high priority 2 1 M y organizat i on values training 22. My organization supported my employee s attempts t o use the training on the job. 26. The training content is consistent with my agencys 118

PAGE 133

l 'ar iable Supervisor reported supeiVis r support .\1 aximum value: 36 SupeiVisor rep ort e d co worker support Maximum value: 24 SupeiVisor reported environmen t support Afaxi mum value: 90 Table 3 8 (Cont) Questionna ire Item. C r on -mission philosoph y and goals bach s A I ha 6 1 supp orted my employee in imp l ementing the .48032 tra inin g on the job. 7 I was familiar w ith th e con tent of this training 8 I m e t with m y employee t o discuss ap plic ation of this tra ining on the j o b 1 1 In m y op inion. training is important 1 2 I expect e d my employee t o use this training o n th e job. 30. I helped my e mployee to prepare for the trainin g b y disc ussing his/her leaming needs. 17. M y s taff value s training .::i742 18 My staff sup ported m y emplo yee s attempts to us e the t ra inin g on the job. 19. My employee made a plan wi th a co-worker as to how to use this training. 23. Whe n staff have trouble i mpl ement in g training on the job. we can count o n others in my agency to help u s 13. In my agency to p mana ge ment values staff tra ining .8091 14. In m y agency to p managemen t vi ews this training as a high priority. 21. My organizati on values training 22 My orga niza tion s upport e d m y employee s attempts to u se t 1e traini n g on th e job. 26. The training content is consistent with my agenc y"s mission philosophy and goals. 6. I supported my empl oyee in i mpl eme nting the training on the job. 7 I wa s f a miliar with th e conten t of tllis training 8. I met with m y empl oyee to dis cuss application of this training on the job. 1 1 I n m y opinion, training is important 12. I expect e d my employee to use this t rain ing on the job. 30 I he l ped my employee to prepare for the training b y discussing his/her l eaming needs 17. My staff values training 18. My staff supported my employee s atte mpts to us e the traini n g on the job. 19. M y employe e made a plan with a co-worker as to 2 This c omposit e va riabl e i s below the threshold of 50 It is presented for illustrative purposes only. 119

PAGE 134

r ariahle Table 3 8 (Cont.) Questionnaire Items how to use this train ing. 23. When staff hav e trouble i mplementing training on the job. we can count on oth rs in my agency to help us. Cron bach s A I ha To increase variability questionnaire items were summed rather than averaged to develop the composite variables for use in the data analysis. To further validate the choice of items for the composite variab les an exploratory factor analysis to confirm the items chosen for the compo ite 3.6.2 Confirma tory Factor Analyses The factor analysis used principal components analysis with varimax rotation and resulted in five significant factors that are quite similar to the factors built from the literature review. Previous studies and relevant literature determin ed the associatons between the composite va riables and the factor anal ysi s help to confirm their relationship This factor analysis is only confirmatory since the sample size is 61 subjects and below the acceptable level to conduct a reliable factor analysis (Kline 1993). Table 3 9 contains the indi v idual items which loaded onto Factor l and the composite va riable in which the item was previously designated Factors with loadings above 6 are considered strong while factors at 3 or above are considered acceptable Factor loadings of below 3 are not used (Kline 1993) 120

PAGE 135

Item ;:: 10 12 24 1 6 5 20 22 !5 25 9 37 Ta ble 3.9 Factor Loa ding of Items Contained within Factor 1 : Transf er !U sed training and Previously Designated Composite Variable I tem Description Factor Previously Loading Designated C omp o site lariable I v.as not able to implement the training on the job 832 Transfer My supervisor expected me to use this training on the .7-l5 Supe1viso r job. I t sed the training on the job 7-l4 Transfer The training was relevant to my job duties. 733 N on e I implemented on the job at least one concept/skill that 725 Transfer I learned. I have had sufficient opportunities to practice the new .706 Transfer ideas/skills/techniques on the job. My organization supponed my attemp ts to use the .595 Organizati raining on the job. on I was mot i a ted to put this training i nto practice on the 5-H MotiYation job. Even if no one noticed o r notices. I will use .531 Motivation learned from this traiPing on the job The training conten t was consistent with my age nc y s 508 None policies and m y in ividual responsibilities I had a plan as to how I would implement the training .483 Motivation Since training_ I have r etained what I learned at the Al5 Transfer training Factor 1 is labeled "Transfer/Used Training since most ofthe items related to w hether transfer occurred, trainees used the training and t heir motivation for using the training This facto r is composed of 12 items five of which were previously designated a the trainee self-reported transfer composite 121

PAGE 136

variable and three which were part of the trainee reported motivation composite variable. It is important to note t hat five of the five items within the trainee selfreported transfer composite variable and th re e of the four items from the trainee reported motivation composite variable compose Factor l These results support the composition of the tra inee self-reponed transfer variable Table 3 .I 0 presents the factor loa dings and previously d esig nat ed composite varia bles for Factor 2 Item :1 21 2 6 1-+ 13 ll 2 25 22 Table 3.10 Factor Loading of Items Contained within Factor 2: Organization Support and Previously Designated Com posite V ariable I t em D e script io n Factor Loading Designated Composile I ariah/e My organization values train ing 774 Organizati on The training content is with my agency's .76-+ Organizati mission. philosophy and goals on In m y age n cy, top management v iews this training as .760 Organi zati a low priority on In my agency. top management va l ues staff training. .729 Organizati on M y supervisor considers training important .671 Supervisor I did learn new skills at this training .647 Ability The training content was consi tent ith my agency" s .506 None po l icies and my individual re spo ns ibilities My organization supported my attempts o use tlte .413 Organizati training on the j ob. on 122

PAGE 137

Factor 2 is easily identified as the "Organiz ation Support factor as five of five items that previously were designated as the organization support composite va r iable are found in this factor. The oth er three items logically synchronize with the organization support composite vari able The composition of the organization support composite variable is supported by these results Item ;;. 7 3 0 8 6 3 1 12 Table 3 .11 presents the results for Factor 3 Table 3.11 Factor L oa ding of Items Contained within Factor 3: Supervisor Support and Previously Designated Composite Va riable Item Description Factor Previously Loading Des i g nate d Composit e r ariab/ e My supervisor was familiar with the content of this .793 Supen, isor training My supen, isor helped to prepare me for the trai n ing 736 Supen,.isor by discussing my learning needs. I met with my supervisor to discuss a pp lication of this 726 Supervisor training on the job. My supen' isor suppo rted me in implementing the 689 S upen, isor tr ainin g on the job. I rely on m y rerun and supervisor for feedback. .-l98 one My supervisor expec ted me to us e this training on the .346 Supervisor ob Factor 3 is obviously designated as the Supervisor Support" factor as the items are all related to the supervisor's support and in fact five of the six items that were previously assigned to the supervisor support composite variable loaded onto this factor thus confirming t he composition of that composite varia ble Only i23

PAGE 138

item I I from the pre viousiy des ig nated composite variable does not appear in t his acto r but ins tead loaded onto Factor 2 It e m ..,. 37 3 6 3 1 33 27 Table 3 12 presents th e results for Factor 4. Table 3 .12 Fac tor L oading of Items Contained within F actor 4: Trainee Characteristi cs and Pre vi ousl y Des ig nated Composite Variable Item Description Factor Loadi n g Designated C o m posite I 'ari abl e As a result of the trainin g I substantially increased m y .721 Ability kno\vle d ge on this topic Sinc e the training. I ha\'e retained what I learned at the 693 Transfer t raining. Prior to a tt ending the trainin g I heard the training v as 642 None .. worthwhile ''/valuab le It is importa nt for me to do well in my job. .514 Motivation I know insi de when I do well a t m y job or don t do .491 None well at m y job As a result of this t raining I am a mo r e effective .423 Ability interv iewer Facto r 4 is not so cle arly assoc iated with the previously designated composite v ariables but a close examination of the items clearly indicate that all the e items relate to intrinsic tra i nee char acteristics Two of the four items previously designated as part ofthe trainee reported perceived ability composite va r i able appear in this facto r as well as one of the items re lated t o train ee report ed motivation The other items were not ass igned to a composite va riable b u t do appea to be related to trainee characteristics 124

PAGE 139

! r em ;= t8 1 7 1 9 1 9 ?"' _.) Table 313 presents factor loadin gs and previous designations for Factor 5 Table 3.13 Factor Loading of Items Contained within Factor 5: Co Worker S upport and Previously Designated Composite Var i able I t em De s c ripti on Factor Previous(v Loading Designated Composite T ariabl e My co worker supported my attempts to use the 662 Co-worker training on the job. !vl y cow orkers valu e trainin g 597 Co-worker I made a plan it.h a co-worker to u e this t raining -+56 Co worker Prior to th e workshop I was motiYated to a tt end .-+ 16 Motivation When T had trouble implementing the t raining on the 396 Co-worker j ob. I could count on others in m y agency to help me Of the five items which loaded onto Factor 5 Co-worker Support ," four of the items were previously designated as belonging to the co-worker support composite variable This fa ct or confirms the choice of items related to co-worker suppon. T he digression into factor analysis has been somewhat useful for c onfirming the choice of items for the composite var iables but furthe r use of these factors would violate bas i c assumptions of factor analysis and so composites based o n this analysis were not used a s the basi s for further analyses The factor anal y s i s d i d show that the composite variables are statistically and conceptually congruent for further use in sta t istical analyses For exploratory analysis the 125

PAGE 140

factors were used as predictor variables for the dependent var iables and the results were essentially the same as those found with the designated composite variables. 3 6 3 Statistical Analyses Data fr m the pretest training, and posttest instruments were entered into one database, and trainee and supervisor questionnaire data were entered into another. Once entered data were merged to form one database. Measures were taken to ensure the confidentia i ty of all research subjects. From the three tests and two questionnaires over 250 variables were collected from each subject Descriptive statistics for each vari ab le were calculated and analyzed to look for data entry errors and to prov ide initial information on data distributions Minor adjustments were made to clean up data by referring to the original questionnaires for the correct responses Data analysis was conducted in several steps using the Statistical Package for the Social Sciences (SPSS), Version 9 .0. Descriptive statistics were used to present the data, and inferential statistics were used to determine whether there were differences between groups The data were analyzed using a paired samples T -tes t to determine whether there were statistically significant differences between scores on the pretest, training and posttest and a multiple regression analysis was conducted to determine w hether research subjects with favorable organizational climates-as de fined by the questionnaire items related to 126

PAGE 141

organ i zation s upervisor and co-worker support retained more information at the posttest. Results from the s uperv iso r and trainee questionnaires were compared to de termi n e their level of agreement and confirm t he trainees opinions of the ir transfer climate Rela tionships be t ween the composite variables demographic variables and scores also were examined for significance This analysis provides a contextual reference to understand the trainees and supervisors opinions about their orga nizational support transfer opportunities, and self-characterist ics that might facilitate or inhibit transfer of training. See Table 3 14 for a summary of research questions hypotheses dependent variables independent variables, and statistical tests that were performed 127

PAGE 142

Table 3.14 Research Questions Hypotheses, Variables, and Statistical Tests R e se ar c h QuestionS! Dependent ariab!es Independent l ariab/es .',' tatistica/ T e sts Hv otheses R 1 Does training 1) Adjusted training T tes t s improve interviewin g scor e skills perfom1a11ce) 2 ) Adjust d posttes t score Which 1) Adjusted reten tion l) Trainee reported self I ) C01relations organizational factors 2) Adjusted training characte risti cs are most influential in score 2) Trainee reported 2) Multiple promoting the transfer 3) Adjusted posttest environment s upport regression of training'/ score 3) Adjusted pretest 4) Self-reported measure transfer composite 4) Num ber of variable investigative interviews conducted 5) Job type 6) Time between training test and posttest H Participants from Primary : Trainee self reported 1) Correlations organiz.ational 1) Adjus t ed retention organization support environments they Seconda r y : compos i te va riabl e 2) Multiple perceive as supportive 2 ) Adjusted training regression will retain more test score inform ati on in the pos t -3) Adj u sted posttcst trainin g interviews Score 4 ) Trainee self-reported transfer H2 Participants with Primary: Trainee self-reported I) Corre l ations supervisors who are l) A djuste retention supervisor support perceived as supportive Secondary composite variable 2) Multiple will retam more 2) Adjusted training regression infonnation in the posttest score training interviews. 3) Adjust d postte s t score 4) Trainee self reported transfer 128

PAGE 143

R esearch Questions Hy p othese s H3 P a rticipan t s with co-\vorkers who ar e percei ve d as supportive will retain more inform a tion in the post tra i nin g interviews Table 3.13 (Cont.) D e p enden! I a riables Primary : I ) Adjusted retention Secondary: 2 ) Adjusted training test score 3 ) Adjusted posttest score 4 ) Trainee self-reponed transfer lnde p endenl r a r i ables Trainee self reported co-worker support composite variable S!alisli ca / Tests 1) Correlations 2) Multiple regressiOn Th e hypotheses (HL H2 and H3 ) that there will be a significant difference on the primary dependent variable retention scores was tested using a multiple r egress i on anal y sis O t her secondar y dep e ndent variables (i. e., training test score posttest score, self-reported transfer composite variable) were used to determine whether differences existed with any of these variables an d were anal y zed i n separate regression equations F u rther explorat ry statistical testing was conducted to de t ermine if there were alternat i ve explanations other than w hat was hypothesized to expla i n the variance in the dependent variabl e s o f retention training test scores, posttest scores and the selfreported transfer composite variable These dependent variab l es were used because while they were not specifically hypothesized they do repre ent a measure of transfer and should be explored Multiple regression equat i ons w ere built that incorpo r ated the v ariables most likely to predict to 129

PAGE 144

transfer The following i ndepe n dent variables we re entered into reg ressi on equations and used to p redict ach dependent varia ble in sepa rate equations : Nu mber of e xual abuse interview s conducted Number of investi g ative interviews conducted Tim e between tests Train ee mo t ivation composi e variable Trainee perceived abi l ity composite v ariable Trainee reported self cha racteristi cs composite v ariable Sup e rvisor reported transfer compo site v a ria ble Supervisor repo rt ed motivation co mpo site variable Superv i sor reported perceived abilit y c om po sit e variable ., Supervisor re p orted traine charact er istics composite variable Pretest scores Addi t i onally, the foliowing demographic va riables we r e entered into the multiple regression equations as c ov ariates to determine if t h ey predic ted any of the dependent variables. Title of subject Child we l fare uni t of subject Lengt h of time in current p o ition Le ngth oftime at S cia! services 130

PAGE 145

Education level Gender Age o Marital status Ethnicity Supervisor's length oftime in current position None of these demographic predictor va r i ables were significant so they will not be discussed further Results of the hypothesis testing and other significant findings are presented i n the next chapter. 131 ..

PAGE 146

' 4. Results It was hypothesized that different types of organizational environments namely those supportive and unsupportive towards training, would result in difference in the retention of interview skills at the posttest. The results did not support the h ypotheses, and in fact, suggest there is little var iability among organ izat !on as measured by trainees' responses to the questio n naire with respect to supportiveness towards training Without variability in organizational envir onments, the hypotheses cannot be supported there being no variance to exp l ain. A detailed presentation of results from the statistical analyses follows in this chapter. Univariate anal y sis of results from all the questionnaires is given first then results from the inferential statistical testing are provided A detailed exploration of the indi v idual item results from the questionnaires helps to paint a conte x tual picture of the degree of satisfaction trainees had with the training how trainees and superv i sors perceive their organ i zational en v ironments and the extent to w hich their opinions match Together with the inferential statistics an alysis, c: picture emerges t o help comprehend trainees' organizational environments and their experience with the transfer of training 132

PAGE 147

4.1 Results from Satisfaction Survey At the conclusion of each SIS training all participants complete a satisfaction survey (Kirkpatrick s level-one and American Humane Association's levels-one and two of training evaluation) Data from persons who agreed to be research subjects for this study are presented in this section Overall research subjects w ere quite sa t isfied with the training and the trainer and thought they learned a great deal at training as show n m Table 4 1 below Trai nees were asked to select two reasons from a list of eight for attending the tra i ning. The top two selections for the most i mportant reason selected were I heard it was interesting / fun '' ( 41 A percent) and I wanted to know more about this area" (50 percent) For th e second reason for attending 39 7 percent said It was required," and 25 9 percent said "I wanted to know more about this area "3 The other responses were distri b uted a c ross the other reasons This information indicates that most participants attended the t ra ining because they thought it was intere sti ng and v aluable not because they wer e mandated to attend although that too was important Trainees were asked questions related to the workshop content and the supp ort they think they will rec eiv e at their agencies Responses were selected 3 Note. the same statement could be used for the first and second reasons i e .. some people ma y haYe selec ted it as the first reason others may have selected it as the second reason 133

PAGE 148

from a Likert scale with 4 = s t rongl y agree 3 = agree 2 = disagree 1 =strongly disagree Results are pre ented in Table 4 1 T able 4.1 Responses from Questions Related to Workshop Content (N= 58 )4 Question M SD St rongl y Disagr ee Agree % Strong l y Disagree % Agree% % The subject ma tt er 3 .75 .44 25.5 74.5 was at the right level of difficulty The wor kshop 3.76 .43 23.6 76.4 content was compatible w ith m y agencys philosophy and polic ies M y agency \ ill 3.73 .4 5 27 3 72 7 support me i n using this trainin g on the job. I learned specific. 3.84 37 15. 8 84.2 job-reb ted knowledge/skills. I will use 3 84 37 16.4 83.6 knowledge/s kill s from this tra inin g on the job I will be able to do 3.82 39 17. 9 82 1 my job better becaus of this training Families will 3.85 36 15 1 84 9 benefit from my taking this c ourse 4 Three trainees did not compl ete the satisfaction surYey at the conclus i on of the SIS tr ai ning. 134

PAGE 149

All participants were in the agree range regarding all the questions related to workshop content. These results indicate that trainees felt that t h e training was relevant and supported by the trainees' organizations Trainees were asked to rate whether they had learned new skills/knowledge related to the intended train i ng competencies using a different Likert scale (4 =yes, a great deal ; 3 =yes, moderately; 2 =yes, a little ; 1 = no) Table 4 2 presents the means and standard deviations for each training competency Table 4.2 Trainees Perceived Achievement of Training Competencies (N=58) Has y our knowledge or skill increased M SD ? zn. ... Policie s related to CPI interviews with the child 3 57 57 victim the non-offending parent. and the alleged perpetrator. Applying knowledge of the interviewing process 3 84 .42 and of child development to the CPI interview with children Utilizin g knowledge of interview i n g children in 3 87 39 conducting a CPI intcniew with a child As evidenced b y trainees perceptions the training had achieved its goal of increasing t rainees competence in those thr ee areas Trai nees generaliy thought highly of the trainer s ability Results from all the survey items related to the trainer are presented in Table 4.3 135

PAGE 150

T abl e 4 3 T r a inee P erc e ptio n s of Train e r s A b ility (N =58) Question M SD Strongly Disagree Agree% Stro ngl y D i sagree % Agree% % The trainer knew 3.97 .18 3.4 966 the subject area. The trainer \\-as 3.91 34 1.7 5.2 93. 1 well prepared an d rgan.ized. The trainer rela t ed 3.98 .13 1 .7 98.3 well to the group. answered questions and responded to concems. The trainer 3.90 36 1.7 6.9 9 1 .4 p r ovi d ed e n oug h explanation and examples. The trdiner gave 3.79 .52 5.2 10.3 84 5 me enough op p ortunities to practice sk i lls. The traine r used 3.78 .46 1.7 19.0 7 9 3 handouts flipcharts and visuals to ill u strate key points The trainer 3.86 .35 13.8 86 2 motiYated me to want t o try out the t r a i ning ideas on the job. The trainer 3 83 42 1.7 1 3.8 84 5 modeled cultural sensitivity. 136

PAGE 151

All the trainer items were weighted towards strong l y agree Based upon the training satisfaction surveys, tra i nees were quite happy with the training and what they learned thus no events at training would have interfered with trainees abilit y to transfer the training to on-the-job performance, so it will not be necessary to test for this during the statistical analysis 4 2 Results from Follow-up Questionnaire All research subjects completed questionnai r es a t the posttes t containing questions related to their : opinions about themselves, organizat i onal environment actual experience conducting interviews since the train i ng, and pe r sonal demographics Opinion items were rated using a Likert scale of 6 = strongly agree 5 = moderately agree, 4 = mildly agree 3 = mildly disagree 2 = moderately disagree and 1 =strong! disagree Some questions were negative l y presen ted in the questionnaire to help subjects pay closer attent i on to the wording o f ea c h question and are shaded in the table below; the mean, however, is presented aft er i t was recoded to more easily compare these items to the other i tems (the frequenc i es are presen ed before the recoding). Table 4.4 presents the means and percentage of responses for each ofthe six possible categories for each question on the Trainee Follow up Questionnaire 137

PAGE 152

Table 4.4 Descriptive Statistics from the Trainee Follow -up Questionnaire (N=61) QuesLionnaire !rem ,\[ Strongly .Hade r Mildly Mildly ModerStrongly DISa!ely D 1 s -Agree urely Agree agree Dis-agree % Agr ee % % a gree % % % 1 A s a r e s ult of the tra ining, I 5.6 1 0 0 0 -l.9 29.5 6 5 6 s ub stan tiall y inc r ea s e d my kn o wledge on this t o pic. 2 I did not l earn any new skills 5 85 88.3 8.3 3.3 0 0 0 at this training. 3. The training h as affect e d m y 5 .00 1.6 0 .., .., 1 8.0 4 5 9 31.1 _) _) attit ud es c o nce rni ng th e topi c area. -l. I was m o t ivate d t o p t thi s 5.4 3 0 0 1.6 -l.9 -l2. 6 50.8 trai ning in t o p r act i ce on t he job I 5 I implemented on th e job at I 5 .-l9 10 0 0 1 8 0 J -l. 8 67.2 least one conce p t/s k ill tha t I l earned 6 My s upe rv isor s up p orted me 5 1 1 0 0 .., .., -'-' 23.0 3 2 .8 4 1.0 in i m p e m enting the tra inin g on I the job I I I 7 M) su p ervi sor was familiar -!-.85 l o 1.7 3.4 32 2 33. 9 1 28.8 wi t h the co nte nt of this tra inin g I I I 8 I met "1\-ith my s u perv isor to 3 92 1 9 8 6.6 1 1.5 39.3 1 9 7 13 I dis c u ss a p pbcation of t l li s t r a i ili n g o n the job. I 9 I h ad a plan as to h o w I -l.31 0 6.6 11.5 41. 0 2 6.2 l-l 8 w oul d m plement the trainin g 10. I was not able to implement 4 .66 47 5 14 8 13. 1 11.5 6 6 6.6 the training on the job. 11. M y supervisor con s iders 5 70 82.0 11.5 3 3 1.6 1.6 0 trainin g a waste of time 5 Mean revers ed for all s h a d e d ite ms. fre q ue nc y perc e ntages presented a s collec te d 138

PAGE 153

Table 4.4 (Cont.) QuestiOn \ 1 ean 1 Strongly ModerMildly .\fildly A!oderS crongly I D1sacely DISAgree a rely Agree agree DIS-agree % Agree % 1 % agree % % % 12. My supervisor expe cted me 4 80 3.3 -1-.9 3 3 19 7 34.4 3-1-.4 to use this training on the job. 13. In my agency, top 5 .02 3.3 1.7 .., .., 16.7 31.7 43.3 ).) management v alues staff I I training 14 In m y agency, top 4 88 4 6 .7 21.7 16.7 5.0 8.3 1.7 manag ment v iews this training as a low priority 15. EYen if no one noticed or 5.64 0 0 0 4 .9 26 .2 68.9 notices. I will use knowledge learned from this training on the I j o b. I 16. The training was relevant 5.02 1.6 1.6 0 26 2 31.1 39.3 I to my job duties. I I 17. My co-workers value 5 .28 0 0 I ... ... 1 3.1 36.1 -+7 5 ).) training. I 18. My coworke r s supported 4.8-1-0 3.3 4.9 3l.l 26.2 34.4 my attempts to use the tra ining I I I I on the job. I I 19. I made a plan with a co3.02 19 .7 1 8.0 23.0 24.6 9.8 4 .9 worker to use this t r aining. I 20. I have ha d sufficient I 3.51 8 2 I 14 .8 31.1 1 9 7 16.4 9.8 opponunitics to practice the I I I new ideas/skills/techniques on I the job. 21. My organiza tion val ues 5 .16 0 0 4 .9 180 32.8 44.3 training i 22. My organization supported 4.89 0 1.6 8 2 24 6 31. I 34 .4 m y atte mpts to use the training on the job. I I 139

PAGE 154

Table 4.4 (Cont.) ["" I"'"' S tr ongly ModerMtldly Mtldly ModerStrongly Diswely Dis-Agree at ely Agree agre e DI>agree % .rlgr ee % % agree % % % 23. When I had trouble 1 4.43 3.3 0 18.0 26.2 34 4 18.0 implementing the training on I the job. l could count on others in my agency to help me. I 24. I used the trairting on the 4 64 I 3.3 11.5 24 6 23.0 1 34.4 .)_.) job. I I I 25_ The trairting content was 5 18 0 3.3 0 19.7 29.5 47 5 consistent with m y age1 cy's polici es and my indi vi dual I responsibilities I 26 The trairting content is 1 5.44 0 I" 1.6 11.5 127.9 59.0 consistent with m y agency's I I I mission. philo so phy, and goals. I I i 2 7. As a result of this training. 5 .5l 0 0 0 9.8 1295 60.7 I am a more effe cti ve m erv1ev. er. 28 My reason for attending the 4.26 39.3 14 8 9 8 13. 1 14 8 8.2 training was due to a mandatory requirement. 29 Prior to the workshop. I I 5 .51 !.6 0 0 6.6 27.9 63.9 was motivated to a ttend 30. My supervisor helped to 2 .66 i 29 5 18. 0 21.3 21.3 8.2 1.6 prepare me for the training by I discussing m y learnin g needs I 3 I It is im p o rtant for me to do 1 5.9o 0 0 0 0 9.8 90.2 well in my job. 32. I rely on m y team and 5.26 1.6 3.3 8.2 26 2 57 4 .)_.) sup er visor for feedback. I 33. I know inside whe n I do 5.44 0 0 1.6 4.9 410 52 5 well at my job or don t do well I I at my job i I 140

PAGE 155

Table 4.4 (Cont ) Question .\1ean Stro ngly Afoder MJ/dly Afildly ModerS t ro ng l y I D1s-a t ely D1s-Agr ee at ely Agr ee I agree D ISag ree % Agr ee % I % agr ee % % % 34 After the training. I felt 5.30 I 1.6 0 1.6 9.8 37.7 49 2 more confident to interview I c hildren I 35. I have low expectations for 4 97 49.2 24 6 4.9 16 .4 4.9 0 my interv iewing performance. 36. Pr i o r t o attending the I 5 39 0 .., .., 4 9 4.9 123.0 I 63.9 .)_.) training 1 heard the training I was .. \\orthwhile. '/v aluable I I I 3 7 Since the training I have 4 .74 0 1.6 4 9 31.1 42.6 19. 7 retained what I learned at the training Resul s from the questionnai r e indicate that trainees generally have a positive opinion about their training experience the extent of transfer to their j o bs the degree of support from their orga n izat ional environments degree of motivation and their interviewing aptitude Most ofthe items (26 of37 items) are negat iv ely skewed (the median was greate r than the mean) signifying that most r esponse s were heavily weighted towards strongly agree Most subjects agreed with the questionnaire items ; for 23 of37 i t ems (62 percent) less than ten percent o f participants responded in the disagree range The questions with the strongest level of agreement dealt with the training s effect on their motivation, skill and knowledge level. All responses for items 1 2 5 and 27 addressing knowledge and skills gained, were in the agree 141

PAGE 156

categories Also for items 15 and 31, dealing with perso nal motivation all responses were in the agree categories Items rated the lowest (8 19, 20 30) were making a plan with a co-worker having sufficient opportunities to practice the skills learned in training and meeting with a supervisor to discuss learning needs and t ransfer. Excep t for these four items (8 1 9, 20 30) all means were in the agree range. 1 rext the analysis looked at the questionnaire items most rele vant to the hypotheses and those that comprise the composite variables It should be noted that since a reliabilit y analysis was conducted with the composite variables no ignificant differences between items would be expected If there were the anomalous items would have be e n dropped from the composite variable to ensure internal consistency The organization support var i able consists of items 13 (mean = 5 02) 14 (mean= 4.88), 21 (mea = 5 16) 22 (m ean= 4 89), and 26 (mean = 5.44) Item 26 (" Training content i s consistent with my agency s mission philosophy, and goal s") w as rated highest while the lowest in this group although not substantially lower, w as item 14 ("I n my agency top management views this training as a high priority ) All items hovered around moderately agree indicating that trainees had an overall positive att i tude towards their organizations' support of training 142

PAGE 157

The supervisor support variable is comprised of items 6 (mean = 5. 11 ), 7 (mean= 4 .85), 8 (mean= 3 92) 11 (mean= 5 70) 12 (mean= 4.80), and 30 (mean = 2 66) These items represent a wider range of opinions regarding their supervisor support. The highest rated was item 11 ("My supervisor considers training important' ) while the lowest rated was ("My supervisor helped to prepare me for the training by discussing my learning needs") These results would indicate that whi le trainees say their supervisors are supportive of training they are not engaged by them in activities that would illustrate good training principles such as meeting with trainees to discuss their learning needs and application of the t raining to the job In other words they talk the talk" but do not "walk the walk." The next composite variable is co-worker support and consists of items 17 (mean= 5.28), 18 (mean= 4.84) 19 (m ean = 4.89), and 23 (mean = 4.43) These items are more similarly rated than the supervisor variable and, like the organization variable, hover aro und moderately agree. The highest rated was item 17 (My co -wo rke rs value training") and lowest was item 23 ("When I had trouble implementing the training on the job I could count on others in my agency to help me"). Again, these results would indicate that the general attitude training is positive but behaviors do not always indicate such strong support as th e other items would suggest 143

PAGE 158

4.2 1 Interven ing Factors on Follow -up Questionnaire Trainees we re asked to provide the number of sexual abuse intervie ws and investig at ive interviews they had conducted since the tr a ining T he y also were asked to provi de frequency information about ho w often they had used specific sections of the interview protocol and whether they had received addit i onal training o r done research on their own about sexual abuse interviewing Together this information would paint a fairly accura te picture about the opportunities trainees had to transfer the training content and if there were intervening factors that may have influenced t heir retention level (e .g., a dditi onal reading and training, opportunity to use training) Descriptive st atis tics for the intervening facto rs are given in Table 4 56 Table 4.5 Descrip t i ve Statistics o f Intervening Factors from the Trainee F ollow-up Questionnaire Qu e stionnaire item M A!dn SD 38. S i n c e the training. I have attended additional train in g on 1 66 1.00 t.r sexual ab us e interv ie mg 39. Since the training I have done additiona l reading on sexual 2.77 2.00 1.68 a bus e int erviewing 41. ince the training 1 have co n ducted sexua l abuse 1.43 00 3.14 interviews with chi ldren. 6 ote. except for questions 38 and 39. these ques t ions aske d for a specific quantit y of behaviors performe d so these results are presented slightly different l y than those in the pre vious tables 144

PAGE 159

Table 4.5 (Cont.) Questionnai r e Item M A !cln ,'D ince th training I have conducted investigati ve 6.55 3.00 8.42 ( re gard ing physical abuse. neglect. emotional abuse or se>..'Ual abuse) interviews with children. 43. Since the training I have used the introduction portion of 6.34 3 .00 8 .14 the interview protocol ___ ti mes during interviews with children -U. Since the training l have used the truth and lie portion of 4.97 2.00 7 .77 the inte rview protocol ___ times during interviews with children 45. Since the training I have used the Body Parts Inventory 1.00 .00 2 .04 portion of the interview protocol ___ times during interviews with children. Since the training. I have used the target event (e.g .. the 4.18 2.00 5.88 i ncident) portjon of th e interview protocol ___ times during interviews w ith children. 47. Smce the tra j ning I have used the summary portion of the 5.80 2.00 7.97 interview protocol __ times during interv ews with children Generally subjects did not receive additional training or read additional material. inety-rwo percent of subjects were in the disagree range 7 for the question about additional training and 59 percent were in the disagree range for the question asking whether they had read additional material about sexual abuse interviewing The results indicated subjects had little opportunity to apply the training to their jobs. On average subjects conducted 1 66 sexual abuse interviews More While it appears to be a nominal response. trainees were asked to choose a response from the Likert scale 145

PAGE 160

importantly 59 percent of subjects had not conducted any sexual abuse interviews at all. Investiga tive interviews are more broadly defined and include interviews to assess for physical abuse and neglect Subjects conducted an average of 6 55 investiga t ive interv iews and for these t y pes of interviews only 25 pe r cent of subjects had no t conducted any investigative interviews The introduction and summary port i ons ofthe interview protocol were most frequently pe r formed in onthe -job interviews while the body parts inventory section was rare l y practiced by participants-69 percent had not used this section at all 4 3 Result s from Supervisor Questionnaire Table 4 6 presents the mean and percentage of responses for each of the s ix possibie categories from the questionnaire filled out by trainees supervisors The s am e response categories were used on this questionnaire as were used in the Trainee Follow-up Questi onnaire Like the trainee questionnaire results table reversed question items are haded but means are presented after the variables were recoded for easier comparison. 146

PAGE 161

Tabl e 4.6 Descriptive Statistics of Supervisor Sur v ey (N = 49) Q u estio nnair e Item M St r o n gly ModerMild l y Mil d ly D 1 sat ely D1sAgree agree D ISag r ee % % agree % % J As a r esult of the training 5 .15 0 0 2 5 1 2 5 m y emp l oyee s u b s ta nti ally inc r eas e d his / h e r knowle d ge on thi s t o pic. 2. As a r esul t o f the trainin g, 59 5 3 1.0 9 5 0 m y emplo y ee has not developed new s kills 3. The training affec t e d my 4 8 6 1 0 2 .-+ 2.4 23.8 em p lo ye e s artitu d e s c on c erning I the t opi c area I 4 M y em p l o ye e wa s m o t i vate d 5 1 3 2 2 0 2 2 2 2 2 to put tltis trainin g i nto practice on th e j o b 5 M y e m p l oye e implement ed 4 .9 5 2.5 0 5 .0 20 0 on the job a t l e a s t o n e I I conce p t/sk ill that h e / she i l earned Q u e sti onna ir e Item .\I I Strongly .HaderMildly \ 1ddly DISa t ely D 1s-Agr ee : ag r ee D1s-agree % 1% agree % % 1 6. I s upport e d m) employ e e in 5.38 i O 0 0 1 4 9 I i mp lememi ng t h e tra ini ng o n I I the job 7 I w as f a m ili ar w i th the -U8 8.5 4.3 6.4 31.9 con t ent o f Lltis tra ining 8 I m e t w i t h m y e mp loyee to I 3 8 1 12.8 1 0 6 8 5 3 4 0 d iscuss a p plica t i on o f till I traini n g on the job. I I 8 Mean reve r se d for all sh ade d ite ms fr equ e nc y p erce n tage s pres e nted as coll ec ted 1 47 Mo d erS tr o n g l y at ely Agree Agr ee % % 5 2 5 32 5 0 0 50 0 2 1.4 24 4 48 9 37 5 3 5 0 AloderS tr o ngly a/ely Agree Agree % % 31.9 53.2 I 19.1 2 9 8 I 1 1 9 1 14 9 I

PAGE 162

Table 4.6 (Cont.) Que stionnair e Item I M Scro n g l y Mo d er-M ildl y M ildl y Mo d er-Scrong ly Drs-a c e l y Des-Agree acel y Agr ee agr ee D c s -agre e % Agr ee % % ag r ee % % % 9 My employee had a plan as 4.17 7 3 : 2.4 17. 1 29 3 126. 8 1 7 1 to how he or she would impl ement the training 10. My employe e was not ab le 4.74 46.5 16.3 16.3 9 3 9.3 2.3 to imp lement the training on the job. 11. In my opinion, training is a 5.91 93.5 4.3 2.2 0 0 0 waste of time. 12. I expected my employee to 5.17 6 3 0 2 1 16.7 12.5 62.5 use this training on the job. J 13. In my agenc y top 5.6-J. 0 10 0 10 6 I 14. 9 74 5 management val ue s staff I I training 14. In my agency, top 5.51 68.9 17.8 11.1 0 2.2 0 management views this training as a low priority 15. My employee is self.:-. 38 0 4 2 4 2 4 2 25 0 62.5 motivated. 16. The training was re le \ ant I 511 2 1 IO 43 19. 1 27 7 4 6.8 to my employee s job duties. I I 17. My staff values training. 5.6 7 IO 0 0 4.2 25.0 70.8 I 18. M y staff supported m y 5 37 1 0 !0 0 14. 3 34 3 51.4 employee s attempt s to use the I I training on the job. 1 19. My employee made a plan 3.48 1 25.9 3 7 14.8 1 8 5 25.9 11.10 with a co-worker as to ho w to use this training. 20 My employee had 3 .91 6 7 20 0 8 9 2-+.4 20 0 20.0 sufficient opportunities to JJ practice the new I I i ideas / skills / tech n iq u es on the job. 148

PAGE 163

Table 4 6 (Cont) Questionnaire i tem l"f I Stro ngly Mod e rM 1 ld l y Mildl y ModerS tr o ngly DISa t ely DISAgr ee a t ely Agre e I agree D1s-ag r ee % Agree % I % a g r ee % % % 21. My organization va l ues I 5.52 0 0 0 10.4 27.1 62.5 training i 22. My organizat i on supported 5.36 10 0 0 11.4 1 40.9 147.7 my employee s attemp t s to use I I the trai ning on the j ob 23. When staff have troubl e ,-l.63 2.1 8.3 8 3 16. 7 35.-l 29.2 implementing training on the job. we can count on o t hers in I my agency to help us I 24 My employee used the 4 56 I -l. 9 2.4 7.3 29 3 29 3 26.8 training on the job. I i I I 25. The training content was I 5.3 9 10 0 2.4 7.3 39.0 I 51.2 I consistent w ith my agency s I policies and m emp lo)ee' s I I I I responsibilities i I 26 The training content is 5.4-l 0 0 0 9.8 36.6 53.7 consistent wi th my agency" s mission philosophy and goals 27. As a result of this training 5 .30 0 0 0 18. 9 32.4 48 6 my employee is a more I effe c tive interviewer i I I I 28 My employee was 3.91 43.5 13.0 4.3 2 2 4.3 32.6 mandated to atten d the training. 1 29. Prior to the v,:orkshop my 5.46 I 2.1 0 0 8 3 27.1 62.5 I employee was motivated to I attend. i I 30 I helped my empl oyee to 3.31 l 18. 8 1 0.4 14.6 35.4 1 8.8 2.1 prepare for the training by I I discussing his/her leJming I needs 149

PAGE 164

Table 4 6 (Cont.) Questionnaire item _I;[ Srrongl y ModerAftldly A-!ildly 1\toder-S rrongl y D1s-a/ely D1s-Agr ee a/ely Agree ag r ee D1sagree % Agr ee % % agree % % % 3 I My employee is self-5 .35 2.0 2 .0 2.0 I 8 .2 24.5 6 1.2 mot v a ted I 32. My employee relies o n our 5 22 0 2 0 0 6.1 57.1 3-U team and myself fo r fee dback. 33. M employee needs .37 16.3 18.4 2 0 20.4 34.7 8 .2 feedback about his/her performance to feel ok. 34. After the training. my 4 .9 8 I 2 3 2.3 4 7 9 3 48 8 32.6 employee felt more confident to interview chi ldre 3 5 M y employee h as high 5 .13 2.1 0 2.1 12. 8 4-U 38.3 expectations for his/her interviewing performance 36. Prior to my empl oyee 5.02 148 4 8 I 7 I 26 2 52.4 attending the training I heard the training was 'worthwhile"/valuable. I I 37. Since the training. my 5 i8 0 !O 0 17 6 47.1 35 3 employee has retained wh t I I he/she !e a rned a t the training I l Supervisors also generally had p o sitive opinions about the employees training exp erience interviewing ability organizational environments and t h e exten t to wh i ch they were invo lv ed in their employees' learning experience Like the trainee questionnaire 26 of37 items were negat iv ely s kewed making the distribution hea vily weighted towards stro n gly agree Also for 23 of37 items (62 150

PAGE 165

percent ) less than 1 0 percent of the supervisors responded in the disagree range For eleven i t ems (30 percent) no responses were in the disagree range Items with the highest means were about supervisors' opinions ofthe importance of training from a personal and agency perspective (items 11, 13, 14, 17) and the employees new sk i lls as a result of training (item 2) Items rated l o w er (items 8 19, 20 30) were regarding whether supervisors had met with their employees about applying the training to their job making a plan with co-worker discussing learning needs w i th emplo y ees and the numbe r of opportunities a vailable to the employee to practice new skills Trainees rated these items lowest as well N ext the i ndiv i dual i tems rele v ant to the hypotheses and the orga n izational environment compos ite variables were analyzed Again because of the reliab i lity anal y sis conducted one would expect to see the individual items that c o mprise the composite variab les rated similarly T he organization support variable consists of items 1 3 (mean = 5 64) 1 4 ( mean= 5 51), 21 ( m e an = 5 52) 22 ( mean = 5 36) and 26 (mean =544) Item 13 ( In m y agency top management values stafftraining") was rated highest while item 22 ("My organization supported my employee s attempts to use the training on the job" ) was rated lowe t but by only a slight margin All means for t he se it e m s were bet w een moderatel y and strongly agree, indicating tha t supervisors had a positive attitude abo t their organizations support of training 151

PAGE 166

Supervisors also were asked to provide opinions about their own support oftraining. This composite variable consists of items 6 (mean= 5.38), 7 (mean= 4 38), 8 (mean = 3 81) 11 (mean= 5 91), 12 (mean= 5.17), and 30 (mean= 3. 31 ) These items express a wider range of responses with item 11 rated the highes t ("I n my opinion training is important") while the lowest was 30 ("I helped my employee to prepare for the training by discussing his / her learning needs") Similar to the results from the trainee questionnaire supervisors express a beli e f in t raining but do not regard the activities that would exemplify good training principles as highly This may not be the result of an inferior attitude it ma y be that supervisors do not feel they ha v e the time t o engage in these activities thus, their responses eflect a practical reality rather than a negative mind set Note, the reliability anal y s is indicated a Cronbach' s alpha below .50 so the internal consistency of this subscale is not sufficient for further analysis The co-worker composite v ariable is composed of items 17 (mean= 5 67), 18 (mean= 5 3 7 ) 19 (mean = 3.48) and 23 (mean= 4 63) The highest rated item was 1 7 ( "My staff values trai ning ) while the lowest was item 19 ( "My employee made a plan with a co-worker as to how to use this training ) Once again the results indicate a general belief in the v alue oftraining but a shortfall w ith respect to i mplementing good training principles The items related to the organizational environment indicate similarity between the beliefs oftrainees and 152

PAGE 167

supervisors whi c h d o e s n ot bode well for finding significant results durin g the infe rential st a tistical testing phase 44 Comp ar iso n of S uperv isor and Trainee Questionnaire Table 4 7 presents a compar son of s uperviso r and trainee me an s for each questionnaire item A paire d sample t t est was performed for each trainee and supervisor quest i on Ta ble 4. 7 Comparison o f Supervisor and T rainee Means for Q u estionn a ire Items Q u e s t ionna i r e fle m Sup ervisor T r ain ee A l ean M ea n (N =-19) (N = 61) 1 As a result of the tr aining. I/ my employee 5 15 5 .61 4 .031 *** subst a ntially i ncreased my kno wledge on this top i c 2 V m y employee did learn new sk i lls at this 5.50 5 .85 3 745** trainin g 3. T h e trai n i n g has affec t ed m y / my emp l o y e e 4 86 5 00 503 a ttitudes c o n c erni n g th e topic area ..J.. Vmy employe e was m oti vate d to put this 5 .13 5.43 2 318** trainin g into practice on the job 5 V m employee implemented on the job at leas t -1.95 5.49 3.365** o n e c o ncept/skill that I learned 6 My s uperviso r / supporte d me in i m pleme nting 5 38 5 ll -1. 000 the training on the job 7 M y supervisor / was fam il iar with the con t ent of 4.38 4 .85 2 544* th i s t r ai ning 8 Vmy employee met w i th my superviso r to 3 8 1 3 9 2 960 dis c uss applica t i on of thi s training on the job. 153

PAGE 168

Table 4. 7 (Cont.) Que tio nnair e Item up erv i sor Trainee }dean ."vlean (N = 49) (N = 61) 9. I/m y mployee had a plan as to hO\, I would 4.17 4.31 1.171 implement tl e training. lO. Jlmy employee was able to implement the 4.74 4 66 .265 training on the job. ll. My supen isor/I thinks training i s important. 5.91 5 70 1.265 12. My supervisor/! expected me to use this 5 .17 4 80 612 training on the j ob 13. In my agency. to p management v alues sraff 5 .6 4 5 02 2.460** t rai11i:1g I .f In my agenc '.top management views t his 5 .51 4 88 2.120** training as a high priority 15. Even if no one noticed or notices. I/my 5 38 5 64 1.706* employee will use kno wledge leamed from this training on the job. 16. The training was relevant to my/my employee 5.11 5 02 -.443 job dull es. 17. M y co-workers/staff value training 5 67 5 28 -2 .293** 18. My co-worker s/staff su pporte d m y a tempts to 5.37 4 84 2.315** use the training on the job. 19. limy employee made a plan with a co-worker 3.48 3.02 .345 to use this training 20 limy employ e has had sufficient opportunities 3 9 1 3 .51 1.911* to practice the new ideas/skills/tech.r tiques on the job 21. My organizatwn Ya1ues training. 5 52 5 1 6 -1.4 73 22. My organization supported my/my employee 5 36 4.89 2.409** attempts to u se the training on the job. 21. When I had trouble implem ntin g the training 4 .63 4.43 262 on the job. 1/my employee could count on others in my agency to help me. 154

PAGE 169

Table 4 7 (Cont.) Queslionnaire Item 24 I/ my employee used the training on the job. Supervisor }.Jean (_V49) 4 56 25. The training content was consistent with my 5 .39 a gency s policies and my/my employee individual responsibilities. 26 The training content is consistent with my 5 .44 agency s mission. philosophy and goals. 27 As a result of this training Vmy employee am 5.30 a more effective interviewer. 28 limy employee attended the training because I 3. 91 wanted to. not because it was a mandatory requirement. 29 Prior to the workshop, limy employee \vas .46 motivated to attend. 30. My superviso r / helped to prepare me for the 3.31 training by d iscussi ng my learning needs 31. It is important for me/my empl oyee to do well 5 35 in my job. 3 2 limy emplo y ee rely on my team and supervisor 5 22 for feedback. 33. limy employee know inside when I do ell at 3.3 7 my job or don t do well at my job 3-l-. After the training. I/my employee felt mor e 4 98 confident to interview children. 35. I/my employee have high ex pectations for my 5 13 in tervi ewing 36. Prior to attending the training. I heard the 5 02 training was --worthwhilc"'/valuabl 3 7 Since the training I/my employee have 5. 18 retained what I learned a t the training. p::; .10. p::; 05 **"' p::; .01. 155 Trainee .Mean ('1 =61) 4 64 5 .18 5.44 5 .51 4 26 5 .51 2.66 5.90 5.26 5.44 5.30 4 97 5 .39 4 74 t 642 -i09l .443 2 089** 392 .326 -1.550 3.464*** 814 8 .31 0* 2 503** 577 1 .387 i 787*

PAGE 170

Seventeen of the 37 items had significantly different means between the trainees and supervisors while 20 items were similar enough to be not significantly different indicating hat supervisors and trainees were more similar tha different in their opinions about their training environment A closer look at the q uestionna ires shows distinct differences of opinions when items are ca tegor ized Traine es were more enthusiastic about their abilities than supervisors, and supervisors had a higher opinion of the organizational environment than d i d trainees All o f the eight items that are classified as train ee self c haracter i stics were mor e highly rated by the trainee than the supervisor se v en o fthem at a statistically significant level. In terms ofthe items related to organizational environment variables twelve of the fifteen variables were rated highe r by 'upervisors than trainees, although onl y five means were statistically significan t, and one item had equal means. Trainees seemed to think more highly of themselves, w h ile supervisors appeared to have had a better opinion of the organizat i onal en v ironment i ncluding themselves. 4.5 Results from Composite Variables For comparative purposes scale means (to more clearly relate to the Likert scale) and sums for all the composite variables are presented in Table 4 8 The Liken scale ranged from 1.0, the value assigned to strongly disagree, to 6 00, the 156

PAGE 171

valu e assigned to strongly agree The sum means were used in the subsequent stati tical analysis to increase va;-iabi lity The responses to negatively -pr esented items were reversed to compute composite scores Tab le 4.8 Descriptive Statistics of Composite Variables ComposiTe Variab l es Scale Mdn SD SumM lvf Trainee selfreported transfer 4 .6 1 4 60 95 23. 03 Trainee selfreported motivation 5.32 5 25 .47 21.28 Trainee reponed perceived ability 5 54 5 .75 50 22 .16 Trainee reported self characteristics 5 .43 5 50 .44 43.44 Trainee reported organization support 5 05 5.20 87 25 .23 Trainee reported supervi sor support 4.48 4 50 .81 26 88 Trainee reported co-worker support 4.38 4 25 .77 17. 56 Trai ne e reported environment support 4 64 4 60 65 69 67 Supervisor reported transfer 4 00 4 20 1.53 20 02 Supervisor reported trainee moti ation 4 70 5 00 109 18.81 Super vi sor reported trainee abi ity 4 71 5 25 1.28 18.82 Supervisor reported trainee elf 4 .51 5.00 133 3610 characteristics Supervisor reported organization support 5 05 5 20 87 25.23 Supervi or reported supervisor support 4.57 4 66 .75 27.44 157

PAGE 172

Table 4 8 (Cont.) Composite Variables Scale Mdn SD SumM M Superv i sor reported co-worker suppon 3 38 3.75 1.62 13.50 Supervisor reported environment suppo rt 4.53 4 .80 96 67 96 As would be expected given the individual item results the composite variables reflect a positive attitude about the trainees' and supervisors' opinions of their abil i ty, motivation, tran sfer of training and organizational environment. Both supervisors and trainees generally had a positive o pinion about their personal characteristics and their organizational environment with respec t to training. Except for the composite variable of supervisor-reported co-worker suppon, all composite variable means were above 4 0 (value assigned to mildly agree) and mo st were negatively s ewe d, indicating that they were weighted to w ard strongly agree The highest rated vari ables were those associated with the trainees' self characteristics while the lowest ra t ed var i ables were regarding the supervisors' opinion of trainees' co-worker support and the supervisors opinion of their trainees' degree of transfer Supervisors and trainees both rated organ i zation support the same (5.05) but supervisors rated their supervisory 158

PAGE 173

support slightly higher than trainees ( 4 57 versus 4.48) Trainees reported their degree oft ransfer more positively than their supervisor (4.6 1 versus 4.00) 4 5 1 Relationships between Trainee and Supervisor Questionnaire Composite Variab les Trainee and supervisors opinions from the composite variables were correlated. First, opinions about the amount of transfer between trainees and supervisors were correlated but found to be weak, (r = 282)9 and not statistically significant Other relationships between t he compos i te variables were investigated and some correla t ions indicate s i gnificant relationships between trainee and superviso r reported i t ems Correlations between trainee and supervisor-reported t ra inee variables (i.e., moti v ation and ab.lity) are given in Table 4 9 Table 4.9 Correlations betw een Trai nee and Supervisor Reported Train ee Va riable s Variable T r ainee self -r epo rted motivat"o n 1.000 T r rune e reported perceived .586** ab i lity Supervisor reported trainee .218 motivation Sup.::rvisor reported tramee 196 ab i!tt y ** p 9 P earson CorrelatiOn 1 2 3 1.000 .046 1.000 101 .699** !59 1.000 -1

PAGE 174

Strong re l atio nships exist between the trainee-reported self variables (i.e., motivation ability) The relat ions hips between trainee-reported perceived ability and motivation were pos i tive and strong (r = 586 p _01) Positive and statistically significant relationships were found bet w een supervisor reported trainee abilit_ and motivation (r = .699, p :-:::;: 01) There were no relationships betwee n trainee and supervisor-reported trai nee self v ariables This finding indicates t hat supervisors and trainees did not agree on the trainees' abilit y a nd motivation so the sup ervisor's opinions will not cross validate the trainees opinions on the traineereported se l f variables_ Correl ati o ns between trainee an d supervisor-reported organizational var iables are presente d in Table 4_1 0 Tabl e 4 .10 Correlations between Trainee and Supervisor Reported Organizational Variables Variable 1 2 3 -1 5 Tra i nee :-eponed 100 organi zat i o n al support Trainee reported .4 75*"' LOGO supervisor support cported co.335** .416 ** 1000 orker support S upervisor reported 1.000** .475** .335** 1 000 organ i za t ion support Supervisor reported .137 .186 .187 137 1000 supervisor support 6 SuperYisor repo rted co .279* .40 7** .511 ** 279 548*"' 1.000 '"orker su ort p :-:::;: .10 ** p :-:::;: 05_ 160

PAGE 175

Among the t rainee-reported organizational environment var iable s (o r gan izat ion superv i sor and co wo rker), strong positive and sta t isticall y significant relationships wer e found (o rganizatio n/ superv iso r : r = .475,p s; .01; organ iz ation/co worker : r = .33 5 p s; 0 1 ; supervisor/co-worker: r = .416 p s; 01 ) A lso strong and significant relationships were found among the supervisor reporte d orga nizati o nal envi ro ment va riables of co-worker support and su pervisor support and organization up port (co -worker / supervisor: r = 548 p s; 01; c o worke r / organization : r = .279, p s; 05) Indi v idu ally, trainees and supervisors te nd to agree in their opinions o f their environment In terms o f the co rrel ations between trai nee and supervisor-reported v ar i ables stron g and pos i tiv e c o r r elations were found between the supervisor reporte d organization support and t r ainee-reported supervisor and co-worker support (o rganizat ion/ superviso r : r = .475, p s; .0 I ; organization/co-worke r : r = 3 3 5 p s; 01). S u pervisor and trainee c o-worker support were positivel y correlat ed ( r = .511, p s; .0 1 ), as was th e relationship between supervisor-reported co-worker s upport and traine -reported organization s upport and supervisor support (c o-worker / organization : r = .279 p s; .05; co-wor k er / s u pervisor support ( r = .4 07 p s; 01) Lastly supervis o r-reponed co -worker support and s up ervisor reported supervisor s upport was positively correlated (r = 548 p s; .01). 161

PAGE 176

In terms ofthe superviso r and trainee reported relationships a perfect p ositiv e r elationsh i p was found bet w een t he trainee and supervisor reported organiz ation support variable ( r = 1.000 p:::; .01 ) Looking more closely at this v ar able, the individual items that comprise this var i able are different Because a perfe ct co r r e la t ion i s s u ch a sta t istical oddity, an i t em by i tem comparison f o r the o rganization support variable is pre se nted in Tab le 4 11. Table 4.11lndividual Ite ms Composing the Organiza t ional Support V ari a ble Q u e s t i o nnair e Item 1 3 I n m y agenc y top management v a lues st aff t rain i n g 1 4 I n m y a gency to p ma n agement v i ew s this t raining as a high p r iority. 2 My organ i zat i on v alues t raining 22 My organization supported m y/ m y employ ee a t te mp t s t o us e the t ra i nin g on t h e job 2 6 Th e tr a i ning c o nten t i s consistent wi t h my agency's miss io n philosophy, a n d goals Tra inee Q u es ti o nnaire M 5 02 4 .88 5 .16 4 89 5.44 Supervis o r Que st i o nnair e M 5 64 5 .51 5 52 5 36 5.44 Except for item 2 6, supervisors had more positive opinions about the ir organizationa l en vi ronment than did the trainees I n g eneral strong relationships existed amongst trainee a nd supervisor r eported v ariables bu t le s s s ignifican t relationships were found between t he 162

PAGE 177

trainee and supervisor-reported variables. Interestingly, when trainees and s u pervisors a re commenting on each o ther the y are not in agreement but when commenting on their organization t hey a re in complete agreemen t If trainees or supervisors agree p osit ively with on e aspect of an environmental variable, they typically agreed with the other varia bles in that category as well. Trainees and superv isor s did not generally agree on t he supervisor support v ariables but did tend to agree on the organizational and co worker environmental variables 4.6 Ag reement Regarding Vid eo tape Viewing The SIS curriculum strongly r ecommends that tra i nees view their vid e o taped interviews wit h the ir supervisors, since this provides additional f eedb ack opportunities and more fully i nvolves the supervisor in the caseworker s learning process-bot h good p r inciples for encouraging the transfer of le arning. Supervisors were a s ke d wh ether they had w atched the videotape wit h thei r trainees 8.2 p er cent of supervisors said they had while 9 1 8 percent sa i d they had no t From the trainees' perspective 3 3 percent said the y had watched the tape while 96.7 percent had not viewe d the tape Two pairs of supervisors and trainees d i d not match in thei r v iew o f whe ther they watch e d the videotape together. 163

PAGE 178

4 7 Test Scores Results Results from the pretest training and postt est are presented in this section using both the raw results and the Rasch-adjusted results The raw results are more easil y interpretab le than the Rasch results so are presented for comp arative purposes. 4. 7 1 Raw Score R e sults The research study analyzed wh ether interview skill performance as re pres e nted b y the posttest score s i mprov ed after training U sing the I ntervie w I n strume nt s ubjects were rated on 50 ke y items for a possible score of 100 Raw test s cores ( adjusted b y a dding a constant to the i nterv i ews rated by the mor e difficult rater) from the p retest tra i n ing and posttes t int e rviews a r e presen t ed in Table 4 .12. 164

PAGE 179

Table 4.12 Comparison of Pret e st, T r a inin g and Posttest R aw Interview Scores Adjusted for Mea n Di ffe r ences in Rate r s Statist i c Prete t Interview Training interview Post/est Interview Mean 60.19 87.71 78 .85 Median 61.34 87.72 80 74 Std Deviation 6.78 7 67 10 12 anance 45 94 58. 84 102.38 Minimum 42 68 .73 54 Maximum 72 34 103.731 0 98 As would be expected, most trainees had their worst performance at the pretest Resuits showed that trainees did not perform as well at the posttest as they did immediately following the training The score at the posttest rep r esents the amount of information retained from the training One would expect that the test sc ores from training to the posttest would go down somewhat over time and indeed, the mean posttest scores were lower than the training test score Caseworkers may not remember the details ofthe protocol and thus forget to cover certain items during the posttest. The test scores ho wev er did not consistently diminish at the posttest and in fact for 30 10 Bec au se of the 9 point adjustment. a score of over 100 was possible 165

PAGE 180

subjects, or 49 percent of the sample, test scores were higher at the posttest than at the training The mean p sttest score for all subjects, however was lower than the training score As evidenced by the pretest scores trainees did come to t he training with some skills, but the var'ability ofthese scores was low By the posttest the variability had increased substantially likely due to the opportunity some trainees had to use their new skills after the training while others did not have it. 4.7.2 Rasch Score Results Table 4 .13 presents a comparison of descriptive statistics using the Rasch and mean adjusted measures (scores) at the pretest training, and posttest. Ability measures are presented in logits which are the trainee's natural log odds for succeeding on items like those at the zero point on the scale defined by the R
PAGE 181

Table 4 .13 Comparison of Pr etest, Training a nd Posttest Rasch Interv i ew Scores Adjusted for Mean Differences in R at ers Statistic Pretest lnt en,iew Train in g lnt e n ,iew Posttest Interview Mean 5200 2 1665 1 5682 Median 5435 2.1467 1.61 Std Deviation 2999 5141 .6331 Minimum .26 1.21 .25 Maximum 1.12 3 .73 4 39 A v isual representation ofthe differences in mean scores is present e d in Figure 4.1. Figure 4.1 Comp a rison of Rasch Adjusted Scores for Pretest, Training, and Posttes t Sco r e DPretest Traini n g 0 P osttest Figure 4.1 clearly shows t he low er level of skill a t the pretest a significa n t increase in skill at the training test, and a decrease in skill level by the time th e postt st w as a dmi nis tered 167

PAGE 182

A paired sample t-test was performed on the Rasch adjusted pretest, training, and posttest scores, and results are presented in Table 4.14 While the use of at-test is not entirely adequate since it uses ordinal level data it does have the dist i nct advantage of comparing trainee and supervisor means on an individual basis T able 4.14 Pa ired Samples TT est for Pretest, Training, and Posttest C ompari so n G r oups Cases AI SD r Val u e ]-Tailed Probahi liry Prete s t and training sco re 60 1.65 54 23 957 000 Pretest and follow-up score 60 1.05 .6 2 13.278 000 Training and follow up score 60 .6 0 67 6 970 000 The differenc es be t ween the pretes t and training and follow-up test scores were statistically significant as was the difference between the training a nd follow-up test scores-indicating that interview performance improved between the tests It should be not ed that this represents the average tre nd in interview skill performance, however, for 49 percent of the sample skills actually improved from the training to posttest. 4.7.3 Pretest and Training Test S cores as Pred i ctors of Posttest Score A multiple re gression test was conducted to analyze the effect of the pretest and training scores on the posttest score. Results of the multiple regression 168

PAGE 183

analysis are presented in Table 4 .15. Table 4.15 Multiple Regression Results with Adjusted Posttest Score on Adjus ed Pretest Score and Adjusted Traini n g Score Var i able jJ I Adjusted pretest score 282 2.288** Adjusted training score 231 1.873 161*** *t value: p :S .10. ** p :S 05. ** F-ratio for multiple R: p :S 01. The F-ratio for multiple R was significant at the .05 level and the pretest and t raining test score accounted for 16 l percent of the variance in the posttest score. The effect for the adjusted training test score ( 282) is slightly larger than thee ect of the pretest score ( .23 i ) This analysis indicates that people who do well during the training te t and pretest are more likely to do well on the posttest and the inverse would be true as well The model was tested for multicollinearity and there was no evidenc of it (the maximum condition index was 1 0 3). Results from he hypotheses testing are presented in the next section 169

PAGE 184

4 8 Primary Anal ysis of Major Hypot he ses (Retenti o n Pre icted by O r ganizationa l Environ m ent) The hypotheses of this re search stud y were : H 1 Participant s from organizational environments the y perceive as supportive will ret ain more informat ion in the post training inte r v ie ws H2 Partici p ants with supervisors who are perceived as supporti v e will r eta i n more info r m ati o n in the post training interviews H3 Participants with co wor kers who are per c eived as supportive will ret ain more information in the post training i nterviews Retention w as the pri ma ry de pendent variable to be tested an d is defined as the amount of sk ill s r etain ed from the training test to the posttest as measured by the Interview Instrument It w as operationalized by subtracting the posttest score from the traini n g test s c ore The organizational en vi ronment variables used as predictors were o rg anizatio n sup erv is o r a nd co worker support The analysis was conduc t ed i n two s t eps. F i rs t, Pea rson s correlations were studied then a mul tipl e regression analysis was c ond ucted to tes t the major hypotheses outlined above Results from all tests a re presen t ed below 1 70

PAGE 185

4 8 1 Relationships between Retention Measures, Test S c ores and Key Va riables The retention measures were calculated by subtracting each resea rch su bject s posttest s core from the training score A retention score of zero means that f ere was no change between the training and posttest and the research subjec t r etained all that they learned in training A negative retention score means that the subject had a loss of interviewing skills while a positive reten tion score could be interpreted to mea n that the subject actually improved between the t raining and posttest Table 4 .16 presents the correlations between retention and the enviro nmental s upport variables None were significant. Table 4.16 Correlations of Key Va riables and Adjusted Retention from Training to Follow-up Variable Correlation Trainee reported organization support 025 Trainee reported supervisor support 085 Trainee reported co -worker support 132 No significant relation hips were found between the adjusted retent ion measures and the t rainee-reponed compo s i t e environmenta l variab les of organization supervisor and co-worker support L o relationships between the supervisorreported organizational environment va riables were found either so these are not 17 1

PAGE 186

presented The lack of correlations in the hypothesized direction does not bode well f o r finding significant results wi h the multiple regression a n alysis 4 8 .2 Multiple Regression Analysis of Retention with Hypothesized Predictor Variables A multiple regression a nal ysi was conducted to test the hypotheses and det e r mine t he amount of variance accounted for by the environmental variables Usi n g t he retenti n measure as the d e pe n dent variable and trainee-reported org an i zational superv i sor, and c o -wor k er support as predictors Tab l e 4 17 presents the results of th i s anal y sis Table 4.17 Multiple Regression Results of Retention on Trainee-Reported Organization Support Supervisor Support, and Co-Worker Support Var i a bl e fJ t T rain e e r e ported organization support 017 1 I 5 Tra in e reported supervisor support 162 -1045 Traine e reported co-worker upport .206 1.421 04 1 T he r atio for multiple R was not stat i stically significant and therefore the hypotheses are not supported The model was tested for multicollinearity and there w a s n o evidence ofit (the maximum condition index was 17. 5) Since trainee-reported variables did not produce significant results the sup ervi sor-172

PAGE 187

reported variables of organiza t ion superv isor and co-worker were ente r ed into the regre ss ion equation to predic t ret ention The F -ratio for multiple R was not significant at ( R2 = 006 F -ra tio: p = 962). 4 9 Secondary Anal y sis of Oth er Transfer Measures (Tes t Scores Predicted by Organizationa l Environment) Since n o significa nt results were found usi ng retentio n as the dependent variable, a seco nda ry analysis as c ondu cted with other dependent variab l es that also are rel a t ed to transfe r name iy the adjusted training test score posttest score and th e self -r eported transfer var iable While these dependent variab l es are not specifically stat ed in t he hypotheses t hey do provide a measure of tra nsfe r t hat is worthy of inv e stig a t ion The same sta tistical process was used to test research questio ns using these dependent measures; Pearson's correlations were examined an d then multiple regression t es ts conduc t ed. 4 9 1 Relationships between Retention Meas ures, Test Sco es and Se lf Reported Transfer Before delving int o the i ndividual relat i onships among the dependent variabies, relat ionships between the ret enti o n measures and the alternative dependent v ariab l es are exp lor ed. Relationships between retention measures t est s c ores and s elf-reported transfer were analyzed an d are presented in Table 4.18 173

PAGE 188

Table 4.18 Relationships between Retention Measures, Test Scores and Self R e ported Transfer Vari able Trainee self-reponed 1.00 t r ansfe r Ad j usted retention 022 fro m tr a i n i ng t o p ostt es t Adjusted t raining .267 sc o re A dju sted po ttes t 240 s co re p 0 ** p .0 5 1 2 3 1.00 -.454** 1.00 690** 331 1.0 0 T h e correlation betwee n retention and the adjusted training score was stro ng and negati v e (r = -.454 p 05) meaning that those who did better in training retained more at the postt e st but thei r retention scores were smaller, and t h os e w h o did w orse on the t rainin0 score had larger retention scores at the posttest This makes sense in that trainees who did better at training would likely do be tt er at the posttest (r = .331 p 01) so the retention score would be s maller. A lso there was a strong and positi v e relationship betw e en retention s co es and t he posttest scores (r = .690 p 01) This fact also makes sen s e since those who performed bette r on the posttest obviously retained more inf o r mation A modera t ely strong and positive relationship also was found 1 7 4

PAGE 189

between training scores and trainees w o self-reported more transfer. There was no r elat i onship however between self-reported transfer and the posttest score This is a surprising result for it indicates that self-reported transfer does not alwa ys equate with performance but there may be unintended benefits to training 4 9 2 Relationships between Adjusted Test Scores and Organizational Environment Variables Correlations between adjusted training and posttest scores and the or g anizational environment variables are presented in Tab l e 4 .19. This analysis was conduct e d to determine ift here was any relationship between the perceived organizational envi r onment variable s and the performance measures since the lat t er may subs t itute for retention as a measure of transfer. Table 4.19 Relationship Betw een Training and Pos ttest Scores for Trainee Reported Organizational Environment Variables Variabl e Tra i ne e reported o rganiz a tion support Trainee reported supervisor support Trainee reported co worker support p s .05 Adjusted T raining Score 082 278 -048 Adjusted Posttest Score 040 136 1 01 Positi v e and significant relationships were found between the adjusted training score and trainee-reported supervisor support (r = 278 p S 05) The more 175

PAGE 190

support trainees perceived from their supervisor, the better they did at the training o other significant relationships were found between the adjusted training score and the trainee-reported organizational environmental composite variables. Also no significant relationships were found between any of the tra inee-reporte d organizational environment composite variables and the adjusted posttest scores. No significant relationships between su perv isor-reported organizational environment composite variables and the training or posttest scores were found. ext, a multiple regression analysis was conducted to determine if the organizational environment variab l es would predict either the adjusted training test scores or posttest score. Results u ing the adjusted training test scores as the depend nt variable and the trainee-reported organization, supervisor, and co-worker support variab les as predictors are presented in Table 4 20. Table 4.20 Multiple Regression Results with Training Test Score on Trainee-Reported Organization Support, Supervi sor Support and Co-Worker Support Variable f3 Trainee reported organizational upport -.031 .212 Trainee reponed supervisor support .373 2498** Trainee reported co-worker support -193 -1.383 .111 F ratio for multip le R : p .1 0 ** t val ue p .05. 176

PAGE 191

The F-ratio for multipl e R is sign ific ant at w i th the adjusted training score as the dependent variable was significant at the l 0 level. 11 The organizational environment predict r variab le s accounted for 11 percent of the variance in the adjusted raining score measure Only the t value for the effect of the traineereported supervisor support was significant. The orga n iza tional environment var iable of supervisor support affected how these trai nees performed at the training Regression re sults using the adjusted posttest scores as the dependent variable are presented in Table 4 21. T able 4.21 Mult iple Regression Results with Adjusted Posttest Score on Train ee-Reported Organization Support Supervisor Support, and CoWorker Support Variab l e f3 1 Trainee reported organizational support .043 285 T rain ee reported supervisor support .131 .840 Trainee reported co wo rker support .06 1 .418 .022 1 A .10 l evel of significance was used here to more aggr essively gua rd against Type II errors that is not detecting a relationship when in fact. one does exist since the sample size. while acceptable. is relatively small 177

PAGE 192

Again, the use of the adjusted posttest score as the dependent variable did not produce significant results for the F-ratio of multiple R. Speculation about the results ofthese analyses and alternative explanations are offered in the next chapter Since retention and test scores yielded few significant results, the selfreported transfer score was correlated with key variables to exp lore relationships. Table 4.22 presents the relationships between trainee selfreported transfer and the organi zational environment variables Table 4 22 Relationships b e tween Trainee R e p o rted Transfer and Key Variable s Variable Trainee repo rted organization support Trainee reported supe rviso r support Trainee reported co-worker support < 10 ** < p-. 'p-. ) Trainee Se(f-Reported Transfer 277* .314* .344** In general trainees who reported more transfer also had higher opinions of their organiz ational environment All of these relationships were positive and significant The strongest relationship was between self-reported transfer and co-worker support and the weakest, (thoug h still significant) was between organization support and self-reported transfer (r = .277, p 05; supervisor r = 178

PAGE 193

. 314 p:::: 05 ; co-worker r = 344 p:::; 01) Trainees who percei ved their coworkers supervisor and organization as supportive reported mo re tra nsfer from t h e ciassroom to on-the-job performance 4 9 3 Multiple Regression Analysis of SelfReported Trans f er Pre dic ted by Organizational Environment Another regression analysis was performed with self -r eported transfer as the de p endent v a r iable and the environmental varia bles as the predictor variables Results are shown in Table 4 23 Table 4.23 Multiple Regression Results with Trainee Self-Reported Transfer on Trainee-Reported Organization Support, Supervisor Support, and Co Worker Support Vari a hle fJ t Trainee reported organization support 123 884 Tra i ee reported supervisor support 157 1 082 Trainee r eported co-worker support .237 1757* 165 t v a l ue : p:::; .10. **F ratio for multiple R p:::: 05 Interesting l y the environmental variables of organization support supervisor support and co-worker support accounted for approximately 16 percent of the 179

PAGE 194

variance in the trainee self-reported transfer variables The F-ratio for multiple R is stat is tically significant beyond the .05 level. The effect for the trainee-reported cowork er support ( .2 37) is larger than the effects of trainee-reported organization support (.1 23) and superviso r support (.157) but only the t value for the coworker support variable was s tatisti cally significant at the 10 level.12 Since the same i ndependent variables were entered in the model multicollinearity was not a probl em. 4.10 Results from Telephone Interviews with Research S ubjects Fourteen telephone interviews were conducted with research participants t o provi de contextual information about their training experiences organizational environments, and ability to transfer the information learned in training to t heir on the j ob performance Li ttl e variab ilit y among organizational environments 'aS found, a nd the telephone inter view s reinforce that there was little variab lity in the organzational en v ironments to d e t ect. But the interviews did shed some interesting light on the overall benefit of the training and how the training content was transferred to the j ob All te lep hone interview participants thought highly of th e SIS training and had positive opinions about support for training from their organizations supervisors, and co work er s 1 A I 0 level of s i g nific ance was used again to more aggressively gua rd against Type II errors 180

PAGE 195

Thirteen of fourteen participants said their organizations were supportive oftrain. n g, and the rema ini ng participan t thought her organization was somewhat pportive of training. Organizations ca n demonst rate support by a variet y of actions, such as allowing casew o rker s time off to attend training providing financial support, and ad ve rtising training classes on agency billboards and sending out fly ers. According t o one partic i pant h er agency director gave up her own reservat i on at a tra ining c on fere nce so the em p l oyee could attend One tra i nee said her org an ization has a clear exp ec t ati o n that she take at least eight hours oft aining a year and d oes no flinch when she takes eight hours a month. In terms of supervisor s upport thirteen of fourteen participants thought their superviso r s we re supportive oftraining and one pa rt icipant said her supervisor was somewhat supportive Sup po rtive supervisors showed their s up port of training by do i ng th i n gs l i ke helping to re l i eve caseworkers' caseloads so they could attend training req ues t ing that they not carry beepers or telephones while at t raining talking abou t training and how caseworkers will apply it in their jobs. One train ee said her supe rvi sor discusses h er training needs and brainstorms ways to apply training to her job Ano t her said that after team members return from training th e y are expected to giv e a short report at team meetings to help other figure o u t whether the y want to attend the training and to help other team since the sample size. while acceptable. is relati vely small 181

PAGE 196

members learn new information. This same person also sa i d that training is an expectation and is evaluated during performance reviews All research participants said their co-workers were supportive of training One person said that co-workers try to attend trainings together Another person said her co-workers are quick to jump in and handle a crisis so the person attending the training does not feel guilty about leaving fo r training. Four other trainees simply expressed a general feeling that their co-workers just have a good attitude about traini g and encourage it. Trainees a l so were asked how they have been able to use the training and what might have helped or hindered their ability to use it. In general, trainees said the y have been able to use the interviewing skills in a limited number of interviews, particularly sexual abuse interviews. Five trainees had used the protocol in a sexual abuse interview while the balance had not. Eleven trainees said that they applied the techniques to othe r types of interviews (such as interviews with adolescents or other family members) beyond those for which the interview protocol was prescribed ( i e interviews with latency age children and about sexual abuse). Three people said a hindrance to utilizing the training was the lack of opportunity to use the interviewing skills on sexual abuse cases with latency age children. The lack of opportunity to-use could be a result of their unit assignments (only the Intake and Sexual Abuse Team un i ts are likely to do sexual-abuse i nt erviews), or it could be that no sexual abuse cases were assigned 182

PAGE 197

to the caseworker during the interim period between the training and posttest. One p rson said her fear about he r own interviewing capabilities kept her from utilizing the training Another person said that while the interview protocol was quite helpful, real interviews with children are much more ambiguous Sexual a buse r arely presents unambiguously and explicitly ; instead caseworkers tend to stumble on things." Two people said that these interviews are conducted with law enforcement, and where there is a lack of coordination with these other agencies the interview d es not occur as presented in the training Overall research subjects were complimentary about the training the trainer and the training's releva nce for their jobs Most meaningfully one person said "I've changed the way I interview kids 4.11 Concluding Observations C lear results from the self-reported transfer variable and minimal results from the c riterio n measures of performance (i .e., retention adjusted training sco re posttest score) seem to be incongruent, but perhaps they are not. The self reported transfer va riab le and the organizational environment variables were all self-reported b y trainees, so it is not surprising that research subjects held consistent attitudes about transferring the training and about the organizational environment. The questionnaires clearly indicated a lack ofvariabi!ity in their opinions abo t their organizational environment, but there was variability in their 183

PAGE 198

test sco res Logically therefore the organizational environment va riables would not have the patency to predict the criterion measures of performance It should be noted tha t the lack of variability in attitudes towards their organizational environment could be a function ofthe questionnaire s design and lack of sensitivity rather than t he reality of the environment. It is possible that the questionnaire s imply did not ask the right qu estions to detect organizational environment variability rather the re wa not any t o discove r (this will be discuss ed more thoroug ly i n the next chapter) W h ile the univ ariate analysis pro v ided some interesting findings the hypoth ese s testing with th e main dependent variable of retent ion failed to produce sign ific an t results and the anal y sis wi t h the secondary hypotheses produced only margi nal results The ne x t chap er int erprets the results fro m the statistical testing and e xp lores alternative explanations to help explain variance in the t ransfer of training It is obv ious from t h e questionnaires and the interviews that while the hypot h e ses were n o t supported there were important unintended benefits to t ra inees for a tte nding the training. Th e next chapter also discusses some of these unintended be ne fits 184

PAGE 199

5. D i s cus si on Much data has been p r ese nted with various statist i cal tests perfor me d with significance ranging from nil to meaningful so t he next question naturall y is, what does it all mean ? The discussion no w turns to interpreting the findings and making sense of what was present and ab sent in the data analysis result s This chapter discuss es the research study findings, alternative explanations to interpret t he results that were detect ed study limitations, implications for professi o nal practice and future research When ap p ropriate the discourse re turns to the lit erature tO understand the factors affecting the transfer of training. 5 l Overvi ew of Signifi ca nt Findings The research sought answers t o two esearch questions Does training imp rove imerview ing skills perfo rma nce.? and Which organizational factors are most ilifluential in promoting the tranifer of training? As shown in the previous chapter, trai nin g does inde ed impro v e interview skills performance as evidenced by t he results on the pre tra inin g, and postt est interviews The mean score (u sing the adjusted ra w score maximum sc o re w as l 00) on the pretest was 60 1 9 rose to 87 .71 after the tra i n i ng intervention and t hen diminished to 78 85 at the posttest still greate r than the pretest leveL 185

PAGE 200

The analysis next assessed the organizational factors that influenced reten ion (the amount of skills retained from the training interview to the posttest) as a measure of the transfer of learning The research hypotheses were H1 Participants from organizational environments they perceive as supportive will r etain more information in the post-training i nte rv iews H2 Part i cipants with supervisors who are perceived as supportive will retain more information in the post-training interviews H3 Participants with co-workers who are perceived as supportive will retain more i n formation in the post-training inte r views. Th e statistical analysis did not suppon the hypotheses that supportive organizationa l environments supervisors o r co-workers result in traine e s who reta i n m ore at the p o sttest intervie w; t hus t he research fails to reject the null h y pothesis. The research did not support the portion ofBald win and Ford s (1988) model that was tested (i. e organiza t ional support as an input for transfer of learn i ng). o s i gnificant results were found using r e t e ntion as the dependent variabie a n d the organ i zational env i r o nment variables of organization supervisor and c o -worker support as predictor variables There is a danger of making a Type II err r, that the null hypothese s reall y is false, yet has not been rejected As evidenced by the responses ofboth trainees and supervisors, there was a genera l percept i on that the organizationa l environment was supportive oftraining. In 186

PAGE 201

fact t here was so little variance in these perceptions that organizational environment support for training is more of a constant than a v ariable for this group of caseworkers It is poss i ble however, that there is variance in perceptions of their organizational environment, but the questionnaires were not sufficiently sensitive enough to detect these differences due to flaws in i nstrumentation (d iscussed more thoroughly in section 5.4 .6). The multiple regression analysis wi t h the adjusted training score as the dependent variable did find that t he F -ratio for multiple R is significant at the .10 level1 3 with t he organizational environment predictor variables accounting for 11 percent of the variance of the adjusted traming score measure Trainees with upport from their organization supervisor and coworkers did better at the t r aining test Only the effect for the trainee-reported supervisor support, however, was significan t Trainees w ho perceive their supervisors as supportive did b ett er at the traini ng test than other trainees Perhaps this is because trainees with supportive supervisors go into the tra ining with an expectation that they will get more out of it t han do workers with iess supportive supervisors. The F-ra t io for multiple R was not significant when the adjusted posttest score was used as the dependent variable and the organizational environment variables (o rg anization, superviso r and co-worker) as predictors l3 The !0 level was again used here to m i nimi z e Type II errors and ensure that relationships that do exis t are detected 187

PAGE 202

The organizational environment predictor variables did not provide any clear r esults u ing any of the criterion measures of performance as the dependent variable. But other data gathered f o r the dissertation indicated that the training was a w orthwh i le endeavor for the caseworkers who attended the training and some variance in the trainee self-reported transfer composite variab le could be explained by the organ i zatio nal environment variables The organizational envir o nment v ariables of organization support supervisor support and co-worker supp o rt accounted for approximatel y 16 perc ent of the variance in the trainee self rep o n e d transfer v ariable and t he F-ratio for multiple R is significant at the 05 leveL Of these variables, only co-worker support had a significant t value This would ind icat e that research subjects had a fairly consistent positive opinion of their organization and supervisors but opinions about co-workers varied, hence this composite variable w as the only s i gnifican t v ariable in the multiple regress' on anal ys is These findings do provide some su pp ort for the Baldwin and Ford model of transfer Also the raw opinion of research subjects (t he self-reported transfer measure) did show that trainees benefited from train in g ; the mean score for the selfr eported transfer var able was 4 6 I (between mildly and mode r ately agree) des p ite the relatively low number of sexual abuse interviews conducted by research subjects (an average of l.66 interviews between the training and posttests) During the telephone interviews with a select number of resear ch 188

PAGE 203

subjec ts, i t was clear that they ap plied t he skills learned a t training to other types of n t erviews or interactions with their clients. This transfer phenomenon is called "fa r t ransfer by Clark and Voogel (1985) and high road transfer by Salomon and Perkins (1989), and provides supplementary value to the training be y ond what was originally an ticipated Yan cey and Kell y (1990) postulated that tra inee s exaggerate their post tr aini n0 performance because of a strong need to justify t heir actions and the findin g s oft his study wo uld support this argument. Trainees self-reported a significant amount of transfer but this did not correlate with their skill level as measured by t he cr i ter io n measure of performance (Inte rv iew Instrument) A s ignificant re lat i onship was found bet een selfreported transfer and the number of s xual ab u se o r inves tiga t iv e interviews conducted (sexual abuse inte rviews: r 55, p::::; .01; in ves tiga t i v e interviews r = .464 p::::; 01), suggesting (as other re earchers have fo n d for instance Ford Quinones Sego & Sarra 1992 ) that t h e opportu nit y t o use skills ha s a n otable effect o n transfer a work en viro nment characteristic i n the Baldwin and Ford model. 5.2 Alte rnative Explanations for Transfer of Training Since the research results did not support the hypotheses the next ques tion becomes, w ha t variables do account for differences in transfer ? In orde r to provide some information to future researchers, and since data on variables other 189

PAGE 204

than the organizational environment were collected, alternative explanations were explored t o detect variances in any of the transfer measures as represented by the four dependent vari ables of I Adjusted re tention m eas ure 2 Adjusted training score 3 Adjusted posttest score 4. Self-reported transfer composite variable The selec t ion of pr ed ictor varia bles to enter into the multiple regression equation w as based upon the lit eratur e re v iew especially Baldwin and Ford's ( 988 ) framework for understanding transfer of learning According to that framework transfer is contingent upon the interaction oftrainee, training design, and org aniz a tional environment fa ctors including the opportunity t o use the trai ing Training design did not var y for this study and res earch subjects were quite satisfied w ith the training as e vi denced by train i ng satisfaction survey resu l ts sho wi ng s li tie v ariability in responses that training design variab l es are precluded from further anal ysis. Opportunity-to use is measured by two var i ables-numb e r of investigative interviews conducted and job type. The form e r is from the Trainee Follow-up Questionnaire and is operationalized by the number o f investigative interviews th at subjects reported t hey had conducted since the training Research subjects completed a relatively small number of sexual abuse interviews but more investigative interviews so the latter was used as 190

PAGE 205

the predictor va riable ins t ead of the number of sexual abuse interv iews. Resea r ch subjects unit assignment may also affect whether they have the opportunit y to us e the tra ining The unit variab l e was dummy-coded into jobs that may have the potential to use the trai ning and th ose that likel y w ould not offer the opportunity to cond uct sexual abus e interviews Another con s i dera t ion is the level of skill t rai nees bring into the training as measured by the adjus t ed pretest score and whether the lengt h of time between the training test and posttest influenced performance so thes e variables a lso will be entered into the equations to assess for significance. Therefore pr edi cto r variables to be used in the explorato ry anal yse s with the four dependent variables are : 1 Tra i nee report ed environment sup port 2. Trainee reported self characteris tic s 3 Numb er of inv es tigative interviews conducted 4 Job p r o v ided o p portunity to use 5 Time be tween t rai ning tes t and posttest 6. Adjusted pretest measure As wi th the p r ev i ous chapter, corre latio ns between the dependent va riabl es and predictor variabl es will be presen t ed then multiple regression results next 191

PAGE 206

5 2 1 Relationship between Adjusted Retention and Predictor Variables Correlations between the predictor variab les and adjusted re t ent ion are presented in Table 5.1. Table 5.1 Correlat ions betw een A d j usted Retention and Predictor Va riables Vari ahle Correlation wit h Adjusted Retention from Trai ning to Posttest Trainee reported environment 0 l2 support Trainee reported self characteristics 020 Number of investigative interv i ews 032 conducted Job provides opportunity to use 114 Time between training and posttest 113 Adjusted p r etest measure for rater 110 o correlations were significant A multiple regres sio n analysis was conducted with the same predictor va riables and, as would be expected given the 192

PAGE 207

poor correlat i ons, results were not significant (R2 ; 031, F-ratio for multiple R p = 941) and so are not presented 5 2 .2 Relat ionship between Adjusted T r aining Test Score and Predictor Variables The next dependent varia ble analyzed was the training test score and it was correlated with the same predictor variables Results are presented in Table 5.2 Table 5.2 Correla t ions between Adjusted Training Test Score and Predictor Variables Variable Trainee reported environment support Trainee reported self characteristics umber of investigative interv ews conducted Job provides opportunity to use Time between training and posnest Adjusted pretest measure for rater p ** p Co rrelation w ith Adjusted Tra in ing Test Score .161 .284** 295** .085 105 215* 1 93

PAGE 208

A strong and pos i tive relationship exists between the composite variab le of trainee self characteristics (i e., ab i iity and motivation) the number of inve s tigat i ve i nterviews conducted and the adjusted pretest measure A closer exam i nation ofthe composite var i ables making up this macro composite variable (train e e reported self cha r acteristics) also yielded significant relationships Pos iti v e and significant relationship. were found between the adjusted training sc o re and trainee reported mot i vation and trainee reported per ce i ved ability The trainees who scored higher on the training test had higher opinions of th eir ability and motivation ievel conducted more investigative interviews, and had a higher score on t he pretest Next a multiple regression analysis was performed with all the predictor va r iables to test the extent to which v ariance was accounted for by this model. Regression results are presented in Table 5.3 194

PAGE 209

Table 5.3 Multiple Regression Results with Adjusted Training Score and Predictor Variables Variable fJ t Trainee reported environment support 1 01 .669 Trainee reported self characteristics .146 939 umber of investigative i nterv i ews 323 2 040** conducted Job provides opportunity to use 199 -1.253 Time between training and posttest 108 835 Adjusted pretest measure for rater 173 1.312 .192* F -Ratio for multiple R p 5:.10. ** t value: p 5:.05. This model predicted ninetee n percent of the variance in the adjusted training score Only one variable, number of investigative interviews conducted had a signifi cant t va lue however The adjusted train ing score was collected at the SIS training, before trainees would have had the opportunity to perform investigative interviews but the adjusted pretest measure for rater was not significant yet was stronger than the other va riables Perhaps these results speak to the level of skill and confidence of workers who have jobs that could potentially allow them to conduct investigative (and more specifically sexual abuse) i nterviews or maybe it is because these workers are more willing to try the new skills on-the-job 195

PAGE 210

5 .2 3 Relationship between Adjus ted Posttest Score and Predictor Variables Next correlations between the adjusted posttest score and the same predic t or variables were performed Results are presented in Table 5.4. Table 5.4 Corr elations between Adjus ed Posttest Score and Predictor Va r iables Vari able Cor relation with Adjusted Pos'ftest S cor e T rainee reported environment support 119 Train e e reported self characteristics .252 1 u m ber of investigative interviews 274** cond u cted J ob pr ovi des opportunit y to u se .189 T i m e betw e en t rain i ng and pos est 0 3 4 Ad j usted pretest measure for rater .291 p .s-.1 0 ** p .s'. 05 Th e same v ariable s ( t rainee r eported self characteristics n umber of investigati v e i nterviews conduct e d a nd adjusted p retest measure for rater) that had s trong a n d positive relationships with the adjusted tra inin g test score also had stron g and positi v e relationships with the adjusted posttest sco r e Once again a closer scrut i ny of the var i ables o f the macro composite variable train ee reported self characterist i cs found that only trainee reported perceived ability was 1 9 6

PAGE 211

significant (r = 242 p 1 0) while trainee reported motivation was just over the .1 0 significance threshold 14 ( r = 206 p 112) A multiple regression analysis was performed next with the adjusted posttest score as the dependent v ari a ble Results were not significant using all the predictor variabies at the 1 0 level15 ( R1 = 061, F -ratio for multiple R : p = 151 ), so are not presented 5 .24 Relations h ip betwee n Trainee S elf-Repor ted Transfer and Predic tor Variables L astly correlations between the trainee self-reported transfer composite variable and the same predictor variables were explored Results are presented in Table 5.5 Table 5 .5 Correlations between Trainee Self-Reported T ransfe r and Predicto r Variables Variable C orrelation with Trainee Se!f Reported Transfer Tra inee repo rted environment support 392*** Trainee reported self characteristics 617* Number of investigative interviews .464 conducted 1 A threshold of. 10 was used here to again more aggressively guard against Type Il errors 1 .10 used as the outside l eve l of significance to more agg ressively guard against Type II errors and detect any relationship that may exist. 197

PAGE 212

Table 5 5 (Cont.) Varia ble Job provides opponunity to use Time between t raining and posttest Adjusted pretest measure for rater *** p .:{_01. Co rrelation with Trainee Self Reported Transfer 491 *** 048 174 The correlations between trainee selfre ported transfer composite va riable and the predictor variables were much stronger than with the other dependent variables All of the predictor va riables which origina ted from the Trainee Follow-up Ques tio nna ire (trainee r e ported en v ironment support trainee reported self c h aracteristics number of investigative interviews cond uct ed job provides opportunity to use) were positively and significantly related to trainees' perceptions about the skills that were transferred to on-the-job performance Trainees' perceptions oftheir ability and motivation were most significantly related to transfer, indicating tha t traine e s with higher reported levels of m tivation and ability also applied the training more than those with less motivation and ability Also trainees whose job assignment (either Intake Sexual Abuse, or General units) an those who actually reported conducting in ves tigative interviews had higher levels of self-reported transfer. Self-reported transfer also 198

PAGE 213

correlated with organizational environment support, indicating that trainees who believe they have more support from their organization supervisor, and co workers also believe the y transfe r red more skills to on-the-job performance The nex t level of analysis looked at the individual composite variables of the macro composite va riable "trai nee reported self characteris ti cs ." The strongest relatio nship was between trainee self-reported transfer and motivation (r = .643,p 01) Trainees who w ere motivated to utilize the training also reported more ransfer Also a s t rong and positive relationships was found between trai nee perceived ability and transfer (r = .462, p 01 ). Though still significant the weaker re l ationships were between self-reported transfer and the organ i zationa l environment variables (organ ization ; (r = 277, p 05; supervisor : (r= 05 ; co-wo rker(r= 01) Finaliy a multiple regression analysis was conducted Results are presented in Table 5 6 199

PAGE 214

Table 5 6 Multiple Regression Results with Trainee Self-Reported Transfer and Predictor Variables Variable f3 Trainee reported environment support 085 .736 Trainee reported self characteristics .449 3.744*** 1 umber of invest" gative interviews .209 1.715* conducted Job provides oppor.unity to use 220 1.798* Time between training and posttest 058 585 Adjusted pretest measure for rater 012 .113 .521 *** t value p .10. ** F-ratio for multiple R/ t value: p .01. This analys is produced the most sig nificant results of all the tests conduc ted. The predictor variables accounted for 52.1 percent of the variance in trainee selfreporte d tra nsfer and the F -ratio for multiple R is sig nifi cant at the 001 level. Of the predictor variables trainee reported self character i stics that consists of all the items related to trainee ability and motivation had the strongest effect and the t v alue was sign ificant at the 001 leveL Also s i gnificant were the t val u es for the variables "j ob p ro vides opportunity to use" and number of investigative 200

PAGE 215

interviews conducted," and significant at the 1 0 level16 T values for the other variables were not significant and in fact, when they were removed from the equation, 51.3 percent of variance was explained by the remaining variables. So the variables oftrainee reported environment support adjusted pretest measure for rater nd time between training an d posttest account for only .9 percent more variance in the self-reported transfer variable when entered into the equation These results lead to the conclusion that trainees motivation to use the training, perce iv ed ability at applying the skills and the opportunity to use the training back at the job were the most salient variables for achieving percei v ed transfer of training for this group of caseworkers Organizational environment support as defined by organization supervisor and co-worker support did not seem to affect trainees perceptions ofwhether transfer occurred Because the opportunity to use the training is an organizational environment characteristic the implication is that if transfer of training is to occur training must be relevant to caseworkers' job responsibilities and they must be able to use the training once they return to their jobs W hile this would seem to b ommon sense the supervisors in this study did not adhere to this basic tenet of training management (i.e., selecting the most relevant training fer the trainee s job responsibilities) as evidenced by 1) the fact that subjects had little opportunity to use the training because of their unit 16 A threshold of .lO was used again to more aggressively guard against Type II errors to maximize the ability of the test to detect a relationship 201

PAGE 216

assignments and 2) the lack of sexual abuse interviews conducted by research subjects during the approximately three month period following the training test No significant relationships between supervisor-reported composite var a b ies and the training or posttest s cores were found ; therefore these results are not presented. 5 2 5 Differe n ces in Opportunity to Use Another question that arises from the research, especially given the results fro 1 th e last analysis, is whether individuals who had the opportunity to conduct inves t igative interviews performed better during the training test and posttest and retai n ed mo re skills at the posttest The median number of investigative i nterview s conducted was three, so research subjects were divided into two groups (no or rare opportunities to use the training versus multiple opportunities to use) Table 5 7 presents a comparison of their test scores and retention level. Table 5. 7 Comparison of Mean A djusted Training Test Scores and Retention by Opportunity to Use Group Subjects with opponunity to use (N = 28 ) Subjects with little or no opponunity to use (N= 3 2) Adjusted Training 'core 2 24 2 09 Adjusted F o llow-up Score 1.72 1.41 202 Adjusted Retention from Training to Follo w 52 68

PAGE 217

This information is visually represe nt ed in Figure 5 1 Figure 5.1 Graphical Representation Comparing Opportunity to Use 2.5 2 1.5 1 0.5 0 -0 .5 -1 .. II!III!BI!BIIBIIIIR!!I"' Ad. Training Score Adj. Posttest Retention Score Score D Opportunity to Use No or Little Opportunity to Use An i n dep endent samples t -tes t was performe d on each group Only the posttest scores exhibited significantly different means (p ::; I 0) with a t val ue of 1 92 and actual sig nificanc e level of 059 Resear c h subjects who had the opportunity to use skills learned at the tra ining once they got back to their jobs scored higher on the training and posttest, with a statistically significantly higher score at the posttest. The opport unity to use v ariable is considered a component of the organizational environment, sinc e the organization does or does not provide opp ortunities to practice. These results imply that administrators and supervisors should select trainees for specialized training based upon their potential to actually u se what is l earned in train i ng on the ir job To do otherwise would be to create unfair expectations of an employee setting them up at the very least for 20 3

PAGE 218

frustration and at the most for failure, not to mention the wasted trainin g time and investment for management. This finding also supports a work environme nt characte r istic ofBaldwin and Ford' model. While the main hypothese s wer e not supported th e find ings do suggest managers' ability to intervene in at least one aspect of the work e nvironment opportunit y to use the training back at the job 5 2 6 S ubjects' Emotional Reactions While the original hypotheses were not s upported it would be difficult to conclude fr om th i s analysis that the organi za tional environment did not have any effec t o n retention. However other va riables may have had a more significant effect o n skill retention and thus transfer in this study For instance trainees may been mo re moti v a ted to d e velop their skills because of the relative imp o rtance of doing thei r j obs mo r e effectively It is not an e x aggeration to say that children's lives could literally be saved b y conduc t ing effe tive i nter vi ews and the consequence of not con du cting effect ive interviews could be horrific ; children could b e e moti o n ally and physically hurt or even killed because caseworkers were not able to eli c i t information from the interviews that could be used to protect the children It is p o sible that trainees who had organizational environme nts t hat we r e no t supportive of training w o uld not have att ended the tra i nin g much less volunteered to participate in the r esearch study 20 4

PAGE 219

Perhaps it was of the sting of the feedback received at the training and on the Interview Instrument sent back to trainee that resulted in performance improvement and thus transfer. Research subjects may have learned from their mistakes because they were vividly pointed out to them Inwardly vowing never to make the same mistakes again the trainees improved their skills This may also b e the reason why some research subjects did better on the posttest than on the t r aining test ; the tra ini ng test itself was in fact an effective learning expenence It may be that the most authentic value of training is emotional-the training gve s caseworkers confidence to i n t ervi e w more effectively with children and famil y !11 mbers, and therefo re they serve their cli ents better. While not an i tended result the outcome is still the same since families and children are better served lt in no way dimini hes the importance and need for training but it would simpl y shift how training is marketed to potential participants or more dramaticall y in a revision ofthe training focus (e.g., develop interview confidence rather than interview s kill). It may be interesting and productive to assess for th e emotional i mpact of training in future studies 5 2 7 Other Types of Transfer are Occurr i ng Other researchers have said that training could lead to far transfer (for ins ance, Clark &Voogel 1985) and results from this study would support that 205

PAGE 220

far transfer is occurr ing The mean score for self reported transfer is 4 .61 (closer to moderately agree ), yet on av rage only 1 66 sexual abuse interviews and 6 55 investigat"v e interviews were conducted by trainees during the period following the training (ave rage of 13.49 weeks) Also during the qualitative interviews research subjects assert that they used he trai ning for other types of interviews and interviews with the family indicating far transfer. 5 2 8 A Consideration of Culture Cu lture is understood and operationalized in many different ways (Sm i rcich 1983) At its most basic and popular conceptualization the core of organizationa l culture is the shared va lues of its members (Peters & Waterman 1984 ) Relevant to this research this means that all the caseworkers and superv i sors placed a high value on the importance of developing th eir casework skills a nd using training t o better serve families and children These shared values produce normative pressures on members ofthat organization to conform (Wiener & Vardi, 1 990) Becau e of th ese values caseworkers would feel pressure to attend training and apply it when they return to the job. More evolved definitions discuss c u lture and the effect on individual behavior as both an active participant and a function of the forces within the culture (for instance Marcoullides & Heck 199 3 ; Meek 1988; Smircich, 1983). In this schema caseworkers also would 206

PAGE 221

contribute to the overall culture regarding training by actions such as ph ysica lly a n e nding the training and con v eying a positive reaction to it by their behavior. According to Schein (1990) culture manifests itself at three levels 1) obse rv able artifacts or what can be s een touched or heard ; 2) values; and 3) assumptions that underlie the collect i ve unconscious and determine p erceptions thought processes feelings and behavior. Support for training and thus transfer would be expressed in artifacts such as flyers advertising upcoming trainings on the a gency bulletin board in vaiues for attending and using training and in assumptions that training is a necessary condition of employment and ca s eworkers should continually seek to improve their skills to better serve families and children. Another perspective looks at the significance of professional culture Ac c ording to Bloor and Dawson ( 1994) the organizational culture and the profess i onal groups w th in them interact to influence the nature of the culture A professional group is an 'occupational group which provides an exclusi ve and essential client service in v olving the discretionary applicat i on of specialized know l edge" (Bloor & Dawson 1 99 4 p 282) Child welfare caseworkers certainly qualify as such a gro up The professional group becomes the primary reference gro up with a distinct pattern of values beliefs, norms, and interpretations for judging behavior. The professional culture is assimilated into t he organizational culture (B loor & Dawson 1994) It may be that the 207

PAGE 222

professional culture of being a child welfare caseworker may have contributed to the overall positive regard for training and the attending support for it. Curry's (1997) study supports the contention that the professional culture of child welfare caseworkers is generally supportive oftraining; as he found that less that less than eight perc ent of participants thought they had an unsupportive organizational envi r onment. In a normative-instrumental framework for understanding the influence of cultu re, a caseworker's transfer of training is a function of his or her intention to perform that beha vior. The beh aviora l intention in turn is determined by two basic factors : 1) the attitude caseworkers have towards performing the interview protocol and 2) the subjective n rm, i.e., perception of the totality of the normative pressures concerning the whether the interview protocol is performed The first component i e the per on's attitude toward performing a particular act, is a fun ction of beliefs concerning consequences of performing th e interview protocoi and its va l ue to that caseworker ; these are referred to as instrument cognitive beliefs. The second el ment the subjective norm is a function of the caseworker's beliefs about what important referents such as administrators supervisors, or co-workers think he or she should do weighted by motivation to comply with these referents (Wiener and Vardi, 1990). Based upon this framework culture can be manipulated to encourage transfer of training 208

PAGE 223

Some researchers treat culture as something an organization has (for instance Peters and Waterman, 1982 Wiener & Vardi, 1990), while the other perspective treats culture as something an organization is (fo r instance, Smircich 1983) There are attending repercussions for behavior depending on the view adopted (Meek 1988) Prior to conducting this study, the author viewed culture as a variab l e to be manipulated by management The results from this s tudy, however indicate that such a view may be naive since it would appear that the organizational culture towar ds training is fairly entrenched and, while it is a positive trait of the organ izationa l culture, it may not be as easily manipulated as first thought Results from the follow-up survey to trainees and supervisors as well as the qualitative interviews indicate that the regard for training fits with any of these perspectives on culture, whichever is chosen as the standard. Because of the overall positive opinions of researc h subj ects and their supervisors about their o r ganization, supervisor and co-worker support, it would appear that support of and belief in the value of training i a constant fea ture shared by all the research subjects, rather than a var iable thus leading to the conclusion that training is a core v alue ofthe research subjects organizational culture(s) It was thought that the wide representation of county departments of Social Services would result in an equally wide ra nge of different perceptions about their organizational culture. Twenty-two different county departments ofvastly different sizes from 209

PAGE 224

Colora o s largest county, Denver, to smaller counties such as Kit Carson were represented in the sample. If this dissertation s postulates were correct this should have resulted in w id ly different attitudes towards training But Colorado is a county -adminis tered system with state oversight and perhaps this state system which sets policies and mon i tors standards, ha s more influence on fosteril!g an overall positive attitude towards training than otherwise thought In stead, the organizational environment v ariable wa s a c o nstant feat ure of t rainees and supervisors opinions, l e ad ing to the conclusion that t he local county department of social servi ces organizational cult ure is conducive to and possibly even o riente d toward t raining as an impo rtan t feature of organizational life Ir. a large study of child welfa re wo r k ers, Vinokur-Kap la n ( 1986) also found w idespre ad suppon fort aining, as did Curry (1997) in a smaller but still ignificant study on ch ild we lfare worke rs attitudes towards training Both of these researc her s found l ittl e variabili t y in the attitude towards trainin g but did find variance i n the beha v iors that a r e indicative of a su p p o rtive environment Such finding s i n dica te that the professional culture may be supportive of trai nin g on the surfac e but actual behaviors do not always match the attitude Gregoire et al s ( 1998) s tud y disp e nsed with using opinions as indicators of a supp ortive environment and went directly to beha vio r s w hich they found were no! indicative of transfer f riendl y orga niza tional environments 210

PAGE 225

The professional culture of child welfare caseworkers may also have more influence than previously considered Some force such as the normative instrumental framework the norms of the professional culture, or any other framework for understanding the role of culture was operating in this situation to create an environment where training was sought encouraged, promoted and valued Perhaps it is for this reason that transfer of training had a better opportunit y to occur. While this discussion is an interesting digression it is speculative since the stl dy was not designed to assess or even understand the larger organizational culture towards training Nonetheless, the study of a training culture is fodder for further discussion and research 5 2 9 Summary of Alternative Explanations Some significant results were found when add i t ional predictor variables were ana yzed to examine differences in t he dependent variables In 1988, Baldwin a n d Ford stated that transfe r is contingent upon multiple factors and this research project would support that contention For this research study, self reported transfer is dependent primarily upon intrinsic trainee factors of their motivation perceived abi.ity and actual skill level as measured by the pretest the organizational nvironment factor of opportunity to use the training affected transfer. From the seemingly ubiquitous support training receives in the counties departments of social services it may a l so be a promising 211

PAGE 226

lead to explore the role of culture for facilitating this environment These significant r esults present promisin g leads for future researchers to build upon 5 3 Consideration of the Relevant Literature W h i le this stud y did not s upport the contention that the organizational env iro nmen t as defined by organization supervisor, and co-worker support influenced the degree oftransfer from the classroom to on-the-job performance other research studies did (for instance Baldwin & Magjuka 1997 ; Baumgartel & Je anapi e r re, 1972 ; Beudin 1987 ; Brinkerhoff & Montesino 1995 ; Burke & Ba l d win, 1999 ; Carne v ale Ga i ner & Villet 1 990 ; Facteau Dobbins Russell L add. & K t dis c h 1 995; F leish m a n 1 9 5 3; Ford Quinones, Sego & Sarra 1 9 92 ; G re goire, Propp & Poertner, 1 99 ; H and Richards & Slocum 1973 ; Buczynski & Le w is 1980 ; Kozlowski 1 997; Lim 1999 ; Miles 1965) Findings rele v ant to this r es e arch study an d i n fact contradict the dearth of significant findings in this re ear c h a re explo r ed next Gregoir e e t a l s ( 1 99 8) st udy a s sessed the supe rvi sor s role in transfer and foun d t ha t c ertain beha v iors by super v isors were associated with trainees ability to t ransfer. Few supervisors h o wever were engaged in the behaviors associated w ith impr o v ing training t ransf er. Like this study the supervisors say it i s important b u t f all short of actuall y engag i ng in the behaviors that are more likely to r esult i n transfer. Curry ( 1 9 97) also found that transfer-related behaviors often 212

PAGE 227

fall short ofthe generally positive opinions regarding the overall training and transfer environment as research subjects gave h igh marks for overall attitudes but low ratings on behaviors which manifest good transfer principles His study found that the most salient fac tors leading to transfer were tra ini ng relevance tra ner adult learning and transfer trategies, and perceived learning To a lesser extent, organizational environment factors pre-training motivation and prior experience with the training were associated with transfer. Brinkerhoff and Montesino's study (1995) found that trainees with supportive manage rs who engaged them in a pre -trainin g expectations discussion and an after-training follow-up discussion had increased transfe r and a more positive perception regarding the forces that encourage transfer of training Burke's (1999) study found that relapse prevention strategies taught during the training were most effec.ive in unsupportive work environments and as the transfer climate became more supportive, there was l ess need for the relapse prevention strategies since favorable work climates were conducive to transfer of learning Facte au et al. (1995) also found that the organizational environment affected pre-training motivation and training transfer. Rouiller and Goldstein ( 1993) determined that the organizational transfer climate influenced the degree of learned behavior transferred to the job In an effort to duplicate Rouiller and Goldstein's findings Tracey, Tannenbaum, and Kavanagh ( 1995) also found similar results i.e. that the transfer-oftraining climate and continuous-learning 213

PAGE 228

culture had direct effects on transfer I n their conclusion, they call for examining the influence o f the work environment on individual behavior for instance self efficacy and moti vation This study s resu l ts support the call for such research since trainee characte r istics were most influential in determining transfer outcomes y et t he link between these characteristics and the organizational env i ronment was not established Othe r stud i es re sear ch ing both trainee characteristics and organizational environment had mixed resul ts identifyi ng the factor that was most important for predictin g transfer of trainin g Tesluk Farr, Mathieu, and Vance ( 1995) found that the organizational environment did predict the extent to which training was generalized to the c ore job acti ities In their study, individual trainee characteris t ics such as attitudes and experience (25 percent ofvariance e plained) we r e more predictive than the organizational environment factors (6 percent of var ia nce ex plained ) B ennett, Leh m an and Fors t (1999) did find that a supportive transfer and change climate led to improved reports of transfer. Their results also indicated the impo rtance of individual empl oy ee attitudes in transcending though not necessar ily er a dic atin g, a poor r a ns er climate S imilarl y the Ford et a!. ( 1 9 9 2) study found that n egative work envir onment factors could impede subjects opportunities to perform but t hese could be mitigated by individual factors such as self-efficacy Machin and Fogarty (1997) studied individual trainee characteris t ics and work environment characteristics and found that self214

PAGE 229

efficacy and motivation to transfer were re l ated to transfer intentions No signific an t results, however, were found between the work environment charact eris t ics and transfer intentions In cont ra st Foxon ( 1994 ) found that manager support for the utiliza t ion of the skills learned at training was more important than individ ual trainee motivation on the transfer process and outcomes. Some s tud ies assessed for the organizational environment but did not consider tra inee characteri s tic s in th study design. For instance Xiao's (1996) study did find suppo rt for the importance of the o rgani z ational envi ro nment for influencing t ransfer but did not assess the effe ct of indi v idual characte r istics Also Lim's study ( 1 999) did not con ider t he trainee characteristics in the study des ign; he came to regret this oversight since the results led him to conclude that trainee charac ter istics could have affected outco mes He also found that th e opportunity to use the trai n ing and relevancy to subjects jobs was a ma j or determinant of perceived learn i ng T h e findings fro m this study rega r ding trainee characteristics support those ofFord, W eis sbein S mith, Gully, a n d Salas (1998) who found that trainee characterist ics of self-efficacy and mot iv ation were related to increased levels of transfer Gist et al. (1991) a l so researched t rainee characteristics and determined t hat trainee s who a re more confident a re more likely to have i mpro ved performance ( a facto r that was acco unted f o r this st udy's variable trainee self 215

PAGE 230

characteristics) Other researchers who found that motivation also resulted in increased transfer included Burke (1997), H i cks and Klimoski (1987) and Machin and Fogarty (1997). Support for the re sults from this study a r e found in the literature knowledge base but are mitigated by the limitations inherent in the study, which are discussed in the next section 5.4 Study Limitations Several stud y limitations ma y in terfere with the interpretation and g e n e r alizability of the study. CDHS originally commissioned the study to assess empirically whether training improved worker skills. The cost of the two year evaluat i on conducted by American Humane A sociation was over $80 000 and included the expen e of project design, p roject manage ment pretests posttests ad itionallodging for research subjects trainer compensation actors to role play the child victim facilities fees curriculum development for the Advanced training, ince ntives for research subjects, data entry and analysis, as well as other items The evalua t ion expenditure does not include th cost of the SIS training, videotape rating or othe r costs incurred by DU-GSSW since t hese were paid to them directly b y CDHS. The evaluation exp end i ture represents a fairly substantial investment by any training evaluation standard The addi tional cost to corre c t for inherent study limi t ations such as more research subjects (and therefore, more posttests train ing s actors lodging etc.) was prohibitive For 216

PAGE 231

discussion purposes, however specific limitations are presented in the following sec t ion 5 4. l Selection Bias Due to ethical and management considerations, it was not possible to mandate participation in the study so subjects self-selected into it Potential research subjects were persuaded to participate by two letters, follow-up telephon calls, and were further enticed by a $30 dollar gift certificate It is not known whether the coercion, i ncentives or none of these efforts (perhaps a magnanimous belie f in the value of management research) enticed research subjects to participate who otherwise would not have done so Also, it is unkno wn w hether the organizational climates, SKills or motivation of those who did participate were different from those who did not participate It is possible that individuals who accepted the $30 gift certificate or participated because they saw an oppo rtunity to have another day away from the office are different than those who declined to participate 5.4.2 Time Frame The time frame between the t raining and posttest ranged from 9 to 19 weeks and therefore could have influenced retention, in that more elapsed time could have reduced the level of skills retained Alternatively, the additional time perio d could have provided more opportunities to practice thus improving 217

PAGE 232

retention A correlat ion analysis between time frames and t he dependent variables of re t ention adj u st e d posttes t score an d self -rep orted tra ns fer, however, did not yield any significan t rel at i on hips. As Ellis (1965) found i n his research for this sample the time per i od between measurements was incon sequential regarding transfer. 54.3 A dditi onal Training or Research The Trainee Follow -up Questio nnai re asked questions to determine if intervening variables (suc h as additional trainin g or additional reading) ma y have affected scor es Mor e training or addit ion al reading could influence the amount of information retained Resea rch subjects however reported minimal additional training or research that may h ave i nfl uenced outcomes 5.4.4 Vide ota p e Rater s The R asch and means ana l ysi s made it app ar en t that videotape rati ng b y the raters was in on sistent result i ng in significant differences in scores An attempt to correct th e se errors was made but it was a standardized correction (rather than indi vidual) so e rror ma y have spread across the entire sample Also the v ideotap e raters did n ot rate the videotapes bli nd ly in t ha t research subjects were clearly differentiated from those i ndividual s who d i d not participate It w a s obvious from the videota p es whether an i nt erview was a pretest o r a posttest since the interviews w er e conducted in a diffe rent setting than the training test, and the 218

PAGE 233

videotape raters saw multiple vid eotapes with the same person conducting an interview with different actors These issues, however could not be avoided and are inh erent limitations ofthe study 5.4.5 Test Conditions Reactive effects occur when a pretest increases the sensitivity of the trainee to the training as he or she may have a heightened awareness of content within the training and thus perform better on the training and /or posttes t (Campbell & Stanley, 1963). All trainees in the study, however, were exposed to the same test conditions, and therefore all research subjects scores would have experienced a reactive effect A comparison oftraining scores from the non resea r ch part i cipants could have been made but the study is descriptive in nature to determine the factors that affected retention so this step was not undertaken Another concern is the use of simulated interviews rather than "real" inter v iews with children as a measurement of actual interview skill performance An interviewer may react differentl y to an adult actor role-playing a child than to a child and to the concomitant factors associated with an actual interview (e.g., dis tractibility of chi ld ren vacillating emo t ional reaction of both child and interviewer) Also the simulated interview tended t o proceed according to the script of both what happened and how the information should be obtained (i. e., the sequential interview protocol) whereas "real inte rviews ma y be much more 219

PAGE 234

ambiguous as children may no t be as forthcoming timeframes may be quite different (fo r instance caseworkers have an hour to build rapport and another for finding out about the alleged in cident), and a joint i nt erview with a law enforcement officer may comp letely change the interview dynamics Additionally, the Interview Instrument measured performance at each ofthe testing per i ods so the transfer measure was of retention rather than act ual interview skill on the job and 49 percent of the sample actually did better on the postte t than at the trai ing test Nevertheless, t he use of real abuse victims, or eve n a child actor for tra ining purposes would o bviously pose insuperable ethical probl ms. 5.4.6 Questionnaire Design The questionnaire failed t o evoke a range of response s from r esearch subjects as shown by the generally po itive responses they had to the posed questions even though t here may have be e n nega t i v e a tt itudes present It might be t hat the orga n izational climates as defined by the agency supervisor, and co worke r attit u des a r e conducive to training and thus tran sfer so th e re was no variabi lit y for the questionnaire to detect A lternatively, the questionnaire may no h av e been provocative enough to discern differences in organizational climates that ma y affect att i tudes towards training Perhaps the wrong questions were asked or they were too ambiguous Such ambig u i t y may be suggested by 220

PAGE 235

the fact that trainees generally had a positive opinion about their organization supervisor, and co-worker support but when a ked about behaviors that are i llustrative of good training and principles, their answers were no t quire so po irive Instead of asking opinio ns about the organizational environment, it may better to operationalize good training and transfer behaviors and survey research ubjects on those items. For instance future research might ask questions such as : How many hours of training does your organization encourage you to attend ye ar! y0 To what extent does your supervisor cover your cases while you attend traini ng ) To what extent does your supervisor ask you to call the office during training to check on the status of your cases? To w hat extent does your supervisor ask you to carry a beeper or cellular telephone to handle case emergencies while you attend training? These behavioral questions may be o r e indicative of attitudes towards training than direct opinio ns. It would appear from t his study that either there is a ubiquitously positive attitude towards training in the organizations studied or i ndi vi dual s are reluctant to criticize because they do not have a comparative perspective, or it not their nature to do so To some extent there was evidence of a lac k of behaviors that may indicate supportiveness towards training For 221

PAGE 236

instance, i t i s strongly recommended that trainees and supervisors view the vi eotape tog e th er to reinforce the t r aining receive additional feedback and pro v ide an opportunity for the superv isor to engage the trainee in one to one co aching. Few trai n ees and supervisors howe ve r engaged in this activity, and the e was even disa greeme n t fro m two sets of r espondents (trainee and supe rviso r ) abou t whethe r they had i nd eed view ed the videotape to g e t he r Obviously a supervisor nee ds to be supportive but it would appear tha t at least some of that s uppo rt is moral rather than beha v ioral. t also is possible th a t research subjects and supervisors i nterpret e d it e ms differently. Perhaps th term "use" (as in did u se the training") mean t in an y c apacity" to one person b u t an ot her per s on may have interpreted it to mean t he full applica tion o f the interview pr t o col. Another area that remains ambiguous is the interpretation rese arch subject made of certain questionnaire items For i nsta nc e the quest ion n aire item "I n m y agency, top management values staff tra n ing could have take n o n d i fferen t mean i ngs for subjec t s For on e person t his might mean th at a dmi nist r ators adverti se traini n gs and allow caseworkers to attend but th ey are expect ed t o main tain their cases while at training even if this means leavi ng the t raining A other person might interpret this to mean that their agency ac t ivel y recruits c ase work ers for trainings includes training as part of the ir perfor mance review, and expect s t hat training will be utilized and to facilitate this admini str ators do not allow ca s eworkers to carry beepers or 222

PAGE 237

telephones to training and ensures that their caseload is covered by other caseworkers during the training Also, the nature of opinions is subject i ve and i ndividuals may base their opinion on prior experience that may have been extraordinary or wretched in comparison to their current situation Therefore the perception of employees and supervisors regarding their organ i zat i onal climate towards training was assessed through self scored questionnaires that may have been inherently flawed 5.4 7 Comparison of Problem Statement Criteria and this Research In t he Literature Revi w chap t er research st u dies focusing on transfer and organizational factors were analyzed based upon a set of criteria thought to be releva n t for assessing their value (see page 67) This study was analyzed using the sam criteria established in the problem statement to discuss further study lim i t r t i o n s h e criteria were sample size type of subjects equivalenc y o f training programs criterion measures of performance and experimental rigor Typically more than 30 subjects is conside r ed adequate for a research study S t atistical power which is the ability of statistica l tests to detect a relationship is reduced with smaller sample sizes and while not ideal for sta t istical power 61 subjects is adequate for this study In terms of the types of sub j ec ts all were child welfare worker at approximately the same level of county 22 3

PAGE 238

employment Also, all subjects attended the same training program so the treatment was equivalent. Most positively, a criterion measure of performance was used to assess skill perfo mance and thus skill retention as a measure oftransfer. Post-hoc anal ysis, however, rev ealed some issues with the instrument. A Rasch facet analy is d is closed problems with a few items on the instrument, so adjustments were made to ensure a rating scale with more integrity and a truer equal-interval scale. Also, efforts were made to compensate for rater differences through a means ana l ysis and subsequent adjustments. In another measure oftransfer, tra i nees gave self-reports of transfer on the Trainee Follow-up Questionnaire, which were correlated with their supervisors perception of transfer behavior as reported on the Supervisor Follow-up Questionnaire The experimental rigor criterion implies that the st udy should be experimental i n nature; this study was descriptive so on this criterion it is rated as low. Study weaknesses, however, a r e mitigated by the study's strength s : the adequacy ofthe sample size; the equivalency ofthe subjects and training program; and the use of a criterion measure of performance to measure ski II s at three points in time Mos t of the criteria previously stated as being important for a research study on the transfer of learning were met 224

PAGE 239

54. 8 Summary of Study Limitations Ethica l and financial constraints did not allow for all potential study defici ncies to be addressed in the research design. For instance, a true experimental design was not possible, and trainees self-selected into the study During the analysis phase ins t rumentation issues also became apparent On the positive side, however, performance was measured using an instrument with pre spe c ified crite ria the sample size was adequate, and the type of subjects and training programs were equivalent Inherent limitations were unavoidable and should be con s idered when inte r preting res dts 5.5 Implica tions of the Study for Professional Practice The research study produced many ideas for improving public management of training programs that are specific to this project and may be generalized for use by any organization Some ofthese ideas are explored in this section 5 5 1 Rec onsidc ation of Training C r ric ulum Few sexual abuse interviews were conducted by research subjects but trainees generally had a high opinion of the training and the extent to wh ic h informafon was tran sferred to on -the-job performance Since "far or high road" transfer is more likely for this p articular training, strategies should be taught in the training that help trainees achieve this type of transfer and emphasized in 225

PAGE 240

the organizational environment after the tra i ning e x per ie nce It may be strategically prudent to re vise the training format to make it more relevant for all types of interviews wi th sexual abu se a s but one of many t y pes of specialized interviews that case workers may conduct Such a change would i ncrease the ut i lit y of the training and ackno w ledge the real ity of the job responsibilitie s of the case w orkers who attend the train in g 5 5 2 Reconsideration of t he Training Participants Th e primary part i c ip ants ofthi stu dy we r e not casewor ker s w ho ty p ically conduc t sexual abu se interviews in their j obs It is not known ifthis is a t yp i cal composit i on ofthe SIS t r aining classes o r an anomaly Since these trainees actuall y performed so few int erviews, it suggest s that managers and supervisors select potential training participants based upon the potential t he y may have to use the t ra i n ing on their jobs. F o r example training participants who are assigned to unit s su h as adoption or core services should be sent to traini n g m ore relevant to thei r job re s ponsibili t ies and not t o the SIS training, while a ttendees at th e SIS training s hould be caseworkers from units such as Intake or the Sexual Abuse Team who w ill have the opportunit y to t ransfer the training to their jobs Th us admi nis t r ato rs and su pervisors sh ould send trainees to t r ainings t hat are relevant to their job responsibilities 226

PAGE 241

Trainee personal characteristics of moti v ation and abil ity were highly associated w ith perce ived transfer as w as the opportunity to use the training From a managerial p erspective th en administrators and s upervi sors will attain the highest re tu rn o n th eir training investment when the y send ca seworkers to the training who will have the opportunity to apply the skills learned in training and then foster motiva ti on and abi l ity so tha t caseworkers want to transfer the skills and act'..lally d o a pply the trai n i n g This recommenda t ion is juxtaposed to the previous ugge stion of revising t he t ra inin g curri ulum to re flect the more diverse a ud ie n ce. It is clear that one or the other decision should be made to improve overall transfer. The advantage of adapting the tr aining curriculum is that it acknowledg es the reality that some counties require t his course of all new workers, despite their un i t assignments and therefore the curriculum would be made relevan t to a wider range of caseworkers who will be atte n ding the training Also, broadening th e curric u lum would better pre p are work er s who ma y chaP..ge job ass ignmen ts w i th this important sk ill. It is clear t hat workers whose job ass ignme nt s may p ro vide them the opportunities to conduct sexual abuse interviews should have the full tr ain ing to ensure they are prepared for this important job task If the tra i ning is restricted to worke rs w ho will use th e trai n .ng (e.g., Intake, S e xual Abuse T e a m) thos e individual s may ha ve a better learning expe rien ce as their l earn ing level will not be lowered b y workers who have no experience and will never have it with these types of interview 227

PAGE 242

Interviewing skills may actuaily improve for workers in th ese classes when the r e is more exper i ence to draw upo n and learn from. Either scenario-revising the curriculum or restricting the training-is preferable to ignoring the reality of the class composition. 5.5.3 Reconsidera tio n ofthe Interview Instrument and Videotape Rating It may be appropriate to reassess the Interview Instrument and the training of videotape raters to ensure the instrument maintains current validity and reliability Child welfare pract i ce has changed in the decade since the instmment was developed, but the instrument has not Close scrutiny of current best practice techniques and their reflection i n the Interview Instru ment will help make the p ro tocol more relevant to today's pra tice reality Und er closer examination of the Interview Instrument's content validi t y each item on it is weighted e qually, wh i le in practice some items may deserve greater emphasis than others For instance, it could be argued that avoiding leading questions is more important than stating your pro fessional role to the child y et they are both weighted equally on the instrumen (i e both items have a total point value of two ) Some conside ration might also be given to weighting interview items that are more important than others on the instrument Any changes in the instrument will require reestablishing inter-rater reliabili ty and developing a new !raining guide for videotape raters. 228

PAGE 243

Because of the inter rater reliabilit y i ssues with the videotape raters even if no change s are made to the ins t ru ment consideration might be given to r e visin g the video tape ra te rs' t r ain ing format and written training guide to achi eve better in terna consistency amongst t he v ideotape raters Otherwise a comparison of scores has relativel y little meaning for administrators supervisors or cas worke rs 5.4 Recons ider ation of t he T ransfer I nstrument Holton Bates Seyler, a n d Carvalho ( 1997) call for the development of valid and gen eral" zable tr an sfer cli mate s ca l e w ith known psychomet r ic properties to help unders t and the transfer process and enable cross study comparisons Such an in trument would also have diagnostic potential and w ould help to answer questions about the nature of t rain i ng transfer and about the for ces (e .g., trainee characte rist ics organiza tio nal enviro ment) that encourage or inhibit transfer of training to onth ejob performan ce. Tra nsfer p erformance should be operation liz ed and anchored b y co ncrete behaviors rather than just by opinions about w hether tran fer o ccurre d The g oal of this effort is to detect differences in t ransf e r performance an d the trans fer environment so more accurate relat i onsh i ps can be establi shed for recommending app ropriate interventions. Instruments shou l d be fully pretested and validated to e ns u r e items reflect the reality of behavior an d intentions. 229

PAGE 244

5 5 5 General Recommendations for Tr aining Managers The li erature, at lea t, is clear tnat trai ning in and of itself, is not sufficient to ensure transfer Organizational development programs should develop climates that are favorable to innovation and change to help maximize the payofffrom an investment in t rain ing Additionally they should help trainees develop the intrinsic factors of confidence self-efficacy and motivation which appear to be closely related to tr an sfer success. Multiple transfer strategies should be used to continue to encourage transfer. Brinkerhoff and Montesino s ( 1995) study utilized an experimental design to test the effects of a s upervisory intervention on transfer and perception o f forc es that encourage transfer It points to th importanc e ofbuilding strong trainer-man ag er trainee partnerships whic h the researchers called "train ing transfer support in the phases before, during, and after training Similarly, Curry Caplan and Knuppel ( 1994) recommend de v eloping a transfer desig n with i tervention methods for use before during, and after training and specified by the person performing the intervention including the trainee, trainer supervisor coworker and admini trator Their model is specific to child welfare caseworkers but can be generaiized for any type of employee or organiza tio n See Table 5 8 for suggested interventions differentia ted by the individual and the time period for improving transfer oftraining. 230

PAGE 245

Table 5.8 Potential Transfer Intervent ion s by Pe r son and Time Period Traine e Transfer Interventions Before l Plan for coverage of cases/un it while in traini ng 2. Id en t ify cases to keep in mind. 3. Begin to formulate objectives for action plan. 4. Foster mind set of geuing involved in the learning and transfer process. I During l Identify barriers and facilitative forces for applicatio!1. 2 Ask 'H ow Can I apply this to m y caseloadT 3 Identify key individuals who can have an impact on barriers and facilitators 4. Make a commitment with cotrainees to support each I other After 1 Hold a timely meeting wit h supervisor to discuss im portance and application 2 Share information with coworkers 3. Follow thro u gh w ith action plan 4. Place v isual rem i nders where easily seen Trainer Transfer Inte rventions Before 1. Conduct a multi-me asure needs a sessmen t 2 Provide information (e.g., objectives and outline of training) to administrators. superviso r s a n d trainees 3. Make tra sfer a pri rity i n all phases of the traini ng cycle During l. Set the stage for focus in g I on transfer I 2 Lea m retention strategies (id entical eleme n ts. ge n eral principles stimulus variability conditions of II practice ) 3 Focus on adult leaming I princ ip s an d trainee leaming styles. 4 Implement action plan. 231 After l Offer re m ind e rs of commitment t o a ctio n plan 2. Use booster shot" training sessions 3. Evaluate transfer.

PAGE 246

Table 5 8 (Cont.) Coworker Fo r ces Befo re During A fter 1 Invoi e teams in the !. Sit toge the r to reinforce I Review training coment needs a s s e sment process le ing and application an d application in team beh aYio r of other team m eet ings memb rs 2. Have t eams identify the 1 2 R evie w Lam me m bers 2 Remind team memb e r s of impact of training o n spec ific action plan s 1 action plans cases I I 3. Enco ura ge active I 3 Make pl a ns to reinforce 3. Provide team recognition involyement in the learning 1 teamma tes action plans for transfer. and transfer process as a I I g roup norm I 4. Identify hmv t rai ning i 4. Develop a team action 4 Integrate training t eam relates to team g o als 1 plan p lans and dec is ion s I I Supen,isor JntenJention s B efore 1 During After I I I. Conduct pre-training 11 Insu r e that there are no I Meet with supervisee rene wi th trainee distractions (cover for within a week to r ev ie w key trainee if necess ary). points in training and action I plan. 2. Discuss wo rks ho p 2. Convey that training i s a 2. Provide worker expectations with supervis e pnon opportunity t o try out new I skills I 13. Provide r einforcement for 3 Begin to p lan for acti n 3. Di cuss training and plan a pplic tion with worker usage of new skills between sessions. if I multip le -da y t raining. 14. Anend uaining. I 4 Convey trainin g as a I 4 Reduce b arri e r s to priori ty. 1 a pplic ati on of new skills No te From .. Transfe r oftrammg and ad ult leammg." b y D H Curry P Caplan. & J. KnupeL 1994.Journa l o[Con(nuing Social Work Educa ti on 6 (l). p 13. 232

PAGE 247

5.6 Recommendations for Further Re search In the tradition of scien t ific research, this study gives direction to other researchers for implementing stud i es and for other paths to exp lo re Areas for further research as well as components of future studies to consider include : Use criterion measures oftransfer performance as outcomes (instead o f self-report) Assess for actual beh viers as w ell as opi nions by organizations, supervisors and cowor kers ind i cating s upport for training Explo r the unintended benefits and "far" or "high road" transfer of training in increasingly sophistic ated evalua ti o n des i gns Examine the effects of various organizationa l cultures on transfer. Explore the role of organizational culture o n individual percept ions Explore t he influence of a s tat e agency on local county culture Research t he links between th e sta te and c ounty systems the presence and re l ative imponance oft he professional culture a n d its effect on organizationa l culture Study the effects of different t ran sfer intervent ion strateg i es o n transfer e g s upe rvi ory behavior befo re training) o Assess for the emotional i mpact of tr aining in future studies

PAGE 248

q Develop and validate instruments to measure transfer climates Ex plore the specific s t rategies that resulted in increased transfer. Study the person and organizational factors that may interact to affect t ransfer. Study the com p lex i nteraction among all of Bald win and Ford's tra ining inputs (trainee, design and work environment) Explore the relati o nship between interventions conducted by the trainer administr at o rs, and supervisor to impr ove transfer. Additio nall y researchers should continue to address the issues Baldwin and Ford n oted in 1988 namel y : 1 Th e criter ion problem of how transfer is defined and operationalized ; 2 The low complexity of the tasks used to examine the impact oftraining d es ign factors on learn ng and tra n sfer; 3 The lack of conceptual fram ewo rks to drive the choice of trainee characteris t i cs s uch a persona lit y motivation and ability to examine in tr ansfer research; and 4 Key wo rk env i ronment factors such as climate, support and opportunity to perform training that are clearl y conceptualized and operationalized. This res earch attempted to addr es s items I and 4 but did not quite achieve those goals due to instrumentation issues measuring transfer and the organizational 234

PAGE 249

environment. The research however did more clearly elucidate the direct ion for future researchers who would like to address these items Namely more careful attention must be paid to the instrumentation issues and how transfer is measured Instruments to measure both items should be thoroughly pretested be reliable and valid with known p sychometric qu ali ties. Effective criterion measures must consider iss ues of generalization (the extent to which sk i lls and behaviors are utilized in the transfer setting in any capa city) and maintenance of skills (mainte nance is the length of time that trained skills and behaviors continue to be used on the job) (Baldwin & Ford 1988) Ford and Weissbein (1997) found that researchers have made great strides in addressing Baldwin and Ford's issues, but more effo n must be given to the se and othe r emerging issues regarding transfer Based upon experiences with this research project, this author would encourage this a tt ention 5 7 Conclusion This research study added another block to the foundation of literature focusing on the transfer oftraining. Many lessons were learned from this study namely the importance ofvalidated c r iterion measures of performance questionnaires that reflect opinion and behaviors, and the study of multiple potential causes to understand transfer Transfer oftraining can b improved by attending to personal character i stics training design factors and the 235

PAGE 250

organizational environment Not all learn ing will be transferred to the job but it can certainly be impro ved to maximize the investment government agencies make in tra i ning. Most importantly better training programs mean that caseworkers will be better prepared to perform an e>-.1:remely important and critical jobprotecting children from harm In Chapter One I no te d that public management researchers have been debating about whether management really matters This research adds another small voice to the chorus that is saying yes, indeed it does matter. The research did not support the contention that management's efforts to provide organizational supportiveness as defined by Baldwin and Ford and for this study as organizational supervisor and co-worker support. It did however, suggest that environmental factors such as opportunity to use the training and trainee characteristics such as motivation and ability be fostered by public managers to impro v e transfer Also, the research did support that a managerial decision to provide the training, did indeed improve caseworkers' intervie w ing skills. Public management techniques and tools guide public administrators to make good decisions and manage effectively. Training is one such tool for improving the efficiency and effectiveness of child welfare and thus helping to va l id ate the importance of public management Without effective public management of training, child welfare caseworkers would conduct their tasks capriciously or i ncompetently And the consequences for not being effective could be enormous: 236

PAGE 251

families couid be tom apart at tremendous social cost, and children could be hurt ore ren killed On the other hand the benefits to f am ilies and children can be equally great, even priceless. As one child said to a Denver Intake caseworker shortly after revealing her abuse, I hav e been waiting and waiting Finally, you came (Colorado Department ofHuman Services 1998, p 3-17).

PAGE 252

Appendix A. Description of Coordination between Research Projects CDHS has been a national leader an d advocate for sophisticated and mean in.:,ful t r aining evaluations In 1997, CDHS contracted with AHA to develop and implement a strategic plan for evaluating all of Colorado s child welfar e training courses One component of that plan was to develop an evaluation of the SIS training conducted by DU-GSSW to empirically test whether training improved caseworkers' interviewing skills AHA's project (Research Project 1) built upon the existing training format and Interview Instrument already in use to assess skills and added a pretest to collect pre-tra inin g baseline data on trainees interviewi ng skills and a post-test interv i ew approximately two to four months after training to assess retention of interviewing skills and thus transfer This author's research project (Research Project 2) utilized the data from AHA's project and went a step further by admi n is t ering q esrionnaires !o cas ew or. ers and their supervisors to understand the factors and specifically the organizational environment factors, that may have affected transfer This research project was coordinated with CDHS and DUGSSW and is the basi for this study See the figure below for a v isual 238

PAGE 253

representation ofthe coordinat i on between AHA CDHS DU-GSSW, and the author. Coordination of Research Project Research Project 1 Research Project 2 CDHS sponsore d and supervised the training and research projects AHA or ganized and conducted Resear ch Project 1 and provided g uidanc e to Research Project 2. DU-GSSW coordinated and pro v ided the SIS training and supervised the graduate students who rated t he simulat ed videotape d interviews using the Interview Instrumen. AHA contracted with the same trainer who teaches the SIS training at DU-GSSW to provide the Advanced Train i ng and consult on the project d esig n of Research Project 1. As a staff member at AHA, this author also wo r k don the design and implementation ofResearch Project 1 239

PAGE 254

It should be noted that this author has been involved in the SIS training in several capacities over the past ten years As a graduate research assistant at DU GSSW the author role-played the child victim during the simulated interviews Subsequent to that assignment she also rated videotapes and provided feedback to training participants Approximately three years ago the author was hired by DU-GSSW to develop a formal written curriculum from the live training presented by the traiP.er (a backwards process, but nonetheless important for the future consistency of the project should it ever be presented by other trainers or agencies) Currently the author is employed b y AHA as researcher/trainer and was assigned to Research Project las an AHA staff member and third-party evaluator. Her role for Research Project l was to coordinate AHA's port ion of the project including recruiting participants overseeing pretests and posttests and coordinating the exchange of information between CDHS, DUGSSW and AHA 240

PAGE 255

Appendix B. Specialized Interviewing Skills Interview Evaluation SPECIALIZED INTERVIEWING SKILL lNTERVIE\V EVALUATION SKILL TESTED YOUR SCORE 1. Greet the child by name 2 Introd uce yourself 3. State professional role. 4 State purpose of interview 5. Ask for the child's perceptions 6. Estab lish rapport 7. Obtain names and relationships of family member s 8 Assess child's und rstand i ng of sequence or progressiOn 9. Have child explain what trut h'' and"lie means 10 Ask child consequences of lying 1 1. Stat you (i nterviewer) will tell only the truth. 12. Ask the child to agree to tell o n ly the truth. 241 (0 2)

PAGE 256

13. Introduce abuse topic 14 Maintain comfort in introduci ng abuse to pic 15. Determine name of the perpetrator. 16 Determine the relation hip of the perpetrator. 1 7 Maintain comfort level in discussing body parts 18. State descriptio n/purpose of doing a body inventory 19. Conduct a body parts inventory 20. Use the child's words for body parts thro ughout the interview 2 i C l arify the ch ild's words for body parts and events mean in describing the abuse 22. Allow the child to demonstrate or describe what h appened/was to uched 24. 25 Avoid personalizing an y aids Ask where the most recent incident of exual abuse occurred. Determine when the most recent Incident occurred. 26. Determine who, if anyone, the child has told of the abuse. 242

PAGE 257

27 Determine who was in the room with the child and if anyone else saw or helped the perpetrator during the most recent incident of abuse 28 Determine whether or not multiple incidents of abuse have occurred. 29. Determine the specific details of the abuse such as whe ther or not ejaculation occurred ; whether or not there was blood ; what, if any prop s or devices were used 30 Determine whethe r or not there have been multiple perpetrators 31. Determine the r isk of physical violence to the child and /or other family members 32. Us the technique ofuniversali zation at least once in the interview 33. Reflect content back to the child at least once in t he interview 34 Retlect feelings back to the child at least once in the interview 35. U se neutral minimal encourager s at appropriate points in the interview 36. First use no n leading quest io n to elicit response 37 Offer the child choices when a leading question is used 38 Avoid asking 'do" or d id" questions which can be answered y es" or no" 243

PAGE 258

39 Avoid asking "why questions abou t the abuse 0 A void judgmental comments or suggestions 41. A void leading o r j ud gmental behaviors 4 2 Ask the child for his / he r questions concerns fear s 43 Affirm to the child that he / she has done the right thing b y r eve ali ng the ab u se 4 4 Offer rea suran c e, ba ed on the chi l d's cues 45. Avoid unrealistic reas sura nce or pro m1ses 46 P ace i nterview appro pri ately for the chiid's ab i lity and comfort 47 Demonstrate continuity and logical equencing throughout the interv ie w 48. Express congruence between content Present e d and affect and beh avi or. 49. A v oi d in i tiating or othe rw is e inappropria tel y touching the chi l d 50 State immediate next steps to be taken in the investigation. Date: Trainee's name: Scenario s name : Actor's name : 244

PAGE 259

Appen dix C. Sat i sfaction Survey Specializ d Interviewing Sk i lls for Chil d Welfare Worke r s Date ___ I. TOP TWO REASONS FOR ATTENDING: ____ I h eard it was interesting/fun area I wan ted to know more about this area. ___ It was requir e d --I recentl y had a change i n iob duties/clients. I was having difficulty in this I needed the training hours. It's a day awa y from the office Other (Please specify)_ II. TRAINER FEEDBACK ame of Trainer(!) ______ ( l) Strongly (2) (3) (4) Strongly Disagree Disa gr ee Agree Agree The trainer knew the subject area. I The trainer w s well prepared and organized. The trainer related well to the group answered questions and responded to concerns The trainer provided enougr. I explanation and examples i The trainer gave me enough I I opportunities t o practice skills. The trainer used handouts. flipchart s and visuals to illustrate key points The trainer motivated me t o want to try out the training ideas on the job The trainer modeled cultural sensitivity. 245

PAGE 260

Name ofTrainer (2) : ____________ ___ (1) Strongly (2) (3) (-+) Strongly D i sagree Dis agree Agree Agree The trainer knew the subject I I area. I I The tra iner was ell prepared I and organized. I The raim:r related well to the group. ans"' ere d questions and I responded to concern s I I The trainer provided enough explanation and examples T he trainer gave me enough i opportunities to practice skills. I The trainer u sed handouts I flipch ar ts and v isu a ls to illustrate i key points The trainer mot iva ted me to wan t ro try out the t raining ideas on the job. T he trame r r :1ode\ed cultural sensiti v ity I I III. WORKSHO P CONTENT (1) Strongly (2) (3) (4) Strongly DK Disagree Disagree Agree Agree 1 The subject matte r was at the I rig_ht level of difficulty. The workshop content was compatible w ith my agency's philosophy and policies My agency w ill support me in using this training on the job 1 !earned specific, job-related knowledge/skills. I will use knowledge/skills I from this training on the job I will be able to do m y job bette r because of this training Familie s will benefit from my takin g tllis course. Explanatory Comments. 246

PAGE 261

IV COMPETENCIES Has your knowledge or skill I (I) No (2) Yes. (3) Yes. (4) Yes increased in .. ? a Litt l e Moder a tel y a Great Deal polici e s related to CPI I inte r views wit h the child victim th e non offending p arent and I the alleged perpetra t or applying knowle d ge o f the I in t erviewing process an d of ch ild develo pment to the CPI I I mterv i ew with children ut ilizi n g know ledge of I interviewi ng children in conducti n g a CPI interview with I I a child i Comments 247

PAGE 262

Appendix D. Trainee Follow-up Questionnaire Please take a few minutes to answer rhe following questions. Circle the number that best describes your opinions on the statements below Please only circle one number. Some questions ask for a spec!fic number (for example. your age). All questions regarding the training r efe r to the Specialized Int erview ing Skills for Ch ild Welfare Workers. Please answer ALL questions. Your responses are completely confidential and w ill not be shared with others. Social Security Number Superv sor Name Strong Moder Mlldly MUdly Mode1 Strong ly a.tely Disagr Agree ately ly Disagr Disagr ee Agree Agree ee ee 1 As a result of the training I 2 3 4 5 6 substantially increased m y kn owledge on this topic 2 I did not le arn any new skills 1 2 3 4 5 6 at this training. "> The training has affected my 2 3 4 5 6 -' attitudes concerning the topic area. 4. I was motivated to put th:is 1 2 ,., 4 5 6 .) training into practice on the j ob 5. I imple mented on the job at 1 2 "' 4 5 6 .) least one concept/skill that I l earned 6 My supervisor supported me 1 2 3 4 5 6 in implementing the training on the job. 7 My supervisor was familiar 1 2 3 4 5 6 wi th the content of this training. 248

PAGE 263

Strong Moder Mildly l \'lildly Moder Strong ly ateJy Disagr Agree ately Jy Dl'lllgr Disagr ee Agn.-e Agree ee ee 8. I met with my supervisor to 1 2 3 4 5 6 discuss application of this training on the job. 9. I had a pl an as to how I would 2 3 4 5 6 imple men t the train ing 10 I was not able to implement 1 2 ,.., 4 5 6 ..) the training on the job. 11. My supe r visor considers 1 2 3 4 5 6 training a waste oftime. 12 My supervisor expecte d me to 1 2 3 4 5 6 use this training on the job. 13. In my agency, top 1 2 3 4 5 6 management valu e s staff training. 14. In m y agenc y top 1 2 3 4 5 6 management views this training as a low priority 15. Even if n o one noticed or 1 2 3 4 5 6 notices I w ill use know l edge learne d from this training on the job. 16 The training w a s relevant to 1 2 3 4 5 6 my job duties. 17. My co-workers value 1 2 .... 4 5 6 .) training 18. My co-workers upported my 1 2 3 4 5 6 attemp ts to use the training on t he job. 19. I mad e a plan with a co -2 3 4 5 6 w o rker to use this training. 20 I have had sufficient 1 2 "l 4 5 6 ..) 249

PAGE 264

Stro g Moder Mildly 1\.fildly Moder Strong ly ately Disagr Agree at.ely Jy Diomgr Disagr ee Agree Agree ee ec opportunities to practice the new ideas/skills/techniques on the job. 2 1 M y organization values 2 4 5 6 .) training. 22. My organization supported 1 2 3 4 5 6 my att empts to u se the training on the job. 23. When I had trouble 1 2 3 4 5 6 implementing the training on the job, I could count on others in my agenc y to help me. 24 I used the training on the job. 1 2 3 4 5 6 25 The training con t ent as 1 2 3 4 5 6 cons ist ent with my agency's p olic i es and my individual responsibilities 26 The training content is 1 2 3 4 5 6 consistent wi th my agency s mission, philosophy and goals 27 As a result of this training, I 2 4 5 6 .) a m a more effectiv e er. 28. I attended the training due to 1 2 3 4 5 6 a mandatory requirement 29. Prior to the workshop I was 2 3 4 5 6 motivated to attend. 30 My supervisor helped to 1 2 3 4 5 6 250

PAGE 265

Strong Moder Mildly Mildly Moder Strong ly ately Di.sagr A gree ately l y Di.ugr Di.sagr ee Agree Agre e ee ee prepare me for the training by discussing my learning needs. 31. It is important for me to do 2 3 4 5 6 well in my job. 32 I rely on my tea m and 1 2 3 4 5 6 supervis or for feedback. _..,.., I know inside when I do well 2 .., 4 5 6 .) .) .) at m y j ob or don "t do wei! at my job 34 After the training, I felt more 1 2 3 4 5 6 confident to interview children. 35 I have low expectations for 2 3 4 5 6 my i nterviewing perfo mance 36. Prior to attending the training 1 2 3 4 5 6 I heard the training was worthwhile"/valuable. 37 Since the training I ha ve 1 2 3 4 5 6 retained what I learne d at the training 38. Since the training I have 1 2 .., 4 5 6 .) attended additional training on sexual abuse interviewing 39. Since the training, I ha v e 1 2 .., 4 5 6 .) done additional reading on sexual abuse interviewing 40. Did y ou watch the tape with Yes No your supervisor? 251

PAGE 266

Strong ly Disagr Moder Mildly ately Disagr Disag r ee ee ee Mildly Moder Agree ately Strong ly Agree 41. Since the training I have conducted ___ sexual abuse interviews with chi ldren 42. Since the training, I have conducted investigative (regarding physical abuse neglect emotional abuse or sexual abuse) interviews with children 4 3 Since the train i ng I h ave used the introdu c tion portion of the interv i ew protocol times during interviews with chi ldren. 44. Since the training I have used the t ruth and lie portion of the intervie\v proto col times during intervie ws with children 45 Since the training I have used the Body Parts Inventory portion of the interv i ew protocol times during int erv i ews wi th children. 4 6 Since the training 1 have used the target event (e.g., the incident) portion of the interview protocol times during interviews with children. 47 Since the tr a ining I have used th e summary portion ofthe inter view protocol ____ times during interviews \\i th children Please answer a few d emographic questions T h is infor matio n i s very important for help ing us to understand your respons es 48 What is y our c u rrent title ? Caseworker I Caseworker IT Caseworker III Caseworker IV 49 What is yo ur current unit? Intake _Ongoing General Yo u th -inConfli c t 252 Caseworker V Supervisor I Supervisor II Other Se xua l Abus e Team _Adop tio n Core S ervi ces _Famil y Therapist Other ---

PAGE 267

50. How long have your worked in your current position? Years Months 51. How many yea rs have y o u wo rked for a county department of social services, in the area of child welfare services? Years Month s 52. How many years have you worked in child and family services, regardless of employer or type of employment? Years Months 53 Which desc ri bes your education level? High School Some College College Degree (4 year college) Some Graduate Work _Graduate Degre e 54. In what year did you receive your last degree? ____ 55 What is y our gender? Male Female 56. How old are you? __ 57 What is your marital status ? __ Single Married __ Separated/Divorced Widowed 58. How many children do you have? __ 59. What is your ethnici ty? Asian / Pacific Islander African American _Hispanic Native American White Other Thank you for your participation!!! 253

PAGE 268

Append x E. Su p erv isor F o llow up Questionnaire An employee of yours rec e ntly complered the Specialized Int erv iewing Skills for Child W elfare Workers Please take a few minutes to answer the following questions. Circle the numbers that bes t describes yo ur opinions on the statements Some questions askfor a specific number (for example, n u mber of interviews). All questions regarding the training refe r t o the Specialized Interviewing Sh!lsfor Child Welfare Workers Please answer ALL que stio ns. Your res ponses a r e completely co nfidential and will not be shared wit h others. Employee name : Strong Moder : Mildly Mildly Mode Strong Don't l y ately Dis-Agree ately ly Know Disagr Dis -agree Agree Agree ee aoree 1. As a result of the 1 2 .., 4 5 6 DK -' training my emplo ye e substantially increased his/ her knowledg e on this top ic. 2. As a result o f the 1 2 3 4 5 6 DK training, my employe e has not developed new skill s ") The train ing affected my 1 2 .., 4 5 6 DK -' .) emp lo yee's attitudes concerning the topic area 4. My employee was 1 2 3 4 5 6 DK motivate d to put this training into practice on the job. 5 My emp lo yee 2 .., 4 5 6 D K -' implemented on the job at least one concept/skill th at he / she learned 254

PAGE 269

Strong Moder Mildly 1 \Uidly Moder Strong Don't ly ately DisAgree ately ly Know Disagr Dis-agree Agree Agree ee agree 6 I supported my 1 2 3 4 5 6 DK employee in implementing the training on the job. 7 I w as fa mi l iar with the 1 2 3 4 5 6 DK content of this t r aini n g 8. I met with m y employee 1 2 3 4 5 6 DK to discuss application of th is training on the job. 9. My emp loyee had a plan 2 4 5 6 DK .) as to how h e o r she ;vould i mplement the training. 10. My employee was not 1 2 3 4 5 6 DK able to implement the training on the job. 11. In m y opinion training 1 2 ..., 4 5 6 DK .) is a w ast e of t ime 12 I ex.rpected my employee 1 2 3 4 5 6 DK to us e th.is training on the job. 1 .) In m y agency t op 1 2 3 4 5 6 DK ma na gemen t val ues staff training. 14 In m y agency top 1 2 3 4 5 6 DK management views this training as a lo priority 15 M y emp l oyee is self-2 .. 4 5 6 DK .) motivate d 16 The training w a s 1 2 3 4 5 6 DK 255

PAGE 270

Strong Moder Mildly Mildly Moder Strong Iy ateJ, r DisAgree ately Jy Know Di:.agr Dis-agree Agree Agree ee agree relevantto my employee' s job duties 17. M y staff val u es training. 2 3 4 5 6 DK 18 My staff supported my 1 2 ..., 4 5 6 DK ..) employee's attempts to use the training on the job. 19. M y employee made a 1 2 .., 4 5 6 DK _, plan with a co w orker as t o how to use th i s training 20 My employee had 1 2 3 4 5 6 DK sufficient opportunities to practice the new ideas/ski lls/techrriques on the job. 21. M y organization values 2 3 4 5 6 DK training 22. My organization 1 2 3 4 5 6 DK supported my em plo yee' s attempts t o use the training on the job. 23 When staff have trouble 2 .., 4 5 6 DK _, mpleme nt i ng tr ai nin g on the job we can count on others in m y agenc y to help us 24. My employee used the 1 2 3 4 5 6 DK training on the job. 25 The training content was 2 4 5 6 DK .) 256

PAGE 271

trong Moder Mildly Mildly Moder Strong Don't ly ateJy Dis -Agree ately Iy Know Disagr Dis-agree Agree ee agree consistent with m y ag e nc y s po l icies and m y emplo y ee s r e sponsibiliti e s 26. The training content is 1 2 3 4 5 6 DK consistent with my age ncy s mission philosophy, and goals 27 As a result o f this 1 2 3 4 5 6 DK training my emplo ye e is a mor e effecti v e i nterviewe r 28 My employee as 1 2 4 5 6 DK .) mandated to attend the training 2 9 Prior to the w orkshop l 2 4 5 6 DK m y emplo y ee was mot iv at e d to attend 30. I helped my employee to 1 2 3 4 5 6 DK prepare for the training b y d iscussing his/her learning needs 31. My employ ee is se f-2 4 5 6 DK .) m oti v at e d 32. M rnployee reli e s on 1 2 4 5 6 DK .) our team and myself f or feedback. 33. My employee needs 2 3 4 5 6 DK fe e dback about his/h e r performance to feel ok. 34. After the training, my 1 2 3 4 5 6 DK 257

PAGE 272

Strong 'Ioder Mildly Mildly Moder Strong Don't. ly ately DisAgree ately ly Know D.isagr Dis-agree Agree Agree ee agree employee fe lt more confident to interview children 35. Mye ployee has high 2 3 4 5 6 DK ex pectations for h i s/h er i nterviewing performance 36 P ri o r to my em plo yee 1 2 3 4 5 6 DK attending the training, I h e ard the training was 'w orthwhile"/valuable 3 7 Since the training my 2 3 4 5 6 DK e mplo y ee has re tained w hat he/she learned at the tra inin g 38 Since the training my 1 2 3 4 5 6 DK em plo ye e attended addition al training on sexual abuse interviewing. 39 Si nc e the training m y 2 3 4 5 6 DK emp lo yee h as don e addition al reading on sex ua l a buse i n terviewing. 40. I watched the in terview Yes N o tape my employee made at the training together w ith him/her Please answer a few demographic questions This informatio n is very important for h elping us to understand your responses. 25 8

PAGE 273

41 What is your current title? Caseworker ill Caseworker IV __ Supervisor I Supervisor II Supervisor III Other 42 What is your curr e nt unit? Intake _Ongoing Gene ra l Youth-in-Conflict Sexual Abuse Team _Ado ption Core Services Family Therapist Other -------------------------4 3. How long ha e y our worked in your current position? Years Months 4 4 How many y ears have you worked for a county department of social servi ces in th e area of child welfare ser v ices ? Y ea r s Months 45 How many years have you worked in child and family services, regardless of employer or type of employment? Years Months Thank you for your participation!!! Ple a se return th i s questionnaire by June, 5 1998 in t he enclosed stamped envelope to: Charmaine Brittain American Humane Association 259

PAGE 274

Appendix F. T ran sfer of Learn ing Qualitative Interview Quest i ons Target: 25 re search participant trainees wh o attended the t r aining 8-15 month" ag o Method : Telephone i n tervi ew) approximately 20-30 minutes Introduction : H ello I'm calling about the SIS trai nin g that you attended several months ago Do you remember t hat training? Yes / no Provide prompts (it was w ith the tra i nin g you atte nded where you interviewed an actor who role played a child, it was v ideotaped .... ). You might recall that yo u were interested i n talking to me about th e factors that might have affected how much y ou learned in the c assroom I' d like to spend about 20-30 minutes tal ing to yo u abou t th e SIS training and the factors that may have affected your ability to apply that information to the job 1. Can yo u tell me about how you've b een able to use the SIS training at your job (or not)? 2. What do you think helped (or hindered) your ability to use what you learned in the SIS training to your job? Were there any barr i ers to using the skills learned at train ing? 3. Is your organization supportive oftraining? How or how not? 4 What kinds of thi ngs d oes your organizat ion do that are supportive or non supportive of training? 5 Is your su perv isor supportive oftrai nin g? Please exp l ain 6. What kinds of th ings does your supervisor do that i s supportive or non supportive oftra ining ? 260

PAGE 275

7 Ifyour supervisor was supportive of training do you recall your supervisor doing anything to prepare or orient you to SIS training? If so what did he or s he do ? 8 I f y our supervisor was support i ve oftraining, what kinds ofthings did your u pervisor do to support you dur i ng the training? 9 Ify our supervisor was supportive oftraining, what kinds ofthings did your supervisor do after the training to help you apply it to your job? 1 0 Ifyour superv i s o r was N O T supportive oftraining, what could he o r she have d o n e to make the t rai n ing e x perience more meaningful to you ? 1 1 Are y our c o -workers supportiv e of training ? How or how not? 1 2 What kinds of things do your co-workers do that are supportive or non supportive oftraining? 261

PAGE 276

Appendix G. Recruitmen t Letter and Consent Form N oYember 20, 199 7 First arne L a st Name Address 1 City State Posta Cod e De a r F irstName You are currently s che duled t o attend the Specialized Interviewing Skills ( S IS) for Child Welfare W orkers (trai ner Larry Wright) on December 10-12 1997 As a partic ipant in SIS training, you a re invited also to participate in a research pro j ec t conducted jointly by the American Humane Association. U ni v ersity of Denv e r a nd Unive r s ity of Colorado -Denver. The res e arch project will e \' aluate the effect i ven ess of srs t raining in helping workers develop and r etain interv iew skills once they ha ve re t urned to t h e job. In exch an g e for your partic i p a tion in the research you are invited to attend Advanced I nt erviewing ( Al) training conducted b y nati onall y acclaimed trainer Larry Wright. This training developed esp eci a lly fo r workers who agree to partic ipat e i n the r esearch project will be held the day after you complete the third i nterview ( described below) Only individuals who participate in t h is resea rch stud y will be eligible for the Advanced In t erv i ewing training There will be no cost t o you o r your age nc y to a ttend this training and all travel and p e r diem e xpenses will be re i mbursed As y ou know during Ll).e Specialized Interviewing Skills training you will interview an actor ro le p lay ing a child vi c t im T his interview will be video-recorded If y ou parti cipa t e in the research project you will do two a dditional 30 minut e videotaped interviews w ith actors The first the pre-tes t". will be done the da y before SIS tra ining. The last interview will be do n e approx i m a t ely three months after S IS training. i e .. the day before your Advanced lnterviewing training You also w ill complete t w o q uestionnaires The first which is a bout y our satisfaction with the S I S training. will b e comp le ted at the en d of SIS training The second which is about y our as sessment of the environmental conditions at y our workplace that might affect your ability t o usc the skills you ac quired in training will be complete d durin g the AI training In addition y our supervisor' ill be asked to fill out a similar questionnaire to assess his /her op inions of th e enviro nmental condit i ons All r esearch results will b e strictly confidentiaL Your responses and those of y our supervisor on the questionnaires as well a s y our performance on the v id e otapes will not be shared with any one All identifying inform at io n w ill be coded and onl y the small research team at 262

PAGE 277

American Humane Association and University of ColoradoDenver will have access to the data The results will be reported in terms of the entire group. We a r e NOT evaluating individuals. but rather the effectiveness of the training. In summary if you participate in the research project. you will conduct three 30 minute videotaped interviews with an actor rol e pl aying a child victim and com plete two que st ionnaires Th e first videota ped intervie\ will be held the day before SIS training which is D ecembe r 9. 1997 anytime between 1:00pm. and 7 00 p.m at American Humane Association ( l3 miles south of Denver offl-2 5 see enclosed map for directions ) The second interview will be the regularly scheduled int rview during the SIS training on December 11, 1997 The final interview will be held the da y before (March 16, 1998) the Advanced Interviewing training scheduled for March 17, 1998 a t American Humane Association At each of the two training sessions you will complete a questionnaire If yo u d cide to participate your supervisor will be notified that y ou are participating in this project and will be asked to complete a questionnaire about his or perceptions of your work environment. Your participation is key to the success of this research project. If you are willing to participate please sign the enclosed consent fonn and return it to me in the enclosed self a ddressed stamped en ve lope by November 29. 1997 Please call me to schedule your video-taped pre-test interview to be held on D ecember 9 1997 between 100 p.m. and 7:00p.m. To schedule the pre-test i otenljew or if y ou have any other questions, p l ease call me at my home office numb r (303) 433-0058. If yo u would like to prepare for the pre-test interview (e.g. read the i ntake scenario and task assignment). please plan on arriving an hour early. Thank you in advance for considering participating in tllis research project The results will help trainers understand what helps workers retain interviewing skills over time and to then make changes in the training to support this. Charmaine R. Brittain M.S W Researcl.er 263

PAGE 278

University of Colorado -Denver Graduate School ofPublic Affairs SUBJECT CONS T FORM FOR PAR TICIPA TIO IN A RESEARCH PROJECT Project Title: Evaluation of Specialized Interviewing Skills for Child Welfare Work ers Res ea rcher: Charma i ne R. Brittain M.S.W. You a e being asked o partic ip ate n a study with the primary goal of understanding th e environmentai facto r s that affect whether training is transferred to the job and the ex1ent training is retained after several m onths If you agree to participate in the study_ you will be asked to participate in three interview s with an actor \-Vho will rol e p lay a child abuse victim. The first interview will be scheduled the day before the training s peciali ze d Interv iewing Skills for Child Welfare Workers begins. This interview is approximately 30 minutes in duration The second intervi e w will be held during this 3 day trai ning. The third int erview (30 minutes in duration) will be held the day before th e advanced one-day training scheduled approximateiy three months later. These interviews will be videotaped for later review and analysis b y the researcher. In add ition y ou will be asked to fill out two brief questionnair s during th e trainings. Measures will be taken to protect your confidentiality (e.g. questionnaires wi ll have a sticker with a bar c ode to identify eac h participant) In addition your supervisor w ill be contacted and asked to fill out a similar questionnaire to the one you will fill out to determine his/her opinions regarding your organizational environm ent in regards to train in g. T he se surveys will be correlated in order to eval uat e whether your agency s organi zational emi ronment is supportive or not supportive of training. There are no major ris ks to y ou personally b y participating in this study As a benefit for your participation you will have the opportunity to participate in an advanced training b y a nationally acclaimed trainer. You rna refuse to answer any questions on the questionnaires You have the right to withdraw from the study a t any time If you have any questions a t any t ime. please contact the r esea rcher Charmaine Brittain. If y ou any ques tions about your rights as a research subject please contact the Office o f Academic Affairs CU Denver Building Suite 700_ 556-2550. A cop y of this consent form will b e return ed to you for your records. The kno vledge gained from this study may u lt imatel y benefit both child welfare administrators and casew orkers by providin g information to improve the effectiveness of tra ining. I have r ead the above and understand the purpose risks and benefits of the study My signature indicates agreement to participate in this study. Participant ____________ Witness : _________ SS# S uperv is or Name : __________ Date: 264

PAGE 279

Appendix H. SIS Data C ollection Plan (Based on 4 scenarios and 8 actors) Fe males Birgitta D., Marta Hilary, Luanne M ales : Sam, Michael 0., Brad J Do ug Scenario 1--Turner Scenario 2--B a rtlett Scenario 3-Martin Scenario 4 ---B ergman Class Traine Pre Session e Scenar group 10 1 A 1 Oct 8-10 B 2 2 A .... -' O ct. 15IB 4 17 I .... i; 4 -' ov 12-1 14 .... A -' Dec 3-5 B 4 A l Dec 10-IB 2 12 I 5 A 2 Jan. 21-B 4 23 6 A Jan 28-B 30 7 A 4 Feb 25-, B 2 27 8 A 2 I Pre Post Post FollowFollow A ct ors Scenari Ac tors up up 0 Scenari Actors 0 BD 2 s 4 D s .... BD 1 M -' L 4 B J 2 s s 1 M ..., H -' s l H .... L -' M 2 BJ 4 MO D .... M 1 L -' H 4 s 2 MO L 2 D 4 MO BJ ..., M 1 L -' BJ 4 D ..., H -' l ..., -' I 2 I ..., -' 1 4 3 1 265

PAGE 280

I I Mar. 18-B j 3 4 2 20 I 9 j A 1 2 4 Mar. 25B 2 13 I 27 I I 10 A 3 4 2 Apr. 15I B 4 I 3 17 11 A 1 H i3 BD 4 BJ May 13-B T j 2 MO 1 BD .) L 15 12 A 3 2 1 May 27 B 2 4 3 29 As it stands now, all actors share some trainees with all other acto rs and play each relevant scena r io about the same number of times Maintaining this connection and bala n ce is the goal to work toward in assigning actors to sess i ons Actors/Numb er of Shared Sess ions BD-S 2 BD-1 S M 1 MO BD-BJ 1 S -B J 1 M -2 MO BD-D 1 S-D 1 M -1 MO1 BJ H BD-M 1 S 2 M-D 1 MO1 H-BJ 1 M O BJ BD-H 1 S-H ') M H 1 MO1 H-D 1 BJL 1 .) D BDL 1 S L 2 M L 2 MO1 H L 1 BJ-D 1 L-D L 266 2

PAGE 281

Actors/Number of Scenarios Played (Females can play only I and 3 Males can play only 2 and 4) Scenario Actor 2 3 4 BD 2 2 M 3 2 I H 2 1 2 L 3 "' -' s 3 2 MO 2 2 BJ 2 2 D 2 2 267

PAGE 282

Appendix I. T ask Letter Dear Trainee. Please rea d the attached Intake Summary for preliminary information a bout your report. You arc abo t to conduct an interv iew with an actor who is ro l e pl aying a child victim of se:\'Ual abuse. Tllis is a pre-test and you are not e pecte d to perform well on this tes t (altho u gh if you do thafs great!). This pretest will establish your base line skill l eve l for int erv i ew ing children During t he v ideo-taped interviewed try to find out t h e following informat ion: 7 Docs the child unde r stand the differ ence between truth and lie? Does th e clli d understand sequencing? What does the child call male and fema l e body parts? Was the chi l d se:mally abused. if not what other e. p lanation was there for the child s statement? Exactly what happened? Who was the perpetrator ? How often did i t happen? "' Where did i t happen ? o Who knew about ii? H a s the child been threatened? Dunng the intervievv try not to use leading questions To tl1e extent possible establish rapporr with the child During the in terv iew. assume that there is only you and tl1e cllild in tl1e room and y ou are in th e location mentioned on the Intake. n t h e room, you will have: Ana t omical dolls Anatomica l drawings Toys Drawing Pape r/Markers You may use these props if you choose. You have 30 minutes to complete the interview Because time limited. it is suggested tilat yo u do not take notes Prior to the ta pin g, you and the actor will discu ss how you w ill know when there is five minutes left. You are participating in a scientific research study so please do not discuss ) our scenario with other trainees either in or outs i d e of tile traini ng. We n ee d to ensure tllat r esults do not b ec om e '"con t am in ated." Thank y ou for your participa t ion Y ou will be contacted wi th tile date of the post-test/advanced training. GOOD LUCK!!! 268

PAGE 283

References Abernathy D J. ( 1 999, February). Thinking outside the evaluation box Training & Development. 19-23 A!lig r, G M. & Janak E. A C 989) Kirkpatrick's levels of training criteria : Thirty y ears later. Personnel Psychology 42 (2) 331-342. American Humane Association (199 7 ) A strategic plan for training evaluation m Colorado (Working Paper). Englewood, CO: Author. Argyri C (1957) Personality and organization. New York Harper & Row. Argyris, C. (1971 ) Managemem and organizational developm enT: The path from X4 to YB. New York M Graw Hill Book Company. Bal d w in T T & Ford J. K ( 1988) Transfer of training : A review and directions for future research Personnel Psychology, -11 ( 1 ) 63-l 05. Bald w in T T, & Magjuka, R J. (1991 ) Organizational training and signals of importance : Linking pretraining perceptions to intentions to transfer. Human Resource Development Quarterly, 2 (l ), 25-36 Baldwin, T. T & Magjuka, R J. (1997) T raining as an organizational episode: Pretraining influences on trainee motivation In J. K Ford (Ed ) Improving t raining effectiveness in work organizations (pp 99-127). Mahwah, New Jersey : Lawrence Erlbaum Associates Publishers Barnard, C (1938) The function s of the executive. Cambridge MA: Harvard University Press Basarab, D J, Sr. & Root, D. K ( 1992) The training evaluation process. Boston: Kluwer Academic Publishers Bassi L. l & Van Buren, M E. (1999, January) Sharpening the leading edge. Trai n ing & Development, 23-33 269

PAGE 284

Baumgartel, H. & Jeanpierre F. (19 72). Applying new knowledge in the back ho me setting : A st ud y of Indian managers adoptive efforts J ournal of Applied Behavioral Science, 8 (2) 674 694 Be n R D (1991) Leadership counts. Cambridge MA Harvard University Pres s Bennett J. B Lehman W. E K. & Forst J. K. (1999 June) Change tra nsfer climate and c us to mer orientation Group & Organizati on Management, 2-1 ( 2), 188-216 Beudin, B. P ( 1 987 November/December) Enhanci ng the transfer of job-related learning from the learning environment to the workplace Pe1jormance and I nstruction, 19-21. Berdie l & England, T ( 1994 ). Components of train in g evaluation systems The Training Connection 1 (3) 5 Birnbrauer H. (1987 July) Evaluation techniques t hat work. Tra inin g and Develop me nt Journal, 53-55 Bloor G & Dawson P (1994). Understanding professional culture in organizational context. Organization S tu d ies, 15 (2) 275-295 Bri kerhoff, R 0 ( 1 987) Achieving result s from tra i ning: How to e val uate human r esource d eve lopm ent to str e ngthen programs and increas e imp act. San Francisco: Jossey-Bas s Publishers Brinkerhoff, R 0., & Montesino, M. U. ( 1 9 95) Partnerships for t rainin g transfer Less ons from a corpor a te stud y Huma n Resource Development Quarterly, 6 ( 3), 263-274 Broad M L. (1997). Ove rvi e w oftransfer oftraining: From learning to performance Perfor man ce Improvement Qua rt erly, 10 (2) 7-21. Broad, M L., & Newstrom J. W. ( 1 99 2) Tramfer of tra inin g: Action-pa c ked strategies to ensu r e high payoff from training investments. Reading MA Addison-Wesley Publish ing Company, Inc. 270

PAGE 285

Brown, A D & Starkey K. (1994) The effect of organizational culture on communication an d informati on Journal of Management S tudies 31 ( 6), 807-828 Burke, L (1997). Improving positive transfer : A test of relapse prevention tra i ning on transfer o ut comes Human Resource Development Quarter l y, 8 ( 2) 115-128 Burke L, & Baldwin, T. (1999) Workforce training transfer : A study of the effect of r elap se prevention training and transfer climate Huma n R es ource Management, 38 (3) 227-242 Campbel D T., & Stanley J. C. (1963) Expe riment a l and quasi-experimental designs for researc h Chicago: Rand McNally College Publishing Company Campbell, J.P. (197 1). Personnel training and dev elopment In P Mussen (Ed. ) Ann ual Review of Psychology (pp 56 5-602 ) Palo Alto, CA: Annu al Reviews I nc C a nnon-Bowers J. A Rhodenizer L Salas, E., & Bowers, C. A (1998) A framework fi r understanding pre-practice con d itions and thei r impact on learning Personnel Psychol o gy, 51 (2) 29 1-320 Carnevale A P, Gainer L. l & Villet, J. (1990) Training in America: T he organization and strategi c role a,[ training San Francisco Jossey-Bass Publishers Chaffin M., Kelleher K Harber G Harper l, & Crone C. (1994) Im pa ct of substance abuse and c hil d maltreatment train in g on service utili zat ion in a rural set ting Jo urn a l ofC hild and Fam i ly Studies, 3 (4), 379-387 Cheung, K. M., Stevenson, K. M & Leung, P (1991) Competency-based evalu ation of case management skills in ch ild sexual abuse intervent ion Child Welfare, 7 0 (4) 425 43 5 Clark R E & Voogel A ( 1 9 85 ) T ransfer oftraining principles for instructional design Educational Co mmunication and Technology Journa l 33 ( 2) 1 13-123 271

PAGE 286

Colorado Department of Human Services ( 1998) Colorado Child Welfare Practice Handbook. Denver CO: Author. Cormier : M & Hagman l D (Eds ) (1987) Transfer of learning: C ontemporary research and applications San Diego CA Academic Press Cronin D. L. & Drake, B. (1995 March) Evaluation of child welfare competencies in social w ork education. St. Louis, MO: Washington University Cruz B J. ( 1997) Measuring the transfer of training Performance Improvement Quarterl y 10 (2), 83-97 Curry, D ( 1997) F actors affec t ing the perceived transfer of learn ing of child prote c tion social w orkers. Unpublished docto ral disserta t ion, Kent State Univers i ty of Akron OH C r ry, D H., Caplan P., & Knupel J. (1994) Transfer oftrainin g and adult learning Journal of C o n t inui n g Social Work Educat ion, 6 (1) 8 14 Darou, W ( 1990, March). Training the people who help troubled kids Training & De v elopm e nt Journ a l 54-58 Dav ey, R.I., & Hill, J. (1995). A study ofthe variability oftraining and beliefs among professionals who interview children to investigate s u spected sexual abuse Child Abu e and Neglect 19 (8) 933-942. Dele w s ki, C. H Pecora P J., Smith G., & Smith Jr. V J. (1986). Evaluating Child Protective Services t r aining : The participant actio n p l an approach Child We( fare, 65 (6), 579-591 Dett erman D K ( 1993 ) The case for the pro ecution : Transfer as an epiphenomenon I n D K. Detterman & R. l Sternberg R. l (Eds ) Transf e r on trial : Intelligence, cognition, and instruct i on (pp 1-24) Norwoord. NJ. Ablex Publishing Corp D i onne, P ( 1996). The evaluation of tra i ning act i vities : A complex issue involving different stakes Human Resource Development Quar t erly 7 (3) 279-286 272

PAGE 287

Di x on r M ( 1990) The relationship between trainee responses on participant reaction forms and postte s t sc o re Human Resource Development Quarterly, 1 (2) 129-137 Druc k er P F. ( 1954). The pra c t i ce ofmanagement. ew York Harper & Row. Dunn M G., Norburn, D. & Birley, S (1994) The impact of organizational values goals and c l imate on marketing effectiveness Journal of Business R es e a r c h 30 131-141. Ellis H C (1965). The transfer of learning New York: The Macmillan Company Fac t eau l D Dobbins G H., Russell J. E A Ladd R. T & Kud i sch, l D ( 1 995) The influence of general perceptio n s ofthe training environment on pretraining motivation and perceived training transfer. The Journal of Management 25 (1), 1-25 F ish e r S & Ford l K ( 1998) Differential effects of learner effort and goal orientation on two learning outcomes Personnel Psychology 51 (2), 39742 0 Fle i shm a n E. (I 953 ) Leadersh i p cli mate human relations training and supervisory behavior P e rsonn e l P s ychology 6 205 222 Follett M P ( 1992) The gi v ing of orders. In J. M Shafritz and A C Hyde (Eds ) C lassics of public administration (pp 66-74) Belmont CA : Wadsworth Publishing Company. F ord l K Quinones M A Sego D l & Sorra l S (1992) Factors affecting t he opportuni t y to perform trained tasks on the job Per s onnel Psy ch o logy, 45 (3) 511-527 F ord l K Weissbein D A Smith E. M., Gully S M., & Salas E. (1998) The r elationships of goal orientat i on metacognitive activity and p r actice s t r a tegies w i th learning outcomes and transfer. Journal of Applied P s ychology 8 3 (2). 218 233. F o x on M ( 199 7) The influence of motivation to transfer, action planning and manager support on t he transfer process Pe1jormance Improvement Quarterl y, 10 (2), 83-97 273

PAGE 288

Gantt H L. (1916). Industria/leadership. New Haven : Yale Universit y Press Garavaglia P L ( 1993, October). How to ensure transfer of training. Training & Development, .:17, 63-68 Garavaglia, P L. (1996). Applying a transfer model to training. Performance and Instruction 35 (4), 4 -8. Georgenson D L. (1982). The problem oftransfer calls for partnership Training and Development Journal, 36 (1 0), 75-78 Gick M L., & Holyoak K l (1987) The cognitive basis ofknowledge transfer InS. M Cormier & J.D. Hagman (Eds. ), Transfer of learning: Comemporary research and applica tions (pp. 9 46) San Diego CA Academic Press. Gilbreth L. M (1914). The psychology ofmanagement. New York: Sturgis and Walton. Gist M. E. (1989). The influence oftra i ning method on selfefficacy and idea generation among managers. Personnel Psychology, -12 (4), 787-805 Gist M E., Stevens C K & Bavetta, A. G. (1991). Effects of self-efficacy and post-training interv ention on the acquisition and maintenance of complex inte rper sonal skills. Personnel Psychology, -1.:1 ( 4), 83 7-861. Golden, K. A. (1992). The individual and organizational culture Strategies for action !n highly-ordered contexts. Journal of Management Studies, 29 ( 1 ), 1-21. Golembiewski R. T., & William E. B. (1978) Organizational development in public administration. ew York: Marcel Dekker, Inc Golembiewski, R. T (1978). Organization development in public agencies In R. T. Golembiewski & E. B. William (Eds. ) Organizational development i n public administration. New York Marcel Dekker, Inc. Golembiewski R. T & Kiepper, A (198 8) High performance and human costs: A public secto r model of organizational development. New York Praeger Ptlblishers. 274

PAGE 289

Gordon, J. ( 1991). Romancing t he bottom line In J. J. Phillips (Ed .), Handbook of training evaluation and measurement methods (2nd ed) (pp. 31 -42) Houston, TX: GulfPublishing Company. Gordon, G. G., & DiTomaso, 1 (1992). P redicting corporate performance from organizational culture Journal ofA1anage ment Studies, 29 (6) 784-798 Gregoire, T. K (1994) Assessing the benefits and increasing the uti l ity of addiction training for public child welfare workers A pilot study Child Welfare, (7 3) I, 69-8 1 Gregoire T. K., Propp, J., & Poertner, J. (1998) The supervisor's role in the transfer of training Administration in Soc ial Work 22 (1), 1-32 Hand H H., Richards M D. & Slocum J. W., III. (1973). Organizational climate and the effectiveness of a human relations training program. Academy of Management Journal 16 (2) 185-195 Hicks, W D .. & Klimoski R (198 7) Entry into training programs and its effects on traini n g outcomes. Academy of Management Journal, 30 (3) 542-552 Holcomb H ( 1 994) Ma k e training worth eve1y p e nny San Die go, CA: Pfeiffer & Co. Holton, E F III ( 1996) The flawed four level evaluation model. Human R eso urce Development Quarterly, 7 ( 1 ) 5-21. Holton, E. F., Bates, R A., Seller D L., & Carvalho M B. (1997). Toward co nstruct validation of a transfer climate instrument Human Resource Development Quarterly, 8 (2), 95-113 Buczynski, A. A & Lewis, J. W (1980) An empi rical study into the learning transfer process in management training Journal of Management Studies, 17, 227-240 Johnson D & Johnson, R. ( 1998) Effective staff development in cooperative l earnin g : Training, transfer and long -term use In C. M Brody & N Davidson (Eds. ) Professiona l development for cooperative learning. Albany, NY: State University ofNew York Press 275

PAGE 290

Johnson, S (1998). Who moved my cheese?. New York G P Putnam's Sons Jones J. F, St ev enson K. M., Leung P, & Cheung, K. M. (1995). Call to competence: Child prot ective services train in g and evaluation Englewood, CO American Humane Association Katz D, & Kahn, R L (1966) The socia l p sycho logy C?! organizations. New York: John Wiley and Sons. Kaufman, R & Keller, l M (1994). Levels of evaluat ion: Beyond Kirkpatrick. Human Resource Development Quarterly, 5 (4), 371380 Khademian, A M. (1997, October). In the long term is silly putty" managea b le? Looking for the links between organizational culture management and institutional context. Paper presented at the meeting of the Fourth National Public Management Research Conference, The University of Georgia Athens GA Kirpatrick, D (1959) Techniques for evaluat i ng training programs Journal C?! the American Society of Training Directors, 13 (39), 21-26 Kirkpatrick, D. (1983, November). Four steps to measuring training effectiveness Personnel Administrator, 19-25. Kir' patrick, D. L ( 1996). Invited reactio n Reaction to Holton a rticle. Human Resource Development Quarterly, 7 (1) 23-25 Kline, P (1994). An easy guide to factor analysis. ew York : Routledge. Kozlowski S W. l (1997). A multilevel organizational systems approach for the implementation and transfer of training. In : J. K Ford (Ed.), Improving training effectiveness in work organizations (pp. 247-287). Mahwah, Nl Lawrence Erlbaum Associates, Publishers Krein T. l, & Weldon K C (1994, April). Mak in g a play for training evaluation Training and Development Journal, 62 67 Lahiry S (I 994, April). Building commitment through organizational culture Traming & Development, 50-52 276

PAGE 291

Lim, D. H. (1999). Organizational and cultural factors affecting international transfer oftraining. Performance Improvement, 38 (3), 30-36 Linacre l M (1994) Many-facet Rasch measurement. Chicago Mesa Press. Macaulay C & Cree, V I. (1999) Transfer of learning : Concept and process. Social Work Education, 18 (2), 183-194 Machin M A & Fogarty G. l (1997). The effects of self-efficacy motivation to transfer and situational constra ints on transfer intentions and transfer of training Performa n ce Improvement Quarterly 10 (2), 98-115 March, J. G., & Simon H. (1958). Organizations. ew York: John Wiley & Sons Inc Marcoullides, G A, & Heck R H. (1993). Organizational culture and performance : Propo ing and testing a modeL Organization Science 4 (2), 2 09 -225 M artin D ( 1989) The guide to the foundations ofpublic administration. New York : Marcel Dekker, I nc Marx, R. ( 1986). Improving management development through relapse p r e v ention strategies Jou rnal of Management Development 5 (2), 27 -4 0 McGregor, Douglas. (1960) The human side of enterprise. New York M c Graw-Hill. Meek V L (1988). Organizational culture Origins and weaknesses Organization Studies, 9 (4) 453-473 Miles M. B. (1965). Changes during and following laboratory training : A clinical experimental study. The Journal qf Applied Behavioral Science 1 215 242 Newstrom, l W (1985). Leveraging management development through the management of transfer Journal ofManagement Deve l opment, 5 (5) 3344 Newstr om l W (199 7) Invited Reaction Progress or Relapse ? Human Resource Development Quarte rly 8 (2) 129-136 277

PAGE 292

Nunnally, J C (1978). theory (2nd ed .) New York : McGraw-Hill Osgood, Dorothy (1995, S ep tember). The effec t s of training on se r v i ce de live1y. Paper presented a t the meet i ng o f the Nat io nal Association ofWelfare R esea r ch and Statistics, Jackson, WY. Ohio Child Welfare Traini ng Program Evaluation Subcommittee (1993). Goal oriented assessment of learning, satiifaction, rel ev ance, application, and trai ing enviro n me nt. Akron OH Author. Olivero G, Bane, K. D, & Ko pelman R E. (1997, Winter) Exec u ti ve coa ching as a t ransfer oftraining tool Eff e ct s on producti v it y i n a publi c agency. Public Personnel Management 26 (4) 461469. O'Toole, L l (1997, Oct ober ) Different public managements ? Implications of structural context in h i erarchies and networks. Paper presented at the mee tin g of the Fou rth National Public Management Research Conference The U n iversity of Georgia Athens GA Ottoson l (199 7). Bey ond tra n sfer oftraining: U sing multiple le nses t o assess community education programs In A D Rose & M. A Leahy (Eds ) Assessing adult learning in diver se settings: C urrent i ssues and approaches (pp.87-96) San Francisco Josse y-Bass Publish e r s Parry C (1999 ) Eva lua tion o f CDHS Specialized Interviewing Skills for child welfare workers training Englewo od CO American Humane Association. Parry C & J. (1999) T rainin g eva luation in the human servi ces. Washingto n D C : American Public Huma n S er vice s Assoc iatio n Parry S (1990, May). Ideas for improvin g transfer of training Adu lt Learning, 19-23. Peco ra, P Dodson A Tea th er, E, & Whittaker l (1983) Assessing work e r training nee ds: Use o f staff surveys and ke y informant interviews. Child Welfa re, 62 ( 5 ), 395-407 278

PAGE 293

Pecora, P J, Delewski C. H. Booth, C., Haapala, D., & Kinney J (1985). Home-based, family-centered services: The impact oftraining on worker attitudes. Child Welfare, 6-1 (5) 529-540 Peters, T & Waterman, R H. (1982) In search of excellence. New York Harper & Row. Pettigrew A M. ( 1979) On studying o rganizational cultures Administrative Sci e nce Quarterly, 2-1, 570-581 Phillips, J. J. (1991). Handbook of training evaluation and measurement methods (2nd ed ) Houston, TX Gulf Publishing Company. Phye, G. D & Sanders C.E. (1992). Accessing strategic knowledge: Individual differences in procedural and strategy transfer Contemporary Educational Psychology, 7 21 1-223 Rae, L. (1991). How to measur e training effec!ive ness (2nd ed ) Brookfield VT Gower Publishing Company Rouillier l Z & Goldstein I. L. (1991). Determinants ofthe climate for t r ansfer of training Paper presented at the Meeting of Social Industrial Organ izat ional Psychologists Rouiller, J. Z., & Goldstein, I. L. (1993). The relationship between organizational transfer climate and positive transfer of training Human Resource Development Quarterly, 4 (4), 377-390. Rubin A., & Babbie, E (1989) Re s ea r ch me thods for socia l work. Belmont, CA: Wadsworth Publishing Company. Russ-Eft, D, & Zenger, l H. (1985 April) Common mistakes in evaluating training effectiveness Personnel Administrator, 57-62 Saari L. M Johnson, T R McLauglin, S D & Zimmerle, D M. (1988) A survey of management training and education practices in U.S compames. Personnel Psycho l ogy -11 (4) 731-743 Salomon G & Perkins D. N (1989) Rocky roads to transfer: Rethinking mechanisms of a neglected phenomenon. Educational P ,ychologist, 2-1 (2), 113-142. 279

PAGE 294

Schein, E. H (1990). Organizational culture A merican Psychologist 45 (2), 109-119. Schinke, S P Smith, T. E., Gilchrist, L. D, & Wong, S E. (1981, Winter) Measuring the impact of continu ing education Journal of Education for Social Work, 17 (1), 59-64. Seaburg, J R (1982, Septemb r). Getting there from here : Revitalizing child welfare training Social Work, 441-447. Senge P M. (1990). The.f1fth di. cipline: The art and practice of the learning organization. ew York: Doubleday Sheridan, l E. (1992). Organizational culture and employee re t ention. Academy of Management Journal, 35 (2) 1036-1056. Simon, H A ( 1945). Administrative behavior: A study of decision-making processes in administrative organ izati ons. New York The MacMillan Company Smircich, L. ( 1983). Concepts of culture and organizational analysis. Administrative Sc ienc e Quarterly, 28, 339-358 Stevens, C K., & Gist, M. E. ( 1997). Effects of self-efficacy and goal orientation training on negotiation skill maintenance: What are the mechanisms? Personnel Psychology 50 (4), 95 5-978 Stokking, K., M. (1996) Levels of evaluation: Kirkpatrick, Kaufman and Keller and beyond. Human Resource development Quarterly, 7 (2), 179-183. Sullivan R., & Clancy T (1990). An expe r imental evaluation of interdisciplinary training in intervention with sexually abused adolescents. Health and Social Work, 15 (3), 207-214. Tannenbaum, S I., & Yuki, G Training and development in work organizations Annual Review of Psycholo gy, -13, 399-441. Taylor, F (1992). Scientific management In J. M. Shafritz and A. C Hyde (Eds.), C lassics of publi c administration (pp 29 30) Be l mont, CA: Wadsworth Publi hing Company 280

PAGE 295

Taylor, P (199 2) Training d irectors' perceptions about the suc cessful implementation of supervisory training. Human Resource Development Quarterly, 3 (3), 243 -259. T esluk P E, Farr, J. L. Mathieu, J. E. & Vance R J. (1995). Generalization of employee involvement training to the job setting : Individual and situational effects Personnel Psychology, -18 (3), 607-632. Thompson, l D (1967). Organizations in action New York : McGraw Hill. Tracey J. B. Tannenbaum, S I., & Kavanagh M J. (1995). Applying trained skills on the job: The importance of the work environment. Journal of Applied P:-,ychology, 80 (2) 239-252. Vestal K. W, Fralicx, R D., & Spreier S W (1997). Organizational culture The critical link between strategy and results Hospital & Health Services Administration, -12 (3) 339-365 VinokurKaplan, D. (1986 Winter). National evaluation of in-service training by child wel fare practitioners Social Work Research & Abstracts, 22 (4), 13-18. Warren, A. R., Woodall, C.E. Nunno, T. M. Keeney, M., Larson, S., & Stadfeld J. C n press) Assessing the effectiveness of a training program for interviewing child witnesses Applied Deve l opmental Science. Weekley, J., & Jones, C. (1997) Video-based situational testing Personnel P syc hology 50 (1 ) 25-49. Wiener, Y., & Vardi Y. (1990). Relationships between organizational culture and individual motivation-A conceptual integration. Psychological Reports 67, 295-306. Wilson W ( 1887 July) The study of public administration Political Science Quarterly 2, 197-222 Wright, B. D., & Stone, M. H. (1979) Best test design. Chicago Mesa Press. 281

PAGE 296

Xiao Jin. (1996). The re la tionship between organizational factors and the transfer of training in the electronics industry in Shenzhen China Human Resource Development Quarterly, 7 ( 1 ), 5 5-73. Yancey, G B & Kelly, L, (1990). The inappropriateness ofusing participants reac tions to evaluate effectiveness of training Psychological Reports, 66, 937-938. Yorks L O'Neil, J., Mars i ck, V. J., Lamm, S, Kolodny, R., & Nilson, G (1998). Transfer of learning from an action reflection learning program Performance Improvement Quarterly, 10 (2), 83-97 Ziglar, Z (1994) Over the top. Nashville, TN: Nelson Zunz S ( 1995). The view from behind the desk: Child welfare managers and their roles Administration in Social Work, 19 (2) 63-80. 282