FAIRNESS AND PERFORMANCE APPRAISAL:
THE MULTISOURCE ASSESSMENT PROCESS
Ann J Ewen
B.A., University of North Dakota, 1979
M R.A., Arizona State University, 1*980
A thesis submitted to the
Faculty of the Graduate School of the
University of Colorado at Denver
in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
1994 by Ann J. Ewen
All rights reserved.
This Thesis for the Doctor of Philosophy
Ann J. Ewen
has been approved for the
Graduate School of Public Affairs
/o /99 y
Ewen, Ann J. (Ph.D., Public Administration)
Fairness and Performance Appraisal: The Multisource Assessment Process
Thesis directed by Assistant Professor Linda deLeon
Performance management is an important factor of organizational life. It
helps organizations identify high-performing individuals and reward them, as well as
identify low- and non-performers for intervention and development. However, it does not
work as well as theory suggests. The literature indicates widespread dissatisfaction with
the traditional single-source (supervisor-only) process, particularly regarding the potential
for unfairness in the process.
The EEOC published guidelines in 1964 calling for fairness in
performance appraisals and other selection decisions. In 1978, the Uniform Guidelines
on Employee Selection procedures legislated that organizations use the fairest
performance appraisal process available. There is, however, little documentation
regarding fairness in performance appraisals. While a number of theories of fairness
exist, applications of these in organizational settings are limited.
Multisource assessment provides an alternative to traditional supervisor-
only appraisal processes. The multisource model recommends that performance
information come from multiple sources, such as work associates, who interact with the
assessment receiver. This field study was undertaken to determine whether multisource
assessment (MSA), appraisal from the supervisor and work colleagues, is perceived as
fairer than traditional supervisor-only appraisal systems.
A pre- and post-test survey was used to collect information from
assessment receivers in a large federal agency site on their perception of process fairness.
The pre-test was given to participants after the completion of their traditional supervisor-
only assessment. After multisource assessment, a post-test assessed the perceived
fairness of that process.
T-tests indicated that the changes in agreement scores on fairness from
pre- to post-test results were significant, and respondents (regardless of age, gender,
ethnicity) perceived the MSA process to be fairer. Thus, the multisource appraisal
process was found to provide a new approach that enhanced participants perceptions of
performance ratings. Additional hypotheses were tested regarding the accuracy of the
process, usefulness of the information, confidence in the assessment results, and process
safeguards. Additional research is warranted regarding longitudinal impacts and results
in other organizational settings. This research project provides the benchmark model
demonstrating that multisource appraisal is a fairer performance appraisal method.
This abstract accurately represents the content of the candidates thesis. I
recommend its publication.
CHAPTER 1 PERFORMANCE APPRAISAL IN CONTEMPORARY
Contemporary Organizational Challenges...........................2
A Changing Environment.........................................4
Cultural Changes and the Changing Workforce....................9
Public Organizations Are Changing...............................12
Importance of Performance Appraisal.............................13
The Evolution of Performance Appraisal..........................14
Problems of Current Appraisals..................................21
Alternative Appraisal Models..................................25
CHAPTER 2 THE SPECIAL QUESTION OF FAIRNESS.........................30
Performance Management in the Public Sector.....................31
A Higher Standard of Fairness...................................35
Mandate for Fair Performance Appraisals.........................37
The Psychological Contract....................................39
Legal Mandate for Fairness....................................41
Fairness as a Desired Objective.................................43
Need for Fairer Performance Appraisals........................49
Importance of This Research...................................50
CHAPTER 3 THE MULTISOURCE ASSESSMENT MODEL: RESEARCH
QUESTIONS AND HYPOTHESES...........................................53
MS A-An Alternative to Single-source Appraisal..................55
Challenges to Implementing MSA..................................63
MSA as a Fairer Performance Appraisal Alternative...............64
CHAPTER 4 A NEW PERFORMANCE APPRAISAL PROCESS IN A
Description of the Organization.................................72
Designing the MSA Process.......................................74
Steps of the Multisource Appraisal Process....................77
Overall Research Design.........................................89
Dependent and Independent Variables...................................90
Pros/Cons of Research Design..........................................91
Evaluation of Perceptual Fairness........................................92
Survey Research-Scope and Limitations....................................95
Multisource Process Reliability and Validity.............................98
CHAPTER 5 DATA ANALYSIS AND RESULTS........................................103
Data AnalysisSurvey Results............................................105
Understanding of the Process.........................................119
Additional Areas of Interest.........................................122
Predictors of Fairness Using Discriminant Analysis...................122
Impact of Training on Perceptions of Fairness........................123
Additional Interview Comments........................................126
Cross-Tabulations and T-Tests........................................128
Groupings of Variables Using Factor Analysis.........................132
Additional Survey Comments: Why?...................................135
Data Analysis: Empirical Results........................................137
CHAPTER 6 RESEARCH OUTPUT AND ITS SOCIAL IMPACT............................145
Discussion of Findings..................................................150
Implication of Study for Applied Use....................................156
Recommendations for Further Research....................................159
Appendix A Behavioral Competencies......................................166
Appendix B Sample Reports...............................................169
Appendix C Pre- and Post-Implementation Survey.................175
Appendix D Discussion Guide....................................181
Appendix E Comments............................................186
Pressures Driving Changes in Performance Appraisal.................16
Comparison of Feedback Systems.....................................56
Single-Source Ratings Distribution for All Employees...............138
Composite Ratings for All Employees................................141
Non-Standardized Supervisory Score for All Employees...............144
Standardized Supervisory Score for All Employees
LI. Rating Perspectives..................................................58
5.1. Evaluation Survey Results...........................................107
5.2. Surveys of Federal Agency...........................................108
5.3. Changes in Agree Scores...........................................109
5.4. Results at 33rd/66th Percentiles....................................110
5.5. Pre-/Post-Implementation Survey Comparisons.........................Ill
; 5.6. Chi-Square Significance.............................................113
: 5.7. Factors of Fairness.................................................123
; 5.8. Impact of Training on Perceived Fairness............................124
: i.9. Comparison of Fairness Evaluation...................................129
! i.10. Correlation Analysis.................................................131
5.11. Factor Analysis........................................................133
5.12. Cluster Analysis.....................................................134
! i.13. Distribution of Ratings by Gender, Ethnicity and Age.................138
t >.14. Ratings for Single-Source vs. Multisource Appraisal
I wish to thank my committee, consisting of Professors Linda deLeon, Kathy
Boyd, Joe Cayer, Mike Cortes and Richard Stillman, for their assistance in focusing my
research efforts. The collegial process is what I was seeking in my decision to pursue
additional graduate work, and they each played important roles in providing that. In
particular, Professor deLeon (in her role as chair) provided guidance that totally exceeded
my expectations. She exhibited tremendous dedication to her task of providing input and
suggestions to me in my efforts. This greatly enhanced my learning in this process, and I
am very appreciative.
In addition, family always provides a tremendous role in ones accomplishment of
such magnitude. The drive for on-going education was instilled at an early age by my
parents, Clark and Jo Ewen, which was of critical importance. I am grateful for that
Finally, my husband, Mark Edwards, assisted me both intellectually and, much
more importantly, emotionally in my quest. For that, my love and thanks.
IN CONTEMPORARY ORGANIZATIONS
Performance appraisal is the key to effective human resources management. In
their influential discussion of performance appraisal, Latham and Wexley (1982)
identified selection, appraisal, training and motivation as the four key systems necessary
for insuring the proper management of an organizations human resources. Of these four,
they argue, performance appraisal is the most important because it is the prerequisite for
establishing the other three (p. 3). The salience of an effective performance appraisal
system as a tool to enhance human productivity is well documented (Gleuck, 1978).
Appraisals serve a critical function in organizations because they enhance productivity by
targeting individual development needs and, at the same time, identifying high, medium
and low performers (Bemardin and Beatty, 1984; Latham and Wexley, 1982; and
But there is substantial evidence that current performance appraisal processes do
not work well. Managers do not like them, employees do not like them, and they dont
produce measurable effects on productivity. This chapter will develop three major
arguments: first, that public organizations are changing, driven by rapid changes
occurring in two vital areasthe workforce and the environment. Second, a review of the
evolution of performance appraisal will show that appraisal processes have not altered in
evolution of performance appraisal will show that appraisal processes have not altered in
ways that are adaptive to these organizational changes. Finally, the problems of current
performance appraisal processes will be discussed and a variety of alternatives currently
on the table will be described.
Contemporary Organizational Challenges
A variety of changes are occurring in American organizations, both public and
private. These large-scale organizational changes are causing many rules to be re-
evaluated (Mohrman et al., 1988), a re-evaluation which raises questions about the
performance appraisal process. Changes driving organizations include those both
environmental and cultural.
Changes in the environment, such as technology and declining global resources,
reflect an overall more rapid pace of change throughout society. Competitiveness exerts
pressures to improve resource efficiency, a virtue of what Denhardt (1982) has called the
rational model. In the rational model, efficiency is the preeminent value, and a
hierarchical organization structurebureaucracyis regarded as the most likely to foster
The rational organizational model, with its emphasis on bureaucratic control, is
often inconsistent with the objectives of reinvention initiatives, however. Cultural
effects, such as environmental changes, require organizational changes. In addition,
todays diverse and changing workforce, for example, resists hierarchical forms of
organization and demands more participation in the decision-making process. Workforce
changes center on the individuals expectations of the employee/employer contract.
These expectations might revolve around issues such as loyalty, equity and diversity.
According to some theorists (Kanter, 1983; Morgan, 1986), organizational culture shapes
how people act within the organizational setting. This enactment view of culture holds
that organizational reality is socially constructed (Morgan, 1986). This reality is
reinforced through slogans, mission statements, and job descriptionsall of which help
the employee behave appropriately. An individuals values, norms, and perceptions of
organizational reality come from interactions with role set members (Katz and Kahn,
1966). These role set members, including a persons immediate work group, immediate
supervisor, and immediate subordinates, are important influences on an individuals
effectiveness in an organization (French, Bell and Zawacki, 1989). So organizational
culture has an important bearing on the assessment process. The velocity and variety of
change are factors that raise more questions about the rational approach to organizations
and the performance appraisal process. The rational model presents a strong value
orientation to efficiency by facilitating the organizations achievement of its goals. It is
anticipated that the organization still wants to pursue its goals of efficiency. Meeting the
expectations of the public interest requires responsible use of public resources to
The following section provides additional discussion on the changing
environment and cultural aspects of organizations. These challenges require balancing the
public interest in efficiency with employee interests in fairness. These changes provide a
framework for the ensuing discussion regarding fairness and how it enhances
A Changing Environment
Major environmental factors which are changing include: enhanced technology,
the use of self-directed work teams, the quality movement, the drive to increase public
sector productivity, and leaner organizations. These are discussed in the following
Impact of Technology. The impact of changing technology has an effect both on
employees access to information and on their ability to manage large amounts of data.
Contemporary workers often have extensive knowledge or access to technology-based
information. As Drucker notes, These types of workers are not like the apprentices in
the past who learned all they need to know about their job by age 17. .. Todays workers
need to continuously learn and update their skills every five years ... or be obsolete
(Martinez, 1994). These knowledgeable workers may oversee day-to-day responsibilities,
have customer contact responsibilities, and be involved in managerial decision-making.
Their access to information has changed the power relationship within the organization-
no longer do the managers have a higher degree of knowledge and technical skills than
their direct reports (subordinates).
The availability of technology, particularly computers, also presents an
opportunity to manage large amounts of data efficiently. The massive amounts of
information required for an effective Human Resources Information System exceed the
capability of traditional filing systems. Technology offers to make human resource
management processes better (Nadler, 1977) through increased speed, increased
anonymity to assessment providers and recipients, and statistical safeguards to minimize
bias and enhance assessment accuracy in performance appraisal (Edwards, 1983).
Technology also provides opportunities for access to increasing amounts of information
and the accompanying need to manage it (Benveniste, 1994).
Self-Directed Work Teams. The increasing use of self-directed work teams
drives new adaptive responses (Mohrman, Mohrman and Lawler, 1987). Traditionally,
managed work groups involve a supervisor and the workers: the supervisor gives the
orders; the workers carry out those orders. Self-directed work teams provide a new model
for getting work accomplished. The self-directed work team developed from quality
initiatives, which fostered employee participation. Theory Z calls for employee input and
multidirectional improvement (Ouchi, 1981). Members of the self-directed team manage
the work themselves, without the presence of a supervising manager. This structure gives
employees more control over their work lives.
Self-directed work teams create new challenges for organizational support, such
as enhancing workers dedication to the success of the team. In order for self-directed
work teams to be effective, new processes need to be adopted within the organization
(Nebeker and White, 1990). One significant requirement for managing self-directed work
teams effectively is ensuring effective team leadership. The leaders role is to empower
the team and its members to lead themselves. Teaching and coaching behaviors are the
actions that typify this new leadership. Team members, not only designated leaders, need
strong skills and sometimes additional training to lead teams effectively.
Performance assessment becomes a critical factor for the self-directed team.
Performance measures are needed to identify the degree to which the team is meeting its
goals and objectives, and who within the team is and is not contributing to that success.
These changes require new approaches to the traditional performance appraisal process,
which relies on the supervisor alone to make the performance assessment.
Quality as an Objective. The quality movement (often referred to as TQM) has
created substantial organizational change, including the new expectation for assessment
from customers, both internal and external. Often, however, public organizations need
assistance in identifying their customers. The Federal Total Quality Management
Handbook (Lewis, 1991, pp. 16-17) identifies the essentials of a public agency quality
initiative as: (1) flatter, more flexible, less hierarchical organization structure, (2) focus
on continuous improvement in systems and processes, (3) workers see supervisors as
coaches, facilitators,( 4) supervisor-subordinate relationship of interdependency, trust,
and mutual commitment, (5) focus of employee efforts on team effort (where workers see
themselves as teammates), (6) management perceives labor as asset and training as
investment, (7) organization asks customers to define quality, develops measures to
determine if requirements are met, and (8) primary basis for decisions shifts to facts and
systems. These are significant changes from the rational model of organizations which
reinforces a hierarchical model of interaction between employees based on rank.
Customers impact organizational direction by purchasing products and services for use.
Quality initiatives focus on the exchange of goods and services, with internal clients
recognized as an important source for defining expectations.
TQM emphasizes the notion that giving and getting feedback on a regular basis is
critical to organizational success. It also rejects many of the fundamental realities of the
hierarchical bureaucracy by opening up communication and dispersing power. A focus
on quantitative measures moves the organization from management-by-intuition to
management-by-numbers. Continuous improvement can be measured only if the starting
point is accurately assessed and subsequent improvements identified relative to that
Bowman (1994) has recently argued that TQM is the appropriate alternative to
performance appraisal. He asserts that continuous feedback will enhance performance of
individuals with a TQM process. His approach argues for a team-based approach to
productivity, and assumes that if the team succeeds, all individuals must be contributing
to their fullest. The TQM approach to performance appraisal shifts the focal point from
individual assessment to system assessment. Rather than the supervisor providing the
sole judge role, the supervisor acts as a coach, with peers, colleagues, and customers also
sought for input. The intent is to create a model for frequent feedback and appraisal.
This argument overlooks the point that without individual feedback, it is possible
that some individuals may hide behind the team with no accountability or requirement
to perform. Theories of motivation also recognize that, while team accomplishment is a
tremendous incentive, individuals also desire personal feedback (Where are my
strengths? where should / improve?).
TQM processes do not provide individual feedback. In fact such feedback is
repudiated as creating fear (Deming, 1986). A failure of TQM programs is that they do
not provide tools for developing individuals to continuously improve human performance.
Much of the TQM literature suggests doing away with individual performance appraisals,
as argued by Deming proponents (Deming, 1986). These arguments suggest that
organizational processes are a better focal point than the individual. Alternatively, it is
suggested that an appraisal process is needed that incorporates performance management
with the philosophy of TQM (Long, 1991). Rather, a systems approach to performance
management that replaces MBOs and other traditional approaches may be the solution. It
is important that individual accountability be present even within the TQM organization.
Increased Drive for Public Sector Productivity. In the federal bureaucracy, a
reevaluation of the status quo is the result of the current drive to increase public sector
productivity. Taxpayers suspicion that their dollars are not being spent well comes at a
time when the need to spend more on such categories as social services, public safety, and
highway infrastructure has increased (DeWitt, Kettle, Dyer and Lovan; 1994). Balancing
such a difficult equation requires increased efficiency and productivity.
This refrain, the call for increased efficiency and productivity, sounds familiar to
public administrators. It is consistent with rational models of organizations, Woodrow
Wilson (1911) first called for government to be run more like a business by increasing its
efficiency. In these days of tax and expenditure limits, efficiency means that more or
better is done with less (Rosen, 1993, p. 6). In order to be successful, two things are
criticalchoosing the right technique and obtaining the necessary cooperation from the
stakeholders to realize the change (p. 7). Both are important for a successful productivity
improvement effort. In order to obtain the necessary cooperation, it is necessary to link
organizational and individual productivity. The performance appraisal facilitates this
Leaner Organizations. Organizations are becoming flatter as costs increase and
budget dollars decline. Competitionglobal and domesticrequires American businesses
to increase profitability, decrease employee alienation and lack of motivation, and bolster
lagging productivity (Kanter, 1983). Downsizing or delayering is undertaken to reduce
costs. This leads to greater spans of control for supervisors of 1:20, 50 or 100 (Edwards,
1989). Comprehensive and knowledgeable performance feedback appraisals are difficult
in many supervisory situations.
Cultural Changes and the Changing Workforce
Just as the workplace environment is changing in significant ways, the workforce
iteself is responding. These changes include resistance to authority; greater diversity;
changing employer/employee relationships; and the trend toward treating the workplace
as a community. Consequently, the contemporary organization requires new models for
Resistance to Traditional Authority. The bureaucracy was attacked in the
1960s as dehumanizing, which set the stage for organizational change (Maccoby, 1988).
Typical contemporary employees are better educated than were their American
counterparts in the Industrial Age. Workers are no longer satisfied with being cogs in the
organizational machine. Employees, at all levels, want more power and autonomy. There
is . a rise of rights consciousness among employees of all kinds (Kanter, 1983, pp.
3-4). Kanter asserts that the social climate also has created an expectation of
participation and voice, with younger employees not as willing to accept strict
hierarchical control. This resistance to authority requires new managerial models for
achieving greater output, replacing the directive bureaucratic model. Traditional
performance appraisal processes have an inherent power/dominance due to the reliance
on the sole evaluator, the supervisor. In order to better respond to changing expectations
regarding employee involvement, appraisals must be revised to be more participative.
Greater Diversity. Litigation arising from perceived discrimination on the basis
of gender, age and race is on the increase. Frequently such factors as an employees age,
sex, or race may lie behind negative comments couched in objective terms (Latham and
Wexley, 1994). A study by the Rand Corporation in 1988 found that employees are three
times more likely to sue their employers than they were in 1980. This litigation
explosion occurs partially as a result of employees growing awareness of their legal and
contractual rights. Studies show that 50% of jurors in employment cases have felt
discriminated against at work (Fairness, 1994). Additionally, 84% of jurors believe
that older people are discriminated against as employees. These jurors tend to identify
with a disgruntled employee when asked to determine the merits of such a case.
Consequently, jurors look carefully at how fairly a company has treated employees in
reaching their decision.
The changing workforce (Johnston and Packer, 1987) will be made up of more
women and an increasing number of people of color. For organizations to be successful
in not only managing this diverse workforce, but retaining and promoting competent
people as well, changes are required in how the personnel process works. This includes
the performance appraisal process to ensure that gender, age, and ethnicity are not used to
discriminate against employees. The organizations that can effectively manage diversity,
not just as a legal mandate, but as a means of keeping the best qualified workers
motivated and productive, will be in a better position to grow and meet their clients and
customers needs. These organizations will see managing diversity as a business decision
and a competitive advantage (Thomas, 1990).
Companies must review their performance appraisal processes to avoid the cost
of litigation, which currently appears predisposed to favor employees. Managing
impressions of fairness (Greenberg, 1990) becomes important. Ensuring that the
personnel process incorporates factors that enhance participants sense of fair treatment
will help organizations avoid litigation.
Changing Emplovee/Emplover Relationship. All of these organizational
changes cited above affect the employee/employer relationship. As recently as ten years
ago workers could expect a lifetime, or near lifetime, of employment opportunity with the
same organization, particularly if that organization was in the public sector. This secure
and stable relationship is referred to as a psychological contract (Schein, 1980). It
represents an emotional bond between the organization and the employee. Organizational
changes affect the psychological contract, or emotional bond between employer and
employee (Schermerhom, Hunt, and Osborn, 1985). The psychological contract of
yesterdays organization was built on a supposition that both the employees and
employers responsibilities were specific, mutually interdependent and conducive to a
long-term relationship (DeMeuse and Tomow, 1992). This psychological contract was
particularly important in white-collar and management ranks where lifetime employment
was often a reality.
Those times have changed. Terms such as organizational layoff, downsizing,
rightsizing, and reduction in force (RIF) have become commonplace. Competitive
business factors are driving many of the tremendous changes occurring in American
business to increase profitability, including global competitive pressure and competition
in domestic markets (Kanter, 1983). In the public sector, pressure is raised by tax and
spending limits, which create similar organization demands, such as reduced head count
and limited budget resources. These limitations lead to employee alienation and lack of
motivation, which serve to drive lagging productivity.
The infrastructure of the bureaucratic organization must change to support the
new culture of empowerment invoked by environmental and cultural changes. The
psychological contract required by contemporary employees and organizations must be
flexible and fair while at the same time offering more of a short-term focus on the current
exchange of value (DeMeuse and Tomow, 1992). This new perspective of the
contemporary organizations psychological contract with employees has a significant
impact on the manner in which employees achievements are appraised and rewarded.
Organization as Community. Todays expectations for enhanced employee
performance and closer work relationships are different from those of just a decade ago.
The role of the workplace as a community does not arise so much because work
accounts for so much of a persons time, but because institutions that used to provide a
sense of community, such as the church, have become less important to people. Most
working people now spend the majority of their days working and interacting with co-
workers. The contemporary organization acts as a community of sorts for employees, and
thus requires a different model for appraisal-one that recognizes both interdependence
and individual contribution.
By responding to these massive organizational changes being posed, the
traditional values of the rational public organization can be supported. The rational
organization may benefit by using the social structure of the organization to better reach
its own ends of enhancing efficiency.
Public Organizations Are Changing
Public organizations are changing how they operate, so drastically in some cases
that the process is often called reinvention (Osborne and Gaebler, 1992). This term
connotes the drive to eliminate bureaucratic waste and foster a new spirit of
entrepreneurism within the public sector. A major impetus for this reinvention is the
publics changing expectations of government (DeWitt, Kettle, Dyer and Lovan; 1994).
Many observers hope that reinvented governments will not only be more productive, but
will provide a better workplace for their employees. In order to increase efficiency in
providing services, new methods and tools are required for enhancing employee
effectiveness, improving morale and assessing individual and team contribution.
The dual necessity of coping with rapid change while remaining competitive
poses significant challenges. It has produced a drive to decentralize and democratize the
workplace, in the hope of thereby achieving greater flexibility and creativity. This
presents substantial challenges to the performance appraisal process, which was originally
developed primarily as a tool for increasing worker efficiency and for motivating
individuals to be more productive. The performance appraisal process needs to be
reinvented to meet these new organizational conditions. The performance appraisal
process has gradually developed techniques to make it more participative, such as
Management by Objectives (MBO), and egalitarian, such as upward review feedback
processes. However, performance appraisals, as currently practiced, receive mixed
reviews as to their effectiveness.
Importance of Performance Appraisal
Performance appraisals serve to link organizational goals with individual job
performance. Without appraisals, organizations would not know whom to reward,
promote, train, discipline or retain. Appraisals, which lead to recognition and rewards,
can also target individual or group training needs and assist with administrative
placement, as well as enhance retention of key employees. Failing accurate performance
measures, the other human resource management actions occur almost at random or based
on supervisory intuition. Participants may feel that they are not being treated fairly under
these conditions, reducing morale and productivity.
Performance appraisal involves a series of steps. These are: to review legal
requirements, conduct the job analysis, develop the appraisal instrument, select observers,
train observers, measure performance, give employee feedback, establish performance
goals, praise/reward performance, and align process and outcome with organizational
justice principles (Latham and Wexley, 1994, p. 8).
The Evolution of Performance Appraisal
Performance appraisals have changed as organizations change. The time line in
Figure 1.1 summarizes these modifications and their relation to historical and
organizational events. The changes reflect a move from the rational view of organizations
to a more behavioral perspective that views the employee on a personal basis. The post-
bureaucratic model (Barzelay, 1992) represents a contemporary ideal model of an
organization that breaks down the hierarchy.
Performance appraisals evolved from a desire to increase productivity (Latham
and Wexley, 1994). To do this, the organization focused on motivating the individual to
produce more. The objective for much of the early development of performance
appraisals was increasing productivity by focusing on efficiency and output. McGregor
(1975) introduced Theory X and Theory Y of motivation. Theory X was rooted in the
rational approach to organizations. Workers needed strict direction in order to maximize
output. Theory Y, based on the behaviorist model, argued that human beings, when
properly motivated through recognition and reward systems, would outperform the
bullied worker of Theory X. Today it is recognized that these theories provided
simplistic models for understanding the performance/output equation; they nonetheless
capture much of the thinking evidenced in the early history of performance appraisals.
The rational organization seeks to increase efficiency, or use of resources.
Scientific Management (Taylor, 1911) was based on a structured approach to job design
that viewed the worker as a machine. Reasoning by analogy, it was an easy step to the
belief that routinizing this human machines actions would lead to increased productivity,
or output. The goal was to eliminate inefficiencies such as wasted movement, laziness or
slowness. By tracking results and focusing on output, employees were forced to
concentrate on production. The logical next step was to reward individuals based on
output, which provided the motivation to create higher output.
Time: 1900 1930 1950 1960 1970 1980 1990 2
i Organizational Planning
Counseling and Development
' ' m Improve Process
Subjective Appraisal (Essays, Traits, Competencies)!
Figure 1.1: Pressures Driving Changes in Performance Appraisal
The traditional work group evolved out of the Scientific Management movement
and bureaucratic models of organizational job design. The bureaucratic organizational
structure reinforces a rigid approach. Taylors influential Scientific Management
movement can be traced to the traditional work team (Bolman and Deal, 1988). Scientific
Management was the first treatment of management as a separate function. Previously
there had been work groups with foremen (first-line supervisors), but until Taylors time,
when industrialization had produced the first giant corporations, there was not widespread
need for the managerial ranks. In this scenario, management is in charge and directs the
work that is undertaken. Taylor (1911) also initiated time studies to identify the most
efficient method of accomplishing a job.
The bureaucracy was identified as the most efficient model for organizing work
and workers (Weber, 1930). This structure perpetuated the concepts of unity of command
and hierarchical control through domination. Whyte (1956) spoke of the Organization
Man, who gave all to his work. In return for his commitment to the company, the
employee, who was typically male and (at the executive level) white, could expect
lifelong employment. Consistent with the philosophy and structure, the appraisal process
traditionally came from the supervisor in a top-down manner.
The human relations movement was a countervailing approach in the 1920s and
1930s and recognized the importance of employees emotional and social needs (Follett,
1918). Asa result, concern for the individual increased. The human relations approach
focuses on increasing the workers integration into the organization, his or her job
satisfaction, and the contribution of each employee to increase efficiency. Rather than
individual employees treated merely as cogs in the organizational machinery, they were
recognized as individuals with emotions and ideas that contributed to the organizations
overall effectiveness. This shift recognized that the rational approach, which assesses
performance results only on how much is produced, does not address the human
components of interaction and interdependency, which leads to how the results are
References to performance appraisals begin in 1842 in the federal government
(Lopez, 1968), where the early performance appraisal was used predominantly in the
military setting for determining promotions based on merit (Swan, 1991). The methods to
determine merit generally relied on forced ranking and trait-rating scales. Forced ranking
consists of identifying objectives for performance, and identifying who is the highest
performer, then of the remaining work group identifying the highest performer, and so on,
until all employees are ranked. Employees sometimes react negatively to forced ranking
because it promotes a competitive atmosphere.
Trait rating evaluates employees on such aspects as personality, and modifiers for
various levels of personality are then displayed along the graphic rating scale. A failure
of this approach is that ratings are oftentimes lenient because administrators time is too
limited to permit careful discrimination and thoughtful judgment (Nigro and Nigro,
1981). Additionally, because traits are difficult to define, inter-rater agreement is
oftentimes low (Austin and Villanova, 1992), so the usefulness of trait rating is called
In nonmilitary organizations, the initial performance appraisals affected only
nonsupervisory personnel (which also served to reinforce managerial power). After
World War II, however, managers in private sector organizations began to receive
Management by Objectives (MBO), which focused on bottom-line results, was
introduced as an organizational planning and strategic integration tool (Drucker, 1954).
The process involves the individual who helps identify appropriate objectives. Today the
predominant type of appraisal in use in U.S. governmental agencies and private
businesses is some variation of MBO. Although MBO addresses problems regarding the
hierarchical and nonparticipative nature of performance appraisal, it is not perfect
because of its failure to provide a motivational or developmental tool for employees. In
one government survey, two-thirds of employees who responded indicated that MBO did
not support training and development activities (Daley, 1987).
Another type of appraisal process is the Behaviorally Anchored Ranking System
(BARS). Behavioral descriptions are defined for each degree of performance (e.g., good,
fair, poor), with specific examples of the behavior modeled by each rating. They are
widely used because the subjectivity of the rater is greatly reduced and the process is
effective for development because it clarifies employee expectations. Disadvantages
include that it may be too cumbersome to document the many possible varieties of
behavior, and matching employees behaviors with anchors is not always easy for raters
to accomplish (Latham and Wexley, 1994).
These methods of traditional single-source performance appraisal reinforce the
power that the supervisor and the organization hold over the individual employee (Folger,
Konousky, and Cropanzo, 1992). As an evaluator of employees, the supervisor exercises
a highly visible form of organizational power (Nalbandian, 1981, p. 395). Inherent in
the model is the fundamental orientation of supervisors watching workers do the work.
The supervisor judges performance and renders a verdict in the form of a performance
rating. The employee has no choice in supervisors, little choice in evaluation criteria, yet
must accept the rating with practically no recourse.
These classical theories of productivity and motivation, as described by rational
and behaviorist approaches, appear to be outmoded and incomplete in light of new
employee expectations and environmental concerns. Rather, values motivate behavior
and determine how and why we work. Additionally, people are motivated to perform
when the atmosphere is mutually supportive (Broterick, 1992). Post-bureaucratic
organization theory recognizes that new models are needed for effectively structuring and
managing organizations (Barzelay, 1992). Values serve an important function because
they form the social character of a work group.
The new employee/employer relationship must provide for continuous
improvement, feedback and employee participation. In Workforce 2000, the emphasis
will be on shared responsibility for group performance, job flexibility and fluidity of
assigned tasks, and coordination flowing from shared values (Bruce and Reed, 1994, p.
42). Employee participation is expected to increase output by creating a sense of
belonging for individual workers.
Since the introduction of participative management, there has been a
tremendous release of energy and creativity. Its true what they say about
this sense of ownership, its very powerful. When people really believe that
what they think, what they say, what they do will make a difference, they
take hold, they make things happen, they look for what needs changing and
they change it, be it in the system or in themselves (Rosen, 1993, p. 169).
Rosen goes on to cite an example of a productivity project in Palantine, Illinois
where crime rates are down 20% and accidents have been reduced 35%, with overall
department activity increasing 50% with the same number of people (1993, p. 190).
Employee involvement can have a dramatic impact on how organizations accomplish
their mission and objectives. In the same manner, the performance appraisal process must
respond to the challenges posed by organizational change in order for organizational
efficiency to increase. To accomplish this, it must effectively enhance employees
motivation to improve by recognizing the impact of values on work behavior, in order to
provide employees fair treatment and opportunity for participation in organizations.
Additionally, removing bias will have a positive impact in creating a more representative
bureaucracy. Thus performance appraisal can meet its longstanding objective of
enhancing productivity to meet organizational requirements of efficient and effective
delivery of services, while providing information that enhances an individuals personal
sense of contribution to the organizations objectives.
Problems of Current Appraisals
There is widespread dissatisfaction with traditional single-source (supervisor-
only) processes. Substantial research indicates that the traditional performance appraisal
process does not work well. The adequacy of performance appraisals has been
traditionally judged in terms of Thorndikes (1949) criteria: validity, reliability, freedom
from bias and practicality (Latham and Wexley, 1994, p. 230). Validity and reliability
characteristics associated with performance appraisal process such as MBO and BARS
are well documented. However, to emphasize solely the reliability and validity of an
appraisal instrument can prove to be a mistake. There are important issues regarding the
evaluator, too. As has been identified, sound instruments either are not used or are soon
abandoned because of researchers failure to take into account user perceptions (Dreher
and Sackett, 1983). If appraisers perceive the instrument to be difficult and cumbersome
to use, it will not be used properly, if at all (Latham and Wexley, 1994, p. 101).
The issues of bias and practicality are of greater interest to this discussion. The
traditional performance appraisal process is subjective and biased (Latham and Wexley,
1994). Other notable flaws are the vagueness of standards, subjectivity of the rater and
inconsistency among appraisers, that is, from one supervisor to another (Basnight and
In addition to concerns posed to the performance appraisal process by
researchers, there is also strong indication that neither employees nor supervisors are
satisfied with the current process (Latham and Wexley, 1994; Nigro and Nigro, 1981).
Employees do not look forward to the annual appraisal because they are being judged and
often they do not understand the process. Supervisors do not look forward to the process,
either. There are a number of reasons why supervisors cannot adequately meet the
requirements of assessing performance accurately or effectively (Glueck, 1978). For
example, as judges of performance, they cannot hope to be viewed positively by the
employee. The following factors are the key problems of performance appraisal
Supervisor biasFavoritism is a natural flaw in supervisor-only appraisals
(Nalbandian, 1981; Bedeian, 1976). Traditional, single-source performance management
processes result in Caucasians receiving higher scores than people of color (Latham and
Wexley, 1982). Many current appraisal processes are unfair to women (White, Crino and
DeSanctis, 1981). This tends to be true when the supervisor is the same ethnicity.
Additionally, the rating is often based on impressions of personality or of service of the
employee, rather than performance (GAO study, cited in Nigro and Nigro, 1981, p.273).
This perceived bias may be based on an accurate identification of lower performance.
However, when the rating comes from one source, it is often difficult for the employee to
feel fairly treated and thus it becomes easy to dismiss the rating as faulty.
Lack of appropriate training/communication for usersEffective rater
training is important to help eliminate potential for bias (Latham and Wexley, 1982).
Supervisors seldom receive this kind of training (Glueck, 1978). Also, communicating to
the employee what the process is and how the information will be used, can affect
employee acceptance but often does not occur (Bemardin and Beatty, 1984).
Objectives not clearly defined or generalized (Nigro and Nigro. 1981)An
Iowa study showed that half of job standards were unmeasurable. In addition, public
sector work relies on service provision, which magnifies the difficulty in quantifying
performance. This creates concerns regarding the reliability and validity of the
performance appraisal process. These concerns pose threats to the organization regarding
the possibilities of personnel grievance. It also undermines the desire to increase
individual productivity and contribution.
Predisposition to indiscriminate or undifferentiated assessmentIn 1978, a
survey of federal employee ratings reviewed indicated that 95 percent received
satisfactory ratings (Nigro and Nigro, 1981, p. 267). Where everyone receives high
ratings, recognition and rewards must be made on a political or random basis.
Role of the supervisorSupervisors are judges of performance (Glueck, 1978)
so they cannot hope to be viewed positively by the employee. As McGregor (1975)
noted, it puts them in the position of gods, which is not desirable for either supervisor or
employee. Another concern posed in the contemporary organization is that there may not
be a supervisor overseeing the work group, such as in the case of a self-directed team.
More organizations are expecting that the supervisor assume a coaching role to enhance
individual performance. This is more supportive of organizations as they create a more
interactive, participative environment. The roles of judge and coach are not readily
compatible. An individual may have skills better suited to judging, such as clear
identification of objectives, while lacking the communication skills to help the individual
develop action plans. Conversely, a strong coach may be better suited to helping an
individual focus on areas of improvement and key contributions to the work group, but
not have the technical skills needed to identify specific areas for technical enhancement.
The limited perspective of the supervisor does not see all of an individuals
performance. Organizations are increasing spans of control, or more humanistically
speaking, spans of supervision. Where spans of supervision used to mean 1:7-10,
organizations are moving to 1:20, 50, 100 or even more (Edwards, 1989). This creates a
situation where it is difficult for the supervisor to know direct reports by name, let alone
what are their performance levels and developmental needs, potential opportunities, and
effective career paths. Additionally, managers spend less than one percent of their time
observing subordinates (Komacki and Desselles, in press).
These concerns indicate significant deficiencies in the performance appraisal
process that must be addressed. While the theoretical models make sense, in practice the
results are often disappointing (Colby and Wallace, 1975). Further, a reliable or valid
process that is not viewed as fair by employees puts the organization at legal risk. More
important, it does not serve a key purpose to motivate and develop the employee. An
impractical performance appraisal process, for whatever reason, will not be viewed
seriously by participants. These conditions seriously undermine an organizations
performance management process.
Much of the research in the past has focused on technique versus process. One
approach focuses on the perfection of the test and includes such analysis as reliability
and validity studies. This focus is losing favor as the need for a more comprehensive
approach to research and application is recognized. The second approach represents a
shift to a comprehensive consideration of the goal of performance improvement to be
achieved by means of a process of appropriate supervisory communication and employee
commitment (Colby and Wallace, 1975, p. 50).
Alternative Appraisal Models
A dramatic change of the status quo is needed. Proposals to enhance the
effectiveness and fairness of performance appraisals include a range of options, nearly
universally targeted at single-source, supervisor-only appraisals. These suggestions
include opening up communications, involving employees in the process, changing the
rating scale, changing the rating format, and clarifying expectations by specifying
objectives (Bernardin and Beatty, 1984; Latham and Wexley, 1994; and Gleuck, 1978).
However, these suggestions often do little more than incrementally modify existing
procedures. Incremental changes have not been shown to be adequate to rectify the
problems inherent in the single-source appraisal process. Indeed, given the limited
success and large volume of research that has been dedicated to improving measurement
accuracy, recent public sector review of the literature concluded that further investment
in a higher degree of precision of measurement would not be justified. Instead it was
suggested that the goal of performance appraisal should be to support and encourage
informed managerial judgment in order to make personnel decisions (Schay, 1993, p.
These changes do not alter one of the primary areas of employee dissatisfaction,
which is the subjective nature of performance appraisal and the traditional reliance on a
single person, i.e., the employees supervisor, as a sole judge of an employees
performance (Schay, 1993, p. 659)
There are five commonly discussed alternatives for improving the appraisal
process. They are: eliminating appraisal, self-appraisal, peer review, subordinate review
and assessment centers. The latter four options focus on changing who provides the
Eliminating appraisalDemine argues against performance appraisals because
they are demotivating and pit individual against individual. Fundamentally, Deming
argues that processes are the problem, not people (Walton, 1986). But eliminating
appraisals ignores the need for organizations to identify high performers for salary and
promotional opportunities and other selection decisions. Without appraisal, an
entitlement culture will predictably replace performance. Deming fails to recognize that
focusing on behavioral competency for appraisal can provide feedback and job
assessment in areas where the individual has control. As Latham and Wexley (1994, p.
2) note, Deming was not aware of the benefit of shifting the appraisal from performance
outcomes ... to employee behaviors that are largely under the persons control. . .
Demings supporters understand the psychological benefit of using appraisers other than
the persons supervisor.
Self-appraisalSelf-appraisal occurs where the individual provides an
assessment of personal performance. The problems with this approach include (1) bias
toward overstating personal contribution, and (2) women tend to rate themselves lower
than men (Edwards, 1991). This may disadvantage people who are honest or provide
more accurate self-assessments. The benefit of this approach is that each individual best
knows himself or herself. It also provides an opportunity for the supervisor to involve the
individual, thus improving participation in the process.
Associate/peer appraisalAnother alternative is peer review, in which lateral
associates, or colleagues, assess one another. The concerns over this alternative include
that (1) competition or friendship among peers may affect ratings, (2) supervisors fear
they may lose control of the process, (3) unions may be concerned about individuals
rating peers or co-workers, and (4) it may have a negative connotation, particularly with
police, law, etc., where peer review is a disciplinary process with potential termination.
Some benefits of this approach are that colleagues have a high degree of
interaction with the individual in order to assess performance, and that it is a participative
process. Additionally, peer appraisal has been found to be a reliable and valid indicator
of performance (Latham and Wexley, 1994).
Subordinate/direct report appraisalHaving subordinates or direct reports
provide evaluations of the supervisor is another alternative. The benefits of this approach
are that these employees are consumers of the supervisors leadership abilities, and thus
are in the best position to assess performance. This also requires supervisors to interact
with employees in ways reinforcing of the organizations values, such as teamwork
(Bemardin and Beatty, 1984).
Negatives of such an approach are that this may expose a supervisor to unfair
evaluation because of hard decisions required of them by the organization. Subordinate
review may also identify severe organizational shortcomings in upper management ranks
that must then be addressed.
Assessment centers as an alternativeThe assessment center provides one
option for expanding input to the performance appraisal process. It is staffed with a
professional group of evaluators. This decreases the difficulties with biased reviews.
Consequently, the assessment center has been found to be more valid than a single-
source, supervisor-only approach. However, the assessment center poses a unique set of
problems (Shafritz, Hyde, and Rosenbloom, 1986). These include that a gatekeeper (the
supervisor) generally recommends an employee for assessment center attendance, that
those rated highly in the process are best modeling the existing organizational value set
and that thus a cloning of leadership takes place, and that those who do well/poorly in the
assessment center will presume that their future is unchangeable.
Each of these options present advantages, as well as certain disadvantages, over
the traditional single-source supervisor appraisal. Each has demonstrated reliability and
validity. Each is offered as a means of providing additional perspectives on performance.
However, they neglect one importance aspect. This is that the fairness of the process is a
critical component of the effective performance appraisal, not just the outcome or rating
results. Employees want to be treated fairly, and they evaluate the performance appraisal
process regarding its fair treatment of them. It is an area that has been overlooked. Too
little recognition has been given to the importance of perceived fairness in performance
appraisals (Latham and Wexley, 1994).
Performance appraisal processes that focus on one perspective do not fully
address the challenges posed by the new public organization. An area that has received
minimal discussion is that of using multiple raters for performance feedback. The best
way to increase the reliability of ratings is to increase the number of individuals providing
input (Schay, 1993, p. 659). However, Schays research posited that federal employees
are not receptive to multirater, or multisource, appraisal at this time, as 87% said they
preferred input from the first-line supervisor. Co-workers input received an 18% agree
rating, and internal customers 24%.
Yet, the multisource feedback approach is receiving greater interest. The federal
Office of Personnel Management has suggested that multisource assessment (MSA), also
called 360 Feedback, be tested by various agencies as part of its efforts to enhance the
performance appraisal process (U.S. Office of Personnel Management, 1992.). The
multisource feedback approach is thus proposed as an alternative model for gathering
performance appraisal feedback that expands input into the performance feedback process
beyond the scope of the lone supervisor. Multisource feedback is the focus of this
research project and is discussed in greater detail in Chapter 3. But first a discussion of
THE SPECIAL QUESTION OF FAIRNESS
The preceding chapter began with the premise that performance appraisals are an
important part of organizational life. There are two fundamental reasons for this
assertion, one based on the need for efficiency and the other on the value of equity as it
relates to treatment of employees within the public sector. Appraisal supports efficiency
when it permits the accurate assessment of individual performance, permits comparisons
between individuals for purposes such as promotion or compensation, and motivates
individuals to perform well. Appraisal supports equity when it produces outcomes which
the people involved regard as equitable for them, and when the process itself is perceived
by employees and managers alike to be fair.
This chapter will begin with consideration of the special salience of fairness as
a value in public sector organizations. It continues with an exploration of the implicit
mandate for fairness that emerges from the psychological contract between employer
and employee, and also from explicit legal requirements. The major section is a review of
the literature on fairness, in general and specifically with respect to performance
appraisal. Fairness is a key component of an effective performance appraisal process, but
has not been sufficiently addressed (Latham and Wexley, 1994). It is suggested that if
people perceive that the performance appraisal process is fair to participants, the
organization will be more efficient in reaching its goals of public sector service because
employees will be motivated to be more productive and efficient in their work. Increasing
perceptions of fairness of the performance appraisal process thus has additional benefits
to the rational organization beyond enhancing employees views of how they are treated.
Performance Management in the Public Sector
The public sector has traditionally been driven by three key values. These are
representativeness, neutral competence and executive leadership (Kaufman, 1956).
However, it is also noted that these various values have all been present to some degree or
another, sometimes all at once, during the course of American public administration.
There is historical precedent, then, for recognizing that certain values are a recurring
theme in the public sector. Balancing these sometimes conflicting values is the challenge.
In addition to the organizational values of the federal public sector agency,
performance appraisal requirements pose an additional set of values to be addressed.
These are proposed to be managerial efficiency, political responsiveness, social equity,
and protection of individual rights (Klingner and Nalbandian, 1993). It is argued that
managerial efficiency is consistent with the underpinnings of the rational organization.
Political responsiveness ties to the quest for representative government. Additionally,
merit forms the reference point for social equity and protection of individual rights.
The identification of these factors is important as a recognition of differences
between public and private sector organizations. The following is intended to provide
insights into challenges within the public sector due to its unique characteristics in
comparison with private sector organizations. These differences include the challenge of
accurate measurement of performance, personnel constraints, lack of a bottom line
orientation, and the orientation to equity and efficiency (Allison, 1980).
The measurement of performance is as difficult for government agencies as it is
for private sector service providers. Client satisfaction and numbers of clients served are
two perspectives of performance. Yet for many public employees, it is difficult to define
the performance standards required, due to the service nature of public sector
employment. Additionally, for many federal employees, their job responsibility falls
along a continuum of activities, making it increasingly difficult to identify who did
what. This service component occurs along a bureaucratic continuum. This reality of
the public organizations environment makes it difficult to isolate individual performance
contribution and thus to accurately measure that contribution.
Public agencies also have many more personnel constraints, generally legislated,
regarding how personnel matters are handled. The Civil Service Reform Act (CSRA), of
1978, for example, has significant procedural requirements. These are additional
constraints over the requirements for private sector organizations and require strict
adherence. The focus on the legal requirements is a driving concern to personnelists in
the public sector. It sometimes prevents significant experimentation and limits options in
an agencys performance management process.
Equity and efficiency are key issues for public agencies. The issue of equity, or
fairness, for employees and the public, is inherent in the values of the public sector. Yet
the need for efficiency drives public agencies to increase the number of services that can
be provided as well as the number of times services are provided, while private
organizations are driven to maximize profit. While balancing the important concern of
equity presents unique considerations within the public sector, private agencies enjoy
tremendous flexibility surrounding whether and how to focus on these issues..
A bottom line orientation to defining the organizations goals in dollars or
revenues is generally not feasible in the public sector. Business use of profitability to
measure performance is viewed as an objective evaluation, thus ostensibly more fair.
Public agencies are charged with providing services to the public constituency for the
public good. The issues of profitability and return on investment are not readily
applicable. Establishing measures for public agencies in a way that minimizes, or ideally,
eliminates subjectivity in the performance assessment process is a critical challenge.
These unique considerations within the public sector thus present challenges to
the performance appraisal process not shared with the private sector. Further, the
limitation of experimentation posed by legal considerations is also a significant factor.
Notably, however, there is an interest in enhancing the performance appraisal
process. This is also in response to the recognition of the problems inherent in the current
process. Several key pieces of legislation regarding performance appraisal in the public
sector create the opportunity to enhance the process. The Classification Act of 1923
mandated the authorization for revising the appraisal process. Its chief purpose was to
establish merit recognition in order to eliminate special privilege (Nebeker and White,
1990). The Performance Rating Act (PRA) of 1959 introduced several major changes to
the performance appraisal process (Nebeker and White, 1990). These changes included
that each agency would adopt its own appraisal plan and that appraisal plans should be
developed with the intent that employee performance would improve.
The Civil Service Reform Act (CSRA) of 1978 provided the modem reference for
performance appraisals in the federal government. Its intent was to improve employee
productivity. It has also been observed that it has another role. Maintenance of a
climate of trust and belief in the fairness of the system is essential (Nigro, 1981, p. 86).
The phrase performance management first entered the civil service lexicon following
the introduction of the CSRA (Nebeker and White, 1990). The provisions of the statute
were expanded to include all civil service employees in the Code of Federal
RegulationsAdministrative Personnel. Parts 7-699.(1980, sect. 430.203-205.)
The CSRA requires that each agency develop one or more systems for the
periodic appraisal of employee performance. Each system must provide for establishment
of performance standards that, to the extent feasible, permit evaluations of job
performance on the basis of objective criteria. Agencies must encourage employee
participation in the development of the performance standards. Employee ratings are to
serve as a basis for training, rewarding, reassigning, promoting, demoting, retaining and
separating employees (Nigro and Nigro, 1981, p. 269). Additionally, individual
performance appraisals must occur on an annual, if not semiannual, basis.
Several definitions regarding performance management are important because
they specifically guide the manner in which performance appraisal is undertaken in
federal agencies. (The following are from the Code of Federal Regulations
Administrative Personnel. Parts 1-699.1980. Sect. 430.203-205.) Performance
means the employees accomplishment of assigned duties and responsibilities.
Appraisal means comparison of an employees performance of duties and
responsibilities with performance standards. The purpose of performance appraisal is to
improve organizational performance by providing management with information about
employees performance and employees with information about how they can improve
their performance. ... To be useful to all concerned, the performance appraisal process
needs to be based on realistic and objective standards derived from the requirements of
the position which are established and communicated in advance so that supervisors and
subordinates have a common understanding of expected performance. And finally it is
stressed, It should be clear that the successful implementation of a performance appraisal
system must ultimately rest on employee participation and acceptance. Employee
participation and acceptance become the building blocks on which the effective
performance appraisal process rests.
This brief review indicates that much of the legislation concerning the
performance appraisal process was predicated upon a desire to enhance the objectivity,
and consequently the fairness, of the performance appraisal process for federal
A Higher Standard of Fairness
Public sector agencies may be held by law to a higher standard of fairness than
private sector organizations. The American form of government is based upon precepts
of a democratic process. The process, however, is reliant upon the participantsthose
who carry out the decisions made by the politicians. This legitimization comes about
through [governmental representation of the public .. and by public participation in
government . The notion that administrative power and democratic government can
be integrated through the composition of a civil service has a long tradition (Krislov
and Rosenbloom, 1983, p. 530).
Representative bureaucracy leads to:
[mjore democratic decision-making and better decisions by expanding
the number and diversity of the views used in the policy-making process[,
and serves to] improve bureaucratic operations and outputs ... by
ensuring that the decisions and services were more responsive to the
needs of clientele and consumers, particularly members of minority
groups; promote a more efficient use of the countrys human resources;
increase, both symbolically and actually, the legitimacy of governmental
institutions; and elevate social equity and justice to prime political values
at least as important as the prevailing paradigm of economy and
efficiency (Kranz, 1976, pp. 110, 112, 115 and 116).
It is through representation that equity occurs.
The impact of the Civil Rights Act of 1967 created a different social context
against which to assess integration. Its goal was to eliminate discrimination. Then in the
1980s the issue of antidiscrimination expanded the concept of merit to reaffirm that
diverse employee bodies respond to two important issues. One is the need to be
representative of the larger population served by the government organization.
Additionally, it was recognized that diversity in the employee population maximized an
organizations effectiveness by capturing a variety of perspectives and skills (Thomas,
Representation is also a concern of the performance appraisal process results,
because the government should be socially representative, that is, made up of a diverse
group of employees based on factors such as age, gender, ethnicity and others (Cohen,
1984). The issue of representation became more critical when the Civil Rights Act of
1964 was passed. Then the Glass Ceiling Act of 1991 identified those artificial barriers
based on attitudinal or organizational bias that prevent qualified individuals from
advancing upward in the organization (U.S. Department of Labor, 1991). The issues of
increasing diversity and eliminating bias thus became clearly articulated as goals for the
Early government appointments reinforced privilege and power. Appointed
officials were either well known or rich, or both. Merit selection replaced this
orientation, with the intent to ensure that government employees represented the highest
quality of employees. This change also reinforced a demand for increased efficiency in
public services. People who could perform the job well should be expected to best meet
the needs of the public they served. In order to fulfill the desire for selection based on
merit, the process was formalized through job classification. Job classification described
the skills needed by a prospective employee and provided a framework for selecting the
applicant best suited for the position.
Merit in the selection process and representation in the public sectors employee
body continue to be important topics for consideration today. One aspect of merit rests
on selecting the best person for the position based on skill and ability, not ethnicity, for
example. Representative government means that government personnel should be
representative of the community served. This orientation led to greater concern that
government employees include people of color and non-male employees. Both of these
issues present challenges to the performance appraisal process.
Mandate for Fair Performance Appraisals
Performance appraisals are an important part of organizational life. In order for
the process to be effective and motivate employees, it is important that appraisals be fair.
There are a number of reasons that compel organizations to be fair, including the drive for
competitiveness and the need to foster diversity. The drive for competitiveness is
resulting in flatter organizations and the increasing knowledge of co-workers. There are
several reasons to foster diversity. These include changing workforce demographics and
resultant issues of diversity, and the drive for increasing organizational productivity.
Diversity awareness, and a resultant change in employee management, is expected to
increase trust in the workplace
Fairness and the perceived fairness (whether people perceive themselves to be
fairly treated) have benefits to the individual and the organization. There is usually a
positive, circular relation between the well-being of the individuals in a group (or society)
and the level of functioning of that group. The more satisfied individuals are, the better
their group functions.. . [T]his proposition suggests that justice is intrinsically
concerned with both individual well-being and social functioning (Deutsch, 1975, p.
Individuals want to be viewed as fair; indeed, fairness is a desired social
identity (Greenburg, 1990, p. 111). We each want to be treated fairly by others. When
fairness is evaluated within the context of organizations, it is referred to as organizational
justice. This area of study provides significant insight in developing processes for
managing employees in the workplace fairly. Procedural justice represents the search for
fairness, or equity, in the workplace (Sheppard, Lewicki and Minton, 1992). Rawls
(1971) indicated that justice can be developed only under conditions that are fair to all
parties involved. This premise is justice as fairness. This research project attempts to
frame the discussion of organizational fairness in the public sector as it relates to the
performance appraisal process.
Organizational justice builds trust. It is dependent upon treating employees
fairly. The organization benefits from developing a performance appraisal system that
employees trust. Lack of trust does not facilitate a high-producing workforce, so the
organizations effectiveness is reduced. Productivity is affected (U.S. Office of Personnel
Management, 1991). Lack of trust can lead to the failure of the performance appraisal
system. No matter how technically and administratively sound the appraisal system, it
will not succeed if employees lack confidence in the integrity and good intentions of an
organizations top leadership (Nigro and Nigro, 1981, p. 281). Employees will view the
performance appraisal process with suspicion and defensiveness (Nalbandian, 1981, p.
394) if they distrust it. In addition, the more subjective the appraisal process, the greater
the degree of trust required to make it work (Lawler, 1968). Thus, if the primary intent of
performance appraisal is indeed maximizing individual contribution and organizational
efficiency, the failure of the performance appraisal system has significant consequences
within the public sector.
A 1979 U.S. Office of Personnel management study found that 51% of federal
employees felt that their performance rating presents a fair and accurate picture of my
actual job performance, while 23% didnt know (Nigro and Nigro, 1981). More recent
research completed in 1991 found an increase in how fair employees viewed their
appraisals to 63% (Schay, 1993). In a Navy demonstration project, perceived fairness of
personnel ratings increased by 19% when a more-objective performance method was
introduced. Regardless, these findings indicate that a significant portion of the workforce
in the public sector does not view the current performance appraisal process as fair.
The Psychological Contract
Organizations expect that satisfaction in work rewards and recognition leads to
higher productivity. Yet, if the organization of the future will be made up of fewer
employees, it makes sense that organizations will want to retain those employees best able
to contribute. The greatest contribution will come from those satisfied with their jobs
(Smith, 1993; Minton, 1988).
Besides the explicit portions of the psychological contract, such as salary, the
implicit part is increasingly recognized as important. Intrinsic motivators, such as job
satisfaction and respect by ones peers, are as important for many contemporary workers
as promotions were to the post-World War II workforce (Daley, 1984).
This implicit portion has been labeled the relational obligation. If
organizations violate this relational obligation, a negative financial impact based on
employees perceptions of managements fairness could well be the result. For example,
people in emotional inner turmoil... are less likely to observe safety precautions
(Sashkin and Williams, 1990, p. 68). Sashkin and Williams research also found that
organizations (in this case retail stores) with high employee illness and accident
compensation (defined as overhead) scored significantly lower on an evaluation of
fairness by their employees than organizations with lower overhead costs. Employees are
also absent more frequently or leave the organization (resulting in more turnover) due to
perceived unfairness (Mowday, Porter, and Steers, 1982). Also, employees sensing pay
inequity do not perform at the same (high) level as those employees sensing pay equity
(Pritchard, Dunnette and Jorgenson, 1972). Thus, increasing employees perception of the
organizations fairness leads to lower cost burdens.
The cost of job dissatisfaction to both organizations and individuals is high when
organizations suffer loss of productivity and employee malaise. The level of
labor/management trust is one of the crucial characteristics that determine which firms
have a competitive advantage relative to others (Kochan, Katz, and McKersie, 1986).
The individual also bears costs due to unfairnessdirect costs from time and energy
expended, indirect costs from lost income, lost psychological investment, atrophy of
skills, retaliation, loss of reputation, emotional costs of action, lost opportunities and a
sense of failure (Sheppard et al., 1992). The public organization as a community must
recognize this cost to the individual. Just as the public deserves honest and competent
service, the millions of people who work in the public sector deserve a humane and
fulfilling workplace (Rosen, 1993, p. 139).
Legal Mandate for Fairness
It is also important that organizations have fair appraisals due to the legal
mandate. In 1964, the Civil Rights Act Title VII was passed (and amended in 1972)
prohibiting discrimination on the basis of race, color, religion, sex, or national origin.
The Equal Employment Opportunity Commission, or EEOC, was formed as a result. The
initial guidelines focused only on selection procedures. To clarify its intent, the EEOC
published the 1978 Uniform Guidelines which established the Civil Rights Act as an
umbrella over all the terms, conditions and privileges of employment inherent in the
hiring process, including the performance appraisal process (EEOC, 1978). It also
created a requirement for performance appraisals that are fair and commands users to seek
out and use the fairest process available. (Fair means that the assessment process be
valid, or job-related, and that the criteria be representative of the significant portions of
Organizations must also demonstrate that their appraisal system is the best
available, that no other system is less discriminatory (EEOC, 1978). The courts
demand a standard of performance that is greater than simply saying, We were fair.
The legal process also requires performance data on the wronged person and also others
similarly situated (Azevedo, 1990, p. 103).
Organizations using discriminatory practices in their performance appraisal
process may find themselves challenged by costly legal actions. The judiciary has
asserted itself, as in Spady v. Mount Vernon, and U.S. v. Richardson, in situations
where individual rights were threatened by performance appraisal systems. Justice
Douglas stated, Todays mounting bureaucracy promises to be suffocating and
repressive unless it is put into the harness of procedural due process (419 U.S. 983,
985). The judiciary will continue to require that public sector performance appraisal
systems are not discriminatory for racial, ethnic or gender factors (Rosenbloom, 1982).
Court cases have found performance ratings to be discriminatory when they are:
subjective or based on ill-defined criteria, affected by sexual/racial bias, not
collected/scored under standardized conditions, not based on a careful job analysis when
identifying the content of the rating instrument, or not shown to be job related (Schneier,
1977, p. 25). These are areas that raise serious issues of discrimination when they are
presented to the courts by individuals.
To respond to these concerns, the following steps have been identified when
developing a performance appraisal process that follows the EEOC guidelines (Schneier,
1977, p. 34): (1) conduct a job analysis to ensure that characteristics under evaluation are
critical to job performance; (2) provide rater training on use of the appraisal instrument,
including written standards on performance objectives; (3) administer the instrument
under standardized conditions; and (4) conduct validity studies. The EEOC guidelines
articulate that when members of one racial, ethnic or sex group characteristically obtain
lower scores on a selection procedure than members of another group, and the differences
are not reflected in differences in a measure of performance measure, use of the selection
procedure may unfairly deny opportunities to members of the group that obtain lower
ratings (Arvey and Faley, 1988, p. 104). These situations identify areas of performance
More than 15 years have passed since the guidelines were published, but
surprisingly little documentation exists to substantiate fairness in performance appraisal
processes. There is clearly no agreement on the most appropriate performance appraisal
methodology to use. While the standards that must be met for adherence to the EEOC
guideline are clear, the specific methods for increasing fairness in performance appraisals
have not been well researched. This thesis is intended to bring focus to that review and
provide a new framework for fairness in performance appraisal.
Fairness as a Desired Objective
In all people, without exception, there lives some instinct for truth, some
attraction for justice.
Franklin D. Roosevelt
Fairness is defined many ways. The research indicates four major approaches to
defining fairness-outcomes (rewards and punishments) or distributive justice (Adams,
1965); equity (what happened to one vis-a-vis another) (Cohen, 1984; Gilliland, 1993);
appropriateness; and morals or ethics (adherence to a code of conduct that defines
appropriate behavior) (Sashkin and Williams, 1990). Fairness can also be determined
empirically by quantifying the results of a process and how it affects certain groups. The
crux of statistical proof in discrimination cases is the presentation of percentage
differences between majority and minority groups sufficiently substantial to support an
inference that such differences would not exist in the absence of discrimination. ... If the
performance appraisal results in assigning employees to rating categories,. . statistical
comparisons of the frequencies of minorities and non-minorities in each category should
be made (Latham and Wexley, 1994, p. 42).
Alternatively, it has been recently argued that fundamentally fairness is a
perceptual measure individually defined (Sheppard, Lewicki and Minton, 1992). This
approach is important when discussing performance appraisal because ultimately
individual employees reach a personal evaluation of fairness based on how they were
treated. The individuals model for fairness represents a three-fold approachoutcome,
procedure, and system. For example, outcome is a pay raise that comes about due to
performance ratings. This is distributive justice. The procedure is the manner by which
the decision is made. This is procedural justice. The system defines the organizational
context in which the procedure operates, such as the organization may informally
distribute the information or it may rely on traditional top-down communication patterns
to distribute the information. The system influences both outcome and procedure.
Evaluating justice on these three factorsoutcome, procedures and systemleads to a
more comprehensive and complicated concept of justice. Significant research exists to
argue that the perception of procedural fairness (how the raise was determined) is at
least as important as the perception of outcome fairness (what the raise amount was) (p.
17; See also Thibaut and Walker, 1975; Lind and Tyler, 1988).
Distributive Justice and Equity Theory. The work of Adams (1965) articulates
the equity perspective of fairness, and lays the groundwork for future empirical work.
Equivalent outcomes act as a key component to this view of fairness. Equity theory holds
that people expect to receive similar treatment for similar behavior: Inequity exists for a
person whenever he [or she] perceives that the ratio of outcomes to inputs and the ratio of
others outcomes to others inputs are unequal (p. 80). The distribution of rewards
(Kidd and Utne, 1978) is important as people consider fairness. Adams also identified
how situations of inequity affect the performance of individuals.
Equity theory argues that fairness is a social interaction based on reciprocal
change where individuals strive to maximize their rewards (Walster, Walster and
Bercheid, 1978). Some question the usefulness of the theory (Lawler, 1967). Others
continue to see its validity (Goodman and Friedman, 1971). Managers view fairness as
very closely linked to equity and a performance/productivity ethic (Sashkin and Williams,
1990, p. 61). The outcomes may be rewards, such as compensation or promotion, or
punishment, such as demotion. Individuals assess whether they feel that the outcome is
suited to the situation in determining fairness.
Procedural Justice and Factors of Fairness. While the results are not yet
conclusive, there does appear to be a connection between the two perspectives on fairness
research-procedural and distributive (Greenburg, 1986; Gilliland, 1993). Fair processes
may be more likely to lead to fair outcomes. However, it has been noted that process
constraints have a greater impact on increasing favorable reactions (Leung and Li, 1990).
As participants determine fairness of the performance evaluation, for example, research
has shown that how the performance evaluation was determined may be more important
than the outcome itself (Greenberg, 1986). This finding is integral to this study, as it
suggests that process used to measure performance may be equal to or more important
than the results themselves. Procedural justice suggests that equity serves as an
associated requirement as participants determine fairness.
An integrated model that incorporates the lessons from procedural justice to
enhance the fairness of performance appraisals has been suggested to include employee
input, enhancing the accuracy of performance data and minimizing rater bias (Greenberg,
1986). Employee participation in the process leads to increased perceptions of fairness.
Part of the fairness model involves reducing bias. It includes the requirement for
complete and accurate information, the motivation of the rater, and clearly defined
policies/practices. Rater training facilitates reduction of bias, and is critical to support the
organizations contention of a fair performance evaluation process in the event of
litigation (Kleiman and Durham, 1981). Reducing bias leads to increased accuracy and
completeness of the performance information.
Lawler (1967) proposes that attitudes toward fairness play a critical role in
employee acceptance. This assumption was tested by Landy, Barnes, and Murphy (1978)
as part of a model to identify variables that enhance perceptions of fairness. They
identified five process variables as significantly affecting perceptual fairness: formal
evaluation program, annual evaluation, supervisors knowledge of performance,
opportunity to express feelings, and action plans related to performance. These factors
had a high impact on perceptual fairness. (However, the organization studied had an
MBO appraisal process, so the generalizability is limited.)
Greenberg (1986, based on Leventhal, 1980) also tested procedural components
of perceptual fairness. He extended on the research of Landy, Barnes and Murphy (1978)
by conducting open-ended interviews to further identify fair process variables. The
factors identified are: soliciting information prior to the evaluation and using it, two-way
communication during interview, ability to challenge/rebut evaluation, rater familiarity
with ratees work and consistent application of standards. While this output is consistent
with theories proposed, it is untested in an organizational setting. Additionally, the
factors were identified by participants in a management workshop, so the generalizability
to the organizational setting is limited.
Leventhal (1976) suggests seven procedural elements that enhance perceptions of
fairness in the organization. These are: selection of agents, ground rules for decisions,
agreed-upon procedures, decision structure, safeguards, appeal process, and mechanisms
for changing the outcome if additional information is identified. Additionally, Leventhal,
Karuza and Fry (1980) identify six questions that employees ask when deciding if a
decision is fair. These include whether the rules are applied consistently, biases are
suppressed, the process provides accurate information, poor decisions are correctable, a
representative sample is chosen, and the process is ethical.
Voice procedures are important to enhancing procedural fairness. Voice allows
people an opportunity to provide inputs to the decision maker. One of the findings of
related research is that people perceive voice procedures as fairer than mute procedures
(Greenburg and Folger, 1983). Additionally, the need for justification is also introduced
as an important factor to enhancing procedural fairness (Bies and Shapiro, 1988). It is not
sufficient to provide a process that is fair; people also need to be told the rationale for
The due process model is an alternative to the power/dominant nature of most
current appraisal processes. It is consistent with the importance of soliciting employee
input as articulated by several other researchers. This model suggests that elements
include adequate notice, fair hearing, and judgment based on evidence (Folger, Konousky
and Cropanzo, 1992, p. 129). It challenges many of the typical performance appraisal
processes based on rationality, particularly recognizing the influence of the social context.
It also recognizes the difficulties of wholly rational decision-making when numerous
value judgments are inherent in the process. Moral obligation should also matter (p. 129).
This approach suggests including self-appraisal and addresses issues of employee rights.
It also reinforces the need for a fair appraisal process that includes knowledge of the
standards, participation in the process, and reduction of rater bias. The self-assessment
further enhances the employees participation in the process.
Gilliland (1993) expands on Greenbergs model (1990) that perceptions of
fairness are enhanced based on formal characteristics of the process, explanation, and
interpersonal treatment. This model incorporates procedural and distributive justice
factors in a comprehensive model of overall fairness. The procedural justice rules consist
of the formal characteristics, explanation, and interpersonal treatment. Formal
characteristics include job relatedness, opportunity to perform, the reconsideration
opportunity, and consistency of administration. Explanation includes feedback, selection
information, and honesty. Interpersonal treatment consists of interpersonal effectiveness
of administrator, two-way communication, and propriety of questions. This focus on
interpersonal treatment expands the procedural justice model to recognize how the human
element affects perception of fairness. While this model attempts to integrate a variety of
factors into a comprehensive model, it is overly cumbersome for testing in the
Need for Fairer Performance Appraisals
Those areas which enhance performance appraisal fairness, as identified by
several researchers, not only include employee input, accuracy of the data, and
minimizing of bias, following Greenburg (1986), but also include established policies and
practices, which is consistent with the due process model. Providing a mechanism for
change and process safeguards may be related, in that the need to rebut inaccurate ratings
is reduced if safeguards are established. These factors are important in increasing
employees perceptions of fairness.
But what does this information mean for practitioners? There is clearly extensive
research on fairness. Numerous theories are presented regarding what factors enhance
perceptions of fairness. Actual tests of perceptual fairness drawing on these factors of
procedural justice are limited, however. Many of the conclusions have been asserted
based on research occurring in classrooms and executive seminars. So these theories
provide scant insight into actual organizational usefulness, [y]et performance evaluation
is an applied subject and, as such, research should eventually lead to improvement in
practice. Continued reliance on student samples and laboratory settings is not facilitating
the transfer of research into application (Bretz, Milkovich, and Read, 1992, p. 338).
This project is intended to advance the discussion of fairness in performance appraisal
with field research.
The performance appraisal process is challenged to respond to the dual goals of
managerial efficiency and protection of individual rights. Managerial efficiency responds
to the core value of efficiency in providing services to the public. Protection of the
individual responds to the challenges regarding merit selection and representative
government. Furthermore, the appraisal process must be fair because managers identify
fairness as the most important performance appraisal issue organizations face (Bretz,
Milkovich and Read, 1992, p. 332). The procedures that make up the appraisal process as
well as the rating itself have been found to be tied to the determination of how fair the
appraisals are perceived to be.
Organizational leadership must demonstrate values of both head and heart-
intellectual excellence, analysis, innovation, and flexibility, as well as critical thinking,
fairness, integrity, perseverance, and respect for individual dignity. Without such
leadership, the American economy will stumble and flounder into the technology age.
(Maccoby, 1988, pp. 231, 238). Leaders must recognize the increasing expectations of
employees by providing for autonomy as employees take control of their own
The way to integrate these concerns into the performance appraisal is to focus on
the individual and how individuals perform within the organization. To be effective
performance appraisal must take into account why man behaves as he does, his
expectations, his relationship to the organization and to the people he works with, and the
many factors that influence his job performance (Olsen and Bennett, 1975, pp. 18-23).
Importance of This Research
The public sector faces significant changes which are forcing a reevaluation of
how performance appraisals are undertaken. Current performance appraisal processes are
being challenged regarding their accuracy, participation and fairness to the individual
being evaluated. The new performance appraisal model must respect the drive and values
that argue for improved productivity and increased efficiency in the public sector. The
appraisal process must respect the contemporary public employee who demands a greater
voice in the process.
There are additional reasons for the importance of fairness in performance
appraisal in public sector agencies. Public organizations rely heavily on appraisal for
personnel management decisions. But why is fairness important? There are several key
reasons. First, employees acceptance of feedback from the performance appraisal
process is based in large measure on whether they trust the process to treat them fairly,
For the feedback to be used to improve performance, this trust is critical. When fairness
is enhanced, employees are motivated to increase their efforts toward meeting the publics
diverse needs. Fair processes increase employee acceptance, fostering organizational
Equity theory argues that people who believe that outcomes (distributive justice)
are not dependent upon appropriate inputs may adjust their performance accordingly to
compensate. There is also indication that organizations that are viewed by employees as
treating them fairly have reduced costs due to absenteeism and turnover. In these
instances, the public organization seeking to maximize its efficiency is helped.
Fairness is also important to avoid employee litigation. If federal sector
employees view their treatment as unfair, they may file suit against the employing agency.
This is costly. If wrongdoing on the part of the organization is found, the cost of the
settlement adds to the financial burden. This scenario does not best serve the public good
through effective management of taxpayer dollars. An additional reason arguing that
fairness in performance appraisal is important is because it is required as defined in the
EEOC Uniform Guidelines. Thus, federal agencies are mandated to fairly assess
employees performance. Failure to do so puts the agency at risk.
From a social benefit perspective, fairness is an important value. A fairer
assessment model serves to more effectively respond to the complex social structure
which is the reality of todays public agency.
Finally, people desire fairness. The concepts of merit and representation are also
rooted in a desire for fairness. The federal governments personnel requirements are
designed to provide recognition of merit in performance and to be representative of the
larger constituent population. Performance appraisals must accommodate legal
requirements to be fair and unbiased, as well as recognizing and accommodating the
diversity of the labor force.
Little research exists regarding public organizations and fairness in performance
appraisals. Research on procedural and distributive fairness provides some initial insight
into the study. This research is not integrated into a model that ties the fairness research
to the performance appraisal process. The intent of this project is to demonstrate that
changing the method by which performance appraisals are undertaken will enhance the
participants view of the fairness of the performance appraisal.
The benefits are that if the process is viewed as more fair, employee acceptance
will increase in support of organizational goals, thereby enhancing productivity. This
serves an important need to meet the public sectors responsibility to effectively and
efficiently provide services to its clients. This research project provides an important
beginning in the search for a fairer performance appraisal method.
THE MULTISOURCE ASSESSMENT MODEL:
RESEARCH QUESTIONS AND HYPOTHESES
The multisource assessment (MSA) model has been proposed as a fairer way of
assessing employees performance. The defining characteristic of the MSA process is that
multiple individuals are evaluators, rather than a single source, usually the employees
supervisor, in the traditional performance assessment process. In MSA, the evaluation team
consists of work associates with whom an individual works closely and who are able to
provide accurate and credible performance assessment. Because more views of an
individuals performance are captured in this model, it is expected that the employee will
view the process as more fair.
Precedent for the multisource assessment model can be found in many settings.
For example, the process is seen in the Supreme Courts judicial rulings. Many
municipalities use a panel review process when selecting a building design for downtown
redevelopment. The Olympics rely on input from multiple judges to decrease the impact
of nationalistic bias. Each of these instances relies on capturing assessment information
from more than one source to increase the accuracy of the assessment.
The MSA process for performance appraisal has evolved in part due to many of
the organizational changes occurring. These include fewer layers of management, larger
spans of control, team-based management, employee involvement and quality initiatives.
MSA serves as a response to these changes. MSA is a tool that facilitates organizational
The MSA process focuses on the organizational culture and its human processes
how people interact. This is consistent with organizational development interventions,
described as facilitating an effective transformation of behaviors. This transformation
requires developing skills for inducing effective and satisfying interpersonal and
intergroup relationships at work; increasing competencies to induce and sustain desired
and desirable organization climates; augmenting skills in conflict resolution; enhancing
skills and attitudes to satisfying and productive choice or change; and encouraging
specific counterbureaucratic attitudes and behaviors, e.g., those related to risk-taking,
openness and so on (Golembiewski, 1985, p. 234).
The MSA process communicates the new competencies required. For example,
an organization that wishes to reinforce the effectiveness of team activities might identify
competencies such as supports the team consensus, shares expertise with others, and
recognizes others contributions. These are different assessment variables than
traditional appraisal processes would address. Such variables communicate to employees
the need to change how they get the job done. Consequently, the MSA process itself
acts as an organizational development intervention, a change agent, to focus on the
This chapter discusses the multisource assessment model used for performance
appraisal. The expansion of the evaluation team from one (the supervisor) to many
represents the most significant change MSA introduces to the appraisal process. The
following section discusses several aspects of implementing an MSA process, including
the supervisors role, and the safeguards that can be incorporated into the MSA process
that are not available in traditional single-source, supervisor-only assessment. Next, some
challenges to implementing MSA are noted. The chapter concludes with a discussion of
the hypotheses being tested in this project and their connection to the literature reviewed.
MSAAn Alternative to Single-source Appraisal
A relatively new appraisal model, multisource assessment, recommends that
performance information come from multiple individuals who interact with the
assessment receiver (Lawler, 1967; Bemardin, 1986; Edwards, 1983). The diagram
below (Figure 3.1) demonstrates how the model expands the number of feedback
providers. [Sjoliciting inputs from multiple sources can frequently be valuable to
capture the multiple needs and perspectives of an employees multiple clients (Schneier,
1977, p. 14). The multisource model may include the supervisors assessment plus that of
other work associates. These work associates may be colleagues, associates, peers, direct
reports, the second level supervisor, and internal and external customers. Multisource
assessment introduces an important concept to the process of performance assessment
that individuals should be assessed by work associates with whom they interact
frequently. This approach combines information from several sources which leads to
more valid and reliable performance appraisal of individual performance (Baruch and
Issues of reliability and validity are critical to any performance appraisal model.
Performance appraisal systems that rely solely on supervisor evaluations assume that
supervisors are able to observe closely their employees behavior. The characterist
change of MSA is that it expands the evaluation team beyond the immediate supervisor,
to include others such as peers, direct reports, internal clients and a self appraisal. A key
concern with the traditional single-source appraisal model is that it is biased, or not
representative of the individuals complete performance, because it relies on a lone
perspective to provide the assessment. In todays organizations, in which employees may
have considerable autonomy and supervisors may have large spans of control, this
assumption may not hold true. Using different viewpoints provides a more
comprehensive picture of individual performance (London and Beatty, 1993). Also,
team-based management structures do not provide for a traditional supervisor/overseer.
The MSA process compensates for the limited perspective of the supervisor by expanding
the evaluation team to include those in the performers circle of influence. Adding
additional raters obviously can help improve the reliability and validity of ratings. But
the MSA process represents not just an incremental change to the performance appraisal
process but one that also fundamentally alters the power dimension of performance
assessment by soliciting input from multiple levels of the organization.
Expanding the evaluation team adds value on other counts. Self-input increases
the subjects perception of process fairness (Thibaut and Walker, 1975) because affected
individuals feel able to influence the decision or offer input. Peer appraisal is a reliable
and valid source of performance measurement, because the ratings of work associates
have been shown to be closely correlated with the subjects successful performance
(Cummings and Schwab, 1973). Direct reports input is highly reliable and may have
more predictive validity than supervisory judgments (Bemardin and Beatty, 1984). The
following Table 3.1 summarizes various rating perspective information.
Table 3.1: Rating Perspectives
Supervisor only Traditional source, subject to biases (Nalbandian, 1981; Bedeian, 1976) Puts supervisor in judge role (Glueck, 1978) Limited perspective (Edwards, 1989)
Self only Impacts perception of process fairness (Thibaut and Walker, 1975) Becoming a popular source but may be adverse to those who self- rate accurately (Cummings and Schwab, 1973)
Peer review (Colleagues) More reliable and valid than single-source supervisor May create competition bias Often creates low user satisfaction Sometimes used for disciplinary actions Highly motivational
Upward review (Direct Reports) More reliable and valid than single-source supervisor (Bemardin and Beatty, 1984) May create discomfort to leaders Disadvantages leader of tough unit or group Must include safeguards against retaliation
Multisource assessment (all the above plus team members, internal and external clients) More reliable and valid than single-source supervisor (Baruch and Harel, 1993) Provides a wide number of perspectives May need to show feedback source (e.g. team members) Highly credible to feedback receiver
One concern regarding the use of multiple raters is that different rating sources
may have different perspectives on performance behavior, and they also may rate
different observed behaviors. It has been argued that while low inter-rater agreement may
result, each rating source has a high degree of validity (Baruch and Harel, 1993).
Research, in fact, shows that inter-rater agreement is quite high (Harris and Schaubroeck,
1988). MSA may increase the accuracy of the performance appraisal rating by adding
additional perspectives to the assessment process. Combining multiple sources of
additional perspectives to the assessment process. Combining multiple sources of
performance appraisal information into one overall rating of performance feedback
enhances the ability of managers to assess employees performance in the organization
(Baruch and Harel, 1993, p. 109) because it broadens the perspectives provided.
Another problem identified with single-source, supervisor-only appraisal, is that
it is predisposed to indiscriminate or undifferentiated assessment. Evaluation teams
address variation in rating distinction by providing more perspectives to the rating
process. Multiple raters may also counteract each others biases. Thus, the multisource
assessment model responds to many of the problems of supervisor bias. It also moves the
supervisor from the role of judge to the role of coach (Latham and Wexley, 1982).
Supervisors as well as employees view the coaching role more positively than the judging
role. It is more consistent with new organizational structures, which include teams and
larger spans of control.
MSA incorporates safeguards to fairness not possible or inherent in traditional
single-source appraisals. These safeguards enhance participation and participants
confidence in the accuracy of the behavior feedback results. Multiple raters generate
more data on which to reach an overall performance appraisal rating. The process may
facilitate the identification of biased ratings because the validity of the ratings can be
assessed with multiple raters if inter-rater agreement is shown. The MSA process
includes technological safeguards (to be discussed later) designed to moderate biases and
overcome employees concerns about fairness. Hence, the MSA process has a better
chance of motivating behavior change than traditional performance appraisal, because
employees are more likely to act on appraisals they think are fair (Edwards, 1985, 1989).
Rater anonymity must be sustained, in order to enhance the perceived fairness of
the MSA process. Care must be taken to ensure rater anonymity. ... In some
organizations managers do not receive subordinate ratings when the number of peers or
subordinates is small because of the risk of revealing the identity of the raters through the
numbers alone. That anonymity is fundamental is not only a common-sense observation
but is also supported by research findings (London and Beatty, 1993, p. 366). London
and Beattys respondents indicated they would have rated differently had the feedback not
Another benefit of MSA is its ability to foster quality initiatives through effective
performance management. Because the feedback is tied to specific core competencies of
job behaviors, individuals can target action plans to improve; thus, MSA reinforces the
value of feedback. The Total Quality Movement has increased the perceived value of
co-worker feedback by emphasizing service to customers and continuous improvement of
customer service through measurement (Kaplan, 1993). Co-workers are identified as
internal clients. They are important because the relationship requires the giving and
receiving of information, technical skills, assistance, etc. The MSA process captures
feedback from these customers to enhance performance. In this way, MSA provides a
tool, that by establishing baseline performance measures, supports incremental and
MSA provides an assessment alternative for TQM proponents (Deming, 1986)
who urged doing away with performance appraisal because the system is the problem, not
the individual. It responds to these concerns by focusing on individual job behaviors,
which are controllable by the individual (Latham and Wexley, 1994). Rather than setting
individual against individual, it benchmarks individual profiles and sets standards of
excellence. This fosters employee participation, continuous improvement, and sustained
The MSA process communicates the vision to the organization by focusing on
forward-looking behavioral aspects (London and Beatty, 1993). The multisource
assessment survey focuses on behaviors, on how the employee accomplishes the job,
rather than on the output (Schneier, 1977). The competencies of MSA can be developed
to reflect the organizations vision and values. Core competencies may be identified that
define behaviors required of successful employees in three years, for example. This helps
people move beyond the this is how weve always done it mentality.
MSA behavioral standards expand beyond the typical quantitative results, such as
the financial or operational objectives typically included in MBO-type appraisal
processes, to include behaviors, or how the job is done. These may be leader and team
behaviors, such as cooperation, planning, delegation and teamwork (London and Beatty,
1993). MSA introduces behavioral competencies to the performance equation. Hence,
how something is accomplished has importance as well as what was accomplished.
This is relevant because [o]ne of the greatest weaknesses of nearly all existing tests is
that they structure the situation in advance and demand a response of a certain kind from
the test taker. . But life outside of tests seldom presents the individual with such clearly
defined alternatives (McClelland, 1973, p. 11). Since the focus of MSA is on
competencies, it supports employee decision-making by recognizing that there is no one
correct answer, but generally requires employees to make the best decision based on the
Defining the values of the organization in behavioral terms reinforces employee
focus on core values such as teamwork, quality or fairness. This is important because:
[T]elling someone to be more fair is pointless advice; we all believe we
are fair.. . However, explicitly and quantitatively identifying a persons
behavioral tendency .. can have a more substantial and specific impact
on this persons behavior, particularly when the person realizes that
those actions are not going unnoticed, and that no one is really fooled
(Sashkin and Williams, 1990, pp. 69-70).
The MSA process increases communication within the organization (Bemardin
and Beatty, 1984). Communication is enhanced through the feedback process which
includes the supervisor and co-workers. The process focuses employee activities because
the behaviors communicate the organizational mission, which aligns employees
behaviors, their jobs and the organizational focus. This helps employees better
understand the organizations goals.
The MSA process is highly participatory because it uses multiple raters and
increases participation beyond the supervisor in the performance appraisal process
(Schneier, 1977). The MSA approach is consistent with the high-involvement-oriented
management approachas opposed to a control-oriented approachsuitable for the
effective contemporary organization.
The involvement-oriented approach argues that employees often can
add significant value to the quality of products and services by
solving problems and responding correctly to unpredicted
events...Instead of holding that employees should be programmed to
respond correctly to problems, the involvement-oriented approach
argues that employees should be given general missions and
philosophies as guides for their behavior and then allowed to
respond to the customer as they see fit (Lawler, 1992, p. 35).
This approach offers a change from a more directive mode of management
because it focuses on appropriate behaviors and provides guidelines for employees, not
one right answer. The process responds to challenges for greater employee involvement.
Communication and participation serve to foster a sense of commitment to the
system. Commitment is a necessary prerequisite for accurate ratings(p. 14). MSA is a
tool for evaluating performance that accommodates the new public organizations
realities and requirements for involving employees. The value orientation of multisource
assessment is consistent with winning adherence to norms (Barzelay, 1992, p. 125),
proposed for the post-bureaucratic view of public sector organizations.
Challenges to Implementing MSA
The uniqueness of MSA raises several critical concerns: it threatens the
supervisors traditional role based on control, adds additional responsibility to work
associates and direct reports regarding evaluation, and may provide different rating
groups scores that do not converge due to different interactions with the individual. The
role of supervisors presents a challenge to implementing MSA in organizations. Many
are threatened by a process that does not rely on their judgment alone and instead requires
integrating different perspectives, including those from employees lower in the ranks of
the hierarchy or in other parts of the organization, such as internal customers. Training
can assist supervisors understanding of their new role.
MSA shifts the burden of evaluation from the supervisor alone to an evaluation
team. This requires that evaluators are knowledgeable of the individual and the process
and also have the time to fill out the evaluation form. Some associates may resent the
imposition on their job responsibilities, which previously did not include giving feedback
MSA receives input from a variety of co-workers, colleagues, peers, direct
reports, and the supervisor. This prototype presents the potential that these perspectives
are different. It is suggested that regarding differences between rating categories, if there
are significant differences, these need to be noted (Schneier, 1977).
MSA requires administrative overhead to manage data analysis and report
generation. When multisource assessment is introduced, the challenges of data
management are multiplied. For example, a feedback receiver with an evaluation team of
nine people, plus self-assessment, using a survey of 30 criteria, has a data set of 300
pieces of information. This changes the process for performance appraisal from
established practice. A comprehensive implementation plan can address these concerns.
One challenge to implementation regards the threat potentially provided to
feedback receivers by the anonymity of raters. This raises the concern of declining
reliability and validity due to a lack of individual accountability. Several safeguards can
be introduced to minimize these threats. One is Windsorized mean scoring to stabilize the
mean and reduce the impact of an outlier rating on the performance score. Also,
calculating the degree of agreement provides an index of the consistency of ratings within
the evaluation team.
MSA as a Fairer Performance Appraisal Alternative
The MSA process provides several unique characteristics to the performance
appraisal process. The most significant change is expanding the evaluation team. This
major change provides the basis for the hypotheses being tested in this project. The
overall thesis is that by expanding the evaluation team from one person (the supervisor) to
a team of co-workers with whom the feedback receiver interacts, it is expected that the
fairness of the appraisal, both perceptually and empirically, will increase. The following
section identifies the various hypotheses, both primary and secondary, relating them to
the research previously cited.
Perceived fairness of performance appraisals may be based upon a number of
factorsorganizational trust, input, methods of data collection, rater training, reliability,
and anonymity. A key factor is the knowledge the rater has of the individual being
assessed. It is proposed that expanding the evaluation team from one perspective to
multiple perspectives, and with those perspectives including people of different status in
relation to the subject, will be perceived as providing a fairer appraisal rating by the
feedback receiver. Multisource assessment appraisal process is expected to be perceived
as fairer than supervisor-only appraisals because an evaluation team made up of
individuals with a high degree of knowledge of the individuals performance provides the
rating. Therefore, the primary hypothesis is as follows,
Hypothesis 1: Tthe multisource appraisal process provides a fairer method of
assessing individual performance than traditional supervisor-only
performance appraisals as perceived by feedback receivers.
Single-source appraisal models are sometimes viewed as biased against older
employees, women and people of color. Even today, as diversity awareness and training
have increased, many people still perceive these biases to be present, particularly since
the employee is dependent upon one individual to render a performance judgment. The
MSA model expands the number of evaluators to include more perspectives. Expanding
the evaluation team from one perspective (the supervisor) to multiple perspectives (co-
workers and clients) is expected to increase the feedback receivers perception of fairness.
Specifically, this enhanced perception of fairness will be noted by women, older
employees, and people of color. Because of perceived discrimination against older
employees, women and people of color, it is expected that issues of gender, age and
ethnicity are affected by the assessment method. While it is anticipated that all
demographic categories will rate the MSA process as fairer, these distinctions are useful
because the intent is not to replace one discriminatory process with another. Thus:
Hypothesis 2: Multisource assessment is rated as fairer by women than
Hypothesis 3: Multisource assessment is rated as fairer by older employees
than supervisor-only appraisals.
Hypothesis 4: Multisource assessment is rated as fairer by people of color
than supervisor-only appraisals.
The evaluation team consists of a selection of co-workers, including the
supervisor and (for supervisors) direct reports. The expansion of the evaluation team is
expected to be viewed as increasing the completeness of the information provided. These
evaluators will be selected from the knowledge network around the individual to ensure
that they have an awareness of the individuals job performance (Edwards, Ewen and
ONeal, 1994). The knowledge network consists of colleagues, peers, direct reports,
internal customers, plus the supervisor. Expanding the performance perspectives to a
team of evaluators is expected to enhance the participants feeling that the process is a
more complete and accurate rating of their performance. The model incorporates
perspectives from co-workers and other clients into an overall rating. These potential
evaluators thus have a high degree of knowledge regarding the individuals work
performance behavior. Thus,
Hypothesis 5: Multisource assessment appraisal process is perceived as
fairer because it represents a more complete and accurate assessment of
performance than the traditional appraisal process.
Traditional appraisal processes, such as MBO, focus on results or output. This is
consistent with the rational model of organization. There is a growing awareness,
however, that how the job is accomplished is also important. This realization is
facilitated by team-management, a service orientation plus a growing awareness of
interdependency within organizations. The multisource assessment model provides
behavioral feedback on those core competencies identified as important to successful
work performance within the organization. This is in order for the individual to be
successful and for the organizational to reach its goals. The inclusion of behaviors, or
how the job is accomplished, is expected to be viewed by feedback receivers as
improving the feedback. The focus on behaviors will be viewed as more useful to
feedback receivers than an evaluation based on output. Thus:
Hypothesis 6: This feedback information from the evaluation team,
including co-workers and internal customers, will be viewed as more useful
to the feedback receiver than feedback only from the supervisor.
The lack of performance differentiation for single-source assessment provides
feedback that is inherently demotivating. Traditional appraisal processes provide little
differentiation among performers. This is because of the number of rating choices, e.g.,
three or five ratings. Supervisor ratings often provide undifferentiated performance levels
where everyone is rated basically the same. High performers are not compelled to
continue to provide high performance because undifferentiated ratings leading to
undifferentiated rewards. This is pointed out in Adams equity theory. Individuals also
cannot focus improvement efforts because they are not able to identify areas needing
The MSA process provides an evaluation team consisting of co-workers, clients,
direct reports and the supervisor. This evaluation team of co-workers provides greater
differentiation among performers than the one perspective of the single-source model.
This differentiation will be viewed as more motivational because multisource assessment
provides clearer identification of strengths and developmental opportunities. The
feedback obviously is provided from a broader perspective. The inclusion of co-workers
is anticipated to be more motivational, also, because respect from co-workers, or being
valued, is important to many people (Rosen, 1993). Thus:
Hypothesis 7: Multisource assessment feedback will be viewed by
individual feedback receivers as being more motivational to improve
performance than traditional single-source appraisal feedback.
The multisource assessment process expands the evaluation team to include co-
workers, in addition to supervisory input traditionally received in other performance
appraisal processes. The earlier test regarding completeness and accuracy of the
performance rating will be built on by asking feedback receivers whether they have
confidence and trust in the performance rating. The expansion of the evaluation team
beyond one perspective is expected to increase feedback receivers confidence in their
performance rating because it incorporates input from multiple sources, thus reducing the
chance of bias or insufficient information, which are drawbacks of the single-source
Hypothesis 8: Individual feedback receivers have more confidence
(trust) in the multisource assessment feedback results than traditional
single-source appraisal feedback.
A performance appraisal process must have safeguards inherent in the model for
feedback receivers to view it as fair. The multisource assessment process expands the
evaluation team to include co-workers and other colleagues, which expands participation
in the process and increases the perspectives on an individuals performance. Thus:
Hypothesis 9: The multisource assessment feedback will be viewed by
feedback receivers as providing more safeguards than traditional single-
source appraisal processes.
An additional hypothesis will be tested empirically. As suggested by the above
discussion on differentiation of ratings, the multisource assessment process is anticipated
to provide greater differentiation of performance ratings. A 10-point scale with 5 anchors
has been used in MSA projects based on users requesting greater choice in assessing
performance. The 10-point scale will be used in this project. It is also anticipated that an
evaluation team providing multiple perspectives will better differentiate performance
levels than a single assessor with limited information. Thus:
Hypothesis 10: The multisource assessment process provides greater
differentiation of performance than traditional single-source appraisal
A NEW PERFORMANCE APPRAISAL PROCESS
IN A FEDERAL AGENCY
The bureaucratic model of the past is not working in the face of current
organizational challenges. The dual task of increasing productivity in order to be
effective while fostering more human and involved action with its workforce is central.
One key change that must be undertaken to motivate productivity and self-worth is a
revision of the performance appraisal method. In order for the new process to gain
acceptance, it must be perceived as providing accurate ratings that are fair to participants.
This study advances the research in the area of fairness in performance appraisal
methodology for several reasons specifically related to perceptual fairness, the first step
in meeting the objective of process fairness. First, it is a study undertaken within an
organizational setting. Little of the research on procedural fairness or of performance
appraisal has occurred within an actual organization. Most studies have been undertaken
in graduate schools and executive programs. Second, it offers a quantitative test, using
pre- and post-test implementation data of perceptual fairness from the perspective of the
participant in a different approach to performance appraisal, the MSA model. Finally, it
provides an empirical test of actual ratings distribution to determine if multisource
assessment evaluations provide greater differentiation of ratings as compared to
traditional single-source appraisal models. Each of these contributions is significant to
assisting public organizations better meet their responsibilities to both constituents and to
The following chapter describes the organization in which this field test occurred.
The methodology is then described for implementing the MSA process in the
organization. Finally, the research issues raised by the methodology are discussed in a
section on research design issues.
Description of the Organization
This research project focuses on a 450-person operations office for a federal
agency. The agencys headquarters personnel staff agreed to this field office acting as a
pilot site for testing the multisource appraisal process. The agreement occurred for a
variety of reasons. The National Performance Review (1994) provided the foundation for
allowing agencies to try new methods of enhancing productivity, including performance
appraisal. This particular agency communicated its desire to participate in the pilot.
Employee climate surveys throughout the entire organization indicated high
levels of discontent with the existing process which relied on a single source (the
supervisor) to provide the individual performance appraisal rating. An employee focus
group in November 1992 pointed out that the current performance appraisal process did
not reflect actual performance and provided little guidance for evaluating. In addition to
the declining employee morale throughout the organization, there were also concerns
noted about the current process. Specifically, an internal data analysis examined
performance ratings in 1989, 1990 and 1991 by ethnicity and gender. The results of this
study showed that people of color, particularly black males, were underrepresented in the
higher rating classifications. This finding caused considerable concern among senior
management and agency personnel specialists. Additionally, the black employees group
had specifically discussed this issue as an area of concern with senior management. Thus,
the need for a change was identified, plus employees desired a new process.
Additionally, the Office of Personnel Management agreed to the federal agencys
participation in a pilot to implement multisource assessment, also referenced as 360
Feedback, as part of its effort to identify and develop alternative and better methods of
assessing performance. The key foci of OPM and the agencys headquarters staff were
regarding project outputs, such as how employees perceived the quality of their
performance feedback, as well as whether the process improved organizational integration
with an existing TQM initiative.
The current performance appraisal process consisted of three areas of
performance standards. The areas are identified as job knowledge, continuous
improvement, and leadership/team building. The rating criteria were written at three
levels (marginal, fully successful and outstanding). The assessment came from the
supervisor. The end-of-year review consisted of the supervisors view of the individuals
performance. Five rating levels would then be assigned as outstanding, highly, fully,
marginal and unsatisfactory. It represents a single-source supervisor model similar to that
of other federal government agencies.
This organization is characterized by several key factors. It is made up of
primarily two types of employees-those in the central office/administrative staff and
those in the field. Both groups are generally well skilled. The education level is
relatively high, as are the technical requirements for the jobs. The office/administrative
group is characterized by general managers and functional administrators, i.e., in areas
such as budgeting, finance, human resources, and research. The field personnel are both
supervisory and technical. They are highly skilled, requiring significant technical
knowledge. Because of the nature of the organizations mission, safety is a critical issue
for all personnel. The almost 500 employees at the federal agency also have oversight
responsibility for approximately 15,000 contract workers, operating primarily for
Westinghouse and Kaiser Engineering.
The researcher is a consultant under contract to the organization to implement a
multisource assessment process for performance management by January 1994.
Designing the MSA Process
Several important elements of the performance appraisal process should be
considered when designing a multisource assessment (MSA) process. These include the
rating content, involvement of employees in system design, the item types and format,
item relevance, the implementation plan, the frequency of the self-assessment rating, and
its uses (London and Beatty, 1993). These elements are discussed in the following
section. They are important because the model for multisource assessment is different
from traditional organizational feedback models. Addressing these issues helps create a
system that is viewed as accurate and fair by employees.
The content of the ratings in MSA focuses on performance behaviors that are
essential to the success of the organization. These may be general categories of behavior,
such as sharing information or meeting commitments, or they may be specific
organizational initiatives such as quality initiatives. This ensures relevancy to the actual
jobs. Job analysis is a critical component when designing a performance appraisal
process. The content of the MSA competencies must be identified based on the
requirements of the jobs for the organization.
Employee involvement in the design process serves multiple functions. First, it
signals the organizations commitment to genuine empowerment of employees. Second,
the behaviors identified for assessment in the MSA processcommunicate organizational
change and direction to employees. Employees involvement in providing input on
performance also serves to increase their assessment of perceived fairness of the process
(Thibaut and Walker, 1975).
The items types and format are another important consideration. Types of items
may include different factors than traditional single-source processes. For example, the
competencies to be rated may be interpersonal (such as written communication),
technical, or managerial (such as developing others). In many cases, the items are
behavioral statements and descriptions that take the organizational values and translate
them into job-related behaviors.
Item relevance may apply to general abilities or specific behaviors of an
individual (London and Beatty, 1993). However, in order to be deemed relevant, items
must be seen by raters as reflective of prototypical organizational behaviors and as
important to the organizations success in reaching its objectives. For example, relevant
items might include behaviors focused on serving the customer or reinforcing employee
involvement in decision-making.
An implementation plan is important if the MSA process is to succeed because
the MSA model is different for employees. To foster understanding and acceptance, the
implementation plan must, first, clearly articulate the goals of the MSA program. Second,
training helps participants understand the process. The training should include the
purpose of the ratings, how data results are compiled, and the method of providing
feedback to individuals (London and Beatty, 1993).
Rating frequency should be decided (Landy, Barnes and Murphy, 1978). Most
formal appraisal processes take place annually. Informal feedback can occur throughout
the year. Ideally, organizations strive to create environments where feedback is given and
received on a constant basis.
Identifying appropriate evaluation participants, including whether or not to
include self-assessment in the process, is a question which must be addressed. For
example, self-ratings provide valuable insight to the individual from a developmental
perspective. These may factor into the rating, or be 0-weighted, to eliminate
opportunity for inflated self-ratings to occur. Also, whether or not in include feedback
from customers, both internal and external, should be addressed.
An additional consideration is summarizing and aggregating the data. There are
issues such as how to summarize and aggregate the various ratings. These include what
weight to assign to different categories of evaluators. It may also be important to identify
who will aggregate the data. Accuracy and confidentiality are important.
The report format is important. The information received from MSA is different
from the traditional single-source format with which most employees are familiar. The
information must help the employees identify their strengths and development
opportunities. Making the information clear and understandable overall is important.
Addressing each of these areas supports implementation of MSA, and is
important because the MSA model differs substantially from the traditional performance
assessment with which employees are familiar.
Steps of the Multisource Appraisal Process
The multisource assessment appraisal process will be used for this research
project. This model for multisource assessment involves the following (Edwards, 1990):
A systematic method of identifying evaluation teams to provide a
comprehensive and integrated view of the individuals performance;
An integrated training process to ensure high-integrity rater input and
motivated response by ratees.
A software-based automated system to reduce administrative costs and ease
A demonstrated method of discriminating among different individuals
performance ratings for selection decisions such as succession planning and
A system of process and technological safeguards to provide anonymity and
confidentiality to raters and feedback receivers.
The project entailed a 10-step work plan. The steps to implement the multisource
appraisal process are shown in Figure 4.1 and represent the key elements in building,
using, and evaluating it. The steps are described below as implemented in the federal
agency. Some references may be to 360 Feedback, which was the name used by the
agency for this project.
1. Application Selection and Promotional Plan
The federal agency selected itself to implement the MSA model for performance
appraisal. The project was targeted to integrate its TQM process with individual
performance appraisal in the organization. Additionally, it also focused on employee
development, performance coaching, and identifying training needs.
A number of key policy decisions were addressed for implementation of the
project. They concerned: use of the information, access, anonymity of feedback
providers, confidentiality of feedback receivers, evaluation team guidelines, data
collection method, and retention of data. The rationale for each of these decisions are as
follows. The use of the information would be for performance appraisal. No one,
including supervisors or feedback receivers, would have access to raw data. Anonymity
of ratings would be protected by not disclosing a category of evaluator score if fewer than
3 scores were captured. The feedback data would be collected on scannable paper forms
and the comments keyed in. Also, feedback receivers would not have access to any raw
scores. Only the individual feedback receiver and the supervisor would see the feedback
report. Each evaluation team would consist of the individual, the supervisor, colleagues
and direct reports, if appropriate.
The process was to be highly participative, with a design team made up of
employees from throughout the organization. Employee briefings occurred throughout
the organization to describe the new MSA process. The initial intent of the human
resources project leaders was to implement MSA as a developmental-only process, that is,
that the individual alone would receive the feedback with no formal review with the
supervisor required. Initially, the intent was that an assessment of whether to next use the
process for performance appraisal would be made after the developmental-only pilot.
However, as employees heard of the MSA process, they reiterated their feeling that the
current process was not developmental and did not help them identify strengths and
weaknesses. Consequently, employees recommended that MSA should be implemented
for use as the formal performance appraisal process. An internal survey was taken
throughout the organization, and employees confirmed their desire for MSA to be used
for performance appraisal.
A promotional plan was developed to communicate the vision, goals and
objectives of the MSA process in a manner understandable to everyone. The objectives
of the communication plan were: establish clear performance expectations, foster
employee awareness of the organizations vision and values, be easy to administer and
understand, support team development, enhance personal awareness and growth, and,
predominantly, to support the organizations TQM initiative. The promotional plan
created a specially designed set of newsletters which were distributed every 4-6 weeks.
At the agency, the employee briefing packet focused on quality. The focus of the
communication plan was that the performance appraisal process was intended to align the
agencys commitment to and support of the Vision and Quality Principles. The design
was intended to align employees self-interest with the organizations commitment to
Figure 4.1: MSA Process
A communications plan supports the process implementation, enabling
participants to understand the process. Understanding improves perceptions of equity:
The [foundation for this understanding for maintaining the appearance of equitya
common vision, true teamwork, participative planning, constant communication, genuine
feedback from customers and peers, more communication and still more communication
(Scholtes, 1990, p. 50).
2. Competency Development
Developing the MSA competencies is the critical first step. The competency
instrument was created by a focus group of 18 members, made up of a cross-section of
employees from throughout the organization. The focus group was assembled from the
design team and included other organizational perspectives.
This important first step is also difficult. A single set of core competencies was
developed for all employees throughout the organization to provide a common reference
on behavior. This ensures consistency of ratings throughout the organization, regardless
of the department in which an employee works (Latham and Wexley, 1982, p. 71). The
competencies were developed by answering the question What kind of employee
behaviors are needed for the organization to be successful today and in the future?
Nominal group technique was used to facilitate competency development and group
agreement. The draft competencies were distributed throughout the organization for
comments, then modified based on comments from employees. Second, an additional set
of criteria was created to supplement the list of supervisory behaviors. Finally,
behavioral definitions of the performance expectations were developed.
An important aspect of competency development is communicating the
competencies widely so that organizational members understand what is expected of
them. The revised version of the competencies was then distributed again for comments.
The validation process included asking respondents (1) if the criteria captured all the core
competencies necessary, (2) if the competencies were redundant or unclear, (3) whether
individuals could be assessed on the competencies, and (4) if they had suggestions for
format changes. The competencies were identified to reinforce the organizations values
and goals. The behavioral surveys for both the supervisory and non-supervisory
employee groups are found in Appendix B.
3. MSA Training 1: Providing Feedback
The first training introduced participants (feedback receivers) to the MSA
process. It was a one-and-a-half hour session. The learning objectives were to help
participants understand the process benefits and advantages and to reinforce the
newsletter information. Process pre-training reviewed the objectives of the pilot as
identified by the agency, the timelines, how to give and get feedback, how to complete
behavior feedback surveys, defining the rating scale, and avoidance of typical respondent
errors. This training was included to enhance accuracy because training individuals on
how to reduce rating errors has a significant impact on actual rating accuracy (Fay and
Latham, 1982). The training also outlined the next action steps for respondents and
participants to take in the MSA process, and defined their roles and responsibilities.
The reasons given for introducing the process were that it (1) reflects multiple
perspectives, (2) enhances teamwork, (3) is keyed to actual performance, (4) is
participative, and (5) includes safeguards. The training also reinforced support of the
organizations TQM initiative. The training did not position the process as specifically
intended to enhance fairness. Rather the focus was on better information for employee
4. Process: Select Evaluation Team
Evaluation team guidelines were established by the MSA Design Team. These
guidelines emphasized the selection of six or more work associates who were in a
position to provide quality (that is, accurate) performance feedback. It was suggested that
there be 6-9 evaluators on each individuals team. The evaluators were to include self
and supervisor on all teams. In addition, individuals were to select 5-8 colleagues.
Finally, supervisors would identify all direct reports for their teams. Evaluation team
categories were categorized as self, supervisor, colleagues, and direct reports.
Contractors at the federal agency site could not be selected for conflict-of-interest
reasons. However, federal agency employees at other locations could be selected as
colleagues to provide input. After identifying the selection of an evaluation team,
individuals were instructed to meet with their immediate supervisor for possible revision
and sign-off approval.
5. Provide Feedback
The next step was for members of the evaluation team to provide their
assessments. The process design was intended to minimize administrative overhead,
especially time cost, for both respondents and the process administrator. Response time
for a typical evaluator was kept at approximately ten to fifteen minutes by keeping the
survey short and focused on core competencies. This served to minimize respondent
fatigue and to maximize respondents willingness to provide quality information. Also,
the survey forms for the agency were pre-printed on optical scan forms printed by a third-
party vendor. This minimized the administrative time required for the software operator
to input the evaluation data.
The process included safeguards to assure that individual feedback was not
identifiable to feedback receivers. For example, the process was designed to maximize
respondent anonymity. This was accomplished by identifying a sufficient number of
respondents. Trimmed mean scoring was also used for the evaluation, so the highest and
lowest scores on each behavior competency were eliminated prior to scoring. It is
important to assure respondent anonymity to prevent a scenario where respondents either
refuse to respond or respond with uniformly inflated responses (Latham and Wexley,
1994), which would undermine the accuracy and fairness of the process.
A ten-point scale was used to help evaluators provide greater differentiation of
feedback regarding performance levels. Five definitional anchors were provided. While
current research exists suggesting that reliability declines with fewer than 3 or more than
9 choices on the scale, this information has not been tested in a multisource process
environment. The rating scale is as follows:
3-4 Below Expectations
5-6 Meets Expectations
7-8 Exceeds Expectations
Additionally, respondents were asked to provide narrative comments if a 1-2 or 9-
10 rating was given. This was to help the individual feedback receiver better target areas
for greater contribution or improvement in the action-planning stage. Boosting the power
of the feedback by combining numbers and verbal feedback increases the chances of
getting through to the recipient so that he or she sees clearly the need to improve and is
truly motivated to improve (Kaplan, 1993, p. 304).
6. Score and Create Reports
The automated scoring and reporting used field-tested methodology that was
developed based on statistically sound mathematical practices and on reports shown by
prior user feedback to best represent the information (Edwards and Verdini, 1986). The
reports were kept understandable and as user-friendly as possible. The design team
reviewed the sample report packet to select various parameters prior to project
implementation to ensure that the agencys employees would understand the information.
The package selected included a ranking report, a profile report, a perspectives report and
The ranking report shows an individuals competencies arrayed from the highest
to the lowest. This quickly answered the question, How did I do? The profile report
shows the actual results with the competencies arrayed from highest to lowest rating
score. It also shows degree of agreement, how many raters responded, average composite
score, self-score, and supervisors score. The perspectives report breaks out the category
rating, i.e., supervisor, colleague, direct report, to identify the various perspectives of
evaluators. Finally, narrative comments provided additional insight to help feedback
receivers understand their scores. (A sample package is shown in Appendix B).
Trimmed mean or Windsorized scoring was used, where the high and low score
are eliminated and the remaining scores are then averaged. The performance score for
each individual competency was then averaged for an overall composite score of
behavioral performance. Windsorized means were used to stabilize the mean of the small
(generally 5-9 respondent) sample. While this potentially resulted in a loss of
information, it also reduced potential bias to the feedback receiver.
All criteria weightings were given equal weight, except self-score, which was
zero-weighted. The supervisors input was thus weighted equally to other evaluation
team members. A degree of agreement for each criterion was shown on the report. The
degree of agreement is one minus the standard deviation, and provides an index of how
much inter-rater agreement there is within the evaluation team on the individuals
performance (Edwards and Verdini, 1986). If degree of agreement was low, that is below
0.5, the individual was referred to the perspectives report to see if his/her actions are
viewed as different by different groups. For example, a supervisor may share
information with colleagues well, but share poorly with direct reports. The degree of
agreement report assists in identifying such differences in performance.
Similar to the importance of assuring respondent anonymity is the assurance of
feedback receiver confidentiality. Safeguards were developed so that the behavior
feedback for each participant was confidential. These safeguards included a rule that the
individual and the supervisor were the only receivers of the actual reports; no other copies
were retained. The data were secured. The raw data of evaluators were not accessible to
anyone, including the operator, due to encrypted programming of the evaluation software.
The data were retained under the standard agency guidelines for personnel data.
7. Training II: Receiving Feedback and Action Planning
When participants received their behavior feedback, they attended a session on
receiving feedback training, called Sustaining Change. This sessions objectives were to
teach people how to receive feedback and to help them understand how behavior
feedback from the MSA process is different from other feedback they may have received
prior to this experience. The sessions key focus was on how to use behavior feedback to
develop action plans. The report format was discussed to ensure that feedback receivers
understood how to interpret their results.
To accomplish this, the training included assistance in creating action plans based
on the behavior feedback. The action plans were intended to tie back to the areas targeted
by the evaluation team for development based on the behavior feedback. To assist
individual feedback receivers in doing this, a group exercise occurred where groups of
three split out to share ideas on action planning. The individual reports were not
reviewed by the group. Each group identified resources in the organization and
community, as well as ideas for improvement. At the conclusion, the individuals were
advised to meet with their supervisor for the formal review.
8. Process Reports and Safeguards
The automated assessment process includes project and management reports
showing a variety of statistics such as response rates and item reliability for each. The
project and management reports were used to create organization intelligence on
aggregate training and development needs, measures of training effectiveness, measures
of process on culture change and alignment of organizational behaviors and values. The
information could also be used for such applications as reorganizing, succession planning,
placement, and other selection decision processes.
9. Target Action Plans
The behavior feedback process targets areas of strength and areas for
development. Action planning uses the behavior feedback to identify the targeted
developmental actions. This ongoing process, initiated in the Receiving Feedback
training, was reinforced in subsequent communication activities. The action-planning
process included sharing the action plan and assessing progress with the supervisor.
To help supervisors understand their new role with MSA, they attended two half-
day sessions titled Leadership Coaching. These sessions were designed to assist the
supervisors in understanding their new role as coaches, versus judges, of performance.
Support systems, such as employee or career development, were developed to offer
suggestions for continuous learning and professional development opportunities both
inside and outside the organization. Various role playing of facilitating action planning
discussions occurred in these sessions. Participants also had an opportunity to voice
concerns regarding the new process, including fears regarding loss of control.
10. Process Evaluation
The MSA process was assessed in a pre-and post-implementation evaluation for
factors such as quality, accuracy, and other aspects that were identified as important to
the organization and to participants. OPM identified several factors in the pre- and post-
test assessment of the pilot to ascertain whether to continue the process. These factors
include whether the process: promotes teamwork, aligns quality principles with agencys
mission, improves cooperation with co-workers, helps individuals serve their customers
better, and reflects the organizations quality initiative. These are different from the
criteria concerning fairness. The criteria developed to address fairness include: the
process is fair; the information will be fairly used for rewards, promotions and training;
and input from work associates improves fairness and provides safeguards. The survey
also includes criteria regarding user perception of fairness (See Appendix B).
The structured survey process was supplemented with one-on-one interviews to
examine various aspects of the MSA process, to make recommendations for continuous
system process improvements, and to gather more information on user perceptions of the
system. A discussion guide was used by the researchers to ensure consistency in
collecting the data. Appendix D presents the discussion guide for reference.
Overall Research Design
This is a field test of the multisource assessment model for performance
appraisal. The empirical data come from a multisource assessment project implemented
at a 450-person federal agency. Fairness will be evaluated by two-methods. The primary
test is regarding user perceptions of fairness. This involves the use of a pre- and post-test
survey to collect information from assessment receivers on their perception of process
fairness. A focus group process developed the fairness questionnaire for the second
phase of the project. The pre-test is given to participants after the completion of their
traditional supervisor-only assessment regarding their perceived level of fairness of the
process and the resultant rating. After implementation of multisource assessment, a post-
test is conducted to assess the perceived level of fairness of the multisource assessment
process and the resultant rating.