Citation
An exploratory study of how policy research is used in the policy process by policymaker staff and public administration

Material Information

Title:
An exploratory study of how policy research is used in the policy process by policymaker staff and public administration
Creator:
Anthes, Catherine Quigley
Place of Publication:
Denver, CO
Publisher:
University of Colorado Denver
Publication Date:
Language:
English
Physical Description:
xvii, 170 leaves : ; 28 cm.

Subjects

Subjects / Keywords:
Political planning -- Research -- United States ( lcsh )
Public administration -- Research -- United States ( lcsh )
Political planning -- Research ( fast )
Public administration -- Research ( fast )
United States ( fast )
Genre:
bibliography ( marcgt )
theses ( marcgt )
non-fiction ( marcgt )

Notes

Thesis:
Thesis (Ph. D.)--University of Colorado at Denver and Health Sciences Center, 2007. Public affairs
Bibliography:
Includes bibliographical references (leaves 162-170).
General Note:
School of Public Affairs
Statement of Responsibility:
by Catherine Quigley Anthes.

Record Information

Source Institution:
University of Colorado Denver
Holding Location:
|Auraria Library
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
166327663 ( OCLC )
ocn166327663

Downloads

This item is only available as the following downloads:


Full Text

PAGE 1

AN EXPLORATORY STUDY OF HOW POLICY RESEARCH IS USED IN THE POLICY PROCESS BY POLICYMAKER STAFF AND PUBLIC ADMINISTRATORS by Catherine Quigley Anthes B.S. University of Oregon, 1996 M.A. University of Colorado at Denver, 200 I A thesis submitted to the University of Colorado at Denver and Health Sciences Center in partial fulfillment of Doctor of Philosophy Graduate School of Public Affairs 2007

PAGE 2

This thesis for the Doctor of Philosophy Degree by Catherine Quigley Anthes has been approved by Paul Teske "Jpatrick Charles R. Coble Date

PAGE 3

Anthes, Catherine, Q. (Ph.D., Graduate School of Public Affairs) An Exploratory Study of How Policy Research is Used in the Policy Process by Policymaker Staff and Public Administrators Dissertation directed by Professor Paul Teske ABSTRACT This research investigates how policy research, analysis and information are used by actors other than policymakers in the political and policy processes. These other actors, particularly policymaker staff members and middleto high-level administrators in state government, are important brokers of the policy process and often neglected in the literature on research usage. The findings of this mixed-method study, using a survey and interviews, provide ideas to policy analysts to better integrate policy research into the political process and understand the way it is conducted, delivered, and offered into the policy process. This is a crucial step in improving policy research to determine the similarities and differences in how policymakers and staff and administrators use policy research when performing their jobs. Overall, I found that policy staffers and administrators reported very high uses of research in their jobs at all stages of the policy process. The most common types of research use for policy staffers and administrators are enlightenment (providing

PAGE 4

needed information to policymakers, the public and other staffers), agenda setting (placing a topic on the policy agenda) and rationalization (defending an argument or certain policy option). In addition, they reported high use of research at the policy stages concerned with defining the policy problem and determining policy options. Research use differed in statistically significant ways between staffers and administrators in the following manners: (1) to provide a rationale for certain policies (higher for administrators), (2) to use as a policy is being considered in the legislative process (higher for staffers), (3) to use in testimony or legislative debates (higher for staffers), ( 4) to raise awareness or understanding of a policy issue with staff (higher for administrators), (5) to help policymakers avoid making decisions on political whims (higher for administrators), (6) to gamer public support for an issue (higher for administrators), and (7) to help guide the implementation process of policies (higher for administrators). Some of these findings (1, 5 and 6) contradict my exploratory theory, which hypothesizes that administrators are less likely to use research for political purposes than policymaker staffers. This.abstract accurately represents the content ofthe candidate's thesis. I recommend its publication. Signed __ Paul Teske

PAGE 5

DEDICATION To my friends and family, who over the last six years listened to me complain about my workload, were patient when I had to say 'no,' and always had no doubt that I could get through this program while working full-time. A special dedication to my mother and father who set me up in life to go this far, who bore the brunt of the complaints, who made me meals and offered playtime with the dog when I needed comfort and support the most. I can't thank you enough.

PAGE 6

ACKNOWLEDGMENTS Many people supported and assisted me in this endeavor. First, and foremost, is Paul Teske, my chair, who has seen me through this process from start to finish, gently offered suggestions, ideas and improvements and who has been incredibly responsive throughout all aspects of this process. Jody Fitzpatrick, who assisted greatly in the development of the proposal as well as with the methodology and organization of the dissertation. Charles Coble, my boss when I was at the Education Commission of the States, has always been an invaluable mentor and supporter of my work. And Christine Martell, who always gave great advice, was a great listener and supporter throughout my career at GSP A. I could not have done this without the assistance of several members of my cohort including: Tim Noe and Nancy VanDeMark for being my support group at Old Chicago, the Rio and Cafe Calore! Nancy was also my SPSS and statistical analysis tutor. Aleah Horstman for the inspiration she provided me as she was raising a newborn child with a full time job and finished her dissertation before me! Cathy Walker for the constant support to me as a friend and invaluable editor over the past six years. Sarah Jamell, Jennifer Gay, Bonnie Kanter and Karl Wunderlich for their unending support, encouragement and ability to listen to me. And Jennie Yeh, for living a parallel Ph.D. life-even if it is across the country! In addition, I am grateful to my past employer, The Education Commission of the States and alJ of my supervisors who supported me, including Ted Sanders, Bob Palaich, Charles Coble and Tricia Coulter.

PAGE 7

TABLE OF CONTENTS Figures ................................................................................................................ xii Tables ................................................................................................................ xiii PREFACE ................................................................................................................ XV CHAPTER 1. INTRODUCTION .......................................................................................... 1 Rationale and Importance of the Research ................................................... 3 Operational Definitions ................................................................................. 5 Research Purpose ........................................................................................... 6 2. REVIEW OF THE LITERATURE ............................................................... 9 Foundations ofPolicy Research: The Policy Sciences ................................ 10 Purpose and Definition of Policy Research ............................................ 10 Methods of the Policy Sciences .............................................................. 12 Policy Research Use .................................................................................... 14 Use Defined ............................................................................................ 14 Policy Research Used by Policymakers ................................................. 15 Instrumental Use of Policy Research ...................................................... 16 VII

PAGE 8

Use of Policy Research to "Enlighten" ................................................... 19 Other Types of Research Use ................................................................. 21 Research Use Affected by the Individual ............................................... 25 How is Research Accessed When It is Used? ........................................ 26 Policy Goals and Political Goals and Their Effect on Research Use .......... 27 The Real Policy Process .............................................................................. 30 Roles Staffers and Administrators Play in the Policymaking Process ... 32 Policy Research Use by Staffers and Administrators ............................. 35 3. THEORY, RESEARCH QUESTIONS AND METHODS ......................... 38 Research Questions ...................................................................................... 38 Exploratory Theory ...................................................................................... 39 Narrative for the policy process depicted in Figure 2 ............................ 42 Exploratory Hypotheses .............................................................................. 44 Research Design and Operational Definitions ............................................. 46 Operational Definitions for Variables in the Exploratory Model.. ......... 46 Methods ....................................................................................................... 48 Data Collection Methods ........................................................................ 50 The Survey Instrument ...................................................................... 51 The Interview Instrument .................................................................. 53 Vlll

PAGE 9

Sample ............................................................................................... 53 Analysis ....................................................................................................... 56 Survey Data Analysis ............................................................................. 57 Survey Data Coding ................................................................................ 57 Interview Data Analysis ......................................................................... 58 Interview Data Coding ............................................................................ 59 Reliability and Validity ............................................................................... 60 Instrument Validity ................................................................................. 62 Method Limitations ..................................................................................... 62 4. RESULTS .................................................................................................... 66 Staffers' and Administrators' Use of Research, Research Question One ... 66 Frequency and Types of Research Use, Research Question Two ............... 71 Policy Research Use and The Policy Process, Research Question Three ... 76 Where Do Staffers and Administrators Get Research? Research Question Four .............................................................................. 81 Staffers' Versus Administrators' Use of Research: Research Question Five ............................................................................... 85 Providing a Rationale for Certain Policies ............................................. 86 Persuading Policymakers to Delay Policy Decisions ............................. 87 Garnering Public Support for a Policy Issue .......................................... 87 lX

PAGE 10

Understanding a Policy Issue in a New Way ......................................... 88 Making Mid-Course Corrections After a Policy is Enacted ................... 90 Guiding the Implementation Process of Policies .................................... 90 Swnmary of Differences Between Role Groups ......................................... 92 Summary Thoughts on the Results ofthe Research Questions ................... 93 Hypotheses ................................................................................................... 94 Hypothesis One ....................................................................................... 94 Hypothesis Two ...................................................................................... 98 l-lypothesis Three .................................................................................. 100 Hypothesis Four .................................................................................... 101 Hypothesis Five .................................................................................... 1 02 5. CONCLUSION ......................................................................................... 105 Contributions to the Field of Public Affairs ............................................. 106 Are policy decisions made in the absence of research ......................... 1 06 Have the purposes and processes of social research adapted in ways that will increase its impact? ................................................................ 1 07 What have we learned about the audience for policy research? ........... 109 Are careers in policy analysis or research in crisis due to a lack of relevance? ................................................................... Ill X

PAGE 11

Conclusions Regarding the Research Purpose .......................................... 113 Conclusions Regarding the Exploratory Theory ....................................... 113 Policy Problem Definition ......................................................................... 115 Policy Goals and Political Goals ............................................................... 115 Political and Democratic Process .............................................................. 119 Policy Adoption or Change ....................................................................... 121 Policy Implementation ............................................................................... 122 Overall Reflections on My Exploratory Research Model ......................... 123 Research Limitations ................................................................................. 126 The Next Step: Further Research on Research Use ................................... 130 Summary .................................................................................................... 133 Hypotheses ................................................................................................. 135 APPENDIX A: Policy Research Use Survey ...................................................................... 139 B: Focus Group Interview Protocol.. .............................................................. 155 C: Interview Protocol .................................................................................... 157 D: Human Subjects Approval Letter and Human Subjects Consent Letter .. 158 BIBLIOGRAPHY ................................................................................................. 162 XI

PAGE 12

LIST OF FIGURES Figure I: Kirkhart's Theory of Evaluation Usage ............................................................. 25 2: Exploratory Theory on Research Use in the Policy Process by Staffers and Administrators ........................................................................ 41, 114 XII

PAGE 13

LIST OF TABLES Table 2.1: Types of Research Use .................................................................................... 22 3.1: Six Types of Research Use-Definitions ........................................................ 48 3.2: Survey Question, Policy State and Type of Research Categorization .......................................................................................... 52 4.1: Frequency of Research Use for Staffers and Administrators By Type of Research Use ........................................................................................ 67 4.2: Survey Factors and Variances ......................................................................... 72 4.3: Frequency ofTypes of Research Use .............................................................. 73 4.4: Sources and Medium of Research for Both Staffers and Administrators .................................................................................................. 81 4.5: Sources of Research for Both Staffers and Administrators ............................. 82 4.6: Medium of Research for Both Staffers and Administrators ............................ 83 4.7: How Access to Research Differs for Policymakers, Staffers and Adininistrators .................................................................................................. 85 4.8: How Does Research Use Differ Between Staffers, Administrators and Policymakers -To provide a rationale? .......................................................... 86 4.9: How Does Research Use Differ Between Staffers, Administrators and Policymakers -To delay policy decisions? ..................................................... 87 4.10: How Does Research Use Differ Between Staffers, Administrators and PolicymakersTo garner public support? ....................................................... 88 Xlll

PAGE 14

Table 4.11: How Does Research Use Differ Between Staffers, Administrators and PolicymakersTo understand or define a policy issue? .................................. 89 4.12: How Does Research Use Differ Between Staffers, Administrators and PolicymakersFor mid-course corrections? .................................................... 90 4.13: How Does Research Use Differ Between Staffers, Administrators and PolicymakersTo guide implementation? ...................................................... 91 4.14: Research Use by Reported Frequency and Policy Stage for Both Staffers and Administrators ..................................................................................... 96 XlV

PAGE 15

PREFACE This thesis investigates how policy research, analysis and information are used by actors other than policymakers in the political and policy process. These other actors, particularly policymaker staff members and middleto high-level administrators in state government, are important brokers of the policy process and are often neglected in the literature on research use. The findings of this mixed method study, using a survey and telephone interviews, will enable policy analysts to better integrate policy research into the political process and understand the way it is conducted, delivered, and offered into the policy process. This is a crucial step in improving policy research to determine the similarities and differences in how policymakers and staff and administrators use policy research when performing their jobs. This research answers the following questions: 1. To what extent do policymaker staff and midto high-level administrators use policy research? 2. Considering the six types of policy research use (agenda setting, enlightenment, instrumental, rationalization, tactical, change agent), which types are most common? XV

PAGE 16

3. When policy research is used, what is the stage of the policy process (problem definition, goal setting, political/democratic process, policy adoption/change, policy implementation) and the type of research used? 4. How do staffers and administrators get access to policy research? From where? 5. How does the use of research differ between policymaker staffers and midto high-level administrators? To answer these questions, I created a survey instrument to measure the amount and type of research use that my sample employs to meet the goals of their jobs. The items on the survey instrument are based on existing survey items from other tested instruments, current research frameworks and a focus group I conducted for the purpose of refining my survey questions. The quantitative data from the survey was analyzed using descriptive and inferential statistical techniques via SPSS. In addition to the data yielded from the survey, I conducted 10 interviews with a variety of policymaker staffers and state-level administrators. These interviews were coded qualitatively using thematic/process coding and provided useful clarification and contextual data to answer my research questions. Overall, I found that policy staffers and administrators reported very high uses of research in their jobs at all stages of the policy process. The most common types of research use for policy staffers and administrators are enlightenment (providing needed information to policymakers, the public and other staffers), XVI

PAGE 17

agenda setting (placing a topic on the policy agenda) and rationalization (defending an argument or certain policy option). In addition, they reported high use of research at the policy stages concerned with defining the policy problem and determining policy options. Research use differed in statistically significant ways between staffers and administrators in the following manners: ( 1) to provide a rationale for certain policies (higher for administrators), (2) to use as a policy is being considered in the legislative process (higher for staffers), (3) to use in testimony or legislative debates (higher for staffers), (4) to raise awareness or understanding of a policy issue with staff (higher for administrators), (5) to help policymakers avoid making decisions on political whims (higher for administrators), (6) to gamer public support for an issue (higher for administrators), and (7) to help guide the implementation process of policies (higher for administrators). Some of these findings (1, 5 and 6) contradict my exploratory theory, which hypothesizes that administrators are less likely to use research for political purposes than policymaker staffers. xvu

PAGE 18

CHAPTER ONE INTRODUCTION A vast amount of policy research is created each year. Hundreds ofpolicy organizations, think tanks, universities, legislative bodies and associations create and disseminate data, information, research and analysis on the public policy issues of our day. Since there is such a vast amount of research, it must be used a lot? Although social policy researchers have been providing information and policy advice to policymakers for decades, researchers argue that policy analysis and other forms of social research are not frequently used by policymakers in making policy decisions (Lindblom & Cohen, 1979; Kirp, 1992; White, 1994; Albaek, 1995; Stone, 1997; Shulock, 1999; Black, 2001; Hird, 2005). Others argue that when research is used, it is either not effective or democratic, nor particularly helpful in crafting public policies that work (Dryzek, 1989; deLeon, 1997; Stone, 1997; Schneider & Ingram, 1997; Hird, 2005). This argument implies-and makes researchers and policy analysts worry-that the information and analyses they help create is not used or useful to the policy process at all. However, as most theories of the policy process articulate, the process is larger than just policies, policymakers, their decisions and the information they use to make those decisions (Sabatier, 1999; Lindblom & Cohen, 1979; Albaek, 1995; White, 1994; Kingdon, 1995, Barmgartner & Jones, 1

PAGE 19

1993; Weimer and Vining, 1999; Bardach, 2000; Birkland, 2001; Dunn, 1994 ). Indeed, there are numerous additional "actors" in the policy process who influence the process individuals such as policymaker staff members, government administrators and policy influencers within administrative ranks. How do these actors participate in getting research "used" in the policy process? The policy process is large and complex and involves both the political goals and the policy goals of the actors within the process each often in conflict with the other (Stone, 1997). These differing goals within the policy process also must influence how, when and what type of research is used for what purposes. Even with this knowledge, many of the teachings and descriptions of policy research assume that the research should be and is simply inserted into the policy decision process and used in a direct way-known as "instrumental" in influencing decisions of policymakers-and thus creating more effective policies (Weiss, 1977). James Rogers in The Impact of Policy Analysis states the goal of policy research "is to reduce ignorance and uncertainty confronting key officials" and that it attempts to employ scientific methods to detennine clear consequences of policy choices, thus making clear choices for policymakers on better public policies (Rogers, 1988. p.3). This description is indicative of the intention of the field of policy research to be rational, uncomplicated, and used in an instrumental way for policy decisions by policymakers. 2

PAGE 20

As such, the research on the research of "use" tries to determine how policymakers use policy analysis and research in this instrumental way in the policy process (i.e., having a direct impact on policy decisions). Recent studies suggest there is not much direct use of research by policymakers (Hird, 2005; Shulock, 1999; Kirp, 1992; Weiss, 1977; Webber, 1986; Bardach, 1984). As a result of this focus on instrumental use of policy research by policymakers, I believe, and much literature suggests, at least four different outcomes: (1) policy researchers believe that many policy decisions are made and implemented in the absence of research; (2) the purpose and processes of social research have not changed or adapted in ways that might increase its integration into the larger policy process; (3) policy research is targeted mainly at policymakers even though they might not be the most important "users" and; ( 4) "policy analysis," or policy research as a career or profession is in crisis about its relevance as a decision-makers' aid to effective policyrnaking. This study seeks to bring greater clarity to these issues by understanding the use of research by other actors-specifically, policyrnakers' staff and midto high-level administrators in state government and how they deliver policy research and information in the policy process. Rationale and Importance of the Research Both the conceptual and empirical policy research literature focus attention on the use of research by policymakers for direct/instrumental purposes, namely, 3

PAGE 21

improving the decisions ofpolicymakers (Bardach, 1984; Weimer, 2002; Weiss, 1977; Rich, 1977; Hird, 2005). Yet, studies indicate that policymakers do not use research very much (Hird, 2005; Kirp, 1992 and 2004; Lindblom & Cohen, 1979). This is a problem for an entire field (public policy analysis and other social science research) that is dedicated to providing research to policymakers. One explanation for these findings might be because policymakers primarily tend to have political goals (i.e., getting reelected) (Stone, 1997), while policy research more likely serves the needs of policy staffers and program administrators who hold policy goals (creating policies that solve social problems). Therefore, research use and theories on research use in the policy process need to be broadened beyond their current focus onpolicymakers and extended to other pertinent actors (and the goals that those actors hold) in the policy process: policy staffers and program administrators. It is also worth noting that these actors (not just policymakers) contribute substantially to the policy decision-making process (Sabatier, 1999; Lindblom & Cohen, 1979; 1988; Kingdon, 1995, Baumgartner & Jones, 1993; Weimer & Vining, 1999; Bardach, 2000; Birkland, 2001; Dunn, 1994 ). Before moving into specific information about my research questions and the literature on these topics, some operational definitions about the terms I use throughout the study are important to note. 4

PAGE 22

Operational Definitions Policymakers: For the purposes of this research, policymakers are considered to be elected or appointed officials who either vote on, or have the authority to make executive decisions that create state or local laws and regulations. Policvmaker Staffers: For the purposes of this research, staffers are considered employees of either policymakers (their advisors) or employees of the state legislative branch employed to support state policymakers in their mission to create legislation and laws. Midto High-level Administrators: For the purposes of this research, midto high-level administrators are considered to be managers who have some decision making authority and who are policy implementers in state-level bureaucracy (both politically appointed and civil service). These workers have responsibility for making decisions about implementation of policies signed by the governor or passed by the state legislature. In my study, this group consists of midto high-level state employees of state departments of education. Policy Research: Lindblom and Cohen ( 1979) define social research in its broadest sense: anything that might contribute to the knowledge of or solutions to social problems. For the purposes of this research, policy "research" or "analysis" is defined as: information and/or data that is gathered, analyzed, organized and communicated to help solve public problems. 5

PAGE 23

Policy Research Use: For my research, Karen Kirkhart's definition of use or influence will be applied: research "use" has occurred when it has influenced, shaped, affected or changed persons or systems (Kirkhart, 2000). Research Purpose The purpose of this research is to study policymaker staffers and midto high-level administrators to determine how they use policy research in their work. This population has not been the focus of study on the use of policy research and could contribute to the knowledge of how policy research is disseminated and used by multiple actors within multiple stages of the policy process. Better understanding of the use of policy research by policy staffers and program administrators is important because it acknowledges and builds on research that suggests administrators have a good deal of discretion and interpretive power, both when they assist with and implement policy decisions (Pressman & Wildavsky, 1984; Sabatier, 1999). Therefore, by focusing on the use of research by staffers and administrators, one learns of another viable way that policy research contributes to the policymaking process (Pressman & Wildavsky, 1984; Palumbo & Calista, 1987; Terpstra & Havinga, 2001; Sabatier, 1999). My study is also intended to further the knowledge base on types of policy research uses and the influence and relation of those uses to policy goals and political goals (Weiss, 1977; Rich, 1977; Shulock, 1999; Stone, 1997). My research 6

PAGE 24

will begin to investigate those connections and help expand the definitions so that policy analysis is considered as a possible tool in the process on policy learning and the policy process as a whole-not just as a decision-makers' aid. I set out to answer the following research questions: 1. To what extent do policymaker staffs and mid-to high-level administrators use policy research? 2. Considering the six types of policy research use expressed in this proposal, what types are most common? 3. When policy research is used, what is the stage of the policy process and the type of research use? 4. How do staffers and administrators get access to policy research? From where? 5. How does the use of research differ between policymaker staffers and midto high-level administrators? In summary, this dissertation attempts to understand policy research use among state-level policymaker staffers and midto high-level administrators so that the role of policy research is better understood in the policy process, thus increasing its effectiveness. With greater knowledge of this process, schools of public affairs, administration and policy can give clear guidance and expectations to both policy researchers in training and those already in the field. The following research study is organized into five chapters. Chapter One provides an overview and rationale for conducting the research. It also introduces the purpose and operational definitions for the study population. Chapter Two 7

PAGE 25

reviews the pertinent literature covering: the foundations of policy research, the purpose and definition of policy research, methods and types of research in the policy sciences, and how policy research "use" is currently understood by policymakers. Chapter Two also addresses elements of the different types of goals in the policy process and how those goals might influence policy research use by the studied population. This chapter also touches on the policy process as it is currently defined in the literature to provide a theoretical basis for the research questions posed in this study. Lastly, Chapter Two will determine what is currently known about how research is used by the population at the core of this study policy staffers, administrators and policy influencers. Chapter lbree presents the research questions, an exploratory theory, preliminary hypotheses and methodology for this study. It discusses the data collection methods, the sample, the analysis techniques and the method limitations. Chapter Four highlights the results of my research and discusses the analyses of those results. Chapter Five presents conclusions from the results and how those conclusions inform the exploratory theory. It also discusses the implications of the results for future study on research use in the policy process. 8

PAGE 26

CHAPTER TWO REVIEW OF THE LITERA TIJRE The review of pertinent literature related to my research questions will cover several areas of study within the policy sciences. First, I briefly discuss the foundations of the field of policy sciences to establish a common understanding for the purposes of this research. In this section, the definition of policy research will be highlighted. Next, I highlight the existing literature regarding how research has been used in the policy process. This will set up the purpose and rationale for my research -why it is needed. Based on what was learned from the previous sections, I will then explain how policymakers use policy research, which is the population of users that has been studied most. This section highlights the various types of research use presented in my exploratory model. After learning about what the literature has to say on research use, I will begin to explore the literature on the policy process. This will build the argument that we must broaden our definition of research use to other actors in the process not just policymakers. In this part I review the policy process, the difference between policy goals and political goals and what is currently known about research use by the population studied staffers and administrators. 9

PAGE 27

Foundations of Policy Research: The Policy Sciences Policy analysis and research as a professional academic field had its beginnings in the conceptualization of the "policy sciences" a term and academic field coined by David Lerner and Harold Lasswell in 1951 (Lerner & Lasswell, 1951 ). These "fathers" of the policy sciences described policy analysis and research as: providing relevant information that is "integrative and interdisciplinary," "anticipatory," "decision-oriented," "value-conscious" and "client-oriented" (Lerner & Lasswell, 1951, 1988). These principles have held up well through the test of the last half-century because policy researchers today still refer to these very goals of policy research (Dunn, 1994; Weimer & Vining, 1999; Birkland, 2001; Schneider & Ingram, 1997). Overall, the emergence of the field of policy sciences (as opposed to political science, which studies the political process) is based on the belief that social problems could be systematically studied and analyzed (a rational approach) and thus policies that were meant to impact them could be improved upon and made more effective. Purpose and Definition of Policy Research Policy research today is many things to many people. The numerous types and styles of policy research, analysis and information include, among others: policy analysis, evaluation, social research and economic analysis. And, of course, there are different purposes and mechanisms for each type of research as well. 10

PAGE 28

In defining policy analysis or policy research one can infer from the overall purpose of research and analysis an attempt to improve social policy. According to Dunn, building on Lasswell, policy research is meant to create knowledge "of and in" the policy process (Dunn, 1994). Dunn goes on to say that in this process, social researchers investigate the "causes, consequences and performance of public policies and programs (1994, p.l)." Charles Lindblom and David Cohen suggest policy research should similarly be defined very broadly as "professional social inquiry" and should include many types of activities, professions and processes, including: theoretical work by researchers and theorists, academic social science, the development of practical and practice-oriented theories, systematic data-gathering and reporting, systematic professional speculative thought, social commentary and criticism, policy analysis, operations research, evaluation, and consulting services by professionals (Lindblom & Cohen, 1979). They define social research in its broadest sense as anything that might contribute to the knowledge of and in social problem solving. Heienman, Bluhm, Peterson and Kearny take a more narrow approach, saying that the purpose of policy analysis is to improve government decisions by applying data that can be used to increase the general public good (2002). Weimer and Vining, authors of a leading textbook on policy analysis, also restrict the definition more than Lindblom and Cohen ( 1979). They write that the goal of policy research is to "craft policy options" for decision-makers and to do this, researchers must gather, organize and 11

PAGE 29

communicate information, put social problems in context and understand political and organizational behavior (Weimer & Vining, 1999). While all of these definitions discuss improving decisions, none of them explicitly say whose decisions research is supposed to improve. This is important because my study contends that policy research is used to inform and improve decisions even though it might mostly inform other actors in the policy process rather than just policymakers. While these definitions of policy research and analysis differ slightly, they are all based on the assumption that research and analysis can contribute to increasing the public good by analytically studying policies, learning about causes and consequences, thus leading to better decision-making (Heienman, 2002; Weimer & Vining, 1999; Bardach, 2000; Webber, 1986). For the purposes of my research, policy ''research" or "analysis" is defined as: information that is gathered, analyzed, organized and communicated to help solve public problems. Methods of the Policy Sciences The traditional methods of policy research are based on the rational model of inquiry and decision-making noted in the definition above (Ukeles, 1977; Stone, 1997; Schneider & Ingram, 1997; Lindblom & Cohen, 1979; Kirkhart, 2000; Ginsberg & Rhett, 2003; Grob, 2003; Bernard, 2000; Kumar, 1996; O'Sullivan & Rassel, 1999). In this model, a professional in the policy sciences field would provide specific and detailed information to a policymaker prior to making a policy 12

PAGE 30

decision. The policymaker would then gather all the pertinent information, methodically evaluate it, and then come to a conclusion based on what maximizes either his/her benefit, the benefits of those the decision is affecting, and the public good. It is this type of rational policy decision structure that was intended for the policy sciences. One can see this illustrated in early descriptions of the policy process that imply a linear step-by-step policy process that allows for in-depth research of this type (Lerner & Lasswell, 1951; de Leon, 1997). This rational policy process method included: identifying the problem, identifying the options for action, predicting the possible outcomes of the options and selecting the option that provides the most value to the most people (Weimer & Vining, 1999; deLeon, 1997; Dunn, 1994; Bardach, 2000). But as many acknowledge, the rational policy research process and the irrational way that policy gets made (messy, iterative, and full of multiple actors with multiple agendas) do not "fit together" and make for a tight connection or path of research use (Sabatier, 1999; Booth, 1990; Hird, 2005). And even though many have started to question the feasibility and logic of using this "rational" research model in an irrational system, the rational model is still very much alive today (Stone, 1997; Bobrow & Dryzek, 1987; Ginsberg & Rhett, 2003). As the goal of policy research is to make "more rational" the messy process of policymaking (Weiss, 1977; Rogers, 1988; Lindblom & Cohen, 1979), analysts can help policymakers make more informed "scientific" decisions about public policy choices by researching social 13

PAGE 31

problems and policy solutions using hard data (Grob, 2003; Ginsberg & Rhett, 2003; Weiss. 1977; Rogers, 1988; Lindblom & Cohen, 1979). For example, Ginsberg and Rhett say there is still an "unprecedented demand for scientifically sound" studies that contribute to policy decision-making (2003 p. 489). A specific example of this is in the education policy sector, where the United States Department of Education requires that all state-directed K-12 programs that use federal education funds be scientifically "research-based" using the traditional criteria of the rational, research methods (Education Commission of the States, 200 I). Policy Research Use "Use" Defined So now that we recognize there is still a demand for a rational research and policy analysis model, what do we know about how it is used? To understand how policy research is used within the policyrnaking process, the term "use" must be defined. In most of the literature reviewed for my project, researchers apply the general term "use" in its most basic form, meaning, "to put into service or to employ" (American Heritage Dictionary, 1994). However, for the breadth of understanding that is needed about how, why, what types and at what points research is integrated into the policy process, a more inclusive, encompassing definition is needed. Along that broader definitional line, Karen Kirkhart argues, in terms of evaluation use, it is necessary to find a term that embodies "how and to what extent 14

PAGE 32

evaluation shapes, affects, supports and changes persons and systems" (in the case of my research, policymakers and their staffers or administrators) (Kirkhart, 2000. p. 7). She argues for the term "influence" as opposed to "use" to capture the previous issues. For my research, Karen Kirkhart's definition will be applied: research "use" has occurred when it has influenced, shaped, affected or changed persons or systems (Kirkhart, 2000). This provides a definitional starting point and basis from which to examine how policy research is integrated into the policy and decision-making process. As already stated, the literature and research on this topic thus far have focused mostly on how policymakers (as opposed to staffers or administrators) have been influenced by research. Therefore, the majority of the following section will review those findings while also addressing the gaps that currently exist in the research. Policy Research Used by Policymakers Interest and concerns about how policy research (such as policy analysis, social science research, or program evaluation) is used to inform policymaking has been occurring since the study of the policy sciences began (Lindblom & Cohen, 1979; Birkland, 2001 ). And, even though many cases have been found where research does inform policymakers, it is still an issue of concern for today's public affairs student and professional (Hird, 2005). Indeed, this concern for understanding 15

PAGE 33

research use is the subject of entire academic journals. For example, one titled Evidence & Policy has articles such as "Bridging research and policy: insight from 50 case studies" as well as "Affecting policy and practice: issues involved in developing an argument catalogue" (an "argument catalogue" is a compiled set of research and policy evidence on a topic of interest) (November 2006). Another example is highlighted in the journal New Directions in Program Evaluation, which has dedicated an entire issue to the question of what makes a policy "useful" evaluation (Sridharan, 2003). In this issue, Christina Segerholm argues that it is important (and long overdue) to start examining research and evaluation as a phenomenon to determine its effectiveness as a policy-informing tool (Segerholm, 2003). Others have also stated that this same type of examination needs to be done for all methods of research that aim to provide decision-makers with useful, pertinent infonnation to make better social policy (Hird, 2005; Lindblom & Cohen, 1979). Instrumental Use of Policy Research Because the "traditional" intention of policy research is to affect policy decisions, most of the research is directed toward assessing that with regard to policymakers, as they have the power to make important decisions in the policy system (Shulock, 1999; Hird, 2005; Kirp, 1992). And it has been found that instrumental research use does happen, albeit less frequently than policy researchers might like. Instrumental use of research "is when a study is used as the direct basis 16

PAGE 34

tor a decision" (Weiss, Murphy-Graham & Birkeland, 2005, p. 13). One of the most widely cited studies on policy research use comes from Carol Weiss and Michael Bucuvalas in the late 1970's (Weiss & Bucuvalas, 1977). They conducted survey and interview research to determine the forms of effective social science research that were useful for decision-making (i.e., trying to determine the extent of instrumental use of research) (Weiss & Bucuvalas, 1977). These two researchers found that instrumental use of policy research was not the only (or even most common) way that social science research was used, even though this is how policy researchers intended it to be used. Instead, they found research typically was used in conceptual ways including defining social problems, looking at issues from other perspectives and understanding problems more completely (Weiss & Bucuvalas, 1977; Weiss et al., 2005). Mostly, researchers find that "pure instrumental use (of research] is not common (Weiss et al., 2005 p. 13; Hird, 2005; Black, 2001; Shulock, 1999; Albaek, 1995; Webber, 1986; Caplan, 1975). In another key analysis Weiss and Bucuvalas suggest that research use depends on how much the research results reinforce the decision-maker's currently held beliefs. When a particular piece of research is seen as supporting a set of values and beliefs that a decision-maker holds it will reinforce a policymaker's inclination to use it. The finding shows when research contradicts a belief that is held, then the policymaker will not only be less likely to use it but will also criticize the credibility 17

PAGE 35

of the research by pointing out flaws in the methodology behind the research (Weiss & Bucuvalas, 1980). More recently, Nancy Shulock conducted a study to determine instrumental use of policy analysis by legislators in congressional committees. She determined "use" by counting the number of times research studies were cited in final committee reports (Shulock, 1999). Her article"The paradox of policy analysis: if it is not used, why do we produce so much of it?" -offers a theory to resolve the paradox. Shulock suggests there is plenty of use of policy analysis, but not in the traditional, direct, and instrumental way that the "science" of traditional policy analysis might hope. She found usage of research in three ways: (1) to frame political discussions, (2) as a rationalization for legislative action, and (3) as a symbol of legitimate decision-making processes in government. Overall she found that the more public attention a policy issue received, and the more competition among legislative committees to govern on the same topic, the more research studies were cited in the committee hearings and reports. This may suggest, as Shulock notes, that research is used more when policymakers have to explain their position to other colleagues or the public. Shulock suggests a "fundamental redefinition of policy analysis and its use in a legislative environment. .. [because] despite its scientific origins, policy analysis may be a more effective instrument of the democratic process than of the problem solving process" (Shulock, 1999, p. 226). Similar to Weiss and Bucuvalas, Shulock also found that, more often, research was used to define policy problems or 18

PAGE 36

to defend a policymaker's position on the solution to the problem, especially when there was more public scrutiny or jurisdictional competition between committees (Shulock, 1999). However, other researchers make a point to illustrate that instrumental use of research does still occur (Ginsberg& Rhett, 2003; Fitzpatrick, 2004). Ginsberg and Rhett suggest that "a useful evaluation of an educational program is one that adds to the body of timely, relevant evidence to increase the likelihood that policy decisions improve program performance" (2003, p. 489). Their research cites examples of instrumental use of research from evaluations in the field of federal education policy (2003). They found a "relevant, timely, body of scientific evidence has been influential in affecting legislation, budget and administrative decisions" related to national education policies (Ginsberg & Rhett, 2003, p. 490). They also found evaluation research did make a direct contribution to decisions that supported budget reductions for the Even Start Program as well as a mandatory requirement for Title I schools to use phonics in literacy programs in the Reading First Initiative (Ginsberg & Rhett, 2003). Use of Policy Research to "Enlighten" Though Weiss and Bucuvalas, Shulock, and Ginsberg and Rhett all intended to measure instrumental impact of policy research, more often they found other uses besides direct/instrumental were evident. Weiss and Bucuvalas described these types 19

PAGE 37

of usage as having an "enlightenment" influence on policy actors. "Enlightenment punctures old myths, offers new perspectives, and changes the priority of issues" (Weiss, 2003). Shulock refers to the same phenomenon as the "interpretive" view of policy analysis use ( 1999). Enlightenment refers to when people learn something new about the policy problem, solution or policy and as a result they re conceptualize how they perceive the problem, and thus how they might perceive the solution. The ideas that make up the meaning of Weiss and Bucuvalas' 1977 "enlightenment" concept continue to influence the literature. For example, both Shulock and Ginsberg and Rhett have acknowledged the effects of enlightenment when they discuss the use of research to provide new information, define a problem or help formulate an argument. Even before these researchers conducted their analyses, Robert Merton wrote about the notion of enlightenment when he discussed how part of a policy analyst's job is to help policymakers find new ways to look at policy problems and thus new possibilities of resolving them (Merton, 1968). "Enlightenment" is a process of gaining new information and concepts about a social problem and how to solve it. Tim Booth ( 1990) also discussed this process of enlightenment when he highlighted the difficulties of research being influential. He suggested that because policymaking is not a one-time event that can be easily separated from other events, and is instead an "evolutionary process[,] ... research creeps into policy use in diffuse ways." (Booth, 1990, p. 80-81 ). In his discussion of the evolutionary policy process 20

PAGE 38

he recognizes the time it takes for new information to "enlighten" actors in the policy process and new concepts and solutions to emerge as a result. Other Types of Research Use Only recently has research validated other types of research use, such as enlightenment, with respect to how policymakers incorporate research knowledge into their policymaking duties. Shulock argues that, for a long time, these other types of research use were not considered important-only "traditional" types of research use were considered worthy (such as instrumental use) (1999). However, the empirical research base now acknowledges policy research has an impact on policymaking through utilization in multiple ways (Hird, 2005; Henry, 2003; Shulock, 1999; Stone, 1997). Now researchers are showing that these other types of research use are influential and important in the policy process. Additional types of research use that surfaced in the late 1970's indicate policy research is more often used during the problem formation stage, as a context setter, or in conceptual learning as opposed to at the decision-making or policy selection stage (Hird, 2005; Weiss, 1977; Lindblom & Cohen, 1979; Shulock, 1999; Rich, 1977). The research originally conducted by Weiss and Bucuvalas has been further developed by other researchers to include at least six categories of research influence: (I) agenda setting (as a way to place a topic on a policy "to-do" list); (2) enlightenment (as a source of information to understand problems, as a way to get 21

PAGE 39

new ideas for solutions, as a way to change attitudes and behaviors); (3) instrumental (research findings being directly used in policy formulation); (4) rationalization (''to justify previously held positions" (Hird, 2005, p. 88); (5) tactical (as a method "in providing analysis to achieve tactical goals, such as delaying policy decisions" (Hird, 2005, p. 88); and, (6) empowering change agents (as a way to influence policy actors to focus and create activism around a topic (Stone, 2002; Hird, 2005; Shulock, 1999; Davis, 1986; Kirkhart, 2000; Henry, 2003; Caplan, 1976). These six categories are presented in Table 2.1. T bl 2 1 T a e .. .ypes o fR esearc hU se Agenda Setting Placing the topic on the policy "to do" list Enlightenment Understanding problems in new waysconceptual Gaining new information on a problem Instrumental Influence on actual policy decisions Rationalization Defending an argwnent Tactical Used in debates and as a strategy to determine or delay certain policy options. As a symbol of legitimate decision-making. Change Agents Creating activism both politically and publicly around a topic These six categories form the existing literature on how research is used by policymakers for policy decisions and creation. Nancy Shulock's research is one of the more recent pieces to validate this more "interpretist" view of policy research use. As previously noted in her study of how Congressional committees utilize 22

PAGE 40

research based on the "publicness" and the committee jurisdiction of the topic, Shulock suggests that policy analysis is used in three ways: (I) to frame political discourse (see "enlightenment" in the table above), (2) as a symbol of legitimate decision-making (see '"tactical" in the table above), and (3) as a rationalization for legislative action (see "rationalization" in the table above) (Shulock, 1999). While Shulock's recent research gives more credence to three of the categories of research use above, all six categories serve as the basis for my exploratory model and research on how policymaker staff and midto high-level administrators use policy research. Through my data collection I have learned how these categories align with research usage patterns of staff and administrators as well as how they fit in with the policy process. To these six categories of research use the program evaluation literature has added a model of how these types of"uses" occur, specifically with program evaluation research (Kirkhart, 2000). Karen Kirkhart's model, called the "integrated theory of influence," adds an element to the six categories explained above (Kirkhart, 2000). This model looks at the influence (as opposed to "use") of evaluation research on three dimensions: intention, source of influence, and time. Within each category are several other sub-categories of how research "influences," including: process-based influence, intended and unintended influence, and short term and long-term influence. The model is inclusive of the multiple ways in which research enters and is influential in the policy process. 23

PAGE 41

Process-based influence is when the evaluation research may have impacts due to the process of conducting the research itself. These impacts might be on a program, organization or policy. For example, questions an interviewer may ask of a program staffer may affect that staffer's view of his/her job and either motivate him/her to do better or make him/her feel worse about the conduct of his/her job. This might translate into behavior that affects the program performance. There are three types of process-based influence: cognitive, affective and political. Cognitive is when the research process enhances the understanding of the program/policy among participants in the process. The affective dimension is the way the process of conducting the research affects the personal feelings and ideas of the participants themselves (how they feel about the worth and value of the research process). The political dimension relates to how the research process itself creates new dialogs, brings attention to new problems or influences the dynamics of power around the people in the program/policy (Kirkhart, 2000). Results-based influence is akin to instrumental use of research and occurs when there are data-based conclusions drawn from the research or there is a direct, visible action taken based on the use of research findings. Kirkart's "results-based influence" is a broader definition than has been defined in this research because she also includes research used in advocacy, argument or political debate as a type of results-based influence (Kirkhart, 2000). 24

PAGE 42

Within both the process and results types of influence, other dimensions come into play such as the intention of use of the research and the part ofthe policy process in which it was used (immediate, end-of-cycle, or long-term). A complete look at Kirkhart's theory is illustrated in the graphic below. Kirkhart's theory adds an important aspect to consider when studying research use in the policy process: what is the intent of the research use and how does it affect both the policies and people who are involved in the process? Intended lntentioa Unintended Long tenn Immediate Figure 1: Kirkhart's Theory of Evaluation Usage Research Use Affected by the Individual Nathan Caplan (1975) also provided a different but relevant framework for understanding how the background of policymakers affects the type of research they are likely to use. He found that if the policymaker approaches a problem from a clinical perspective (i.e., if his/her profession is practice and clinically based), he/she 25

PAGE 43

is more likely to use scientifically based research and apply unbiased information to a problem to make decisions (as opposed to policymakers who come from an academic or advocacy position). If the policymaker comes from an academic context or education background then he/she is more likely to apply theory and grand solutions without thinking too much about the feasibility of such solutions. Lastly, if a policymaker comes at policy decisions from an advocacy position, then he/she is likely to use information to support already-formed conclusions about a problem (Caplan, 1975). This knowledge adds to the research use literature the perspective of individual differences among people using research, and suggests that an actor's background is important for understanding research use in the policy process. How is Research Accessed When It is Used? In addition to individual backgrounds and attributes affecting if, how, and why policy research is used, the access to research is also a factor with respect to the types of research most often tapped and integrated into the policy process. For example, the prevalence of technology via the internet, email, research databases and online research journals has influenced access to research information. One might guess that these technologies provide access to a larger number of actors in the policy process compared to when most research on use was conducted in the 1970's (Backer, Salasin & Rich, 1991). Interestingly, this new element is not discussed very much in the literature with regard to how it has altered or affected research 26

PAGE 44

infonnation use with various actors. Policymaker staffers and midto high-level administrators likely now use technology in sophisticated ways because of their access to computers and their technologically upgraded work environments. This important contextual element of technology will affect how, why and when research is integrated into the policy process. Policy Goals and Political Goals and Their Effect on Research Use As the different types of research use and influence are found, one can imagine that another variable the goals of the person using it might also influence the way research is used. Actors in the policy process hold different goals and thus will likely have different patterns of research use to reach those goals. Two of the main types of goals in play are policy goals and political goals (Stone, 1997). A policy goal has the objective of finding a technical policy solution to a public problem. When this goal is held, policymakers are trying to achieve the goal by selecting a policy that solves the problem and maximizes benefits to the most people (Stone, 1997). In this case, the use of research might be more prevalent because often research reports recommend possible policy solutions or suggest findings that articulate whether a certain policy is feasible or based on specific research evidence. Political goals are the goals to which policymakers pay attention to order to satisfy their constituency and get reelected. In this set of goals, the policymaker wants to maximize the happiness of his/her constituency sometimes over and above the 27

PAGE 45

benefit of other constituencies (Stone, 1997; Schneider & Ingram, 1997). In the case of political goals, the "best" policy solution may not be chosen or even considered at all. Therefore, the use of research in this case is largely conceptual either as a rationale, political cover or to clarify a policy problem. The difficulty in researching the impact of policy goals and political goals is that they are not mutually exclusive, so it is difficult to measure or quantify the effects these different goals have on research use. Even though policymakers can hold both goals simultaneously, they will choose (consciously or unconsciously) to maximize one set of goals over the other in any given circumstance (Stone, 1999). Deborah Stone investigates this exact problem of simultaneously having policy goals and political goals in her book, The Policy Paradox: The Art of Political Decision-Making (1997). She concluded that policy analysis and research techniques were only geared to policy goals and those conceptualized in the "rational model" at that. And, she contends, until political goals are considered and integrated into the research and delivery process, policy research will continue to be a questionable use of time, energy and money (Stone, 1997). Another way to look at the issue of policy goals and political goals is to consider the actors in the process who are more apt to have either one or the other type of goals. For instance, policymaker staffers and government administrators probably identify more with having policy goals as opposed to their "bosses" politicians who have political goals. This is so because in performing their jobs 28

PAGE 46

staffers and administrators need to be somewhat predictive, analytical and able to determine costs of policy solutions as well as the feasibility of the implementation of the policy solution. As such, policy goals may be more influenced by social research, thus suggesting a better entry for research into the policy process might be through staffers. For example, Daniel Dreyfus commented in the 1970's on the important influence of staffers by saying that often "congressional policy will be established by informal comments [from staffers] made during committee hearings, [and] by staff contacts with agency officials" (Dreyfus, 1977 in Weiss, p. 101 ). He goes on to say that in this respect, Congress basically ratifies the policy proposals of others, namely, staffers and agency officials. Research helps both of these audiences though possibly for different reasons. Research is often used by policymakers as a rationale for political decisions while it may be used by staffers and administrators to help advise policymakers, design policy options and implement the policies. However, as previously noted, there is not a direct dichotomy between policy and political goals and the actors in the process who hold them. Policymakers also hold policy goals and quite possibly use research to further them. They often hold both goals at the same time but have to choose (whether consciously or unconsciously) which goals to prioritize at any given decision point. Thus far, in the field of the policy science, it has been through the lens of policy goals that policy research is often conceptualized, defined, intended and assumed to be used (Stone, 1997). In this realm, policy researchers hope their work 29

PAGE 47

is extracted from the messy political process and can provide policymakers with relevant, research-based information from which they can choose the best policy option to make decisions and meet policy goals (Beckman, 1977; Hahn, 1973 ). And while this may be the optimal world for the policy researcher, as noted in the previous pages, it is certainly not the real one. In the real world, policymakers are pressured by constituents or lobbyists to make certain decisions, or they are constrained by budgets, or controlled by rules and laws outside of the current decision, or forced to compromise with other sets of constituents who do not share their view, as well as being beholden to other political power structures (Heineman, 2002; Stone, 1997; Schneider & Ingram, 1997; Dreyfus, 1977). Given all these contextual conditions, even if policymakers wanted to choose the "right" policy option based on the research knowledge to solve a problem, they often do not because of political goals as well as the political system as a whole. The Real Policy Process Given the multiple policy actors who each may hold dual goals, one can see why policy research is often times discussed in absence ofthe context of the full policy process (Heineman et al., 2002; Lindblom & Cohen; Stone, 1997; Rogers, 1981; Schneider & Ingram, 1997). Very often research is discussed in a way that makes practitioners think it is conducted separately from the political process to inform the beginning of the policy process (problem definition) and the selection part 30

PAGE 48

of the process (where a policy alternative is chosen) (Hird, 2005; Stone, 1997). What is not usually discussed is how policy research fits into the messy, non-linear way that policy really gets made. Several policy process theorists such as Sabatier, Kingdon, Baumgartner and Jones, Schneider and Ingraham have all noted the distinct absence of a linear process in the way public policy is made in the United States (Sabatier, 1999). Kingdon notes in his multiple streams theory that the right events, actors and timing all have to converge at the same time to make new policies (Kingdon, 1984 ). Similarly, Sabatier and Jenkins-Smith's theory called the Advocacy Coalition Framework discusses multiple subsystems, actors, strategies and external events that all Ct'ntribute to a policy being made (Sabatier & Jenkins-Smith, 1999). Baumgartner and Jones talk about the need for a "focusing event" that "punctuates" the rational system and thus creates a new construction or concept of the social problem attempting to be solved through policy (Baumgartner & Jones, 1993). Each of these theories of the policy process discusses being "in the right place at the right time" with the right policy solution. There is less discussion about how comprehensive policy research ensures the correct selection of a policy to solve a stated problem. The barriers of bounded rationality, complex decision-making environments and events beyond any one person's control lead us to realize that information gets infused into the policy process in many ways and by many actors. 31

PAGE 49

Charles Lindblom and David Cohen, in Usable Knowledge: Social Science and Problem Solving, argue that public policy researchers have focused only on parts of the policy process rather than looking at the whole context of social problem solving, including necessary citizen participation. They argue researchers need to study the full policy process and decision structures of our political system as a whole in its relation to social problem solving to make policy research more useful and used by actors within the policy and political process (Lindblom & Cohen, 1979). Part of studying this full process and decision-making structure is to take into consideration the audience for research usage (including the general citizenry, staffers, administrators and other actors in the policy process) and the roles those audience members might play in the policy system. Roles Staffers and Administrators Play in the Policymaking Process Each of the above-noted theories of the policy process acknowledges the multiple actors and the importance of their roles in the creation of public policy in the United States. Kingdon, in his "streams" model, for example, discusses the role of "policy entrepreneurs" as the actors in the policy process who can take advantage of an opportunity (a policy window) to create new policies (Zahariadis, 1999). These entrepreneurs are individuals with the knowledge, expertise and issue focus to promote certain policy ideas (Zahariadis, 1999; Schneider, Teske & Mintron, 1995). These actors play the important role of bringing together policy problems, solutions, 32

PAGE 50

and information to politicians who might be willing to take action on the ideas. Similarly, Sabatier and Jenkins-Smith note the role of "subsystem actors" and "policy brokers" in their Advocacy Coalition Framework (ACF) (Sabatier & Jenkins-Smith, 1999). In the ACF policy process model, the policy subsystem is actually the most important part of the policymak.ing process, where actors from multiple public and private organizations influence policies in their specific topic domain (Sabatier & Jenkins-Smith, 1999). In this subsystem, a nwnber of different role groups are evident: journalists, researchers, policy analysts and all government actors throughout the system who have any role in policy development or implementation (Sabatier & Jenkins-Smith, 1999). These actors play many roles in the policy process, such as providing the needed information on a topic, creating the public pressure for action on a policy issue and "brokering" the necessary relationships, information and resources to create policies. Again, the importance of other actors is noted in the punctuated equilibriwn model, such as staffers and government administrators who help to define public problems, survey the data related to problems and then convey that information during the stages of problem definition and policy implementation (Baumgartner & Jones; 1993). Given these important roles that other actors play in the policy process, policy researchers need to attend to the different audiences and types of policy research use, as well as how and when policy research might serve different goals of the policy process. All of these theories highlight that the most important users of 33

PAGE 51

research might be other actors in the process such as staffers, administrators and other policy influencers or entrepreneurs. By understanding the most opportune times and ways that research is utilized in the policy process, the policy researcher can improve the type of research as well as the integration of it at the most useful times, with the most useful actors of the process. For example, in The World of the Policy Analyst, Heineman et al. (2002) examine policy analysts as actors in the policy process as opposed to external agents providing information to inform the process. The authors discuss factors that influence policy researchers in terms of how they get their information into the policy process, suggesting that researchers and analysts who understand the political process and strategize the best way to get information and research into the system have better success at affecting policy decisions than others (Heineman et. al., 2002). They also maintain that researchers who build relations with the pertinent "actors" in the process have more access and influence than researchers who remain separate from these individuals (Heineman, et al., 2002; Grob, 2003). This argument supports the premise that if theories of the policy process such as the Advocacy Coalition Framework, the streams model and punctuated equilibrium were coupled and aligned with the policy research process and the actors who are users of research, the impact and use of policy research might be much greater. 34

PAGE 52

Policy Research Use by Staffers and Administrators Robert Rich, based on his research in 1977, created a theoretical framework related to the use of research by these other actors in the policy process in which he posits that instrumental use of policy analysis flows upward from administrators and staffers to policymakers, and that conceptual use of analysis flows downward from policymakers to administrators and staffers (Rich, 1977). He also found two "waves" ofutilization: the first consists of short-term use of research; the second wave is research use over the long term. Within these two waves of utilization, "intended" uses of research (typically described as instrumental) usually take place in the first wave for about three months the time it takes for the information to flow through the organization hierarchy. In this first wave, the information flows upward through the decision-making hierarchy; the use of research is primarily instrumental and action-oriented. The second wave of utilization is from three to six months after the information is initially received (Rich, 1977). This flow of information is downward from policymakers to others and is usually conceptual. For example, policymakers use research information to tailor their argument for colleagues and their constituents. They use it to define and refine their message about a particular policy problem to both the public and the staffers and administrators who work for them (Rich, 1977). These findings lend support to the notion that policy staffers and administrators utilize research in the more traditional, intended use of policy research for instrumental purposes. While this research is interesting and starts to identify 35

PAGE 53

how policy staffers and agency administrators might use research, many questions are left unanswered. Nevertheless, these findings on policy research use by other actors in the policy process led me to investigate the policy implementation literature, which may shed light on how staffers and administrators utilize research in their jobs to implement policy decisions. Upon examination of the literature through the policy implementation lens, I found that it focuses on policy outcomes and the process of implementation not on how information or policy analysis contributes to administrators' judgments when implementing or interpreting policy. Therefore, to fill this knowledge gap, and to build on the elements of research highlighted in this literature review, in my thesis I examine the use of policy research among policyma.ker staffers and midto high-level administrators (often the designers and implementers of policy) to reach a better understanding of the role of policy research in the policy process. My research questions and exploratory model build on the past research on how policyma.kers use research in their decision making (less for instrumental purposes and more for enlightenment and rationalization purposes) and on how different theories of the policy process articulate the role of other actors in it (other policy influencers are key players in the policy process and include actors such as staffers and administrators). 36

PAGE 54

In the next chapter, I discuss my exploratory theory and hypotheses that build otT of this literature review. In addition, I explain the methods used to conduct this research. 37

PAGE 55

CHAPTER3 RESEARCH QUESTIONS, THEORY AND METHODS This chapter describes the exploratory theory and hypotheses derived from the literature, the methodological considerations and analytical approach, including the study design, rationale, and limitations; the research questions and hypotheses; a description of the population and sample; and the analyses conducted. In addition, Chapter lbree highlights and describes the exploratory theory in which the data are meant to inform in a preliminary way. My methods are meant to accomplish three outcomes: (1) answer the research questions, (2) create an exploratory model, and (3) begin to test the model to see if it is viable. The exploration of the theory was guided by a series of research questions and organizational hypotheses. It occurred through a mixed method research design using a survey instrument that yielded quantitative data and a set of qualitative interviews. Research Questions To gain some knowledge and understanding of how and if my model could be used as a theory of research use in the policymaking process, I intend to answer the following questions: 38

PAGE 56

I. To what extent do policymaker staffs and midto high-level administrators use policy research? 2. Considering the six types of policy research use expressed in this proposal, what types are most common? 3. When policy research is used, what is the stage of the policy process and the type of research use? 4. How do staffers and administrators get access to policy research? From where? 5. How does the use of research differ between policymaker staffers and midto high-level administrators? These questions are meant to explore the use of policy research by the multiple actors who influence both policymakers and the implementation of policy policymaker staffers and midto high-level administrators. The theory and hypotheses below are preliminary and illustrate my ideas regarding aspects of the theory presented. Exploratory Theory Figure 2 illustrates my exploratory theory of research use and the policy process. It was conceived based on past research and literature on the policy process (deLeon, 1988; Sabatier, 1999; Stone, 1997; Birkland, 2001). In the synthesis of other policy theories and theories on use of research, I have integrated a number of dimensions of policy research and actors in the policy process into this one consistent model. The theory was created to have a model in which to ''react based 39

PAGE 57

on the data collected for the research questions. In Chapter Five, I discuss my findings with respect to this preliminary exploratory theory of research use in the policy process by policymaker staffers and midto high-level administrators. The theory is explained in narrative form on the page following Figure 2. 40

PAGE 58

Policy actors interact with each other and with research in the following basic process. Although the process looks linear in this graphic, it is well established that it is not. Within each area, specific research uses of the six highlighted in the literature review are more prevalent (hypotheses in parentheses). Staffers (s) and administrators (a) are more active in certain parts of this process (as noted by the codes within the stages), suggesting their research use is more specific to certain types. (Enlightenment) (Instrumental) Policy Problem Definition (s & a) Information gathered Identified and defined Placed on the agenda Policy Goal (a) Political Goal (s) Analysis Analysis Policy options based on Stakeholders evidence "'----""" Policy options based Public discourse on constituent Mostly prioritized for policy satisfaction influencers Mostly prioritized for policymakers Political/ Democratic Process (s) Stakeholder input Information exchange -Compromise Policy Change/ Adoption -Voting Executive order Ballot initiative Administrative discretion Policy Implementation (a) (Agenda setting) (Enlightenment) (Change Agent) Rationalization) (Tactical) (Rationalization) (Tactical) (Enlightenment) (Change Agent) (Instrumental) (Rationalization) (Instrumental) (Enlightenment) Figure 2: Exploratory Theory on Research Use in the Policy Process by Staffers and Administrators 41

PAGE 59

Narrative for the Policy Process Depicted in Figure 2 At the very beginning of the policy process, a policy problem surfaces, whether from current events, an advocacy group, citizens, a policymaker or government organization (Sabatier, 1999). The problem is identified, defined and placed on the policy and/or political agenda. Within this part of the policy process are both policy goals and political goals associated with the policy problem (Stone, 1997). I hypothesize that policy staffers and administrators are more aligned with the policy goals and policymakers are more aligned with the political goals. Depending on which goals they hold, actors will use information differently to achieve them. I hypothesize that the types of research use in this stage may include agenda setting, enlightenment and change agent. There are multiple policy goals for any given problem (Stone, 1997) that also must be identified and defined. Various types of analysis, stakeholder interaction and events occur to determine different policy goals. Once those are defined, different policy solution options are identified to meet the goals. Research and analysis contribute to this process. Again, policy staffers and administrators typically have policy goals in mind, while policymakers are trying to balance both political and policy goals. Depending on the role they play in the process and their priority goals (i.e., policy or political), each group of actors will use information differently and access it from different sources. I hypothesize that the types of research use to forward these goals might include informational and instrumental. 42

PAGE 60

Just as there are multiple policy goals, there are also multiple political goals for any given policy problem (Stone, 1997). Different stakeholders hold different goals and the job of the policymaker, at least partly, is to engage certain constituencies and stakeholders around various goals. Political goals are related to policy goals but are much more associated with the need to get re-elected and with certain value beliefs than with solving a technical social policy problem. I hypothesize that the types of research use to forward these goals might include tactical and rationalization. After policy and political goals are identified (which is certainly not a linear process), a political and democratic process is undertaken (Sabatier, 1999). Compromise and conflict negotiation occur and power structures come in to play when determining which policy is selected. As in other stages of the policy process, there is public discourse, input and active engagement from citizens. During this stage goals are shifted, compromised and changed to serve the interests of those at the table negotiating the policy solutions, thus possibly using research for tactical, rationalization, informational and change agent reasons. Eventually, after the non-linear process of analysis, goal setting and democratic engagement, a policy is created, changed or adopted. This step occurs through a variety of ways including voting, executive order, citizen ballot or administrative discretion. I hypothesize that this part of the process might reflect a 43

PAGE 61

traditional use of policy research-instrumental or as a rationale to defend the position taken. Once this policy process is complete, the policy must be carried out or implemented. Most policy and political academics agree this is an area about which little is known and which is very hard to study (Palumbo, 1987; Pressman & Wildavsky, 1984, 1999; Sabatier, 1999). At this stage the administrator uses a lot of discretion in implementing policy (Pressman & Wildavsky, 1984). It is often difficult to determine whether policies are implemented in a way that meets the intended policy and political goals. At this stage research may be more apt to be used in an instrumental or informational manner. This exploratory theory forms the framework for a complete research agenda. It helps inform several questions to be answered in this dissertation that will then create the information necessary to test the model in future phases of research. Exploratory Hypotheses The following hypotheses are exploratory and are embedded in the model just described. They serve a secondary purpose to this research-the primary purpose being to answer the research questions. The hypotheses are based on the literature regarding the policy process and how policy research is used in this process. They were developed based on my research questions and are meant to organize my data and information based on my preliminary exploratory theory of research use within 44

PAGE 62

the policy process. The hypotheses were created to gain insight into how the types of research use interact with the policy process and the actors within it as a whole. The hypotheses are also meant to help shape the next phase of the research by providing some preliminary insights into their directional nature. They are not mean to be "accepted" or "rejected" for the purposes of this research. Hypothesis 1: Policy staffers and midto high-level administrators use research in different ways depending on the stage of the policy process, as depicted in the exploratory model (related to research questions one and two). Hypothesis 2: Policy staffers and midto high-level administrators are more likely to use research at the policy problem, policy goal and implementation stages than at the political goal, democratic process or policy change stages (related to research question three). Hypothesis 3: Policy staffers and midto high-level administrators are more likely to use policy research with respect to reaching policy goals as opposed to political goals (related to research question three). Hypothesis 4: Policy staffers and midto high-level administrators will access research via newer technologies (such as the internet) (related to research question four). Hypothesis 5: When comparing groups, policy staffers will be more closely aligned with political goals than midto high-level administrators (related to research question five). 45

PAGE 63

Research Design and Operational Definitions The research design employed in this dissertation is mainly descriptive. The methods used are meant to provide the field of public policy with descriptive information about a population that has not been studied with regard to policy research use. The survey and interview data are meant to investigate my exploratory research use theories. The terms and variables used in the model are defined below. Operational Definitions for Variables in the Exploratory Model Policymakers: For the purposes of this research, policymakers are considered to be elected or appointed officials who either vote on or have the authority to make executive decisions that create state or local laws and regulations. Policymaker Staffers: For the purposes of this research, staffers are considered employees of either policymakers (their advisors) or employees of the state legislative branch employed to support state policymakers in their mission to create legislation and laws. Midto High-level Administrators: For the purposes of this research, midto high-level administrators are considered to be mid-level to high-level policy implementers in state-level bureaucracy (both politically appointed and civil service). These workers have responsibility for making decisions on the implementation of policies signed by the governor or passed by the state legislature. 46

PAGE 64

In this research, this group consists of midto high-level state employees of state departments of education. Policy Research: Lindblom and Cohen define social research in its broadest sense-as anything that might contribute to the knowledge of and solutions to social problems ( 1979). For the purposes of this research, policy "research" or "analysis" is defined as: information that is gathered, analyzed, organized and communicated to help solve public problems. Policy Research Use: For my research, Karen Kirkhart's definition will be applied: research "use" has occurred when it has influenced, shaped, affected or changed persons or systems (Kirkhart, 2000). Policy Goals: A policy goal is the objective of finding a technical policy solution to a public problem. When this goal is held, policymakers are trying to achieve the goal by selecting a policy that solves the problem and maximizes benefits to the most people (Stone, 1997). Political Goals: Political goals are those held by policymakers in order to satisfy their constituency and get reelected. In this set of goals, the policymaker wants to maximize the happiness of his/her constituency sometimes over and above the benefit of other constituencies (Stone, 1997; Schneider & Ingram, 1997). However, policymakers can hold both types of goals simultaneously. Research Use Types: Table 3.1 defines the six types of policy research use that I describe and study in this thesis. 47

PAGE 65

T bl 3 1 s T a e .. IX types o fR esearc hU se-D fi .. e tmttons Agenda Setting Placing the topic on the policy "to do" list Putting a policy topic on the legislative docket for review and bill creation Enlightenment Understanding problems in new ways-conceptual Gaining new information on a problem Instrumental Influence on actual policy decisions such as votes on legislation Rationalization Defending an argument or position on a policy topic Tactical Used in debates and as a strategy to determine new policy options. As a symbol of legitimate decision-making Change Agents Creating activism both politically and publicly around a policy topic Now that the basic exploratory theory, research questions, exploratory hypotheses and terms have been explained and defined, the following section will explain the research design and methods. Methods To answer the research questions and conduct preliminary tests on the exploratory hypotheses and frameworks, a multi-method, cross-sectional design was used. This study was largely descriptive, as my purpose was descriptive the current research on this topic mainly describes the use of policy research by policy-makers as opposed to their staff members or midto high-level administrators. Therefore, this study is adding to the literature by describing whether and how 48

PAGE 66

policymaker staff and public administrators use policy research. However, the interview results, coupled with the descriptive statistics and some inferential statistics from the survey results, do test a preliminary exploratory model of policy research use and its outcomes by policymaker staff and administrators (see Figure 2). Quantitative descriptive statistics were used based on results of an administered survey instrument (Appendix A) to determine frequency of research use and types of research use by policymaker staff and administrators. Qualitative interviews, the second method, were conducted with a variety of individuals who use policy research in their different roles in the policy system. The interviews allowed me to gain deeper knowledge of complex situations and the context of survey answers. These interviews also proved to be very important in clarifying terms that were difficult to clearly define in the survey instrument and in understanding the non-linear way in which the respondents use research-a factor that could not be captured adequately in the survey instrument. The data collected through the survey and interviews added knowledge to the exploratory model and created a more solid model to be tested by further research. Many researchers who have explored this same topic have used the research methods employed. The survey/interview format is one that is tested and accepted when investigating the type of research questions posed in this study (Caplan, 1975; Weiss & Bucuvalas, 1977; Rich, 1977; Hird, 2005; Mooney, 1991). 49

PAGE 67

population (from a database of 1,155 contacts) and a semi-structured interview with a smaller subset of individuals via I 0 telephone interviews. The Survey Instrument The survey instrument was created by combining items from a survey instrument created by John Hird on the use of research for decision-making (2005) and is supported by Karen Kirkhart's theory on evaluation use, specifically the aspects regarding types of influence and timing in the policy cycle (Figure I). In addition to using items and ideas from Hird and Kirkhart, I created items from my theory (Figure 2) that serves as an exploratory model and organizing framework for understanding the data. I conducted a focus group with a sample of this population to determine pertinent questions that will help to achieve the research aims (focus group protocol in Appendix B). Most questions used a Likert scale response that ranged from 1-5 (strongly agree, agree, neither agree or disagree, disagree, strongly disagree or very frequently, somewhat frequently, occasionally, rarely, never). Responses were averaged instead of summed to get a score for each variable for the purposes of comparison. I piloted the survey with a sample of similar participants and adjusted the survey questions to reflect their concerns, including shortening the survey to increase the response rate. In addition, I conducted factor analyses to determine validity of the survey instrument. The following table shows the final linkages between my 51

PAGE 68

survey questions, the type of research use I assigned to each and the policy stage into which each was categorized. T bl 3 2 S a e .. urvey Q t ues Ion, P r st 0 ICY age an dT ype o fR esearc hC ategonzation Survey question wordin2 of policy research Policy Problem To infonn or raise awareness ofpolicymakers Agenda setting and enlightenment To provide needed information that lacks data Agenda setting and enlightenment To understand or define a policy issue in a new way Agenda setting and enlightenment To define the policy_problem Agenda setting Policy Goal To infonn or raise awareness of policymakers Agenda setting and enlightenment To analy_ze a particular policy issue of program Enlightenment To provide needed information that lacks data Enlightenment To develop policy before a specific policy is ever created Agenda setting, instrumental and or considered enlightenment To determine policy options Agenda setting and enlightenment To identify potential problems with certain policy options Enlightenment To provide policy options to policymakers Agenda setting and enlightenment To raise awareness or understanding of a policy issue with Enlightenment your staff To help policymakers avoid making decisions on political Tactical, enlightenment, instrumental whims Political Goal To _provide a rationale for certain policies Rationalization To develop policy before a specific policy is ever created Agenda setting, instrumental and or considered enlightenment To determine policy options Agenda setting and enlightenment To identify potential problems with certain policy options Enlightenment To provide _l)olicy_ to policymakers Agenda setting and enlightenment To gamer support for a policy solution Rationalization, change agent To gamer support for an issue Agenda setting, enlightenment, rationalization and change agent To persuade policymakers to delay policy decisions Tactical and instrumental To provide information that supports certain policies Rationalization The Political and Democratic Process To provide a rationale for certain policies Rationalization To infonn policymakers as they are deciding on a policy Instrumental To provide policy options to policymakers Enlightenment To use as a policy is being considered in the legislative Instrumental and enlightenment, process, testimonies or legislative debates rationalization, tactical 52

PAGE 69

Table 3.2 (continued) Policy Change or Adoption To provide a rationale for certain policies Rationalization To determine policy options Enlightenment To inform policymakers as they are deciding on a policy Instrumental To use as a policy is being considered in the legislative Instrumental and enlightenment, process, testimonies or legislative debates rationalization, tactical To persuade policymakers to delay policy decisions Tactical and instrumental To alter existing policies Instrumental To make midcourse corrections after a policy_ is enacted Instrumental Policy Implementation To help policymakers consider the practical implications of Enlightenment different policy options To provide a rationale for a _policy implementation strategy Rationalization To help guide the implementation process Instrumental and enlightenment To make midcourse corrections after a policy is enacted Instrumental The Interview Instrument The semi-structured interview questions focused on eliciting responses from the interviewees regarding the process in which they use research and the outcomes of their research use. These questions were based on the current research knowledge and the exploratory theory I created. The questions were directly aligned to my research questions and exploratory hypotheses. See Appendix C for the interview protocol. Sample I had access to a sample of policymaker staff and midto high-level administrators in state departments of education through the Education Commission of the States (ECS) constituent database. ECS is a national, nonprofit education 53

PAGE 70

policy organization that serves multiple constituencies (governors, state senators and representatives, state departments of education, state board members, and education administrators) for 49 states and the four U.S. territories. I was an employee ofECS from 1999 through June of 2006 and had unlimited access and permission to use this database. Using the database, I surveyed the entire population of governors' education policy advisors (50), plus a sample of state legislative staffers from 50 states (470). Additionally, at least 300 midto high-level K-12 education department staffers and executive administrators of higher education agencies were surveyed. The entire sample population I selected for the survey totaled 820. In addition, the database categories I selected included various other members of the policy community-such as policymakers and other education policy influencers who could not be omitted from the overall sample group. In total, 1,155 invitations to participate in the survey were sent. I received survey responses from 222 individuals; after discarding responses that were inappropriate to use because of technical error or incompletion, my total sample equaled 213. The response rate to the survey was approximately 20%, which is a consistent response rate for both mail :md internet-based surveys (Dillman, 2000). The 1,155 invitations were sent on ECS letterhead (to elicit a higher response rate) via U.S. mail. Of those letters, 34 were returned due to incorrect addresses or because the addressee no longer worked at that location. There were an estimated 45 duplicates (addresses recorded in the ECS database twice so two letters were 54

PAGE 71

received). The invitation to participate in the research study and the survey included all the necessary University of Colorado at Denver human subjects review protocols and information about my study (Appendix D), and followed the prescribed protocol in Don Dillman's Tailored Method (2000). Included in the invitation letter was a web address that linked the respondent directly to the survey, which was hosted by the online survey tool Survey Monkey. In addition to the letter, I sent the same text from the letter in an email format to everyone in the database sample with an email address (approximately 735). I also sent a postcard reminder, with the link to the online survey, three weeks after the first letter was sent. I sent three subsequent email reminders in two-week intervals from when the letter and postcards were sent. In all, the people in my sample were contacted four times. According to Don Dillman, the single most important factor in increasing the response rate is additional contact with the potential participant (2000). Four contacts with potential participants is an accepted amount of effort to elicit an appropriate response rate (Dillman, 2000). My sample consisted of98 females (58.7 %) and 88 (41.3%). It included 104 policymaker staffers (48.8%) and 61 administrators (28.6 %) as well as 48 policymakers, which comprised 22.5% of the total sample size. An unexpected aspect of my sample was that it included a fair number of policymakers. My research does not focus on policymakers' use of research, however, in some instances, I compared policymakers with staffers and administrators because it 55

PAGE 72

offered some interesting insights. I did not make these comparisons for all research questions as it was not a part of my original study. Based on survey respondents who offered ( 4 7), via their survey form, to be interviewed, a small sub-set of respondents was asked to participate in in-depth telephone interviews. I selected I 0 respondents based on the criteria of role group, including both administrators and staffers, and gender, including both males and females. I used two types of sampling procedures to determine the interviewees: convenience and critical case sampling (Miles & Huberman, 1994). Convenience sampling refers to a method of sampling that takes advantage of opportunities that exist (such as willingness to be interviewed and providing the necessary information in the survey) (Miles & Huberman, 1994). Critical case sampling permits logical generalization and maximum application of information to other cases (Miles & Huberman, 1994). The 10 interview subjects were selected because of their specific job title and role and responsibilities that were most closely aligned to my definitions of policy staffer and administrator. This way I could maximize the application of their comments to others in the field. Analysis Two types of data analysis will be discussed below. First, I will describe how I analyzed the survey data and then how the interview data were collected, coded and analyzed. 56

PAGE 73

Survey Data Analysis The survey results were analyzed using descriptive statistics and some basic inferential statistic techniques to determine frequency, demographics and trends of respondents' answers to all survey questions. The analyses were conducted using SPSS Student Version 12.0. A series of Chi square tests were applied to describe potential similarities or differences between staffers, administrators and also policymaker role groups. For each Chi square test an alpha of .05 was used. Although policymakers were not the intended study population, my sample yielded 48 of them, so they were included in relationship analyses for comparison purposes. The independent variable is policy role group (staffer, administrator or policymaker) and the dependent variable is research use and type of research use. For all analysis an alpha of .05 was selected. Survey Data Coding All data were transferred from Survey Monkey into Microsoft Excel, and then from Excel into SPSS using the correct data export protocols described in the web survey tool database. Seventeen entries that did not include data were removed, and I ensured that all the data "made sense" according to the survey questions. Once the data were in SPSS, many of the survey questions were recoded into numeric data and I ensured that all reverse-ordered survey questions were coded as such. Each variable was coded to be able to obtain a maximum of information (for 57

PAGE 74

example, data categories or groups were not collapsed) and I ensured there was a code or value for each participant (Leech, Barrett & Morgan, 2005). Data were coded so that the following types of respondents could be grouped: policymakers, staffers and administrators. After all the data were coded, an exploratory data analysis was performed to check for any visible problems in the data set such as outliers, missing values or errors in the transfer of data (Leech, Barrett & Morgan, 2005) Frequencies on all survey questions that included means and standard deviations were completed for the whole sample combined, staffers only, administrators only and policymakers only. The results presented in Chapter Four are considered exploratory due to the untested nature of the models and frameworks I used, as well as some of the lessons learned regarding the operational definitions of research, policy goals and political goals. Interview Data Analysis Ten in-depth interviews were conducted with individuals who identified themselves on the survey as users of policy research. I selected individuals who have differing jobs in the policy process that accurately reflect my intended study population. I interviewed seven staffers (two were partisan, meaning they worked for either a Democratic or Republican policymaker; two were nonpartisan, meaning they worked for the entire chamber of policymakers, both Democrats and 58

PAGE 75

Republicans) and three administrators (one was a higher education administrator, two were K-12 administrators). Each interviewee was from a different state and an equal number of males and females took part in the interviews. Interview Data Coding The interviews were analyzed using qualitative analysis techniques on key parts of the exploratory framework using the methods discussed in Miles and Huberman's Qualitative Data Analysis Handbook (1994). The interviewees were asked to participate in a semi-structured interview that followed a protocol (Appendix C) but that allowed for flexibility in "probing" areas of interest as brought up by the interviewee. The interviews were recorded, transcribed and analyzed and coded qualitatively. The interviewees were told via telephone that they were being recorded and their names, states and organizational affiliations would remain confidential. The interview transcriptions were analyzed through coding, which, in qualitative research, is a strategy to find themes and patterns in qualitative data. "Codes" are meant to be a data reduction technique to capture and organize the meaning and topic of the interview segment (Miles & Huberman, 1994). My qualitative interview data was coded according to "thematic/process analysis" in a deductive manner. To do this, pre-set codes derived from my exploratory theory (the six research types and policy process stages) were used to organize and make sense 59

PAGE 76

of my qualitative data. Thematic/process analysis is a type of content analysis that examines text or interviews for major themes, metaphors or stories that give meaning to other data (including my quantitative data). This type of coding fits the purpose of my interview data in that it is a supplemental, contextual set of information that adds more depth of knowledge and clarity to the story that my quantitative, survey data is describing. Reliability and Validity The reliability of this study will be bolstered by the multi-method approach. Reliability refers to how stable and consistent a measurement instrument is in gathering consistent data (i.e., are the same results achieved if the data collection method/instrument is administered more then once) (Coleman & Briggs, 2002). Factors affecting this study's reliability might be: the wording of questions in the survey, the operational definitions assigned or understood by the participant, the way the interviews were conducted, and the respondent's current environment or most recent memory of research use (Kumar, 1996). However, by using both methods survey and interview techniques-the data can serve to "check" both techniques and highlight possible inconsistencies that can be reported in the research findings. The validity of this study was also improved by the implementation of the multi-method approach (Ragin, 1987; Kumar, 1996; Huberman & Miles, 2002). Validity is the strength of research conclusions and inferences the researcher makes 60

PAGE 77

based on the data collected. Validity refers to how well the research describes the phenomenon it is intended to describe (Coleman & Briggs, 2002). Cook and Campbell ( 1979) define validity as the "best available approximation to the truth or falsity of a given inference, proposition or conclusion." Four types of validity are commonly examined in social research (Coleman & Briggs, 2002) and my research is concerned mostly with conclusion, construct and external validity issues: 1. Conclusion or statistical validity asks if there is a true a relationship between one particular condition being measured and the observed outcome. Or the chance that the researcher has made a type I or type II error. 2. Internal validity asks if there is a causal relationship between variables where certain conditions lead to other conditions. 3. Construct validity asks if there is there a true relationship between how the concepts in the study were operationalized to the actual relationship of the phenomenon being studied. 4. External validity refers to our ability to generalize the results of our study to other settings. My research takes in important validity issues. For example, the external validity could be compromised because my sample is largely focused in the policy area of public education. This might affect the ability of my conclusions to be applied or generalized to other policy staffers and administrators in other policy fields. Also, my construct validity is vulnerable due to the difficulties in clearly defining variables such as research use, policy goals and political goals. Each of these terms has a wide range of possible interpretation from participants in my study. 61

PAGE 78

Conclusion validity can be measured based on the p values listed in each separate analysis. Instrument Validity Many survey items were taken from the survey instrument developed by John Hird (2005), however, additional survey items were added to the instrument based on other theoretical models (Kirkhart's) and on the focus group results. These additional questions were tested for face and construct validity via expert review and assistance from professionals in this field. In addition, I conducted a pilot of the survey instrument with a sample of 20 education policy organization staff members. The results of that pilot were incorporated into the wording of the survey questions, the organization of the survey and the removal of certain items. To bolster the face and content validity tests for my survey items, a factor analysis was conducted on the survey instrument after the data were collected. The analysis resulted in a set of five factors that accurately reflect my dependent variables. More discussion of the factor analysis is presented in Chapter Four when I discuss my results. Method Limitations The sample of informants for this study possibly has a bias toward policy research usage because it is already in the ECS database-meaning these individuals either have contacted ECS as a policy organization for some reason or ECS has 62

PAGE 79

contacted them for some reason. This potential bias is a common one that researchers encounter due to the nature of the data sets available to them. Also, as mentioned with respect to external validity concerns, this research will be biased toward policymaker staffers and administrators who work in the field of education policy. It would be important to conduct a similar study with a different set of people who work in another policy field to determine whether education policy staffers and administrators utilize research in a different way than others. One other weakness of this study related to clarity on the types of research use by staffers and administratorsis the difficult and overlapping definitions of these policy research use "types." For example, it may be hard to determine the differences between a "rationalization" use of research and a "tactical" one via the measurement instrument of the survey. Other operational definitions have been problematic within this type of research. In reviewing the data results from the survey, and attempting to determine composite variables or factors, it was difficult to clearly distinguish between "policy goals" and "political goals" in the survey instrument. These two terms are very difficult to measure due to their interrelated nature that the interviews raised. The open-ended questions to which some survey participants responded suggested that many of the questions could be interpreted in multiple ways. Other limitations in the research design include: 63

PAGE 80

The database from which I drew my sample was not completely aligned with the population I wanted to study. The database included some other actors in the policy process including researchers, policy analysts and policymakers. Once the sample was divided into groups of either staffers or administrators, the result was a low sample size for each population. The response rate on the survey (20%) was lower then the goal of 2535%. This response rate was low given my sample's connection and loyalty to the organization of ECS. One explanation for this is that the people in this database have highly variable workloads based on the legislative session in their state. When their legislature is in session, they rarely have time to respond to any requests except immediate ones from their supervisors. In addition, I believe the email and web-based format of the invitation and survey can add to the internet "spam" that many of these individuals try to avoid. Lastly, many of these people receive multiple requests to participate in surveys, focus groups and other various committees to provide feedback. As such it may be an issue of being tired of providing their ideas and feedback to yet one more project. Because I could not find an existing survey instrument that measured the same variables in which I was interested, I had to create a new instrument that had not been tested for validity and reliability. Though I did run factor and reliability analyses after the data were compiled, additional changes to the survey could be made based on the results of those analyses. The pilot test of the survey did not provide enough data to run these tests prior to implementing the survey to my full sample. Due to time and expense of transcription, the interviews were limited to I 0 interviewees. The research would have been strengthened had I been 64

PAGE 81

able to conduct more interviews with a broader number of the survey responders. I would also have ensured that equal numbers of partisan and nonpartisan staffers as well as administrators participated in the interviews. In addition, I would have had a second researcher code the interview transcripts to test for inter-rater reliability of the coding techniques. Despite these weaknesses, the research did achieve its descriptive and exploratory purposes of learning more about how other important policy actors within the policy process use research and for what reasons. We can gain a lot of insight and lessons from this study even with its limitations to apply to future studies investigating these issues. 65

PAGE 82

CHAPTER FOUR RESULTS Data received from 213 survey respondents, combined with 1 0 in-depth interviews with a variety of policy staffers and educational administrators, yielded results that can inform the field of policy analysis and public affairs. This chapter presents the results as organized by the posited research questions outlined in the previous chapters. The second part of Chapter Four discusses how the results of the research questions inform my exploratory hypotheses. Staffers' and Administrators' Use of Research: Research Question One Research question number one inquired, to what extent do policymaker staffers and midto high-level administrators use policy research? The overarching answer, according to the staffer and administrator respondents to the survey, is they use research to a large extent. In over three quarters of the survey research questions asking about frequency of research use, respondents answered that they use research l!ither frequently or very frequently in this manner. The answer to this research question was determined by reviewing the frequency of responses to a series of questions that highlight different types of research use. The mean responses to each type of research use are listed in Table 4.1 below by how frequently staffers and 66

PAGE 83

administrators reported research use (1 =never, 5= very frequently). In addition the right column in Table 4.1 lists the percentage of respondents that answered either very frequently or somewhat frequently to the survey question. Table 4.1: Frequency of Research Use for Staffers & Administrators by Type of Research Use Question description of research type: Mean (SD) Do you use research/information ... to infonn or raise awareness of policymakers 4.41 (.740) %Very frequent or freQuent 89% to analyze a particular policy issue or program 4.40 (.729) 89% to provide a rationale for certain policies 4.3 I (.744) 87% to provide needed infonnation that lacks data 4.28 (.787) 86% to develop policy before a policy is ever created or considered 4.15 (.911) 85% to detennine policy options 4.11 (.684) 89% to inform policymakers as they are deciding on a policy 4.10 (.796) 83% to identify potential problems with certain policy options 3.95 (.815) 74% to provide policy options to policymak.ers 3.94 (.926) 74% to use as a policy is being considered in the legislative process, 3.92 (1.035) 74% testimonies or legislative debates to raise awareness/understanding of a policy issue with your staff 3.89 (1.038) 72% to help policymakers consider the practical implications of 3.89 (.850) 72% different policy options to alter existing policies 3.76 (.748) 75% to understand or define a policy issues in a new way 3.75 (.844) 65% to provide a rationale for a policy implementation strategy 3.73 (1.023) 67% to help policymakers avoid making decisions on political whims 3.73 (1.017) 64% Scale: I= never, 2= occasionally, 3= sometimes, 4= frequently, 5= very frequently 67

PAGE 84

Table 4.1 (continued) Question description of research type: Mean %Very (SD) frequent or Do you use research/information ... frequent to select policies 3.73 (.847) 71% to gamer support for a policy solution 3.72 (.840) 76% to gamer public support for an issue 3.57 (1.414) 65% to help guide the implementation process of policies (in selecting 3.56 (1.097) 63% certain programs to implement policy directives). Scale: I= never, 2= occastonally, 3= sometimes, 4= frequently, 5= very frequently The data presented in Table 4.1 are the combined results from highest reported use to lowest reported use of both staffers and administrators. In this table we see that staffers and administrators report using research most often to inform or raise awareness of policymakers, analyze a particular policy, provide a rationale for certain policies, provide needed information that lacks data, develop policy and detennine policy options. Also, contrary to what most of the literature shows on research use, staffers and administrators do report a high use of research to inform policymakers as they are deciding on a policy (i.e., instrumental use of research). The responses of these groups reflect that they do, indeed, feel that frequently or very frequently the research they conduct and gather is used in the policy process. This implies that they use research frequently in their jobs because when asked about the responsibilities of their job, they stated their job is meant to "inform state level 68

PAGE 85

policy decisions," "provide guidance to elected or appointed officials" and "provide research to infonn state policies" (from survey question 40). One survey question in particular illustrates a slightly different perspective on research question number one. Survey question 32 asks how much the respondent agreed with the following statement: "The infonnation and research I conduct or gather gets used in some way in the policy process (the policy process may include but is not limited to defining, debating, selecting and implementing public policy)." The measure for this question was a Likert scale that ranged from one (strongly disagree) to five (strongly agree). Eighty-eight percent of all respondents either agreed or strongly agreed with this statement. One interesting finding related to this research question is a comparison of what this population said about how they use research in their jobs with how they think research is used in the policy process. When staffers and administrators were asked whether they believe the "information and research they conduct is used in the policy process," the majority said they either strongly agree or agree (87.7%), but when asked if they believe that "policy decisions are often made without policy analysis or research," only 60.5% agreed or strongly agreed with this statement. The explanation for this apparent contradiction could be due to the difference between the tenns "policy decisions" and "policy process." It could also be based on the micro verses macro wording of those two questions. For example a respondent might feel that the research they conduct and gather is used in the policy decisions 69

PAGE 86

that they are involved in but when asking them about the whole "universe" of policy decisions, on average they feel research is not used. However, if it is possible to generalize the findings of this study to other populations of policy staffers and administrators, then we should find that all staffers and administrators feel their research is used in policy decisions thus research is used to inform all or most policy decisions. This idea warrants further investigation. The interviews added other important information to answer research question number one. Participants reported using research, data and policy reports "all the time." They stated that their job responsibilities revolve around finding, collecting, conducting and reporting research use. For example, one partisan staffer said, "We strive to use research and data throughout the [policy] process. Most fundamentally, this administration is data driven in [its] decision-making." Another high-level administrator interviewee stated: We use [research and data] all along the continuum [of the policy process]. There are a variety of policy initiatives that are coming into fruition, some that are being designed, some that are being implemented and some that are being changed. Therefore, our work focuses on data and research for all parts of that process (since we advocate for changes and monitor policies). All interviewees discussed the fact that part of their job was to take raw data, expansive research reports or policy briefs and distill them to their key points to make them more manageable for policymakers to react to and understand. One partisan staffer interviewee stated that they do "quite a bit of repackaging to more precisely fit the needs that a legislator has." 70

PAGE 87

When considering both the survey outcomes and qualitative data from the interviewees, the results show that research is, indeed, used a lot in the policy process by policy staffers and administrators. Every interviewee discussed the importance of the research they use, the training they received in policy analysis (so that they could conduct their own research and analysis), and the direct dependence many of them had on organizations and universities that produce policy research, policy reports and other types of policy information. Frequency and Types of Research Use: Research Question Two The second research question asked: "Considering the six types of policy research use expressed in this proposal, what types are most common?" To answer this question, I made judgments on each survey question regarding the type of research use it reflected based on the six types of use highlighted in the literature review (agenda setting, instrumental, enlightenment, rationalization, tactical, and change agents). To determine if my survey measured the types of research use that I intended, I conducted a factor analysis once all the data were collected. The factor analysis found at least five types of research use that accurately reflect many aspects of my six research use types (listed in parentheses within Table 4.2 below). I labeled these five factors as the following variables: (I) providing information, (2) persuading, (3) finding policy solutions, (4) the influence of politics, (5) whether research is used in 71

PAGE 88

decision-making. The cumulative variance that each factor explains in my research, the connections of those factors to my six research types, and the survey questions that loaded onto each of the five factors is presented in Table 4.2: T bl 4 2 S a e F urvey actors an dV anances % of variance Factor explained Factor 1: Providing information 17.462 (Enlightenment) Survey questions that loaded onto this factor: 2,3, 4, 5,6, 7,10, II, 12, 13, 14, 15, 16, 17, 18,34c Factor 2: Persuading 29.001 (Rationalization and Change Agent) Survey questions that loaded onto this factor: 3, 7, 8, 9, 13, 16, 18, 19,27 Factor 3: Finding policy solutions 36.488 (Instrumental) Survey questions that loaded onto this factor: 16, 3 I, 32,34a,34c,34d,34e Factor 4: Influence of politics 42.932 (Tactical and Rationalization) Survey questions that loaded onto this factor: 16, 27, 30,33,34b Factor 5: Whether research is used in 49.278 decision-making (Instrumental) Survey questions that loaded onto this factor: 1, 28, 29 This factor analysis shows that almost half of the variance in survey responses is accounted for in these five factors. There are eight eigenvalues (a measure of explained variance) greater than 1.0, which suggests there might be three additional factors worth investigating (Leech, Barrett & Morgan, 2005). However, when analyzing the additional three factors, they did not seem to have any clear, explained 72

PAGE 89

concepts that I could decipher. The factor analysis suggests that if my survey were altered, it might want to include more items that reflect the fifth factor-instrumental use -to ensure that each of the types of research use had adequate opportunity to be measured in the survey instrument. After determining that my survey items were a close measure of my six research types, I then looked at the survey items that reported the highest frequency and determined the type of research each was referencing. To operationalize the phrase "most common" I created a "cut" score of 4.0 (mean). The most common types of policy research use are presented below (in order of frequency). The mean score received from all staffer and administrator respondents follows each statement. T bl 4 3 F a e .. requency o fT ypes o fR esearc hU se I Type of Research Use Question Mean Enlightenment and To inform or raise awareness of 4.41 Agenda Setting policymakers Enlightenment To analyze a particular policy issue or 4.40 program Rationalization To provide a rationale for certain 4.31 policies Enlightenment To provided needed information that 4.28 lacks data Agenda Setting To develop policy before a specific 4.15 policy is ever created or considered Enlightenment To determine policy options 4.11 Instrumental To inform policymakers as they are 4.10 deciding on a policy Scale: 1= never, 2 =rarely, 3 = occastonally, 4 =somewhat frequently, 5 =very frequently 73

PAGE 90

The results ofthe survey support the research expressed in the literature review that one of the main uses of research is for enlightenment purposes that is, understanding a policy problem in new or more in-depth ways and gaining new information on a policy problem. The research in the literature review focused on how policymakers use research-and now my research supports the premise that staffers and administrators also use research mostly for enlightenment purposes. However, one interesting note in these results is that research use for instrumental purposes ranked very high. This finding contradicts some previous studies but not all (see Fitzpatrick, 2004; Ginsberg & Rhett, 2003) regarding the use of research for instrumental purposes by policymakers. The interviews provided more insight into the types of research use by staffers and administrators. Indeed, when asked how they use research the most, the staffers and administrators interviewed responded with answers that reflect use of research for agenda setting (policy problem definition and placing the issue on the policymakers list of topics to address), enlightenment (answering specific questions that policymakers have related to a policy issue) and policy option creation (both driven by political goals and policy goals). One nonpartisan staffer said that his job was to "create options to address a policy issue that was brought before him from a legislator." He also said that even though policymakers usually identify the problem for which they want research, sometimes the research conducted by legislative staff identifies a problem not known before. One administrator noted that she is deeply 74

PAGE 91

involved in policy agenda setting and policy enlightenment through the use of research, but that the ways in which she "enters" that information into the policy process are "under the radar." For example, she gives research findings to key powerful people in the state (sometimes prominent business executives) and lets them highlight certain research implications to policymakers. She does this, she said, because she tries to get the most influential people in the state to "deliver the policy message." The interviews revealed a distinction between how partisan staff and nonpartisan staff or administrators responded to questions. As expected, partisan staff used research more overtly in rationalization or tactical ways (one might say a more political way) to dissuade policymakers from certain options or by trying to discredit a certain political party's policy options. One political staffer noted that her use of research differs depending on whether her own political party is in power or not. She said that if the legislators for whom she works are in the minority, then the use of research is much more political, and in that case '"(research] is not used to craft solutions, it is used to make amendments and find the holes in bills to exploit them for their own constituency." However, even nonpartisan staffers reported research uses that could be described as tactical and political in a covert way in the policy process. For example, one administrator described how careful he has to be with the data he collects, and more importantly how he presents that data to policymakers. He said that he could present the same set of education data on 75

PAGE 92

student achievement for the purpose of either arguing for the disintegration of the public education system or the investment in it, saying that the "careful selection of adjectives is of paramount importance." He said: Because I work for the government, I believe in the Hippocratic oath as well-' first do no harm.' And since I believe the public education system is important in this country to alleviate harm, I will advocate for it in my own way through data and research presentations. This statement illustrates a long-term discussion in the field of public administration regarding the discretion that administrators have in implementing and, in this case, shaping public policy based on their own values. These examples from the interviews illustrate some nuances of how staffers and administrators use research most commonly in setting policy and political agendas, dissuading certain policy options and providing information to policymakers through a number of different ways. Policy Research Use and The Policy Process: Research Question Three My study reflects that policymaker staffers and administrators frequently use policy research and that the most frequent types of use are related to agenda setting, enlightenment and rationalization reasons. Now, how do these types of research use fit with the policy process? Research question three asks: "When policy research was used by staffers and administrators, what was the stage of the policy process and the type of research use [according to the process described in Figure 2]? 76

PAGE 93

Staffers and administrators who participated in the survey reported they used policy research at all stages of the policy process (which included policy problem definition, policy agenda setting, policy option creation-for both policy and political goals-policy selection, policy change and policy implementation). When presented with multiple stages of the policy process in one question (survey question 34), respondents suggested that they use research most in determining policy options (89% either agreed or strongly agreed that the research they conduct or gather gets used at this stage) and defining the policy problems (82% either agreed or strongly agreed that the research they conduct or gather gets used at this stage). These results are also consistent with respondent reports of using research most often for enlightenment (related to determining policy options) and agenda setting purposes (related to defining policy problems). Respondents indicated their use of research was fairly equal among the next three policy stages: garnering support for a policy solution (76% either agreed or strongly agreed that the research they conduct or gather gets used at this stage) (this relates to my political goal and political process stages); altering existing policies (75% either agreed or strongly agreed that the research they conduct or gather gets used at this stage) (this relates to my policy change stage); and selecting policies (71% either agreed or strongly agreed that the research they conduct or gather gets used at this stage) (this relates to my policy selection stage). 77

PAGE 94

One of the most interesting findings from the results of this research question is the high number of reports of using research for instrumental purposes in the policy selection stage of the policy process. This is notable because most of the literature suggests that research is infrequently used for direct purposes, at least by policymakers (Weiss & Bucuvalas, 1977). Indeed, if one asks a different population of policy actors (staffers and administrators) whether policy research is used in policy decision-making, the answer is, "yesquite often." Fitzpatrick also found a high use of instrumental use in her research on exemplary evaluators (2004). A complete list of the policy process and the research use type within that stage of the policy process is below. When a judgment was made on intent of all the survey questions, it shows in descriptive terms this population uses research slightly more in the policy problem and policy goal stages but not as much in the implementation stage (as compared to the democratic process stage). The list below shows how I categorized the survey questions into the policy process stages (the underlined headings) and the types of research use I associated with each survey question; sometimes survey questions were categorized in multiple places. Policy Problem Definition QS: to inform or raise awareness ofpolicymakers (agenda setting and enlightenment) Q6: to provide needed information that lacks data (agenda setting and enlightenment) 78

PAGE 95

Ql 0: To understand or define a policy issue in a new way (agenda setting and enlightenment) Q34a: to define the policy problem (agenda setting) Policy Goal Q5: to inform or raise awareness of policymak.ers (enlightenment) Q2: to analyze a particular policy issue or program (enlightenment) Q6: to provide needed information that lacks data (enlightenment) Q4: to develop policy before a specific policy is ever created or considered (agenda setting, instrumental, enlightenment) Q34c: to determine policy options (enlightenment) Ql2: to identify potential problems with certain policy options (enlightenment and maybe tactical) Q II: to provide policy options to policymak.ers (instrumental, enlightenment) Q9: to raise awareness or understanding of a policy issue with your staff (agenda setting, instrumental, enlightenment) Q 13: to help policymak.ers avoid making decisions on political whims (tactical, instrumental and enlightenment) Political Goal Q3: to provide a rationale for certain policies (rationalization) Q4: to develop policy before a specific policy is ever created or considered (agenda setting, instrumental, enlightenment) Ql2: to identify potential problems with certain policy options (enlightenment and maybe tactical) Qll: to provide policy options to policymak.ers (Instrumental, enlightenment) Q8: to garner public support for an issue (agenda setting, enlightenment, rationalization and possibly change agent) Q7: to persuade policymak.ers to delay policy decisions (tactical and instrumental) Q27: to provide information that supports certain policies (rationalization) Democratic/Political Process Q3: to provide a rationale for certain policies (rationalization) Ql5: to inform policymakers as they are deciding on a policy (instrumental) Qll: to provide policy options to policymakers (instrumental, enlightenment) QI4: to use as a policy is being considered in the legislative process, testimonies or legislative debates (instrumental and enlightenment) 79

PAGE 96

Policy Change Q3: to provide a rationale for certain policies (rationalization) Q34c: to determine policy options (enlightenment) Ql5: to inform policymakers as they are deciding on a policy (instrumental) Ql4: to use as a policy is being considered in the legislative process, testimonies or legislative debates (instrumental and enlightenment) Q7: to persuade policymakers to delay policy decisions (tactical and instrumental) Q34e: to alter existing policies (instrumental) Q34d: to select policies (instrumental) Implementation Stage Ql7: to help policymakers consider the practical implications of different policy options Ql8: to provide a rationale for a policy implementation strategy (rationalization) Q 19: to help guide the implementation process of policies (in selecting certain programs to implement policy directives) (instrumental and enlightenment) Ql6: to make midcourse corrections after a policy is enacted (instrumental) The interviews highlighted the difficulty in separating the types of research use discretely into the stages of the policy process. For example, one nonpartisan staffer was confused by the following question: "When in the policy process do you use research and how?" Even after I listed several stages of the policy process, he stated: Umm, I'm not sure about this question .. .I probably use it most in the policy development stage because it is important that I understand the complexity of issues so that I can answer questions and provide information at all parts of the policy process. The interviews, and this response in particular, highlighted the nonlinear way that policy is developed and selected in states across the country. Most interviewees said 80

PAGE 97

that they usc research all through the policy process and that most often it is used to infonn or enlighten policymakers with whom they work. Where Do Staffers and Administrators Get Their Research? Research Question Four Research question four asks: How and where do policy staffers and midto high-level administrators get access to policy research? To answer this question, I analyzed the results of survey question 22 (asking how often respondents use a variety of web/internet sources) and questions 24 and 25 (inquiring about other ways to get research through journals, conferences, meetings, staff conversations, etc.). Table 4.4 shows the sources and medium of research and how often respondents reported using each in order of highest to lowest reported use. Table 4.4: Sources and Medium of Research for Both Staffers and Administrators Type of research source MEAN (SD) Organization reports and publications 4.22 (1.083) Government websites 4.11 (1.134) Conversations with colleagues 4.05 (1.074) Policy organization websites 4.00 (1.143) Meetings 3.98 (1.101) Journals: practitioners or association journals 3.78 (1.203) Professional association websites 3.65 (1.222) Conferences 3.58 (1.110) Reports compiled by your staff 3.55 (1.346) Journals: Academic 3.48 (1.200) Newspapers 3.38 (1.291) University websites 3.12 (1.094) Memos from staff or polic}'makers 3.08 (1.094) Think tank websites 2.95 (1.181) Scale: 5= Very Frequently 4= Somewhat Frequently 3= Occasionally 2= Rarely 1= Never 81

PAGE 98

Table 4.4 (continued) Type of research source MEAN (SO) Mag_azines (such as Time or Newsweek) 2.82 ( 1.068) Political Think tank websites 2.58 (1.100) Lexis Nexis or Westlaw 2.35 (1.251) Scale: 5= Very Frequently 4= Somewhat Frequently 3= OccasiOnally 2= Rarely I= Never Respondents said they use policy organization wcbsites most frequently and they also use government websitcs and professional association websites "somewhat frequently." Interestingly, respondents said they use partisan policy "think tank" websites rarely. When illustrating the above data from the perspective of only source of information, results show: Table 4.5: Sources of Research for Both Staffers and Administrators Type of research source MEAN (SO) Government websites 4.11 (1.134) Conversations with colleagues 4.05 (1.074) Policy organization websites 4.00 (1.143) Meetings 3.98 (1.101) Professional association websites 3.65 (1.222) Conferences 3.58 (1.110) Reports by your staff 3.55 (1.346) N eWSJ>ai>_ers 3.38 (1.291) Universi!Y_ websites 3.12 (1.094) Think tank websites 2.95 (1.181) Magazines (such as.Time or Newsweek) 2.82 (1.068) Political Think tank websites 2.58 (1.100) Lexis Nexis or Westlaw 2.35 (1.251) Scale: 5= Very Frequently 4= Somewhat Frequently 3= Occasionally 2= Rarely I= Never 82

PAGE 99

An interesting result of this research question is that it highlights the importance of networking. having face-to-face meetings and interacting with colleagues. Even with the introduction of the internet and the ability of many actors in the policy process to now have research and information at their fingertips. there is still a premium placed on face-to-face interactions with trusted colleagues. "Conversations with colleagues." "meetings" and "conferences" were all among the top I 0 ways or places that staffers and administrators get research and information. When looking at the data with respect to only the medium of research we find the following results (note that some descriptions can be classified as both a source and a type of medium of research): Table 4.6: Medium of Research for Both Staffers and Administrators Type of research medium MEAN (SD) Organization reports and publications 4.22 (1.083) Conversations with colleagues 4.05 (1.074) Journals: practitioners or association journals 3.78 (1.203) Reports compiled by your staff 3.55 (1.346) Journals: Academic 3.48 (1.200) NewsJ:>a_Qers 3.38 (1.291) Memos from staff or policymakers 3.08 (1.094) Magazines (such as Time or Newsweek) 2.82 (1.068) Scale: 5= Very Frequently 4= Somewhat Frequently 3= Occasionally 2= Rarely I= Never It is clear that the internet is having an impact on the way staffers and administrators gather research. At least two and possibly three of the top five ways in which respondents gather research utilize the internet. When taking into account 83

PAGE 100

both the survey information and the interview responses, the internet emerged as the main source of information and data collection. For example, one partisan staffer stated: I am so thankful for all the resources out there and all the things I can get off of the web. All these organizations are working to produce information on very small areas and that is a wonderful resource to have for policy stafferswe rely on that information to inform the [policy] discussion. Another administrator commented: "We [referring to himself and other administrators with whom he works] all use the web a great deal." Another nonpartisan staffer highlighted the importance of organizations dedicated to collecting policy information and conducting policy research. He uses information from various policy organizations "all the time" saying ;'it is very important to have literature reviews on policy topics that are already done by many policy organizations." He relies on this information to save time during the quick pace of the legislative session. I ran Chi squares on each item to detect any significant differences in the way staffers, administrators and also policymakers answered these questions on sources and medium of research. I found statistically significant differences in only one area: how often the respondent reported using government websites. Table 4. 7 shows the results of that analysis, including percentage results for each role group. The relationship is strong to moderate; the largest difference among cells is 34%. Staffers report they very frequently use government websites almost twice as much 84

PAGE 101

as policymakers. The differences are not as strong between staffers and administrators. This result is statistically significant at p=.036. Table 4.7: How Access to Research Differs for Policymakers, Staffers and Administrators "How often do you use government websites?" Staffers Administrators Policymakers Frequency (n = 48) (n = 32) (n = 27) Never 0% 0% 4% Rarely 4% 16% 22% Occasionally 25% 37% 37% Very Frequently 71% 47% 37% Chi sq.= 13.457, p. < .05 Staffers' Versus Administrators' Use of Research: Research Question Five As the interview results of previous research questions have illustrated, the responses of policy staffers (both partisan and nonpartisan) and midto high-level administrators differ with respect to some research use issues. When distinguishing between the two groups and analyzing the survey results, I identified any signiticant differences. Research question number five asks: How does research use differ between policymaker staffers and midto high-level administrators? I conducted cross tabs and Chi square tests to compare the groups and to determine if any statistically significant differences existed. The results are presented in Tables 4.84.13 and include the types of research use that had a statistically significant 85

PAGE 102

difference in how policymakers, staffers and administrators responded to the particular survey question. The significance level (p value) is listed below each table. Providing a Rationale for Certain Policies The Chi square in Table 4.8 shows percentage results regarding how each group uses research to provide a rationale for certain policies. The results indicate significant differences in the way that policymakers, staffers and administrators use research to provide a rationale for policies. The strength of the relationship is moderate with respect to the difference in how policymakers use research as a rationale, but there is a weak relationship between the differences among staffers and administrators. Policymakers report about a 20% higher frequency of using research to provide a rationale for policies (at the "very high" frequency) as compared to how staffers and administrators report research use in this way. The differences here are mostly between policymakers and how they use research for this purpose as compared to how staffers and administrators use research in this way. Table 4.8: How Does Research Use Differ Between Staffers, Administrators d P r k T d 1 ? an o 1cyma ers-o prov1 e a rat10na e. ''Do you use research to provide a rationale for certain policies? Staffers Administrators Pol icymakers Frequency (n = 79) (n=61) (n = 48) Very Frequently 46% 48% 67% Somewhat frequently 33% 44% 31% Occasionally or rarely 21% 8% 2% Chi sq.= 15.27, p. < .05 86

PAGE 103

Persuading Policymakers to Delay Policy Decisions The Chi square in Table 4.9 below shows percentage results regarding how much each group uses research to persuade policymakers to delay policy decisions. Analyses revealed that the groups' responses differed significantly in whether they used research for these purposes. Responses indicate that policymakers are almost twice as likely to use research for this purpose than the other two groups, although even with policymakers only 30% report using results in this manner very frequently. In contrast, a majority of staffers and administrators report they rarely, if ever use research for this purpose. Staffers report the lowest use of research for this purpose. Table 4.9: How Does Research Use Differ Between Staffers, Administrators and P r ak T d 1 r d ? o tcym ers 0 e ay po tcy eclSlons. "Do you use research to persuade makers to delay policy decisions?" Frequency Staffers Administrators Policymakers (n = 79) (n = 60) (n = 48) Very Frequently 14% 15% 31% Somewhat frequently 24% 20% 35% Occasionally 35% 52% 25% Rarely or Never 27% 13% 8% Chi sq.= 21.064, p. < .01 Garnering Public Support for a Policy Issue Table 4.10 below illustrates how using research to garner public support for a policy issue has statistically significant differences for each role group. 87

PAGE 104

Administrators report significantly higher use of research for this purpose than either statiers or policymakers. The differences reported here show a low to moderate relationship. Administrators use research for this purpose the most-significantly more than staffers do, but only slightly more so than policymakers. Table 4.10: How Does Research Use Differ Between Staffers, Administrators an d P r ak T br ? o 1cym ers-o gamerpu 1c support. "Do you use research to gamer public su_pQ9rt for an issue?" Staffers Administrators Policy makers Frequency (n = 79) (n = 601 (n = 48) Very Frequently 29% 42% 36% Somewhat frequently 19% 35% 29% Occasionally 18% 17% 17% Rarely 6% 2% 5% Never 28% 5% 13% Cht sq. = 31.034, p. < .001 Understanding a Policy Issue in a New Way Statistically significant differences in how staffers, administrators and policymakers report use of research in understanding policy issues are shown in Table 4.11 below. 88

PAGE 105

Table 4.11: How Does Research Use Differ Between Staffers, Administrators an dPr ak T d d dfi r ? o 1cym ers-o un erstan or e me a po ICY ISSUe. "Do you use research to understand or define a policy issue in a new way?" StatTers Administrators Policymakers (n = 78) (n = 60) (n = 48) Very Frequently 22% 17% 33% Somewhat frequently 33% 52% 44% Occasionally 42% 23% 21% Rarely 3% 8% 2% Chi sq. = 15.456, p. < .05 Table 4.11 shows that policymakers use research more than staffers or administrators to understand or define a policy issue in a new way, as indicated at the very frequent level. The results of this relationship are significant at the p=.Ol7 level. If the frequency rates of "somewhat frequently" and "frequently" are combined, we see that 77% ofpolicymakers use research in this way, compared to 69% of administrators and 55% of staffers. This indicates a moderate relationship between policymakers and statiers, with a 22% difference in reporting. The difference between policymakers and administrators (8%) is lower as is the difference between administrators and staffers (14%) in terms of how they use research for this purpose. 89

PAGE 106

Making Mid-course Corrections After a Policy is Enacted A strong relationship exists with regard to the differences in very frequent use of research by policymakers after a policy is enacted to make mid-course corrections to an existing policy. Table 4.12 below highlights these differences, showing that policymakers say they use research in this manner very frequently a 20% difference over staffers and a 26% difference over administrators. Staffers and administrators do not significantly differ in how they use research in this manner. Table 4.12: How Does Research Use Differ Between Staffers, Administrators and P r ak F d ? o Icym; ers-or m1 -course correctiOns "Do you use research after a policy is enacted, for example, to troubleshoot and/or to make mid-course corrections to an existin,l;l policy? Staffers Administrators Policymakers Frequency (n = 79) (n =59) (n = 48) Very Frequently 13% 7% 33% Somewhat frequently 33% 41% 33% Occasionally 42% 42% 25% Rarely 10% 8.5% 8% Never 2.5% 2% 0% Ch1 sq. = 17.339, p. < .05 Guiding the Implementation Process of Policies Table 4.13 below shows the differences in how staffers, administrators and policymakers use research in guiding the implementation process. 90

PAGE 107

Table 4.13: How Does Research Use Differ Between Staffers, Administrators an d P I' k T "d I t ? o tcyma ers-o gm e 1mp1 emen atton. "Do you use research to help guide the implementation process of policies (for example, in selecting certain programs to implement policy directives?" Staffers Administrators Policymakers (n = 79) _(n = 60) (n = 48) Very Frequently 23% 18% 31% Somewhat frequently 28% 50% 48% Occasionally 25% 23% 15% Rarely 16.5% 5% 6% Never 8% 3% 0% Chi sq.= 18.313, p. < .05 Lastly, there was significant difference at p= .019 in how staffers, administrators and policymakers used research to guide the implementation of policies, for example, in selecting certain programs to implement policy directives. Unexpectedly, administrators reported the lowest use of research in this manner-at the "very frequent" level. However, in combining both "very frequently" and "somewhat frequently," we see that 68% of administrators report frequent use of research in this manner, compared to 79% of policymakers and only 51% of staffers. This reflects a low strength of difference between policymakers and administrators, and administrators and staffers, but a moderate strength of difference between policymakers and staffers. 91

PAGE 108

Summary of Differences Between Role Groups Overall the results show that when combining the responses of "very frequently" or "somewhat frequently," the groups reported different use of research in statistically significant ways: To provide a rationale for policies Policymak.ers67% Administrators 48% Staffers-46% To persuade policymak.ers to delay policy decisions Policymak.ers 66% Staffers -38% Administrators -35% To gamer public support for a policy issue Administrators 77% Policymak.ers 65% Staffers48% To define a policy issue in a new way Policymak.ers 77% Administrators 69% Staff55% To make mid-course corrections to policies already enacted Policymak.ers 66% Administrators48% Staffer-46% To guide the implementation of policies Policymak.ers 79% Administrators 68% Staffers-51% 92

PAGE 109

Summary Thoughts on Results of the Research Questions Overall, policy staffers and administrators reported very high uses of research in their jobs at all stages of the policy process. The most common types of research use for policy staffers and administrators are enlightenment (providing needed infonnation to policymakers, the public and other staffers), agenda setting (placing a topic on the policy agenda) and rationalization (defending an argument or certain policy option). In addition, they reported high use of research at the policy stages concerned with defining the policy problem and determining policy options. Research use differed in statistically significant ways between staffers and administrators in the following manners: (1) to provide a rationale for certain policies (higher for administrators), (2) to use as a policy is being considered in the legislative process (higher for staffers), (3) to use in testimony or legislative debates (higher for staffers), ( 4) to raise awareness or understanding of a policy issue with staff (higher for administrators), (5) to help policymakers avoid making decisions on political whims (higher for administrators), (6) to gamer public support for an issue (higher for administrators), and (7) to help guide the implementation process of policies (higher for administrators). Some of these findings (1, 5 and 6) contradict my exploratory theory, which hypothesizes that administrators are less likely to use research for political purposes than policymaker staffers. 93

PAGE 110

Hypotheses The proposed hypotheses reflect my own ideas and inclinations about my presented exploratory theory on research use and the policy process. My inclinations are based on the literature regarding the policy process and how policy research is used in this process. They are meant to guide the analysis of my data and provide direction for further research projects based on the preliminary data gathered through this research. Due to the design of my research, which is mainly descriptive, the following hypotheses are not meant to be accepted or rejected, but merely to help guide the analysis and information gathered with respect to the research questions, and to eventually inform the testing of the exploratory theory. Both the qualitative and quantitative data gathered through this research project have provided some possible directions for these hypotheses. In order to accept or reject the hypotheses, additional research would be warranted to gather more directional data. Following are some preliminary information and ideas regarding each hypothesis presented in Chapter lbree. Hypothesis One Hl: Policy staffers and midto high-level administrators utilize research in different ways depending on the stage of the policy process as depicted in the exploratory theory. This hypothesis was supported by the data available but further research and exploration are necessary. After categorizing the survey questions according to the 94

PAGE 111

stages of the policy process, as depicted in the graphic, and averaging the results for each stage, the following results were found (Table 4.14). Overall, the types of research use tend to focus on the areas of agenda setting (policy problem stage), enlightenment (policy problem and policy goal stage), and the political process findings that are consistent with past study on research use by policymakers. However, some answers to survey questions suggest there is more instrumental use of research than one may expect based on past research (Hird, 2005; Kirp, 1992; Bardach, 1984). Table 4.14 below indicates which type ofresearch use is meant by each survey question again listed from the highest reported use to the lowest and categorized by policy process. It is important to note that many survey questions are linked to multiple parts of the policy process and multiple types of policy research use. There is some overlap in categories they are not discrete. The table shows that when the survey responses are averaged for each policy stage, the policy problem and policy goal stage report the highest frequency of research use at an average of 4.09. The political and democratic process stage is very close showing an average frequency of 4.06. The policy change or adoption stage comes next at an average of 3.83. Lastly, the political goal and policy implementation stages show frequency of use averages at 3.77 and 3.64 respectively. These results offer support for hypothesis one, in that there do seem to be some stages of the process where research use is more frequent and different in purpose. 95

PAGE 112

Table 4.14: Research Use by Reported Frequency and Policy Stage for Both Staffers and Administrators Survey question wording Mean Type of policy research Policy Problem To inform or raise awareness ofpolicymakers 4.41 Agenda setting and enlightenment To provide needed information that lacks data 4.28 Agenda setting and enlightenment To understand or define a policy issue in a new 3.75 Agenda setting and enlightenment way To defme the policy problem 3.93 Agenda setting Average 4.09 Policy Goal To inform or raise awareness ofpolicymakers 4.41 Agenda setting and enlightenment To analyze a particular policy issue or program 4.40 Enlightenment To provide needed information that lacks data 4.28 Enlightenment To develop policy before a specific policy is ever 4.15 Agenda setting, instrumental and created or considered enlightenment To determine policy options 4.11 Agenda setting and enlightenment To identify potential problems with certain policy 3.95 Enlightenment o{>tions To provide policy options to policymakers 3.94 Agenda setting and enlightenment To raise awareness or understanding of a policy 3.89 Enlightenment issue with _your staff To help policymakers avoid making decisions on 3.73 Tactical, enlightenment, instrumental political whims Average 4.09 Political Goal To provide a rationale for certain policies 4.31 Rationalization To develop policy before a specific policy is ever 4.15 Agenda setting, instrumental and created or considered enlightenment To determine policy options 4.11 Agenda setting and enlightenment To identify potential problems with certain policy 3.95 Enlightenment options To provide policy options to policymakers 3.94 Agenda setting and enlightenment To gamer support for a policy solution 3.72 Rationalization, change agent To gamer support for an issue 3.57 Agenda setting, enlightenment, rationalization and change agent To persuade policymakers to delay policy 3.24 Tactical and instrumental decisions To provide information that supports certain 2.99 Rationalization policies Average 3.77 Scale: S= Very Frequently 4= Somewhat Frequently 3= OccasJOnally 2= Rarely 1= Never 96

PAGE 113

Table 4.14 (continued) The Politital and Democratic: Process To provide a rationale for certain policies 4.31 Rationalization To inform policymakers as they are deciding on a 4.10 Instrumental To provide policy options to policymakers 3.94 Enlightenment To use as a policy is being considered in the 3.92 Instrumental and enlightenment, legislative process, testimonies or legislative rationalization, tactical debates AveJl!ge 4.06 Policy Change or Adoption To provide a rationale for certain policies 4.31 Rationalization To determine policy options 4.11 Enlightenment To inform policymakers as they are deciding on a 4.10 Instrumental policy To use as a policy is being considered in the 3.92 Instrumental and enlightenment, legislative process, testimonies or legislative rationalization, tactical debates To persuade policymakers to delay policy 3.24 Tactical and instrumental decisions To alter existing policies 3.76 Instrumental To make midcourse corrections after a policy is 3.40 Instrumental enacted Average 3.83 Policy Implementation To help policymakers consider the practical 3.89 Enlightenment implications of different policy options To provide a rationale for a policy 3.73 Rationalization implementation strategy To help guide the implementation process 3.56 Instrumental and enlightenment To make midcourse corrections after a policy is 3.40 Instrumental enacted Average 3.64 Scale: 5= Very Frequently 4= Somewhat Frequently 3= OccasiOnally 2= Rarely I= Never The results of the interviews also support hypothesis one. Almost all interviewees described different ways they use research depending on the goal and part of the policy process. In the agenda setting phase, these staffers and administrators focused on collecting state-specific data to paint a picture of a policy problem. They also described having to put research together in response to a 97

PAGE 114

policymaker's request (usually based on the policymaker's values and beliefs and ideas to serve his/her constituency). Surprisingly, most interviewees used research for policy evaluation purposes. Many nonpartisan legislative offices are often required by law to hold agencies accountable and monitor progress of policy and program implementation. They are tasked with the "evaluation" of these policies at certain time intervals after the policy is passed and the program is implemented. Still others (only partisan staffers) said that when their political party was not in session, they used research for a whole different purpose: completely tactical in nature, to "find the political holes and exploit them" in the other party's platforms or policies. Hypothesis Two H2: Policy staffers and midto high-level administrators are more likely to use research at the policy problem, policy goal and implementation stages than at the political goal, democratic process or policy change stages. This hypothesis is inconclusive due to difficulties in defining the stages of the policy process in my survey questions. I used professional judgment regarding which survey items fit within or "defined" each part of the policy process. There are limitations to these judgments and many survey items could fit within each part of the policy process since the policy process is not linear nor are the stages discrete. Some possible ''trends" might be occurring when reviewing the averages of Table 4.14. The trends might suggest that staffers and administrators use research more in 98

PAGE 115

the policy goal stage (average frequency of 4.09) than in the political goal stage (average frequency of3.77). There is some evidence to the contrary of my hypothesis that staffers and administrators would use research more for implementation purposes than other policy process purposes. These data lend some evidence to the opposite staffers and administrators do not use research for implementation as much as they do in all other stages of the policy process (the average for frequency of research use in the implementation stage was the lowest of any stage at 3.64). Again, it is difficult to compare these numbers because the variable being measured by the survey question could be categorized in multiple stages of the policy process. The data from the interviews did provide additional information on this hypothesis. In fact, I was able to glean more information from the interviews because I could verbally describe and define what I meant by the different stages of the policy process. Basic analysis of the qualitative data suggests there might be some truth to this hypothesis, especially when referring to whether this group of staffers and administrators are more connected to policy goals or political goals. The exception would be the responses from partisan staffers they seemed more connected to political goals. Following is a sample of responses from each group when asked if they consider their job more policy related or politically related: Partisan staffer: "I think it is both, it very much depends on where you are sitting. If your party is in the minority it is much more political." 99

PAGE 116

Partisan staffer: "This is a tricky question but inherently we are political. We work for the governor." Nonpartisan staffer: "Policy related, we are nonpartisan. We work for the entire legislative membership but we are here to assist members get their legislation advanced, which gets into the political." Nonpartisan staffer: "More policy-related yet clearly recognizing that politics are the vast majority of the considerations here." Administrator: "My research gets used both ways, but we try hard to just put the research or numbers out there and try to be independent in our interpretation." Administrator: "In the sense that no one knows I am a Democrat, it is policy related. In the sense that I don't think policy happens outside of the political context, it is politically related." These interactions between the actors' policy goals and political goals raise issues that warrant further research, though similar challenges of defining the terms will remain. Hypothesis Three HJ: Policy staffers and midto high-level administrators are more likely to use policy research with respect to achieving policy goals as opposed to political goals. This hypothesis is closely aligned with hypothesis number two and is largely inconclusive with this set of data due to definitional problems between policy goals and political goals. Interview data from policymaker staff and administrators confirm these are very difficult terms and concepts to separate. Each interviewee 100

PAGE 117

mentioned that policy goals and political goals are interrelated. In general, interviewees associated themselves mostly with one or the other goal (partisan staffers with political goals and administrators and nonpartisan staffers with policy goals), but they all stated that the goals "cross over" all the time. The quantitative survey data expressed through both the lens of my research questions and qualitative interview data suggest that this hypothesis might be rejected upon further research. Hypothesis Four H4: Policy staffers and midto high-level administrators will access research via newer technologies (such as the internet). This hypotheses is supported, but due to the lack of comparative data regarding how these groups accessed data in the past, there is no way to describe differences or changes over time. However, at least two and possibly three of the top five ways that staffers and administrators said they accessed research utilized the internet. When combining this information with the interview data, the internet emerged as the main source of information and data collection. The overall findings that inform this hypothesis are in Table 4.3 presented earlier in this chapter. Interview data support the idea that the internet is a key component of how staffers and administrators find information through organization websites, journals and legislative databases. However, none of the interviewees were working in their current positions before the internet was created, therefore a comparison of whether 101

PAGE 118

this change has enabled more research would be difficult. For example, one partisan staffer stated: I am so thankful for all the resources out there and all the things I can get off of the web. All these organizations are working to produce information on very small areas and that is a wonderful resource to have for policy staffers. We rely on that information to inform the [policy] discussion. Another administrator commented, "We [referring to himself and other administrators he works with] all use the web a great deal." Hypothesis Five HS: When comparing groups, policy staffers will be more closely aligned with political goals than midto high-level administrators. This hypothesis is inconclusive. However, my data do suggest some descriptive trends and areas in which to dig deeper through further research. My data suggest the following statements in response to these hypotheses. Administrators use research more to (percentage frequencies are based on Chi square results reported earlier in this chapter): Provide a rationale for policies Administrators 48% Staffers 46% Garner public support for a policy issue Administrators 77% Staffers 48% Define a policy issue in a new way Administrators 69% Staffers 55% Make mid-course corrections to policies already enacted Administrators 48% 102

PAGE 119

Staffers 46% Guide the implementation of policies Administrators 68% Staffers 51% Staffers use research more to: Persuade policymakers to delay policy decisions Staffers-38% Administrators-35% Some of these findings contradict the hypothesis that administrators are less likely to use research for political purposes than staffers. I think the bigger picture this suggests is that administrators do get involved in politics and the process of convincing, recommending and garnering public support for policy issues. It suggests that the line between policy and politics is a very thin one-if it is there at all. Another interesting aspect of these results is that in general, administrators report a higher use of research in almost all areas as compared to staffers. This may suggest that administrators may be the policy analyst's main consumer of research. The main findings of my hypotheses suggest that the policy problem, policy goal and the political process are the stages of the policy process where staffers and administrators use research the most. Use of research during the implementation stage was not as prevalent as it was in other policy stages, even for administrators. Also, staffers and administrators do not necessarily differ in research use on the basis of administrators being more aligned with policy goals and staffers being more 103

PAGE 120

aligned with political goals. I found that administrators use research to further political goals as well. The following chapter discusses what I learned about my exploratory model through my research. It highlights the lessons I learned about my research design and the process of conducting research. In addition, it suggests areas for further research and development on the topic of policy research use by actors other then policymakers in the process. 104

PAGE 121

CHAPTERS CONCLUSION In Chapter One, I discussed four important issues facing the field of public affairs as a result of the past focus on policymaker use of research in the policy process. Because of this focus on instrumental use of policy research by policymakers, four outcomes have resulted in the field of public affairs: (1) policy researchers believe that many policy decisions are made and implemented in the absence of research; (2) the purpose and processes of social research have not changed or adapted in ways that might increase its integration into the larger policy process; (3) policy research is targeted mainly at policymakers even though they might not be the most important "users" and; (4) policy analysis" or policy research as a career or profession is in crisis about its relevance as a decision-makers' aid to effective policymaking. My research, through the research questions and exploratory model I created, was intended to bring greater clarity to these broad based issues with which our field struggles. Indeed, my research has provided important insights into these and the other research purposes that I set out to answer. This chapter highlights what I have learned about these four issues and discusses my overall conclusions about the research I conducted, and suggests areas for further investigation. 105

PAGE 122

Contributions to the Field of Public Affairs Are policy decisions made in the absence of research? The answer to this question, based on my findings, is, no, most certainly not. My research shows that policymaker staffers and midto high-level administrators use research all the time and that they consider it an integral part of their work. In 1992, David Kirp essentially dooms the field of policy analysis to complete irrelevance, saying that anecdotes, personal beliefs and media are the instruments that mainly shape policy decisions. However, my research finds that policy staffers and administrators use various types of research-including policy analysis in their daily jobs. They describe and report their use of research as directly informing the policy process-from the beginning in defining the problem, to designing policies as well as in the selection stage when they help policymakers examine and understand multiple options in the policy process. Overall, based on the interviews I conducted, this population believes the research they use gets plugged into the policy process they described all the ways in which they use research to inform the policy process but they are still leery about the actual decisions of policymakers themselves and whether the research is used by them. Staffers and administrators use research in drafting bills for legislators or in preparing legislative testimony on a certain policy issue. Many staffers and administrators I interviewed distinguished between the terms "research use" and "research influence." When I pressed some interviewees on whether they felt that research is used in policy decision-making, many said no, 106

PAGE 123

but they did feel that research was influential in the policy decision-making process. One administrator said, "Research frames policy decisions-it doesn't inform it." Overall, though, both the survey results and interview results make very clear that research is used in the policy process by staffers and administrators and it has at least some influence over policy decisions. Research use by this population is an area that requires further study in the field of public affairs. We now know that staffers and administrators are very important groups to consider when teaching public affairs to practitioners who will eventually produce research for use in the policy process. Have the purposes and processes of social research adapted in ways that will increase its impact? The answer to this question is not as clear from the results of my research. Even though my research questions were not specifically seeking information on whether the process of creating and disseminating social research has adapted to other conditions and policy actors (besides policymakers) in the policy process, we now know that it should. Similar to Nancy Shulock's (1999) desire to recalibrate policy analysis to be more aligned with the political and democratic process, my research has demonstrated that policy analysis should be recalibrated with other policy actors in mind. My research suggests there is a large audience for policy research, and yet policy analysts, university policy programs, government 107

PAGE 124

institutions and policy think tanks mostly produce research for policymakers. The past literature refers almost solely to policymakers' use of research and very little about how other actors use and influence the process with it (Hird, 2005; Kirp, 1992; Shulock, 1999; Weiss, 1977). Yet, theories on the policy process indicate there are other important actors in the policy process, and thus there might be other users of research to inform policymakers in the process. Now we know, at least with respect to policy staffers and administrators, this is the case. As a result, the field of public affairs should pay attention to the broader audience of policy research customers and tailor some research practices and purposes to that population. The concerns of this broader audience are reflected in the results of my interviews, in which staffers and administrators said time and time again that policy does not happen outside of a political context. They agree with Deborah Stone in that if policy analysts do not craft research to more accurately reflect the broader linkages between policy, political and public processes and the actors who work within that context, it might be or become irrelevant. Another conclusion related to this question is around the process of research creation and dissemination. The results of my interviews suggest that the easier research is to read, understand and access (via the web), the more it is used in the policy process. Most interview participants indicated they have a harder time incorporating "big research" (by this they meant academic research) into discussions with policymakers, policy briefs, testimony and memos because of access difficulties 108

PAGE 125

due to the high cost of academic journals. This does have implications for many policy researchers and the field in general. If our goal is for research to inform the public policy process to increase the public good, then we should do everything possible to make research easy to understand, actionable and free via the internet. What have we learned about the audience for policy research? As noted in the previous section, the audience for policy research is indeed much broader then just policymakers. In fact, staffers and administrators might use research in more purposeful and relevant ways. The key academic researchers on this topic-Carol Weiss, Nancy Shulock and John Hird-all focus on policymakers in their analyses and we clearly need to expand that audience to include others in the policy process. Much of what they learned in their research I corroborated in mine (use of research for enlightenment and rationalization purposes), but my analysis of a different group of consumers adds to the knowledge of who, when and why others consume policy research. Perhaps organizations, university public affairs programs and government agencies should consider creating more research networks, research reports, conferences and websites specifically for staffers and administrators. Policy research could be tailored to the needs and issues of staffers and administrators to increase its use on public policy issues. Clearly, because staffers and administrators report such a high use of research in meeting the goals and objectives of their jobs, 109

PAGE 126

the field of public affairs should support their roles in finding solutions to public policy issues. When examining how they conduct their jobs, it becomes clear that staffers and administrators have a large impact on policy development, formulation and implementation. Because of this, at least some research should be geared toward this type of audience and use. As a result, the type of research might look very different than simple policy recommendation options. An unexpected result of this research is that it highlighted a population of policymakers who indicate they use research a great deal, which is contrary to my assumptions and to the existing literature in the field on this topic. Indeed, I chose to study staffers' and administrators' use of research because of this assumption -I intended to find out who does use research. My research refutes one of the major concerns for the field of public affairs. I found that policymakers at least think they use research for many different reasons, including designing, creating and selecting public policies. This finding adds a new dimension to several studies and a major debate going on in this field that David Kirp frequently refers to in his articles and commentary, such as "When we 'speak truth to power,' does anyone listen?" and The End of Policy Analysis (Kirp, 2004, 1992). According to my results, policymakers do listen (even more so than the staffers and administrators in my study) when we "speak truth to power." The real learning for our field is that research is used frequently, by policymakers and other actors but probably in more 110

PAGE 127

complicated, nonlinear ways than as a direct influence on decision-making (though many instances point to this type of use as well). Are careers in policy analysis or research in crisis due to lack of relevance? One of the most interesting findings of my study is that previous research on "research use" may over inflate the "problem" that policy research is not adequately used by policymakers (Lindblom & Cohen, 1979; Kirp, 1992; White, 1994; Albaek, 1995; Stone, 1997; Shulock, 1999; Black, 1997; Hird, 2005). My findings suggest that research is, in fact, utilized by policymakers, though it may be inserted into the process (by staffers and administrators) long before policymakers use it in ajinal decision-making step (such as a vote). In other words, when considering policy options and policy design and drafting bills, policymakers do use research to inform those decisions. By the time a policy comes to vote, the research is already embedded in the crafting of the policy. When staffers and administrators were asked whether they believed that the "information and research they conduct gets used in the policy process," the majority (87.7%) said they either strongly agree or agree. And, from my interviews with staffers, it became clear that they are the architects of policy options, legislation and other components of the policy process. The research they use and gather is constantly reflected on, refined and integrated into policy language. By the end of the policy process, the research has already been used and reflected on in the previous stages of the process. 111

PAGE 128

As a conclusion then, practitioners of public policy do not need to be overly worried about the research they conduct getting "used" in the policy process. On the contrary, I found that research from policy-related organizations, think tanks and universities is a key source of information for staffers' work and administrators' influence in the design and implementation phases of the policy process. Even my unintended sample of policymakers reported a very high use of research higher than staffers and administrators in almost every respect. Still, a re-tooling of the purpose, format and processes for policy research should be considered, in light of this research, for the population of staffers and public administrators. One implication is that researchers who build relations with the pertinent "actors" in the process (such as staffers and administrators) might have more access and influence than researchers who remain separate trom these individuals as Heineman et al. suggested in 2002. These overarching questions facing the field of public affairs are important to revisit as new research emerges, the political process shifts and new actors and innovations emerge. My research has shed light on these questions and offered new directions to think about in the field of public affairs. Now I will review what I learned about my exploratory theory throughout this research process. The following sections highlight lessons learned and important next steps for research on this topic of study. 112

PAGE 129

Conclusions Regarding the Research Purpose The specific purpose of this research is to study policymaker staffers, midto high-level administrators and policy influencers to determine how they use policy research in their work. As noted above, better understanding the use of policy research by policy staffers and program administrators is important because, as the results of my work suggest, administrators have a good deal of discretion and interpretive power, both when they assist with and implement policy decisions. Consequently, policy researchers should have a clear understanding of how this population uses research. By focusing on the use of research by staffers and administrators, one learns of another viable way that policy research contributes to the policymaking process. My study also was intended to further the knowledge base on types of policy research uses and the influence and the relation of those uses to policy goals and political goals. This was the most difficult aspect of my research-distinguishing between policy and political goals-and how the use of research affected those goals. Conclusions Regarding the Exploratory Theory Given these outlined purposes of my research, what do the answers to my research questions say about the exploratory model and hypotheses within it that was put forth here? Again, the exploratory model is below. 113

PAGE 130

Policy actors interact with each other and with research in the following basic process. Within each area, specific research uses of the six highlighted in the literature review are more prevalent (hypotheses are in parentheses and if italicized then there was support for that hypotheses). Staffers (s) and administrators (a) are more active in certain parts of this process (as noted by the codes within the stages), suggesting their research use is more specific to certain types. (Enlightenment) (Instrumental) Policy Problem Definition (s & a) Information gathered Identified and defmed Placed on the agenda Policy Goal (a) Political Goal (s) Analysis Analysis -Policy options based on -Stakeholders evidence 1'\----U' -Policy options based Public discourse on constituent Mostly prioritized for policy satisfaction influencers Mostly prioritized for policymakers PoliticaV Democratic Process (s) Stakeholder input Information exchange Compromise Policy Change/ Adoption -Voting Executive order Ballot initiative Administrative discretion Policy Implementation (a) (Agenda setting) (Enlightenment) (Change Agent) Rationalization) (Tactical) (Rationalization) (Tactical) (Enlightenment) (Change Agent) (Instrumental) (Rationalization) (I nstrumentaf) (Enlightenment) Figure 2: Exploratory Theory on Research Use in the Policy Process by Staffers and Administrators: Aspects ofthe Model that are Supported 114

PAGE 131

Policy Problem Definition My study has shown that, in general, research is used by policy staffers and administrators most frequently to inform or raise awareness of policymakers, to analyze a particular policy issue or programs, to provide a rationale for certain policies and to provide needed information that lacks data. These uses for research equate most closely to enlightenment and rationalization (two types of research use highlighted in the six types put forth in this research). As described in the exploratory model, it is clear that research is most often used by this population within the policy problem definition stage to gather information about a policy issue requested by a policymaker or by an administrator who is trying to influence the policy agenda by proposing certain policy issues. In the exploratory model, I note that the type of policy research use labeled "change agent" might also come into use during this stage in the policy process. The data I gathered did not allow me to make any determinations about whether and how research was used for the change agent purpose. Partially this was due to an unclear definition of "change agent" and also, in retrospect, partially because my survey did not ask any questions that would allow the respondent to report this type of research use. Policy Goals and Political Goals In the next stage of the policy process as outlined in my exploratory model, I differentiated between policy goals and political goals, thinking that depending on 115

PAGE 132

which of these goals were held more strongly by the particular actor in the process, the type of research use would be different. In my exploratory theory, I included these goals as a distinct part of the policy process because I wanted to explore the idea that the type of goal that drives the majority of the decisions about a policy issue also will affect the type of policy analysis, development and public discourse on the issue. For example, in the analysis of policy options, if policy goals were driving decisions regarding a certain set of options, the analysis might be much different driven by data, research and implementation considerations. Whereas if the policy option were driven by political goals, then the analysis might be more focused on what constituents say about the issue or what information is provided by special interest groups. However, in this section of the exploratory model there were limitations in my research design and methods that made understanding this interaction between policy goals and political goals difficult. The distinction between policy goals and political goals proved to be very difficult to operationalize and subsequently measure with accuracy. Therefore, some aspects of my research questions and hypotheses are difficult to answer. Through the interviews with policy staffers and administrators I was able to have more dialog on the two types of goals and found that interviewees also had trouble distinguishing between the type of goals, and how holding one type or the other might affect their choice and use of research. This is likely the case because the distinction between policy and political goals is complex and not 116

PAGE 133

mutually exclusive. This made it very difficult to craft survey questions that reliably measured respondents' opinions, thus adequately measuring whether a person is more likely to use research to further a policy goal or a political one. Upon analysis of the survey data, it was nearly impossible to completely distinguish between the two types of goals and thus intent of research use in the different stages of the policy process. Nevertheless, the interviews did shed some light on this question, so I think there is some promise to study further how different types of goals affect the type of research used to achieve those goals. Partisan policy staffers were more comfortable aligning themselves with the political goals of their party (or the particular politician for whom they work). When I asked whether they thought their job was more aligned with policy or politics, partisan staffers were quick to talk about the political implications of their job. One partisan staffer said that she serves as "a liaison between the governor, government agencies and special interest groups" that pertain to subject areas that the governor cares about. Another partisan staffer said, "We are appointed by the governor so we are inherently political, we do the governor's work." However, partisan staffers also felt very sure that their goals and their jobs were policy related as well. They felt strongly about their responsibility as staffers to analyze policy to a high standard and provide all information, even if it wasn't particularly supportive of a policy that their political party would support. On the other hand, the administrators whom I interviewed were quick to identify their jobs 117

PAGE 134

as mostly aligned with policy. One high-level administrator answered this question with a point-blank: "My job is about policy and planning." Another administrator said, "I would say my job is more aligned toward policy goals, except that policy goals are heavily dependent on the politics." And one even said that his job was "not political at all," he was a budget analyst. Except for this last respondent, when pressed, all administrators and nonpartisan policy staffers said their jobs were both political and policy related and that sometimes it was hard to distinguish between when they were working solely on policy issues and finding the best possible policy solution or when they were being driven by political timing, goals or interests. Overall, the survey research found that administrators use research to provide a rationale for certain policies slightly more than staffers did (which could be interpreted as an alignment toward political goals), that staffers use research to persuade policymakers to delay policy decisions more then administrators (could be considered political), and that administrators use research more to gamer support for a policy issue then staffers (could be considered political). The survey research also found that administrators use research more to define a policy issue in a new way (which might be considered aligned with policy or political goals), and to make mid course corrections to policies that have been enacted (could be considered aligned with policy goals). Administrators also reported a higher frequency of research use to help guide the policy implementation process (considered a policy goal). Some of these findings, such as administrators reporting a high use of research to provide a 118

PAGE 135

rationale for policies and to gamer public support for an issue, contradict my exploratory theory, which hypothesizes that administrators are less likely to use research for political purposes than policymaker staffers. I think the bigger picture this suggests is that administrators do get involved in politics and the process of convincing, recommending, and garnering public support for policy issues. It also suggests that the line between policy and politics is a very thin one, if it is present at all. Political and Democratic Process The next stage in my exploratory model is the political and democratic process, where potential policies are put forth by committee, discussed and debated, and given the chance for public input. This stage entails an information exchange between different policymakers, policy actors, experts, staffers, administrators and the public that shapes and develops the final policies to be enacted. In this stage I learned that policy staffers and administrators use research a great deal to provide a rationale for certain policies, to provide needed information for a policy option that lacks data, to determine policy options and to inform policymakers as they decide on a policy. From the interview results I found that usually this state of the policy process happens behind the scenes, with a lot of phone calls made and emails sent before a legislator goes into a committee setting where there will be debates, testimony by experts and public input. However, many staffers mentioned that they 119

PAGE 136

are the ones who set up the committee meeting and thus they have influence on which experts testify and who is invited to the hearing. The exploratory theory posits that during this stage, research will be used for rationalization of policies and for tactical and informational reasons, as well as a change agent. The survey results and interviews confirm that research is used in all of these ways. One partisan statler explained the whole process like this: "We identified a [policy] need, then used data to justify a particular program, which turned into a policy through legislation, and then we will be using outcome data to determine whether we should continue that program or not." Again, the research use for the purpose of being a change agent on a specific policy topic was not well defined or measured in this research so this type of research use is unknown in this model. However, one partisan staffer said, "I definitely place more faith in certain research, and you know, tend to disregard certain conclusions based on who's done the research, to support [political beliefs] that are compatible with ours." This type of research use could be considered in the vein of wanting to create "change" through activism both politically and publicly on a policy issue. Stakeholder (public) input is a process that occurs in this stage. However, there was no mention of this in the interviews and no opportunity for it to be mentioned in the survey responses (except in some of the open-ended questions). One element of this research that could be explored further is how public input and 120

PAGE 137

interaction affects the use of research in the policy process. In fact, the public may be an interesting population to study in terms of if and how they use research or information to understand policy issues. Policy Adoption or Change The policy adoption stage is where a policy is put into law. It may take the form of a vote by a legislative or rulemaking body or a governor's executive order. It is in this stage that much of the previous research on policymakers' use of research has occurred to measure instrumental use of research, i.e., used for an actual decision on a policy option. As discussed in the literature review, much of the past research but not all (see Fitzpatrick, 2004) has suggested that this is not a high use of research for policymakers. Indeed, I did not expect a high reported use of research in this stage. However, the survey results did reveal a high frequency of use by policymaker staffers and administrators during this part of the process. They reported they use research either "frequently" or "very frequently" to "inform policymakers as they are deciding on a policy" (mean of 4.10 on a scale of 5, with 5 being the highest frequency). They also reported a high use of research as a "policy is being considered in the legislative process" (mean of 3.92 on a scale of 5). These results indicate that research might be used for instrumental purposes more then originally thought. 121

PAGE 138

Another type of research use that my theory suggests might be used in this stage would be rationalization. As mentioned previously, staffers and administrators reported a high use of research "to provide a rationale for certain policies" (mean of 4.31 on a scale of 5). This suggests that statiers and administrators might be more involved in this stage of the process-in terms of helping policymakers make political arguments related to the policy choices they will end up making than originally anticipated in my hypotheses. Policy Implementation The policy implementation stage is an area of the policy process that is ripe for more understanding through continued research. My intention in studying public administrators was to gain more insight on this stage of the process and how research is used in the implementation of policies. The survey results suggest that this population does use research for the purposes of informing the implementation process including: raising awareness or understanding of a policy issue with staff, helping policymakers consider the practical implications of different policy options, and providing policy implementation strategies. The results of the Chi square (listed below) that compared the way administrators and staffers responded to questions of implementation showed much higher levels of research use frequency with regard to implementation issues from the administrator perspective. This makes good sense as administrators are 122

PAGE 139

responsible for the implementation phase of the policy process. The following results highlight the areas related to implementation and indicate a higher use of research in these areas by administrators. Administrators use research more than staffers to: gamer public support for a policy issue Administrators 77% Staffers -48% make mid-course corrections to policies already enacted Administrators -48% Staffers -46% guide the implementation of policies Administrators 68% Staffers 51% Overall Reflections on My Exploratory Research Model The model that I put forward holds up well when assessing the various types of research use during each part ofthe policy process. Overall, I learned that the research use "types" that I delineated are most likely used at every stage in the process because each stage has its own miniature political process, but still some (the ones noted in the model) are used more then others. The model needs some additional research with respect to who uses research at what stage in the process because I found more crossover with administrators in the political process than I anticipated. In terms of future research, I am not sure my current model would be a good place to start. Though various parts of my model were supported by my research, I 123

PAGE 140

found that it did not highlight the differences in research use enough to be instructive to the field of public affairs. I learned that the policy process is too fluid to fit specific types of research use into certain stages of the process. The more useful part of my research refers to how the actors use research and how they described their use of research linking it to the policy stages might not be worth the time and effort. Some areas of substantial research use by staffers and administrators are worth additional discussion as they relate to the model. Staffers and administrators reportedly use research a lot to help identify potential problems with certain policy options as well as to provide policymakers with policy options from which to choose. In addition, many staffers and administrators report research is used when a policy is being considered within the legislative process (such as testimonies and debates) to identify problems or criticisms. Taken together, these findings suggest that research is used substantially to identify options and highlight potential problems before policies are selected. The identification of potential problems with policies might be considered a new way research is used in the process one not yet highlighted in this model. Also interesting, a lot of staffers and administrators use research to inform their own staff about policy issues, implying that an important part of policy research is to generate understanding among everyone in the policy process -especially those involved in the discussion and implementation of policy. One type of research use for which respondents reported a low frequency of use, as compared to the other types, was using research to inform the implementation of 124

PAGE 141

policies. I expected a higher use of research for this purpose because of the nature of administrator jobs as mainly executing policy decisions. What was learned at each stage in the model is important because it is likely able to be generalized to other areas of public policy besides education. Many policy staffers in my data set serve policymakers who are responsible for making decisions on policy issues in areas other than education and there is no reason to think that their use of research would be too different. The administrators in my sample work primarily in the field of education. However, I believe my findings are applicable to other areas as well because, in general, administrators are tasked with implementing policies and they face similar responsibilities in other policy fields when implementing and influencing policy directives. Likely, they conduct research and look for program examples that assist their implementation decisions as do education administrators. There might be less room to generalize from my sample to policy areas that are highly technical or scientifically based such as climate change, water use or city planning policies. I would expect these policy fields to be more heavily reliant on research for instrumental policy decisions. When considering the ability to generalize my study population of policy staffers and administrators to other levels of the policy system, such as federal and local, I think some important differences in research use would emerge. One hypothesis would be that federal policyrnaker staffers are more political and thus use policy research much more for tactical and rationalization purposes. I think the finds 125

PAGE 142

of Weiss and Bucuvalas ( 1980) would be upheld: research that supports a previously held belief would be more apt to be used and touted as sound research. At the local level, I think, data might be used more frequently than research. For example, cities and municipalities trying to make the case for a new program or policy would need to have the data from their community to support the need for that new program. In this case, program evaluation research might also be used more at the local level. In general, I think agenda setting, information gathering, rationalization and instrumental use would happen in basically similar patters at the federal, state and local levels. Research Limitations Throughout the process of designing and conducting original research on how policymaker staffers and administrators use research in the policy process, I have learned how this research could be improved. As I went through the process of developing my survey and analyzing the data, I recognized some limitations in my research design and methods that made understanding the interactions with my exploratory model problematic. The most difficult aspect of this research was the lack of clarity regarding the operational definitions and differentiation between the stages of the policy process and types of research use as outlined in my theoretical model. One of the preeminent researchers on this topic notes the same problem stating, '"It is exceedingly unclear 126

PAGE 143

what constitutes a 'use' and there are a thousand questions one can ask when trying to decipher between various uses" (Weiss, 1977, p. 213). This lack of clarity made it problematic to develop a measurement tool that would accurately reflect the research uses of staffers and administrators within each part of the policy process. For example, when I categorized the answers to the survey questions, I found (without surprise due to the non-linearity of the policy process) multiple overlaps of the policy and research use categories, so it was not possible to make a sound judgment about the intent of the survey respondent. Specific examples include the relatively close definitional nature of some of the research use types such as "agenda setting" and "change agents" and the potential crossover between "instrumental" and "tactical" as well as "change agents," "tactical" and "rationalization." Though differences exist between these types of research, I believe some could be masked by the bluntness of a survey instrument that cannot explain or define all the particular nuances of the terms. It was also difficult to distinguish between the policy problem, policy goal and political goal stages of the policy process; this difficulty is supported by research on the policy process, which indicates it is not a linear process. I was unable to compare the stages of the policy process and which ones used research "more" or "less" because the survey question variable could be categorized in multiple stages of the policy process. Survey question three, for example, asked how frequently respondents use policy research to provide a rationale for certain policies. Results indicated very high frequency, at a mean of 4.31 (between "very frequently" and 127

PAGE 144

"frequently"), but could be categorized as using research in the policy goal, political goal or democratic process stage. Another weakness of the study design was the distinction between "data driven" and "research driven" in policy development. Many interviewees felt policies were created because of what the data in their state illustrated. They reflected that a large part of their job was to put data in front of legislators to paint a picture of policy problems to address. The research was used within the context of that data, but the data was the driver of decision-making, not necessarily the research. In my research design, I did not define research terms specifically enough to make the distinction between data and research, partially because the survey questions would have to be asked twice for each version (data use and research use). However, according to Lindblom and Cohen's definition of"professional social inquiry," all types of information are included-both data and research-and should be used to inform the policy process and the decisions that come from that process (1979). In future research, I would pilot the survey to a larger sample and have multiple focus groups to determine the best way to communicate the terms regarding the types of research use and policy stages. I would then run a reliability analysis and factor analysis to test my conceptual connections between the definitions I intended and the survey data results. At that point I would have made adjustments to the survey instrument. This may have helped sharpen the focus of the definitions, 128

PAGE 145

but I think it would still be difficult to eradicate this problem all together. One of the challenges of research in this area is the fact that the policy stages are not mutually exclusive. In addition to gaining more clarity around my variable definitions, I would have added a section to the survey that measures the extent to which research is used at all by this population. My survey instrument was biased by the assumption that this population did use research in their jobs. In retrospect, I would include questions on the extent to which research is used at all. For example, ''Do you use research daily? Multiple times a day? Weekly? Monthly?" This line of questioning would allow me to test my assumption that research use is already present in this population. There were also obvious limitations ofthe statistical power of my study given my relatively small sample size. Because of this, the ability to generalize my research results to other staffers and administrators might be questionable. Another important research limitation to consider is the potential bias of my sample. Previously explained in the methods section, the sample might create bias in two ways. First, because this sample was already in the ECS database, it is likely these individuals already have been involved with ECS in some capacity seeking policy information, participating in a conference or requesting a newsletter. This means the sample already has a bias toward research use in some form or another and might not be representative of other staffers and administrators in the field. A second bias 129

PAGE 146

exists because some of these staffers serve solely in the field of education policy. Despite these biases, however, I do not believe they compromise the integrity or the validity of this study. And though this sample may have an inclination toward research use, no more so than participating in a professional association or other professional membership organization. As this study was mostly descriptive and addressed the population of policy staffers and administrators, the five research questions I posited were able to be answered in an informative way. However, the next phase of research should identitY a testable model, definitions and plausible relationships to be explored. For this type of research work the survey instrument would need to be refined, the operational definitions of the variables clarified and the sample size increased to gain the needed statistical power to examine relationships between variables and groups. That is, the next phase of research would be the more refined with specific testing of the exploratory hypotheses that I put forth. The Next Step: Further Research on Research Use Based on the descriptive information I gathered as well as what I have learned on research design, the following section discusses implications for further research. When I originally designed this research, I considered staffers and administrators to be somewhat similar in terms of the way they use research. I was 130

PAGE 147

especially interested in investigating how policy actors who support, work for, or influence policymakers use research. In this respect, I grouped policy staffers and high-level administrators together to learn how this population as a whole uses research. In the next phase of research, I would do more to separate these groups, increase the sample size of each, and learn more about the ditTerences between them. My results hinted at interesting differences between how staffers and administrators use research, though not always in ways I anticipated. For example, administrators indicated a higher use of research in political ways than I expected. In future research I would like to investigate this finding more thoroughly. In addition :o distinguishing more between staffers and administrators, I would also want to look more closely at the differences between partisan and nonpartisan staffers. In this study I did not have a large enough sample size to differentiate between these groups. In the interviews, however, I did hear distinct differences in how partisan and nonpartisan staffers answered certain questions on research use. The interviews with staffers and administrators provided a great deal of nuanced information that gave me ideas for future research. One is the need to distinguish between using "data" or "research" to inform policy decisions. Several policymakers and administrators said their legislature and governor's office is very "data driven" in decision-making, but when asked if they were research-driven, they said no, not as much. This is an important distinction of information use to investigate. Additional research questions to explore might be, "What is considered 131

PAGE 148

data? What is considered research? Why are policymaking bodies data driven (if, indeed they are) rather then research driven?" Another future research strand should be on the nature of"instrumental" or "direct" research use. This type of research use seems to be the type that people want to "see" in policy development. But given my research, I think it is far too limiting a term when considering all the ways in which research is used in the policy process. Why do we care so much if research is used in a direct way? Doesn't the focus on this distract from all the other important ways that research is used in the process? It is intuitive that use of information is not clear-cut in the policy process. Democracy in the United States is based on the premise of discussion, debate and multiple influences within the policy process, therefore, I do not think we should measure "research use" only in the manner of direct use. My data suggest that research is used in multiple ways-including in a direct, instrumental way -to directly inform decisions, directly design policies, and directly influence the decision-making process, so dissecting this term to craft a more comprehensive definition would be helpful. Based on what I have learned from my work in this study, I would investigate other hypotheses in addition to reassessing the ones I put forth in this research: HI: Administrators use research in tactical ways to persuade policymakers toward certain policy options and issues. 132

PAGE 149

H2: Policy staffers use research that is most readily available to them through websites and policy think tanks that disseminate research in accessible, clear and frequent ways. H3: Administrators are involved in using research in the political process of developing, and shaping policy in the policy definition and development stages. H4: Legislators will receive policy information based on different types of research (some more biased then others) depending on whether their state has a nonpartisan policy research staff supporting legislators (as opposed to only partisan staffers). Summary Through the findings of this mixed-method study, policy analysts are better able to integrate policy research into the political process by understanding the way it is offered and used by policy staffers and administrators. Policy analysts in government, policy organizations, think tanks and universities should strongly consider developing a relationship with policy staffers and public administrators in their field. By doing so, they will gain information and insight into the types of research needed and will have a powerful avenue for disseminating research for use in the policy process. A quick summary of the answers to my research questions and hypotheses follows. 133

PAGE 150

I. To what extent do policymaker staffs and mid-to high-level administrators use policy research? To a large extent! This population reported most frequent use related to enlightenment, agenda setting, rationalization and instrumental use. They reported most frequent use during the policy problem definition, policy goal and politicaVdemocratic process stage. 2. Considering the six types of policy research use expressed in this research, what types are most common? Enlightenment, agenda setting, rationalization and instrumental. 3. When policy research is used, what is the state of policy process and the type of research use? Respondents reported use at all stages ofthe policy process: policy problem definition (4.09 average frequency of use on a scale of 1-5 (most))(agenda setting and enlightenment), policy goal (4.07 average frequency of use) and political goal (3. 77 average frequency of use) (enlightenment, rationalization and possible tactical), implementation (3.64 average frequency of use) (enlightenment, instrumental, rationalization), political/democratic process ( 4.07 average frequency of use) (rationalization, enlightenment and instrumental), policy change (3.83 average frequency of use) (rationalization, enlightenment, instrumental and tactical). 134

PAGE 151

4. How do staffers, administrators and other influencers get access to policy research? From most frequently reported method to least: organizational web sites, government websites, conversations with colleagues, meetings, practitioner journals, professional association web sites, conferences, reports compiled by staff, academic journals, newspapers, university web sites, memos from staff or policymakers, think tank websites, magazines, political think tank websites, Lexis-Nexis or Westlaw. 5. How do policymaker staffers and midto high-level administrators differ in their use of research? Administrators use research more to (percentage frequencies are based on Chi squares results reported earlier in this chapter): provide a rationale for policies (administrators-48%, staffers-46%); gamer public support for a policy issue (administrators-77%, staffers-48%); define a policy issue in a new way (administrators-69%, staffers-55%); make mid-course corrections to policies already enacted (administrators-48%, staffers-46%); and to guide the implementation of policies (administrators-68%, staffers-51%). Staffers use research more to persuade policymakers to delay policy decisions (staffers-38%, administrators-35%). Hypotheses Hl: Policy staffers and midto high-level administrators utilize research in different ways depending on the stage of the policy process as outlined in Figure 2. 135

PAGE 152

This hypothesis is supported as the results suggest that staffers and administrators do get involved in differing stages of the process and use research more frequently depending on the different stages. The stages where staffers and administrators appear to use research the most are the policy problem, policy goal and political and democratic process stages. H2: Policy staffers and midto high-level administrators are more likely to use research at the policy problem, policy goal and implementation stages than at the political goal, democratic process or policy change stages. Inconclusive due to problems with exclusively defining policy goals and political goals. However, there was support in the data that suggested this population uses research more during the policy problem and policy goal stages yet contrary evidence regarding the frequency of use of research at the implementation and democratic process stage. H3: Policy staffers and midto high-level administrators are more likely to use policy research with respect to reaching policy goals as opposed to political goals. Inconclusive due to problems defining policy goals and political goals. However, elements of both survey and interview data from policymaker staffers and administrators support this hypothesis. H4: Policy staffers and midto high-level administrators will access research via newer technologies (such as the internet). 136

PAGE 153

Supported, but due to the lack of comparative data regarding how they accessed data previously, there is no way to describe differences or changes over time. HS: When comparing groups, policy staffers will be more closely aligned with political goals than mid-to high-level administrators. Inconclusive due to problems defining policy goals and political goals. However, quantitative survey data and interview data from policymaker staff and administrators support the rejection of this hypothesis because administrators were found to be closely aligned with political goals. Given the results ofthis research it is imperative that public policy and public affairs programs consider the specific population of policy staffers and administrators in the research process. They are key actors in the entire policy process and providing them with needed research on public policy issues in ways that help them achieve their goals will do much to serve the public good. This means that the field of public affairs should study the job functions, roles and responsibilities of staffers and administrators more deeply to find out what research might serve them better in their quest to improve the policies that are delivered out of the policy process. In addition, this research suggests that the line between policy and politics is very blurry. Administrators are very involved in using research to shape the political debate and policymakers use research in this manner very 137

PAGE 154

frequently as well. Students and professionals that conduct policy analysis must understand that their research will be used for these political purposes-as well as for policy purposes. Overall, my research is good news for public affairs students in that it supports the idea that those who produce policy research and analysis should recognize that their research is used (by policymakers too!). Yet, we should all recalibrate our expectations for how it is used to understand that it will be incorporated into the policy process largely as a defining and conceptual tool for understanding policy problems, as well as used as a tool in the democratic process of debate and rationalization of differing policy solutions. 138

PAGE 155

APPENDIX A Policy Research Use Survey Policy Research Use Survey We need you! You play a critical role in shaping education policy. Because of that role, I am requesting your feedback and opinions in a research study being conducted as part of my dissertation. This study is partially supported by The Education Commission of the States. Only you can help me answer these questions! The purpose of this research project is to understand policy research usage in the policy process. The survey contains questions about the type of job you have and how you use research in your job. It should take about 15 minutes to complete. The information we glean from the survey results will help ECS, and organizations like it, create a line of products and services that will better help you carry out your role in the education policy process. Research Use For the purposes of this survey "policy research" or "analysis" is defined as: Information that is gathered, analyzed, organized and communicated to help solve public problems. How much do you agree with the following statement? 1. Policy decisions are often made without policy analysis or research. Strongly A Neither Agree Disagree Strongly Agree gree or Disagree Disagree 139

PAGE 156

How do you most often use policy research and/or information? Again, for the purposes of this survey "policy research" is defined as: Information that is gathered, analyzed, organized and communicated to help solve public problems. 2. Do you use policy research/information for analysis of a particular policy issue or program? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y 3. Do you use research/information to provide a rationale for certain policies? Very Somewhat Frequently frequently Occasionally Rarely Never 4. Do you use research/information to develop policy (for example, before a specific policy is ever created or considered)? Very Somewhat R 1 Frequently frequently Occasionally are Y Never ..;' 5. Do you use research/information to inform or raise awareness of policymakers? Very Somewhat Frequently frequently Occasionally Rarely Never 140

PAGE 157

How do you most often use policy research and/or information? (continued) 6. Do you use research/information to provide needed information for an issue that lacks data? Very Somewhat Occasionally Rarely Never Frequently frequently 7. Do you use research/information to persuade policymakers to delay policy decisions? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y ... i 8. Do you use research/information to garner public support for an issue? Very Somewhat F tl f tl Occasionally Rarely Never requen y requen y 9. Do you use research/information to raise awareness or understanding of a policy issue with your staff? Very Somewhat Frequently frequently Occasionally Rarely Never 141

PAGE 158

How do you most often use policy research and/or information? (continued) 10. Do you use research/information to understand or define a policy issue in a new way? Very Somewhat Occasionally Rarely Never Frequently frequently 11. Do you use research/information to provide policy options to policymakers? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y .I 12. Do you use research/information to identify potential problems with certain policy options? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y 13. Do you use research/information to help policymakers avoid making decisions on political whims? Very Somewhat Frequently frequently Occasionally Rarely Never 142

PAGE 159

How do you most often use policy research and/or information? (continued) 14. Do you use research/information as a policy is being considered (for example, in legislative sessions, testimonies or legislative debates)? Very Somewhat F tl f tl Occasionally Rarely Never requen y requen y 15. Do you use research/information to inform policymakers when they are deciding on a policy? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y 16. Do you use research/information after a policy is enacted (for example, to troubleshoot and/or make midcourse corrections to an existing policy)? Very Somewhat 0 11 Rarel N Frequently frequently ccaslona Y Y ever 17. Do you use research/information to help policymakers consider the practical implications of different policy options? Very Somewhat Frequently frequently Occasionally Rarely Never 143

PAGE 160

How do you most often use policy research and/or information? (continued) 18. Do you use research/information as a rationale for a policy implementation strategy? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y 19. Do you use research/information to help guide the implementation process of policies (for example, in selecting certain programs to implement policy directives)? Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y 20. Comments or other ways that you use research and information? Your Job and Use of Research 21. Can you think of a particular piece of research or report you used in the last month? Please describe it and how you used it 144

PAGE 161

Acquiring Research -Web/Internet In the last three months, what were the most common ways in which you acquired new research/data/analysis/policy information? 22. How often do you use: Government Web sites (such as U.S. Department of Education or state government sites) Very Somewhat Occasionally Rarely Never Frequently frequently Policy Organization Web sites (for example, but not limited to National Conference of State Legislatures, Education Commission of the States, National Governor's Association) Very Somewhat Frequently frequently Occasionally Rarely Never Think tank Web sites (for example, but not limited to RAND and Brookings) Very Somewhat Frequently frequently Occasionally Rarely Never Political think tank Web sites (for example, but not limited to, Heritage Foundation, Fordham Foundation, Progressive Policy Institute) Very Somewhat 0 11 Rarel N Frequently frequently ccaslona Y Y ever 145

PAGE 162

Acquiring Research -Web/Internet (continued) 22. How often do you use (continued): Professional Association Web sites Very Somewhat Frequently frequently Occasionally Rarely Never University Web sites (for example, but not limited to, university policy centers and research divisions) Very Somewhat 0 11 R 1 Never Frequently frequently ccaslona Y are Y j J j J Lexis Nexis or Westlaw Web sites Very Somewhat Occasionally Rarely Never Frequently frequently ,; Jl' ,.J' ..1 23. Are there other Web sites you use? Please describe: 146

PAGE 163

Acquiring Research Publications Now that we have determined how you use the internet to acquire research, through what other venues do you acquire research? 24. How often do you get research/information from the following sources? Journals (academic) Very Somewhat Frequently frequently Occasionally Rarely Never "" Journals (practitioner or association publications) Very Somewhat Frequently frequently Occasionally Rarely Never Magazines (Time, Newsweek, The Economist, etc.) Very Somewhat 0 11 R 1 Frequently frequently ccaslona Y are Y Never Newspapers Very Somewhat Occasionally Rarely Never Frequently frequently ./ ,.1 ..1 ""' J Organization reports or publications Very Somewhat Occasionally Rarely Never Frequently frequently _, ...; .. i j 147

PAGE 164

Acquiring Research Other Ways? In addition to Web sites and other publications that you use, how else to you get access to research? 25. Do you get access to research in any of the following ways? Reports or publications compiled by your staff Very Somewhat 0 11 Rarel Frequently frequently ccaslona Y Y Never Conversations with colleagues Very Somewhat 0 11 Frequently frequently ccaslona Y Rarely Never Meetings (internally in your organization or externally) Very Somewhat 0 11 Rarel Frequently frequently ccaslona Y Y Never Conferences (such as national gatherings of professionals or on specific policy topics Very Somewhat Frequently frequently Occasionally Rarely Never .;i Memos from policymakers or other staff Very Somewhat Occasionally Frequently frequently Rarely Never 148

PAGE 165

26. Other (describe) The Policy and Political Process How much do you agree with the following statements? 27. You feel a lot of pressure in your job to provide information that supports certain policies. Strongly Agree Neither Agree Disagree Agree or Disagree .) _, .,.;' Strongly Disagree 28. Implementation decisions often are made without policy analysis or research. Strongly Agree Agree Neither Agree or Disagree Disagree Strongly Disagree 29. Debates and discussions about policy options often occur without research or information to inform the conversation. Strongly Agree Neither Agree Disagree Agree or Disagree 149 Strongly Disagree

PAGE 166

The Policy and Political Process (continued) 30. A large part of my job is geared toward satisfying a constituent or customer base. Strongly Agree Agree Neither Agree or Disagree Disagree Strongly Disagree 31. A large part of my job is geared toward finding and creating policies that solve public education issues. Strongly A ree Neither Agree Disagree Agree g or Disagree Strongly Disagree 32. The information and research I conduct or gather gets used in some way in the policy process (the policy process may include but is not limited to defining, debating, selecting and implementing public policy). Strongly Agree Neither Agree Disagree Strongly Agree or Disagree Disagree 33. Your job is political or highly influenced by partisan interests. Strongly Ag e Neither Agree Disagree Strongly Agree r e or Disagree Disagree .. i 150

PAGE 167

Research Use in the Policy Process 34. If the research you conduct or gather gets used in the policy process, it most often gets used at the following points: Defining the policy problem Strongly Agree Neither Agree Disagree Strongly Agree or Disagree Disagree .,., ..I ..1 .,/ ..I Garnering support for a policy solution Strongly Agree Neither Agree Disagree Strongly Agree or Disagree Disagree ..1 .,; ..1 ..1 ..,; Determining policy options Strongly Agree Neither Agree Disagree Strongly Agree or Disagree Disagree ..1 ., "" Selecting policies Strongly Agree Neither Agree Disagree Strongly Agree or Disagree Disagree .., ..,-..,; J .,.; 151

PAGE 168

Research Use in the Policy Process (continued) 34. If the research you conduct or gather gets used in the policy process, it most often gets used at the following points (continued): Altering existing policies Strongly Agree Agree Neither Agree or Disagree Disagree Strongly Disagree 35. If the research you conduct or gather gets used in the policy process in other ways, not reflected in the question above, please explain: You and Your Job Please tell me a little bit about yourself and your job. You are almost done! 36. Your job title 37. Male/Female Male Female 38. Year of Birth 152

PAGE 169

39. Highest degree earned .J High school completion ..1 Less than bachelors ..1 Graduate degree Doctorate .i Other (please specify) 40. What are the responsibilities of your job? (check all that apply) Deciding on state level policies Implementing state level public policies Informing state level policy decisions Creating/drafting state level policies Providing guidance to elected or appointed officials Managing people that implement state policies Conducting research that informs state policies Providing research to inform state policies Other (please specify) Are you a state policymaker? Yes No Please specify 153

PAGE 170

42. Do you work for or directly with an elected official or a group of elected officials? Yes No 43. Would you be willing to participate in an interview to explore these issues deeper? If so, please enter your e-mail address and contact information in the box below. By doing so, your name can be connected to your survey results, however the information will still remain confidential. Thank youl Thank you so much for your time, I hope the results of this survey help policy analysts and policy organizations provide more useful research that can be embedded in the policy process! For more information, contact: Katy Anthes at kanthes@ecs.org or kanthes10@yahoo.com 154

PAGE 171

APPENDIXB Focus Group Interview Protocol Focus Group conducted November 1, 2005 in Washington D.C. Topic: Research and the Policy Process Participants: Lois Adams-Rodgers, Council of Chief State School Officers (CCSSO), former Deputy State Superintendent, Kentucky Joe Simpson, Council of Chief State School officers, Former Deputy State Superintendent, Wyoming Mariana Haynes, National Association of State Boards of Education (NASBE) Arika Long, Education Commission of the States (ECS) Charlie Toulmin, National Governors Association (NGA) Dane Linn, NGA Piedad Robertson, ECS Brenda Welburn, NASBE David Griffith, NASBE Jane Best, National Conference of State Legislatures (NCSL) Melissa Johnston, CCSSO Julie Bell, NCSL 155

PAGE 172

Protocol: I used open-ended, semi-structured questions consisting of the questions below. In addition, I probed deeper on certain elements of the questions when respondents touched on something especially pertinent to my research or if I misunderstood the intent of their comments. 1. How do you use research in your jobs? 2. How do you use information and pass it on to others? 3. What type of information do you use? 4. How do you present that information to others? 5. What types of research do you use? 6. What is the source of your information? 156

PAGE 173

APPENDIXC Interview Protocol 1. How do you use research/analysis and information in your job? 2. How do you use information and pass it on to others? 2.1. Whom do you pass it on to? 3. For what purposes do you mostly use information/research/analysis? 3.1 Agenda setting, instrumental, enlightenment/informational, change agent, tactical, rationalization 4. \\'hen in the policy process do you use research? When do you deliver research information? 4.1 : Policy problem formation, policy goal, political goal (garnering support for a policy solution), determining policy options, selecting policies, altering (or evaluating existing policies) 4.2: probe about the policy goal vs. political goal and the differences between them 5. How do you present that information to others? 6. What is the source of your information? 6.1 Internet, library, organization, conferences, meetings, phone calls, etc. Prompt to see if it is partisan, from think tanks, university-based, etc. 7. Do you consider your job policy-related or political? 7.1 Find out if they are closer to a staffer or administrator 7.2 What are the goals of your job? 157

PAGE 174

APPENDIX D Human Subjects Approval Letter And Human Subjects Consent Letter 158

PAGE 175

University of Colorado at Denver and Health Sciences Center Human Subjects Research Committee InstitutiOnal Review Board Downtown Denver Campus Box 120, P.O. Box 173364 Denver, Colorado 80217-3364 Phone: 303-556-4060, Fax: 303-556-5855 UATE: TO: FROM: January 20. 2006 Catherine Anthes It\ J\ Dorothy Yatcs.IISRC St..:BJI':CT: lluman Subjects Research #2006-065 An Exploratory Study of II ow Policy Rl!search is in the Policy Process by Policy-Maker Sta1T & Administrators Your protocol has been approved us non-exempt. This approval is good for up to one year !'rom this date. Your responsibilities as a researcher include: If you make changes to yuur research protocol or design you should contact the IISRC. You are responsible for maintaining all documentation of consent. Unless specitied differently in your protocol, all data and consents should be maintaim:d for tlm:e ye:.trs. If you should encounter adverse human subjects issues, please contact us immediately. If your research continues beyond one year from the above date. contact the liSRC lor an extension. The IISRC may audit your at any time. Good Luck with your research. 159

PAGE 176

April 24, 2006 Dear : You play a critical role in shaping education policy. Because of that, the Education Commission of the States needs a little help in getting your important feedback in a research study. You are the one who knows the answers to these questions! The purpose of this research project is to understand policy research usage in the policy process. Your information will help ECS improve the types and formats of information that we give to you! I hope you'll take a few minutes to till out the survey. The survey will last about 1 5 minutes. It contains questions about the type of job you have and how you use research in your job. I hope that in taking this survey you might receive benefits from reflecting on your methods of using information and thus consider new ways to use research in your job. Results of survey will be summarized in my dissertation and possibly in other publication formats. However, no one's identity will be revealed. Any information provided by you in the survey will be afforded the professional standards for protection of confidentiality. Proper university research protocols require that I inform you of any risks in participating in this research. There is a slight risk for minor discomfort or embarrassment if confidentiality should be accidentally breached. Your participation is entirely voluntary. You will suffer no penalty if you choose not to participate. This study is my dissertation at the Graduate School of Public AtTairs at the University of Colorado at Denver and is taking place from January through December 2006. If you have any questions about the research, you may contact Katy Anthes at the Education Commission of the States, at (303) 299-3635 or kanthes@ecs.org. If you have any questions about your rights as a research subject contact the Human Subjects Research Committee Administrator, 1380 Lawrence Street, Suite 1425, or at (303) 556-4060. 160

PAGE 177

By completing this Web-based survey, you are consenting to the terms of this research stated above. This notice serves as your copy of the consent agreement. If you can take the survey, please enter this link into your web browser and follow the instructions! Thank you for your valuable help! If you would like to receive a copy of the research results, please send me an e-mail. Again, many thanks in advance! Sincerely, Katy Anthes Principal Investigator 161

PAGE 178

BIBLIOGRAPHY Albaek, E. (1995). Between knowledge and power: utilization of social science in public policy Making. Policy Sciences, 28, 79-100. Axelrod, R. M. (1976). Structure of Decision: The Cognitive Maps of Political Elites. Princeton, NJ: Princeton University Press. Backer, T., Salasin, S., & Rich, R. (1991). Federal policy and knowledge utilization. Knowledge: Creation, Diffusion, Utilization, 12, 220-224. Bardach, E. ( 1984 ). The dissemination of policy research to policy makers. Knowledge: Creation, Diffusion, Utilization, 6, 125-144. Bardach, E. (2000). A Practical Guide for Policy Analysis. New York: Chatham House Publishers. Baumgartner, F. R., & Jones, B. D. (1993). Agendas and Instability in American Politics. Chicago: University of Chicago Press. Beckman, N. ( 1977). Policy analysis in government: alternatives to "muddling through." Public Administration Review, Symposium Issue, May/June 1977, 221-244. Bernard, H. R. (2000). Social Research Methods. Thousand Oaks: Sage Publications. Birch, S. (1997). As a matter of fact: evidence-based decisionmaking unplugged. Health Economics, 6, 547-559. Birkland, T. (2001). An Introduction to the Policy Process. Armonk, New York: M.E. Sharpe. Black, N. (2001). Evidence based policy: proceed with care. Department of Public Health and Policy, London School Of Hygiene and Tropical Afedicine, 323 (August 2001), 275-280. Bobrow, D. B., & Dryzek, J. S. (1987). Policy Analysis By Design. Pittsburgh: University of Pennsylvania. 162

PAGE 179

Booth, T. (1990). Researching policy analysis. Knowledge: Creation, Diffusion, Utilization, 12, 80-100. Busenberg, G. (2001). Learning in organizations and public policy. Journal of Public Policy, 21 (2), 173-189. Caplan, N. (1975). Social research and national policy: what gets used, by whom, for what purposes and with what effects? International Social Sciences Journal, 28, 187-194. Caracelli, V. J. (2000). Evaluation use at the threshold ofthe twenty-first century. New Directions for Evaluation, Winter 2000 (88), 99-111. Chalmers, I. (2003). Trying to do more harm than good in policy and practice: the role of rigorous, transparent, up-to-date evaluations. The Annuals of the American Academy, 589 (September 2003), 22-40. Chelimsky, E. ( 1991 ). On the social sciences contribution to government decision making. Sciences, 254, 226-231. Clark, J., & Little, T. (2002). National organizations as sources of information to state legislative leaders. State and Local Government Review, 34, 38-44. Coleman, M., & Briggs, A. (2002). Research Methods in Education Leadership and Management, London: Paul Chapman Publishing, Ltd. Davidson, R., & White, P. (1988). Information and Government: Studies in the Dynamics of Policy-Making. Edinburgh: Edinburgh University Press. deLeon, P. (1988). Advice and Consent: The Development of the Policy Sciences. New York: Russell Sage Foundation. deLeon, P. (1997). Democracy and the Policy Sciences. Albany: State University ofNew York Press. de Leon, P. ( 1999). The missing link revisited: contemporary implementation research. Policy Studies Review, 1999 (Fall/Winter), 312-338. Dillman, D. (2000). Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley and Sons, Inc. 163

PAGE 180

Dreyfus, D. ( 1977). The limitations of policy research in congressional decision making. In C. Weiss (Ed.), Using Social Research in Public Policy Making. Lexington, MA: Lexington Books. Dunn, W. (1994). Public Policy Analysis: An Introduction. Englewood Cliffs, NJ: Prentice Hall. Education Commission of the States. (200 1 ). No Child Left Behind: Report to the States. Denver, CO: Author. Engelbert, E. ( 1977). University education for public policy analysis. Public Administration Review, 1977 (May/June), 228-236. Feldman, M., & March, J. ( 1981 ). Information as signal and symbol. Administrative Science Quarterly, 26, 171-186. Fitzpatrick, J. (2004). Exemplars as case studies: reflections on the links between theory, practice, and context. American Journal of Evaluation, 25 ( 4 ), 541-559. Frye, A. ( 1976). Congressional politics and policy analysis: bridging the gap. Policy Analysis, 2, 265-282. Ginsberg, A., & Rhett, N. (2003). Building a better body of evidence: new opportunities to strengthen evaluation utilization. American Journal of Evaluation, 24 (4), 489-498. Goldman, A., & Rojot, J. (2003). Negotiation: Theory and Practice. Norwell, MA: Kluwer Law International. Grasso, P. (2003). What makes evaluation useful? Reflections from experience in large organizations. American Journal of Evaluation, 24 (4), 507-514. Grenzke, J. M. (1982). Influence, Change and the Legislative Process. Westport, CT: Greenwood Press. Grob, G. F. (2003). A truly useful bat is one found in the hands of a slugger. American Journal of Evaluation, 24 (4), 499-505. Hanney, S., Gonzalez-Block, M., Buxton, M., & Kogan, M. (2003). The utilisation of health research in policy-making: concepts, examples and methods of 164

PAGE 181

assessment. Health Research Policy and Systems, 1:2, Health Economics Research Group, Brunei University, Uxbridge, Middlesex. Heineman, R. A., Bluhm, W. T., Peterson, S. A., & Kearny, E. N. (Eds.). (2002). The World of the Policy Analyst: Rationality, Values and Politics. New York: Chatham House Publishers. Henry, G. T. (2000). Why not use? New Directions for Evaluation, Winter 2000 (88), 85-98. Henry, G. T. (2003). Influential evaluations. American Journal of Evaluation, 24 (4), 515-524. Henry, G. T., & Mark, M. M. (2003). Beyond use: understanding evaluation's influence on attitudes and actions. American Journal of Evaluation, 24 (3), 293-314. Hird, J. A. (2005). Policy analysis for what? The effectiveness of nonpartisan policy research organizations. The Policy Studies Journal, 33 ( 1 ), 83-105. Hird, J. A. (2005). Power, Knowledge and Politics: Policy Analysis in the States. Washington, DC: Georgetown University Press. John, P. (2003). "Is there life after policy streams, advocacy coalitions and punctuations: using evolutionary theory to explain policy change? The Policy Studies Journal, 31 ( 4 ), 481-498. Kirkhart, K. (2000). "Reconceptualizing evaluation use: an integrated theory of influence. New Directions for Evaluation, Winter 2000 (88), 5-23. Kirp, D. L. (1992). The end of policy analysis. Journal of Policy Analysis and Management, 11 (4), 693-696. Kirp, D. L. (2004). When we "speak truth to power," does anyone listen? Policy Matters, I (I), 693-696. Kumar, R. (1996). Research Methodology: A Step-By-Step Guide for Beginners. London: Addison Wesley Longman Australia Pty Limited, Sage Publications. Lavis, J. N., Robertson, D., Woodside, J.M., McLeod, C. B., & Abelson, J. (2003). 165

PAGE 182

How can research organizations more effectively transfer research knowledge to decision makers? The Milbank Quarterly, 81 (2), 221-248. Leech, N., Barrett, K., & Morgan, G.A. (2005). SPSSfor Intermediate Statistics: Use and Interpretation. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. Leech, N., & Onwuegbuzie, A. (2005). A typology of mixed methods research designs. American Educational Research Association, 23. Leviton, L. (2003). Evaluation use: advances, challenges and applications. American Journal of Evaluation, 24 (4), 525-535. Lindblom, C. ( 1986). Who needs what social research for policy making. Knowledge: Creation, Diffusion, Utilization, 7, 345-366. Lindblom, C., & Cohen, D. K. (1979). Usable Knowledge: Social Science and Problem Solving. New Haven, CT: Yale University Press. Lord, M.D. (2000). Corporate political strategy and legislative decision making: the impact of corporate legislative influence activities. Business & Society, 39 (1), 49-75. Martin, A. (200 1 ). Congressional decision making and the separation of powers. American Political Science Review, 95, 361-378. Martin, B., & Richards, E. (1995). Scientific knowledge, controversy, and public decision-making. InS. Jasanoff, G. E. Markle, J. C. Petersen & T. Pinch (Eds.), Handbook of Science and Technology Studies. Newbury Park, CA: Sage, 506-526. Mellers. B. A., Schwartz, A., & Cooke, ADJ. (1998). Judgment and decision making. Annual Review of Psychology, 49,447-477. Merton, R. K. ( 1968). Social Theory & Social Structure. New York: The Free Press. Miles, M. B., & Huberman, A. M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage Publications. Mooney, C. (1991 ). Information sources in state legislative decisiorunaking. 166

PAGE 183

Legislative Studies Quarterly, XVI, 445-455. O'Sullivan, E., & Rassel, G. R. (1999). Research Methods for Public Administrators. New York: Longman. Palumbo, D. (1987). Symposium: implementation: what have we learned and still need to know? Policy Studies Review, 7 (1), 91-102. Pinkleton, B. E. ( 1998). Effects of print comparative political advertising on political decision-making and participation. International Communication Association, 24-36. Pressman, J., & Wildavsky, A. (1984). Implementation. Berkeley: University of California Press. Putnam, L. L., & Roloff, M. E. (Eds.). (1992). Communication and Negotiation. Newbury Park, CA: Sage Publications. Ragin, C. C. ( 1987). The Comparative Method: Afoving Beyond Qualitative and Quantitative Strategies. Berkeley: University of California Press. Rich, R. (1977). Uses of social science information by federal bureaucrats: knowledge for action versus knowledge for understanding. In C. Weiss (Ed.), Using Social Research in Public Policy Making. Lexington, MA: Lexington Books. Robertson, A. ( 1988). Policy-making and the use of research. In R. Davidson & P. White (Eds.), Information and Government: Studies in the Dynamics of Policy-making. Edinburgh: Edinburgh University Press. Robinson, W. H. ( 1989). Policy analysis for congress: lengthening the time horizon. Journal of Policy Analysis and Management, 8, 1-9. Roe, E. (1994). Narrative Policy Analysis: Theory and Practice. Durham, NC: Duke University Press. Rogers, J. M. ( 1981 ). Policy analysis and governing: the impact of impact analysis. Political Science. University Park: Peilllsylvania State University Press, 273. Rogers, J. M. (1988). The Impact of Policy Analysis. Pittsburgh: University of Pittsburgh Press. 167

PAGE 184

Sabatier, P. A. (Ed.). (1999). Theories of the Policy Process: Theoretical Lenses on Public Policy. Boulder, CO: Westview Press. Sabatier, P. A., & Jenkins-Smith, H. C. (1999). The advocacy coalition framework. In P.A. Sabatier (Ed.), Theories of the Policy Process. Boulder, CO: Westview Press. Sanderson, I. (2002). Evaluation, policy learning and evidence-based policy making. Public Administration, 80, 1-22. Schneider, A. L., & Ingram, H. (1997). Policy Designfor Democracy. Lawrence, KS: University of Kansas. Schneider, M., Teske, P., & Mintrom, M. (1995). Policy Entrepreneurs: Agents for Change in American Government. Princeton, NJ: Princeton University Press. Segerholm, C. (2003). Researching evaluation in national (state) politics and administration: a critical approach. American Journal of Evaluation, 24 (3}, 553-572. Sherman, L. W. (2003). Misleading evidence and evidence-led policy: making social science more experimental. The Annuals of the American Academy, 589 (September 2003), 6-19. Shulock, N. ( 1999). The paradox of policy analysis: if it is not used, why do we produce so much of it? Journal of Policy Analysis and .Management, 18 (2), 226-244. Sridharan, S. (2003). Introduction to special section on "What is a Useful Evaluation?" American Journal of Evaluation, 24 (4), 483-487. Stone, D. (1997). The Policy Paradox, New York: W.W. Norton & Co. Inc. Stone, D. (2000). "Non-governmental policy transfer: the strategies of independent policy institutes. Governance, 13 ( 1 ), 45-70. Terpstra, J., & Havinga, T. (2001). Implementation between tradition and management: structuration and styles of implementation. Law and Policy, 23 (1), 95-116. 168

PAGE 185

Theodoulou, S. z .. & Cahn, M. A. (Eds.). (1995). Public Policy: The Essential Readings. Englewood Cliffs, NJ: Prentice Hall. Ukeles, J. B. (1977). Policy analysis: myth or reality. Public Administration Review, 1977 (May/June): 223-228. Vickers, G., Sir. (1995). The Art of Judgment: A Study of Policy Making. Thousand Oaks, CA: Sage Publications. Webber, D. (1986). Explaining policymakers' use of policy information. Knowledge: Creation, Diffusion, Utilization, 7, 249-290. Weimer, D. (2002). Enriching public discourse: policy analysis in representative democracies. The Good Society, 11 ( 1 ), 61-66. Weimer, D., & Vining, A. (1999). Policy Analysis: Concepts and Practice. Upper Saddle River, NJ: Prentice Hall. Weiss, C. H. ( 1977). Using Social Research in Public Policy Making. Lexington, MA: Lexington Books. W ciss, C. H . & Bucuvalas, M. J. ( 1977). The challenge of social research to decision making. In C.H. Weiss (Ed.), Using Social Research in Public Policy },,faking. Lexington, MA: Lexington Books. Weiss C. H., & Bucuvalas, M.J. ( 1980) Truth tests and utility tests: decision-makers' frames of reference for social science research. American Sociological Review, 45, 302-313. Weiss, C. H., & Bucuvalas, M. J. (1980). Social Science Research and Decision "\faking. New York: Columbia University Press. Weiss, C.H. (2003). In conversation: Carol Weiss and Evert Lindquist on policymaking and research. Interview by Stephen Dale. Retrieved April 6, 2005 from http:/ /web.idrc.ca/en/ev-43607 -20 1-1-do topic.html. Weiss, C. H., Murphy-Graham, E., & Birkeland, S. (2005). An alternate route to policy influence: how evaluations affect D.A.R.E. American Journal of Evaluation, 26 (1 ), 12-30. 169

PAGE 186

White. L. G. (1994). Policy analysis as discourse. Journal of Policy Analysis and Afanagement, 13 (3), 506-525. Zahariadis, N. ( 1999). Ambiguity, time and multiple streams. In P .A. Sabatier (Ed.), Theories of the Policy Process. Boulder, CO: Westview Press. 170