Citation
Assessing the value of the vaccine social media intervention through the re-aim framework implementation dimension

Material Information

Title:
Assessing the value of the vaccine social media intervention through the re-aim framework implementation dimension
Creator:
Wagner, Nicole Marie
Place of Publication:
Denver, CO
Publisher:
University of Colorado Denver
Publication Date:
Language:
English

Thesis/Dissertation Information

Degree:
Doctorate ( Doctor of philosophy)
Degree Grantor:
University of Colorado Denver
Degree Divisions:
Department of Health and Behavioral Sciences, CU Denver
Degree Disciplines:
Health and behavioral sciences
Committee Chair:
Krueger, Patrick
Committee Members:
Yeatman, Sara
Glanz, Jason M.
Ritzwoller, Debra P.

Notes

Abstract:
A growing number of parents are delaying and refusing vaccination. This is increasing the risk for contracting vaccine preventable diseases.1-3 The Vaccine Social Media (VSM) intervention trial was found to increase the number of children up to date on vaccinations compared to usual care.4 Factors important for implementation decision makers were not assessed including the cost-effectiveness of the intervention and changes to health system resource requirements. Despite substantial health and economic benefits due to increased vaccination,5-7 the lack of information on these important implementation factors prevented intervention implementation. In this dissertation, I used the RE-AIM framework implementation dimension as a guide to assess factors important for health systems adopters including the health system resource requirements and cost effectiveness of the VSM intervention. I evaluated the health system resource requirements through differences in the child’s health system encounters between the VSM and usual care study arms using a poisson regression analysis. I assessed differences in well child visits, inpatient, emergency department, phone, and email encounters. I evaluated cost of the intervention using an incremental cost-effectiveness evaluation per additional child vaccinated and pertussis cases prevented. The only identified difference in health system resource requirements was a 5.56 increased rate of emergency department visits associated with the intervention (p=.0043). The incremental cost effectiveness evaluation limited to costs for administration of the intervention and health benefit cost reduction, were $165 per additional child vaccinated and $35,325 per pertussis case prevented. Costs reduced further when the model was extrapolated to a national health system at $22 per additional child vaccinated and $6,740 per pertussis case prevented. The intervention was not cost savings compared to usual care. However, taking the significant population health impact of additional children vaccinated into consideration, costs and resource requirements for administration of the VSM intervention are minimal. This indicates a costeffective vaccine hesitancy intervention for many health systems.

Record Information

Source Institution:
University of Colorado Denver
Holding Location:
Auraria Library
Rights Management:
Copyright Nicole Marie Wagner. Permission granted to University of Colorado Denver to digitize and display this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.

Downloads

This item has the following downloads:


Full Text
ASSESSING THE VALUE OF THE VACCINE SOCIAL MEDIA INTERVENTION THROUGH
THE RE-AIM FRAMEWORK IMPLEMENTATION DIMENSION
by
NICOLE MARIE WAGNER B.A., University of Colorado Boulder, 2003 M.P.H., San Diego State University, 2006
A thesis submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfillment of the requirements for the degree of Doctor of Philosophy Health and Behavioral Sciences Program
2019


This thesis for the Doctor of Philosophy degree by Nicole Marie Wagner has been approved for the Health and Behavioral Sciences Program by
Patrick Krueger, Chair Sara Yeatman Jason M. Glanz Debra P. Ritzwoller
Date: May 18, 2019


Wagner, Nicole Marie (PhD, Health and Behavioral Sciences)
Assessing the Value of the Vaccine Social Media Intervention Through the RE-AIM Framework Implementation Dimension
Thesis directed by Associate Professor Patrick Krueger
ABSTRACT
A growing number of parents are delaying and refusing vaccination. This is increasing the risk for contracting vaccine preventable diseases.1-3 The Vaccine Social Media (VSM) intervention trial was found to increase the number of children up to date on vaccinations compared to usual care.4 Factors important for implementation decision makers were not assessed including the cost-effectiveness of the intervention and changes to health system resource requirements. Despite substantial health and economic benefits due to increased vaccination,5-7 the lack of information on these important implementation factors prevented intervention implementation.
In this dissertation, I used the RE-AIM framework implementation dimension as a guide to assess factors important for health systems adopters including the health system resource requirements and cost effectiveness of the VSM intervention. I evaluated the health system resource requirements through differences in the child’s health system encounters between the VSM and usual care study arms using a poisson regression analysis. I assessed differences in well child visits, inpatient, emergency department, phone, and email encounters. I evaluated cost of the intervention using an incremental cost-effectiveness evaluation per additional child vaccinated and pertussis cases prevented.
The only identified difference in health system resource requirements was a 5.56 increased rate of emergency department visits associated with the intervention (p=.0043). The incremental cost effectiveness evaluation limited to costs for administration of the intervention and health benefit cost reduction, were $165 per additional child vaccinated and $35,325 per


pertussis case prevented. Costs reduced further when the model was extrapolated to a national health system at $22 per additional child vaccinated and $6,740 per pertussis case prevented. The intervention was not cost savings compared to usual care. However, taking the significant population health impact of additional children vaccinated into consideration, costs and resource requirements for administration of the VSM intervention are minimal. This indicates a cost-effective vaccine hesitancy intervention for many health systems.
The form and content of this abstract are approved. I recommend its publication.
Approved: Patrick Krueger
IV


ACKNOWLEDGEM ENTS
First, I’d like to thank my dissertation committee for their continued support and scientific expertise throughout my education. My dissertation chair, Patrick Krueger and committee member Sara Yeatman provided an immense amount of time and methodologic expertise as I went through multiple topic areas of focus. Their patience, support, and critical feedback were essential in my scientific development and direction of career focus. Debra Ritzwoller’s support and guidance in implementation science and cost evaluation were critical to the success of my dissertation and greatly expanded my scientific skillset. Jason Glanz, inspired me to continue my education, taught me a rigorous process of developing and testing effective health intervention programs, and continues to guide my development and growth as a scientist.
I would also like to thank multiple people at Kaiser Permanente Colorado (KPCO) central to the completion of this dissertation. Stan Xu and Komal Narwaney were always available last minute to answer questions and provided critical development of my analytic skills essential to this dissertation. Kris Wain and Christina Clarke greatly assisted in the development of my SAS coding skillset needed to disentangle health system encounters. Matt Daley’s pediatric and vaccine expertise were essential to the interpretation of findings. The Vaccine Social Media (VSM) team, including Jo Ann Shoup, Courtney Kraus, Chris Boyd, and Kathy Gleason, contributed to the development of time estimates and translation of findings.
Lastly, I’d like to thank my family. My parents taught me that perseverance always pays off. My husband, Eric, consistently supported and encouraged me throughout this process without a single complaint despite the many hours away. My son Frankie’s constant smile always inspired me to persevere and be the very best mom and scientist I could be.
*This research has been approved by COMIRB (Protocol Number: 18-2147) and by the Kaiser Permanente Colorado Institutional Review Board (IRBNet ID #s: 1224885-34).
v


TABLE OF CONTENTS
CHAPTER
I. INTRODUCTION AND SPECIFIC AIMS.................................................1
Background......................................................................l
Specific Aims...................................................................2
II. BACKGROUND AND REVIEW OF THE LITERATURE.......................................4
Vaccine Hesitancy...............................................................4
Implementation Challenges.......................................................4
Implementation Theory...........................................................5
Addressing Vaccine Hesitancy: The Vaccine Social Media Project..................7
RE-AIM Implementation Value....................................................10
RE-AIM: Reach, Adoption and Maintenance........................................18
Background Overview............................................................19
III. METHODS.....................................................................21
Review of Aim 1................................................................21
Aim 1 Follow-up Period.........................................................22
Aim 1 Study Population.........................................................22
Aim 1 Measures.................................................................22
Aim 1 Missing Data.............................................................25
Aim 1 Analytic Plan............................................................25
Aim 1 Power....................................................................27
Review of Aim 2................................................................27
Aim 2 Analytic Plan............................................................28
Aim 2 Follow-up Time Period....................................................30
Aim 2 Study Population.........................................................30
Aim 2 Measures.................................................................30
Aim 2c Reference Case Comparison...............................................47
Sensitivity Analysis...........................................................48
IV. RESULTS......................................................................52
Aim 1 Review...................................................................52
Aim 1: Study Population........................................................53
Aim 1 Results..................................................................55
Aim 2: Implementation Cost Effectiveness.......................................58
vi


Sensitivity Analysis...........................................................66
V. CONCLUSIONS...................................................................73
Implementation Resource Evaluation.............................................73
Implementation Cost Evaluation: Public Health Impact...........................75
Adoption Considerations for Health System Adopters.............................77
VSM RE-AIM Value and Next Steps................................................79
Informing Implementation Science...............................................80
Limitations...................................................................81
REFERENCES.......................................................................84
APPENDIX.........................................................................97
vii


CHAPTER I
INTRODUCTION AND SPECIFIC AIMS
Background
A growing number of parents are delaying and refusing vaccination for their children. This is increasing the risk for contracting vaccine preventable diseases.1-3 There is a call to action in the public health community to develop interventions to address this growing health concern.8-11 Researchers at Kaiser Permanente Colorado recently (KPCO) conducted a pragmatic intervention trial, the Vaccine Social Media project (VSM). The trial was found to increase the number of children up to date on vaccinations by 5.9 percentage points compared to usual care.4 This increase in vaccination could have substantial population health and economic benefits.5-7 However, factors important for implementation decision makers were not assessed, including the cost-effectiveness of the intervention and changes to health system resource requirements through health system utilization. Without assessing the VSM intervention’s full value, including the cost and resource implications, health plan decision makers were unable to appropriately weigh the pros and cons of the intervention. Thus, movement toward implementation of the program was stalled.
Costs and health system resource requirements not assessed in the VSM trial are common barriers to implementation for many effective interventions.12 There is a growing number of interventions found to be effective at positively impacting health behavior, but very few are adopted, successfully implemented, or maintained in the health care settings.1314 Reach, effectiveness, adoption, implementation and maintenance (RE-AIM) is an implementation and dissemination framework designed to assist in the implementation process by evaluating the value of health interventions.1516 The implementation dimension of RE-AIM focuses on factors contributing to successful implementation of an intervention including assessment of required resources and costs.17
1


Using the RE-AIM framework as a guide, this dissertation will evaluate implementation factors of the VSM intervention trial important for health system adopters. I will evaluate the impact of the VSM intervention on health system resource requirements through an evaluation of changes in patient health system encounters that may be associated with the intervention. I will then conduct a cost-effectiveness analysis of the intervention in terms of number of pertussis cases avoided and the incremental increase in the number of children vaccinated attributable to the intervention.
Specific Aims
Aim 1: Resource requirements: Assess the health system resource requirements associated with the VSM intervention trial by comparing the rate of encounters (inpatient, ER, outpatient, phone, and email) for children up to 395 days old between participants in the intervention arm (VSM) and the usual care (UC) arm of the trial.
Hypothesis 1: Children in the intervention (VSM) arm of the study will have higher rates of well child care encounters through 395 days of life compared to children in the UC arm.
Hypothesis 2: Children in the intervention (VSM) arm of the study will have lower rates of inpatient encounters through 395 days of life compared to children in the UC arm. Hypothesis 3: Children in the intervention (VSM) arm of the study will have higher rates of email encounters through 395 days of life compared to children in the UC arm. Hypothesis 4: Children in the intervention (VSM) arm of the study will have higher rates of phone encounters through 395 days of life compared to children in the UC arm. Hypothesis 5: There will be no differences in emergency department encounters between children in the intervention (VSM) arm of the study compared to children in the UC arm.
2


Aim 2: Cost: Assess implementation cost-effectiveness of the VSM intervention trial by assessing the incremental cost effectiveness ratio between the VSM intervention and Usual Care arm in a) dollars per additional child vaccinated, b) dollars per pertussis cases prevented, and c) estimating the replication incremental costs-effectiveness of the intervention when scaled to a large national managed care system.
Hypothesis 6: The VSM intervention trial will be cost savings compared to usual care at preventing pertussis cases.
3


CHAPTER II
BACKGROUND AND REVIEW OF THE LITERATURE
Vaccine Hesitancy
Vaccines are known as one of the greatest public health advancements in the 20th century.1819 Despite their success, a growing number of parents are choosing to delay or decline vaccinations.3’20’21 This has been shown to significantly increase the risk for contracting and spreading vaccine preventable diseases.322'28 In addition, under-vaccination has implications for health system and public health costs. A cost analysis by Zhou and colleagues found a $13.5 billion direct medical cost savings associated with following the recommended childhood vaccine schedule.7 Similarly, it was found that a 5% decline in MMR vaccination coverage would result in an additional 2.1 million dollars in public sector costs.29 To address this growing public health concern, there is currently a call to action in the public health community to develop interventions for health systems to reduce vaccine hesitancy.8-11 In order to meet this call to action, implementation of effective interventions into healthcare systems needs to occur. Implementation Challenges
Implementation challenges are common in intervention research. A study conducted by Balas and colleagues found that after 17 years only 14% of research has turned into practice.30 Research in implementation science has consistently found that turning research into practice is rare, difficult, and needs research attention.12-14 The research process is often slow and underfunded to adequately address implementation needs.3132 Additionally, health system adopters rarely use solely effectiveness results to make implementation decision. Resource requirements are often an important consideration.33-35 Yet, research investigators focus on effectiveness results and rarely assess outcomes, such as cost and resources, important for adopters.15’3536
A recent emphasis on implementation of effective health interventions has begun to emerge.1214 Developing studies with implementation in the strategy has become important
4


criteria for grant funding. Additionally, new funding sources to advance the dissemination and implementation scientific knowledge base have formed.14’3738 However, there has been less focus on identifying strategies to implement the numerous interventions that have already been conducted or were found to be effective.12 39 Implementation Theory
One of the longest standing frameworks in implementation science, is the diffusion of innovations theory, first published by Everett Rogers in 1962.40 It outlines the steps to dissemination of innovations into practice. Rogers describes a five-stage decision making process in which dissemination of an idea occurs: knowledge, persuasion, decision, implementation and confirmation. The knowledge stage is when the adopter is first made aware of the innovation. Persuasion is the next stage when the individual actively seeks information and details. The last stage before implementation is the decision making stage when the adopter weighs the pros and cons of the intervention to accept or reject implementation of the innovation.41 At both the second information seeking persuasion stage and the third decision making stage, detailed information about the intervention is needed. Specifically, decision makers require information on the overall value of the intervention.
RE-AIM is an evaluation framework designed to assess the overall population health value of an intervention.15’1642 The goal of RE-AIM is to assist in identifying program elements that can improve implementation and sustained adoption of effective health interventions.42 The framework provides key factors for evaluation that are important to intervention adopters at the persuasion and decision-making stage. There are 5 dimensions to the RE-AIM framework; reach, effectiveness, adoption, implementation, and maintenance.1516 All dimensions of the REAIM framework are important to evaluate the overall population health value of an intervention. However, the RE-AIM implementation dimension focuses on the factors important for health system adopters that contribute to successful implementation.17 The RE-AIM implementation
5


dimension refers to how consistently a health system can deliver a program or policy including the resources and cost required to implement the intervention.17
The intervention impact on resource requirements is a critical component to implementation of an intervention.15 36 A pilot intervention designed to improve management and measurement of low-density lipoprotein cholesterol (LDL-c) levels in patients with coronary heart disease were implemented in six Veterans Health Administration Northwest Network medical centers. Sharp and colleagues assessed barriers and facilitators to implementation and found resources needed for the intervention, including staff time, was a barrier to implementation. They found identifying and allocating required staffing needs was an underutilized step in the implementation process.43 Staff implementing the intervention and stakeholders reported a realistic estimate of resource requirements was needed to adequately staff for successful implementation. As indicated by the VA study, providing accurate information on the resource requirements of an intervention gives health system adopters an opportunity to plan and implement with a greater opportunity for success.
Directly related to the resource requirements, cost is often the first question asked by intervention adopters. However, due to expertise needed and the time required for assessment of long term outcomes, it is rarely evaluated.44 Additionally, when cost is evaluated it often does not incorporates costs important for adopters such as implementation costs, relevant time-frames, and specific intervention resource and staffing requirements.4445 This is particularly true for online health interventions that are often assumed to reach a large population at a low cost.31 In a review article by Tate and colleagues in 2011, only eight internet health interventions reported cost outcomes. Of these studies, only one was conducted in the United States and provided cost details required for implementation and maintenance of the program.46
This lack of cost information for online health interventions becomes increasingly important as the evidence of online behavioral health interventions continues to grow.47 48 Additionally, evidence suggests online interventions are more effective when they incorporate
6


human interaction.49'52 This could require additional resources. However, the evidence also suggests online interaction can be as effective as face to face interaction potentially providing opportunity for health system adopters to shift resources.53 Limited information on implementation and maintenance costs of online interventions make it difficult for health system adopters to determine the most cost-effective approach to address behavioral health concerns. Addressing Vaccine Hesitancy: The Vaccine Social Media Project
Conducting research within an integrated healthcare system is one strategy to promote implementation of research findings into practice. Integrated healthcare systems are networks of healthcare providers offering coordinated care to a patient population.54 Intervention research within an integrated health environment requires collaboration with clinicians. This step encourages modification of interventions to work within the health system, increasing the opportunity for translation.55 Kaiser Permanente Colorado (KPCO) is an integrated healthcare system serving approximately 600,000 members in Colorado, approximately one-fifth of the Colorado Denver Metro Area.56 To address the growing vaccine hesitancy concern and need for effective interventions, our research team at KPCO developed a vaccine hesitancy intervention, the Vaccine Social Media project (VSM).
Currently, vaccinations are recommended for children in the first-year life during well-child visits at 2 months, 4 months, 6 months and 12 months of age as part of Bright Futures program of American Academy of Pediatrics guidelines for pediatric care.57 Vaccines are one of many topics providers address during these visits. More than half of pediatrician’s report spending at least 10 minutes discussing vaccines with vaccine hesitant parents, limiting time available to address other health topics.58 Physicians are a trusted source of vaccine information indicating parent’s desire for expert input.59 However, vaccine hesitant parents also report seeking vaccine information online and begin making vaccine decision during pregnancy before the opportunity to discuss vaccines with the provider at the well child visits.59 60 Additionally, when asked about their experiences receiving vaccines, parents reported a desire
7


for vaccine information prior to well-child visits.61 This indicates an online vaccine information resource with interactive capability allowing discussion with vaccine experts, could be an effective intervention strategy. This formulated the development of the VSM intervention.
In addition to conducting research in an integrated healthcare system, using an implementation framework in the design of an intervention can also aid in the implementation process.12 Using the RE-AIM framework as a guide in intervention development, we designed VSM using an iterative approach. Stakeholders, including pediatricians, obstetricians, and pediatric nurses, were interviewed at different stages in development to identify features of the intervention important for adoption. In agreement with the literature, providers desired an intervention that would alleviate the amount of time spent addressing vaccine questions with parents and a resource to provide patients who had vaccine questions.
The VSM intervention was a web-based resource for parents designed to provide accurate, up to date vaccine information. The interactive components in the VSM arm included a chat room, discussion forum, blog, and “ask an expert” portal. The experts overseeing interaction and contributing to content included a pediatrician, a vaccine safety researcher, and a risk communication specialist. Each month, the expert team generated 1 to 2 blog posts and conducted an online chat session allowing participant to engage with the vaccine experts in real time. Expert staff responded to blog comments and would correct misinformation or respond to participant questions. Chat session transcripts were redacted and posted on the discussion forum. Participants were encouraged to ask questions privately using the “ask an expert” portal where the expert team provided personalized responses within 2 business days. The website was moderated to prevent bullying, abusive language, and disclosure of personal identifying health information. Expert staff used a communication framework in all responses to participants.62
We conducted a randomized trial of the VSM intervention recruiting all pregnant women enrolled at Kaiser Permanente Colorado (KPCO). It was conducted as a pragmatic trial
8


designed to test interventions effectiveness in real-life situations as opposed to optimal situations.63 Participants were randomized 3:2:1 to receive the interactive website with vaccine information (VSM) (n=542), just the vaccine information website (VI) with no interactive features (n=371), or usual well child care (UC) (n=180). As a pragmatic trial, participants were not required to use the intervention, but were still included in the analysis. This more accurately reflects how the intervention would impact a population in a real-world setting. This is an intention to treat analysis which includes all participants in the trial randomized to a study arm regardless of exposure to the intervention or variation from protocol.64 A modified intention to treat analysis was conducted comparing all participants randomized with KPCO membership through the child’s first 200 days of life regardless of exposure to the intervention. The intervention was found to be effective with a 5.9 percentage point increase in children up to date on vaccination by the 6 month well child (200 days) (OR=1.92, 95% Cl, 1.07-3.47).4 Congruent with literature indicating human interaction increases effectiveness of online interventions, up to date status did not differ statistically between the UC and VI arms.52
Vaccination coverage is high, so a small increase in vaccination may not seem to have a large impact. However, even a very small increase in vaccination can have a large impact on the spread of disease. This is called herd immunity.5 When the herd immunity threshold is met, the opportunity for spread of disease is greatly reduced.6 For example, in Colorado in 2017, 87.4% of 13 month-olds were up to date on pertussis vaccination.65 A 2.6 percentage point vaccination increase in pertussis would reach the herd immunity threshold of 90-94%.5 Thus, even if a smaller proportion of the population is impacted, such as in only one health system, a small increase in vaccination could still impact the spread of disease in the community.
Vaccination rates increased in the intervention population at the 12-15 month vaccine visit (489 days) indicating continued vaccination protection.4 This outcome directly impacts an important measure for health systems that often leads to implementation, Healthcare Effectiveness Data and Information Set (HEDIS). HEDIS is a performance measure widely used
9


by health systems to measure quality of care.66 These measures impact the choices employers make on health systems offered in their benefits packages.35 Increase in vaccination positively impacts two HEDIS measures, vaccination coverage and receipt of recommended preventive care (well child visits).5767 For smaller healthcare systems that may not see the same benefit from disease prevention, HEDIS measures provide an observable outcome with direct positive impact on a healthcare system, increasing likelihood of adoption.
In addition to these observable positive outcomes within the health system, providers and pediatric leadership at KPCO remained engaged with the project throughout the trial. I provided status updates to providers annually. During data collection for the trial, providers requested written materials from the intervention they could provide to patients. At study completion, effectiveness results were presented to the pediatricians and pediatric leadership groups. Both groups indicated interest in implementation of the project.
Despite the effective results and positive reaction from stakeholders, we encountered barriers to adoption. Specifically, budgets for department initiatives are limited. KPCO leadership asked for specific costs, including estimates on additional resource requirements for additional staffing requirements for the intervention and use of the health system. Without evaluating the value of the VSM intervention, including the cost and resource implications important for implementation, decision makers were unable to appropriately weigh the pros and cons of the intervention preventing implementation.
RE-AIM Implementation Value
RE-AIM includes assessing cost and resource factors as part of the implementation dimension, with an emphasis on identifying components of the intervention required for successful implementation in a variety of settings.17 42 Health System Resource Requirements
The resources required within a health system in terms of staffing for additional health encounters has an important impact on healthcare institutions willingness to implement.34’68
10


Children less than one year of age are frequent utilizers of the health system. Evidence suggests approximately 83.2% of children less than one year of age met recommendations for well child visits, including visits at age 2 weeks, 2 months, 4 months, 6 months, and 12 months of age.69 Children under 1 year of age also have the highest rate of emergency department and inpatient visits among all age groups.70 71 However, utilization of the healthcare system may vary during that time period if vaccination behavior changes. In a national study assessing healthcare utilization in vaccine hesitant parents, we found that vaccine hesitant parents are more likely to have hospital inpatient visits and less likely to have outpatient/well child visits for their child.2 Increased inpatient stays could be related to stays associated with vaccine preventable diseases or extra caution by healthcare providers when unvaccinated children are seen in the emergency room to ensure it is not a vaccine preventable disease. Thus, I hypothesize children in the VSM arm will be less likely to use inpatient services. Parents who are vaccine hesitant have also reported lack of trust in the healthcare system.60 This could lead to lower utilization of the healthcare system for outpatient and well child visit services as we found in the national study. Therefore, I hypothesize children in the VSM arm will be more likely to use outpatient services (well child visits).
There was no statistically significant difference in emergency department visits (ED).2 This suggest, ED visits are unlikely to change with vaccine hesitancy. However, a decrease in vaccine hesitancy could also lead to a decrease or increase in ED visits. A decrease in hesitancy and increase in vaccination could increase ED visits due to additional vaccine adverse events. Similarly, vaccine hesitant parents may be more concerned about their child for fear of a vaccine preventable disease or experience more vaccine preventable diseases increasing use of the ED.60 Thus, a decrease in vaccine hesitancy could also decrease ED visits. Since the direction of change is unclear, I hypothesize ED visits will not change consistent with the national study. The VSM intervention increased vaccination in children and lessened parents’ concerns about vaccination,72 but it is unclear how that change in attitudes and
11


behavior impacted use of the healthcare system. These potential changes in the way patients use the healthcare system would require adjustments to staffing with direct cost implications.
One of the goals of the VSM trial was to decrease the time needed from the provider to address vaccination concerns. However, studies have found that, initially, online communication with providers may have the opposite impact. A study at KPCO found that patients’ access to secure email communication with the provider initially increased use of clinical services including office visits and telephone encounters.73 74 Technology based intervention trials have also been shown to increase email encounters with providers. We found in a home blood pressure monitoring study that used an online blood pressure tracking system, significantly greater email (median 5 vs 1) encounters during the 6- month study period.75 This increase did not correspond with the expected decrease in office visits, thus increasing the workload for clinic staff. Providers are a key source of vaccine information for vaccine hesitant parents.59 Parents seek information on vaccination prior to the well child visits,61 and email communications with pediatricians were shown to be used frequently in pediatrics for non-urgent medical questions.76 Thus, email communications could be used as a potential resource to validate or further explore vaccine concerns. As such, I hypothesize children in the VSM arm will be more likely to have email encounters than children in the usual care arm.
As discussed previously, an increase in well child care would benefit the KPCO HEDIS quality measures. On the contrary, an increase in email encounters would be a negative outcome for a health system for both staffing requirements and costs. Email encounters are not currently reimbursable at KPCO, as is true for many health systems.77 To identify the impact of the VSM intervention on clinic staffing requirements, assessing changes in encounters due to the VSM intervention is needed.
While the VSM intervention has potential to change the frequency of health system encounters, multiple factors may influence a difference in use of the health system and should be considered in evaluation of health system utilization. Specifically, patients with chronic
12


conditions are more likely to have higher utilization of the health system than patients without chronic conditions.78 Patients who use the internet for medical information frequently, or who are more comfortable with technology, may be more likely to use email communications with providers. Maternal age and family size have also been shown to independently predict use of preventive childhood healthcare services where utilization increased with maternal age and decreased with a increased family size.79 80 Marital status has also been shown to independently impact healthcare utilization, where dual parent homes are more likely to receive preventative services for their children.79 While mother’s employment status has not been independently linked to healthcare utilization,80 employment status could contribute to the time available for reviewing health information ultimately impacting utilization of services.
Patients in the United States with a lower socioeconomic position, measured by income and education, are less likely to use the health system.81 This occurs despite health insurance status. Members of minority racial/ethnic groups are more likely to have less intensive and lower quality of care.81 Similarly, members of racial/ethnic minorities and lower education were found to be less likely to use the online health portal system for email communications with providers.82 Health insurance coverage also plays a key role in health system utilization.83 While all participants in the study have some type of health insurance, the specific type of insurance may contribute to health system utilization.84 For example, patients with high deductible health plans have lower health system utilization rates,85 but have also been more likely to report use of secure email with their provider.82 Lastly, as reported earlier, vaccine hesitant parents are less likely to use well child care and more likely to use inpatient services.2 Each of these factors provide potential unique contributions to encounter utilization.
Cost
Understanding the resource requirements alone does not provide enough information for health system adopters. The first question asked by the leadership at KPCO when the intervention outcomes were presented was, “What is the cost?.” This is often found to be the
13


first and most important question asked by health system adopters, but is rarely evaluated.31’35’46 86 The implementation dimension of RE-AIM includes assessment of the cost spent implementing the intervention. RE-AIM focuses on valuing an intervention for sustained adoption and implementation in a variety of settings.42 This requires assessing costs important for implementation in health systems with outcomes, time-frames, and intervention costs that are relevant to a variety of health system adopters.15 45 Cost-effectiveness analysis for vaccine interventions are often evaluated in the entire population of a country over a lifetime7’87 88 This reflects the goal of the interventions evaluated to incorporate a new vaccination with direct impact on society as a whole. However, these cost-effectiveness results could not easily be translated to the cost and health benefit in one health system.89 Instead, using a health system perspective in conduct of cost evaluations allow for the focus on outcomes that can be directly applied to the overall cost of health systems.90
Increased vaccination provides a positive impact for health systems through an improvement in HEDIS measures benefiting health systems of all sizes. Thus, the incremental improvement in the number of additional children vaccinated, that can be attributed to the intervention, is the primary health outcomes of this cost evaluation. However, health systems may also be interested in understanding the disease prevention due to the intervention. The VSM intervention resulted in increased vaccination in the first 200 days of life impacting eight different infectious diseases.91 However, because vaccine preventable diseases are so rare, a single health system will likely not see any cases for a majority of the diseases. Pertussis is the vaccine preventable diseases impacted by the VSM intervention with the highest rate of transmission. In 2017 in the United States, there were 2,237 cases of pertussis in children less than 1 year of age and 4,994 cases during an outbreak in 2012.92 93 Pneumococcal had the next highest rate of transmission with only 435 cases in children less than 5 in 2017.93 Additionally, an economic analysis of the entire childhood vaccination schedule found pertussis vaccination had the highest direct cost savings for inpatient services, outpatient services and outbreak
14


control.7 Observable outcomes are an important attribute for intervention adoption.34 41 Thus, a secondary health outcome of pertussis cases averted provides an observable health outcome with direct monetary value to health system adopters.
Similarly, using a time periods for cost evaluations that are relevant to a health system is also needed. The VSM intervention implemented into one health system would have health benefits limited to a shorter time period than would be expected with a change in vaccine policy. There was a 5.9 percentage point increase in vaccinations up to date after 200 days. While the rates of Measles Mumps and Rubella (MMR) vaccination coverage at 489 days was higher in the VSM arm compared to the UC arm, it was not statistically significant (95.3 vs 91.8, OR 1.95, p=0.10).4 As vaccination coverage continues to increase with time,65 health benefits due to the intervention reduce. From a health system perspective, pertussis outcomes in the first year of life have the largest impact.94 Pertussis health outcomes include, respiratory illness, neurologic disease, and death.94-96 In 2017, 33% of children under one year of age with pertussis were hospitalized, whereas only 2.5% of children and adults older than one were hospitalized.97 Of the 13 deaths from Pertussis in 2017, 9 of them were in children less than 1 year of age.97 Evaluating implementation cost-effectiveness for the VSM intervention over the child’s first year of life provides a time-frame with measurable benefits and relevant health outcomes for health system adopters.
Implementation cost-effectiveness also includes intervention cost required to implement and maintain the intervention.35’44 45 Health intervention cost evaluations often do not include all costs required for replication to a variety of healthcare settings. Specifically, implementation costs have historically not been included in cost effectiveness analyses of interventions.44 45 These costs could be substantial, particularly for an online intervention that may require resources not currently in place.46 For example, the VSM intervention required a secure login system for the interactive components. Developing a secure portal for the website requires specific expertise and technology capacity for implementation. Costs unique to online
15


interventions are also important to distinguish. Online interventions present a unique circumstance where enhancements and updates are needed as technology changes.32 46 The VSM intervention required annual domain name registration, website security, server space from a website hosting company, and one design update. Including these implementation and maintenance costs in a cost-effectiveness evaluation provides health system adopters the information needed to assess the cost and staffing impacts of the intervention on their health system more accurately.
To effectively evaluate the VSM interventions implementation cost and cost-effectiveness using the RE-AIM framework as a guide includes taking a health system perspective, assessing cost in terms of additional children vaccinated and pertussis cases prevented in the child’s first year of life, and incorporating costs of the intervention necessary to implement and maintain the intervention.
The RE-AIM perspective also emphasizes implementation measures for a variety of healthcare settings.42 To effectively address this RE-AIM component, translating the implementation cost-effectiveness to health systems with varied size and scope is needed. The 2nd Panel on Cost Effectiveness in Health and Medicine was comprised of experts in design and conduct of cost-effectiveness analyses. The panel developed a set of recommendations for cost-effectiveness analyses in healthcare.98 They recommend conducting a reference comparison between healthcare system costs and societal costs. The goal of including societal costs is to also identify the intervention impact on the patient and public health system. From a health system perspective, the societal costs have less value as there is no direct monetary impact on the healthcare institution. Implementation cost-effectiveness evaluations have not always included a societal cost reference case comparison. Instead, they have used a reference case comparison between different size healthcare systems.99'101 This comparison allows for easy identification of the cost and health benefit in a variety of health systems.45 An intervention may not be cost-effective in a small healthcare system, but when scaled to a large national
16


organization it may be cost-effective or vice versa. For example, Shoup and colleagues found that an automated influenza vaccination reminder system was less cost effective then mailed reminder postcards at increasing influenza vaccination at KPCO. When scaled to a larger population size in the reference case comparison, however, the automated system was more cost-effective."
To address these gaps in knowledge of online health intervention costs, an implementation cost-effectiveness evaluation of the VSM trial is needed. Using the RE-AIM framework as a guide, an implementation cost evaluation of the VSM intervention should be evaluated from a health system perspective with cost considerations for a variety of health systems. Specifically: assessing incremental cost-effectiveness of the VSM intervention between the usual care and VSM study arms within the child’s first year of life with intervention costs that incorporate implementation and maintenance expenses in terms of a) additional children vaccinated b) pertussis cases prevented; and c) a larger healthcare system to estimate cost effectiveness in various environments.
Since vaccinations have been successful and the spread of infectious disease is rare93 and the online intervention includes interactive components that could be resource intensive, it may cost more to implement the VSM intervention than the cost reduction in pertussis cases prevented. Cost evaluations often use a pre-determined maximum value that indicates an intervention is cost effective. There has been large variation in what the pre-determined maximum value, or threshold, of a cost-effectiveness analysis should be, such as $50,000 per quality adjusted life year.102 When taking a RE-AIM approach, healthcare institutions should determine the total cost worthwhile for the health benefit received.44 45 However, that value may mean different things to different health systems. There is a lot of variation in terms of what a health system values based on needed resources and associated outcomes. For example, a nonprofit health system with a mission to improve health in the community, may value interventions that address the spread of infectious disease and thus, impacting the community
17


outside the healthcare system. Placing a specific value indicating if the intervention is worthwhile does not address the goal of this dissertation, which is to provide health systems adopters with information important to their specific circumstances to inform implementation decisions.
RE-AIM: Reach, Adoption and Maintenance
The remaining four dimensions for a RE-AIM assessment of value, including reach, effectiveness, adoption, and maintenance, will not be assessed as part of this dissertation. These factors are important for both healthcare decision makers and for determining the overall value of an intervention. However, for the VSM intervention trial they have either already been done or are not yet relevant. Reach assesses the representative of the population willing to participate in the intervention.16103 While there are reasons other than vaccine hesitancy a parent may not obtain vaccines for their child, such as barriers to care, this intervention was not designed to address logistical barriers to vaccination. Rather, the intervention was designed to address vaccine hesitancy in parents with concerns about vaccination. The rate of vaccine hesitant parents enrolled in the study was comparable to the rate of vaccine hesitant parents in the general population (14.1%).14 This indicates the intervention reached the target population. Effectiveness measures the impact of an intervention on outcomes and potential negative outcomes.104 The VSM trial was found to positively impact vaccination rates.4 Since patients could use the intervention at their own discretion, potential negative outcomes to the patient unrelated to vaccination are unlikely. Adoption measures the proportion and representativeness of the sites that adopt the intervention at both the provider and setting level.105 The VSM intervention was implemented by the research team and the clinic staff was not involved. Thus, measures of adoption are not yet applicable. In addition to the resource and cost measures of the RE-AIM implementation dimension, it also assesses factors important for health system adopters that contribute to successful implementation. Specifically, the implementation dimensions also addresses how consistently a health system can deliver a program or policy
18


including the resources and cost required to implement the intervention.17 The research team delivered the intervention during the trial per protocol. Since the research team delivered the intervention, it cannot be determined how consistently a health system would deliver the intervention at this time. Similarly, maintenance measures continued use of the intervention by sites implementing the intervention.106 Since there were no sites implementing the intervention at this point, maintenance is also not applicable. This dissertation will focus on addressing the RE-AIM implementation dimension which assesses factors important for health system adopters that are relevant to implementing the intervention in a healthcare setting, including staff requirements and implementation costs.
Background Overview
In summary, vaccine hesitancy is a growing concern that needs effective interventions implemented in healthcare systems. The VSM trial was found to be effective at increasing vaccination.4 Congruent with literature in implementation science, lack of information on the factors important for implementation of the intervention created implementation barriers.1516 Implementation barriers included lack of information on the cost and resource requirements to address changes in health system utilization required for implementation and maintenance of the intervention. The RE-AIM framework provides a guide to assess the value of an intervention including the specific barriers to implementation experienced with the VSM intervention.16 Using the RE-AIM framework as a guide I will assess 1) the impact of the VSM intervention on resource requirements by testing differences in encounter types between the VSM arm and UC arm and 2) the implementation cost-effectiveness of VSM intervention in terms of cost of the intervention per additional children vaccinated and pertussis case prevented in children less than a year at KPCO. The results of this study will inform health systems considering adopting the VSM intervention. Additionally, this dissertation will advance the literature in implementation science by assessing these often-overlooked measures for intervention implementation
19


including resource requirements and costs of online interventions and increasing knowledge on costs of effective online behavioral interventions with interactive components.
20


CHAPTER III
METHODS
Review of Aim 1
For Aim 1 I conducted secondary analyses on data collected for the Vaccine Social Media (VSM) intervention trial evaluating healthcare utilization for patients enrolled in the trial. Results of the aim will provide information on potential changes to staffing resource requirements due to the intervention for health systems considering adoption. For example, aim 1 will test if patients in the intervention have higher email encounters with their pediatrician than patients in the usual care arm, or if there is a higher rate of well child visits for patients in the intervention arm. I tested the following aim and hypotheses:
Aim 1: Resource requirements: Assess the health system resource requirements associated with the VSM intervention trial by comparing the rate of encounters (inpatient, ER, outpatient, phone, and email) for children up to 395 days old between participants in the intervention arm (VSM) and the usual care (UC) arm of the trial.
Hypothesis 1: Children in the intervention (VSM) arm of the study will have higher rates of well child care encounters through 395 days of life compared to children in the UC arm.
Hypothesis 2: Children in the intervention (VSM) arm of the study will have lower rates of inpatient encounters through 395 days of life compared to children in the UC arm. Hypothesis 3: Children in the intervention (VSM) arm of the study will have higher rates of email encounters through 395 days of life compared to children in the UC arm. Hypothesis 4: Children in the intervention (VSM) arm of the study will have higher rates of phone encounters through 395 days of life compared to children in the UC arm. Hypothesis 5: There will be no differences in emergency department encounters between children in the intervention (VSM) arm of the study compared to children in the UC arm.
21


Aim 1 Follow-up Period
The VSM intervention was administered to children in the first year of life. Thus, changes to the way the health system would be expected to occur due to the intervention would occur during the time period of intervention exposure. Additionally, the intervention is intended to increase vaccination which occurs during routine pediatric care. At Kaiser Permanente Colorado (KPCO) this includes structured well-child visits at 2, 4, 6, and 12 months of age. Vaccines are routinely administered at these in-person visits.57 Thus, changes in encounters due to changes in vacce behaviors and related medical conditions including email and telephone outreach to providers with vaccine questions or concerns, increased ED, inpatient, and outpatient visits due to vaccine adverse events or vaccine preventable diseases, and well child visits to receive vaccination would be expected to occur in the time period when both the intervention and vaccine administration occurs. Consistent with previous literature assessing under-vaccination and health system utilization related to undervaccination, 30 additional days were applied to the 12 month exposure time period allowing for well-child visits that may occur late due to scheduling variation.2’4’107 Aim 1 Study Population
All participants enrolled in the VSM trial from the VSM and UC arms with at least 1 day of KPCO insurance coverage were included in the study population. Participants were dropped from the study due to loss of KPCO insurance coverage, child death, and requests to end participation. Participants dropped from the study were included through the date dropped from the trial.
Aim 1 Measures
Independent variable (IV): Intervention arms.
The independent variable includes patients in the study population randomized to one of
the following study arms:
VSM arm: Patients with access to the online intervention with interactive components
22


UC arm: Patients without access to the online intervention
Dependent Variable (DV): Healthcare Utilization
All health system encounters for the child in the first 395 days of life were extracted from the KPCO databases including the electronic health record (EHR), claims, and enrollment data. This includes outpatient events with KPCO providers that are captured in the EHR, as well as events including ED and hospital admissions for care received outside KPCO health system captured via claims data. Each of the following types of encounters were evaluated in a separate analysis: emergency department (ED), inpatient (IP), email (EM), telephone (TE), well child visit (WCV), and other outpatient visits not including well child visits (OP).108 While email and telephone encounters are made by the parent or guardian of the child, the data is linked specifically to encounter for the child and does not necessarily specify who made the contact. Thus, email and telephone encounters will include all telephone and email encounters made on behalf of the child. For each encounter type, a value for the total number of encounters from birth through days enrolled in the study up to 395 days will be given to each study participant.2
Well child visits are not a specific encounter type available within the EHR dataset. Instead, well child visit encounters were identified using diagnosis codes. The International Classification of Diseases (ICD) Diagnosis codes, is a system used by healthcare providers to identify symptoms and procedures that occur in conjunction with health care received. The ICD codes used to identify encounters in which a well child visit occurred are listed in Appendix Table 1. The table includes ICD9 codes for encounters occurring prior to October 2015 and ICD10 codes for encounters occurring after October 2015.109 All visits containing an ICD code listed in the table were categorized as a well child visit encounter (WCV).
Covariates: Demographics and Pre-existing Conditions
Since the study population assessed for this aim does not include the entire randomized population as participants were dropped from the study (due to fetal demise, no insurance coverage, or loss of interest in participation), these covariates may not be equally distributed
23


between study arms. To address this issue, each potential covariate was compared between the VSM and UC arms. Comparisons were conducted using the Chi-square and t-test appropriate for each measure. When the data indicated a non-normal distribution, Wilcoxon-Mann-Whitney test will used instead of t-test. Similarly, when cells in categorical measures had less than 5, Fischer’s exact test was used instead of chi-square.110 Outcomes indicating a statistically significant difference in covariates at baseline between study arms were included as covariates in the poisson regression model described below.
The potential covariates were captured using the same methods for both UC and VSM arms and are listed below with a description of the measurement parameters and measurement source.
1. Pre-existing conditions in the child: Pre-existing conditions will be extracted from the electronic health record using diagnosis codes. Pre-existing conditions will be categorized in three categories; 1=Non-Chronic, 2=Non-complex Chronic (such as type 1 diabetes), and 3=Complex Chronic (such as type 1 diabetes and chronic pulmonary disease). The pre-existing conditions used in this analysis are those derived from previous literature.111
2. Insurance type at 90 days old (child’s insurance coverage): Deductible Co-Insurance (D/CO), Deductible/Coinsurance HMP Plus (D/CO+), High Deductible Health Plans (HDHP), Traditional HMO(TRD+), Medicaid (MD), Self-Funded (SF), HMO Plus (TRD+)
3. Covariates collected on the baseline survey representing parent response:
a. Parity: 0 (currently pregnant), 1, 2, 3 or more)
b. Education: (Grade school, c. Age: >18 (continuous)
d. Marital status: married, separated, divorced, not married, single, living with partner, widowed
24


e. Race (parent): White, Black or African American, Asian, American Indian/Alaska Native, Native Hawaiian or Pacific Islander
f. Ethnicity (parent), Do you consider yourself to be Latino Spanish or Hispanic: Yes, No, Don’t know
g. Employment: full-time, part-time, unemployed, stay at home parent, retired, student
h. Income (household): <40,000, 40-80, 81-120, 121-150, >150, prefer not to answer
i. Frequency of internet use for health information in the last 2 months: (not at all, less than once a month, about once a month, every week, every day)
j. Vaccine hesitancy: Parent’s Attitudes and Childhood Vaccines (PACV), a validated 15-tem instrument assessing vaccine hesitancy on a scale of 0 to 100.112 Consistent with prior studies, participants with a score of <50 were categorized as “non-hesitant” and >50 as “vaccine hesitant”.113
Aim 1 Missing Data
As participants are included in the analysis for the days enrolled at KPCO, a zero value for each encounter type is assumed to have no encounters, thus there will be no missing data for the dependent variable. As the number of missing data values in the survey is small (< 3 for each question), listwise deletion will be used for surveys with missing data if covariates are included in the model.114 Aim 1 Analytic Plan
I used a poisson regression to assess the rate of utilization for each type of encounter in the child’s first year of life. Outcomes are expressed using incident rate ratios (IRRs) and 95% confidence intervals. Covariates were controlled for in the analysis if significantly different between groups. The model was tested for overdispersion, or higher variance than expected.
25


When found, a negative binomial poisson regression analysis was used to address the overdispersed data. The equation below describes the poisson model used for anlaysis.115
\ /
Where n is the count of visits for the given individual
T=time it was followed-up (up to 395 days)
Xi=applicable covariates (such as mother’s age, parity, preexisting conditions)
Model paramters Bi are log relative risks
Results with an IRR with a confidence interval that does not include 1 and a value greater than 1 indicates an increased risk (supporting hypothesis 1, 3, and 4) ora value less than 1 indicating a reduced risk (supporting hypothesis 2).115 Results with an IRR containing a confidence interval that includes 1, indicates an insignificant result and will provide support for the hypothesis that ED visits are not different between arms (hypothesis 5).
Excessive zeros. Utilization counts that are found to have a higher than expected variance, may be due to excessive zeros. In these situations, a zero-inflated poisson model may be a better model fit.116 For example, to have an email encounter, a parent is required to have a kp.org account. If a parent does not have a kp.org account they could not have an email encounter. In this case, a zero-inflated poisson model may better describe the outcomes. I used the zero-inflated poisson model to compare utilization between the VSM and UC arms where excessive zeros exist and tested between the appropriate poisson model to determine which is the best fit using the Vuong Statistic.117 Results of a zero-inflated model will provide two model estimates. The first models those with a zero probability of having the outcome, such as those without a kp.org account. The second model assess those with a probability of having the outcome in the population who have some probability of an email encounter.
26


Aim 1 Power
Power was calculated to estimate the minimum detectable coefficient for the available encounter data. Power was calculated using PASS software118 using the encounter rate for the usual care study arm in person years.
Well Child Visit
A rate of 4.98 (837/168 person years) well child visit encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.55 will be detectable. Emergency Department
A rate of 0.13 (23/168 person years) emergency department encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.1 will be detectable. Inpatient
A rate of 0.065 (11/168 per years) inpatient encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.07 will be detectable.
Email
A rate of 2.59 (438/168 person years) email encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.4 will be detectable.
Telephone
A rate of 6.58 (1105/168 person years) telephone encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.63 will be detectable.
Review of Aim 2
The goal of aim two was to evaluate the cost of the VSM intervention in order to provide comprehensive detail on all potential intervention costs and benefits for health system adopters to use when evaluating implementation. I addressed the following aim and hypothesis for the cost evaluation:
Aim 2: Cost: Assess implementation cost-effectiveness of the VSM intervention trial by assessing the incremental cost effectiveness ratio between the VSM intervention and Usual
27


Care arm in a) dollars per additional child vaccinated, b) dollars per pertussis cases prevented, and c) estimating the replication incremental costs-effectiveness of the intervention when scaled to a large national managed care system.
Hypothesis 6: The VSM intervention trial will be cost savings compared to usual care at preventing pertussis cases.
Aim 2 Analytic Plan
I conducted an incremental cost effectiveness ratio (ICER) of the total intervention costs per expected health benefit. Specifically, ICER was used to assess difference in total costs between VSM and UC study arms per pertussis cases prevented in the KPCO population birth cohort over a three-year time period. Consistent with cost-effectiveness analyses of health system interventions using a health system perspective, I calculated ICER using the difference in total cost between the UC and VSM arm divided by the incremental health benefit44’90"'101 Health benefit was evaluated for each individual in the first year of life. Cost was evaluated over a three-year time period to incorporate implementation and maintenance costs to assess expected changes in annual costs over time.
Let
IC=incremental cost;
TCvsm=Total costs for the VSM intervention TCuc=Total costs for usual care.
Then IC=TCvsm-TCuc, where TCvsm-TCuc, is represented by the following cost values:44’119
TC1= Intervention costs: The cost of the intervention excluding recruitment and costs associated with research, such outreach and consent for the study enrollment and follow up surveys. Development costs are described, but not included in the analysis as the intervention is already developed. Intervention costs were categorized as fixed or
28


variable. Variable costs were estimated based on the population size of KPCO and Kaiser Permanente National (KP national) in the reference case comparison. TC2=lntervention cost for vaccine administration: the cost increase to the health system for increased vaccination and treating vaccine adverse events.
TC3= Utilization: The health system cost for addition or reduction in visit types due to the intervention (results of aim 1)
TC4= Health outcome costs: The cost reduction to the health system for pertussis cases prevented in the intervention arm.
IC=TC1 +TC2+or-TC3-TC4 Let
IE=incremental health outcome (additional children vaccinated (2a) or pertussis case prevented (2b)) over the three-year time period
2a-Evsm=children vaccinated in the 3 KPCO birth cohorts over the three-year time-period when exposed to the VSM intervention
2a-Euc= children vaccinated in the 3 KPCO birth cohorts over the three-year time period of KPCO in children less than 1 receiving Usual KPCO Care.
2b-Evsm=expected number of pertussis cases in the 3 KPCO birth cohorts over the three-year time- period when exposed to the VSM intervention
2b-Euc= expected number of pertussis cases in the 3 KPCO birth cohorts over the three-year time period of KPCO in children less than 1 receiving Usual KPCO Care.
Then
IE=Evsm-Euc, or the number of additional children vaccinated (2a) or pertussis cases prevented (2b) due to the intervention.
Let
ICER=incremental cost-effectiveness ratio Then
29


ICER=IC/IE or (TC1+TC2+or-TC3-TC4)/(Evsm-Euc)
Aim 2 Follow-up Time Period
Costs and health benefits were evaluated for a birth cohort through the child’s first year of life. The full cost evaluation will include costs and health benefits over three years of time to incorporate implementation and maintenance costs. This will address variation in costs over time including required design updates, decreased staff requirements as repeat questions increase, implementation costs, and increase in population impacted by the intervention over time. In total, the cost evaluation includes all costs and health benefits in the child’s first year of life from 3 birth cohorts over a three-year period of time.
Aim 2 Study Population
The study population includes a KPCO birth cohort for each of the three years in the cost evaluation (3 birth cohorts). The KPCO population averages 6,200 births a year. The KPCO population of children was used as pertussis is rare and the study population is too small to identify cases between arms (VSM=536 and UC=178). Additionally, extrapolating to the KPCO populations provides health system values relevant to potential adopters.
Some health systems may choose to implement the intervention to all patients increasing the number of questions. However, the trial was assessed in a population of pregnant women and parents of young children. Results indicated it was only effective in the pregnant population, not in parents who had already made vaccine decisions for their children. Effectiveness results indicate the intervention is most effective when offered during pregnancy. As such, I focus the cost evaluation in the population known to be positively impacted by the intervention.
Aim 2 Measures Health Benefit
The health benefit, represents the denominator in the incremental cost effectiveness equation above. I evaluated the health benefit in terms of additional children vaccinated (aim 2a)
30


and pertussis cases prevented (aim 2b). I evaluated the health benefit over 4 steps. In step 1, I determined the population potentially impacted by the health outcome for each birth cohort over the 3 year follow up period. In step 2, I assessed the number of children vaccinated over the follow up period. In step 3, I identified the proportion of children vaccinated and unvaccinated at risk for pertussis infection. In the final step 4, I applied the pertussis incidence to determine the number of children infected with pertussis. I evaluated the children vaccinated (part 2 below) and total pertussis cases (part 4 below) for both the VSM and Usual Care arms. Below, I describe details for each step of the analysis. Once the health benefit was determined for each study arm, the difference between study arms (Evsm-Eu) provided the incremental health outcome (IE).
1. Health benefit step 1: population size.
Implementation year 1. The VSM intervention was effective at increasing vaccination after 200 days (7 months of exposure).4 Additionally, the last childhood pertussis vaccination in the first year of life is recommended when the child is 6 months old.57 For the known impact of the intervention to occur, the child would need to be at least 6 months old. Therefore, the pertussis cases prevented and vaccination health benefits were evaluated for 50% of the population in the year 1 birth cohort (n=3,100).
Maintenance year 2 and 3. The pertussis and vaccination health benefit were evaluated in year 2 and 3 from 50% of the population from the previous year birth cohort and 50% of the population from the current year birth cohort (n=6,200 per year).
2. Health benefit step 2: number of children vaccinated (IE for 2a). Vaccination coverage in the usual care group (2-Euc) was estimated at 84.2% coverage. This rate reflects the percentage of the KPCO population up to date on the 6 vaccines (heptatits B; rotavirus; diptheria-tetanus-acceular pertussis; haemophilius influenzae type b; pneumococcal conjugate vaccine; and polio) assessed in the VSM trial at 200 days of age prior to the start of the intervention trial (2008-2012). The VSM intervention followed the same vaccine
31


recommendations as usual care at KPCO with a 1.92 increased odds of vaccination.4 Vaccination coverage due to the intervention was estimated by converting the probability of vaccination at KPCO to odds of vaccination. The odds of vaccination were then multiplied by the odds of increased vaccination due to the intervention. The new odds of vaccination were then converted to probability of vaccination in the KPCO population, with an estimated 91% vaccination coverage (2a-Evsm). For aim 2a assessing a health benefit of additional children vaccinated, the health benefit assessments methods stop here. The vaccination rate applied to the study population in the VSM arm minus the vaccination rate in the study population of the Usual Care arm, provide the incremental health outcome for aim 2a.
To evaluate cost in terms of pertussis cases prevented (2b), vaccination coverage due to usual care (2b-Eu) was estimated at 88.83% using the percentage of children at 200 days of age up to date on DTaP vaccination at KPCO prior to the intervention trial (2008-2012). VSM vaccination rates were increased using the VSM intervention trial 1.92 increased odds of vaccination.4 Vaccination coverage due to the intervention was estimated by converting the probability of vaccination in Colorado to odds of vaccination. The odds of vaccination were then multiplied by the odds of increased vaccination due to the intervention. The new odds of vaccination were then converted to probability of vaccination in the KPCO population, with an estimated 94% vaccination coverage (2b-Evsm).
3. Health benefit step 3 (2b): proportion of children at risk. The population at risk for pertussis infection (2b) was estimated using the vaccination coverage rates for Pertussis in Colorado and the effectiveness of three doses of the pertussis vaccine (98.1%).1201 estimated a total of 13% of the population at risk for contracting pertussis in the UC arm. Assuming a 94% vaccination coverage rate due to the intervention, the population at risk after intervention implementation decreased to 8% for the VSM intervention arm.
4. Health benefit step 4 (2b): population infected with pertussis (IE for 2b). I assessed the pertussis health benefit by applying the 2017 Colorado incidence of pertussis, 71.4/100,000
32


in children less than 1 year of age, to the proportion of children at risk of infection (Step 3 above).121 Incidence calculations are established based on the entire Colorado population of infants. However, only the population at risk could contract the disease. Using the 2017 Colorado vaccination rate from the National Immunization Survey at 13 months of age (87.4%), and the assumptions of the population at risk in Health Benefit Step 3 above, I calculated incidence of pertussis in the at risk population. Results estimated at 71.4/14,261 (0.5%) in children less than 1 year of age.651 determined the number of pertussis cases in the UC and VSM arm by applying this incidence rate to the proportion of children at risk of pertussis infection identified in Step 3 above. The pertussis cases for each study arm was then subtracted to determine the incremental health outcome for aim 2b.
Intervention Costs (IC)
Intervention costs, or the numerator of the incremental cost effectiveness ratio, is determined through an evaluation of 4 different costs. The increment cost (IC) was a sum of the following calculated costs.
1. Cost to administer and maintain the intervention (TC1)
2. Costs for vaccine administration (TC2)
3. Costs addition or reduction due to changes in utilization of the health system from specific aim 1 (TC3)
4. Cost Reduction due to the costs associated with pertussis cases prevented. (TC4)
Usual care at KPCO includes vaccination following the Advisory Committee on Immunization
Practices (ACIP) recommendations. This includes vaccination at the 2 month, 4 month, and 6 month well child visits.57 91 The VSM intervention arm also has access to the same usual care services cancelling out costs associated with those visits. Thus, the intervention costs (IC) in the incremental cost effectiveness evaluation are limited to those associated with the intervention (TC1, TC2, TC3, and TC4) described in detail below. Differences in well child visits due to the intervention identified in Study Aim 1 are incorporated in the TC3 costs described below.
33


1. Intervention administration and maintenance costs (TC1). Development costs were not included in the cost estimates of the intervention as they are considered sunk costs since the process is complete and the resource available to adopters.45
In the intervention administration cost analysis, I assumed the intervention was administered as designed for the intervention trial in all environments. This includes incorporating; 1) a secure access portal to the website, 2) a vaccine expert overseeing the website (intervention manager) and responding to patient questions, participating in chats, moderating the website, monitoring vaccine news, writing bimonthly blogs, developing and recording podcast (2 per year), and updating the content as appropriate, 3) a technology person to maintain the website and address any problems identified, and 4) a pediatrician participating in chats and developing/recording podcasts.
Measuring Intervention Staff Time. Staff time estimates for intervention activities were measured by the study team monthly throughout the intervention trial. Study team tracked time spent on all study activities in a spreadsheet.45 For example, the number of hours in a month spent monitoring news for new vaccine information and responding to participant questions was tracked on the spreadsheet. To supplement the data available in the time tracking document, each study team member was interviewed. Interviews provided an estimate of time on specific activities not available in the time tracking document such as the time spent responding to a new versus repeat question. In interviews, I asked study team members to estimate the average and range of time spent on blog development, question response for a new question, question response for a repeat question, daily website moderation, monthly website updates, daily news monitoring, a podcast recording and podcast development. To assist in accuracy of time estimates, in interviews I provided examples relevant to each staff member with specific patient questions responded to, blog and podcast’s written, and website issues addressed. Time estimates provided in interviews were compared to the monthly tracking and labor reports for further confirmation of accuracy. After time estimates for each activity were collected and
34


confirmed, I averaged time across all study team members. The average time for each activity was used in the labor estimates for the intervention administration costs. Using the high and low values from the ranges of time provided by all staff members, a range of time for each activity was also determined. The ranges were used in sensitivity analyses described below. Results of time used for labor estimates in the cost evaluation are found in the table of intervention cost values (table 1) below.
The range of time estimates for question response (10-240 minutes) reflects the wide variation in type of questions received. For example, one parent asked a simple question for information on what to expect from shots at her child’s 4 month visit:
Question: “My child will have his 4 month well check in a month. I can't remember from my first child whether a fever is common for this immunization. From what I remember, it is not common at 2 months but is at 4 months... I want an idea of what to expect in terms of response to shots.”
Response: “That is a great question. Fever is definitely common and expected after vaccination. A fever is part of the anticipated immune response to the vaccine, similar to when someone mounts a fever while fighting off an infection like the flu. Roughly one-third of infants will have a low-grade fever (temperature greater than or equal to 100.4) after the routine 2-month, 4-month, and 6-month vaccines given at Kaiser Colorado. Roughly 2% to 4% of infants will have a fever higher than 102.2. Based on clinical trials (the type of studies done to prove that vaccines are effective and safe), it doesn't seem like the risk of fever is a lot higher at 4 or 6 months compared to at 2 months of age.” However, there were also more complicated questions that required the study team to spend much more time working on a response.
Question: Hi. My intentions are to vaccinate my child and I am confident it is the right thing to do. Unfortunately, I have a family member who is adamant that I would be putting him at risk by vaccinating. She sent me a link to the following website: [website
35


here]. It appears to be research based and makes points regarding infant immunity that I do not know how to rebut. I would like to provide a fact based response to this family member as well as be reassured that I am doing the right thing for my child by vaccinating but as a mom of a newborn I don't have time to do extensive research.
In this case, the website provided was written by a physician who cited multiple research studies in immunology in support of her argument against vaccination. They were valid scientific articles, but the claims on the website were misinterpreting the results. Due to the large number of scientific studies in immunology that the study team had not previously read, a large amount of time was required to interpret the results and respond in an easy to understand approach for the parent. Where the first question may have only taken a few minutes to respond, the second question took much more time and multiple team members to address. Because of this variation the average time to respond provides a good estimate of time required annually, but large variation exists in addressing the complexity of questions. One of the sensitivity analysis below addresses this potential variation.
The labor estimates for staff time were based on national wage rates from the Bureau of Labor Statistics for a comparable position, with a fringe rate of 31.7% and an overhead rate of 40%.122’123 The fringe rate was determined using the national average and the overhead rate indirect rate was the average across partner institutions in Colorado (ranging from 10%-67%). Specifically, a nurse position was used for the intervention manager, a pediatrician for the vaccine expert in blog development and chat sessions, and a computer programmer for the person implementing and maintaining the online intervention. Expertise of these positions is comparable to the study team member conducting these activities. Additionally, these staff members represent a person within a health system likely to conduct these activities.
Measuring Quantity of Intervention Activities. A database was used for the intervention trial to track all study activities. Data collected included participant questions and response, blog posts and all comments associated with the blog post, and chats sessions transcripts. Tracking
36


for each activity included the dates of interaction, the staff members, and the study participants involved in the interaction. This data was used to quantify the number of events for interactive activities. The rate of interaction was assessed by quantifying the total number of participant questions and blog comments (staff responded to all blog comments) by the total number of participants.
To determine quantity of new and repeat questions, I entered all questions and responses into a spreadsheet. Questions were coded into a specific topic area, such as “alternative vaccine schedule.” Responses to similar topic areas were compared to identify consistent language used. In cases where similar responses were used, the question from the first date available was coded as a new question and all following were coded as a repeat question.
In addition to costs associated with labor, development and maintenance of the online intervention had cost requirements. Wordpress, is a free, web development tool used to build the VSM website. Wordpress is open source, allowing developers to design features for websites, called “plugins,” that can be used to enhance the website and provide additional functionality.124 The wordpress system and plugins require regular updates to enhance security and upgrade features. The VSM intervention included additional programming to enhance design and functionality. As updates to the plugins could influence this additional programming, testing was required with each update to ensure continued functionality and security. In addition, there are costs associated with hosting and securing the website. The VSM trial’s requirements for space were limited and thus we used a web hosting company on a shared server. Additional security measures were taken to address the risk posed by using an open source system with multiple plug-in features. A security system was installed for automated security testing and updates. Identified issues were addressed by the study web developer. In summary, costs required for the VSM website include programming features and design for implementation, website hosting costs, plugin features, security software, updates to the website, and staff time to address functionality and security issues.
37


Intervention implementation costs year 1. Year 1 Intervention costs includes implementation costs. As implementation has not yet occurred, these will be estimated based on similar activities tracked during the study in the monthly time tracking document. Interaction is assumed to increase with time as an entire birth cohort becomes exposed to the intervention. As such, the implementation year of the intervention would be expected to have a lower rate of interaction. In year 1, the rate of interaction expected in the population was limited to interaction (question response and blog comments) in the first half of the intervention trial time period (3.14%). That rate was then applied to the KPCO population size (n=6,200).4 Repeat questions received from participants, often require a lower response time than new questions and the proportion of repeat questions received will increase with time. The proportion of repeat questions received in year 1 were estimated using the proportion of questions received in the first half of the intervention trial time period that were repeats (17.6%).
Intervention maintenance costs year 2 and 3. As the participants in year 2 and 3 include cohorts from the previous year, the total number of question/comments expected in year 2 and 3 were determined based on the rate of interaction (questions and blog comments) for the entire VSM intervention trial time period (5.54%). The proportion of repeat questions received in year 2 were estimated using the proportion of of questions received in the second half of the intervention trial time period that were repeats (46.2%). The proportion of repeat questions received in the second half of the intervention increased by 28.6 percentage points. The proportion of repeats questions received in year 3 were conservatively estimated to increase an additional 20 percentage points (66.2%). As technology continues to change, it was assumed one design update will be required over the three-year time period. Costs for the design update were incorporated in maintenance costs of year 3.
Costs can be fixed or variable in that a fixed cost will not change with the population size and a variable cost will. Table 1 lists each cost to be included in the intervention costs (TC1),
38


the years the cost will be applied, whether the cost is fixed or variable, the measurement values for the cost assessment, and source for each estimate.
Table 1: Sources and Values for Intervention Costs (TC1) for Each Follow-Up Year
Cost description Year Fixed or variable Value Value Source
Implementation Study tracking
Website Development 1 Fixed 50 hrs Study tracking
Website Design 1 Fixed $4930 Study Tracking-Invoice
Content Updates 1 Fixed 8 hrs Study Tracking
Intervention Costs
# of questions/comments (first half of the 1 Variable (17/542) 3.14% of the Study Tracking
intervention) population
# New questions/comments (first half of the intervention) 1 Variable 82.4% of the questions Study Tracking
# Repeat questions/comments (first half of 1 Variable 17.6% of the question Study Tracking
intervention) # of questions/comments (full intervention 2, 3 Variable (30/542) 5.5% of the Study Tracking
time period) population
# New questions/comments (2nd half of 2 Variable 53.8% of the questions Study Tracking
intervention) # Repeat question/comments (2nd half of the intervention) 2 Variable 46.2% of the questions Study Tracking
# New questions/comments (2nd half of the intervention -20%) 3 Variable 33.8% of the questions Study Tracking
# Repeat questions/comments (2nd half of the intervention +20%) 3 Variable 66.2% of the questions Study Tracking
New Question response time 1,2, 3 Fixed 62 (30-240) Study Tracking,
min/response Interviews
Repeat Question response time 1,2, 3 Fixed 26.67 (10-90) Study Tracking,
min/response Interviews
Newsletter Development 1,2, 3 Fixed 90 (60-120) Study Tracking,
min/news letter Interviews
Newsletters/ year 1,2, 3 Fixed 12 Glanz, 20174
Blog development 1,2, 3 Fixed 135 (60-420) min/blog Study tracking, Interviews
Blog import 1,2, 3 Fixed 10 (5-15) min/blog Study tracking, Interviews
Blogs in a year 1,2, 3 Fixed 22 Glanz, 20174
Podcast Recording-Pediatrician 1,2, 3 Fixed 15 (10-15) min/podcast Study Tracking, Interviews
Podcast Recording-Intervention Manager 1,2, 3 Fixed 45 (30-60) min/podcast Study Tracking, Interviews
Podcast content development-Pediatrician 1, 2, 3 Fixed 150 (100-300) Study Tracking,
min/podcast Interviews
Podcasts/year 1,2,3 Fixed 2 Study Tracking
# of Forum updates 1,2,3 Fixed 12 Study Tracking
Forum updates time 1,2,3 Fixed 30 min/update Study Tracking
# of Chats per year 1,2,3 Fixed 12 Study Tracking
Time on chat 1,2,3 Fixed 1 hr/staff Study Tracking
Vaccine news monitoring 1,2, 3 Fixed 120 (60-300) min/month Study tracking, Interviews
Website moderation 1,2, 3 Fixed 12.5 min/day (10-15) Study tracking, Interviews
Website Maintenance 1,2, 3 Fixed 25 hours/year Study tracking, Interviews
Addressing website functionality issues 1,2, 3 Fixed 50 hours/year Study tracking, Interviews
39


table 1 Cont’d
Cost description Year Fixed or variable Value Value Source
Maintenance
Content updates 1,2, 3 Fixed 120 (10-480) min/update Study Tracking, Interviews
# Content updates 1,2, 3 Fixed 4 Study Tracking
Website Design 1,2, 3 Fixed $4930 Study Tracking-Invoice
Plug-in Software 1,2, 3 Fixed $47 Study tracking-invoice
Website hosting 1,2, 3 Fixed $275 Study tracking-invoice
In the results section I provide total intervention administration cost results and staffing requirements in hours for administration of the intervention per year of administration. Sensitivity analyses described below, will provide variation in some of the cost assumptions described above to identify costs in a different environments and circumstances. This will provide additional detail important to health system adopters.
2. Costs for vaccine administration (TC2). To evaluate costs for 2a (additional children vaccinated) I conservatively estimate vaccine administration costs using private sector costs, as opposed to less expensive public sector costs, for the 8 vaccines administered from the KPCO vaccine formulary in the first 200 days of life.125 This includes 2 doses (2 and 4 months) of 4 vaccines protecting against 8 vaccine preventable diseases. Vaccine administration costs were estimated using the Colorado fee schedule costs for a vaccine administration procedural codes.109126 The combination of vaccines and vaccine administration costs are listed in Appendix Table 2 and were applied to the total number of additional children vaccinated due to the intervention from Health Benefit step 2 to get the total cost of vaccine administration. To evaluate costs for aim 2b (pertussis cases prevented), vaccines administered were limited to the DTaP vaccine and costs associated with administration of DTaP in the first 200 days of life.
Vaccine adverse events are rare, particularly when a small percentage of the population is impacted. This limits the potential impact of adverse events on costs. There are currently no known adverse reactions linked to polio vaccination.127 For the hepatitis B vaccine, anaphylaxis is the only known adverse event and it occurs in 1 case per 1.1 million vaccines given, too rare
40


to impact the additional population vaccinated.128 Pneumococcal conjugate vaccine (PCV) has been associated with fever and irritability. However, one study found rates were not higher between vaccine recipients and controls.129 Due to this evidence and consistent with a previous PCV cost analyses, these adverse events will not be incorporated into the costs.7130 Rotavirus has been linked in rare instances to intussusception, vomiting and diarrhea. However, a study conducted by the vaccine safety datalink (VSD), a group of managed care organizations charged with monitoring postlicensure vaccine safety, did not find an association with the vaccine and adverse events. There was found to be a small risk of intussusception too rare to impact the additional children vaccinated and thus will not be included in the cost evaluation.131 Fever after administration of haemophilus influenzae type b (hib) vaccine has been reported in 5-30% of cases. However these often resolve within 24 hours and are treated at home.128 As such, previous cost evaluations for hib vaccine limited costs for fever adverse events to antipyretics, available over the counter.7’132133 They did not include health systems costs. Since the focus of this evaluation is from the health system perspective, no costs associated with fever adverse events for the hib vaccine will be included.
On the contrary to the other 5 vaccines, diphtheria, tetanus and pertussis (DTaP) vaccine has been associated with the following conditions that impact health system costs; medically attended local reactions (0.4%),134 and medically attended events associated with fever (0.27%).135 As recent studies have found no association with seizures or encephalopathy and DTaP or Pediarix (the combination vaccine on the formulary at KPCO containing DTaP), these adverse outcomes were not be included.136-138 A severe reaction of anaphylaxis was not included in the cost evaluation of increased vaccination as it occurs in less than 1 in a million cases, too rare to impact the intervention costs in the KPCO and KP national populations.139 The percentage of medical events occurring in each potential location (urgent care, inpatient, emergency department, and outpatient) were determined based on the literature for each outcome and are listed in Table 2 below.134135
41


Table 2: Proportion of the Population Experiencing an Adverse Outcome From DTaP by Event Type and Location
Description of Adverse Event Type % of population vaccinated Source
Medically attended events associated with fever (fever) 0.4% Zangwill, 2008135
Medically attended events for local reactions (local reaction) 0.27% Jackson, 2011134
Description of Fever Visit Type % of Medically Attended Fevers Source
Outpatient 75% Zangwill, 2008135
Emergency Department 21% Zangwill, 2008135
Inpatient 4% Zangwill, 2008135
Description of Local Reaction Visit Type % of Local Reactions Source
Outpatient 60.6% Jackson, 2011134
Emergency department 31.3% Jackson, 2011134
Inpatient 7.9% Jackson, 2011134
Urgent care 0.2% Jackson, 2011134
Costs for DTaP adverse outcomes visits were determined using the fee schedule for current procedural terminology (CPT) codes associated with medically attended fever and medically attended local reactions.126 The appropriate procedural codes for each type of health outcome and location were identified using events extracted from the KPCO electronic health record in children less than one year of age with a primary diagnosis of fever due to a vaccine or local reaction due to a vaccine. Table 3 lists the primary diagnosis codes used to identify cases of fever or local reaction. CPT codes were extracted for inpatient, emergency department and outpatient visits.
Table 3: ICD Codes Used to Identify Cases of Fever and Local Reactions Within the KPCO Electronic Health Record
Code Type Definition
R50.83 10 Postvaccination fever
T50.B95A 10 Adverse effect of other vaccines
780.63 9 Postvaccination fever
E949.6 9 Adverse Effect of a viral vaccine
Consistent codes for each health condition and location were identified and confirmed with a pediatric expert to ensure they accurately reflected expected medical treatment of patients experiencing a medically attended fever and location reaction. After confirming the CPT codes identified for each condition and location, costs were derived from the fee schedules published by the Centers for Medicare & Medicaid Services (CMS).140 Colorado fee schedules were used for the KPCO cost analysis and the CMS cost estimates were used for the KP
42


National analysis (reference case comparison for aim 2c). No inpatient encounters of either adverse health outcome occurred in KPCO in the last 20 years. The average inpatient cost for children less than 1 year of age, was found to be $5,000 on average from the Healthcare Cost and Utilization Project in 2012.71 For any medically attended fevers or local reactions occurring in the inpatient setting due to the intervention, a cost of $5,000 updated to 2017 US dollars using the Bureau of Labor Statistics Personal Consumption Expenditures Price Index was applied.141 Costs used for the vaccine administration and adverse event outcomes are described in Table 4 below, including the source for the cost value.
Table 4: Costs Associated with Vaccine Administration and Vaccine Adverse Event Outcomes (TC2)
Cost Description Cost Value Source
Cost of vaccines in the first 200 days of life $712.80 CDC125
Vaccine administration $18.93; $16.94 Colorado Fee Schedule126, National Fee Schedule140
Fever-outpatient (low complexity outpatient visit) $79.30 CO Fee schedule, 2017 126 KPCO data
Fever-emergency department (moderate complexity ED visit and urinalysis) $96.84 CO Fee Schedule, 2017126 KPCO data
Fever-inpatient $5,728 Weiss, 2012 71 Bureau of Labor Statistics, 2017141
Local reaction-outpatient (low complexity outpatient visit) $79.30 CO Fee schedule, 2017126 KPCO data
Local reaction-emergency department (moderate complexity ED visit and bacterial culture) $96.84 CO Fee schedule, 2017126 KPCO data
Local reaction-inpatient $5,728 Weiss, 201271 Bureau of Labor Statistics, 2017141
3. Additional or reduced encounters (TC3): results from specific aim 1. Costs associated with changes in health system utilization were applied to the intervention costs. In study aim 1, I evaluated changes in utilization due to the intervention. Any significant differences in number of encounters were then applied to costs as a reduction or addition, appropriate for the aim 1 findings. To accurately capture the expected differences in encounters, a two part
43


model was used. The first part was a logit model comparing those with no events (ie: ED count=0) to those with any events (ie: ED count >1). This provides an estimate for the increase in having a visit. The second stage used a zero truncated poisson where patients with no visits were excluded. This model provided an estimate for the increase in visits if one had occurred. Using the two stage model provided a more accurate estimate of the increase or decrease expected as it captures the expected difference in having a visit at all as well as the expected difference if one has already had that type of visit. The coefficient and standard errors for the two models were used to determine the annual increase or decrease in encounter events using a Monte Carlo simulation.142 A Monte Carlo simulation projects the probability of the risk for an event to occur by simulating different scenarios. The model is given the parameters, in this case the coefficient and standard errors, then randomly selects possible outcome values based on distribution of known parameters and repeats by the number of requested simulations. Monte Carlo simulations are most often used when forecasting situations of uncertainty. Using this approach, I can more accurately obtain an estimate that reflects the expected change in encounters due to the intervention in a variety of circumstances. Using 10,000 simulations, I was able to get an average estimate of change in number of encounters. That total was then applied to the KPCO and KP National population of children each year to get the increase or decrease in number of encounters.
Costs for the encounters were estimated using a moderate complexity visit. The visits costs were derived from the fee schedules developed by the offices of Medicaid and Medicare, Colorado and national fee schedule estimates for the KPCO and KP National cost evaluations respectively.126140
Email encounters are not billable events. Thus, an identified change in email encounters had costs estimated using pediatrician hourly wage, email response time, and difference in email encounters. Provider email response time varies, but was previously determined to be 3.5
44


minutes on average.143 Provider wages were estimated using national wage rates from the Bureau of Labor Statistics for comparable positions.122
4. Health outcome cost benefit (TC4). Health outcome costs represent the total health system cost reduction due to the decrease in number of pertussis cases. The total number of pertussis cases prevented determined in the final phase of health outcome evaluation described in detail above, provided the number of events. Cost values were then applied to the number of pertussis events reduced. Using data from a large geographically diverse health system population, a cost evaluation estimated the increased health system costs associated with pertussis infection in the first year of life.144 Costs estimates included health system costs associated with inpatient stays, emergency room visits, ambulatory visits, pharmacy, and other medical costs. On average, infants less than one year of age with a pertussis diagnosis had $8,271 increased health system costs. Average pertussis costs were updated to 2017 US dollars by using the Personal Consumption Expenditures Price Index.141145 The total cost for a pertussis event times the total number of events prevented provide the health outcome cost benefit value.
Incremental cost analysis. Total increment cost (IC or the numerator in the incremental cost effectiveness ratio) was valued by summing costs calculated from each evaluation above associated with intervention related costs (TC1+TC2+ or-TC3) and subtracting the health system costs associated with the health benefit-pertussis cases prevented (-TC4). The total number provides an implementation and maintenance cost of the intervention over a three year time period.
Incremental Cost Effectiveness Evaluation (ICER)
Results of the incremental costs evaluation (IC) and incremental health outcome (IE) are presented for each component of the evaluation described above. Specifically, costs and health benefit were provided for each year of assessment to give additional detail on how costs and health outcomes change overtime from implementation to maintenance of the intervention.
45


Results include the health outcomes of additional children vaccinated (2a) and pertussis cases prevented (2b), costs to administer and maintain the intervention (TC1), costs for vaccine administration (TC2), costs for changes to utilization of the health system from specific aim 1 (TC3), and cost reduction for the health system due to pertussis cases prevented (TC4)
I will present Incremental Cost Effectiveness Ratio for additional children vaccinated and pertussis cases prevented in two ways:
1. The total incremental costs derived from all components of the intervention over the health benefit (TC1+TC2+TC3-TC4)/(Evsm-Euc). A zero or negative value support the hypothesis that the intervention is cost savings compared to usual care. This analysis includes all potential costs associated with intervention implementation and maintenance. Results will be presented as the total cost per additional child vaccinated and the total cost per pertussis case prevented. If the intervention is not cost savings, the total cost worthwhile to reduce pertussis cases or increase vaccination is valued by the health system considering adoption.
2. The total incremental costs derived from intervention administration and health benefit cost reduction over the health benefit (TC1-TC4)/(Evsm-Euc). The results of this evaluation provide a total cost of administering the intervention (TC1) and the cost reduction from the health benefit (TC4). This evaluation may have more relevance to health system adopters. ICER evaluation 1 incorporates costs for the vaccines provided. This is an assumed outcome of the intervention. Additionally, provisions in the affordable care act require coverage of preventative services including vaccination.146 This indicates the cost of vaccines actually incurred by the health system will likely be much lower if there is a cost at all. Lastly, health systems may be most interested in only the additional cost associated with administration of the intervention. Thus, the changes in utilization and vaccine administration will unnecessarily inflate that direct cost measure. To address the potential desire of health system adopters to understand the total cost of
46


administration of the intervention alone, the second cost effectiveness analysis will be limited to intervention administration and health benefit costs. Results will be presented as the total cost per additional child vaccinated and pertussis case prevented.
Aim 2c Reference Case Comparison
For the reference case comparison, disease cases and intervention costs were adjusted to the birth cohort of Kaiser Permanente national (KP national) (n=116,000). KP national represents a healthcare system covering a large population of patients and geographic area allowing assessment of costs in a system with variation in size and scope. Additionally, the care provided at KP national is comparable to KPCO offering a larger environment where effectiveness results of the VSM trial are likely to translate increasing the accuracy of cost extrapolation.
The population infected with pertussis (health benefit step 4) was adjusted using national vaccination coverage rates in children 13 months of age65 and national pertussis incidence rates for children less than 1 year of age.93 Table 5 lists the specific variable adjustments made to the population at risk calculations.
Table 5: Reference Case Adjustments_____________________________________
Description of adjustment Value Source
Pertussis Incidence 50.88 CDC93
100,000
Pertussis Vaccination Rates 89.2% CDC65
As content specific to the geographic location of the participants (Colorado) may have been an important part of the success to the VSM intervention, intervention costs were adjusted to incorporate local content. Specifically, local content updates and local blog development costs were applied to each of the 9 states where Kaiser Permanente is located (California, Colorado, Georgia, Hawaii, Oregon, Washington, Maryland, Virginia, Washington DC). Additionally, building intervention tailoring to link patients to information specific to their location would be required in the implementation costs of the intervention. For example, patients in
47


Kaiser Permanente Northern California (KPNC) would see information specific to the vaccines in the KPNC vaccine formulary and clinic locations and hours within the KPNC system. The VSM intervention required tailoring to the website in order for participants in each study arm to receive access to varied interventions. Study tracking of time to build tailored components were used to estimate tailoring development for implementation at KP National. Lastly, expanding the population size and usage of the website would require additional space from the website hosting company. Cost for website hosting annual fees were adjusted to reflect a dedicated server that provides additional space. Table 6 lists the fixed and variable intervention costs that will be adjusted specific to the reference case comparison. The variable measures in the Table are variable by the number of geographic locations. Variable measures in the Intervention costs table (Table 1) above will be adjusted by the size of the KP National population.
Table 6: Reference Case Cost Adjustments Including Year Cost Applied, Fixed or Variable to Geographic Location, Value for Analysis, and Value Source__
Cost description Year Fixed or variable Value Value Source
Implementation
Tailored login 1 Fixed 75 hrs Study tracking
Local content 1 Variable 8 hrs/location Study tracking
development
Plug-in for membership 1 Fixed $9/year Study tracking-invoice
tailoring
Intervention costs
Local content blogs 1,2, 3 Variable 1 Study Tracking
Local content updates 2, 3 Variable 1 Study Tracking
Hosting services 1,2, 3 Fixed $7,200 Hosting company estimate
Results will be provided for each evaluation described in the incremental cost effectiveness methods above for the reference case comparison. Comparisons between the local (KPCO) and National (KP National) health systems allow for assessment of a health system size that is most cost-effective for implementation. Additionally, results will provide cost and staffing estimates for implementation on a larger scale.
Sensitivity Analysis
The final step to a cost analysis is determining the appropriate sensitivity analyses used to test assumptions and estimates the range in costs for varied environments.44’45’90 98
48


Specifically, incidence rates of pertussis change each year.147 Using pertussis incidence from both an outbreak and a non-outbreak year provide cost estimates for a variety of disease spread situations. Vaccination coverage rates also vary by year and location.148 The known range of vaccination coverage rates in infants across the United States provide cost estimates that can be applied to health systems in variety of geographic locations. Costs per pertussis case to a health system may vary widely. As pertussis is rare, this variation could have a large impact on health system costs. Using a range of pertussis treatment costs provides a more comprehensive assessment of potential costs. Vaccine expert staff could vary depending on the environment where the intervention is implemented. A health system such as KPCO with an immunization task force, would likely use the nursing staff to respond to questions and moderate the website. However, a local health system may use community health workers in a similar role. Additionally, there is also large variation in provider email response time.77143 As the costs to a health system will vary based on the staff response time, a range of response times were used. Lastly, adjusting intervention estimates of moderation time required of the staff implementing the intervention were applied. A website moderation protocol was built to 1) address inappropriate behavior on the website, such as responding with abusive language and 2) identify participant interactions requiring response, including blog and discussion forum comments. Moderation time may need to increase with varied patient populations interacting with the website. Each of these sensitivity analyses can be used by health system adopters to identify the cost relevant to their respective healthcare system and cost adjustments made accordingly. In summary, the sensitivity analyses will include:
1. Pertussis incidence during an outbreak year, 2012.
2. Vaccination coverage variation
3. Pertussis disease cost variation
4. Adjustments to the intervention costs including
a. Staffing variation
49


b. Number of questions
c. Response time variation
d. Moderation time increase
Outbreaks
Pertussis cases prevented were revised in this sensitivity analysis by incorporating incidence rates of pertussis from an outbreak year in Colorado (178.9/100,000) and nationally (126.7/100,000).149'150 Vaccination Coverage Variation
Vaccination coverage rates vary by location. To help distinguish cost impact in varied geographic locations, vaccination coverage rates were adjusted to reflect the variation in state and vaccine coverage rates. The low and high end of the vaccination rates for each of the 6 vaccines administered in the first 200 days of life were identified using data from the national immunization survey at 7 months of life. Rates varied between 52.8% and 95.1% coverage respectively.65’151'155 Pertussis Disease Treatment Costs
Health system costs were found to be $8,271 higher in infants with pertussis compared to matched comparators. However, there was large variation based on age of the child from $3772 to $18,781.144 This range of values was applied to health system disease costs for the sensitivity analysis. Costs were updated to 2017 US dollars by using the Personal Consumption Expenditures Price Index.141 Intervention adjustments
a. Staffing variation. To address the variation that may occur in staff overseeing and responding to intervention questions, the intervention manager salary was adjusted from a nurse to the following positions: pediatrician, community health specialist, research scientist, and social scientist research assistant.
50


b. Number of questions. The quantity of questions per year was based on rates from the intervention. To address the potential variation in this rate, the percent of questions was halted and doubled to identify the potential variation.
c. Response time. As the time to respond to patient interactions may vary based on having responded to similar questions and staff expertise, response time was adjusted using the range of response time values identified in study tracking (30-240min for each new response and 10-90 minutes for each repeat response). Study staff included a variety of vaccine expertise and communication skill levels providing an accurate range of variability.
d. Moderation time. Estimates for moderation time per day were doubled (25 minutes per day) to incorporate potential increased need for moderation with increased population size.
51


CHAPTER IV
RESULTS
Aim 1 Review
Aim 1: Resource requirements: Assess the health system resource requirements associated with the Vaccine Social Media (VSM) intervention trial by comparing the rate of encounters (inpatient, ER, outpatient, phone, and email) for children up to 395 days old between participants in the intervention arm (VSM) and the usual care (UC) arm of the trial.
The goal of aim 1 was to provide information on potential changes to staffing resource requirements due to the intervention. I hypothesized the following:
Hypothesis 1: Children in the intervention (VSM) arm of the study will have higher rates of well child care encounters through 395 days of life compared to children in the UC arm.
Hypothesis 2: Children in the intervention (VSM) arm of the study will have lower rates of inpatient encounters through 395 days of life compared to children in the UC arm. Hypothesis 3: Children in the intervention (VSM) arm of the study will have higher rates of email encounters through 395 days of life compared to children in the UC arm. Hypothesis 4: Children in the intervention (VSM) arm of the study will have higher rates of phone encounters through 395 days of life compared to children in the UC arm. Hypothesis 5: There will be no differences in emergency department encounters between children in the intervention (VSM) arm of the study compared to children in the UC arm.
Support for these hypotheses would include an IRR greater than 1 and confidence interval that does not include 1 for hypothesis 1, 2 and 4, an IRR less than 1 and confidence interval that does not include 1 for hypothesis 3, and an IRR confidence interval that includes 1 for hypothesis 5.
52


Aim 1: Study Population
A total of 1093 pregnant women were enrolled in the study with 542 participants in the VSM arm and 180 participants in the UC arm. Participants were dropped from analysis due to 4 fetal demise or child deaths, 1 not interested in participating before the birth of their child, and 3 had no Kaiser Permanente Colorado (KPCO) insurance coverage in the first 395 days of the child’s life (figure 1). There were 536 participants in the VSM arm and 178 in the UC arm included in the analysis.
Randomized
n=1093
Analyzed n=178 Loss to follow-up Child death (n=1) no KPCO coverage (n=1) Analyzed n=536 Loss to follow-up Fetal demise (n=3), not interested (n=1) no KPCO coverage (n=2)
Figure 1
Baseline characteristics are found in Table 7 below. Employment was the only observed baseline characteristic different between study arms in the analyzed cohort. VSM participants had a higher percentage of stay at home mothers than the UC arm (21.3% vs 12.4%). Mean parental age at enrollment was 31.6 years. A majority of the population was white (88.8%), non-hispanic (89.5%), college educated (83.1%), married (89.1%), employed full time (64%), and the child had no chronic conditions (89.9%). Approximately half of the population was employed full time (64%), made greater than $80,000 annual household income (55.9%), was on an HMO insurance plan (40%), and had previous children (54.5%). At enrollment 13.4% of the
53


population was vaccine hesitant based on PACV screener score113 and 62.7% reported using the internet for health information at least weekly. Some responses were combined to address the privacy risk in reporting a small number of respondents. However, each characteristic was compared between study arms as measured and the statistical significance of the comparison between arms did not change. Based on the baseline characteristic estimates, randomization appeared to work as expected except for employment. As such, employment was included as a covariate in the poisson regression analysis.
Table 7 Parent Baseline Characteristics
Characteristic Social Media N=536 Usual Care N=178 Total N=714 Pvalue
Pre Existing Conditions**
Non-Chronic 474 (88.4%) 161 (90.4%) 635 (88.9%) 0.5544
Non-complex Chronic 48 (9%) 11 (6.2%) 59 (8.3%)
Complex Chron 13 (2.4%) 6 (3.4%) 19 (2.7%)
Hispanic**
Missing <5 (0.4%) <5 (0.6%) <5 (0.4%) 0.161
Yes 44 (8.2%) 22 (12.4%) 66 (9.2%)
No 486 (90.7%) 153 (86%) 639 (89.5%)
Don't know <5 (0.7%) <5 (1.1%) 6 (0.8%)
Race*
Missing 2 (0.4%) 1 (0.6%) 3 (0.5%) 0.9062
Non-White 53 (11%) 16 (10.2%) 69 (10.8%)
White 428 (88.6%) 140 (89.2%) 568 (88.8%)
Education*
Missing 0 (0%) 1 (0.6%) 1 (0.1%) 0.221
College or more 446 (83%) 147 (82.6%) 593 (83.1%)
Less than college 90 (17%) 30 (16.8%) 120 (16.8%)
Marital Status**
Missing 1 (.19%) <5% 2(28%) 0.0697
Married or living with partner 19 (3.54%) 173 (97%) 683 (95.66%)
Not married or living with partner 25 (4.66%) <5% 29 (4.06%)
Employment*
Missing 1 (0.2%) 1 (0.6%) 2 (0.3%) 0.0314*
Employed full time 339 (63.2%) 118 (66.3%) 457 (64%)
Others 82 (15.3%) 37 (20.8%) 119 (16.7%)
stay at home parent 114 (21.3%) 22 (12.4%) 136 (19%)
54


Table 7 Cont’d
Characteristic Social Media N=536 Usual Care N=178 Total N=714 Pvalue
Income*
Missing 1 (0.19%) 1 (0.56%) 2 (0.3%) 0.8382
<$40,000 44 (8.21) 15 (8.43%) 59(8.3%)
$40,000-$80,000 161 (30.04%) 59 (33.15%) 220(30.8%)
$81,000-$120,000 186 (34.70%) 57 (32.02) 243(34.1%)
$121,000-$150,000 58 (10.82%) 23 (12.92%) 81 (11.3%)
>$150,000 59 (11.01%) 16 (8.99%) 75 (10.5%)
Declined to Answer 27 (5.04%)) 7 (3.93%) 34 (4.8%)
Insurance*
Missing 60 (11.19%) 23 (12.92%) 83 (11.62%) 0.506
Deductible 218 (40.67%) 81 (45.51%) 299 (41.88%)
HMO 222 (41.42%) 64 35.96%) 286 (40.06%)
Other (Medicaid and Self Funded) 36 (6.72%) 10 (5.62%) 46 (6.44%)
Hesitancy*
Hesitant 65 (12.1%) 31 (17.4%) 96 (13.4%) 0.0837
Nonhesitant 471 (87.9%) 147 (82.6%) 618 (86.6%)
Internet use for Health information*
Missing 0 (0%) 1 (0.6%) 1 (0.1%) 0.1652
Less than weekly 207 (38.6%) 58 (32.6%) 265 (37.1%)
More than weekly 329 (61.4%) 119 (66.9%) 448 (62.7%)
Parity*
Missing 0 (0%) 1 (0.6%) 1 (0.1%) 0.2203
>1 st pregnancy 293 (54.7%) 96 (53.9%) 389 (54.5%)
1st pregnancy 243 (45.3%) 81 (45.5%) 324 (45.4%)
Age***
mean (SD) 31.7 (4.3936) 31.4 (4.0658) 31.6 (4.31) 0.5144
‘Statistically Significant difference between study arms AComparison done using a Chi Square “Comparison done using Fisher's Exact AAAComparison done using ttest
Aim 1 Results
The mean number of each type of visit and variance is listed in Table 8 below. There were a mean number of 4.75 well child visits, 2.73 outpatient visits, 0.23 emergency department visits, 0.045 inpatient visits, 2.38 email encounters, and 6.45 telephone encounters in the VSM arm. The number of each encounter type with no visits and the range in number of visits is presented in the Appendix Table 3.
55


Table 8 Mean Number of Visits and Variance by Study Arm
Type of Visit Social Media n=536 Usual Care n=178 Total n=714
Well Child 4.75 (1.747) 4.70 (1.815) 4.74 (1.762)
Other Outpatient 2.73 (8.589) 2.58 (9.420) 2.69 (8.787)
Emergency Department 0.23 (0.482) 0.13 (0.136) 0.21 (0.397)
Inpatient 0.04 (0.050) 0.06 (0.092) 0.05 (0.061)
Email 2.38 (14.863) 2.46(15.674) 2.40 (15.045)
Telephone 6.45 (33.874) 6.21 (27.171) 6.39 (32.174)
I began assessing changes in encounters by using a poisson regression analysis. I controlled for employment in all models. Goodness of fit statistics indicated over dispersion in other outpatient visits, email encounters and telephone encounters. In each case, I used negative binomial regression analysis to address the over-dispersed data. Goodness of fit statistics indicated negative binomial regressions analysis was a better fit than poisson regression analysis in all cases. Additionally, excessive zeros were identified in other outpatient visits, emergency department visits, email encounters, and telephone encounters. I used zero inflated poisson regression analysis to test the excessive zero in emergency department and inpatient visits and zero inflated negative binomial poisson regression in other outpatient, email, and telephone encounters. To test the best model fit, I compared the zero inflated model and the poisson or negative binomial regression model using a Vuong statistic. The model with the best fit was used for the analysis and is reported in Table 9 below.
The only significant differences in visits between the VSM and UC arms was emergency department encounters. In the poisson regression model, when controlling for employment there was a 1.81 (1.16-2.83, p=.0087) increased rate of emergency department visits in the VSM arm compared to UC. The model was a goodness of fit model indicated a poisson regression was a good model fit (p=0.98131). The Vuong statistic indicated the zero inflated poisson regression analysis had a better model fit than a poisson regression analysis. In the zero inflated analysis the count model indicated a 5.5 (1.711-18.079, p=0.0043) increased rate of additional emergency department visits in the VSM arm compared to the UC arm. The zero inflated model
56


was not statistically significant indicating that there were no differences between study arms in the probability of having zero chance of having an emergency department visit (p=0.29). This indicates the intervention may only have increased used of emergency department encounters in patients already using emergency department services.
Table 9: Difference in Healthcare Utilization Between VSM and UC Study Arms From Birth Up to 395 Days of Life_______________________________________________________
Type of Visit IRR Cl pvalue
Well Child VisitA 1.018061 0.940-1.101 0.6565
Other OutpatientAA 1.06844 0.898-1.271 0.4552
Emergency DepartmentAAA
count model 5.562235 1.711-18.079 0.0043*
zero inflated model 9.967203 0.1409-705.129 0.29
lnpatientAAA
count model 0.270495 0.046-1.569 0.1449
zero inflated model 0.322227 0.038-2.763 0.3017
EmailAA 0.967539 0.743-1.260 0.8069
TelephoneAA 2.284164 1.976-2.640 0.2638
*statistically significant difference Apoisson regression AAnegative binomial regression AAAzero inflated poisson regression
Emergency Department Visits
I did not expect to see an increase in emergency department visits in the intervention arm. Therefore, additional steps were taken to describe the data and determine if the increase was due to adverse outcomes due to vaccination. All emergency department visits were categorized into health conditions based on the primary diagnosis of the visit. The health condition categories were determined by a Pediatric expert. Table 10 describes the frequency of each health condition by study arm and the difference in frequencies between study arms. A negative value indicates a higher percentage of events in the specific health category in the Usual Care arm. The most notable differences occurred in the vaccine related events (15.9% more in UC than VSM) and Gastrointestinal events (15.5% more in VSM than UC). However, the vaccine related events were in the opposite direction of what would be expected. There
57


were more children vaccinated in the VSM arm than the UC arm providing more opportunity for adverse outcomes. Additionally, diarrhea, a potential gastrointestinal event related to vaccines, only occurred once among the 15 events.156 As the resource evaluation was a secondary analysis and increase in ED visits for the VSM arm do not link to vaccine related events, rather appear to have a protective effect for vaccine events, the difference in study arms may be due to a spurious finding. However, there may also something about the intervention that increases a participant’s likelihood for bringing their child into the emergency department. Additional ED visits were incorporated into the cost analysis using a moderate complexity Emergency Department visit encounter cost. However, to address the potential for spurious findings, cost results will also be reported excluding Emergency Department visits.
Table 10: Percentage of Each Primary Diagnosis in Emergency Department Visits per Study Arm and the difference between arms
Study Arm Respiratory Potentially Vacci Related Injury Newborn Conditions Allergy Gastrointestinal Poisoning Genitourologic Heart Dermatologic Hematologic Neurological
Social 41.6 5.2 9.1 3.9 3.9 20.8 2.6 2.6 0.0 3.9 1.3 5.2
Media
Usual Care 36.8 21.1 15.8 0.0 5.3 5.3 0.0 10.5 5.3 0.0 0.0 0.0
Difference 4.7 -15.9 -6.7 3.9 -1.4 15.5 2.6 -7.9 -5.3 3.9 1.3 5.2
Aim 2: Implementation Cost Effectiveness
The goal of Aim 2, was to assess the implementation cost effectiveness of the VSM intervention in terms of additional children vaccinated and pertussis cases prevented. I hypothesized, VSM would be cost savings compared to usual care.
Health Benefit
The first step of the incremental cost effectiveness analysis was to determine the health benefit of the intervention (IE). The VSM intervention administered over 3 years to 6,200 pregnant women per year in the third trimester of pregnancy, was estimated to prevent 5 total
58


pertussis cases, 1 in year 1, and 2 in years 2 and 3. There were 1,068 additional children vaccinated over the three-year time period, 214 in year 1 and 427 in years 2 and 3.
Extrapolating to a large national healthcare system, the intervention administered to 116,000 pregnant women per year is estimated to prevent 65 total cases of pertussis, 13 in year 1 and 26 in years 2 and 3 respectively. There were 19,878 additional children vaccinated over the three-year time period, 3,996 in year 1 and 7,991 in years 2 and 3. Results are presented in Table 11 below.
Table 11 VSM Intervention Health Benefit per year
Health Benefit Year 1 Implementation Year 2 Maintenance Year 3 Maintenance Total (all yrs)
Local Health System
Pertussis Cases Prevented 1 2 2 5
Additional Children Vaccinated 214 427 427 1,068
National Health System Health Benefit
Pertussis Cases Prevented 13 26 26 65
Additional Children Vaccinated 3,996 7,991 7,991 19,878
Intervention Costs
The second step of the incremental cost effectiveness evaluation evaluated the 4 components of intervention costs including intervention administration and maintenance (TC1), costs for vaccine administration (TC2), costs for additional encounters (TC3), and cost reduction due to the health benefit from pertussis cases presented (TC4).
1. Intervention administration and maintenance costs (TC1). The total cost of administering the VSM intervention was $223,998. Intervention costs were highest in year 1 with implementation of the intervention ($81,206), lowest in year 2 ($69,723). In year 3, intervention costs were reduced due to less time required for question response as repeat question increased. However, a design update increased total intervention costs slightly above year 2 ($73,068). Highest intervention costs were attributable to question response ($43,576.68) and website moderation ($87,116). Results are found in Table 12 below.
59


Table 12 Intervention Administration Implementation and Maintenance Costs
Intervention Costs Year 1 Year 2 Year 3 Total Costs
Implementation $18,169.86 $0 $0 $18,169.86
Question Response $11,651.34 $17,112.69 $14,812.65 $43,576.68
Newsletter $1,177.73 $1,206.00 $1,234.94 $ 3,618.67
Blogs $9,484.85 $9,712.49 $9,945.59 $29,142.93
Forum (updates) $392.58 $402.00 $411.65 $1,206.22
Chat $3,721.49 $3,810.81 $3,902.27 $11,434.57
Vaccine news monitoring $1,570.31 $1,608.00 $1,646.59 $4,824.90
Website moderation $28,352.83 $29,033.30 $29,730.10 $87,116.23
Content updates $523.44 $536.00 $548.86 $1,608.30
Addressing website functionality issues $1,946.61 $1,993.32 $2,041.16 $5,981.09
Website maintenance $3,893.21 $3,986.65 $4,082.33 $11,962.18
Hosting $275.00 $275.00 $275.00 $825.00
Design $47.00 $47.00 $47.00 $141.00
Design Updates $0 $0 $4,390.00 $4,390.00
Total costs for VSM administration $81,206.25 $69,723.26 $73,068.14 $223,997.65
Staff time required for the intervention is represented in Table 13. Total hours for the intervention manager was lowest in year 1 (691 hours), increasing in year 2 to address the growing number of questions (761 hours) and decreasing again in year 3 from an increased number of repeat questions (721 hours). Total hours for the intervention manager represents approximately 1/3 of a full-time staff member’s time. Time from a technology staff was highest in year for implementation (137 hours) decreasing to a consistent 87 hours in years 2 and 3. Hours for a vaccines expert (Pediatrician) were consistently 62 hours per year.
Table 13 Intervention Staff Hours for Implementation and Maintenance of the Intervention
Staff Year 1 Year 2 Year 3
Intervention Manager Total 691 761 721
Vaccine Expert Total 62 62 62
Tech Person Total 137 87 87
Total questions requiring response was 192 in year 1 and 336 in years 2 and 3. Of the questions requiring response, the total number of repeat questions increased each year from 15 in year 1 to 69 in year 2 and 99 in year 3.
60


Extrapolating to a large national health system total costs of administering the VSM intervention was $1,053,975. Unlike the local health system, intervention costs decreased over time after the initial increase from year 1 to 2 due to the influx of additional questions. In year 1 costs were estimated at $316,949 increasing in year 2 to $387,529 and decreasing in year 3 to $349,497. Results are found in Table 14 below.
Table 14 National Health System Implementation and Maintenance Costs
Intervention Costs Year 1 Year 2 Year 3 Total Costs
Implementation Costs $34,560.42 $0 $0 $34,560.42
Question Response $221,566.55 $325,421.65 $281,683.12 $828,671.32
Newsletter $1,177.73 $1,206.00 $1,234.94 $3,618.67
Blogs $10,809.80 $11,069.24 $11,334.90 $33,213.94
Forum (updates) $392.58 $402.00 $411.65 $1,206.22
Chat $3,721.49 $3,810.81 $3,902.27 $11,434.57
Vaccine news monitoring $1,570.31 $1,608.00 $1,646.59 $4,824.90
Website moderation $28,352.83 $29,033.30 $29,730.10 $87,116.23
Content updates $1,701.17 $1,742.00 $1,783.81 $5,226.97
Addressing website functionality issues $1,946.61 $1,993.32 $2,041.16 $5,981.09
Website maintenance $3,893.21 $3,986.65 $4,082.33 $11,962.18
Hosting $7,200.00 $7,200.00 $7,200.00 $21,600.00
Tailoring Plug-In $9.00 $9.00 $ 9.00 $27.00
Design $47.00 $47.00 $47.00 $141.00
Design Updates $0 $0 $4,390.00 $4,390.00
Total costs for VSM administration $316,948.71 $387,528.97 $349,496.86 $1,053,974.53
Total hours for the intervention manager was lowest in year 1 (4,010 hrs, 2 full time staff) increasing in year 2 (5,401, 2 1/2 full time staff) and decreasing again in year 3 (4649 2 1/3 full time staff). Time for a technology staff was higher in year 1 than a local health system addressing the modification for local content at 212 hours. In years 2 and 3 technology staff time was consistent with local health system requirements at 87 hours per year. Hours for a vaccines expert (Pediatrician) were 62 hours per year, consistent with estimates for a local health system. Results are found in Table 15 below.
61


Table 15 National Health System Intervention Staff Hours for Implementation and Maintenance of the Intervention
Staff Year 1 Year 2 Year 3
Intervention Manager Total 4010 5401 4649
Vaccine Expert Total 62 62 62
Tech Person Total 212 87 87
There were a total of 3,642 question requiring a response in year 1 and 6,380 in years 2 and 3. The total number of repeat questions increased annually from 285 in year 1 to 1,310 in year 2, and 1,877 in year 3. Highest costs were attributable to question response ($828,671) and website moderation ($87,116).
Reference Case Comparison local content costs. In the large national healthcare system comparison group, additional costs were incorporated into the intervention implementation and maintenance for local content tailoring and updates. The costs were $12,690 in year 1 including $10,250 in build and local content updates for implementation and $2,440 for intervention maintenance including local content blogs and tailoring plug ins. Intervention maintenance costs were $2,499 and $2,559 in year 2 and 3 respectively. Results are found in Table 16 below.
Table 16: National Health System Local Content Updates Cost and Staff Hours
Implementation Year 1 Year 2 Year 3 Total
Local Content Costs $10,550.75 $2,511.68 $2,571.75 $2,633.25 $18,267.43
Intervention Manager hours 72 38 38 38 115
Tech Person hours 75
2. Vaccine administration and adverse outcome costs (TC2). Vaccine administration and adverse outcome costs are found in Table 17 below. Additional vaccines administered in the local KPCO population were estimated to include 2 additional vaccine adverse outcomes for a medically attended fever treated in an outpatient setting. There were none in year 1 and 1 in years 2 and 3. Treating adverse outcomes was a total of $158.60 increasing from no cost in year 1 to $79.30 in years 2 and 3. Total costs for administration of the vaccines were $922,827 for all 3 years.
62


Extrapolating to a large national healthcare system, additional vaccines were estimated to include 120 vaccine adverse outcomes, 18 in year 1 and 51 in years 2 and 3. Adverse outcomes included outpatient and ED visits for medically attended fever and outpatient, ED, and inpatient visits for local reactions. Treating vaccine adverse outcomes was a total of $29,563, increasing from $1,212 in year 1 to $14,176 in years 2 and 3. Total costs for administration of vaccines was $16,977,330 for all 3 years.
Table 17: Vaccine Administration and Adverse Outcomes Cost
Year 1 Year 2 Year 3 TotaI Costs
Local Health system
Vx Administration Costs $184,566 $369,131 $369,131 $922,828
Vx Adverse Outcome Costs $0 $79 $79 $159
Total Vx admin and Adverse Outcomes $184,566 $369,211 $369,211 $922,988
National Health system
Vx Administration Costs $3,389,553 $6,799,107 $6,779,107 $16,947,767
Vx Adverse Outcome Costs $1,212 $14,176 $14,176 $29,563
Total Vx Admin and Adverse Outcomes $3,390,765 $6,793,282 $6,793,282 $16,977,330
3. Additional or reduced encounters (TC3). A two part model was used to estimate the number of increased ED visit encounters. The first part was a logit model comparing those with no ED events (ED count=0) to those with any ED events (ED count >1). This provides an estimate for the increase in having an ED visit. The second stage used a zero truncated poisson where patients with no ED visits were excluded. This model provided an estimate for the increase in ED visits if one had occurred. The coefficient and standard error for the employment and study arm were input in a Monte Carlo simulation. A population of 12,400 people was input in the simulation where half were in the UC arm and half were in the VSM arm. There were 10,000 simulations done to estimate the number of ED visits per arm. The average number of visits in a population of 6,200 participants with the estimated confidence interval was determined for each study arm and found in Table 18 below. There were an estimated 1,424 additional ED visits in the KPCO population of 6,200 due to the intervention. The cost for a
63


moderate complexity ED visit was applied to the additional ED visits and incorporated into intervention costs.
Table 18: Mean # of ED Visits per Study Arm Over 10,000 Simulations
Study Arm Mean Confidence Interval
VSM 2304.3 871.0-890.0
Usual Care 880.4 2232.1-2381-5
Difference 1,423.9
Costs found in Table 19 below were estimated at $334,260, ranging from $66,852 in year 1 to $133,704 in years 2 and 3. When extrapolating to a large national healthcare system, ED costs were $7,935,601, increasing from $1,587,120 in year 1 to $3,174,241 in years 2 and
3.
Table 19 VSM Intervention Costs for Increase in ED Visits per Year
Health Benefit Year 1 Year 2 Year 3 Total
Implementation Maintenance Maintenance (all yrs)
Local Health System $66,852 $133,704 $133,704 $334,260
National Health System $1,587,120 $3,174,241 $3,174,241 $7,935,601
4. Health outcome cost benefit (TC4). Costs associated with pertussis cases prevented are found in Table 20 below. Total health system cost savings due to cases of pertussis prevented were $47,375. In year 1 the cost savings was $9,475 increasing to $18,950 in years 2 and 3. Extrapolating to a large national healthcare system, total health system cost savings due to cases of pertussis prevented were $615,875. In year 1 cost savings was $123,175 increasing to 24,350 in years 2 and 3.
Table 20: Health System cost reduction for Pertussis Cases Prevented
Health System Year 1 Year 2 Year 3 Total
Implementation Maintenance Maintenance (all yrs)
Local Health System Cost Savings $9,475 $18,950 $18,950 $47,375
National Health System Costs savings $123,175 $246,350 $246,350 $615,875
Implementation Cost Effectiveness
Incremental cost effectiveness was evaluated in two ways and results are found in Table 21 below. First, incremental costs incorporating a cost reduction due to the pertussis health
64


benefit, and cost increase due to vaccine administration and adverse outcome costs, additional ED visits, and intervention implementation and maintenance were $1,433,870. Costs were lowest in year 1 ($323,149) with intervention administration and vaccine administration costs increasing in years 2 and 3 with the double the number of impacted children. Total incremental cost effectiveness over the three year time period per pertussis case prevented were $286,774 and $1,343 per additional child vaccinated.
Second, incremental costs limited to intervention administration and health benefit, excluding ED visits, vaccine administration, and vaccine adverse outcomes, had a total cost of $176,623 over the three-year time period. Incremental cost effectiveness per pertussis case prevented was $35,325 and cost per additional child vaccinated was $165.
Table 21: VSM Intervention Implementation Cost Effectiveness
Year 1 Year2 Year3 Total
All intervention components
1-Total Intervention costs $323,149 $553,688 $557,033 $1,433,870
Cost per pertussis case prevented $323,149 $276,844 $278,516 $286,774
Cost per additional child vaccinated $1,513 $1,296 $1,304 $1,343
2-Intervention administration costs $71,731 $50,773 $54,118 $176,623
Cost per pertussis case prevented $71,731 $25,387 $27,059 $35,325
Cost per additional child vaccinated $336 $119 $127 $165
Extrapolating to a large national health system results are found in Table 22 below. Total intervention costs were $25,351. Annual costs were lowest in year 1 ($5,171,659) nearly doubling in years 2 and 3 with double the children impacted by the intervention. The total cost per pertussis case prevented was $390,016 and $1,269 per additional child vaccinated. Incremental cost limited to costs of administering the intervention were $438,099. Using limited incremental costs, incremental cost effectiveness was $6,740 per pertussis case prevented and $22 per additional child vaccinated.
65


Table 22: VSM Intervention Implementation Cost Effectiveness in National Healthcare System
Year 1 Year2 Year3 Total
1-Total Intervention costs $5,171,659 $10,108,702 $10,070,670 $25,351,031
Cost per pertussis case prevented $397,820 $388,796 $387,333 $390,016
Cost per additional child vaccinated $1,294 $1,265 $1,260 $1,269
2-Intervention administration costs $193,774 $141,179 $103,147 $438,099
Cost per pertussis case prevented $14,906 $5,430 $3,967 $6,740
Cost per additional child vaccinated $49 $18 $13 $22
Sensitivity Analysis
Sensitivity Analyses were conducted to test the model assumptions and assess costs in varied implementation circumstances.
Implementation Cost Effectiveness Excluding ED visits
Incremental cost effectiveness excluding ED visits costs (TC2) are presented in Table 23 below. The total costs were $1,099,609, a reduction of $334,261. Costs per pertussis case prevented were reduced by $66,852 and $313 per additional child vaccinated. When extrapolated to a national health system, costs were reduced by $7,935,601, $122,086 per pertussis case prevented ad $397 per additional child vaccinated.
Table 23 Sensitivity Analysis: Total Incremental Cost Effectiveness Excluding ED Visits
Total costs Cost per pertussis Case Prevented Cost per additional Child vaccinated
Local Base Case $1,433,870 $286,774 $1,343
Local Excluding ED visits $1,099,609 $219,922 $1,030
Difference $334,261 $66,852 $313
National Base Case $25,351,031 $390,016 $1,269
National Excluding ED visits $17,415,429 $267,929 $872
Difference $7,935,601 $122,086 $397
Pertussis Outbreak Year
Results of sensitivity analysis for a pertussis outbreak year are presented in Table 24
below. In a pertussis outbreak year, the total number of pertussis cases prevented increased from 5 to 8 cases, 2 in year 1, and 3 in year 2 and 3 respectively. Total health benefit costs
66


increased $28,425. This represents a lower implementation cost-effectiveness than the original model, reflecting additional cost-effectiveness with an increased number of pertussis cases prevented. This reflects a reduction of $111,093 per pertussis case prevented. When modeling cost effectiveness using only intervention administration and health benefit reduction costs, the costs per pertussis case prevented were reduced to $18,524 from $35,324.
Extrapolating to a large national health system, in an outbreak year the intervention prevents an additional 10 pertussis cases. This is a total health system cost savings of $1,004,350. This reflects a reduction of $53,265 per pertussis case prevented in total intervention cost effectiveness. When modelling cost effectiveness using only intervention administration and health benefit reduction costs, the intervention reduces costs in an intervention year saving $94,750, with a cost effectiveness of $4,578 per pertussis case prevented and $17.19 per additional child vaccinated.
Table 24: Health Benefit and Costs in a Pertussis Outbreak Year Compared to the Local and National Base Case
Pertussis Cases Prevented Health system cost Savings Intervention Administration and Health Benefit Costs Total Intervention Costs
Local Base Case 5 $47,375 $176,623 $1,433,870
Local Pertussis Outbreak 8 $75,800 $148,198 $1,405,445
Difference 3 $28,425
National Base Case 65 $615,875 $438,100 $25,351,031
National Pertussis Outbreak 75 $710,625 $343,350 $25,256,280
Difference 10 $94,750
Vaccination Coverage Variation
Results of the vaccination coverage variation sensitivity analysis are presented in Table 25 below. With vaccination coverage ranging from 52.8% to 95.1% in the local health care system, pertussis cases prevented range from 3-10 with additional children vaccinated ranging from 496-2,126. The largest impact on cost is for vaccine administration ranging from $428,259-$1,836,974. Estimated costs from the base case model fall in the middle of this range. Vaccination coverage has a large impact on total intervention costs ranging from $986,517-
67


$2,301,389 based on population level of vaccination coverage. However, a majority of the total intervention costs are linked to vaccine administration. When limiting costs to intervention administration and health benefit costs, total costs ranged from $129,247-$195,572.
Extrapolating to a large national health system, the number of pertussis cases prevented ranges from 25-177 and additional children vaccinated from 9,260-39,757. The larger variation in pertussis cases prevented from the national sample increases the range of potential costs for health benefit savings ($236,875-$1,677,075). The wide variation in additional children vaccinated has substantial impact on the total intervention costs ranging from $8.6 million to $33.2 million. As vaccine administration has the largest impact on cost, when limiting total costs to intervention administration and health benefit, costs range from a cost savings of $623,100 to a total cost of $817,099. The cost savings highlights the importance of maintaining high vaccination rates to maintain health system costs.
Table 25: Health Benefit and Costs in Low and High Vaccination Coverage Compared to the Local and National Base Case
Pertussis Cases Prev Add I Children Vx Health system Cost Savings Vx Admin Vx Adverse Events Intervention Admin & Health Benefit Costs Total Costs
Local Base Case 5 1068 $47,375 $922,828 $159 $176,623 $1,433,870
52.8% Vax 10 2126 $94,750 $1,836,974 $907 $129,247 $2,301,389
95.1% Vax 3 496 $28,425 $428,259 $0 $195,572 $986,517
National Base Case 65 19,978 $615,875 $16,977,330 $29,563 $438,100 $25,351,031
52.8% Vax 177 39,757 $1,677,075 $33,726,363 $100,354 -$623,100* $33,203,617
95.1% Vax 25 9260 $236,875 $7,855,280 $2,771 $817,099 $8,675,151
‘indicates cost savings
Variation in Pertussis disease costs
Results of the pertussis disease cost variation are presented in Table 26 below. Applying
a range of pertussis treatment costs of $3,772-$18,721, Health Benefit Cost Savings ranged from $21,605-$107,575. This impacted the overall costs t o range from $1,373,670-$1,459,640. When limiting to intervention administration and health benefit costs, the range of
68


costs was $116,423-$202,393, where high disease costs reduced cost per pertussis case prevented to $23,284 from $35,325.
Health Benefit Cost variation had a larger impact when extrapolating to a large national healthcare system. Health Benefit Cost Savings ranged from $280,865 -$1,398,475. When limiting to intervention administration and health benefit costs, higher pertussis disease treatment costs led to a cost savings of $344,500, reducing cost per pertussis case prevented from $6,740 to a savings of $5,300.
Table 26: Health System and Intervention Costs using Low and High Range of Pertussis Disease Costs
Health System Cost Savings Intervention Admin and Health Benefit Costs Total Intervention Costs
Local Base Case $47,375 $176,623 $1,433,870
Local Low Disease costs $21,605 $202,393 $1,459,640
Local High Disease Costs $107,575 $116,423 $1,373,670
National Base Case $615,875 $438,100 $25,351,031
National Low Disease costs $280,865 $773,110 $25,686,041
National High Disease Costs $1,398,475 -$344,500* $24,568,431
*negative values indicate a cost reduction Intervention Adjustments
Multiple adjustments were made to intervention cost estimates to evaluate the costs under different circumstances. Results are found in Table 27 and 28 below.
Staffing variation. Staffing variation had a large impact on intervention costs. Intervention administration costs ranged from $162,024-$450,409. Notably, a Pediatrician managing the intervention doubled the intervention costs, but a local community health specialist reduced costs by a little over $60,000. The impact of staffing on intervention administration had larger impact in the national scale were intervention administration costs ranged from $653,697-$2,516,323. When modelling cost effectiveness limited to intervention administration and health benefit cost reduction, a community health specialist managing the intervention was cost savings in year 2 and 3 saving between $12,374 and $32,225 respectively. This was an total incremental cost effectiveness of $582 per pertussis case
69


prevented and $1.89 per additional child vaccinated. A research assistant managing the intervention was also cost savings in year 3 of the intervention saving $3,255.
Table 27 Sensitivity Analysis: Staffing Variation in Intervention Manager Position
Intervention Intervention Admin Total
Admin Costs and Health Benefit Intervention
Costs Costs
Local Base Case $223,998 $176,623 $1,433,870
Local Pediatrician $450,409 $403,034 $1,660,281
Local Community Health Specialist $162,024 $114,649 $1,371,896
Local Research Scientist $266,429 $219,054 $1,476,301
Local Research Assistant $175,286 $127,911 $2,042,169
National Base Case $1,053,975 $438,100 $25,351,031
National Pediatrician $2,516,323 $1,900,448 $26,813,379
National Community Health Specialist $653,697 $37,821 $24,950,753
National Research Scientist $1,328,031 $712,156 $25,625,088
National Research Assistant $739,356 $123,481 $25,036,412
Number of questions. When the number of questions addressed by the intervention manager were reduced by half and doubled, intervention administration costs ranged from ($202,209-$267,574). The cost adjustment was due to approximately 80 hours of intervention manager work a year when the number of questions were halted and between 178 and 256 hours a year when questions were doubled. When extrapolating to a large nation health system intervention administration costs ranged from $636,638-$1,882,646. When questions were halted, total intervention administration and health benefit costs were $23,736 reflecting a cost savings in year 2 (-$21,5313) and year 3 (-$37,695). Staffing requirements at the national level would require just over one full time person to manage the intervention when questions were halted and nearly 5 full time employees if questions were doubled.
Response time. Similar variation was found when the time to respond to questions varied. Intervention administration costs ranged from $200,159-$342,965. The cost adjustment was due to a reduction of 94-139 hours per year of the intervention manager with an increased speed in response time and 504-700 additional hours per year when response time was slowed
70


down. In both cases, this is less than a full-time staff person, but could be as much as an additional 1/3 of a full-time employee.
When extrapolating to a large national healthcare system total intervention administration costs ranged from $600,653-$3,316,298. This cost adjustment was due to a decrease in annual intervention manager hours of 1778-2650 when response time sped up and 9,581-13,293 additional hours per year when response time slowed down. This reflects approximately one full time staff reduction with fast response time and between 4 and 7 additional staff members for a slow response time. When response time was sped up, incremental cost effectiveness measured in terms of intervention administration and health benefit was cost savings, reducing costs by $15,222 over the three year time period. The savings were concentrated in years 2 (-$36,342) and 3 ($-56,267) respectively.
Moderation time (double). As moderation is a daily activity a consistent increase in a moderation time increased total intervention costs. The total intervention administration costs when moderation time increased to $311,114 reflecting 434 additional hours a year. Extrapolating to a large national healthcare system, was $1,141,091 total intervention administration costs reflecting the same additional hours a year as the local health system since moderation was estimated using a daily number of hours, not based on population size.
Table 28 Sensitivity Analyses: Intervention Administration Assumption Variation
Analysis Intervention Admin Costs Intervention Admin and Health Benefit Costs Total Intervention Costs Range of Annual Intervention Manager Hours
Local Base Case $223,998 $176,623 $1,433,870 691-760
Local Questions Halted $202,209 $154,834 $1,412,082 602-633
Local Questions Doubled $267,574 $220,199 $1,477,447 869-1016
Local Fast Response Time $200,159 $152,784 $1,410,031 597-621
Local Slow Response Time $342,965 $295,590 $1,552,837 1195-1460
Local Moderation Time Doubled $311,114 $263,739 $1,568,361 1125-1194
71


Table 28 Cont’d
Analysis Intervention Admin Costs Intervention Admin and Health Benefit Costs Total Intervention Costs Range of Annual Intervention Manager Hours
National Base Case $1,053,975 $438,100 $25,351,031 4009-5401
National Questions Halted $639,639 $23,764 $24,936,695 2316-2972
National Questions Doubled $1,882,646 $1,266,771 $26,179,702 7396-10258
National Fast Response Time $600,653 -$15,222* $24,897,709 2231-2751
National Slow Response Time $3,316,298 $2,700,423 $27,613,354 13590-18694
National Moderation Time Doubled $1,141,091 $525,216 $25,438,147 4443-5833
72


CHAPTER V
CONCLUSIONS
This dissertation evaluated the implementation dimension of the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework for the Vaccine Social Media (VSM) intervention through resource requirements and implementation cost effectiveness. The only change in health system utilization identified was an increase in emergency department (ED) encounters due to intervention. When evaluating costs over a three-year time period, the intervention was estimated to increase vaccination by 1,068 additional children and prevent 5 pertussis cases in a local health system. In a national health system, VSM resulted in vaccination of an additional 19,978 children and prevent 65 pertussis cases. The intervention was not found to be cost savings compared to usual care due to pertussis cases prevented in either a local or national healthcare system. Total incremental costs of administration of the intervention and health benefit over the three-years were approximately $175,000 in a local health system and $430,000 in a national health system.
The goal of the RE-AIM implementation measure is to assess how well an intervention can be implemented across a variety of environments. Challenges to implementation including high costs and increased ED visit resource requirements could be a deterrent for implementation of the VSM intervention. However, the goal of RE-AIM is to provide a comprehensive and realistic assessment of intervention implementation.17 In addition to the resource and cost results found in this evaluation, potential adopters also need to consider important contextual factors specific to vaccination behavior and the VSM intervention to inform their decisions.
Implementation Resource Evaluation
Changes in utilization due to an intervention is an important consideration for potential health system adopters as it impacts both costs and staffing. However, when looking at
73


vaccination behavior, the nature of the results makes it difficult to accurately estimate the impact to a specific health system. Thus, changes in utilization due to the VSM intervention need to be interpreted with caution.
For example, the lack of differences in well child visits results do not necessarily indicate there were no changes in well child visit utilization due to the intervention. One national study found 1,399 distinct vaccination patterns in parents not following the recommended schedule.2 This represents use of the healthcare system in a variety of ways. Some parents use the system following normal well child recommendations and just delay or don’t get some of their vaccines. Other parents either do not use the health system for well child visits receiving no vaccines, or use it more frequently to obtain vaccines following a different schedule.1 Thus, an increase in vaccination could result in a variety of changes to well child visit utilization including an increase, decrease, or any combination of the two. Changes in well child visit utilization due to increased vaccination are therefore more likely to be health system specific depending on how the vaccine hesitant population of that health system uses preventative care.
Similarly, there are three potential situations that could have led to the increase in emergency department (ED) visits due to the VSM intervention. First, an increase in vaccination could lead to additional vaccine adverse outcomes, increasing the number of ED visits in the intervention study arm group. However, diagnosis codes for the ED visit do not indicate a link to vaccine adverse outcomes, making this outcome unlikely. Second, evidence suggests anxiety mediates the relationship between health information seeking and utilization.157 As such, a health information intervention describing potential adverse outcomes of vaccines may increase anxiety for the health of the child leading to increased ED visits. This circumstance is also unlikely as there are limited vaccine adverse outcomes and the communication approach used for the intervention focused on addressing parents’ concerns and avoiding scare tactics. Lastly, an increase in ED visits could be a random effect of conducting multiple secondary analyses on
74


outcomes that were not the intention of the intervention.158 This is the most likely outcome, but further testing of changes in ED utilization due to the VSM intervention in a variety of environments is needed to confirm this hypothesis. Thus, health systems do not necessarily need to play for additional ED visits due to VSM intervention implementation.
Implementation Cost Evaluation: Public Health Impact
In addition to considering the challenges to interpretation of resource requirements, the public health benefit of vaccination was minimized in the implementation cost-effectiveness analysis and is an important context consideration for potential adopters. Vaccinations are one of the most significant public health contributions in the last century.159 By maintaining high levels of vaccination, vaccine preventable diseases have limited ability to spread.5 The success of vaccines has minimized the direct impact an increase in vaccination can have on costs to one health system. However, if vaccination rates are not maintained, the impact on both disease spread and costs are substantial. An economic analysis of the full vaccine schedule found it prevented 20 million cases of disease and saved $13.5 billion in direct costs.7
This dissertation cost evaluation was designed to assess cost a health system could reasonably expect to see with the known outcomes of the intervention, the known population, in one endemic vaccine preventable disease. Because of the focused cost evaluation, two important implications for health systems were underestimated and should be included in adoption considerations.
First, I limited the spread of disease in this analysis to the birth cohort as estimating the impact in other populations would be difficult to do with accuracy. Thus, the health benefit in the community, family members, and patients sitting in a waiting room were not considered.
Vaccine preventable diseases are highly contagious and can quickly spread to all members of a community. For example, an estimated 90% of people close to a person with measles who are not immune from the disease will become infected.128 In less than two months in 2019, there have been five outbreaks with 127 cases of measles across 3 different states.160 These cases
75


can spread quickly in communities with low immunization rates impacting community members across all age groups.161162 The sensitivity analyses conducted help inform scenarios where additional members of the population are impacted. Both the outbreak scenario and decreased vaccination coverage rates highlights a larger population impact in situations where a vaccine preventable disease occurs in a health system community. Results indicated a cost reduction in both health systems with cost savings nationally when vaccination rates are low.
Second, the vaccination schedule prevents twelve diseases. In this evaluation I only assessed the currently endemic disease of pertussis. Pertussis is the disease most likely to impact a health system annually. However, as just described with the measles outbreak, other vaccine preventable diseases could experience outbreaks, particularly where vaccination rates are low. The sensitivity analysis during an outbreak year provides an example of costs associated with increased disease burden, indicating limited costs locally and nationally. The overall benefit of the VSM intervention increases when considering all vaccine preventable diseases.
When assessing costs from a health system perspective, understanding the expected costs is critical to adoption. The cost estimates in this analysis provide an accurate assessment of costs for the VSM intervention with outcomes likely to impact a health system. Most of the diseases vaccinated against in the first year of life would not be expected to occur as a regular expected cost within a health system, with the exception of pertussis.93 It is difficult to predict when disease outbreaks may occur and how they will spread. However, the sensitivity analyses highlight the importance of increasing vaccination rates and maintaining a high level of vaccination coverage. This allows for maintenance of costs within a health system and limit the ability for outbreaks to occur. Improved vaccination rates impact the entire health system population and community population health for the better.
76


Adoption Considerations for Health System Adopters
Taking the important public health benefit of the intervention costs and considerations of limitations with health system utilization findings into account, the RE-AIM implementation value of the VSM intervention increases for potential health system adopters. However, another important component of RE-AIM includes consideration of potential variation in intervention adoption.42 Three important adoption considerations for health system adopters were highlighted in this cost-effectiveness analysis of VSM.
The first consideration is vaccine administration costs. When evaluating all costs associated with the intervention there were just under $1.5 million for three years in the local setting and over $25 million dollars nationally. A majority of these costs were associated with vaccine administration. This overestimates the cost of vaccine administration as most health systems negotiate lower rates for bulk orders and the affordable care act requires insurance to pay for preventative services including vaccinations, limiting the direct impact on health systems.146163 Additionally, increased vaccination is a health system benefit improving Healthcare Effectiveness Data and Information Set (HEDIS) scores that many health systems will view as a benefit, lowering the concern for these costs. However, there are situations where this cost could be an important consideration of potential adopters. Evidence suggests, even with the Affordable Care Act provisions for insurance coverage of vaccination, some private practices have chosen not to provide vaccinations as the costs associated with stocking adequate vaccines and limited reimbursement becomes too costly for the provider.164 Increasing childhood vaccination may not be a priority of these health systems.
The second consideration is staffing the intervention. A majority of the intervention administration costs of VSM are associated with the human interaction intervention requirements (responding to questions). Adopters may consider excluding interaction to limit intervention costs. However, when interaction was not included, the VSM intervention did not significantly increase vaccination.4 This indicates interaction was a central component to
77


intervention effectiveness making it a critical part of implementation. The sensitivity analyses highlight ways health system adopter can manage resource requirements for the intervention while still offering interaction. Specifically, costs were greatly impacted by the staff salary and response time, making it important to find a balance between vaccine knowledge and salary level. I found the most cost-effective implementation approach used someone with vaccine expertise to respond in a short amount of time, but also had a lower salary than a pediatrician. For example, using a nurse or community health specialist with expertise in vaccination can help manage intervention costs. In a national health system, this implementation approach was cost savings.
The third consideration is the technical capacity of the website. The intervention implementation build assumed designing a secure login system for accessing the website.
Health systems with limited technical capacity may consider removing the secure login approach. This is likely to increase reach of the intervention, as website login reduces the number of people willing to access the intervention. However, removing secure login could also reduce the effectiveness and increase resource requirements. Analysis of the VSM intervention discourse found that having a secure login program with communication between patient and healthcare providers/vaccine experts, facilitates an environment for civil discourse.165 This may have been an important part of effectiveness of the trial by maintaining an environment where participants felt safe from negative discourse that is often associated with online vaccination websites.166 As such, staffing requirements for moderation were limited for the VSM intervention. This implies the secure program may limit reach, but minimizes costs associated with moderation and maintains civil discourse important for intervention effectiveness. Thus, the secure login is more likely to increase the RE-AIM value of the intervention despite the decrease in reach.
Requiring staffing resources, technical expertise and capacity to include secure login may limit certain health systems from being able to offer the VSM intervention. Specifically,
78


small private health systems may not have this capacity. However, the design and cost estimates in this analysis were built with a model for a grouped health system program in mind. The national health systems costs could be translated to a group of providers instead of one national health system. In a model such as this, a central location, such as a county health department, could administer the intervention. Log in information could be provided to each participating health system tailoring information to their vaccine formulary and office location. Total costs could either be paid for by the county health department or shared between participating health systems. The national costs could be applied to this scenario to estimate total administration costs. Thus, the intervention approach is not limited to larger health systems for use in a variety of environments.
VSM RE-AIM Value and Next Steps
Previous evaluation of the VSM study indicated the vaccine hesitant population was reached by the intervention, and the intervention led to a clinically meaningful increase in vaccination.4 Particularly when taking the overarching public health benefit of vaccination into consideration, the resource impact and cost of the VSM intervention is low. This is particularly true, in a large national healthcare system that could see cost savings from implementation of the intervention in multiple situations such as outbreak situations, when vaccine preventable disease treatment costs are high, and when implemented with staff that has a lower salary and high vaccine knowledge.
The RE-AIM results of the VSM intervention known at this point indicate a high value of the intervention with potential for successful implementation in a variety of settings. However, adoption and maintenance could not be measured at this time as the intervention was conducted by the research staff. Understanding potential barriers and facilitators to adoption and maintenance would provide additional information on the potential success of the intervention in a variety of settings.
79


These additional outcomes directly reflect the next stage of implementation research. There are five phases (T0-T4) advancing scientific discoveries into practice.14 Phase 1 (TO), phase 2 (T1), and phase 3 (T2) are focused on discovering the problem and testing efficacy of the new intervention. The last 2 phases, phase 4 (T3) and phase 5 (T4) are the focus of implementation and dissemination science. T3 research, such as this dissertation, is designed to increase uptake and implementation of the intervention. The next stage, T4 research, evaluates effectiveness and cost-effectiveness of an intervention in real world settings. Specifically, T4 evaluates the cost and effectiveness in a variety of environments after implementation. This provides the value of the intervention when implemented in the real world.
It offers a more detailed look at potential variation in implementation that may occur impacting effectiveness and cost. To fully understand the impact of the VSM intervention in real world application, the next step to this dissertation is a T4 research evaluation of VSM.
Informing Implementation Science
In addition to the information specific to the RE-AIM value of the VSM intervention, this evaluation expands on our current understanding of the resources and costs required for implementation of effective online interventions. Implementation is often found to be difficult when additional staffing requirements are needed.43167 However, as found with the VSM intervention, online interventions that include human interaction are often more effective than online interventions without interaction, making staffing requirements an important consideration.4 52 Online human interaction is also shown to be as effective as face to face interaction.53 This presents an opportunity for interactive online behavior change interventions to shift staffing resources instead of just requiring additional staff. In the case of VSM, pediatricians report spending at least 10 minutes on average discussing vaccines with vaccine hesitant parents.58 We were unable to measure this change in provider time due to the interruption in care when measuring this outcome. However, VSM could have alleviated some of this time from pediatricians allowing additional time in well child visits to focus on other
80


important health topics. As healthcare costs continue to increase, finding new approaches to shift staffing requirements could present opportunities for cost savings or more efficient healthcare administration.168 Limitations
This dissertation is limited in several ways. First, the VSM trial is complete, and therefore the data is limited to what was already collected as part of the intervention trial. This limits the data available to examine changes in staffing requirements and cost estimates. Specifically, a direct measures of intervention impact on provider time with patients in the well child visits was not captured limiting the ability to evaluate impact on pediatrician time.
Second, the cost-effectiveness evaluation is limited to implementing the intervention as designed. In real-world application, modifications to the intervention may be required to make it compatible with different environments. Subaims and sensitivity analysis and were done to address this concern, but additional revisions could be made that were not considered. Further evaluation using T4 research methods would assist in examining effectiveness and cost of the intervention after implementation in a real world setting.
Third, public sector costs for vaccine administration were not represented in this implementation cost-effectiveness analysis. This was done to reflect the circumstances at KPCO and KP national. Additionally, this was a conservative estimate as the private sector costs are higher than the public sector.7125
Fourth, the cost analysis is likely to underestimate the cost-effectiveness of the intervention. The VSM intervention will impact multiple infectious disease in addition to pertussis.57 Similarly, the vaccination coverage in infants may also impact other populations including family and community members that are not assessed.128 Also, increased vaccination could impact the health outcomes of the population over a longer period than assessed in this evaluation. While these are important considerations, the data available for this analysis limit the accuracy of evaluations incorporating these considerations. Additionally, the goal of this study
81


was to provide costs accurately reflecting what a health system could regularly expect and many of the other cost considerations could not be predicted or assumed to occur on a regular basis.
Fifth, the full value of the intervention using all five measures of RE-AIM cannot be assessed as the research team implemented the intervention. This limits the ability to identify potential barriers to adoption and maintenance. Reach, effectiveness, and implementation have or will be assessed with this dissertation to provide the RE-AIM value of the intervention known at this point.
Sixth, additional factors important to implementation, such as characteristics of potential adopters,34 will also not be assessed as part of this dissertation. While this may miss important factors for health care decision makers, the goal of this dissertation is to focus on the RE-AIM dimensions that are often requested by adopters but rarely evaluated, cost and staffing impacts.
Seventh, assessing changes in encounters were limited to the study population. Differences in encounter utilization may not have been detectable. For example, with the small proportion of inpatient encounters (11 in the usual care group), the analysis could only detect a difference if the encounter rate almost doubled. This is a limitation of the data available and would need to be further explored in a larger population implementing the intervention to confirm findings.
Lastly, the VSM trial was conducted in one integrated healthcare system. Results may not be representative of implementation in different health systems. For example, email encounters are not available in all health systems, so the interventions impact on utilization could look different in these environments. Additionally, effectiveness of the trial and cost estimates will also vary by location. Sensitivity analyses were conducted to address some of these issues in the cost evaluation, however there will be variation that was not considered.
82


Conclusion
In conclusion, vaccines are important public health intervention that requires effective intervention strategies to reach herd immunity and prevent the spread of infectious disease. The VSM intervention is a effective intervention strategy for health systems to reach and maintain high levels of vaccination in children at a low implementation and maintenance cost. Implementing the VSM intervention requires staff resources and is most cost effective when applied on a large health system national scale with staff who have lower salaries, but a high vaccination knowledge to efficiently address vaccine questions. Next steps are to evaluate the intervention cost, effectiveness, and adoption as implemented in a variety of settings.
83


REFERENCES
1. Dempsey AF, SchafferS, Singer D, Butchart A, Davis M, Freed GL. Alternative vaccination schedule preferences among parents of young children. Pediatrics.
2011 ;128(5):848-856.
2. Glanz JM, Newcomer SR, Narwaney KJ, Hambidge SJ, Daley MF, Wagner NM,
McClure DL, Xu S, Rowhani-Rahbar A, Lee GM. A population-based cohort study of undervaccination in 8 managed care organizations across the United States. JAMA pediatrics. 2013; 167(3):274-281.
3. Omer SB, Salmon, D.A., Orenstein, W.A., deHart, P., Halsey, N.. Vaccine Refusal, Mandatory Immunization, and the Risks of Vaccine-Preventable Diseases. New England Journal of Medicine. 2009;360(19):1981-1988.
4. Glanz JM, Wagner NM, Narwaney KJ, Kraus CR, Shoup JA, Xu S, O’Leary ST, Omer SB, Gleason KS, Daley MF. Web-based Social Media Intervention to Increase Vaccine Acceptance: A Randomized Controlled Trial. Pediatrics. 2017:e20171117.
5. Plans-Rubio P. Evaluation of the establishment of herd immunity in the population by means of serological surveys and vaccination coverage. Hum Vaccin Immunother. 2012;8(2):184-188.
6. Hill HA, Elam-Evans LD, Yankey D, Singleton JA, Kang Y. Vaccination Coverage Among Children Aged 19-35 Months -United States, 2016. Morbidity and Mortality Weekly Report (MMWR) Centers for Disease Control and Prevention. November 3,
2017;66(42):1171-1177.
7. Zhou F, Shefer A, Wenger J, Messonnier M, Wang LY, Lopez A, Moore M, Murphy TV, Cortese M, Rodewald L. Economic evaluation of the routine childhood immunization program in the United States, 2009. Pediatrics. 2014;133(4):577-585.
8. Sadaf A, Richards JL, Glanz J, Salmon DA, Omer SB. A systematic review of interventions for reducing parental vaccine refusal and vaccine hesitancy. Vaccine. 2013;31(40):4293-4304.
9. Jarrett C, Wilson R, O'Leary M, Eckersberger E, Larson HJ, Sage Working Group on Vaccine Hesitancy. Strategies for addressing vaccine hesitancy - A systematic review. Vaccine. 2015;33(34):4180-4190.
10. Salmon DA, Dudley MZ, Glanz JM, Omer SB. Vaccine hesitancy: causes, consequences, and a call to action. Vaccine. 2015;33:D66-D71.
11. Assessing the state of vaccine confidence in the United States: recommendations from the National Vaccine Advisory Committee. Public Health Reports. 2015; 130:573.
12. Brownson RC, Colditz GA, Proctor EK. Dissemination and implementation research in health: translating science to practice. Oxford University Press; 2017.
13. Westfall JM, Mold J, Fagnan L. Practice-based research—“Blue Highways” on the NIH roadmap. Jama. 2007;297(4):403-406.
84


14. Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C. National Institutes of Health approaches to dissemination and implementation science: current and future directions. American journal of public health. 2012;102(7):1274-1281.
15. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM Framework: A Systematic Review of Use Over Time. American Journal of Public Health. 2013;103(6):e38-e46.
16. Glasgow RE, Vogt TM, Boles SM. Evaluating the Public Health Impact of Health Promotion Interventions: The RE-AIM Framework. American Journal of Public Health.
1999; 89(9): 1322-1327.
17. Glasgow RE. RE-AIM: Implementation of Health Behavior Interventions. 2018; http://www.re-aim.org/about/what-is-re-aim/implementation/. Accessed June 13, 2018.
18. Centers for Disease Control and Prevention. Achievements in public health 1900-1990. Impact of Vaccines Universally Recommended for Children—United States, 1900-1999. MMWR Morb Mortal Wkly Rep. 1999(48):243-248.
19. Ten Great Public Health Achievements -- United States, 2001-2010. Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report (MMWR) Centers for Disease Control and Prevention. 2011 ;60(19):619-623.
20. Dempsey AF, SchafferS, Singer D, Butchart A, Davis M, Freed GL. Alternative vaccination schedule preferences among parents of young children. Pediatrics.
2011 ;128(5):848-856.
21. Glanz JM, Newcomer SR, Narwaney KJ, Hambidge SJ, Daley MF, Wagner NM,
McClure DL, Xu S, Rowhani-Rahbar A, Lee GM, Nelson JC, Donahue JG, Naleway AL, Nordin JD, Lugg MM, Weintraub ES. A population-based cohort study of undervaccination in 8 managed care organizations across the United States. JAMA pediatrics. 2013; 167(3):274-281.
22. Glanz JM, McClure DL, Magid DJ, Daley MF, France EK, Hambidge SJ. Parental refusal of varicella vaccination and the associated risk of varicella infection in children. Archives of pediatrics & adolescent medicine. 2010;164(1):66-70.
23. Glanz JM, McClure DL, Magid DJ, Daley MF, France EK, Salmon DA, Hambidge SJ. Parental refusal of pertussis vaccination is associated with an increased risk of pertussis infection in children. Pediatrics. 2009; 123(6): 1446-1451.
24. Glanz JM, McClure DL, O'Leary ST, Narwaney KJ, Magid DJ, Daley MF, Hambidge SJ. Parental decline of pneumococcal vaccination and risk of pneumococcal related disease in children. Vaccine. 2011;29(5):994-999.
25. Glanz JM, Narwaney KJ, Newcomer SR, Daley MF, Hambidge SJ, Rowhani-Rahbar A, Lee GM, Nelson JC, Naleway AL, Nordin JD, Lugg MM, Weintraub ES. Association between undervaccination with diphtheria, tetanus toxoids, and acellular pertussis (DTaP) vaccine and risk of pertussis infection in children 3 to 36 months of age. JAMA pediatrics. 2013;167(11):1060-1064.
85


26. Feiken DR, Lezotte, D.C., Hamman, R.F., Salmon, D.A., Chen, R.T., Hoffman, R.E. Vaccine Refusal, Mandatory Immunization, and the Risks of Vaccine-Preventable Diseases. Journal of the American Medical Association. 2000;284(24):3145-3150.
27. Robison SG, Groom H, Young C. Frequency of alternative immunization schedule use in a metropolitan area. Pediatrics. 2012;130(1):32-38.
28. Phadke VK, Bednarczyk RA, Salmon DA, Omer SB. Association between vaccine refusal and vaccine-preventable diseases in the United States: a review of measles and pertussis. JAMA. 2016;315(11):1149-1158.
29. Lo NC, Hotez PJ. Public Health and Economic Consequences of Vaccine Hesitancy for Measles in the United States. JAMA pediatrics. 2017;171(9):887-892.
30. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearbook of medical informatics 2000: Patient-centered systems. 2000.
31. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med.
2007;32(5 Suppl):S119-126.
32. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clinical and translational medicine. 2013;2(1):10.
33. Glasgow RE, Lichtenstein E, Marcuse AC. Why Don’t We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition. American Journal of Public Health. 2003;93(8):1261-1267.
34. Dearing JW. Applying Diffusion of Innovation Theory to Intervention Development. Res Soc WorkPract. 2009;19(5):503-518.
35. Rhodes J, Whitney C, Ritzwoller DP, Glasgow RE. Stakeholder perspectives on costs and resource expenditures: tools for addressing economic issues most relevant to patients, providers, and clinics. Translational behavioral medicine. 2018.
36. Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB, Brito F, Blackman KC, Zoellner JM, Hill JL, Almeida FA, Glasgow RE, Estabrooks PA. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev. 2015;4:155.
37. Neta G, Sanchez MA, Chambers DA, Phillips SM, Leyva B, Cynkin L, Farrell MM, Heurtin-Roberts S, Vinson C. Implementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions. Implementation Science. 2015;10(1):4.
38. Mensah GA, Engelgau M, Stoney C, Mishoe H, Kaufmann P, Freemer M, Fine L. News from NIH: a center for translation research and implementation science. Translational behavioral medicine. 2015;5(2):127-130.
39. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015; 10:53.
86


40. Rogers EM. Diffusion of innovations. New York: Free Press of Glencoe; 1962.
41. Rogers EM. Diffusion of Innovations. Fifth ed. New York, NY: The Free Press; 2003.
42. Glasgow RE. What is RE-AIM? http://www.re-aim.org/about/what-is-re-aim/. Accessed June 27, 2018.
43. Sharp ND, Pineros SL, Hsu C, Starks H, Sales AE. A qualitative study to identify barriers and facilitators to implementation of pilot interventions in the Veterans Health Administration (VHA) Northwest Network. Worldviews on Evidence-Based Nursing. 2004;1(2):129-139.
44. Raghavan R. The Role of Economic Evaluations in Dissemination and Implementation Research. In: Brownson RC, ColditzGA, Proctor EK, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. Second ed. New York, NY: Oxford University Press; 2018:89-199.
45. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing Behavioral Interventions: A Practical Guide to Enhance Translation. Annals of Behavioral Medicine. 2009;37(2):218-227.
46. Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med. 2009;38(1):40-45.
47. Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res. 2004;6(4):e40.
48. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12(1):e4.
49. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cognitive behaviour therapy. 2009;38(4): 196-205.
50. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression: systematic review. Journal of medical Internet research. 2009; 11 (2).
51. Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. American journal of preventive medicine. 2007;33(4):336-345. e316.
52. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from critical interpretive synthesis. Telemedicine and e-Health. 2012;18(2):137-144.
53. Santarossa S, Kane D, Senn CY, Woodruff SJ. Exploring the Role of In-Person Components for Online Health Behavior Change Interventions: Can a Digital Person-to-Person Component Suffice? Journal of medical Internet research. 2018;20(4).
87


54. Enthoven AC. Integrated delivery systems: the cure for fragmentation. American Journal of Managed Care. 2009;15(12):S284.
55. Vogt TM, Lafata JE, Tolsma DD, Greene SM. The role of research in integrated health care systems: the HMO Research Network. The Permanente Journal. 2004;8(4):10.
56. United States Census Bureau: Quick Facts Colorado. 2017; https://www.census.gov/quickfacts/fact/map/CO/PSTQ45217. Accessed June 4, 2018.
57. Workgroup BFPS. 2015 Recommendations for Preventive Pediatric Health Care Committee on Practice and Ambulatory Medicine and Bright Futures Periodicity Schedule Workgroup. Pediatrics. 2015;136(3).
58. Kempe A, O’Leary ST, Kennedy A, Crane LA, Allison MA, Beaty BL, Hurley LP,
Brtnikova M, Jimenez-Zambrano A, Stokley S. Physician response to parental requests to spread out the recommended vaccine schedule. Pediatrics. 2015:peds. 2014-3474.
59. Kennedy A, Basket M, Sheedy K. Vaccine attitudes, concerns, and information sources reported by parents of young children: results from the 2009 HealthStyles survey. Pediatrics. 2011:peds. 2010-1722N.
60. Glanz JM, Wagner NM, Narwaney KJ, Shoup JA, McClure DL, McCormick EV, Daley MF. A mixed methods study of parental vaccine decision making and parent-provider trust. AcadPediatr. 2013;13(5):481-488.
61. Lieu TA, Zikmund-Fisher BJ, Chou C, Ray GT, Wittenberg E. Parents’ perspectives on howto improve the childhood vaccination process. Clin Pediatr. 2017;56(3): 238-246.
62. Shoup JA, Wagner NM, Kraus CR, Narwaney KJ, Goddard KS, Glanz JM. Development of an Interactive Social Media Tool for Parents With Concerns About Vaccines. Health education & behavior: the official publication of the Society for Public Health Education. 2014.
63. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues in clinical neuroscience. 2011; 13(2):217.
64. Gillings D, Koch G. The application of the principle of intention-to-treat to the analysis of clinical trials. Drug Information Journal. 1991 ;25(3):411-424.
65. Centers for Disease Control and Prevention 2017 Childhood Diphtheria toxoid, Tetanus toxoid, acellular Pertussis (DTaP) Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imz-managers/coverage/childvaxview/data-reports/dtap/reports/2017.htmI. Accessed February 13, 2019.
66. HEDIS Measures, http://www.ncqa.org/hedis-quality-measurement/hedis-measures. Accessed 5/3/2018, 2018.
67. Summary Table of Measures, Product Lines and Changes. HEDIS. 2018;2.
88


68. Legare F, Ratte S, Gravel K, Graham ID. Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals' perceptions. Patient Educ Couns. 2008;73(3):526-535.
69. Selden TM. Compliance with well-child visit recommendations: evidence from the Medical Expenditure Panel Survey, 2000-2002. Pediatrics. 2006;118(6):e1766-e1778.
70. Centers for Disease Control and Prevention National Hospital Ambulatory Medical Care Survey: 2015 Emergency Department Summary Tables. 2015;
https://www.cdc.gov/nchs/data/nhamcs/web tables/2015 ed web tables.pdf. Accessed February 13, 2019.
71. Weiss AJ, Elixhauser A. Overview of hospital stays in the United States, 2012. HCUP Statistical Brief# 180. Agency for Healthcare Research and Quality, Rockville, MD October 2014.
72. Daley MF, Narwaney KJ, Shoup JA, Wagner NM, Glanz JM. Addressing Parents’ Vaccine Concerns: A Randomized Trial of a Social Media Intervention. American Journal of Preventive Medicine. 2018.
73. Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. Jama. 2012;308(19).
74. Meng D, Palen TE, Tsai J, McLeod M, Garrido T, Qian H. Association between secure patient-clinician email and clinical services utilisation in a US integrated health system: a retrospective cohort study. BMJ open. 2015;5(11):e009557.
75. Billups SJ, Moore LR, Olsone KL, Magid DJ. Cost-effectiveness evaluation of a home blood pressure monitoring program. Am J Manag Care. 2014;20(9):e380-e387.
76. Anand SG, Feldman MJ, Geller DS, Bisbee A, Bauchner H. A content analysis of e-mail communication between primary care providers and parents. Pediatrics.
2005; 115(5): 1283-1288.
77. Hoonakker PL, Carayon P, Cartmill RS. The impact of secure messaging on workflow in primary care: results of a multiple-case, multiple-method study. International journal of medical informatics. 2017; 100:63-76.
78. Krist AH, Woolf SH, Bello GA, Sabo RT, Longo DR, Kashiri P, Etz RS, Loomis J, Rothemich SF, Peele JE. Engaging primary care patients to use a patient-centered personal health record. The Annals of Family Medicine. 2014;12(5):418-426.
79. Gorman BK, Braverman J. Family structure differences in health care utilization among US children. Social Science & Medicine. 2008;67(11):1766-1775.
80. Alio AP, Salihu HM. Maternal determinants of pediatric preventive care utilization among blacks and whites. Journal of the National Medical Association. 2005;97(6):792.
81. Fiscella K, Franks P, Gold MR, Clancy CM. Inequality in quality: addressing socioeconomic, racial, and ethnic disparities in health care. Jama. 2000;283(19):2579-2584.
89


82. Garrido T, Raymond B, Wheatley B. Lessons From More Than a Decade in Patient Portals. Health Affairs Blog. April 7, 2016.
83. BuchmuellerTC, Grumbach K, Kronick R, Kahn JG. Book review: The effect of health insurance on medical care utilization and implications for insurance expansion: A review of the literature. Medical care research and review. 2005;62(1):3-30.
84. Newhouse JP. Consumer-directed health plans and the RAND Health Insurance Experiment. Health Affairs. 2004;23(6): 107-113.
85. Wharam JF, Landon BE, Galbraith AA, Kleinman KP, Soumerai SB, Ross-Degnan D. Emergency department use and subsequent hospitalizations among members of a high-deductible health plan. Jama. 2007;297(10):1093-1102.
86. Bennett GG, Glasgow RE. The Delivery of Public Health Interventions via the Internet: Actualizing Their Potential. Annual Review of Public Health. 2009;30(1):273-292.
87. Lee GM, Murphy TV, Lett S, Cortese MM, Kretsinger K, Schauer S, Lieu TA. Cost effectiveness of pertussis vaccination in adults. American journal of preventive medicine. 2007;32(3):186-193. e182.
88. Ehreth J. The global value of vaccination. Vaccine. 2003;21(7-8):596-600.
89. Ribisl KM, Leeman J, Glasser AM. Pricing health behavior interventions to promote adoption: lessons from the marketing and business literature. American journal of preventive medicine. 2014;46(6):653-659.
90. Owens DK, Siegel JE, Sculpher MJ, Salomon JA. Designing a Cost-Effectiveness Analysis. In: Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, eds. Cost-Effectiveness in Health and Medicine. Second ed. New York, NY: Oxford University Press; 2017:75-104.
91. Kroger A, Duchin J, Vazquez M. Best practices guidance of the Advisory Committee on Immunization Practices (ACIP). Atlanta, GA: US Department of Health and Human Services, CDC; 2017.
92. Adams DA, Jajosky RA, Ajani U, Kriseman J, Sharp P, Onwen D, Schley AW, Anderson WJ, Grigoryan A, Aranas AE. Summary of notifiable diseases-United States, 2012. MMWR Morbidity and mortality weekly report. 2014;61(53):1-121.
93. Centers for Disease Control and Prevention. National Notifiable Disease Surveillance System, 2017 Annual Tables of Infectious Disease Data. Atlanta, GA. . CDC Division of Health Informatics and Surveillance. 2018.
94. Liang JL, Tiwari T, Moro P, Messonnier NE, Reingold A, Sawyer M, Clark TA.
Prevention of Pertussis, Tetanus, and Diphtheria with Vaccines in the United States: Recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recommendations and Reports. 2018;67(2):1.
90


95. Cortese MM, Baughman AL, Zhang R, Srivastava PU, Wallace GS. Pertussis hospitalizations among infants in the United States, 1993 to 2004. Pediatrics.
2008; 121 (3):484-492.
96. Tanaka M, Vitek CR, Pascual FB, Bisgard KM, Tate JE, Murphy TV. Trends in pertussis among infants in the United States, 1980-1999. Jama. 2003;290(22):2968-2975.
97. Centers for Disease Control and Prevention 2017 Final Pertussis Surveillance Report. 2018.
98. Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, Kuntz KM, Meltzer DO, Owens DK, Prosser LA, Salomon JA, Sculpher MJ, Trikalinos TA, Russell LB,
Siegel JE, Ganiats TG. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA. 2016;316(10):1093-1103.
99. Shoup JA, Madrid C, Koehler C, Lamb C, Ellis J, Ritzwoller DP, Daley MF. Effectiveness and Cost of Influenza Vaccine Reminders for Adults With Asthma or Chronic Obstructive Pulmonary Disease. American Journal of Managed Care. 2015;21(7):e405-e413.
100. Ritzwoller DP, Toobert D, Sukhanova A, Glasgow RE. Economic analysis of the Mediterranean Lifestyle Program for postmenopausal women with diabetes. The Diabetes Educator. 2006;32(5):761-769.
101. Ritzwoller DP, Sukhanova AS, Glasgow RE, Strycker LA, King DK, Gaglio B, Toobert DJ. Intervention costs and cost-effectiveness for a multiple-risk-factor diabetes selfmanagement trial for Latinas: economic analysis of jViva Bien! Translational Behavioral Medicine. 2011; 1 (3):427-435.
102. Neumann PJ, Russell LB, Siegel JE, Prosser LA, Krahn M, Mandelblatt JS, Daniels N, Gold MR. Using Cost-Effectiveness Analysis in Health and Medicine: Experiences since the Original Panel. In: Newmann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, eds. Cost-Effectiveness in Health and Medicine. Second ed. New York, NY: Oxford University Press; 2017:1-37.
103. Glasgow RE. RE-AIM: REACH of Health Behavior Interventions. 2018; http ^/re-aim.orq/about/what-is-re-aim/reach/. Accessed May, 16, 2018.
104. Glasgow RE. RE-AIM: EFFECTIVENESS/EFFICACY of Health Behavior Interventions. 2018; http://re-aim.org/about/what-is-re-aim/effectiveness-or-efficacy/. Accessed May 16, 2018, 2018.
105. Glasgow RE. Adoption of Health Behavior Interventions. 2018; http://www.re-aim.org/about/what-is-re-aim/adoption/. Accessed May 16, 2018, 2018.
106. Glasgow RE. Maintenance of Health Behavior Interventions. 2018; http://www.re-aim.org/about/what-is-re-aim/maintenance/. Accessed May 16, 2018, 2018.
107. Luman ET, Barker LE, Shaw KM, McCauley MM, Buehler JW, Pickering LK. Timeliness of childhood vaccinations in the United States: days undervaccinated and number of vaccines delayed. Jama. 2005;293(10):1204-1211.
91


108. The Health Care Systems Research Network Virtual Data Warehouse. http://www.hcsm.orq/en/Tools%20&%20Materials/VDW/VDWDataModel/VDWSpecificati ons.pdf. Accessed February 13, 2019.
109. American Academy of Pediatrics Coding for Pediatric Prventive Care, 2016. Bright Futures. 2016.
110. What Statistical Analysis Should I use? Statistical Analyses Using SAS: UCl_A statistical Analysis Consulting Group, https://stats.idre.ucla.edu/sas/whatstat/what-statistical-analvsis-should-i-usestatistical-analvses-usinq-sas/. Accessed October 1, 2018.
111. Simon TD, Cawthon ML, Stanford S, Popalisky J, Lyons D, Woodcox P, Hood M, Chen AY, Mangione-Smith R. Pediatric medical complexity algorithm: a new method to stratify children by medical complexity. Pediatrics. 2014;133(6):e1647-e1654.
112. Opel DJ, Taylor JA, Zhou C, Catz S, Myaing M, Mangione-Smith R. The relationship between parent attitudes about childhood vaccines survey scores and future child immunization status: a validation study. JAMA pediatrics. 2013;167(11):1065-1071.
113. Opel DJ, Taylor JA, Mangione-Smith R, Solomon C, Zhao C, Catz S, Martin D. Validity and reliability of a survey to identify vaccine-hesitant parents. Vaccine.
2011 ;29(38):6598-6605.
114. Acock AC. Working with missing values. Journal of Marriage and family. 2005;67(4):1012-1028.
115. Gardner W, Mulvey EP, Shaw EC. Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. Psychological bulletin.
1995;118(3):392.
116. Hall DB. Zero-inflated Poisson and binomial regression with random effects: a case study. Biometrics. 2000;56(4): 1030-1039.
117. Vuong ST, Ko KC. A novel approach to protocol test sequence generation. Paper presented at: Global Telecommunications Conference, 1990, and
Exhibition.'Communications: Connecting the Future', GLOBECOM'90., IEEE1990.
118. PASS 15 Power Analysis and Sample Size Software (2017). NCSS, LLC. Kaysville,
Utah, USA, ncss.com/software/pass.
119. Gold MR. Cost-effectiveness in health and medicine. Oxford university press; 1996.
120. Misegades LK, Winter K, Harriman K, Talarico J, Messonnier NE, Clark TA, Martin SW. Association of childhood pertussis with receipt of 5 doses of pertussis vaccine by time since last vaccine dose, California, 2010. Jama. 2012;308(20):2126-2132.
121. Colorado Department of Public Health and Environment Numbers and Rates of Reported Pertussis Cases by Age Group, Colorado, 1/1/2017 - 12/31/2017. 2017; https://drive.qooqle.com/drive/folders/1K34Z4CwGuWckY1C8lqo7YvlFxq84GZiG. Accessed February 6, 2019, 2019.
92


122. Bureau of Labor Statistics. 2018; https://www.bls.gov/mwe/. Accessed June 3, 2018.
123. Economic News Release: Table2. Civilian workers, by occumpational and industry group. Bureau of Labor Statistics. September 2018.
124. WordPress.org Our Mission. 2018; https://wordpress.org/about/. Accessed June 3, 2018.
125. Centers for Disease Control and Prevention. Vaccines for Children Program (VFC) price list. 2018; https://www.cdc.gov/vaccines/programs/vfc/awardees/vaccine-management/price-list/index.html. Accessed February 13, 2019.
126. Colorado Department of Health Care Policy and Financing: Provider Rates & Fee Schedule. 2017; https://www.colorado.gov/pacific/hcpf/provider-rates-fee-schedule. Accessed June 26, 2018.
127. Chen RT, Hadler SC, Terracciano GJ, Tuttle J, Watson JC. Update; vaccine side effects, adverse reactions, contraindications, and precautions: recommendations of the Advisory Committee on Immunization Practices (ACIP).
128. Hamborsky J, Kroger A, Wolfe C. Epidemiology and Prevention of Vaccine-preventable Diseases: The Pink Book: Course Textbook. Public Health Foundation; 2015.
129. Shinefield HR, Black S, Ray P, Chang I, Lewis N, Fireman B, Hackell J, Paradiso PR, Siber G, Kohberger R. Safety and immunogenicity of heptavalent pneumococcal CRM 197 conjugate vaccine in infants and toddlers. The Pediatric infectious disease journal. 1999;18(9):757-763.
130. Lieu TA, Ray GT, Black SB, Butler JC, Klein JO, Breiman RF, Miller MA, Shinefield HR. Projected cost-effectiveness of pneumococcal conjugate vaccination of healthy infants and young children. Jama. 2000;283(11):1460-1468.
131. Belongia EA, Irving SA, Shui IM, Kulldorff M, Lewis E, Yin R, Lieu TA, Weintraub E, Yih WK, Li R. Real-time surveillance to assess risk of intussusception and other adverse events after pentavalent, bovine-derived rotavirus vaccine. The Pediatric infectious disease journal. 2010;29(1):1-5.
132. Zhou F, Bisgard KM, Yusuf HR, Deuson RR, Bath SK, Murphy TV. Impact of universal Haemophilus influenzae type b vaccination starting at 2 months of age in the United States: an economic analysis. Pediatrics. 2002;110(4):653-661.
133. Zhou F, Santoli J, Messonnier ML, Yusuf HR, Shefer A, Chu SY, Rodewald L, Harpaz R. Economic evaluation of the 7-vaccine routine childhood immunization schedule in the United States, 2001. Archives of pediatrics & adolescent medicine. 2005;159(12):1136-1144.
134. Jackson LA, Yu O, Nelson JC, Dominguez C, Peterson D, Baxter R, Hambidge SJ, Naleway AL, Belongia EA, Nordin JD. Injection site and risk of medically attended local reactions to acellular pertussis vaccine. Pediatrics. 2011;127(3):e581-e587.
93


Full Text

PAGE 1

ASSESSING THE VALUE OF THE VACCINE SOCIAL MEDIA INTERVENTION THROUGH THE RE-AIM FRAMEWORK IMPLEMENTATION DIMENSION by NICOLE MARIE WAGNER B.A., University of Colorado Boulder, 2003 M.P.H., San Diego State University, 2006 A thesis submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfillment of the requirements for the degree of Doctor of Philosophy Health and Behavioral Sciences Program 2019

PAGE 2

ii This thesis for the Doctor of Philosophy degree by Nicole Marie Wagner has been approved for the Health and Behavioral Sciences Program by Patrick Krueger, Chair Sara Yeatman Jason M. Glanz Debra P. Ritzwoller Date: May 18, 2019

PAGE 3

iii Wagner, Nicole Marie (PhD, Health and Behavioral Sciences) Assessing the Value of the Vaccine Social Media Intervention Through the RE-AIM Framework Implementation Dimension Thesis directed by Associate Professor Patrick Krueger ABSTRACT A growing number of parents are delaying and refusing vaccination. This is increasing the risk for contracting vaccine preventable diseases. 1-3 The Vaccine Social Media (VSM) intervention trial was found to increase the number of children up to date on vaccinations compared to usual care. 4 Factors important for implementation decision makers were not assessed including the cost-effectiveness of the intervention and changes to health system resource requirements. Despite substantial health and economic benefits due to increased vaccination, 5-7 the lack of information on these important implementation factors prevented intervention implementation. In this dissertation, I used the RE-AIM framework implementation dimension as a guide to assess factors important for health systems adopters including the health system resource requirements and cost effectiveness of the VSM intervention. I evaluated the health system resource requirements through differences in the child’s health system encounters between the VSM and usual care study arms using a poisson regression analysis. I assessed differences in well child visits, inpatient, emergency department, phone, and email encounters. I evaluated cost of the intervention using an incremental cost-effectiveness evaluation per additional child vaccinated and pertussis cases prevented. The only identified difference in health system resource requirements was a 5.56 increased rate of emergency department visits associated with the intervention (p=.0043). The incremental cost effectiveness evaluation limited to costs for administration of the intervention and health benefit cost reduction, were $165 per additional child vaccinated and $35,325 per

PAGE 4

iv pertussis case prevented. Costs reduced further when the model was extrapolated to a national health system at $22 per additional child vaccinated and $6,740 per pertussis case prevented. The intervention was not cost savings compared to usual care. However, taking the significant population health impact of additional children vaccinated into consideration, costs and resource requirements for administration of the VSM intervention are minimal. This indicates a costeffective vaccine hesitancy intervention for many health systems. The form and content of this abstract are approved. I recommend its publication. Approved: Patrick Krueger

PAGE 5

v ACKNOWLEDGEMENTS First, I’d like to thank my dissertation committee for their continued support and scientific expertise throughout my education. My dissertation chair, Patrick Krueger and committee member Sara Yeatman provided an immense amount of time and methodologic expertise as I went through multiple topic areas of focus. Their patience, support, and critical feedback were essential in my scientific development and direction of career focus. Debra Ritzwoller’s support and guidance in implementation science and cost evaluation were critical to the success of my dissertation and greatly expanded my scientific skillset. Jason Glanz, inspired me to continue my education, taught me a rigorous process of developing and testing effective health intervention programs, and continues to guide my development and growth as a scientist. I would also like to thank multiple people at Kaiser Permanente Colorado (KPCO) central to the completion of this dissertation. Stan Xu and Komal Narwaney were always available last minute to answer questions and provided critical development of my analytic skills essential to this dissertation. Kris Wain and Christina Clarke greatly assisted in the development of my SAS coding skillset needed to disentangle health system encounters. Matt Daley’s pediatric and vaccine expertise were essential to the interpretation of findings. The Vaccine Social Media (VSM) team, including Jo Ann Shoup, Courtney Kraus, Chris Boyd, and Kathy Gleason, contributed to the development of time estimates and translation of findings. Lastly, I’d like to thank my family. My parents taught me that perseverance always pays off. My husband, Eric, consistently supported and encouraged me throughout this process without a single complaint despite the many hours away. My son Frankie’s constant smile always inspired me to persevere and be the very best mom and scientist I could be. *This research has been approved by COMIRB (Protocol Number: 18-2147) and by the Kaiser Permanente Colorado Institutional Review Board (IRBNet ID #s: 1224885-34).

PAGE 6

vi TABLE OF CONTENTS CHAPTER I.INTRODUCTION AND SPECIFIC AIMS .................................................................................. 1 Background ............................................................................................................................................. 1 Specific Aims ........................................................................................................................................... 2 II. BACKGROUND AND REVIEW OF THE LITERATURE ......................................................... 4 Vaccine Hesitancy .................................................................................................................................. 4 Implementation Challenges .................................................................................................................. 4 Implementation Theory .......................................................................................................................... 5 Addressing Vaccine Hesitancy: The Vaccine Social Media Project .............................................. 7 RE-AIM Implementation Value ........................................................................................................... 10 RE-AIM: Reach, Adoption and Maintenance ................................................................................... 18 Background Overview ......................................................................................................................... 19 III. METHODS ...........................................................................................................................21 Review of Aim 1 .................................................................................................................................... 21 Aim 1 Follow-up Period ....................................................................................................................... 22 Aim 1 Study Population ....................................................................................................................... 22 Aim 1 Measures .................................................................................................................................... 22 Aim 1 Missing Data .............................................................................................................................. 25 Aim 1 Analytic Plan .............................................................................................................................. 25 Aim 1 Power .......................................................................................................................................... 27 Review of Aim 2 .................................................................................................................................... 27 Aim 2 Analytic Plan .............................................................................................................................. 28 Aim 2 Follow-up Time Period ............................................................................................................. 30 Aim 2 Study Population ....................................................................................................................... 30 Aim 2 Measures .................................................................................................................................... 30 Aim 2c Reference Case Comparison ................................................................................................ 47 Sensitivity Analysis ............................................................................................................................... 48 IV. RESULTS ............................................................................................................................52 Aim 1 Review ........................................................................................................................................ 52 Aim 1: Study Population ...................................................................................................................... 53 Aim 1 Results ........................................................................................................................................ 55 Aim 2: Implementation Cost Effectiveness ...................................................................................... 58

PAGE 7

vii Sensitivity Analysis ............................................................................................................................... 66 V. CONCLUSIONS ...................................................................................................................73 Implementation Resource Evaluation ............................................................................................... 73 Implementation Cost Evaluation: Public Health Impact ................................................................. 75 Adoption Considerations for Health System Adopters ................................................................... 77 VSM RE-AIM Value and Next Steps ................................................................................................. 79 Informing Implementation Science .................................................................................................... 80 Limitations ............................................................................................................................................. 81 REFERENCES .........................................................................................................................84 APPENDIX ................................................................................................................................97

PAGE 8

1 CHAPTER I INTRODUCTION AND SPECIFIC AIMS Background A growing number of parents are delaying and refusing vaccination for their children. This is increasing the risk for contracting vaccine preventable diseases. 1-3 There is a call to action in the public health community to develop interventions to address this growing health concern. 8-11 Researchers at Kaiser Permanente Colorado recently (KPCO) conducted a pragmatic intervention trial, the Vaccine Social Media project (VSM). The trial was found to increase the number of children up to date on vaccinations by 5.9 percentage points compared to usual care. 4 This increase in vaccination could have substantial population health and economic benefits. 5-7 However, factors important for implementation decision makers were not assessed, including the cost-effectiveness of the intervention and changes to health system resource requirements through health system utilization. Without assessing the VSM intervention’s full value, including the cost and resource implications, health plan decision makers were unable to appropriately weigh the pros and cons of the intervention. Thus, movement toward implementation of the program was stalled. Costs and health system resource requirements not assessed in the VSM trial are common barriers to implementation for many effective interventions. 12 There is a growing number of interventions found to be effective at positively impacting health behavior, but very few are adopted, successfully implemented, or maintained in the health care settings. 13,14 Reach, effectiveness, adoption, implementation and maintenance (RE-AIM) is an implementation and dissemination framework designed to assist in the implementation process by evaluating the value of health interventions. 15,16 The implementation dimension of RE-AIM focuses on factors contributing to successful implementation of an intervention including assessment of required resources and costs. 17

PAGE 9

2 Using the RE-AIM framework as a guide, this dissertation will evaluate implementation factors of the VSM intervention trial important for health system adopters. I will evaluate the impact of the VSM intervention on health system resource requirements through an evaluation of changes in patient health system encounters that may be associated with the intervention. I will then conduct a cost-effectiveness analysis of the intervention in terms of number of pertussis cases avoided and the incremental increase in the number of children vaccinated attributable to the intervention. Specific Aims Aim 1: Resource requirements: Assess the health system resource requirements associated with the VSM intervention trial by comparing the rate of encounters (inpatient, ER, outpatient, phone, and email) for children up to 395 days old between participants in the intervention arm (VSM) and the usual care (UC) arm of the trial. Hypothesis 1: Children in the intervention (VSM) arm of the study will have higher rates of well child care encounters through 395 days of life compared to children in the UC arm. Hypothesis 2: Children in the intervention (VSM) arm of the study will have lower rates of inpatient encounters through 395 days of life compared to children in the UC arm. Hypothesis 3: Children in the intervention (VSM) arm of the study will have higher rates of email encounters through 395 days of life compared to children in the UC arm. Hypothesis 4: Children in the intervention (VSM) arm of the study will have higher rates of phone encounters through 395 days of life compared to children in the UC arm. Hypothesis 5: There will be no differences in emergency department encounters between children in the intervention (VSM) arm of the study compared to children in the UC arm.

PAGE 10

3 Aim 2: Cost: Assess implementation cost-effectiveness of the VSM intervention trial by assessing the incremental cost effectiveness ratio between the VSM intervention and Usual Care arm in a) dollars per additional child vaccinated, b) dollars per pertussis cases prevented, and c) estimating the replication incremental costs-effectiveness of the intervention when scaled to a large national managed care system. Hypothesis 6: The VSM intervention trial will be cost savings compared to usual care at preventing pertussis cases.

PAGE 11

4 CHAPTER II BACKGROUND AND REVIEW OF THE LITERATURE Vaccine Hesitancy Vaccines are known as one of the greatest public health advancements in the 20 th century. 18,19 Despite their success, a growing number of parents are choosing to delay or decline vaccinations. 3,20,21 This has been shown to significantly increase the risk for contracting and spreading vaccine preventable diseases. 3,22-28 In addition, under-vaccination has implications for health system and public health costs. A cost analysis by Zhou and colleagues found a $13.5 billion direct medical cost savings associated with following the recommended childhood vaccine schedule. 7 Similarly, it was found that a 5% decline in MMR vaccination coverage would result in an additional 2.1 million dollars in public sector costs. 29 To address this growing public health concern, there is currently a call to action in the public health community to develop interventions for health systems to reduce vaccine hesitancy. 8-11 In order to meet this call to action, implementation of effective interventions into healthcare systems needs to occur. Implementation Challenges Implementation challenges are common in intervention research. A study conducted by Balas and colleagues found that after 17 years only 14% of research has turned into practice. 30 Research in implementation science has consistently found that turning research into practice is rare, difficult, and needs research attention. 12-14 The research process is often slow and underfunded to adequately address implementation needs. 31,32 Additionally, health system adopters rarely use solely effectiveness results to make implementation decision. Resource requirements are often an important consideration. 33-35 Yet, research investigators focus on effectiveness results and rarely assess outcomes, such as cost and resources, important for adopters. 15,35,36 A recent emphasis on implementation of effective health interventions has begun to emerge. 12,14 Developing studies with implementation in the strategy has become important

PAGE 12

5 criteria for grant funding. Additionally, new funding sources to advance the dissemination and implementation scientific knowledge base have formed. 14,37,38 However, there has been less focus on identifying strategies to implement the numerous interventions that have already been conducted or were found to be effective. 12,39 Implementation Theory One of the longest standing frameworks in implementation science, is the diffusion of innovations theory, first published by Everett Rogers in 1962. 40 It outlines the steps to dissemination of innovations into practice. Rogers describes a five-stage decision making process in which dissemination of an idea occurs: knowledge, persuasion, decision, implementation and confirmation. The knowledge stage is when the adopter is first made aware of the innovation. Persuasion is the next stage when the individual actively seeks information and details. The last stage before implementation is the decision making stage when the adopter weighs the pros and cons of the intervention to accept or reject implementation of the innovation. 41 At both the second information seeking persuasion stage and the third decision making stage, detailed information about the intervention is needed. Specifically, decision makers require information on the overall value of the intervention. RE-AIM is an evaluation framework designed to assess the overall population health value of an intervention. 15,16,42 The goal of RE-AIM is to assist in identifying program elements that can improve implementation and sustained adoption of effective health interventions. 42 The framework provides key factors for evaluation that are important to intervention adopters at the persuasion and decision-making stage. There are 5 dimensions to the RE-AIM framework; reach, effectiveness, adoption, implementation, and maintenance. 15,16 All dimensions of the REAIM framework are important to evaluate the overall population health value of an intervention. However, the RE-AIM implementation dimension focuses on the factors important for health system adopters that contribute to successful implementation. 17 The RE-AIM implementation

PAGE 13

6 dimension refers to how consistently a health system can deliver a program or policy including the resources and cost required to implement the intervention. 17 The intervention impact on resource requirements is a critical component to implementation of an intervention. 15,36 A pilot intervention designed to improve management and measurement of low-density lipoprotein cholesterol (LDL-c) levels in patients with coronary heart disease were implemented in six Veterans Health Administration Northwest Network medical centers. Sharp and colleagues assessed barriers and facilitators to implementation and found resources needed for the intervention, including staff time, was a barrier to implementation. They found identifying and allocating required staffing needs was an underutilized step in the implementation process. 43 Staff implementing the intervention and stakeholders reported a realistic estimate of resource requirements was needed to adequately staff for successful implementation. As indicated by the VA study, providing accurate information on the resource requirements of an intervention gives health system adopters an opportunity to plan and implement with a greater opportunity for success. Directly related to the resource requirements, cost is often the first question asked by intervention adopters. However, due to expertise needed and the time required for assessment of long term outcomes, it is rarely evaluated. 44 Additionally, when cost is evaluated it often does not incorporates costs important for adopters such as implementation costs, relevant timeframes, and specific intervention resource and staffing requirements. 44,45 This is particularly true for online health interventions that are often assumed to reach a large population at a low cost. 31 In a review article by Tate and colleagues in 2011, only eight internet health interventions reported cost outcomes. Of these studies, only one was conducted in the United States and provided cost details required for implementation and maintenance of the program. 46 This lack of cost information for online health interventions becomes increasingly important as the evidence of online behavioral health interventions continues to grow. 47,48 Additionally, evidence suggests online interventions are more effective when they incorporate

PAGE 14

7 human interaction. 49-52 This could require additional resources. However, the evidence also suggests online interaction can be as effective as face to face interaction potentially providing opportunity for health system adopters to shift resources. 53 Limited information on implementation and maintenance costs of online interventions make it difficult for health system adopters to determine the most cost-effective approach to address behavioral health concerns. Addressing Vaccine Hesitancy: The Vaccine Social Media Project Conducting research within an integrated healthcare system is one strategy to promote implementation of research findings into practice. Integrated healthcare systems are networks of healthcare providers offering coordinated care to a patient population. 54 Intervention research within an integrated health environment requires collaboration with clinicians. This step encourages modification of interventions to work within the health system, increasing the opportunity for translation. 55 Kaiser Permanente Colorado (KPCO) is an integrated healthcare system serving approximately 600,000 members in Colorado, approximately one-fifth of the Colorado Denver Metro Area. 56 To address the growing vaccine hesitancy concern and need for effective interventions, our research team at KPCO developed a vaccine hesitancy intervention, the Vaccine Social Media project (VSM). Currently, vaccinations are recommended for children in the first-year life during wellchild visits at 2 months, 4 months, 6 months and 12 months of age as part of Bright Futures program of American Academy of Pediatrics guidelines for pediatric care. 57 Vaccines are one of many topics providers address during these visits. More than half of pediatrician’s report spending at least 10 minutes discussing vaccines with vaccine hesitant parents, limiting time available to address other health topics. 58 Physicians are a trusted source of vaccine information indicating parent’s desire for expert input. 59 However, vaccine hesitant parents also report seeking vaccine information online and begin making vaccine decision during pregnancy before the opportunity to discuss vaccines with the provider at the well child visits. 59,60 Additionally, when asked about their experiences receiving vaccines, parents reported a desire

PAGE 15

8 for vaccine information prior to well-child visits. 61 This indicates an online vaccine information resource with interactive capability allowing discussion with vaccine experts, could be an effective intervention strategy. This formulated the development of the VSM intervention. In addition to conducting research in an integrated healthcare system, using an implementation framework in the design of an intervention can also aid in the implementation process. 12 Using the RE-AIM framework as a guide in intervention development, we designed VSM using an iterative approach. Stakeholders, including pediatricians, obstetricians, and pediatric nurses, were interviewed at different stages in development to identify features of the intervention important for adoption. In agreement with the literature, providers desired an intervention that would alleviate the amount of time spent addressing vaccine questions with parents and a resource to provide patients who had vaccine questions. The VSM intervention was a web-based resource for parents designed to provide accurate, up to date vaccine information. The interactive components in the VSM arm included a chat room, discussion forum, blog, and “ask an expert” portal. The experts overseeing interaction and contributing to content included a pediatrician, a vaccine safety researcher, and a risk communication specialist. Each month, the expert team generated 1 to 2 blog posts and conducted an online chat session allowing participant to engage with the vaccine experts in real time. Expert staff responded to blog comments and would correct misinformation or respond to participant questions. Chat session transcripts were redacted and posted on the discussion forum. Participants were encouraged to ask questions privately using the “ask an expert” portal where the expert team provided personalized responses within 2 business days. The website was moderated to prevent bullying, abusive language, and disclosure of personal identifying health information. Expert staff used a communication framework in all responses to participants. 62 We conducted a randomized trial of the VSM intervention recruiting all pregnant women enrolled at Kaiser Permanente Colorado (KPCO). It was conducted as a pragmatic trial

PAGE 16

9 designed to test interventions effectiveness in real-life situations as opposed to optimal situations. 63 Participants were randomized 3:2:1 to receive the interactive website with vaccine information (VSM) (n=542), just the vaccine information website (VI) with no interactive features (n=371), or usual well child care (UC) (n=180). As a pragmatic trial, participants were not required to use the intervention, but were still included in the analysis. This more accurately reflects how the intervention would impact a population in a real-world setting. This is an intention to treat analysis which includes all participants in the trial randomized to a study arm regardless of exposure to the intervention or variation from protocol. 64 A modified intention to treat analysis was conducted comparing all participants randomized with KPCO membership through the child’s first 200 days of life regardless of exposure to the intervention. The intervention was found to be effective with a 5.9 percentage point increase in children up to date on vaccination by the 6 month well child (200 days) (OR=1.92, 95% CI, 1.07-3.47). 4 Congruent with literature indicating human interaction increases effectiveness of online interventions, up to date status did not differ statistically between the UC and VI arms. 52 Vaccination coverage is high, so a small increase in vaccination may not seem to have a large impact. However, even a very small increase in vaccination can have a large impact on the spread of disease. This is called herd immunity. 5 When the herd immunity threshold is met, the opportunity for spread of disease is greatly reduced. 6 For example, in Colorado in 2017, 87.4% of 13 month-olds were up to date on pertussis vaccination. 65 A 2.6 percentage point vaccination increase in pertussis would reach the herd immunity threshold of 90-94%. 5 Thus, even if a smaller proportion of the population is impacted, such as in only one health system, a small increase in vaccination could still impact the spread of disease in the community. Vaccination rates increased in the intervention population at the 12-15 month vaccine visit (489 days) indicating continued vaccination protection. 4 This outcome directly impacts an important measure for health systems that often leads to implementation, Healthcare Effectiveness Data and Information Set (HEDIS). HEDIS is a performance measure widely used

PAGE 17

10 by health systems to measure quality of care. 66 These measures impact the choices employers make on health systems offered in their benefits packages. 35 Increase in vaccination positively impacts two HEDIS measures, vaccination coverage and receipt of recommended preventive care (well child visits). 57,67 For smaller healthcare systems that may not see the same benefit from disease prevention, HEDIS measures provide an observable outcome with direct positive impact on a healthcare system, increasing likelihood of adoption. In addition to these observable positive outcomes within the health system, providers and pediatric leadership at KPCO remained engaged with the project throughout the trial. I provided status updates to providers annually. During data collection for the trial, providers requested written materials from the intervention they could provide to patients. At study completion, effectiveness results were presented to the pediatricians and pediatric leadership groups. Both groups indicated interest in implementation of the project. Despite the effective results and positive reaction from stakeholders, we encountered barriers to adoption. Specifically, budgets for department initiatives are limited. KPCO leadership asked for specific costs, including estimates on additional resource requirements for additional staffing requirements for the intervention and use of the health system. Without evaluating the value of the VSM intervention, including the cost and resource implications important for implementation, decision makers were unable to appropriately weigh the pros and cons of the intervention preventing implementation. RE-AIM Implementation Value RE-AIM includes assessing cost and resource factors as part of the implementation dimension, with an emphasis on identifying components of the intervention required for successful implementation in a variety of settings. 17,42 Health System Resource Requirements The resources required within a health system in terms of staffing for additional health encounters has an important impact on healthcare institutions willingness to implement. 34,68

PAGE 18

11 Children less than one year of age are frequent utilizers of the health system. Evidence suggests approximately 83.2% of children less than one year of age met recommendations for well child visits, including visits at age 2 weeks, 2 months, 4 months, 6 months, and 12 months of age. 69 Children under 1 year of age also have the highest rate of emergency department and inpatient visits among all age groups. 70,71 However, utilization of the healthcare system may vary during that time period if vaccination behavior changes. In a national study assessing healthcare utilization in vaccine hesitant parents, we found that vaccine hesitant parents are more likely to have hospital inpatient visits and less likely to have outpatient/well child visits for their child. 2 Increased inpatient stays could be related to stays associated with vaccine preventable diseases or extra caution by healthcare providers when unvaccinated children are seen in the emergency room to ensure it is not a vaccine preventable disease. Thus, I hypothesize children in the VSM arm will be less likely to use inpatient services. Parents who are vaccine hesitant have also reported lack of trust in the healthcare system. 60 This could lead to lower utilization of the healthcare system for outpatient and well child visit services as we found in the national study. Therefore, I hypothesize children in the VSM arm will be more likely to use outpatient services (well child visits). There was no statistically significant difference in emergency department visits (ED). 2 This suggest, ED visits are unlikely to change with vaccine hesitancy. However, a decrease in vaccine hesitancy could also lead to a decrease or increase in ED visits. A decrease in hesitancy and increase in vaccination could increase ED visits due to additional vaccine adverse events. Similarly, vaccine hesitant parents may be more concerned about their child for fear of a vaccine preventable disease or experience more vaccine preventable diseases increasing use of the ED. 60 Thus, a decrease in vaccine hesitancy could also decrease ED visits. Since the direction of change is unclear, I hypothesize ED visits will not change consistent with the national study. The VSM intervention increased vaccination in children and lessened parents’ concerns about vaccination, 72 but it is unclear how that change in attitudes and

PAGE 19

12 behavior impacted use of the healthcare system. These potential changes in the way patients use the healthcare system would require adjustments to staffing with direct cost implications. One of the goals of the VSM trial was to decrease the time needed from the provider to address vaccination concerns. However, studies have found that, initially, online communication with providers may have the opposite impact. A study at KPCO found that patients’ access to secure email communication with the provider initially increased use of clinical services including office visits and telephone encounters. 73,74 Technology based intervention trials have also been shown to increase email encounters with providers. We found in a home blood pressure monitoring study that used an online blood pressure tracking system, significantly greater email (median 5 vs 1) encounters during the 6month study period. 75 This increase did not correspond with the expected decrease in office visits, thus increasing the workload for clinic staff. Providers are a key source of vaccine information for vaccine hesitant parents. 59 Parents seek information on vaccination prior to the well child visits, 61 and email communications with pediatricians were shown to be used frequently in pediatrics for non-urgent medical questions. 76 Thus, email communications could be used as a potential resource to validate or further explore vaccine concerns. As such, I hypothesize children in the VSM arm will be more likely to have email encounters than children in the usual care arm. As discussed previously, an increase in well child care would benefit the KPCO HEDIS quality measures. On the contrary, an increase in email encounters would be a negative outcome for a health system for both staffing requirements and costs. Email encounters are not currently reimbursable at KPCO, as is true for many health systems. 77 To identify the impact of the VSM intervention on clinic staffing requirements, assessing changes in encounters due to the VSM intervention is needed. While the VSM intervention has potential to change the frequency of health system encounters, multiple factors may influence a difference in use of the health system and should be considered in evaluation of health system utilization. Specifically, patients with chronic

PAGE 20

13 conditions are more likely to have higher utilization of the health system than patients without chronic conditions. 78 Patients who use the internet for medical information frequently, or who are more comfortable with technology, may be more likely to use email communications with providers. Maternal age and family size have also been shown to independently predict use of preventive childhood healthcare services where utilization increased with maternal age and decreased with a increased family size. 79,80 Marital status has also been shown to independently impact healthcare utilization, where dual parent homes are more likely to receive preventative services for their children. 79 While mother’s employment status has not been independently linked to healthcare utilization, 80 employment status could contribute to the time available for reviewing health information ultimately impacting utilization of services. Patients in the United States with a lower socioeconomic position, measured by income and education, are less likely to use the health system. 81 This occurs despite health insurance status. Members of minority racial/ethnic groups are more likely to have less intensive and lower quality of care. 81 Similarly, members of racial/ethnic minorities and lower education were found to be less likely to use the online health portal system for email communications with providers. 82 Health insurance coverage also plays a key role in health system utilization. 83 While all participants in the study have some type of health insurance, the specific type of insurance may contribute to health system utilization. 84 For example, patients with high deductible health plans have lower health system utilization rates, 85 but have also been more likely to report use of secure email with their provider. 82 Lastly, as reported earlier, vaccine hesitant parents are less likely to use well child care and more likely to use inpatient services. 2 Each of these factors provide potential unique contributions to encounter utilization. Cost Understanding the resource requirements alone does not provide enough information for health system adopters. The first question asked by the leadership at KPCO when the intervention outcomes were presented was, “What is the cost?.” This is often found to be the

PAGE 21

14 first and most important question asked by health system adopters, but is rarely evaluated. 31,35,46,86 The implementation dimension of RE-AIM includes assessment of the cost spent implementing the intervention. RE-AIM focuses on valuing an intervention for sustained adoption and implementation in a variety of settings. 42 This requires assessing costs important for implementation in health systems with outcomes, time-frames, and intervention costs that are relevant to a variety of health system adopters. 15,45 Cost-effectiveness analysis for vaccine interventions are often evaluated in the entire population of a country over a lifetime 7,87,88 This reflects the goal of the interventions evaluated to incorporate a new vaccination with direct impact on society as a whole. However, these cost-effectiveness results could not easily be translated to the cost and health benefit in one health system. 89 Instead, using a health system perspective in conduct of cost evaluations allow for the focus on outcomes that can be directly applied to the overall cost of health systems. 90 Increased vaccination provides a positive impact for health systems through an improvement in HEDIS measures benefitting health systems of all sizes. Thus, the incremental improvement in the number of additional children vaccinated, that can be attributed to the intervention, is the primary health outcomes of this cost evaluation. However, health systems may also be interested in understanding the disease prevention due to the intervention. The VSM intervention resulted in increased vaccination in the first 200 days of life impacting eight different infectious diseases. 91 However, because vaccine preventable diseases are so rare, a single health system will likely not see any cases for a majority of the diseases. Pertussis is the vaccine preventable diseases impacted by the VSM intervention with the highest rate of transmission. In 2017 in the United States, there were 2,237 cases of pertussis in children less than 1 year of age and 4,994 cases during an outbreak in 2012. 92,93 Pneumococcal had the next highest rate of transmission with only 435 cases in children less than 5 in 2017. 93 Additionally, an economic analysis of the entire childhood vaccination schedule found pertussis vaccination had the highest direct cost savings for inpatient services, outpatient services and outbreak

PAGE 22

15 control. 7 Observable outcomes are an important attribute for intervention adoption. 34,41 Thus, a secondary health outcome of pertussis cases averted provides an observable health outcome with direct monetary value to health system adopters. Similarly, using a time periods for cost evaluations that are relevant to a health system is also needed. The VSM intervention implemented into one health system would have health benefits limited to a shorter time period than would be expected with a change in vaccine policy. There was a 5.9 percentage point increase in vaccinations up to date after 200 days. While the rates of Measles Mumps and Rubella (MMR) vaccination coverage at 489 days was higher in the VSM arm compared to the UC arm, it was not statistically significant (95.3 vs 91.8, OR 1.95, p=0.10). 4 As vaccination coverage continues to increase with time, 65 health benefits due to the intervention reduce. From a health system perspective, pertussis outcomes in the first year of life have the largest impact. 94 Pertussis health outcomes include, respiratory illness, neurologic disease, and death. 94-96 In 2017, 33% of children under one year of age with pertussis were hospitalized, whereas only 2.5% of children and adults older than one were hospitalized. 97 Of the 13 deaths from Pertussis in 2017, 9 of them were in children less than 1 year of age. 97 Evaluating implementation cost-effectiveness for the VSM intervention over the child’s first year of life provides a time-frame with measurable benefits and relevant health outcomes for health system adopters. Implementation cost-effectiveness also includes intervention cost required to implement and maintain the intervention. 35,44,45 Health intervention cost evaluations often do not include all costs required for replication to a variety of healthcare settings. Specifically, implementation costs have historically not been included in cost effectiveness analyses of interventions. 44,45 These costs could be substantial, particularly for an online intervention that may require resources not currently in place. 46 For example, the VSM intervention required a secure login system for the interactive components. Developing a secure portal for the website requires specific expertise and technology capacity for implementation. Costs unique to online

PAGE 23

16 interventions are also important to distinguish. Online interventions present a unique circumstance where enhancements and updates are needed as technology changes. 32,46 The VSM intervention required annual domain name registration, website security, server space from a website hosting company, and one design update. Including these implementation and maintenance costs in a cost-effectiveness evaluation provides health system adopters the information needed to assess the cost and staffing impacts of the intervention on their health system more accurately. To effectively evaluate the VSM interventions implementation cost and costeffectiveness using the RE-AIM framework as a guide includes taking a health system perspective, assessing cost in terms of additional children vaccinated and pertussis cases prevented in the child’s first year of life, and incorporating costs of the intervention necessary to implement and maintain the intervention. The RE-AIM perspective also emphasizes implementation measures for a variety of healthcare settings. 42 To effectively address this RE-AIM component, translating the implementation cost-effectiveness to health systems with varied size and scope is needed. The 2 nd Panel on Cost Effectiveness in Health and Medicine was comprised of experts in design and conduct of cost-effectiveness analyses. The panel developed a set of recommendations for cost-effectiveness analyses in healthcare. 98 They recommend conducting a reference comparison between healthcare system costs and societal costs. The goal of including societal costs is to also identify the intervention impact on the patient and public health system. From a health system perspective, the societal costs have less value as there is no direct monetary impact on the healthcare institution. Implementation cost-effectiveness evaluations have not always included a societal cost reference case comparison. Instead, they have used a reference case comparison between different size healthcare systems. 99-101 This comparison allows for easy identification of the cost and health benefit in a variety of health systems. 45 An intervention may not be cost-effective in a small healthcare system, but when scaled to a large national

PAGE 24

17 organization it may be cost-effective or vice versa. For example, Shoup and colleagues found that an automated influenza vaccination reminder system was less cost effective then mailed reminder postcards at increasing influenza vaccination at KPCO. When scaled to a larger population size in the reference case comparison, however, the automated system was more cost-effective. 99 To address these gaps in knowledge of online health intervention costs, an implementation cost-effectiveness evaluation of the VSM trial is needed. Using the RE-AIM framework as a guide, an implementation cost evaluation of the VSM intervention should be evaluated from a health system perspective with cost considerations for a variety of health systems. Specifically: assessing incremental cost-effectiveness of the VSM intervention between the usual care and VSM study arms within the child’s first year of life with intervention costs that incorporate implementation and maintenance expenses in terms of a) additional children vaccinated b) pertussis cases prevented; and c) a larger healthcare system to estimate cost effectiveness in various environments. Since vaccinations have been successful and the spread of infectious disease is rare 93 and the online intervention includes interactive components that could be resource intensive, it may cost more to implement the VSM intervention than the cost reduction in pertussis cases prevented. Cost evaluations often use a pre-determined maximum value that indicates an intervention is cost effective. There has been large variation in what the pre-determined maximum value, or threshold, of a cost-effectiveness analysis should be, such as $50,000 per quality adjusted life year. 102 When taking a RE-AIM approach, healthcare institutions should determine the total cost worthwhile for the health benefit received. 44,45 However, that value may mean different things to different health systems. There is a lot of variation in terms of what a health system values based on needed resources and associated outcomes. For example, a nonprofit health system with a mission to improve health in the community, may value interventions that address the spread of infectious disease and thus, impacting the community

PAGE 25

18 outside the healthcare system. Placing a specific value indicating if the intervention is worthwhile does not address the goal of this dissertation, which is to provide health systems adopters with information important to their specific circumstances to inform implementation decisions. RE-AIM: Reach, Adoption and Maintenance The remaining four dimensions for a RE-AIM assessment of value, including reach, effectiveness, adoption, and maintenance, will not be assessed as part of this dissertation. These factors are important for both healthcare decision makers and for determining the overall value of an intervention. However, for the VSM intervention trial they have either already been done or are not yet relevant. Reach assesses the representative of the population willing to participate in the intervention. 16,103 While there are reasons other than vaccine hesitancy a parent may not obtain vaccines for their child, such as barriers to care, this intervention was not designed to address logistical barriers to vaccination. Rather, the intervention was designed to address vaccine hesitancy in parents with concerns about vaccination. The rate of vaccine hesitant parents enrolled in the study was comparable to the rate of vaccine hesitant parents in the general population (14.1%). 1,4 This indicates the intervention reached the target population. Effectiveness measures the impact of an intervention on outcomes and potential negative outcomes. 104 The VSM trial was found to positively impact vaccination rates. 4 Since patients could use the intervention at their own discretion, potential negative outcomes to the patient unrelated to vaccination are unlikely. Adoption measures the proportion and representativeness of the sites that adopt the intervention at both the provider and setting level. 105 The VSM intervention was implemented by the research team and the clinic staff was not involved. Thus, measures of adoption are not yet applicable. In addition to the resource and cost measures of the RE-AIM implementation dimension, it also assesses factors important for health system adopters that contribute to successful implementation. Specifically, the implementation dimensions also addresses how consistently a health system can deliver a program or policy

PAGE 26

19 including the resources and cost required to implement the intervention. 17 The research team delivered the intervention during the trial per protocol. Since the research team delivered the intervention, it cannot be determined how consistently a health system would deliver the intervention at this time. Similarly, maintenance measures continued use of the intervention by sites implementing the intervention. 106 Since there were no sites implementing the intervention at this point, maintenance is also not applicable. This dissertation will focus on addressing the RE-AIM implementation dimension which assesses factors important for health system adopters that are relevant to implementing the intervention in a healthcare setting, including staff requirements and implementation costs. Background Overview In summary, vaccine hesitancy is a growing concern that needs effective interventions implemented in healthcare systems. The VSM trial was found to be effective at increasing vaccination. 4 Congruent with literature in implementation science, lack of information on the factors important for implementation of the intervention created implementation barriers. 15,16 Implementation barriers included lack of information on the cost and resource requirements to address changes in health system utilization required for implementation and maintenance of the intervention. The RE-AIM framework provides a guide to assess the value of an intervention including the specific barriers to implementation experienced with the VSM intervention. 16 Using the RE-AIM framework as a guide I will assess 1) the impact of the VSM intervention on resource requirements by testing differences in encounter types between the VSM arm and UC arm and 2) the implementation cost-effectiveness of VSM intervention in terms of cost of the intervention per additional children vaccinated and pertussis case prevented in children less than a year at KPCO. The results of this study will inform health systems considering adopting the VSM intervention. Additionally, this dissertation will advance the literature in implementation science by assessing these often-overlooked measures for intervention implementation

PAGE 27

20 including resource requirements and costs of online interventions and increasing knowledge on costs of effective online behavioral interventions with interactive components.

PAGE 28

21 CHAPTER III METHODS Review of Aim 1 For Aim 1 I conducted secondary analyses on data collected for the Vaccine Social Media (VSM) intervention trial evaluating healthcare utilization for patients enrolled in the trial. Results of the aim will provide information on potential changes to staffing resource requirements due to the intervention for health systems considering adoption. For example, aim 1 will test if patients in the intervention have higher email encounters with their pediatrician than patients in the usual care arm, or if there is a higher rate of well child visits for patients in the intervention arm. I tested the following aim and hypotheses: Aim 1: Resource requirements: Assess the health system resource requirements associated with the VSM intervention trial by comparing the rate of encounters (inpatient, ER, outpatient, phone, and email) for children up to 395 days old between participants in the intervention arm (VSM) and the usual care (UC) arm of the trial. Hypothesis 1: Children in the intervention (VSM) arm of the study will have higher rates of well child care encounters through 395 days of life compared to children in the UC arm. Hypothesis 2: Children in the intervention (VSM) arm of the study will have lower rates of inpatient encounters through 395 days of life compared to children in the UC arm. Hypothesis 3: Children in the intervention (VSM) arm of the study will have higher rates of email encounters through 395 days of life compared to children in the UC arm. Hypothesis 4: Children in the intervention (VSM) arm of the study will have higher rates of phone encounters through 395 days of life compared to children in the UC arm. Hypothesis 5: There will be no differences in emergency department encounters between children in the intervention (VSM) arm of the study compared to children in the UC arm.

PAGE 29

22 Aim 1 Follow-up Period The VSM intervention was administered to children in the first year of life. Thus, changes to the way the health system would be expected to occur due to the intervention would occur during the time period of intervention exposure. Additionally, the intervention is intended to increase vaccination which occurs during routine pediatric care. At Kaiser Permanente Colorado (KPCO) this includes structured well-child visits at 2, 4, 6, and 12 months of age. Vaccines are routinely administered at these in-person visits. 57 Thus, changes in encounters due to changes in vacce behaviors and related medical conditions including email and telephone outreach to providers with vaccine questions or concerns, increased ED, inpatient, and outpatient visits due to vaccine adverse events or vaccine preventable diseases, and well child visits to receive vaccination would be expected to occur in the time period when both the intervention and vaccine administration occurs. Consistent with previous literature assessing under-vaccination and health system utilization related to undervaccination, 30 additional days were applied to the 12 month exposure time period allowing for well-child visits that may occur late due to scheduling variation. 2,4,107 Aim 1 Study Population All participants enrolled in the VSM trial from the VSM and UC arms with at least 1 day of KPCO insurance coverage were included in the study population. Participants were dropped from the study due to loss of KPCO insurance coverage, child death, and requests to end participation. Participants dropped from the study were included through the date dropped from the trial. Aim 1 Measures Independent variable (IV): Intervention arms . The independent variable includes patients in the study population randomized to one of the following study arms: VSM arm: Patients with access to the online intervention with interactive components

PAGE 30

23 UC arm: Patients without access to the online intervention Dependent Variable (DV): Healthcare Utilization All health system encounters for the child in the first 395 days of life were extracted from the KPCO databases including the electronic health record (EHR), claims, and enrollment data. This includes outpatient events with KPCO providers that are captured in the EHR, as well as events including ED and hospital admissions for care received outside KPCO health system captured via claims data. Each of the following types of encounters were evaluated in a separate analysis: emergency department (ED), inpatient (IP), email (EM), telephone (TE), well child visit (WCV), and other outpatient visits not including well child visits (OP). 108 While email and telephone encounters are made by the parent or guardian of the child, the data is linked specifically to encounter for the child and does not necessarily specify who made the contact. Thus, email and telephone encounters will include all telephone and email encounters made on behalf of the child. For each encounter type, a value for the total number of encounters from birth through days enrolled in the study up to 395 days will be given to each study participant. 2 Well child visits are not a specific encounter type available within the EHR dataset. Instead, well child visit encounters were identified using diagnosis codes. The International Classification of Diseases (ICD) Diagnosis codes, is a system used by healthcare providers to identify symptoms and procedures that occur in conjunction with health care received. The ICD codes used to identify encounters in which a well child visit occurred are listed in Appendix Table 1. The table includes ICD9 codes for encounters occurring prior to October 2015 and ICD10 codes for encounters occurring after October 2015. 109 All visits containing an ICD code listed in the table were categorized as a well child visit encounter (WCV). Covariates: Demographics and Pre-existing Conditions Since the study population assessed for this aim does not include the entire randomized population as participants were dropped from the study (due to fetal demise, no insurance coverage, or loss of interest in participation), these covariates may not be equally distributed

PAGE 31

24 between study arms. To address this issue, each potential covariate was compared between the VSM and UC arms. Comparisons were conducted using the Chi-square and t-test appropriate for each measure. When the data indicated a non-normal distribution, WilcoxonMann-Whitney test will used instead of t-test. Similarly, when cells in categorical measures had less than 5, Fischer’s exact test was used instead of chi-square. 110 Outcomes indicating a statistically significant difference in covariates at baseline between study arms were included as covariates in the poisson regression model described below. The potential covariates were captured using the same methods for both UC and VSM arms and are listed below with a description of the measurement parameters and measurement source. 1. Pre-existing conditions in the child: Pre-existing conditions will be extracted from the electronic health record using diagnosis codes. Pre-existing conditions will be categorized in three categories; 1=Non-Chronic, 2=Non-complex Chronic (such as type 1 diabetes), and 3=Complex Chronic (such as type 1 diabetes and chronic pulmonary disease). The pre-existing conditions used in this analysis are those derived from previous literature. 111 2. Insurance type at 90 days old (child’s insurance coverage): Deductible Co-Insurance (D/CO), Deductible/Coinsurance HMP Plus (D/CO+), High Deductible Health Plans (HDHP), Traditional HMO(TRD+), Medicaid (MD), Self-Funded (SF), HMO Plus (TRD+) 3. Covariates collected on the baseline survey representing parent response: a. Parity: 0 (currently pregnant), 1, 2, 3 or more) b. Education: (Grade school, 18 (continuous) d. Marital status: married, separated, divorced, not married, single, living with partner, widowed

PAGE 32

25 e. Race (parent): White, Black or African American, Asian, American Indian/Alaska Native, Native Hawaiian or Pacific Islander f. Ethnicity (parent), Do you consider yourself to be Latino Spanish or Hispanic: Yes, No, Don’t know g. Employment: full-time, part-time, unemployed, stay at home parent, retired, student h. Income (household): <40,000, 40-80, 81-120, 121-150, >150, prefer not to answer i. Frequency of internet use for health information in the last 2 months: (not at all, less than once a month, about once a month, every week, every day) j. Vaccine hesitancy: Parent’s Attitudes and Childhood Vaccines (PACV), a validated 15-tem instrument assessing vaccine hesitancy on a scale of 0 to 100. 112 Consistent with prior studies, participants with a score of <50 were categorized as “non-hesitant” and >50 as “vaccine hesitant”. 113 Aim 1 Missing Data As participants are included in the analysis for the days enrolled at KPCO, a zero value for each encounter type is assumed to have no encounters, thus there will be no missing data for the dependent variable. As the number of missing data values in the survey is small (< 3 for each question), listwise deletion will be used for surveys with missing data if covariates are included in the model. 114 Aim 1 Analytic Plan I used a poisson regression to assess the rate of utilization for each type of encounter in the child’s first year of life. Outcomes are expressed using incident rate ratios (IRRs) and 95% confidence intervals. Covariates were controlled for in the analysis if significantly different between groups. The model was tested for overdispersion, or higher variance than expected.

PAGE 33

26 When found, a negative binomial poisson regression analysis was used to address the overdispersed data. The equation below describes the poisson model used for anlaysis. 115 Where n is the count of visits for the given individual T=time it was followed-up (up to 395 days) Xi=applicable covariates (such as mother’s age, parity, preexisting conditions) Model paramters Bi are log relative risks Results with an IRR with a confidence interval that does not include 1 and a value greater than 1 indicates an increased risk (supporting hypothesis 1, 3, and 4) or a value less than 1 indicating a reduced risk (supporting hypothesis 2). 115 Results with an IRR containing a confidence interval that includes 1, indicates an insignificant result and will provide support for the hypothesis that ED visits are not different between arms (hypothesis 5). Excessive zeros. Utilization counts that are found to have a higher than expected variance, may be due to excessive zeros. In these situations, a zero-inflated poisson model may be a better model fit. 116 For example, to have an email encounter, a parent is required to have a kp.org account. If a parent does not have a kp.org account they could not have an email encounter. In this case, a zero-inflated poisson model may better describe the outcomes. I used the zero-inflated poisson model to compare utilization between the VSM and UC arms where excessive zeros exist and tested between the appropriate poisson model to determine which is the best fit using the Vuong Statistic. 117 Results of a zero-inflated model will provide two model estimates. The first models those with a zero probability of having the outcome, such as those without a kp.org account. The second model assess those with a probability of having the outcome in the population who have some probability of an email encounter.

PAGE 34

27 Aim 1 Power Power was calculated to estimate the minimum detectable coefficient for the available encounter data. Power was calculated using PASS software 118 using the encounter rate for the usual care study arm in person years. Well Child Visit A rate of 4.98 (837/168 person years) well child visit encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.55 will be detectable. Emergency Department A rate of 0.13 (23/168 person years) emergency department encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.1 will be detectable. Inpatient A rate of 0.065 (11/168 per years) inpatient encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.07 will be detectable. Email A rate of 2.59 (438/168 person years) email encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.4 will be detectable. Telephone A rate of 6.58 (1105/168 person years) telephone encounters occurred in the usual care group. With 80% power, and alpha of 0.05, a difference of 0.63 will be detectable. Review of Aim 2 The goal of aim two was to evaluate the cost of the VSM intervention in order to provide comprehensive detail on all potential intervention costs and benefits for health system adopters to use when evaluating implementation. I addressed the following aim and hypothesis for the cost evaluation: Aim 2: Cost: Assess implementation cost-effectiveness of the VSM intervention trial by assessing the incremental cost effectiveness ratio between the VSM intervention and Usual

PAGE 35

28 Care arm in a) dollars per additional child vaccinated, b) dollars per pertussis cases prevented, and c) estimating the replication incremental costs-effectiveness of the intervention when scaled to a large national managed care system. Hypothesis 6: The VSM intervention trial will be cost savings compared to usual care at preventing pertussis cases. Aim 2 Analytic Plan I conducted an incremental cost effectiveness ratio (ICER) of the total intervention costs per expected health benefit. Specifically, ICER was used to assess difference in total costs between VSM and UC study arms per pertussis cases prevented in the KPCO population birth cohort over a three-year time period. Consistent with cost-effectiveness analyses of health system interventions using a health system perspective, I calculated ICER using the difference in total cost between the UC and VSM arm divided by the incremental health benefit 44,90,99-101 Health benefit was evaluated for each individual in the first year of life. Cost was evaluated over a three-year time period to incorporate implementation and maintenance costs to assess expected changes in annual costs over time. Let IC=incremental cost; TCvsm=Total costs for the VSM intervention TCuc=Total costs for usual care. Then IC=TCvsm-TCuc, where TCvsm-TCuc, is represented by the following cost values: 44,119 TC1= Intervention costs: The cost of the intervention excluding recruitment and costs associated with research, such outreach and consent for the study enrollment and follow up surveys. Development costs are described, but not included in the analysis as the intervention is already developed. Intervention costs were categorized as fixed or

PAGE 36

29 variable. Variable costs were estimated based on the population size of KPCO and Kaiser Permanente National (KP national) in the reference case comparison. TC2=Intervention cost for vaccine administration: the cost increase to the health system for increased vaccination and treating vaccine adverse events. TC3= Utilization: The health system cost for addition or reduction in visit types due to the intervention (results of aim 1) TC4= Health outcome costs: The cost reduction to the health system for pertussis cases prevented in the intervention arm. IC=TC1+TC2+or-TC3-TC4 Let IE=incremental health outcome (additional children vaccinated (2a) or pertussis case prevented (2b)) over the three-year time period 2a-Evsm=children vaccinated in the 3 KPCO birth cohorts over the three-year timeperiod when exposed to the VSM intervention 2a-Euc= children vaccinated in the 3 KPCO birth cohorts over the three-year time period of KPCO in children less than 1 receiving Usual KPCO Care. 2b-Evsm=expected number of pertussis cases in the 3 KPCO birth cohorts over the three-year timeperiod when exposed to the VSM intervention 2b-Euc= expected number of pertussis cases in the 3 KPCO birth cohorts over the threeyear time period of KPCO in children less than 1 receiving Usual KPCO Care. Then IE=Evsm-Euc, or the number of additional children vaccinated (2a) or pertussis cases prevented (2b) due to the intervention. Let ICER=incremental cost-effectiveness ratio Then

PAGE 37

30 ICER=IC/IE or (TC1+TC2+or-TC3-TC4)/(Evsm-Euc) Aim 2 Follow-up Time Period Costs and health benefits were evaluated for a birth cohort through the child’s first year of life. The full cost evaluation will include costs and health benefits over three years of time to incorporate implementation and maintenance costs. This will address variation in costs over time including required design updates, decreased staff requirements as repeat questions increase, implementation costs, and increase in population impacted by the intervention over time. In total, the cost evaluation includes all costs and health benefits in the child’s first year of life from 3 birth cohorts over a three-year period of time. Aim 2 Study Population The study population includes a KPCO birth cohort for each of the three years in the cost evaluation (3 birth cohorts). The KPCO population averages 6,200 births a year. The KPCO population of children was used as pertussis is rare and the study population is too small to identify cases between arms (VSM=536 and UC=178). Additionally, extrapolating to the KPCO populations provides health system values relevant to potential adopters. Some health systems may choose to implement the intervention to all patients increasing the number of questions. However, the trial was assessed in a population of pregnant women and parents of young children. Results indicated it was only effective in the pregnant population, not in parents who had already made vaccine decisions for their children. Effectiveness results indicate the intervention is most effective when offered during pregnancy. As such, I focus the cost evaluation in the population known to be positively impacted by the intervention. Aim 2 Measures Health Benefit The health benefit, represents the denominator in the incremental cost effectiveness equation above. I evaluated the health benefit in terms of additional children vaccinated (aim 2a)

PAGE 38

31 and pertussis cases prevented (aim 2b). I evaluated the health benefit over 4 steps. In step 1, I determined the population potentially impacted by the health outcome for each birth cohort over the 3 year follow up period. In step 2, I assessed the number of children vaccinated over the follow up period. In step 3, I identified the proportion of children vaccinated and unvaccinated at risk for pertussis infection. In the final step 4, I applied the pertussis incidence to determine the number of children infected with pertussis. I evaluated the children vaccinated (part 2 below) and total pertussis cases (part 4 below) for both the VSM and Usual Care arms. Below, I describe details for each step of the analysis. Once the health benefit was determined for each study arm, the difference between study arms (Evsm-Eu) provided the incremental health outcome (IE). 1. Health benefit step 1: population size. Implementation year 1. The VSM intervention was effective at increasing vaccination after 200 days (7 months of exposure). 4 Additionally, the last childhood pertussis vaccination in the first year of life is recommended when the child is 6 months old. 57 For the known impact of the intervention to occur, the child would need to be at least 6 months old. Therefore, the pertussis cases prevented and vaccination health benefits were evaluated for 50% of the population in the year 1 birth cohort (n=3,100). Maintenance year 2 and 3. The pertussis and vaccination health benefit were evaluated in year 2 and 3 from 50% of the population from the previous year birth cohort and 50% of the population from the current year birth cohort (n=6,200 per year). 2. Health benefit step 2: number of children vaccinated (IE for 2a). Vaccination coverage in the usual care group (2-Euc) was estimated at 84.2% coverage. This rate reflects the percentage of the KPCO population up to date on the 6 vaccines (heptatits B; rotavirus; diptheria-tetanus-acceular pertussis; haemophilius influenzae type b; pneumococcal conjugate vaccine; and polio) assessed in the VSM trial at 200 days of age prior to the start of the intervention trial (2008-2012). The VSM intervention followed the same vaccine

PAGE 39

32 recommendations as usual care at KPCO with a 1.92 increased odds of vaccination. 4 Vaccination coverage due to the intervention was estimated by converting the probability of vaccination at KPCO to odds of vaccination. The odds of vaccination were then multiplied by the odds of increased vaccination due to the intervention. The new odds of vaccination were then converted to probability of vaccination in the KPCO population, with an estimated 91% vaccination coverage (2a-Evsm). For aim 2a assessing a health benefit of additional children vaccinated, the health benefit assessments methods stop here. The vaccination rate applied to the study population in the VSM arm minus the vaccination rate in the study population of the Usual Care arm, provide the incremental health outcome for aim 2a. To evaluate cost in terms of pertussis cases prevented (2b), vaccination coverage due to usual care (2b-Eu) was estimated at 88.83% using the percentage of children at 200 days of age up to date on DTaP vaccination at KPCO prior to the intervention trial (2008-2012). VSM vaccination rates were increased using the VSM intervention trial 1.92 increased odds of vaccination. 4 Vaccination coverage due to the intervention was estimated by converting the probability of vaccination in Colorado to odds of vaccination. The odds of vaccination were then multiplied by the odds of increased vaccination due to the intervention. The new odds of vaccination were then converted to probability of vaccination in the KPCO population, with an estimated 94% vaccination coverage (2b-Evsm). 3. Health benefit step 3 (2b): proportion of children at risk. The population at risk for pertussis infection (2b) was estimated using the vaccination coverage rates for Pertussis in Colorado and the effectiveness of three doses of the pertussis vaccine (98.1%). 120 I estimated a total of 13% of the population at risk for contracting pertussis in the UC arm. Assuming a 94% vaccination coverage rate due to the intervention, the population at risk after intervention implementation decreased to 8% for the VSM intervention arm. 4. Health benefit step 4 (2b): population infected with pertussis (IE for 2b). I assessed the pertussis health benefit by applying the 2017 Colorado incidence of pertussis, 71.4/100,000

PAGE 40

33 in children less than 1 year of age, to the proportion of children at risk of infection (Step 3 above). 121 Incidence calculations are established based on the entire Colorado population of infants. However, only the population at risk could contract the disease. Using the 2017 Colorado vaccination rate from the National Immunization Survey at 13 months of age (87.4%), and the assumptions of the population at risk in Health Benefit Step 3 above, I calculated incidence of pertussis in the at risk population. Results estimated at 71.4/14,261 (0.5%) in children less than 1 year of age. 65 I determined the number of pertussis cases in the UC and VSM arm by applying this incidence rate to the proportion of children at risk of pertussis infection identified in Step 3 above. The pertussis cases for each study arm was then subtracted to determine the incremental health outcome for aim 2b. Intervention Costs (IC) Intervention costs, or the numerator of the incremental cost effectiveness ratio, is determined through an evaluation of 4 different costs. The increment cost (IC) was a sum of the following calculated costs. 1. Cost to administer and maintain the intervention (TC1) 2. Costs for vaccine administration (TC2) 3. Costs addition or reduction due to changes in utilization of the health system from specific aim 1 (TC3) 4. Cost Reduction due to the costs associated with pertussis cases prevented. (TC4) Usual care at KPCO includes vaccination following the Advisory Committee on Immunization Practices (ACIP) recommendations. This includes vaccination at the 2 month, 4 month, and 6 month well child visits. 57,91 The VSM intervention arm also has access to the same usual care services cancelling out costs associated with those visits. Thus, the intervention costs (IC) in the incremental cost effectiveness evaluation are limited to those associated with the intervention (TC1, TC2, TC3, and TC4) described in detail below. Differences in well child visits due to the intervention identified in Study Aim 1 are incorporated in the TC3 costs described below.

PAGE 41

34 1. Intervention administration and maintenance costs (TC1). Development costs were not included in the cost estimates of the intervention as they are considered sunk costs since the process is complete and the resource available to adopters. 45 In the intervention administration cost analysis, I assumed the intervention was administered as designed for the intervention trial in all environments. This includes incorporating; 1) a secure access portal to the website, 2) a vaccine expert overseeing the website (intervention manager) and responding to patient questions, participating in chats, moderating the website, monitoring vaccine news, writing bimonthly blogs, developing and recording podcast (2 per year), and updating the content as appropriate, 3) a technology person to maintain the website and address any problems identified, and 4) a pediatrician participating in chats and developing/recording podcasts. Measuring Intervention Staff Time. Staff time estimates for intervention activities were measured by the study team monthly throughout the intervention trial. Study team tracked time spent on all study activities in a spreadsheet. 45 For example, the number of hours in a month spent monitoring news for new vaccine information and responding to participant questions was tracked on the spreadsheet. To supplement the data available in the time tracking document, each study team member was interviewed. Interviews provided an estimate of time on specific activities not available in the time tracking document such as the time spent responding to a new versus repeat question. In interviews, I asked study team members to estimate the average and range of time spent on blog development, question response for a new question, question response for a repeat question, daily website moderation, monthly website updates, daily news monitoring, a podcast recording and podcast development. To assist in accuracy of time estimates, in interviews I provided examples relevant to each staff member with specific patient questions responded to, blog and podcast’s written, and website issues addressed. Time estimates provided in interviews were compared to the monthly tracking and labor reports for further confirmation of accuracy. After time estimates for each activity were collected and

PAGE 42

35 confirmed, I averaged time across all study team members. The average time for each activity was used in the labor estimates for the intervention administration costs. Using the high and low values from the ranges of time provided by all staff members, a range of time for each activity was also determined. The ranges were used in sensitivity analyses described below. Results of time used for labor estimates in the cost evaluation are found in the table of intervention cost values (table 1) below. The range of time estimates for question response (10-240 minutes) reflects the wide variation in type of questions received. For example, one parent asked a simple question for information on what to expect from shots at her child’s 4 month visit: Question: “My child will have his 4 month well check in a month. I can't remember from my first child whether a fever is common for this immunization. From what I remember, it is not common at 2 months but is at 4 months... I want an idea of what to expect in terms of response to shots.” Response: “That is a great question. Fever is definitely common and expected after vaccination. A fever is part of the anticipated immune response to the vaccine, similar to when someone mounts a fever while fighting off an infection like the flu. Roughly onethird of infants will have a low-grade fever (temperature greater than or equal to 100.4) after the routine 2-month, 4-month, and 6-month vaccines given at Kaiser Colorado. Roughly 2% to 4% of infants will have a fever higher than 102.2. Based on clinical trials (the type of studies done to prove that vaccines are effective and safe), it doesn't seem like the risk of fever is a lot higher at 4 or 6 months compared to at 2 months of age.” However, there were also more complicated questions that required the study team to spend much more time working on a response. Question: Hi. My intentions are to vaccinate my child and I am confident it is the right thing to do. Unfortunately, I have a family member who is adamant that I would be putting him at risk by vaccinating. She sent me a link to the following website: [website

PAGE 43

36 here]. It appears to be research based and makes points regarding infant immunity that I do not know how to rebut. I would like to provide a fact based response to this family member as well as be reassured that I am doing the right thing for my child by vaccinating but as a mom of a newborn I don't have time to do extensive research. In this case, the website provided was written by a physician who cited multiple research studies in immunology in support of her argument against vaccination. They were valid scientific articles, but the claims on the website were misinterpreting the results. Due to the large number of scientific studies in immunology that the study team had not previously read, a large amount of time was required to interpret the results and respond in an easy to understand approach for the parent. Where the first question may have only taken a few minutes to respond, the second question took much more time and multiple team members to address. Because of this variation the average time to respond provides a good estimate of time required annually, but large variation exists in addressing the complexity of questions. One of the sensitivity analysis below addresses this potential variation. The labor estimates for staff time were based on national wage rates from the Bureau of Labor Statistics for a comparable position, with a fringe rate of 31.7% and an overhead rate of 40%. 122,123 The fringe rate was determined using the national average and the overhead rate indirect rate was the average across partner institutions in Colorado (ranging from 10%-67%). Specifically, a nurse position was used for the intervention manager, a pediatrician for the vaccine expert in blog development and chat sessions, and a computer programmer for the person implementing and maintaining the online intervention. Expertise of these positions is comparable to the study team member conducting these activities. Additionally, these staff members represent a person within a health system likely to conduct these activities. Measuring Quantity of Intervention Activities . A database was used for the intervention trial to track all study activities. Data collected included participant questions and response, blog posts and all comments associated with the blog post, and chats sessions transcripts. Tracking

PAGE 44

37 for each activity included the dates of interaction, the staff members, and the study participants involved in the interaction. This data was used to quantify the number of events for interactive activities. The rate of interaction was assessed by quantifying the total number of participant questions and blog comments (staff responded to all blog comments) by the total number of participants. To determine quantity of new and repeat questions, I entered all questions and responses into a spreadsheet. Questions were coded into a specific topic area, such as “alternative vaccine schedule.” Responses to similar topic areas were compared to identify consistent language used. In cases where similar responses were used, the question from the first date available was coded as a new question and all following were coded as a repeat question. In addition to costs associated with labor, development and maintenance of the online intervention had cost requirements. Wordpress, is a free, web development tool used to build the VSM website. Wordpress is open source, allowing developers to design features for websites, called “plugins,” that can be used to enhance the website and provide additional functionality. 124 The wordpress system and plugins require regular updates to enhance security and upgrade features. The VSM intervention included additional programming to enhance design and functionality. As updates to the plugins could influence this additional programming, testing was required with each update to ensure continued functionality and security. In addition, there are costs associated with hosting and securing the website. The VSM trial’s requirements for space were limited and thus we used a web hosting company on a shared server. Additional security measures were taken to address the risk posed by using an open source system with multiple plug-in features. A security system was installed for automated security testing and updates. Identified issues were addressed by the study web developer. In summary, costs required for the VSM website include programming features and design for implementation, website hosting costs, plugin features, security software, updates to the website, and staff time to address functionality and security issues.

PAGE 45

38 Intervention implementation costs year 1. Year 1 Intervention costs includes implementation costs. As implementation has not yet occurred, these will be estimated based on similar activities tracked during the study in the monthly time tracking document. Interaction is assumed to increase with time as an entire birth cohort becomes exposed to the intervention. As such, the implementation year of the intervention would be expected to have a lower rate of interaction. In year 1, the rate of interaction expected in the population was limited to interaction (question response and blog comments) in the first half of the intervention trial time period (3.14%). That rate was then applied to the KPCO population size (n=6,200). 4 Repeat questions received from participants, often require a lower response time than new questions and the proportion of repeat questions received will increase with time. The proportion of repeat questions received in year 1 were estimated using the proportion of questions received in the first half of the intervention trial time period that were repeats (17.6%). Intervention maintenance costs year 2 and 3. As the participants in year 2 and 3 include cohorts from the previous year, the total number of question/comments expected in year 2 and 3 were determined based on the rate of interaction (questions and blog comments) for the entire VSM intervention trial time period (5.54%). The proportion of repeat questions received in year 2 were estimated using the proportion of of questions received in the second half of the intervention trial time period that were repeats (46.2%). The proportion of repeat questions received in the second half of the intervention increased by 28.6 percentage points. The proportion of repeats questions received in year 3 were conservatively estimated to increase an additional 20 percentage points (66.2%). As technology continues to change, it was assumed one design update will be required over the three-year time period. Costs for the design update were incorporated in maintenance costs of year 3. Costs can be fixed or variable in that a fixed cost will not change with the population size and a variable cost will. Table 1 lists each cost to be included in the intervention costs (TC1),

PAGE 46

39 the years the cost will be applied, whether the cost is fixed or variable, the measurement values for the cost assessment, and source for each estimate. Tab le 1: Sources and Values for Inte rvention Costs (TC1) for Each Follow Up Year Cost description Year Fixed or variable Value Value Source Implementation Study tracking Website Development 1 Fixed 50 hrs Study tracking Website Design 1 Fixed $4930 Study TrackingInvoice Content Updates 1 Fixed 8 hrs Study Tracking Intervention Costs # of questions/comments (first half of the intervention) 1 Variable (17/542) 3.14% of the population Study Tracking # New questions/comments (first half of the intervention) 1 Variable 82.4% of the questions Study Tracking # Repeat questions/comments (first half of intervention) 1 Variable 17.6% of the question Study Tracking # of questions/comments (full intervention time period) 2, 3 Variable (30/542) 5.5% of the population Study Tracking # New questions/comments (2 nd half of intervention) 2 Variable 53.8% of the questions Study Tracking # Repeat question/comments (2 nd half of the intervention) 2 Variable 46.2% of the questions Study Tracking # New questions/comments (2 nd half of the intervention -20%) 3 Variable 33.8% of the questions Study Tracking # Repeat questions/comments (2 nd half of the intervention +20%) 3 Variable 66.2% of the questions Study Tracking New Question response time 1, 2, 3 Fixed 62 (30-240) min/response Study Tracking, Interviews Repeat Question response time 1, 2, 3 Fixed 26.67 (10-90) min/response Study Tracking, Interviews Newsletter Development 1, 2, 3 Fixed 90 (60-120) min/newsletter Study Tracking, Interviews Newsletters/ year 1, 2, 3 Fixed 12 Glanz, 2017 4 Blog development 1, 2, 3 Fixed 135 (60-420) min/blog Study tracking, Interviews Blog import 1, 2, 3 Fixed 10 (5-15) min/blog Study tracking, Interviews Blogs in a year 1, 2, 3 Fixed 22 Glanz, 2017 4 Podcast Recording-Pediatrician 1, 2, 3 Fixed 15 (10-15) min/podcast Study Tracking, Interviews Podcast Recording-Intervention Manager 1,2, 3 Fixed 45 (30-60) min/podcast Study Tracking, Interviews Podcast content development-Pediatrician 1, 2, 3 Fixed 150 (100-300) min/podcast Study Tracking, Interviews Podcasts/year 1,2,3 Fixed 2 Study Tracking # of Forum updates 1,2,3 Fixed 12 Study Tracking Forum updates time 1,2,3 Fixed 30 min/update Study Tracking # of Chats per year 1,2,3 Fixed 12 Study Tracking Time on chat 1,2,3 Fixed 1 hr/staff Study Tracking Vaccine news monitoring 1, 2, 3 Fixed 120 (60-300) min/month Study tracking, Interviews Website moderation 1, 2, 3 Fixed 12.5 min/day (10-15) Study tracking, Interviews Website Maintenance 1, 2, 3 Fixed 25 hours/year Study tracking, Interviews Addressing website functionality issues 1, 2, 3 Fixed 50 hours/year Study tracking, Interviews

PAGE 47

40 table 1 Cont’d Cost description Year Fixed or variable Value Value Source Maintenance Content updates 1, 2, 3 Fixed 120 (10-480) min/update Study Tracking, Interviews # Content updates 1, 2, 3 Fixed 4 Study Tracking Website Design 1, 2, 3 Fixed $4930 Study TrackingInvoice Plug-in Software 1, 2, 3 Fixed $47 Study trackingInvoice Website hosting 1, 2, 3 Fixed $275 Study trackingInvoice In the results section I provide total intervention administration cost results and staffing requirements in hours for administration of the intervention per year of administration. Sensitivity analyses described below, will provide variation in some of the cost assumptions described above to identify costs in a different environments and circumstances. This will provide additional detail important to health system adopters. 2. Costs for vaccine administration (TC2). To evaluate costs for 2a (additional children vaccinated) I conservatively estimate vaccine administration costs using private sector costs, as opposed to less expensive public sector costs, for the 8 vaccines administered from the KPCO vaccine formulary in the first 200 days of life. 125 This includes 2 doses (2 and 4 months) of 4 vaccines protecting against 8 vaccine preventable diseases. Vaccine administration costs were estimated using the Colorado fee schedule costs for a vaccine administration procedural codes. 109,126 The combination of vaccines and vaccine administration costs are listed in Appendix Table 2 and were applied to the total number of additional children vaccinated due to the intervention from Health Benefit step 2 to get the total cost of vaccine administration. To evaluate costs for aim 2b (pertussis cases prevented), vaccines administered were limited to the DTaP vaccine and costs associated with administration of DTaP in the first 200 days of life. Vaccine adverse events are rare, particularly when a small percentage of the population is impacted. This limits the potential impact of adverse events on costs. There are currently no known adverse reactions linked to polio vaccination. 127 For the hepatitis B vaccine, anaphylaxis is the only known adverse event and it occurs in 1 case per 1.1 million vaccines given, too rare

PAGE 48

41 to impact the additional population vaccinated. 128 Pneumococcal conjugate vaccine (PCV) has been associated with fever and irritability. However, one study found rates were not higher between vaccine recipients and controls. 129 Due to this evidence and consistent with a previous PCV cost analyses, these adverse events will not be incorporated into the costs. 7,130 Rotavirus has been linked in rare instances to intussusception, vomiting and diarrhea. However, a study conducted by the vaccine safety datalink (VSD), a group of managed care organizations charged with monitoring postlicensure vaccine safety, did not find an association with the vaccine and adverse events. There was found to be a small risk of intussusception too rare to impact the additional children vaccinated and thus will not be included in the cost evaluation. 131 Fever after administration of haemophilus influenzae type b (hib) vaccine has been reported in 5-30% of cases. However these often resolve within 24 hours and are treated at home. 128 As such, previous cost evaluations for hib vaccine limited costs for fever adverse events to antipyretics, available over the counter. 7,132,133 They did not include health systems costs. Since the focus of this evaluation is from the health system perspective, no costs associated with fever adverse events for the hib vaccine will be included. On the contrary to the other 5 vaccines, diphtheria, tetanus and pertussis (DTaP) vaccine has been associated with the following conditions that impact health system costs; medically attended local reactions (0.4%), 134 and medically attended events associated with fever (0.27%). 135 As recent studies have found no association with seizures or encephalopathy and DTaP or Pediarix (the combination vaccine on the formulary at KPCO containing DTaP), these adverse outcomes were not be included. 136-138 A severe reaction of anaphylaxis was not included in the cost evaluation of increased vaccination as it occurs in less than 1 in a million cases, too rare to impact the intervention costs in the KPCO and KP national populations. 139 The percentage of medical events occurring in each potential location (urgent care, inpatient, emergency department, and outpatient) were determined based on the literature for each outcome and are listed in Table 2 below. 134,135

PAGE 49

42 Table 2: Proportion of the Population Experiencing an Adverse Outcome From DTaP by E vent Type and L ocation Description of Adverse Event Type % of population vaccinated Source Medically attended events associated with fever (fever) 0.4% Zangwill, 2008 135 Medically attended events for local reactions (local reaction) 0.27% Jackson, 2011 134 Description of Fever Visit Type % of Medically Attended Fevers Source Outpatient 75% Zangwill, 2008 135 Emergency Department 21% Zangwill, 2008 135 Inpatient 4% Zangwill, 2008 135 Description of Local Reaction Visit Type % of Local Reactions Source Outpatient 60.6% Jackson, 2011 134 Emergency department 31.3% Jackson, 2011 134 Inpatient 7.9% Jackson, 2011 134 Urgent care 0.2% Jackson, 2011 134 Costs for DTaP adverse outcomes visits were determined using the fee schedule for current procedural terminology (CPT) codes associated with medically attended fever and medically attended local reactions. 126 The appropriate procedural codes for each type of health outcome and location were identified using events extracted from the KPCO electronic health record in children less than one year of age with a primary diagnosis of fever due to a vaccine or local reaction due to a vaccine. Table 3 lists the primary diagnosis codes used to identify cases of fever or local reaction. CPT codes were extracted for inpatient, emergency department and outpatient visits. Table 3: ICD Codes Used to Identify Cases of Fever and Local Reactions Within the KPCO E lectroni c Health R ecord Code Type Definition R50.83 10 Postvaccination fever T50.B95A 10 Adverse effect of other vaccines 780.63 9 Postvaccination fever E949.6 9 Adverse Effect of a viral vaccine Consistent codes for each health condition and location were identified and confirmed with a pediatric expert to ensure they accurately reflected expected medical treatment of patients experiencing a medically attended fever and location reaction. After confirming the CPT codes identified for each condition and location, costs were derived from the fee schedules published by the Centers for Medicare & Medicaid Services (CMS). 140 Colorado fee schedules were used for the KPCO cost analysis and the CMS cost estimates were used for the KP

PAGE 50

43 National analysis (reference case comparison for aim 2c). No inpatient encounters of either adverse health outcome occurred in KPCO in the last 20 years. The average inpatient cost for children less than 1 year of age, was found to be $5,000 on average from the Healthcare Cost and Utilization Project in 2012. 71 For any medically attended fevers or local reactions occurring in the inpatient setting due to the intervention, a cost of $5,000 updated to 2017 US dollars using the Bureau of Labor Statistics Personal Consumption Expenditures Price Index was applied. 141 Costs used for the vaccine administration and adverse event outcomes are described in Table 4 below, including the source for the cost value. Table 4: Costs Associated with Vaccine Administration and Vaccine Adverse Event O utcomes (TC2) Cost Description Cost Value Source Cost of vaccines in the first 200 days of life $712.80 CDC 125 Vaccine administration $18.93; $16.94 Colorado Fee Schedule 126 , National Fee S chedule 140 Fever-outpatient ( low complexity outpatient visit ) $79.30 CO Fee schedule, 2017 126 KPCO data Fever-emergency department (moderate complexity ED visit and urinalysis) $96.84 CO Fee Schedule, 2017 126 KPCO data Fever-inpatient $5,728 Weiss, 2012 71 Bureau of Labor Statistics, 2017 141 Local reaction-outpatient (low complexity outpatient visit) $79.30 CO Fee schedule, 2017 126 KPCO data Local reaction-emergency department (moderate complexity ED visit and bacterial culture) $96.84 CO Fee schedule, 2017 126 KPCO data Local reaction-inpatient $5,728 Weiss, 2012 71 Bureau of Labor Statistics, 2017 141 3. Additional or reduced encounters (TC3): results from specific aim 1 . Costs associated with changes in health system utilization were applied to the intervention costs. In study aim 1, I evaluated changes in utilization due to the intervention. Any significant differences in number of encounters were then applied to costs as a reduction or addition, appropriate for the aim 1 findings. To accurately capture the expected differences in encounters, a two part

PAGE 51

44 model was used. The first part was a logit model comparing those with no events (ie: ED count=0) to those with any events (ie: ED count >1). This provides an estimate for the increase in having a visit. The second stage used a zero truncated poisson where patients with no visits were excluded. This model provided an estimate for the increase in visits if one had occurred. Using the two stage model provided a more accurate estimate of the increase or decrease expected as it captures the expected difference in having a visit at all as well as the expected difference if one has already had that type of visit. The coefficient and standard errors for the two models were used to determine the annual increase or decrease in encounter events using a Monte Carlo simulation. 142 A Monte Carlo simulation projects the probability of the risk for an event to occur by simulating different scenarios. The model is given the parameters, in this case the coefficient and standard errors, then randomly selects possible outcome values based on distribution of known parameters and repeats by the number of requested simulations. Monte Carlo simulations are most often used when forecasting situations of uncertainty. Using this approach, I can more accurately obtain an estimate that reflects the expected change in encounters due to the intervention in a variety of circumstances. Using 10,000 simulations, I was able to get an average estimate of change in number of encounters. That total was then applied to the KPCO and KP National population of children each year to get the increase or decrease in number of encounters. Costs for the encounters were estimated using a moderate complexity visit. The visits costs were derived from the fee schedules developed by the offices of Medicaid and Medicare, Colorado and national fee schedule estimates for the KPCO and KP National cost evaluations respectively. 126,140 Email encounters are not billable events. Thus, an identified change in email encounters had costs estimated using pediatrician hourly wage, email response time, and difference in email encounters. Provider email response time varies, but was previously determined to be 3.5

PAGE 52

45 minutes on average. 143 Provider wages were estimated using national wage rates from the Bureau of Labor Statistics for comparable positions. 122 4. Health outcome cost benefit (TC4). Health outcome costs represent the total health system cost reduction due to the decrease in number of pertussis cases. The total number of pertussis cases prevented determined in the final phase of health outcome evaluation described in detail above, provided the number of events. Cost values were then applied to the number of pertussis events reduced. Using data from a large geographically diverse health system population, a cost evaluation estimated the increased health system costs associated with pertussis infection in the first year of life. 144 Costs estimates included health system costs associated with inpatient stays, emergency room visits, ambulatory visits, pharmacy, and other medical costs. On average, infants less than one year of age with a pertussis diagnosis had $8,271 increased health system costs. Average pertussis costs were updated to 2017 US dollars by using the Personal Consumption Expenditures Price Index. 141,145 The total cost for a pertussis event times the total number of events prevented provide the health outcome cost benefit value. Incremental cost analysis. Total increment cost (IC or the numerator in the incremental cost effectiveness ratio) was valued by summing costs calculated from each evaluation above associated with intervention related costs (TC1+TC2+ or -TC3) and subtracting the health system costs associated with the health benefit-pertussis cases prevented (-TC4). The total number provides an implementation and maintenance cost of the intervention over a three year time period. Incremental Cost Effectiveness Evaluation (ICER) Results of the incremental costs evaluation (IC) and incremental health outcome (IE) are presented for each component of the evaluation described above. Specifically, costs and health benefit were provided for each year of assessment to give additional detail on how costs and health outcomes change over time from implementation to maintenance of the intervention.

PAGE 53

46 Results include the health outcomes of additional children vaccinated (2a) and pertussis cases prevented (2b), costs to administer and maintain the intervention (TC1), costs for vaccine administration (TC2), costs for changes to utilization of the health system from specific aim 1 (TC3), and cost reduction for the health system due to pertussis cases prevented (TC4) I will present Incremental Cost Effectiveness Ratio for additional children vaccinated and pertussis cases prevented in two ways: 1. The total incremental costs derived from all components of the intervention over the health benefit (TC1+TC2+TC3-TC4)/(Evsm-Euc). A zero or negative value support the hypothesis that the intervention is cost savings compared to usual care. This analysis includes all potential costs associated with intervention implementation and maintenance. Results will be presented as the total cost per additional child vaccinated and the total cost per pertussis case prevented. If the intervention is not cost savings, the total cost worthwhile to reduce pertussis cases or increase vaccination is valued by the health system considering adoption. 2. The total incremental costs derived from intervention administration and health benefit cost reduction over the health benefit (TC1-TC4)/(Evsm-Euc). The results of this evaluation provide a total cost of administering the intervention (TC1) and the cost reduction from the health benefit (TC4). This evaluation may have more relevance to health system adopters. ICER evaluation 1 incorporates costs for the vaccines provided. This is an assumed outcome of the intervention. Additionally, provisions in the affordable care act require coverage of preventative services including vaccination. 146 This indicates the cost of vaccines actually incurred by the health system will likely be much lower if there is a cost at all. Lastly, health systems may be most interested in only the additional cost associated with administration of the intervention. Thus, the changes in utilization and vaccine administration will unnecessarily inflate that direct cost measure. To address the potential desire of health system adopters to understand the total cost of

PAGE 54

47 administration of the intervention alone, the second cost effectiveness analysis will be limited to intervention administration and health benefit costs. Results will be presented as the total cost per additional child vaccinated and pertussis case prevented. Aim 2c Reference Case Comparison For the reference case comparison, disease cases and intervention costs were adjusted to the birth cohort of Kaiser Permanente national (KP national) (n=116,000). KP national represents a healthcare system covering a large population of patients and geographic area allowing assessment of costs in a system with variation in size and scope. Additionally, the care provided at KP national is comparable to KPCO offering a larger environment where effectiveness results of the VSM trial are likely to translate increasing the accuracy of cost extrapolation. The population infected with pertussis (health benefit step 4) was adjusted using national vaccination coverage rates in children 13 months of age 65 and national pertussis incidence rates for children less than 1 year of age. 93 Table 5 lists the specific variable adjustments made to the population at risk calculations. Table 5 : Reference Case A djustments Description of adjustment Value Source Pertussis Incidence 50.88_ 100,000 CDC 93 Pertussis Vaccination Rates 89.2% CDC 65 As content specific to the geographic location of the participants (Colorado) may have been an important part of the success to the VSM intervention, intervention costs were adjusted to incorporate local content. Specifically, local content updates and local blog development costs were applied to each of the 9 states where Kaiser Permanente is located (California, Colorado, Georgia, Hawaii, Oregon, Washington, Maryland, Virginia, Washington DC). Additionally, building intervention tailoring to link patients to information specific to their location would be required in the implementation costs of the intervention. For example, patients in

PAGE 55

48 Kaiser Permanente Northern California (KPNC) would see information specific to the vaccines in the KPNC vaccine formulary and clinic locations and hours within the KPNC system. The VSM intervention required tailoring to the website in order for participants in each study arm to receive access to varied interventions. Study tracking of time to build tailored components were used to estimate tailoring development for implementation at KP National. Lastly, expanding the population size and usage of the website would require additional space from the website hosting company. Cost for website hosting annual fees were adjusted to reflect a dedicated server that provides additional space. Table 6 lists the fixed and variable intervention costs that will be adjusted specific to the reference case comparison. The variable measures in the Table are variable by the number of geographic locations. Variable measures in the Intervention costs table (Table 1) above will be adjusted by the size of the KP National population. Table 6: Reference Case Cost Adjustments Including Year Cost Applied, Fixed or Variable to Geographic Location, Value f or Analysis, and Value Source Cost description Year Fixed or variable Value Value Source Implementation Tailored login 1 Fixed 75 hrs Study tracking Local content development 1 Variable 8 hrs/location Study tracking Plug-in for membership tailoring 1 Fixed $9/year Study tracking-Invoice Intervention costs Local content blogs 1, 2, 3 Variable 1 Study Tracking Local content updates 2, 3 Variable 1 Study Tracking Hosting services 1, 2, 3 Fixed $7,200 Hosting company estimate Results will be provided for each evaluation described in the incremental cost effectiveness methods above for the reference case comparison. Comparisons between the local (KPCO) and National (KP National) health systems allow for assessment of a health system size that is most cost-effective for implementation. Additionally, results will provide cost and staffing estimates for implementation on a larger scale. Sensitivity Analysis The final step to a cost analysis is determining the appropriate sensitivity analyses used to test assumptions and estimates the range in costs for varied environments. 44,45,90,98

PAGE 56

49 Specifically, incidence rates of pertussis change each year. 147 Using pertussis incidence from both an outbreak and a non-outbreak year provide cost estimates for a variety of disease spread situations. Vaccination coverage rates also vary by year and location. 148 The known range of vaccination coverage rates in infants across the United States provide cost estimates that can be applied to health systems in variety of geographic locations. Costs per pertussis case to a health system may vary widely. As pertussis is rare, this variation could have a large impact on health system costs. Using a range of pertussis treatment costs provides a more comprehensive assessment of potential costs. Vaccine expert staff could vary depending on the environment where the intervention is implemented. A health system such as KPCO with an immunization task force, would likely use the nursing staff to respond to questions and moderate the website. However, a local health system may use community health workers in a similar role. Additionally, there is also large variation in provider email response time. 77,143 As the costs to a health system will vary based on the staff response time, a range of response times were used. Lastly, adjusting intervention estimates of moderation time required of the staff implementing the intervention were applied. A website moderation protocol was built to 1) address inappropriate behavior on the website, such as responding with abusive language and 2) identify participant interactions requiring response, including blog and discussion forum comments. Moderation time may need to increase with varied patient populations interacting with the website. Each of these sensitivity analyses can be used by health system adopters to identify the cost relevant to their respective healthcare system and cost adjustments made accordingly. In summary, the sensitivity analyses will include: 1. Pertussis incidence during an outbreak year, 2012. 2. Vaccination coverage variation 3. Pertussis disease cost variation 4. Adjustments to the intervention costs including a. Staffing variation

PAGE 57

50 b. Number of questions c. Response time variation d. Moderation time increase Outbreaks Pertussis cases prevented were revised in this sensitivity analysis by incorporating incidence rates of pertussis from an outbreak year in Colorado (178.9/100,000) and nationally (126.7/100,000). 149,150 Vaccination Coverage Variation Vaccination coverage rates vary by location. To help distinguish cost impact in varied geographic locations, vaccination coverage rates were adjusted to reflect the variation in state and vaccine coverage rates. The low and high end of the vaccination rates for each of the 6 vaccines administered in the first 200 days of life were identified using data from the national immunization survey at 7 months of life. Rates varied between 52.8% and 95.1% coverage respectively. 65,151-155 Pertussis Disease Treatment Costs Health system costs were found to be $8,271 higher in infants with pertussis compared to matched comparators. However, there was large variation based on age of the child from $3772 to $18,781. 144 This range of values was applied to health system disease costs for the sensitivity analysis. Costs were updated to 2017 US dollars by using the Personal Consumption Expenditures Price Index. 141 Intervention adjustments a. Staffing variation. To address the variation that may occur in staff overseeing and responding to intervention questions, the intervention manager salary was adjusted from a nurse to the following positions: pediatrician, community health specialist, research scientist, and social scientist research assistant.

PAGE 58

51 b. Number of questions. The quantity of questions per year was based on rates from the intervention. To address the potential variation in this rate, the percent of questions was halfed and doubled to identify the potential variation. c. Response time. As the time to respond to patient interactions may vary based on having responded to similar questions and staff expertise, response time was adjusted using the range of response time values identified in study tracking (30240min for each new response and 10-90 minutes for each repeat response). Study staff included a variety of vaccine expertise and communication skill levels providing an accurate range of variability. d. Moderation time . Estimates for moderation time per day were doubled (25 minutes per day) to incorporate potential increased need for moderation with increased population size.

PAGE 59

52 CHAPTER IV RESULTS Aim 1 Review Aim 1: Resource requirements: Assess the health system resource requirements associated with the Vaccine Social Media (VSM) intervention trial by comparing the rate of encounters (inpatient, ER, outpatient, phone, and email) for children up to 395 days old between participants in the intervention arm (VSM) and the usual care (UC) arm of the trial. The goal of aim 1 was to provide information on potential changes to staffing resource requirements due to the intervention. I hypothesized the following: Hypothesis 1: Children in the intervention (VSM) arm of the study will have higher rates of well child care encounters through 395 days of life compared to children in the UC arm. Hypothesis 2: Children in the intervention (VSM) arm of the study will have lower rates of inpatient encounters through 395 days of life compared to children in the UC arm. Hypothesis 3: Children in the intervention (VSM) arm of the study will have higher rates of email encounters through 395 days of life compared to children in the UC arm. Hypothesis 4: Children in the intervention (VSM) arm of the study will have higher rates of phone encounters through 395 days of life compared to children in the UC arm. Hypothesis 5: There will be no differences in emergency department encounters between children in the intervention (VSM) arm of the study compared to children in the UC arm. Support for these hypotheses would include an IRR greater than 1 and confidence interval that does not include 1 for hypothesis 1, 2 and 4, an IRR less than 1 and confidence interval that does not include 1 for hypothesis 3, and an IRR confidence interval that includes 1 for hypothesis 5.

PAGE 60

53 Aim 1: Study Population A total of 1093 pregnant women were enrolled in the study with 542 participants in the VSM arm and 180 participants in the UC arm. Participants were dropped from analysis due to 4 fetal demise or child deaths, 1 not interested in participating before the birth of their child, and 3 had no Kaiser Permanente Colorado (KPCO) insurance coverage in the first 395 days of the child’s life (figure 1). There were 536 participants in the VSM arm and 178 in the UC arm included in the analysis. Baseline characteristics are found in Table 7 below. Employment was the only observed baseline characteristic different between study arms in the analyzed cohort. VSM participants had a higher percentage of stay at home mothers than the UC arm (21.3% vs 12.4%). Mean parental age at enrollment was 31.6 years. A majority of the population was white (88.8%), nonhispanic (89.5%), college educated (83.1%), married (89.1%), employed full time (64%), and the child had no chronic conditions (89.9%). Approximately half of the population was employed full time (64%), made greater than $80,000 annual household income (55.9%), was on an HMO insurance plan (40%), and had previous children (54.5%). At enrollment 13.4% of the

PAGE 61

54 population was vaccine hesitant based on PACV screener score 113 and 62.7% reported using the internet for health information at least weekly. Some responses were combined to address the privacy risk in reporting a small number of respondents. However, each characteristic was compared between study arms as measured and the statistical significance of the comparison between arms did not change. Based on the baseline characteristic estimates, randomization appeared to work as expected except for employment. As such, employment was included as a covariate in the poisson regression analysis. Table 7 Parent Baseline Characteristics Characteristic Social Media N=536 Usual Care N=178 Total N=714 Pvalue Pre Existing Conditions^^ Non-Chronic 474 (88.4%) 161 (90.4%) 635 (88.9%) 0.5544 Non-complex Chronic 48 (9%) 11 (6.2%) 59 (8.3%) Complex Chron 13 (2.4%) 6 (3.4%) 19 (2.7%) Hispanic^^ Missing <5 (0.4%) <5 (0.6%) <5 (0.4%) 0.161 Yes 44 (8.2%) 22 (12.4%) 66 (9.2%) No 486 (90.7%) 153 (86%) 639 (89.5%) Don't know <5 (0.7%) <5 (1.1%) 6 (0.8%) Race^ Missing 2 (0.4%) 1 (0.6%) 3 (0.5%) 0.9062 Non-White 53 (11%) 16 (10.2%) 69 (10.8%) White 428 (88.6%) 140 (89.2%) 568 (88.8%) Education^ Missing 0 (0%) 1 (0.6%) 1 (0.1%) 0.221 College or more 446 (83%) 147 (82.6%) 593 (83.1%) Less than college 90 (17%) 30 (16.8%) 120 (16.8%) Marital Status^^ Missing 1 (.19%) <5% 2(.28%) 0.0697 Married or living with partner 19 (3.54%) 173 (97%) 683 (95.66%) Not married or living with partner 25 (4.66%) <5% 29 (4.06%) Employment^ Missing 1 (0.2%) 1 (0.6%) 2 (0.3%) 0.0314* Employed full time 339 (63.2%) 118 (66.3%) 457 (64%) Others 82 (15.3%) 37 (20.8%) 119 (16.7%) stay at home parent 114 (21.3%) 22 (12.4%) 136 (19%)

PAGE 62

55 Table 7 Cont’d Characteristic Social Media N=536 Usual Care N=178 Total N=714 Pvalue Income^ Missing 1 (0.19%) 1 (0.56%) 2 (0.3%) 0.8382 <$40,000 44 (8.21) 15 (8.43%) 59(8.3%) $40,000-$80,000 161 (30.04%) 59 (33.15%) 220(30.8%) $81,000-$120,000 186 (34.70%) 57 (32.02) 243(34.1%) $121,000-$150,000 58 (10.82%) 23 (12.92%) 81 (11.3%) >$150,000 59 (11.01%) 16 (8.99%) 75 (10.5%) Declined to Answer 27 (5.04%)) 7 (3.93%) 34 (4.8%) Insurance^ Missing 60 (11.19%) 23 (12.92%) 83 (11.62%) 0.506 Deductible 218 (40.67%) 81 (45.51%) 299 (41.88%) HMO 222 (41.42%) 64 35.96%) 286 (40.06%) Other (Medicaid and Self Funded) 36 (6.72%) 10 (5.62%) 46 (6.44%) Hesitancy^ Hesitant 65 (12.1%) 31 (17.4%) 96 (13.4%) 0.0837 Nonhesitant 471 (87.9%) 147 (82.6%) 618 (86.6%) Internet use for Health information^ Missing 0 (0%) 1 (0.6%) 1 (0.1%) 0.1652 Less than weekly 207 (38.6%) 58 (32.6%) 265 (37.1%) More than weekly 329 (61.4%) 119 (66.9%) 448 (62.7%) Parity^ Missing 0 (0%) 1 (0.6%) 1 (0.1%) 0.2203 >1st pregnancy 293 (54.7%) 96 (53.9%) 389 (54.5%) 1st pregnancy 243 (45.3%) 81 (45.5%) 324 (45.4%) Age^^^ mean (SD) 31.7 (4.3936) 31.4 (4.0658) 31.6 (4.31) 0.5144 *Statistically Significant difference between study arms ^Comparison done using a Chi Square ^^Comparison done using Fisher's Exact ^^^Comparison done using ttest Aim 1 Results The mean number of each type of visit and variance is listed in Table 8 below. There were a mean number of 4.75 well child visits, 2.73 outpatient visits, 0.23 emergency department visits, 0.045 inpatient visits, 2.38 email encounters, and 6.45 telephone encounters in the VSM arm. The number of each encounter type with no visits and the range in number of visits is presented in the Appendix Table 3.

PAGE 63

56 Table 8 Mean Number of Visits and Variance by Study Arm Type of Visit Social Media n=536 Usual Care n=178 Total n=714 Well Child 4.75 (1.747) 4.70 (1.815) 4.74 (1.762) Other Outpatient 2.73 (8.589) 2.58 (9.420) 2.69 (8.787) Emergency Department 0.23 (0.482) 0.13 (0.136) 0.21 (0.397) Inpatient 0.04 (0.050) 0.06 (0.092) 0.05 (0.061) Email 2.38 (14.863) 2.46 (15.674) 2.40 (15.045) Telephone 6.45 (33.874) 6.21 (27.171) 6.39 (32.174) I began assessing changes in encounters by using a poisson regression analysis. I controlled for employment in all models. Goodness of fit statistics indicated over dispersion in other outpatient visits, email encounters and telephone encounters. In each case, I used negative binomial regression analysis to address the over-dispersed data. Goodness of fit statistics indicated negative binomial regressions analysis was a better fit than poisson regression analysis in all cases. Additionally, excessive zeros were identified in other outpatient visits, emergency department visits, email encounters, and telephone encounters. I used zero inflated poisson regression analysis to test the excessive zero in emergency department and inpatient visits and zero inflated negative binomial poisson regression in other outpatient, email, and telephone encounters. To test the best model fit, I compared the zero inflated model and the poisson or negative binomial regression model using a Vuong statistic. The model with the best fit was used for the analysis and is reported in Table 9 below. The only significant differences in visits between the VSM and UC arms was emergency department encounters. In the poisson regression model, when controlling for employment there was a 1.81 (1.16-2.83, p=.0087) increased rate of emergency department visits in the VSM arm compared to UC. The model was a goodness of fit model indicated a poisson regression was a good model fit (p=0.98131). The Vuong statistic indicated the zero inflated poisson regression analysis had a better model fit than a poisson regression analysis. In the zero inflated analysis the count model indicated a 5.5 (1.711-18.079, p=0.0043) increased rate of additional emergency department visits in the VSM arm compared to the UC arm. The zero inflated model

PAGE 64

57 was not statistically significant indicating that there were no differences between study arms in the probability of having zero chance of having an emergency department visit (p=0.29). This indicates the intervention may only have increased used of emergency department encounters in patients already using emergency department services. Table 9: Difference in Healthcare Utilization Between VSM and UC Study Arms From Birth Up to 395 Days of L ife Type of Visit IRR CI pvalue Well Child Visit^ 1.018061 0.940-1.101 0.6565 Other Outpatient^^ 1.06844 0.898-1.271 0.4552 Emergency Department^^^ count model 5.562235 1.711-18.079 0.0043* zero inflated model 9.967203 0.1409-705.129 0.29 Inpatient^^^ count model 0.270495 0.046-1.569 0.1449 zero inflated model 0.322227 0.038-2.763 0.3017 Email^^ 0.967539 0.743-1.260 0.8069 Telephone^^ 2.284164 1.976-2.640 0.2638 *statistically significant difference ^poisson regression ^^negative binomial regression ^^^zero inflated poisson regression Emergency Department Visits I did not expect to see an increase in emergency department visits in the intervention arm. Therefore, additional steps were taken to describe the data and determine if the increase was due to adverse outcomes due to vaccination. All emergency department visits were categorized into health conditions based on the primary diagnosis of the visit. The health condition categories were determined by a Pediatric expert. Table 10 describes the frequency of each health condition by study arm and the difference in frequencies between study arms. A negative value indicates a higher percentage of events in the specific health category in the Usual Care arm. The most notable differences occurred in the vaccine related events (15.9% more in UC than VSM) and Gastrointestinal events (15.5% more in VSM than UC). However, the vaccine related events were in the opposite direction of what would be expected. There

PAGE 65

58 were more children vaccinated in the VSM arm than the UC arm providing more opportunity for adverse outcomes. Additionally, diarrhea, a potential gastrointestinal event related to vaccines, only occurred once among the 15 events. 156 As the resource evaluation was a secondary analysis and increase in ED visits for the VSM arm do not link to vaccine related events, rather appear to have a protective effect for vaccine events, the difference in study arms may be due to a spurious finding. However, there may also something about the intervention that increases a participant’s likelihood for bringing their child into the emergency department. Additional ED visits were incorporated into the cost analysis using a moderate complexity Emergency Department visit encounter cost. However, to address the potential for spurious findings, cost results will also be reported excluding Emergency Department visits. Table 10: Percentage of Each Primary Diagnosis in Emergency Department Visits per Study A rm and the difference between arms S t u d y A r m R e s p i r a t o r y P o t e n t i a l l y V a c c i n e R e l a t e d I n j u r y N e w b o r n C o n d i t i o n s A l l e r g y G a s t r o i n t e s t i n a l P o i s o n i n g G e n i t o u r o l o g i c H e a r t D e r m a t o l o g i c H e m a t o l o g i c N e u r o l o g i c a l Social Media 41.6 5.2 9.1 3.9 3.9 20.8 2.6 2.6 0.0 3.9 1.3 5.2 Usual Care 36.8 21.1 15.8 0.0 5.3 5.3 0.0 10.5 5.3 0.0 0.0 0.0 Difference 4.7 -15.9 -6.7 3.9 -1.4 15.5 2.6 -7.9 -5.3 3.9 1.3 5.2 Aim 2: Implementation Cost Effectiveness The goal of Aim 2, was to assess the implementation cost effectiveness of the VSM intervention in terms of additional children vaccinated and pertussis cases prevented. I hypothesized, VSM would be cost savings compared to usual care. Health Benefit The first step of the incremental cost effectiveness analysis was to determine the health benefit of the intervention (IE). The VSM intervention administered over 3 years to 6,200 pregnant women per year in the third trimester of pregnancy, was estimated to prevent 5 total

PAGE 66

59 pertussis cases, 1 in year 1, and 2 in years 2 and 3. There were 1,068 additional children vaccinated over the three-year time period, 214 in year 1 and 427 in years 2 and 3. Extrapolating to a large national healthcare system, the intervention administered to 116,000 pregnant women per year is estimated to prevent 65 total cases of pertussis, 13 in year 1 and 26 in years 2 and 3 respectively. There were 19,878 additional children vaccinated over the three-year time period, 3,996 in year 1 and 7,991 in years 2 and 3. Results are presented in Table 11 below. Table 11 VSM Intervention Health Benefit per year Health Benefit Year 1 Implementation Year 2 Maintenance Year 3 Maintenance Total (all yrs) Local Health System Pertussis Cases Prevented 1 2 2 5 Additional Children Vaccinated 214 427 427 1,068 National Health System Health Benefit Pertussis Cases Prevented 13 26 26 65 Additional Children Vaccinated 3,996 7,991 7,991 19,878 Intervention Costs The second step of the incremental cost effectiveness evaluation evaluated the 4 components of intervention costs including intervention administration and maintenance (TC1), costs for vaccine administration (TC2), costs for additional encounters (TC3), and cost reduction due to the health benefit from pertussis cases presented (TC4). 1. Intervention administration and maintenance costs (TC1). The total cost of administering the VSM intervention was $223,998. Intervention costs were highest in year 1 with implementation of the intervention ($81,206), lowest in year 2 ($69,723). In year 3, intervention costs were reduced due to less time required for question response as repeat question increased. However, a design update increased total intervention costs slightly above year 2 ($73,068). Highest intervention costs were attributable to question response ($43,576.68) and website moderation ($87,116). Results are found in Table 12 below.

PAGE 67

60 Table 12 Intervention Administration Implementation and Maintenance Costs Intervention Costs Year 1 Year 2 Year 3 Total Costs Implementation $18,169.86 $0 $0 $18,169.86 Question Response $11,651.34 $17,112.69 $14,812.65 $43,576.68 Newsletter $1,177.73 $1,206.00 $1,234.94 $ 3,618.67 Blogs $9,484.85 $9,712.49 $9,945.59 $29,142.93 Forum (updates) $392.58 $402.00 $411.65 $1,206.22 Chat $3,721.49 $3,810.81 $3,902.27 $11,434.57 Vaccine news monitoring $1,570.31 $1,608.00 $1,646.59 $4,824.90 Website moderation $28,352.83 $29,033.30 $29,730.10 $87,116.23 Content updates $523.44 $536.00 $548.86 $1,608.30 Addressing website functionality issues $1,946.61 $1,993.32 $2,041.16 $5,981.09 Website maintenance $3,893.21 $3,986.65 $4,082.33 $11,962.18 Hosting $275.00 $275.00 $275.00 $825.00 Design $47.00 $47.00 $47.00 $141.00 Design Updates $0 $0 $4,390.00 $4,390.00 Total costs for VSM administration $81,206.25 $69,723.26 $73,068.14 $223,997.65 Staff time required for the intervention is represented in Table 13. Total hours for the intervention manager was lowest in year 1 (691 hours), increasing in year 2 to address the growing number of questions (761 hours) and decreasing again in year 3 from an increased number of repeat questions (721 hours). Total hours for the intervention manager represents approximately 1/3 of a full-time staff member’s time. Time from a technology staff was highest in year for implementation (137 hours) decreasing to a consistent 87 hours in years 2 and 3. Hours for a vaccines expert (Pediatrician) were consistently 62 hours per year. Table 13 Intervention Staff Hours for Implementation and Maintenance of the I ntervention Staff Year 1 Year 2 Year 3 Intervention Manager Total 691 761 721 Vaccine Expert Total 62 62 62 Tech Person Total 137 87 87 Total questions requiring response was 192 in year 1 and 336 in years 2 and 3. Of the questions requiring response, the total number of repeat questions increased each year from 15 in year 1 to 69 in year 2 and 99 in year 3.

PAGE 68

61 Extrapolating to a large national health system total costs of administering the VSM intervention was $1,053,975. Unlike the local health system, intervention costs decreased over time after the initial increase from year 1 to 2 due to the influx of additional questions. In year 1 costs were estimated at $316,949 increasing in year 2 to $387,529 and decreasing in year 3 to $349,497. Results are found in Table 14 below. Table 14 National Health System Implementation and Maintenance Costs Intervention Costs Year 1 Year 2 Year 3 Total Costs Implementation Costs $34,560.42 $0 $0 $34,560.42 Question Response $221,566.55 $325,421.65 $281,683.12 $828,671.32 Newsletter $1,177.73 $1,206.00 $1,234.94 $3,618.67 Blogs $10,809.80 $11,069.24 $11,334.90 $33,213.94 Forum (updates) $392.58 $402.00 $411.65 $1,206.22 Chat $3,721.49 $3,810.81 $3,902.27 $11,434.57 Vaccine news monitoring $1,570.31 $1,608.00 $1,646.59 $4,824.90 Website moderation $28,352.83 $29,033.30 $29,730.10 $87,116.23 Content updates $1,701.17 $1,742.00 $1,783.81 $5,226.97 Addressing website functionality issues $1,946.61 $1,993.32 $2,041.16 $5,981.09 Website maintenance $3,893.21 $3,986.65 $4,082.33 $11,962.18 Hosting $7,200.00 $7,200.00 $7,200.00 $21,600.00 Tailoring Plug-In $9.00 $9.00 $ 9.00 $27.00 Design $47.00 $47.00 $47.00 $141.00 Design Updates $0 $0 $4,390.00 $4,390.00 Total costs for VSM administration $316,948.71 $387,528.97 $349,496.86 $1,053,974.53 Total hours for the intervention manager was lowest in year 1 (4,010 hrs, 2 full time staff) increasing in year 2 (5,401, 2 1/2 full time staff) and decreasing again in year 3 (4649 2 1/3 full time staff). Time for a technology staff was higher in year 1 than a local health system addressing the modification for local content at 212 hours. In years 2 and 3 technology staff time was consistent with local health system requirements at 87 hours per year. Hours for a vaccines expert (Pediatrician) were 62 hours per year, consistent with estimates for a local health system. Results are found in Table 15 below.

PAGE 69

62 Table 15 National Health System Intervention Staff Hours for Implementation and Maintenance of the I ntervention Staff Year 1 Year 2 Year 3 Intervention Manager Total 4010 5401 4649 Vaccine Expert Total 62 62 62 Tech Person Total 212 87 87 There were a total of 3,642 question requiring a response in year 1 and 6,380 in years 2 and 3. The total number of repeat questions increased annually from 285 in year 1 to 1,310 in year 2, and 1,877 in year 3. Highest costs were attributable to question response ($828,671) and website moderation ($87,116). Reference Case Comparison local content costs . In the large national healthcare system comparison group, additional costs were incorporated into the intervention implementation and maintenance for local content tailoring and updates. The costs were $12,690 in year 1 including $10,250 in build and local content updates for implementation and $2,440 for intervention maintenance including local content blogs and tailoring plug ins. Intervention maintenance costs were $2,499 and $2,559 in year 2 and 3 respectively. Results are found in Table 16 below. Table 16: National Health System Local Content Updates Cost and Staff Hours Implementation Year 1 Year 2 Year 3 Total Local Content Costs $10,550.75 $2,511.68 $2,571.75 $2,633.25 $18,267.43 Intervention Manager hours 72 38 38 38 115 Tech Person hours 75 2. Vaccine administration and adverse outcome costs (TC2). Vaccine administration and adverse outcome costs are found in Table 17 below. Additional vaccines administered in the local KPCO population were estimated to include 2 additional vaccine adverse outcomes for a medically attended fever treated in an outpatient setting. There were none in year 1 and 1 in years 2 and 3. Treating adverse outcomes was a total of $158.60 increasing from no cost in year 1 to $79.30 in years 2 and 3. Total costs for administration of the vaccines were $922,827 for all 3 years.

PAGE 70

63 Extrapolating to a large national healthcare system, additional vaccines were estimated to include 120 vaccine adverse outcomes, 18 in year 1 and 51 in years 2 and 3. Adverse outcomes included outpatient and ED visits for medically attended fever and outpatient, ED, and inpatient visits for local reactions. Treating vaccine adverse outcomes was a total of $29,563, increasing from $1,212 in year 1 to $14,176 in years 2 and 3. Total costs for administration of vaccines was $16,977,330 for all 3 years. Table 17: Vaccine Administration and Adverse Outcomes Cost Year 1 Year 2 Year 3 Total Costs Local Health system Vx Administration Costs $184,566 $369,131 $369,131 $922,828 Vx Adverse Outcome Costs $0 $79 $79 $159 Total Vx admin and Adverse Outcomes $184,566 $369,211 $369,211 $922,988 National Health system Vx Administration Costs $3,389,553 $6,799,107 $6,779,107 $16,947,767 Vx Adverse Outcome Costs $1,212 $14,176 $14,176 $29,563 Total Vx Admin and Adverse Outcomes $3,390,765 $6,793,282 $6,793,282 $16,977,330 3. Additional or reduced encounters (TC3). A two part model was used to estimate the number of increased ED visit encounters. The first part was a logit model comparing those with no ED events (ED count=0) to those with any ED events (ED count >1). This provides an estimate for the increase in having an ED visit. The second stage used a zero truncated poisson where patients with no ED visits were excluded. This model provided an estimate for the increase in ED visits if one had occurred. The coefficient and standard error for the employment and study arm were input in a Monte Carlo simulation. A population of 12,400 people was input in the simulation where half were in the UC arm and half were in the VSM arm. There were 10,000 simulations done to estimate the number of ED visits per arm. The average number of visits in a population of 6,200 participants with the estimated confidence interval was determined for each study arm and found in Table 18 below. There were an estimated 1,424 additional ED visits in the KPCO population of 6,200 due to the intervention. The cost for a

PAGE 71

64 moderate complexity ED visit was applied to the additional ED visits and incorporated into intervention costs. Table 18 : Mean # of ED Visits per Study A r m Over 10,000 S imu lations Study Arm Mean Confidence Interval VSM 2304.3 871.0-890.0 Usual Care 880.4 2232.1 2381 5 Difference 1,423.9 Costs found in Table 19 below were estimated at $334,260, ranging from $66,852 in year 1 to $133,704 in years 2 and 3. When extrapolating to a large national healthcare system, ED costs were $7,935,601, increasing from $1,587,120 in year 1 to $3,174,241 in years 2 and 3. Table 19 VSM Intervention Costs for Increase in ED Visits per Year Health Benefit Year 1 Implementation Year 2 Maintenance Year 3 Maintenance Total (all yrs) Local Health System $66,852 $133,704 $133,704 $334,260 National Health System $1,587,120 $3,174,241 $3,174,241 $7,935,601 4. Health outcome cost benefit (TC4). Costs associated with pertussis cases prevented are found in Table 20 below. Total health system cost savings due to cases of pertussis prevented were $47,375. In year 1 the cost savings was $9,475 increasing to $18,950 in years 2 and 3. Extrapolating to a large national healthcare system, total health system cost savings due to cases of pertussis prevented were $615,875. In year 1 cost savings was $123,175 increasing to 24,350 in years 2 and 3. Table 20: Health System cost reduction for Pertussis Cases Prevented Health System Year 1 Implementation Year 2 Maintenance Year 3 Maintenance Total (all yrs) Local Health System Cost Savings $9,475 $18,950 $18,950 $47,375 National Health System Costs savings $123,175 $246,350 $246,350 $615,875 Implementation Cost Effectiveness Incremental cost effectiveness was evaluated in two ways and results are found in Table 21 below. First, incremental costs incorporating a cost reduction due to the pertussis health

PAGE 72

65 benefit, and cost increase due to vaccine administration and adverse outcome costs, additional ED visits, and intervention implementation and maintenance were $1,433,870. Costs were lowest in year 1 ($323,149) with intervention administration and vaccine administration costs increasing in years 2 and 3 with the double the number of impacted children. Total incremental cost effectiveness over the three year time period per pertussis case prevented were $286,774 and $1,343 per additional child vaccinated. Second, incremental costs limited to intervention administration and health benefit, excluding ED visits, vaccine administration, and vaccine adverse outcomes, had a total cost of $176,623 over the three-year time period. Incremental cost effectiveness per pertussis case prevented was $35,325 and cost per additional child vaccinated was $165. Table 21: VSM Intervention Implementation Cost Effectiveness Year 1 Year2 Year3 Total All intervention components 1-Total Intervention costs $323,149 $553,688 $557,033 $1,433,870 Cost per pertussis case prevented $323,149 $276,844 $278,516 $286,774 Cost per additional child vaccinated $1,513 $1,296 $1,304 $1,343 2-Intervention administration costs $71,731 $50,773 $54,118 $176,623 Cost per pertussis case prevented $71,731 $25,387 $27,059 $35,325 Cost per additional child vaccinated $336 $119 $127 $165 Extrapolating to a large national health system results are found in Table 22 below. Total intervention costs were $25,351. Annual costs were lowest in year 1 ($5,171,659) nearly doubling in years 2 and 3 with double the children impacted by the intervention. The total cost per pertussis case prevented was $390,016 and $1,269 per additional child vaccinated. Incremental cost limited to costs of administering the intervention were $438,099. Using limited incremental costs, incremental cost effectiveness was $6,740 per pertussis case prevented and $22 per additional child vaccinated.

PAGE 73

66 Table 22: VSM Intervention Implementation Cost Effectiveness in National Healthcare System Year 1 Year2 Year3 Total 1-Total Intervention costs $5,171,659 $10,108,702 $10,070,670 $25,351,031 Cost per pertussis case prevented $397,820 $388,796 $387,333 $390,016 Cost per additional child vaccinated $1,294 $1,265 $1,260 $1,269 2-Intervention administration costs $193,774 $141,179 $103,147 $438,099 Cost per pertussis case prevented $14,906 $5,430 $3,967 $6,740 Cost per additional child vaccinated $49 $18 $13 $22 Sensitivity Analysis Sensitivity Analyses were conducted to test the model assumptions and assess costs in varied implementation circumstances. Implementation Cost Effectiveness Excluding ED visits Incremental cost effectiveness excluding ED visits costs (TC2) are presented in Table 23 below. The total costs were $1,099,609, a reduction of $334,261. Costs per pertussis case prevented were reduced by $66,852 and $313 per additional child vaccinated. When extrapolated to a national health system, costs were reduced by $7,935,601, $122,086 per pertussis case prevented ad $397 per additional child vaccinated. Table 23 Sensitivity Analysis: Total Incremental Cost Effectiveness Excluding ED V isits Total costs Cost per pertussis Case Prevented Cost per additional Child vaccinated Local Base Case $1,433,870 $286,774 $1,343 Local Excluding ED visits $1,099,609 $219,922 $1,030 Difference $334,261 $66,852 $313 National Base Case $25,351,031 $390,016 $1,269 National Excluding ED visits $17,415,429 $267,929 $872 Difference $7,935,601 $122,086 $397 Pertussis Outbreak Year Results of sensitivity analysis for a pertussis outbreak year are presented in Table 24 below. In a pertussis outbreak year, the total number of pertussis cases prevented increased from 5 to 8 cases, 2 in year 1, and 3 in year 2 and 3 respectively. Total health benefit costs

PAGE 74

67 increased $28,425. This represents a lower implementation cost-effectiveness than the original model, reflecting additional cost-effectiveness with an increased number of pertussis cases prevented. This reflects a reduction of $111,093 per pertussis case prevented. When modeling cost effectiveness using only intervention administration and health benefit reduction costs, the costs per pertussis case prevented were reduced to $18,524 from $35,324. Extrapolating to a large national health system, in an outbreak year the intervention prevents an additional 10 pertussis cases. This is a total health system cost savings of $1,004,350. This reflects a reduction of $53,265 per pertussis case prevented in total intervention cost effectiveness. When modelling cost effectiveness using only intervention administration and health benefit reduction costs, the intervention reduces costs in an intervention year saving $94,750, with a cost effectiveness of $4,578 per pertussis case prevented and $17.19 per additional child vaccinated. Table 24: Health Benefit and Costs in a Pertussis Outbreak Year Compared to the Local and National Base Case Pertussis Cases Prevented Health system cost Savings Intervention Administration and Health Benefit Costs Total Intervention Costs Local Base Case 5 $47,375 $176,623 $1,433,870 Local Pertussis Outbreak 8 $75,800 $148,198 $1,405,445 Difference 3 $28,425 National Base Case 65 $615,875 $438,100 $25,351,031 National Pertussis Outbreak 75 $710,625 $343,350 $25,256,280 Difference 10 $94,750 Vaccination Coverage Variation Results of the vaccination coverage variation sensitivity analysis are presented in Table 25 below. With vaccination coverage ranging from 52.8% to 95.1% in the local health care system, pertussis cases prevented range from 3-10 with additional children vaccinated ranging from 496-2,126. The largest impact on cost is for vaccine administration ranging from $428,259$1,836,974. Estimated costs from the base case model fall in the middle of this range. Vaccination coverage has a large impact on total intervention costs ranging from $986,517-

PAGE 75

68 $2,301,389 based on population level of vaccination coverage. However, a majority of the total intervention costs are linked to vaccine administration. When limiting costs to intervention administration and health benefit costs, total costs ranged from $129,247-$195,572. Extrapolating to a large national health system, the number of pertussis cases prevented ranges from 25-177 and additional children vaccinated from 9,260-39,757. The larger variation in pertussis cases prevented from the national sample increases the range of potential costs for health benefit savings ($236,875-$1,677,075). The wide variation in additional children vaccinated has substantial impact on the total intervention costs ranging from $8.6 million to $33.2 million. As vaccine administration has the largest impact on cost, when limiting total costs to intervention administration and health benefit, costs range from a cost savings of $623,100 to a total cost of $817,099. The cost savings highlights the importance of maintaining high vaccination rates to maintain health system costs. Table 25: Health Benefit and Costs in Low and High Vaccination Coverage Compared to the Local and National Base Case Pertussis Cases Prev Addl Children Vx Health system Cost Savings Vx Admin Vx Adverse Events Intervention Admin & Health Benefit Costs Total Costs Local Base Case 5 1068 $47,375 $922,828 $159 $176,623 $1,433,870 52.8% Vax 10 2126 $94,750 $1,836,974 $907 $129,247 $2,301,389 95.1% Vax 3 496 $28,425 $428,259 $0 $195,572 $986,517 National Base Case 65 19,978 $615,875 $16,977,330 $29,563 $438,100 $25,351,031 52.8% Vax 177 39,757 $1,677,075 $33,726,363 $100,354 -$623,100* $33,203,617 95.1% Vax 25 9260 $236,875 $7,855,280 $2,771 $817,099 $8,675,151 *indicates cost savings Variation in Pertussis disease costs Results of the pertussis disease cost variation are presented in Table 26 below. Applying a range of pertussis treatment costs of $3,772-$18,721, Health Benefit Cost Savings ranged from $21,605-$107,575. This impacted the overall costs t o range from $1,373,670$1,459,640. When limiting to intervention administration and health benefit costs, the range of

PAGE 76

69 costs was $116,423-$202,393, where high disease costs reduced cost per pertussis case prevented to $23,284 from $35,325. Health Benefit Cost variation had a larger impact when extrapolating to a large national healthcare system. Health Benefit Cost Savings ranged from $280,865 -$1,398,475. When limiting to intervention administration and health benefit costs, higher pertussis disease treatment costs led to a cost savings of $344,500, reducing cost per pertussis case prevented from $6,740 to a savings of $5,300. Table 26: Health System and Intervention Costs using Low and High Range of Pertussis Disease Costs Health System Cost Savings Intervention Admin and Health Benefit Costs Total Intervention Costs Local Base Case $47,375 $176,623 $1,433,870 Local Low Disease costs $21,605 $202,393 $1,459,640 Local High Disease Costs $107,575 $116,423 $1,373,670 National Base Case $615,875 $438,100 $25,351,031 National Low Disease costs $280,865 $773,110 $25,686,041 National High Disease Costs $1,398,475 -$344,500* $24,568,431 *negative values indicate a cost reduction Intervention Adjustments Multiple adjustments were made to intervention cost estimates to evaluate the costs under different circumstances. Results are found in Table 27 and 28 below. Staffing variation. Staffing variation had a large impact on intervention costs. Intervention administration costs ranged from $162,024-$450,409. Notably, a Pediatrician managing the intervention doubled the intervention costs, but a local community health specialist reduced costs by a little over $60,000. The impact of staffing on intervention administration had larger impact in the national scale were intervention administration costs ranged from $653,697-$2,516,323. When modelling cost effectiveness limited to intervention administration and health benefit cost reduction, a community health specialist managing the intervention was cost savings in year 2 and 3 saving between $12,374 and $32,225 respectively. This was an total incremental cost effectiveness of $582 per pertussis case

PAGE 77

70 prevented and $1.89 per additional child vaccinated. A research assistant managing the intervention was also cost savings in year 3 of the intervention saving $3,255. Table 27 Sensitivity Analysis: Staffing Variation in Intervention Manager Position Intervention Admin Costs Intervention Admin and Health Benefit Costs Total Intervention Costs Local Base Case $223,998 $176,623 $1,433,870 Local Pediatrician $450,409 $403,034 $1,660,281 Local Community Health Specialist $162,024 $114,649 $1,371,896 Local Research Scientist $266,429 $219,054 $1,476,301 Local Research Assistant $175,286 $127,911 $2,042,169 National Base Case $1,053,975 $438,100 $25,351,031 National Pediatrician $2,516,323 $1,900,448 $26,813,379 National Community Health Specialist $653,697 $37,821 $24,950,753 National Research Scientist $1,328,031 $712,156 $25,625,088 National Research Assistant $739,356 $123,481 $25,036,412 Number of questions. When the number of questions addressed by the intervention manager were reduced by half and doubled, intervention administration costs ranged from ($202,209-$267,574). The cost adjustment was due to approximately 80 hours of intervention manager work a year when the number of questions were halfed and between 178 and 256 hours a year when questions were doubled. When extrapolating to a large nation health system intervention administration costs ranged from $636,638-$1,882,646. When questions were halfed, total intervention administration and health benefit costs were $23,736 reflecting a cost savings in year 2 (-$21,5313) and year 3 (-$37,695). Staffing requirements at the national level would require just over one full time person to manage the intervention when questions were halfed and nearly 5 full time employees if questions were doubled. Response time. Similar variation was found when the time to respond to questions varied. Intervention administration costs ranged from $200,159-$342,965. The cost adjustment was due to a reduction of 94-139 hours per year of the intervention manager with an increased speed in response time and 504-700 additional hours per year when response time was slowed

PAGE 78

71 down. In both cases, this is less than a full-time staff person, but could be as much as an additional 1/3 of a full-time employee. When extrapolating to a large national healthcare system total intervention administration costs ranged from $600,653-$3,316,298. This cost adjustment was due to a decrease in annual intervention manager hours of 1778-2650 when response time sped up and 9,581-13,293 additional hours per year when response time slowed down. This reflects approximately one full time staff reduction with fast response time and between 4 and 7 additional staff members for a slow response time. When response time was sped up, incremental cost effectiveness measured in terms of intervention administration and health benefit was cost savings, reducing costs by $15,222 over the three year time period. The savings were concentrated in years 2 (-$36,342) and 3 ($-56,267) respectively. Moderation time (double). As moderation is a daily activity a consistent increase in a moderation time increased total intervention costs. The total intervention administration costs when moderation time increased to $311,114 reflecting 434 additional hours a year. Extrapolating to a large national healthcare system, was $1,141,091 total intervention administration costs reflecting the same additional hours a year as the local health system since moderation was estimated using a daily number of hours, not based on population size. Table 28 Sensitivity Analyses: Intervention Administration Assumption Variation Analysis Intervention Admin Costs Intervention Admin and Health Benefit Costs Total Intervention Costs Range of Annual Intervention Manager Hours Local Base Case $223,998 $176,623 $1,433,870 691-760 Local Questions Halfed $202,209 $154,834 $1,412,082 602-633 Local Questions Doubled $267,574 $220,199 $1,477,447 869-1016 Local Fast Response Time $200,159 $152,784 $1,410,031 597-621 Local Slow Response Time $342,965 $295,590 $1,552,837 1195-1460 Local Moderation Time Doubled $311,114 $263,739 $1,568,361 1125-1194

PAGE 79

72 Table 28 Cont’d Analysis Intervention Admin Costs Intervention Admin and Health Benefit Costs Total Intervention Costs Range of Annual Intervention Manager Hours National Base Case $1,053,975 $438,100 $25,351,031 4009-5401 National Questions Halfed $639,639 $23,764 $24,936,695 2316-2972 National Questions Doubled $1,882,646 $1,266,771 $26,179,702 7396-10258 National Fast Response Time $600,653 -$15,222* $24,897,709 2231-2751 National Slow Response Time $3,316,298 $2,700,423 $27,613,354 13590-18694 National Moderation Time Doubled $1,141,091 $525,216 $25,438,147 4443-5833

PAGE 80

73 CHAPTER V CONCLUSIONS This dissertation evaluated the implementation dimension of the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework for the Vaccine Social Media (VSM) intervention through resource requirements and implementation cost effectiveness. The only change in health system utilization identified was an increase in emergency department (ED) encounters due to intervention. When evaluating costs over a three-year time period, the intervention was estimated to increase vaccination by 1,068 additional children and prevent 5 pertussis cases in a local health system. In a national health system, VSM resulted in vaccination of an additional 19,978 children and prevent 65 pertussis cases. The intervention was not found to be cost savings compared to usual care due to pertussis cases prevented in either a local or national healthcare system. Total incremental costs of administration of the intervention and health benefit over the three-years were approximately $175,000 in a local health system and $430,000 in a national health system. The goal of the RE-AIM implementation measure is to assess how well an intervention can be implemented across a variety of environments. Challenges to implementation including high costs and increased ED visit resource requirements could be a deterrent for implementation of the VSM intervention. However, the goal of RE-AIM is to provide a comprehensive and realistic assessment of intervention implementation. 17 In addition to the resource and cost results found in this evaluation, potential adopters also need to consider important contextual factors specific to vaccination behavior and the VSM intervention to inform their decisions. Implementation Resource Evaluation Changes in utilization due to an intervention is an important consideration for potential health system adopters as it impacts both costs and staffing. However, when looking at

PAGE 81

74 vaccination behavior, the nature of the results makes it difficult to accurately estimate the impact to a specific health system. Thus, changes in utilization due to the VSM intervention need to be interpreted with caution. For example, the lack of differences in well child visits results do not necessarily indicate there were no changes in well child visit utilization due to the intervention. One national study found 1,399 distinct vaccination patterns in parents not following the recommended schedule. 2 This represents use of the healthcare system in a variety of ways. Some parents use the system following normal well child recommendations and just delay or don’t get some of their vaccines. Other parents either do not use the health system for well child visits receiving no vaccines, or use it more frequently to obtain vaccines following a different schedule. 1 Thus, an increase in vaccination could result in a variety of changes to well child visit utilization including an increase, decrease, or any combination of the two. Changes in well child visit utilization due to increased vaccination are therefore more likely to be health system specific depending on how the vaccine hesitant population of that health system uses preventative care. Similarly, there are three potential situations that could have led to the increase in emergency department (ED) visits due to the VSM intervention. First, an increase in vaccination could lead to additional vaccine adverse outcomes, increasing the number of ED visits in the intervention study arm group. However, diagnosis codes for the ED visit do not indicate a link to vaccine adverse outcomes, making this outcome unlikely. Second, evidence suggests anxiety mediates the relationship between health information seeking and utilization. 157 As such, a health information intervention describing potential adverse outcomes of vaccines may increase anxiety for the health of the child leading to increased ED visits. This circumstance is also unlikely as there are limited vaccine adverse outcomes and the communication approach used for the intervention focused on addressing parents’ concerns and avoiding scare tactics. Lastly, an increase in ED visits could be a random effect of conducting multiple secondary analyses on

PAGE 82

75 outcomes that were not the intention of the intervention. 158 This is the most likely outcome, but further testing of changes in ED utilization due to the VSM intervention in a variety of environments is needed to confirm this hypothesis. Thus, health systems do not necessarily need to play for additional ED visits due to VSM intervention implementation. Implementation Cost Evaluation: Public Health Impact In addition to considering the challenges to interpretation of resource requirements, the public health benefit of vaccination was minimized in the implementation cost-effectiveness analysis and is an important context consideration for potential adopters. Vaccinations are one of the most significant public health contributions in the last century. 159 By maintaining high levels of vaccination, vaccine preventable diseases have limited ability to spread. 5 The success of vaccines has minimized the direct impact an increase in vaccination can have on costs to one health system. However, if vaccination rates are not maintained, the impact on both disease spread and costs are substantial. An economic analysis of the full vaccine schedule found it prevented 20 million cases of disease and saved $13.5 billion in direct costs. 7 This dissertation cost evaluation was designed to assess cost a health system could reasonably expect to see with the known outcomes of the intervention, the known population, in one endemic vaccine preventable disease. Because of the focused cost evaluation, two important implications for health systems were underestimated and should be included in adoption considerations. First, I limited the spread of disease in this analysis to the birth cohort as estimating the impact in other populations would be difficult to do with accuracy. Thus, the health benefit in the community, family members, and patients sitting in a waiting room were not considered. Vaccine preventable diseases are highly contagious and can quickly spread to all members of a community. For example, an estimated 90% of people close to a person with measles who are not immune from the disease will become infected. 128 In less than two months in 2019, there have been five outbreaks with 127 cases of measles across 3 different states. 160 These cases

PAGE 83

76 can spread quickly in communities with low immunization rates impacting community members across all age groups. 161,162 The sensitivity analyses conducted help inform scenarios where additional members of the population are impacted. Both the outbreak scenario and decreased vaccination coverage rates highlights a larger population impact in situations where a vaccine preventable disease occurs in a health system community. Results indicated a cost reduction in both health systems with cost savings nationally when vaccination rates are low. Second, the vaccination schedule prevents twelve diseases. In this evaluation I only assessed the currently endemic disease of pertussis. Pertussis is the disease most likely to impact a health system annually. However, as just described with the measles outbreak, other vaccine preventable diseases could experience outbreaks, particularly where vaccination rates are low. The sensitivity analysis during an outbreak year provides an example of costs associated with increased disease burden, indicating limited costs locally and nationally. The overall benefit of the VSM intervention increases when considering all vaccine preventable diseases. When assessing costs from a health system perspective, understanding the expected costs is critical to adoption. The cost estimates in this analysis provide an accurate assessment of costs for the VSM intervention with outcomes likely to impact a health system. Most of the diseases vaccinated against in the first year of life would not be expected to occur as a regular expected cost within a health system, with the exception of pertussis. 93 It is difficult to predict when disease outbreaks may occur and how they will spread. However, the sensitivity analyses highlight the importance of increasing vaccination rates and maintaining a high level of vaccination coverage. This allows for maintenance of costs within a health system and limit the ability for outbreaks to occur. Improved vaccination rates impact the entire health system population and community population health for the better.

PAGE 84

77 Adoption Considerations for Health System Adopters Taking the important public health benefit of the intervention costs and considerations of limitations with health system utilization findings into account, the RE-AIM implementation value of the VSM intervention increases for potential health system adopters. However, another important component of RE-AIM includes consideration of potential variation in intervention adoption. 42 Three important adoption considerations for health system adopters were highlighted in this cost-effectiveness analysis of VSM. The first consideration is vaccine administration costs. When evaluating all costs associated with the intervention there were just under $1.5 million for three years in the local setting and over $25 million dollars nationally. A majority of these costs were associated with vaccine administration. This overestimates the cost of vaccine administration as most health systems negotiate lower rates for bulk orders and the affordable care act requires insurance to pay for preventative services including vaccinations, limiting the direct impact on health systems. 146,163 Additionally, increased vaccination is a health system benefit improving Healthcare Effectiveness Data and Information Set (HEDIS) scores that many health systems will view as a benefit, lowering the concern for these costs. However, there are situations where this cost could be an important consideration of potential adopters. Evidence suggests, even with the Affordable Care Act provisions for insurance coverage of vaccination, some private practices have chosen not to provide vaccinations as the costs associated with stocking adequate vaccines and limited reimbursement becomes too costly for the provider. 164 Increasing childhood vaccination may not be a priority of these health systems. The second consideration is staffing the intervention. A majority of the intervention administration costs of VSM are associated with the human interaction intervention requirements (responding to questions). Adopters may consider excluding interaction to limit intervention costs. However, when interaction was not included, the VSM intervention did not significantly increase vaccination. 4 This indicates interaction was a central component to

PAGE 85

78 intervention effectiveness making it a critical part of implementation. The sensitivity analyses highlight ways health system adopter can manage resource requirements for the intervention while still offering interaction. Specifically, costs were greatly impacted by the staff salary and response time, making it important to find a balance between vaccine knowledge and salary level. I found the most cost-effective implementation approach used someone with vaccine expertise to respond in a short amount of time, but also had a lower salary than a pediatrician. For example, using a nurse or community health specialist with expertise in vaccination can help manage intervention costs. In a national health system, this implementation approach was cost savings. The third consideration is the technical capacity of the website. The intervention implementation build assumed designing a secure login system for accessing the website. Health systems with limited technical capacity may consider removing the secure login approach. This is likely to increase reach of the intervention, as website login reduces the number of people willing to access the intervention. However, removing secure login could also reduce the effectiveness and increase resource requirements. Analysis of the VSM intervention discourse found that having a secure login program with communication between patient and healthcare providers/vaccine experts, facilitates an environment for civil discourse. 165 This may have been an important part of effectiveness of the trial by maintaining an environment where participants felt safe from negative discourse that is often associated with online vaccination websites. 166 As such, staffing requirements for moderation were limited for the VSM intervention. This implies the secure program may limit reach, but minimizes costs associated with moderation and maintains civil discourse important for intervention effectiveness. Thus, the secure login is more likely to increase the RE-AIM value of the intervention despite the decrease in reach. Requiring staffing resources, technical expertise and capacity to include secure login may limit certain health systems from being able to offer the VSM intervention. Specifically,

PAGE 86

79 small private health systems may not have this capacity. However, the design and cost estimates in this analysis were built with a model for a grouped health system program in mind. The national health systems costs could be translated to a group of providers instead of one national health system. In a model such as this, a central location, such as a county health department, could administer the intervention. Log in information could be provided to each participating health system tailoring information to their vaccine formulary and office location. Total costs could either be paid for by the county health department or shared between participating health systems. The national costs could be applied to this scenario to estimate total administration costs. Thus, the intervention approach is not limited to larger health systems for use in a variety of environments. VSM RE-AIM Value and Next Steps Previous evaluation of the VSM study indicated the vaccine hesitant population was reached by the intervention, and the intervention led to a clinically meaningful increase in vaccination. 4 Particularly when taking the overarching public health benefit of vaccination into consideration, the resource impact and cost of the VSM intervention is low. This is particularly true, in a large national healthcare system that could see cost savings from implementation of the intervention in multiple situations such as outbreak situations, when vaccine preventable disease treatment costs are high, and when implemented with staff that has a lower salary and high vaccine knowledge. The RE-AIM results of the VSM intervention known at this point indicate a high value of the intervention with potential for successful implementation in a variety of settings. However, adoption and maintenance could not be measured at this time as the intervention was conducted by the research staff. Understanding potential barriers and facilitators to adoption and maintenance would provide additional information on the potential success of the intervention in a variety of settings.

PAGE 87

80 These additional outcomes directly reflect the next stage of implementation research. There are five phases (T0-T4) advancing scientific discoveries into practice. 14 Phase 1 (T0), phase 2 (T1), and phase 3 (T2) are focused on discovering the problem and testing efficacy of the new intervention. The last 2 phases, phase 4 (T3) and phase 5 (T4) are the focus of implementation and dissemination science. T3 research, such as this dissertation, is designed to increase uptake and implementation of the intervention. The next stage, T4 research, evaluates effectiveness and cost-effectiveness of an intervention in real world settings. Specifically, T4 evaluates the cost and effectiveness in a variety of environments after implementation. This provides the value of the intervention when implemented in the real world. It offers a more detailed look at potential variation in implementation that may occur impacting effectiveness and cost. To fully understand the impact of the VSM intervention in real world application, the next step to this dissertation is a T4 research evaluation of VSM. Informing Implementation Science In addition to the information specific to the RE-AIM value of the VSM intervention, this evaluation expands on our current understanding of the resources and costs required for implementation of effective online interventions. Implementation is often found to be difficult when additional staffing requirements are needed. 43,167 However, as found with the VSM intervention, online interventions that include human interaction are often more effective than online interventions without interaction, making staffing requirements an important consideration. 4,52 Online human interaction is also shown to be as effective as face to face interaction. 53 This presents an opportunity for interactive online behavior change interventions to shift staffing resources instead of just requiring additional staff. In the case of VSM, pediatricians report spending at least 10 minutes on average discussing vaccines with vaccine hesitant parents. 58 We were unable to measure this change in provider time due to the interruption in care when measuring this outcome. However, VSM could have alleviated some of this time from pediatricians allowing additional time in well child visits to focus on other

PAGE 88

81 important health topics. As healthcare costs continue to increase, finding new approaches to shift staffing requirements could present opportunities for cost savings or more efficient healthcare administration. 168 Limitations This dissertation is limited in several ways. First, the VSM trial is complete, and therefore the data is limited to what was already collected as part of the intervention trial. This limits the data available to examine changes in staffing requirements and cost estimates. Specifically, a direct measures of intervention impact on provider time with patients in the well child visits was not captured limiting the ability to evaluate impact on pediatrician time. Second, the cost-effectiveness evaluation is limited to implementing the intervention as designed. In real-world application, modifications to the intervention may be required to make it compatible with different environments. Subaims and sensitivity analysis and were done to address this concern, but additional revisions could be made that were not considered. Further evaluation using T4 research methods would assist in examining effectiveness and cost of the intervention after implementation in a real world setting. Third, public sector costs for vaccine administration were not represented in this implementation cost-effectiveness analysis. This was done to reflect the circumstances at KPCO and KP national. Additionally, this was a conservative estimate as the private sector costs are higher than the public sector. 7,125 Fourth, the cost analysis is likely to underestimate the cost-effectiveness of the intervention. The VSM intervention will impact multiple infectious disease in addition to pertussis. 57 Similarly, the vaccination coverage in infants may also impact other populations including family and community members that are not assessed. 128 Also, increased vaccination could impact the health outcomes of the population over a longer period than assessed in this evaluation. While these are important considerations, the data available for this analysis limit the accuracy of evaluations incorporating these considerations. Additionally, the goal of this study

PAGE 89

82 was to provide costs accurately reflecting what a health system could regularly expect and many of the other cost considerations could not be predicted or assumed to occur on a regular basis. Fifth, the full value of the intervention using all five measures of RE-AIM cannot be assessed as the research team implemented the intervention. This limits the ability to identify potential barriers to adoption and maintenance. Reach, effectiveness, and implementation have or will be assessed with this dissertation to provide the RE-AIM value of the intervention known at this point. Sixth, additional factors important to implementation, such as characteristics of potential adopters, 34 will also not be assessed as part of this dissertation. While this may miss important factors for health care decision makers, the goal of this dissertation is to focus on the RE-AIM dimensions that are often requested by adopters but rarely evaluated, cost and staffing impacts. Seventh, assessing changes in encounters were limited to the study population. Differences in encounter utilization may not have been detectable. For example, with the small proportion of inpatient encounters (11 in the usual care group), the analysis could only detect a difference if the encounter rate almost doubled. This is a limitation of the data available and would need to be further explored in a larger population implementing the intervention to confirm findings. Lastly, the VSM trial was conducted in one integrated healthcare system. Results may not be representative of implementation in different health systems. For example, email encounters are not available in all health systems, so the interventions impact on utilization could look different in these environments. Additionally, effectiveness of the trial and cost estimates will also vary by location. Sensitivity analyses were conducted to address some of these issues in the cost evaluation, however there will be variation that was not considered.

PAGE 90

83 Conclusion In conclusion, vaccines are important public health intervention that requires effective intervention strategies to reach herd immunity and prevent the spread of infectious disease. The VSM intervention is a effective intervention strategy for health systems to reach and maintain high levels of vaccination in children at a low implementation and maintenance cost. Implementing the VSM intervention requires staff resources and is most cost effective when applied on a large health system national scale with staff who have lower salaries, but a high vaccination knowledge to efficiently address vaccine questions. Next steps are to evaluate the intervention cost, effectiveness, and adoption as implemented in a variety of settings.

PAGE 91

84 REFERENCES 1. Dempsey AF, Schaffer S, Singer D, Butchart A, Davis M, Freed GL. Alternative vaccination schedule preferences among parents of young children. Pediatrics. 2011;128(5):848-856. 2. Glanz JM, Newcomer SR, Narwaney KJ, Hambidge SJ, Daley MF, Wagner NM, McClure DL, Xu S, Rowhani-Rahbar A, Lee GM. A population-based cohort study of undervaccination in 8 managed care organizations across the United States. JAMA pediatrics. 2013;167(3):274-281. 3. Omer SB, Salmon, D.A., Orenstein, W.A., deHart, P., Halsey, N.. Vaccine Refusal, Mandatory Immunization, and the Risks of Vaccine-Preventable Diseases. New England Journal of Medicine. 2009;360(19):1981-1988. 4. Glanz JM, Wagner NM, Narwaney KJ, Kraus CR, Shoup JA, Xu S, O’Leary ST, Omer SB, Gleason KS, Daley MF. Web-based Social Media Intervention to Increase Vaccine Acceptance: A Randomized Controlled Trial. Pediatrics. 2017:e20171117. 5. Plans-Rubio P. Evaluation of the establishment of herd immunity in the population by means of serological surveys and vaccination coverage. Hum Vaccin Immunother. 2012;8(2):184-188. 6. Hill HA, Elam-Evans LD, Yankey D, Singleton JA, Kang Y. Vaccination Coverage Among Children Aged 19-35 Months --United States, 2016. Morbidity and Mortality Weekly Report (MMWR) Centers for Disease Control and Prevention. November 3, 2017;66(42):1171-1177. 7. Zhou F, Shefer A, Wenger J, Messonnier M, Wang LY, Lopez A, Moore M, Murphy TV, Cortese M, Rodewald L. Economic evaluation of the routine childhood immunization program in the United States, 2009. Pediatrics. 2014;133(4):577-585. 8. Sadaf A, Richards JL, Glanz J, Salmon DA, Omer SB. A systematic review of interventions for reducing parental vaccine refusal and vaccine hesitancy. Vaccine. 2013;31(40):4293-4304. 9. Jarrett C, Wilson R, O'Leary M, Eckersberger E, Larson HJ, Sage Working Group on Vaccine Hesitancy. Strategies for addressing vaccine hesitancy A systematic review. Vaccine. 2015;33(34):4180-4190. 10. Salmon DA, Dudley MZ, Glanz JM, Omer SB. Vaccine hesitancy: causes, consequences, and a call to action. Vaccine. 2015;33:D66-D71. 11. Assessing the state of vaccine confidence in the United States: recommendations from the National Vaccine Advisory Committee. Public Health Reports. 2015;130:573. 12. Brownson RC, Colditz GA, Proctor EK. Dissemination and implementation research in health: translating science to practice. Oxford University Press; 2017. 13. Westfall JM, Mold J, Fagnan L. Practice-based research—“Blue Highways” on the NIH roadmap. Jama. 2007;297(4):403-406.

PAGE 92

85 14. Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C. National Institutes of Health approaches to dissemination and implementation science: current and future directions. American journal of public health. 2012;102(7):1274-1281. 15. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM Framework: A Systematic Review of Use Over Time. American Journal of Public Health. 2013;103(6):e38-e46. 16. Glasgow RE, Vogt TM, Boles SM. Evaluating the Public Health Impact of Health Promotion Interventions: The RE-AIM Framework. American Journal of Public Health. 1999;89(9):1322-1327. 17. Glasgow RE. RE-AIM: Implementation of Health Behavior Interventions. 2018; http://www.re-aim.org/about/what-is-re-aim/implementation/ . Accessed June 13, 2018. 18. Centers for Disease Control and Prevention. Achievements in public health 1900-1990. Impact of Vaccines Universally Recommended for Children—United States, 1900-1999. MMWR Morb Mortal Wkly Rep. 1999(48):243-248. 19. Ten Great Public Health Achievements -United States, 2001-2010. Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report (MMWR) Centers for Disease Control and Prevention. 2011;60(19):619-623. 20. Dempsey AF, Schaffer S, Singer D, Butchart A, Davis M, Freed GL. Alternative vaccination schedule preferences among parents of young children. Pediatrics. 2011;128(5):848-856. 21. Glanz JM, Newcomer SR, Narwaney KJ, Hambidge SJ, Daley MF, Wagner NM, McClure DL, Xu S, Rowhani-Rahbar A, Lee GM, Nelson JC, Donahue JG, Naleway AL, Nordin JD, Lugg MM, Weintraub ES. A population-based cohort study of undervaccination in 8 managed care organizations across the United States. JAMA pediatrics. 2013;167(3):274-281. 22. Glanz JM, McClure DL, Magid DJ, Daley MF, France EK, Hambidge SJ. Parental refusal of varicella vaccination and the associated risk of varicella infection in children. Archives of pediatrics & adolescent medicine. 2010;164(1):66-70. 23. Glanz JM, McClure DL, Magid DJ, Daley MF, France EK, Salmon DA, Hambidge SJ. Parental refusal of pertussis vaccination is associated with an increased risk of pertussis infection in children. Pediatrics. 2009;123(6):1446-1451. 24. Glanz JM, McClure DL, O'Leary ST, Narwaney KJ, Magid DJ, Daley MF, Hambidge SJ. Parental decline of pneumococcal vaccination and risk of pneumococcal related disease in children. Vaccine. 2011;29(5):994-999. 25. Glanz JM, Narwaney KJ, Newcomer SR, Daley MF, Hambidge SJ, Rowhani-Rahbar A, Lee GM, Nelson JC, Naleway AL, Nordin JD, Lugg MM, Weintraub ES. Association between undervaccination with diphtheria, tetanus toxoids, and acellular pertussis (DTaP) vaccine and risk of pertussis infection in children 3 to 36 months of age. JAMA pediatrics. 2013;167(11):1060-1064.

PAGE 93

86 26. Feiken DR, Lezotte, D.C., Hamman, R.F., Salmon, D.A., Chen, R.T., Hoffman, R.E. Vaccine Refusal, Mandatory Immunization, and the Risks of Vaccine-Preventable Diseases. Journal of the American Medical Association. 2000;284(24):3145-3150. 27. Robison SG, Groom H, Young C. Frequency of alternative immunization schedule use in a metropolitan area. Pediatrics. 2012;130(1):32-38. 28. Phadke VK, Bednarczyk RA, Salmon DA, Omer SB. Association between vaccine refusal and vaccine-preventable diseases in the United States: a review of measles and pertussis. JAMA. 2016;315(11):1149-1158. 29. Lo NC, Hotez PJ. Public Health and Economic Consequences of Vaccine Hesitancy for Measles in the United States. JAMA pediatrics. 2017;171(9):887-892. 30. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearbook of medical informatics 2000: Patient-centered systems. 2000. 31. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med. 2007;32(5 Suppl):S119-126. 32. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clinical and translational medicine. 2013;2(1):10. 33. Glasgow RE, Lichtenstein E, Marcuse AC. Why Don’t We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition. American Journal of Public Health. 2003;93(8):1261-1267. 34. Dearing JW. Applying Diffusion of Innovation Theory to Intervention Development. Res Soc Work Pract. 2009;19(5):503-518. 35. Rhodes J, Whitney C, Ritzwoller DP, Glasgow RE. Stakeholder perspectives on costs and resource expenditures: tools for addressing economic issues most relevant to patients, providers, and clinics. Translational behavioral medicine. 2018. 36. Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB, Brito F, Blackman KC, Zoellner JM, Hill JL, Almeida FA, Glasgow RE, Estabrooks PA. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev. 2015;4:155. 37. Neta G, Sanchez MA, Chambers DA, Phillips SM, Leyva B, Cynkin L, Farrell MM, Heurtin-Roberts S, Vinson C. Implementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions. Implementation Science. 2015;10(1):4. 38. Mensah GA, Engelgau M, Stoney C, Mishoe H, Kaufmann P, Freemer M, Fine L. News from NIH: a center for translation research and implementation science. Translational behavioral medicine. 2015;5(2):127-130. 39. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.

PAGE 94

87 40. Rogers EM. Diffusion of innovations. New York: Free Press of Glencoe; 1962. 41. Rogers EM. Diffusion of Innovations. Fifth ed. New York, NY: The Free Press; 2003. 42. Glasgow RE. What is RE-AIM? http://www.re-aim.org/about/what-is-re-aim/ . Accessed June 27, 2018. 43. Sharp ND, Pineros SL, Hsu C, Starks H, Sales AE. A qualitative study to identify barriers and facilitators to implementation of pilot interventions in the Veterans Health Administration (VHA) Northwest Network. Worldviews on Evidence Based Nursing. 2004;1(2):129-139. 44. Raghavan R. The Role of Economic Evaluations in Dissemination and Implementation Research. In: Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and Implementation Research in Health: Translating Science to Practice . Second ed. New York, NY: Oxford University Press; 2018:89-199. 45. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing Behavioral Interventions: A Practical Guide to Enhance Translation. Annals of Behavioral Medicine. 2009;37(2):218227. 46. Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med. 2009;38(1):40-45. 47. Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res. 2004;6(4):e40. 48. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12(1):e4. 49. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cognitive behaviour therapy. 2009;38(4):196-205. 50. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression: systematic review. Journal of medical Internet research. 2009;11(2). 51. Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. American journal of preventive medicine. 2007;33(4):336-345. e316. 52. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from critical interpretive synthesis. Telemedicine and e-Health. 2012;18(2):137-144. 53. Santarossa S, Kane D, Senn CY, Woodruff SJ. Exploring the Role of In-Person Components for Online Health Behavior Change Interventions: Can a Digital Person-toPerson Component Suffice? Journal of medical Internet research. 2018;20(4).

PAGE 95

88 54. Enthoven AC. Integrated delivery systems: the cure for fragmentation. American Journal of Managed Care. 2009;15(12):S284. 55. Vogt TM, Lafata JE, Tolsma DD, Greene SM. The role of research in integrated health care systems: the HMO Research Network. The Permanente Journal. 2004;8(4):10. 56. United States Census Bureau: Quick Facts Colorado. 2017; https://www.census.gov/quickfacts/fact/map/CO/PST045217 . Accessed June 4, 2018. 57. Workgroup BFPS. 2015 Recommendations for Preventive Pediatric Health Care Committee on Practice and Ambulatory Medicine and Bright Futures Periodicity Schedule Workgroup. Pediatrics. 2015;136(3). 58. Kempe A, O’Leary ST, Kennedy A, Crane LA, Allison MA, Beaty BL, Hurley LP, Brtnikova M, Jimenez-Zambrano A, Stokley S. Physician response to parental requests to spread out the recommended vaccine schedule. Pediatrics. 2015:peds. 2014-3474. 59. Kennedy A, Basket M, Sheedy K. Vaccine attitudes, concerns, and information sources reported by parents of young children: results from the 2009 HealthStyles survey. Pediatrics. 2011:peds. 2010-1722N. 60. Glanz JM, Wagner NM, Narwaney KJ, Shoup JA, McClure DL, McCormick EV, Daley MF. A mixed methods study of parental vaccine decision making and parent–provider trust. Acad Pediatr. 2013;13(5):481-488. 61. Lieu TA, Zikmund-Fisher BJ, Chou C, Ray GT, Wittenberg E. Parents’ perspectives on how to improve the childhood vaccination process. Clin Pediatr. 2017;56(3): 238-246. 62. Shoup JA, Wagner NM, Kraus CR, Narwaney KJ, Goddard KS, Glanz JM. Development of an Interactive Social Media Tool for Parents With Concerns About Vaccines. Health education & behavior : the official publication of the Society for Public Health Education. 2014. 63. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues in clinical neuroscience. 2011;13(2):217. 64. Gillings D, Koch G. The application of the principle of intention–to–treat to the analysis of clinical trials. Drug Information Journal. 1991;25(3):411-424. 65. Centers for Disease Control and Prevention 2017 Childhood Diphtheria toxoid, Tetanus toxoid, acellular Pertussis (DTaP) Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imz-managers/coverage/childvaxview/datareports/dtap/reports/2017.html . Accessed February 13, 2019. 66. HEDIS Measures. http://www.ncqa.org/hedis-quality-measurement/hedis-measures . Accessed 5/3/2018, 2018. 67. Summary Table of Measures, Product Lines and Changes. HEDIS. 2018;2.

PAGE 96

89 68. Legare F, Ratte S, Gravel K, Graham ID. Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals' perceptions. Patient Educ Couns. 2008;73(3):526-535. 69. Selden TM. Compliance with well-child visit recommendations: evidence from the Medical Expenditure Panel Survey, 2000–2002. Pediatrics. 2006;118(6):e1766-e1778. 70. Centers for Disease Control and Prevention National Hospital Ambulatory Medical Care Survey: 2015 Emergency Department Summary Tables. 2015; https://www.cdc.gov/nchs/data/nhamcs/web_tables/2015_ed_web_tables.pdf . Accessed February 13, 2019. 71. Weiss AJ, Elixhauser A. Overview of hospital stays in the United States, 2012. HCUP Statistical Brief# 180. Agency for Healthcare Research and Quality, Rockville, MD October 2014. 72. Daley MF, Narwaney KJ, Shoup JA, Wagner NM, Glanz JM. Addressing Parents’ Vaccine Concerns: A Randomized Trial of a Social Media Intervention. American Journal of Preventive Medicine. 2018. 73. Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. Jama. 2012;308(19). 74. Meng D, Palen TE, Tsai J, McLeod M, Garrido T, Qian H. Association between secure patient–clinician email and clinical services utilisation in a US integrated health system: a retrospective cohort study. BMJ open. 2015;5(11):e009557. 75. Billups SJ, Moore LR, Olsone KL, Magid DJ. Cost-effectiveness evaluation of a home blood pressure monitoring program. Am J Manag Care. 2014;20(9):e380-e387. 76. Anand SG, Feldman MJ, Geller DS, Bisbee A, Bauchner H. A content analysis of e-mail communication between primary care providers and parents. Pediatrics. 2005;115(5):1283-1288. 77. Hoonakker PL, Carayon P, Cartmill RS. The impact of secure messaging on workflow in primary care: results of a multiple-case, multiple-method study. International journal of medical informatics. 2017;100:63-76. 78. Krist AH, Woolf SH, Bello GA, Sabo RT, Longo DR, Kashiri P, Etz RS, Loomis J, Rothemich SF, Peele JE. Engaging primary care patients to use a patient-centered personal health record. The Annals of Family Medicine. 2014;12(5):418-426. 79. Gorman BK, Braverman J. Family structure differences in health care utilization among US children. Social Science & Medicine. 2008;67(11):1766-1775. 80. Alio AP, Salihu HM. Maternal determinants of pediatric preventive care utilization among blacks and whites. Journal of the National Medical Association. 2005;97(6):792. 81. Fiscella K, Franks P, Gold MR, Clancy CM. Inequality in quality: addressing socioeconomic, racial, and ethnic disparities in health care. Jama. 2000;283(19):25792584.

PAGE 97

90 82. Garrido T, Raymond B, Wheatley B. Lessons From More Than a Decade in Patient Portals. Health Affairs Blog. April 7, 2016. 83. Buchmueller TC, Grumbach K, Kronick R, Kahn JG. Book review: The effect of health insurance on medical care utilization and implications for insurance expansion: A review of the literature. Medical care research and review. 2005;62(1):3-30. 84. Newhouse JP. Consumer-directed health plans and the RAND Health Insurance Experiment. Health Affairs. 2004;23(6):107-113. 85. Wharam JF, Landon BE, Galbraith AA, Kleinman KP, Soumerai SB, Ross-Degnan D. Emergency department use and subsequent hospitalizations among members of a highdeductible health plan. Jama. 2007;297(10):1093-1102. 86. Bennett GG, Glasgow RE. The Delivery of Public Health Interventions via the Internet: Actualizing Their Potential. Annual Review of Public Health. 2009;30(1):273-292. 87. Lee GM, Murphy TV, Lett S, Cortese MM, Kretsinger K, Schauer S, Lieu TA. Cost effectiveness of pertussis vaccination in adults. American journal of preventive medicine. 2007;32(3):186-193. e182. 88. Ehreth J. The global value of vaccination. Vaccine. 2003;21(7-8):596-600. 89. Ribisl KM, Leeman J, Glasser AM. Pricing health behavior interventions to promote adoption: lessons from the marketing and business literature. American journal of preventive medicine. 2014;46(6):653-659. 90. Owens DK, Siegel JE, Sculpher MJ, Salomon JA. Designing a Cost-Effectiveness Analysis. In: Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, eds. CostEffectiveness in Health and Medicine . Second ed. New York, NY: Oxford University Press; 2017:75-104. 91. Kroger A, Duchin J, Vzquez M. Best practices guidance of the Advisory Committee on Immunization Practices (ACIP). Atlanta, GA: US Department of Health and Human Services, CDC; 2017. 92. Adams DA, Jajosky RA, Ajani U, Kriseman J, Sharp P, Onwen D, Schley AW, Anderson WJ, Grigoryan A, Aranas AE. Summary of notifiable diseases--United States, 2012. MMWR Morbidity and mortality weekly report. 2014;61(53):1-121. 93. Centers for Disease Control and Prevention. National Notifiable Disease Surveillance System, 2017 Annual Tables of Infectious Disease Data. Atlanta, GA. . CDC Division of Health Informatics and Surveillance. 2018. 94. Liang JL, Tiwari T, Moro P, Messonnier NE, Reingold A, Sawyer M, Clark TA. Prevention of Pertussis, Tetanus, and Diphtheria with Vaccines in the United States: Recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recommendations and Reports. 2018;67(2):1.

PAGE 98

91 95. Cortese MM, Baughman AL, Zhang R, Srivastava PU, Wallace GS. Pertussis hospitalizations among infants in the United States, 1993 to 2004. Pediatrics. 2008;121(3):484-492. 96. Tanaka M, Vitek CR, Pascual FB, Bisgard KM, Tate JE, Murphy TV. Trends in pertussis among infants in the United States, 1980-1999. Jama. 2003;290(22):2968-2975. 97. Centers for Disease Control and Prevention 2017 Final Pertussis Surveillance Report. 2018. 98. Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, Kuntz KM, Meltzer DO, Owens DK, Prosser LA, Salomon JA, Sculpher MJ, Trikalinos TA, Russell LB, Siegel JE, Ganiats TG. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA. 2016;316(10):1093-1103. 99. Shoup JA, Madrid C, Koehler C, Lamb C, Ellis J, Ritzwoller DP, Daley MF. Effectiveness and Cost of Influenza Vaccine Reminders for Adults With Asthma or Chronic Obstructive Pulmonary Disease. American Journal of Managed Care. 2015;21(7):e405-e413. 100. Ritzwoller DP, Toobert D, Sukhanova A, Glasgow RE. Economic analysis of the Mediterranean Lifestyle Program for postmenopausal women with diabetes. The Diabetes Educator. 2006;32(5):761-769. 101. Ritzwoller DP, Sukhanova AS, Glasgow RE, Strycker LA, King DK, Gaglio B, Toobert DJ. Intervention costs and cost-effectiveness for a multiple-risk-factor diabetes selfmanagement trial for Latinas: economic analysis of Viva Bien! Translational Behavioral Medicine. 2011;1(3):427-435. 102. Neumann PJ, Russell LB, Siegel JE, Prosser LA, Krahn M, Mandelblatt JS, Daniels N, Gold MR. Using Cost-Effectiveness Analysis in Health and Medicine: Experiences since the Original Panel. In: Newmann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, eds. Cost-Effectiveness in Health and Medicine . Second ed. New York, NY: Oxford University Press; 2017:1-37. 103. Glasgow RE. RE-AIM: REACH of Health Behavior Interventions. 2018; http://reaim.org/about/what-is-re-aim/reach/ . Accessed May, 16, 2018. 104. Glasgow RE. RE-AIM: EFFECTIVENESS/EFFICACY of Health Behavior Interventions. 2018; http://re-aim.org/about/what-is-re-aim/effectiveness-or-efficacy/ . Accessed May 16, 2018, 2018. 105. Glasgow RE. Adoption of Health Behavior Interventions. 2018; http://www.reaim.org/about/what-is-re-aim/adoption/ . Accessed May 16, 2018, 2018. 106. Glasgow RE. Maintenance of Health Behavior Interventions. 2018; http://www.reaim.org/about/what-is-re-aim/maintenance/ . Accessed May 16, 2018, 2018. 107. Luman ET, Barker LE, Shaw KM, McCauley MM, Buehler JW, Pickering LK. Timeliness of childhood vaccinations in the United States: days undervaccinated and number of vaccines delayed. Jama. 2005;293(10):1204-1211.

PAGE 99

92 108. The Health Care Systems Research Network Virtual Data Warehouse. http://www.hcsrn.org/en/Tools%20&%20Materials/VDW/VDWDataModel/VDWSpecificati ons.pdf . Accessed February 13, 2019. 109. American Academy of Pediatrics Coding for Pediatric Prventive Care, 2016. Bright Futures. 2016. 110. What Statistical Analysis Should I use? Statistical Analyses Using SAS: UCLA statistical Analysis Consulting Group. https://stats.idre.ucla.edu/sas/whatstat/what-statisticalanalysis-should-i-usestatistical-analyses-using-sas/ . Accessed October 1, 2018. 111. Simon TD, Cawthon ML, Stanford S, Popalisky J, Lyons D, Woodcox P, Hood M, Chen AY, Mangione-Smith R. Pediatric medical complexity algorithm: a new method to stratify children by medical complexity. Pediatrics. 2014;133(6):e1647-e1654. 112. Opel DJ, Taylor JA, Zhou C, Catz S, Myaing M, Mangione-Smith R. The relationship between parent attitudes about childhood vaccines survey scores and future child immunization status: a validation study. JAMA pediatrics. 2013;167(11):1065-1071. 113. Opel DJ, Taylor JA, Mangione-Smith R, Solomon C, Zhao C, Catz S, Martin D. Validity and reliability of a survey to identify vaccine-hesitant parents. Vaccine. 2011;29(38):6598-6605. 114. Acock AC. Working with missing values. Journal of Marriage and family. 2005;67(4):1012-1028. 115. Gardner W, Mulvey EP, Shaw EC. Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. Psychological bulletin. 1995;118(3):392. 116. Hall DB. Zero inflated Poisson and binomial regression with random effects: a case study. Biometrics. 2000;56(4):1030-1039. 117. Vuong ST, Ko KC. A novel approach to protocol test sequence generation. Paper presented at: Global Telecommunications Conference, 1990, and Exhibition.'Communications: Connecting the Future', GLOBECOM'90., IEEE1990. 118. PASS 15 Power Analysis and Sample Size Software (2017). NCSS, LLC. Kaysville, Utah, USA, ncss.com/software/pass. 119. Gold MR. Cost-effectiveness in health and medicine. Oxford university press; 1996. 120. Misegades LK, Winter K, Harriman K, Talarico J, Messonnier NE, Clark TA, Martin SW. Association of childhood pertussis with receipt of 5 doses of pertussis vaccine by time since last vaccine dose, California, 2010. Jama. 2012;308(20):2126-2132. 121. Colorado Department of Public Health and Environment Numbers and Rates of Reported Pertussis Cases by Age Group, Colorado, 1/1/2017 12/31/2017. 2017; https://drive.google.com/drive/folders/1K34Z4CwGuWckY1C8Iqo7YvlFxg84GZjG . Accessed February 6, 2019, 2019.

PAGE 100

93 122. Bureau of Labor Statistics. 2018; https://www.bls.gov/mwe/ . Accessed June 3, 2018. 123. Economic News Release: Table2. Civilian workers, by occumpational and industry group. Bureau of Labor Statistics. September 2018. 124. WordPress.org Our Mission. 2018; https://wordpress.org/about/ . Accessed June 3, 2018. 125. Centers for Disease Control and Prevention. Vaccines for Children Program (VFC) price list. 2018; https://www.cdc.gov/vaccines/programs/vfc/awardees/vaccinemanagement/price-list/index.html . Accessed February 13, 2019. 126. Colorado Department of Health Care Policy and Financing: Provider Rates & Fee Schedule. 2017; https://www.colorado.gov/pacific/hcpf/provider-rates-fee-schedule . Accessed June 26, 2018. 127. Chen RT, Hadler SC, Terracciano GJ, Tuttle J, Watson JC. Update; vaccine side effects, adverse reactions, contraindications, and precautions: recommendations of the Advisory Committee on Immunization Practices (ACIP). 128. Hamborsky J, Kroger A, Wolfe C. Epidemiology and Prevention of Vaccine-preventable Diseases: The Pink Book: Course Textbook. Public Health Foundation; 2015. 129. Shinefield HR, Black S, Ray P, Chang I, Lewis N, Fireman B, Hackell J, Paradiso PR, Siber G, Kohberger R. Safety and immunogenicity of heptavalent pneumococcal CRM197 conjugate vaccine in infants and toddlers. The Pediatric infectious disease journal. 1999;18(9):757-763. 130. Lieu TA, Ray GT, Black SB, Butler JC, Klein JO, Breiman RF, Miller MA, Shinefield HR. Projected cost-effectiveness of pneumococcal conjugate vaccination of healthy infants and young children. Jama. 2000;283(11):1460-1468. 131. Belongia EA, Irving SA, Shui IM, Kulldorff M, Lewis E, Yin R, Lieu TA, Weintraub E, Yih WK, Li R. Real-time surveillance to assess risk of intussusception and other adverse events after pentavalent, bovine-derived rotavirus vaccine. The Pediatric infectious disease journal. 2010;29(1):1-5. 132. Zhou F, Bisgard KM, Yusuf HR, Deuson RR, Bath SK, Murphy TV. Impact of universal Haemophilus influenzae type b vaccination starting at 2 months of age in the United States: an economic analysis. Pediatrics. 2002;110(4):653-661. 133. Zhou F, Santoli J, Messonnier ML, Yusuf HR, Shefer A, Chu SY, Rodewald L, Harpaz R. Economic evaluation of the 7-vaccine routine childhood immunization schedule in the United States, 2001. Archives of pediatrics & adolescent medicine. 2005;159(12):11361144. 134. Jackson LA, Yu O, Nelson JC, Dominguez C, Peterson D, Baxter R, Hambidge SJ, Naleway AL, Belongia EA, Nordin JD. Injection site and risk of medically attended local reactions to acellular pertussis vaccine. Pediatrics. 2011;127(3):e581-e587.

PAGE 101

94 135. Zangwill KM, Eriksen E, Lee M, Lee J, Marcy SM, Friedland LR, Weston W, Howe B, Ward JI. A population-based, postlicensure evaluation of the safety of a combination diphtheria, tetanus, acellular pertussis, hepatitis B, and inactivated poliovirus vaccine in a large managed care organization. Pediatrics. 2008;122(6):e1179-e1185. 136. Moore D, Le Saux N, Scheifele D, Halperin S. Members of the Canadian Paediatric Society/Health Canada Immunization Monitoring Program, ACTive (IMPACT). Lack of evidence of encephalopathy related to pertussis vaccine: Active surveillance by IMPACT, Canada, 1993-2002. Pediatr Infect Dis J. 2004;23(6):568-571. 137. Ray P, Hayward J, Michelson D, Lewis E, Schwalbe J, Black S, Shinefield H, Marcy M, Huff K, Ward J. Encephalopathy after whole-cell pertussis or measles vaccination: lack of evidence for a causal association in a retrospective case–control study. The Pediatric infectious disease journal. 2006;25(9):768-773. 138. Huang W-T, Gargiullo PM, Broder KR, Weintraub ES, Iskander JK, Klein NP, Baggs JM, Team VSD. Lack of association between acellular pertussis vaccine and seizures in early childhood. Pediatrics. 2010;126(2):263-269. 139. Centers for Disease Control and Prevention. Vaccine Information Sheet: DTaP Vaccine. 5/1/2007. 140. Centers for Medicare and Medicaid Services: Fee Schedule: General Information. 2018; https://www.cms.gov/Medicare/Medicare-Fee-for-ServicePayment/FeeScheduleGenInfo/index.html . Accessed June 26, 2018. 141. Consumer Price Index: Medical care services in U.S. city average, all urban consumers, not seasonally adjusted. U.S. city average. Bureau of Labor Statistics. 2018. 142. Briggs AH, Mooney CZ, Wonderling DE. Constructing confidence intervals for cost effectiveness ratios: an evaluation of parametric and non parametric techniques using Monte Carlo simulation. Statistics in medicine. 1999;18(23):3245-3262. 143. Zhou YY, Kanter MH, Wang JJ, Garrido T. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health affairs. 2010;29(7):1370-1375. 144. Masseria C, Martin CK, Krishnarajah G, Becker LK, Buikema A, Tan TQ. Incidence and burden of pertussis among infants less than 1 year of age. The Pediatric infectious disease journal. 2017;36(3):e54. 145. Dunn A, Grosse SD, Zuvekas SH. Adjusting health expenditures for inflation: a review of measures for health services research in the United States. Health services research. 2018;53(1):175-196. 146. Koh HK, Sebelius KG. Promoting prevention through the affordable care act. New England Journal of Medicine. 2010;363(14):1296-1299. 147. Centers for Disease Control and Prevention, National Notifiable Disease Surveillance System and Sypplemental Pertussis Surveillance System. Reported pertussis incidence by age group: 1990-2016. https://www.cdc.gov/pertussis/images/incidence-graphage.png . Accessed June 3, 2018.

PAGE 102

95 148. Centers for Disease Controla nd Prevention. Diphtheria toxoid, Tetanus toxoid, acellular Pertussis (DTaP) vaccination coverage among children 19-35 months by State, HHS Region, and the United States, National Immunization Survey-Child (NIS-Child), 1995 through 2016. https://www.cdc.gov/vaccines/imz-managers/coverage/childvaxview/datareports/dtap/trend/index.html . Accessed June 3, 2018. 149. Colorado Department of Public Health and Environment Numbers and Rates of Reported Pertussis Cases by Age Group, Colorado, 1/1/2012 12/31/2012. 2012; https://drive.google.com/drive/folders/1gm7XqZCFrrRZ7VLnG47Zq8fzP0FOsDuc . Accessed June 3, 2018. 150. Centers for Disease Control and Prevention 2012 Final Pertussis Surveillance report. 2015. 151. Centers for Disease Control and Prevention 2017 Childhood Polio Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imz-managers/coverage/childvaxview/datareports/polio/reports/2017.html . Accessed March 14, 2019. 152. Centers for Disease Control and Prevention 2017 Childhood Haemophilus influenzae type b (Hib) Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imzmanagers/coverage/childvaxview/data-reports/hib/index.html . Accessed March 14, 2019. 153. Centers for Disease Control and Prevention 2017 Childhood Hepatitits B (HepB) Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imzmanagers/coverage/childvaxview/data-reports/hepb/reports/2017.html . 154. Centers for Disease Control and Prevention 2017 Childhood Pneumococcal Conjugate Vaccine (PCV) Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imzmanagers/coverage/childvaxview/data-reports/pcv/reports/2017.html . Accessed March 14, 2019. 155. Centers for Disease Control and Prevention 2017 Childhood Rotavirus Vaccine (PCV) Vaccination Coverage Report. 2017; https://www.cdc.gov/vaccines/imzmanagers/coverage/childvaxview/data-reports/rotavirus/reports/2017.html . 156. Centers for Disease Control and Prevention. Vaccine Information Statement: Rotavirus Vaccine. 2018. 157. Eastin MS, Guinsler NM. Worried and wired: effects of health anxiety on informationseeking and health care utilization behaviors. CyberPsychology & Behavior. 2006;9(4):494-498. 158. Austin PC, Mamdani MM, Juurlink DN, Hux JE. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. Journal of clinical epidemiology. 2006;59(9):964-969. 159. Centers for Disease Control and Prevention. Ten Great Public Health Achievements -United States, 1900-1999. Morbidity and Mortality Weekly Report (MMWR) Centers for Disease Control and Prevention. 1999;48(12):241-243. 160. Centers for Disease Control and Prevention: Measles Cases in 2019. 2019.

PAGE 103

96 161. Omer SB, Enger KS, Moulton LH, Halsey NA, Stokley S, Salmon DA. Geographic Clustering of Nonmedical Exemptions to School Immunization Requirements and Associations With Geographic Clustering of Pertussis. American Journal of Epidemiology. 2008;168(12):1389-1396. 162. Omer SB, Salmon DA, Orenstein WA, deHart MP, Halsey N. Vaccine Refusal, Mandatory Immunization, and the Risks of Vaccine-Preventable Diseases. New England Journal of Medicine. 2009;360(19):1981-1988. 163. Chait N, Glied S. Promoting prevention under the Affordable care act. Annual review of public health. 2018;39:507-524. 164. Allison MA, O'Leary ST, Lindley MC, Crane LA, Hurley LP, Beaty BL, Brtnikova M, Jimenez-Zambrano A, Babbel C, Berman S. Financing of Vaccine Delivery in Primary Care Practices. Academic pediatrics. 2017;17(7):770-777. 165. Shoup JA, Narwaney KJ, Wagner NM, Kraus CR, Gleason KS, Albright K, Glanz JM. Social Media Vaccine Websites: A Comparative Analysis of Public and Moderated Websites. Health Education & Behavior. 2018:1090198118818253. 166. Kata A. A postmodern Pandora's box: anti-vaccination misinformation on the Internet. Vaccine. 2010;28(7):1709-1716. 167. Kahwati LC, Lance TX, Jones KR, Kinsinger LS. RE-AIM evaluation of the Veterans Health Administration's MOVE! weight management program. Translational behavioral medicine. 2011;1(4):551-560. 168. Centers for Medicare & Medicaid Services, Office of the Actuary, National Health Statistics Group, National Health Expenditure Accounts, National health expenditures. Table 94. National health expenditures, average annual percent change, and percent distribution, by type of expenditure: United States, selected years 1960–2016. 2018.

PAGE 104

97 APPENDIX Table 1: ICD Codes Used to Identify Well Child Visits in the EHR E ncounters Code ICD type Definition V20.31 9 Health supervision for newborn under 8 days old V20.32 9 Health supervision for newborn 8 28 days old V20.2 9 Routine infant or child health check Z00.110 10 Health supervision for newborn under 8 days old Z00.111 10 Health supervision for newborns 8 to 28 days old Z00.121 10 Routine child health exam with abnormal findings Z00.129 10 Routine child health exam without abnormal findings Table 2: Private Sector Costs for Vaccines Administered in the first 200 Days of L ife Vaccine Diseases Prevented by Vaccine Quantity of Vaccines Cost Full Vaccine Schedule Pediarix Diphtheria, Tetanus, Pertussis, Hepatitis B, Polio 2 $153.90 Act Haemophilus influenzae t ype b 2 $33.02 Prevnar Pneumococcal 2 $360.10 Rotateq Rotavirus 2 $165.78 Vx Admin 8 $151.44 2a Total $ 864.24 Pertussis Vaccination Infanrix Diphtheria, Tetanus, Pertussis 2 $48.10 Vx Admin 2 $37.86 2b Total $85.96 Table 3: Encounter counts by Encounter Type including the Number and Percentage of Participants with No Encounters and Range in Encounter Frequency Type of E ncounter Number with No Encounters Range of E ncounters Well child visit 8 (1%) 0-10 Other outpatient 157 (22%) 0-22 ED 615 (86%) 0-6 Inpatient 684 (96%) 0-2 Email 324 (45%) 0-38 Telephone 33 (5%) 0-53