Citation
Healthcare effectiveness using information technology

Material Information

Title:
Healthcare effectiveness using information technology
Creator:
Alnsour, Yazan
Place of Publication:
Denver, Colo.
Publisher:
University of Colorado Denver
Publication Date:

Thesis/Dissertation Information

Degree:
Doctorate ( Doctor of Philosophy)
Degree Grantor:
University of Colorado Denver
Degree Divisions:
Department of Computer Science and Engineering, CU Denver
Degree Disciplines:
Computer science

Record Information

Source Institution:
University of Colorado Denver Collections
Holding Location:
Auraria Library
Rights Management:
Copyright YAZAN ALNSOUR. Permission granted to University of Colorado Denver to digitize and display this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.

Downloads

This item has the following downloads:


Full Text
HEALTHCARE EFFECTIVENESS USING INFORMATION TECHNOLOGY
by
YAZAN ALNSOUR B.S., University of Jordan, 2007 M.B.A., New York Institute of Technology, 2009
A thesis submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfilment of the requirements for the degree of Doctor of Philosophy
Computer Science and Information Systems Program
2016


This thesis for the Doctor of Philosophy degree by Yazan Alnsour has been approved for the
Computer Science and Information Systems Program
by
Dawn Gregg, Chair Jiban Khuntia, Advisor Onook Oh Ilkyeun Ra
Date: December 17, 2016


Alnsour, Yazan (Ph.D. Computer Science and Information Systems)
Healthcare Effectiveness Using Information Technology Thesis directed by Assistant Professor Jiban Khuntia
ABSTRACT
The effectiveness of healthcare technology plays an important role as the predictor of its success or failure. This thesis proposal focuses on this phenomenon and is comprised of two essays that delve into the impact that information technology can have on healthcare delivery. The first essay explores the effect of medical alerts and complementary comorbid alerts when viewed via a web portal, on the closure time of a focal medical alert. The objective of the first essay is to quantify the impact of the web-based alerts on patients. The second essay investigates what type functionalities of mobile health applications (apps) have the greatest impact on patients and how this leads to better health apps evaluation. The combination of these two essays sets the foundation for the statement that the usage and subsequent effectiveness of technology are dependent on the patient-technology interaction characteristics and functionalities.
Data for the first essay comes from a national healthcare provider and includes of two datasets consisting of 2,164 diabetic patients and 8,473 chronic kidney disease patients. The data for the second essay comes from a dataset of 185 mobile health applications from the Android App Store, tracked for 14 weeks to form a panel dataset. These two studies will contribute to the existing literature in exploring how health IT can positively impact healthcare and how particular health IT features and attributes enhance the efficiency of healthcare delivery. Additionally, we highlight managerial implications to support in the development and design of health IT with the goal of providing more efficient and


sustainable healthcare delivery and services. This dissertation also includes research and practical implications that are paramount to gain an understanding of the strategies needed when striving to enhance the design of health IT artifacts, which can potentially increase the overall effectiveness of healthcare delivery.
The form and content of this abstract are approved. I recommend its publication.
Approved: Jiban Khuntia
IV


TABLE OF CONTENTS
CHAPTER
I. THESIS OVERVIEW....................................................................1
Objectives and Structures of the Essays in the Thesis...............................3
II. EFFECTIVENESS OF ALERTS ON TEST CLOSURE TIME FOR DIABETES AND
KIN DMA PATIENTS....................................................................6
Abstract...........................................................................6
Introduction.......................................................................7
Prior Literature....................................................................9
Theoretical Framework..............................................................12
Information Cues and Decision-Making............................................12
ELM Central and Peripheral Routes...............................................17
Hypotheses Development.............................................................19
Method............................................................................25
Data and Variables..............................................................25
Estimation Models and Specifications............................................29
Results...........................................................................31
Results of Regression Analysis..................................................31
Results of Survival Analysis....................................................33
Discussion........................................................................36
Findings........................................................................36
v


Implications........................................................................38
Contributions.......................................................................40
Limitations and Suggestions for Future Research.....................................40
Conclusion............................................................................41
Figures and Tables....................................................................43
III. THE ROLE OF EFFECTIVENESS, EASE OF USE, AND FUNCTIONALITY ON
EVALUATION OF HEALTH APPS............................................................56
Abstract..............................................................................56
Prior Literature......................................................................59
Theoretical Framework.................................................................62
Task and Technology Fit and Information System Success..............................62
Task and Technology Fit in Health Apps..............................................66
Assimilation and Contrast Effects...................................................67
Conceptual Model and Hypotheses Development...........................................68
Method................................................................................74
Data and Variables..................................................................74
Empirical Analysis..................................................................76
Results...............................................................................77
Discussion and Conclusion.............................................................79
Figures and Tables....................................................................82
VI


REFERENCES
89
APPENDIX
A. Focal and Comorbid Alerts for Diabetes and CKD.....................................110
B. Detailed Estimation Results........................................................113
C. GMM Estimation Approach............................................................117
D. Interaction Graphs.................................................................118
E. Text analytics.....................................................................121
F. Literature review of apps effectiveness and impact studies in healthcare...........126
G. Detailed Results of Panel Estimation Models........................................129
H. Detection of Fake Reviews..........................................................131
vii


LIST OF TABLES
TABLE
1: Descriptive Statistics.............................................................50
2: Correlation amongst Variables......................................................50
3: T-Test of Variables by Alert Closed for Diabetes and CKD Samples.................50
4: Linear Estimation Models...........................................................50
6: Parametric and Semi-Parametric Estimation Models.................................50
5: Description of Variables...........................................................55
7: Classification of Health App Functionalities and Features..........................83
8: Key Variables Description..........................................................84
9: Descriptive Statistics amongst Key Variables.....................................85
10: Panel Estimation Fixed Effect Models............................................77
11: Description of Variables..........................................................87
12: Correlation amongst Key Variables.................................................88
13: Comorbid Alerts for Diabetes.....................................................112
14: Comorbid Alerts for CKD..........................................................112
15: Linear Estimation Models with Details Results...................................113
16: Parametric and Semi-Parametric Estimation Models with Detailed Results..........114
17: Summary Statistics of Sample for Survivor Models................................115
18: Log-rank Test for Equality of Survivor Functions, Compared by Comorbid Alerts...115
19: Ethno-Nationality Distribution for Diabetes Patients............................115
20: Results of Estimation Models GMM..............................................117
21: Literature Review on App Functionalities/Effectiveness..........................126
22: Results of Panel Estimation Models................................................129
viii


LIST OF FIGURES
FIGURE
1: Conceptual Model..................................................................43
2: Interaction Plot Diabetes Sample..................................................43
3: Interaction Plot for CKD Sample...................................................43
4: The Smoothed Hazard Estimation Plot Diabetes Sample............................43
5: The Smoothed Hazard Estimation Plot CKD Sample.................................43
6: The K-M Plot Showing with Comorbid vs. Non-Comorbid Diabetes Patients.............44
7: The K-M Plot Showing with Comorbid vs. Non-Comorbid CKD Patients..................44
8: Diabetes Sampling Process.........................................................45
9: CKD Sampling Process..............................................................46
10: Alert Closure Flow Chart.........................................................47
11: The Alert Page of Portal Showing No Alerts....................................48
12: An Alert Page Showing a Childs Vaccination Alert................................48
13: An Alert Page Showing a Flu Vaccine Alert........................................48
14: An Alert Page Showing Multiple Alerts along with A1C Test Alert...............49
15: Multiple Peripheral Alerts.......................................................49
16: Conceptual Model.................................................................82
17: Plot for the interaction effect of functionalities and health effectiveness.....118
18: Plot for the interaction effect of integrative quality and health effectiveness.119
19: Plot for the interaction effect of instructive quality and health effectiveness.120
20: Data Cleaning and Variables Coding..............................................122
21: Coding of Key Variables using Text Mining.....................................125
IX


CHAPTER I
THESIS OVERVIEW
The evolution in the field of information technology (IT) has resulted in the various improvements in the healthcare industry (Goh et al. 2011; Iribarren et al. 2016; Lenz and Kuhn 2004; Yang et al. 2015). One of the notable contributions of information technology to healthcare is access to health information and services (Ahem et al. 2011). This access gave patients access to the health information and electronic services that are available via the internet which was not possible in the past. Today, with just a few clicks, patients can learn more about their diagnosis as well as self-management and care by accessing the information through different Health IT artifacts (Krist and Woolf 2011).
To this day, more and more people are opting to access health information and services through the internet (de Lusignan et al. 2014; Zach et al. 2012). Accessibility and connectivity are the main driving factors for the enablement of information, service access of the end-users, and technological access to healthcare services such as teleconsultation (Komak 2016; Vukovac et al. 2015). According to Thompson and Brailer (2004), with the increasing level of connectivity and utilization of the internet, patients are becoming more and more dependent and reliant on the internet to find healthcare information and health-related services. This reliance is especially true in a system where increasing demands for connectivity has resulted in the reduction of the cost of the internet related services such as data plans, and the emergence of competitive prices of smartphones. With this, the electronic and online access to healthcare information has minimized the patients physical barriers in accessing healthcare information and care (Wald et al. 2007).
Using the internet and other methods of digital communication can enhance the efficiency of the delivery of healthcare and can empower patients to manage their health
1


better and take ownership of their ailments (Black et al. 2011). Access to healthcare information and services are pivotal in patients abilities to make better decisions regarding their health and conditions (Randeree 2009). This is the very reason why the electronic health-related services that healthcare workers and professionals provide to their patients are critical in influencing positive health outcomes from the patients (Goldzweig et al. 2013). Such contains the key to greater initiatives towards being responsible for ones health and patients empowerment (Demiris et al. 2008). Hence, the effectiveness of information technology can be measured by how patients are using it. Therefore, the design of a particular technology and its advanced features and functionalities can be pivotal for its success in enhancing the efficiency of healthcare delivery.
Importance of the Topic
As the cost of healthcare and the patient-to-doctor ratio continues to increase, the need for efficient and effective tools to deliver healthcare services has become a priority for the majority of healthcare providers (Douglas et al. 2011; Goldzweig et al. 2013; Jamal et al. 2009). Engaging patients in managing their diseases will reduce some of the major healthcare overheads and burdens related to communication and information dissemination (Oshima Lee and Emanuel 2013). The use of HIT can empower the patients while at the same time allowing providers to deliver more efficient care (Graffigna et al. 2013).
As functional advancements in Health IT have been made over the years, healthcare providers are working to identify specific characteristics of engagement that positively impact patients, empower them, and make them more involved in the healthcare process (Or and Karsh 2009). In recent years, patient web portals and health applications (apps) have become a popular topic amongst both researchers and healthcare communities, with the idea
2


behind them being better to facilitate the communication between healthcare providers and patients and deliver care in a more efficient way.
Objectives and Structures of the Essays in the Thesis
The essays in this thesis focus on two research questions and are held together by a unifying common theme, which is to explore how technology can have a positive impact on the efficiency of healthcare delivery.
The first essay of this thesis asks the question: how do web-based medical reminders impact the closure time of focal diagnostic tests. As health providers are using portals to provide test alerts to patients, and portals facilitate a patients remote access to view the test alerts, it is proposed that increased engagement and views of the test alerts have a positive effect on patients taking responsibility for the management of their health. In the first essay of this thesis, we investigate the complementary effects of the number of focal alerts viewed, and the presence of multiple comorbid alerts within a portal, on the closure time of the focal alert. We argue that these visual alerts provide informational cues that influence a patients decision-making process and motivate him or her to proceed toward conducting the needed critical diagnostic test. With every view of the alert, the information provided reinforces the influence on the patients decision to take action and conduct the prescribed test at the earliest possible opportunity. The theoretical framework and hypotheses for this study are grounded in the decision-making literature and positioned with the understanding that external information provided to the patient will influence the decision-making process and impact the patients behavior. When compared to existing studies regarding patient portal impact, this study includes a deeper investigation into portals and analyzes what particular
3


features positively impact patient behavior and improve patients care. The study finds supporting evidence for using data from a national provider.
The second essay addresses the research question: how do health application (apps) functionalities and effectiveness impact the apps evaluation. Patients evaluate applications based on a combination of criteria, which is formed from comparing the applications functionality, appeal, and effectiveness of existing or previous experiences. The comparison of the users expectations versus existing experience draws on the theoretical foundation of the Task-Technology Fit and the Assimilation and Contrast theory. Determining the functionalities and effectiveness of applications that most greatly influence users evaluations of health applications remains an unexplored question in the existing literature. 188 health applications were analyzed to explore the impact of functionalities and their appeal on a patients evaluation of the application, through candid user reviews. To test the hypotheses, an empirical analysis is proposed to be conducted using data from Android market place for 188 applications, collected over a period of three months.
The combination of the two essays sets the foundation for the argument that candid user reviews of the functionality and effectiveness of an application can greatly influence other users decisions in adopting the application, such as a patient portal. It is paramount that users feel motivated to adopt the above prescribed patient portal, which in some cases is an application-based channel of communication, in order to receive more personalized information regarding their health, which promotes greater self-responsibility, and at the same time assists healthcare providers in delivering better and more efficient services.
To sum up, the objective of this thesis is to explore the effectiveness of health information technologies. The first essay effectiveness of alerts on test closure time for
4


diabetes and kidney patients is already completed to its best, and is included in this thesis proposal. The second study, The Role of Effectiveness, Appeal and Functionality on Evaluation of Health Apps is being proposed as work in progress as we continue to conduct our analysis.
5


CHAPTER II
EFFECTIVENESS OF ALERTS ON TEST CLOSURE TIME FOR DIABETES AND
KINDNEY PATIENTS Abstract
Health providers are using internet-based patient portals to enable information access and communication to patients. Providers can initiate and inform patients about the necessary laboratory and diagnostic tests for treatment and disease management through the portals, known as test alerts. Portals facilitate a patients remote access to view the test alerts. However, whether portal based alerts are effective to impact patients decision and time to conduct the test (i.e., test closure) is not explored in existing literature. This study investigates the complementary effects of some clicks, and multiple comorbid alerts in a portal, on the closure time of a focal alert. We argue that alerts provide cues to influence a patients decision-making process to conduct a test, and every click on the alert reinforces the influence the cues on patients decision to conduct the test earlier. We posit that multiple comorbid alerts intensify the influence of the cues and complement the effect of clicks to conduct a test earlier to reduce test closure time. We empirically examine and find support for the hypothesized relationships using two sample of archival data sets for 2,164 diabetes patients, and 8,473 chronic kidney patient (CKD). Our additional exploration using survival analysis for the variant time effect of the alerts on the closure time suggests a span of high probability threshold for alert closures that we recommend as intervention periods. We discuss the practice implications and contributions of the findings.
Keywords: patient portal, alert, diabetes, chronic kidney disease, comorbid alerts, focal alert, survival analysis.
6


Introduction
Patient portals enable information access and communication to patients from the providers. Portals pull necessary medical information from the patients electronic health record and digital resources at the providers end. On the patients end, portals help to access and view information regarding disease and care management. Portals allow patients to communicate electronically and securely with their providers, request prescription refills, pay bills, review lab results and schedule medical appointments (Ammenwerth et al. 2012). Researchers note that portals have the potential to empower patients, improve care efficiency and effectiveness through information sharing, access and guiding patients for timely actions (Goldzweig et al. 2013).
A distinct feature of a patient portal is the provision for doctors to recommend test alerts to patients. Providers initiate and inform patients about the necessary laboratory and diagnostic tests through the portal. Portals facilitate a patients remote access to view the test alerts. Patients should follow the instructions, and conduct tests for proper diagnosis and management of diseases (Goldzweig et al. 2013; Kruse et al. 2015). Once the tests are done, the providers interconnected systems automatically remove the alerts in the patient portal, which is commonly known as test closure or alert closure (see Figures 11, 12, 13, and 14 for images on alert pages and Figure 10 for flow chart regarding alert placing and closure). Some examples of alerts are albumin limit tests, blood sugar tests, uric acid tests, ferritin levels test for iron deficiency, etc.
Alerts serve as a mechanism to empower patients to take actions regarding their health condition and be aware of the severity of their condition. Alerts also help providers to strategically develop a plan for patients disease management and monitoring process. Without a mechanism to inform the patients to conduct the tests, the disease management
7


process often becomes reactive to a situation where the disease progression would have reached to an unmanageable stage (McCoy et al. 2012). For example, when a patient gets alerts and conducts regular cholesterol tests, he or she can take medication to control cholesterol levels, otherwise of which he or she may suffer from a severe heart attack.
The patient alert feature is an important one to establish the portal effectiveness. Existing research has mainly explored the usability, usage, and adoption of portals (see reviews of studies on patient portals by Amante et al. 2014; Ammenwerth et al. 2012; Goldzweig et al. 2013; Kruse et al. 2015). Prior research suggests that patients are using portals frequently (Weingart et al. 2006), gradually overcoming the barriers that used to persist in the initial days of the introduction of portals in the healthcare system (Britto et al. 2009). However, Empirical investigation of portal effectiveness is sparse in the extant literature.
In this study, we ask the research question, do alerts influence a patients decision to do a test? If so, how do patients differ this decision making? And, what is the threshold period of the alert effect on closure? We contextualize this study to diabetes and chronic kidney disease (CKD) patients, two widespread chronic diseases. The economic burden of diabetes and CKD is huge in the United States (US), with 12% to 14% of the United States population being affected by diabetes, and 40% with pre-diabetes (Menke et al. 2015), and 14 % of US population have some form of kidney disease (de Boer et al. 2011). These diseases have an estimated impact of more than $200 billion annual burdens (American Diabetes Association 2013). Early diagnosis and intervention for diabetes and CKD may aid in patients to adopt medication and lifestyle changes to return their lifestyle to normal (American Diabetes Association 2014; Drawz and Rahman 2015; Hussain et al. 2007). To
8


aid in the diagnosis and management of diabetes and CKD, routine tests are prescribed. For diabetes, the diagnostic test is Glycated hemoglobin (e.g., A1C test) that tests the blood sugar level, and for CKD, the diagnostic test is Glomerular Filtration Rate (GFR) that checks the bold creatinine test. Because of the diagnostic importance, the two tests are considered as focal alerts in this study. Along with the A1C and GFR, providers also prescribe a set of associated test, when it is suspected that the patient may have other diseases along with diabetes or CKD. These associated test alerts are defined as comorbid alerts. We provide a detail description of the comorbid alerts taken for this study in Appendix A.
Theoretically, we argue that with every click on the alert, a patient derives a higher motivation to conduct the test through a cue-influenced decision-making process. We posit that the influence of alert clicks on the patients decision making will be high due to the concern with multiple diseases, and therefore, the number of comorbid alerts will lead to an early closure of the alert. Therefore, we further hypothesize for complementary effects of some alert clicks and number comorbid alerts on test closure time. We empirically examine and find support for the hypothesized relationships using an archival data set for two distinct samples of 2,164 diabetes and 8,473 CKD patients. Also, we use survival analysis to explore the time-based effect of alert clicks on test closures and identify threshold periods with high alert closure probabilities. We suggest the threshold period as an intervention spans for the alerts to be more effective. We discuss the practice implications and contributions of the findings.
Prior Literature
Patient portals play a significant role for access to health information. A health care provider normally manages portals. Technology enables transforming data into information
9


(Tallon et al. 2013). Patient web portals provide health information access, interactive services, decision support functionalities, and health reminders to patients (Carrell and Ralston 2006). Researchers suggest that patient portals have a positive impact on health outcomes. With more patients opting in to use the portals and participate in the data collection and monitoring process from their home using devices, care managers receive insights derived from analysis of a patients integrated data streams (Byczkowski et al. 2011; Jones et al. 2015; Neuner et al. 2015). Mostly, with increasing use of portals, they are emerging as avenues for maintaining the continuing of care beyond hospitals and clinics and are helpful in managing diseases that need continuous monitoring and care (Ammenwerth et al. 2012). As a result, portals are enabling providers to achieve the goal of enhancing engagement with the patient, so potential health problems are spotted and addressed early. There is an increased interest among both petitioners and academic to better understand the impact of the healthcare information technology on healthcare delivery (Hah and Bharadwaj
2012).
We ae seeing an increased reliance on healthcare providers on information technology to increase patients adherence to guidelines (Agarwal et al. 2010; Wu et al. 2006), disease management with chronic conditions (Grant et al. 2008; Ross et al. 2004) and ambulatory care (Romano and Stafford 2011). Patient portals are emerging as plausible to provide a means to enhance the role of information technology in healthcare. Emerging information systems such as electronic health records, clinical information systems, and patient health records are offering new opportunities for efficient and high-quality patient care (Wu et al. 2006). Subsequently, the patients role is also changing from a directed patient to an informed patient who is responsible and competent for managing his health
10


(Buntin et al. 2011; Holroyd-Leduc et al. 2011). In this process, patient portals are arguably emerging as an important means to involve patients in his or her treatment. Patient Portals can plausibly provide multiple potential benefits such as conducting timely tests, compliance to medication and treatment regime; and emerge as an inherent tool for patient engagement (Alpay et al. 2011; Gianchandani 2011; Wilson et al. 2014).
Existing studies suggest that Web-based interventions are effective in increasing patient empowerment compared with usual care or face-to-face interventions for diseases, such as increasing self-efficacy or mastery with the disease management process (for a review of existing literature on web-based interventions, see Samoocha et al. 2010). Emont (2011) explained how patient portals can help bridging care issues if it was linked correctly with the electronic medical records, and note that external factors like meeting the meaningful use requirements will expedite the adoption of patient web portals. Researchers suggest that the outcomes of patient empowerment using patient portals are derived from patient insight into information and intervention of providers to use this mechanism for patient engagement (Otte-Trojel et al. 2015; Otte-Trojel et al. 2014). Interventions through patient portals will facilitate the instructions to reach to the patients irrespective of his or her visit schedule to the provider. In this regard, alerts to patients through the portals serve as a communication element that has the potential to work as a successful intervention (Neuner et al. 2015; Weingart et al. 2006).
Irrespective of the value potential of patient portals, empirical evidence on the impact of portals on patient decision making and any efficiency or effectiveness of the practice is limited. In a review, Goldzweig et al. (2013) note that there is insufficient evidence about the effect of patient portals on health outcomes, with only a few studies evaluating the
11


outcome of care. Specific to the impact of alerts in a portal on health outcomes, a few studies have used clinical trials to investigate the impact of physicians alert systems on patient outcomes, and has shown how clinical information systems with extra capabilities yield to more benefits. For example, multifunctional systems contributed toward increasing delivery of preventive care according to healthcare guidelines (Wu et al. 2006), and Ross et al. (2004) found an improvement trend in guidelines adherence of an intervention group that used an information reminder system over a control group. Specific to diabetes, a few studies show that there is the positive impact of patient portals on managing diabetes, such as patient-provider communication, overall satisfaction, health management or patient outcomes (see review by Osborn et al. 2010). Overall, empirical evidence on how alerts in patient portals influence patient decision remains a gap in the existing literature that the current study tries to address using population-wide disease based sampled datasets.
Theoretical Framework
Alerts contain a textual description of the prescribed tests with three parts. Figure 11 to Figure 14 in Appendix-A show the alert pages as seen by a patient for several single and multiple alerts. In the alert views, the first column provides information about the action plan on the part of the patient in regards to the test. The second column provides information about what exactly the patient should do to conduct the test. The third column provides information on why the test is important.
Information Cues and Decision-Making
We posit the number of information bits in an alert work as informational cues for patients. The action plan mentioned in the first column of the alert page view contain the description such as your doctor recommends to conduct an A1C test, or you are due for a
12


flu vaccine test. This type of information makes a patient infer a state of disease affliction, i.e., the patient may interpret that he or she is sick, or his or her health condition needs attention and doctor intervention (McCoy et al. 2012). A state of disease affliction or indication of affliction is an unpleasant state-of-mind of the patient due to her health condition which induces emotional distress as well as physical pain or distress (Bowman et al. 2006). After the identification of the disease affliction, the patient would be thinking about the test and its future implications, i.e., what happens if the test becomes positive or negative, or the patient is diagnosed with an advanced stage of the disease.
The second part in the alert recommendation indicates or directs the patient on what exactly the patient should do. For example, this portion uses statements: go to the nearest doctor (with address) with an appointment or go the (specific lab with address) to conduct the test. This information provides the cue in regards to what need to be done, and the logistical issues or challenges associated with the action to conduct the test. Further, it makes the patient think the logistics and scheduling plans involved with the test. The directions lead the patient to weigh on a number of factors that are relevant to the test-taking process itself, such as affordability of the test, his or her availability to spend time for the test, the accessibility of a testing lab near to him, and the flexibility of his time to go and really do the test (Adams et al. 2013; Rothman 2000). Often some tests need a substantial preparation regarding fasting or preparing for some hours before the test. The patient may consider whether the tests are affordable. Purely economic factors such as insurance premium costs or patient co-pays or deductibles, although may not be too large, may have a bearing on the patients readiness to go for the treatment. In other words, the interpretation of the cues provided through the logistical direction and information may lead to a set of
13


rationale stimuli on the patients mind, related to affordability, availability, and accessibility of the tests.
The third part contains information on why the test is important, with some statements such as flu vaccine it important for or A1C is important for ... Such information provides clarity to the disease regime and the role of the test concerning the disease treatment (Han et al. 2006). At the same time, the interpretation of these cues may influence the patients behaviors. This potential influence is beneficial to the disease management process, such as how to deal with medication issues or treatment regime in the post-test situations (Viswanathan et al. 2012; Weingarten et al. 2002), or subsequent health-related behavior adjustments or lifestyle changes that may be needed to manage the diagnosis outcome (Koenigsberg et al. 2004).
Thus, alert clicks provide patients a set of cues on which test needs to be completed, why it is important and what logistical process need to be done to conduct the test. The question then remains how these cues influence a patients decision to conduct a test. The decision-making process as a problem-solving activity terminated by a solution deemed to be satisfactory to the decision maker (Kahneman and Tversky 1984). The involved process may be a reasoning or emotional process which can be rational or irrational, and can be based on an information-based approach (e.g., tacit), or a cognitive approach (e.g., explicit), and may include a cost-benefit analysis (Schacter et al. 2010). Some researchers describe a multiple phase process to the information processing and the subsequent decision-making process, arguing that it is a step-wise, time taking cognitive and rational evaluation process. For example, studies in the contexts of acquisition of a cognitive skill (Anderson 1982), or taking decisions to adjusting to a social environment (Crick and Dodge 1994) suggest that
14


information processing and decision making involves steps. These steps include encoding and interpretation of the cues, generation, and evaluation of options, the decision to enact on a possible response and the final phase of actually doing the needed action. Specifically, in the context of a personal problem solving scenario such as managing ones own health, researchers suggest that the information processing and decision-making path is reliant on how the problem is presented, how an individual takes in information, processes that information and plans for solutions to personal problems, and carries out those plans (Heppner and Krauskopf 1987; Heppner and Lee 2009). We extend the prior research to the context informational cues through the alerts and patients decision making to conduct the test.
Anchoring to the prior research on cue based information processing and decisionmaking process (Hilligoss and Rieh 2008; Kahneman and Tversky 1984; Sillence et al.
2007), we propose a conceptual model (see Figure 1) for this study. The conceptual model suggests that clicks on alerts and number of comorbid alerts have direct effects on test closure time. We argue that with each click, the informational cues intensify the momentum to take a decision, thereby decreasing the time to conduct the test. In addition, we argue that in the presence of comorbidity, the concern for the patients health condition increases, due to the details and more bits of information associated with multiple alerts. This concerning effect further reduces the time to conduct the test. Also, the model posits that comorbid conditions have a complementary role to decrease the effect of clicks on the test closure time. We contextualize and elaborate on these arguments further do draw testable hypotheses.
The theoretical framing of this study relies on the early work on process theory of decision making that proposes that decisions are outcomes of mental operations occurring
15


between the presentation of the decision problem and ultimate choice. The decision is dependent on the individuals pre-existing knowledge, experience, and perspective (Crozier and Ranyard 1997; Thomas 1999; Wagenaar et al. 1988). Applying to the context of health decisions, the process of making a decision involves tradeoffs between the quality of life and length of life, or, treatment with side effects and no treatment, depending on the individual preferences (Fraenkel 2013; Kassirer 1994; Pieterse et al. 2013).
The process views of decision-making highlight three points. First, decision making is a time-extended process inclusive of a number of stages. Second, during the process, the decision makers understanding of the problem changes with the time to aid the decision maker to reach decisions (Brownstein, 2003). This mental restructuring depends on the evaluation and re-evaluation of the problem attributes (Brounstein, Ostrove, & Mills, 1979; Brownstein, 2003; Mann, Janis, & Chaplin, 1969). Third, a decision maker follows compensatory or non-compensatory decision strategies (Pieterse et al. 2013). While compensatory decision strategy weighs information cues so that positive attributes can counterbalance negative attributes, the non-compensatory decision strategy depends on the threshold value of options than information on the problem (Pieterse et al. 2013). For example, a patient may choose surgery as a decision for a disease (e.g., cancer, ovarian cysts) as a compensatory strategy to get well, although it involves side effects or further complications. A non-compensatory strategy would be to monitor actively (or, surveillance) the disease progression that does not get rid of the disease but results in no side effects or complications with leading life.
We apply the three points of the decision-making process, i.e., the time-extended stages, mental restructuring, and compensatory or non-compensatory decision strategy to the
16


context of alerts. We argue that the portal based alert context influences the three points through three different factors: (1) time spent on the portal providing a time-extended understanding of the disease or state of health, (2) mental representation and restructuring of the disease attributes due to the information cues contained in the alerts, and (3) compensatory or non-compensatory decision strategy based on the comorbidity states. We draw our hypotheses mainly based on these three parameters associated with the process of decision making.
ELM Central and Peripheral Routes
Developed by Richard E. Petty and John T. Cacioppo in the 1980s the Elaboration Likelihood Model (ELM) works to explain a typical persuasion situation. A situation where an individual reads, listens, or views a message that presents an argument from another party in a certain setting (Petty et al. 2015; Petty et al. 1983). The ELM posits different routes to process an individuals motivation and its impact on changing attitudes and behavior (Petty
2013). ELM focuses on attitudes which guide decisions.
The ELM proposes two routes to persuasion: the central route and the peripheral route (Andrews and Shimp 1990). If the recipient follows the central route, persuasion will result from the recipients logical and conscious thinking of the information presented to him or her. Recipients who are highly motivated will be more likely to follow the central route. Motivation can come from recipients who have a high level of attention and comprehension of the information carried in the message presented to them (Petty et al. 1983). Fear of consequences can also be a reason to increase the attention of the recipients. Strong fear can lead to the negative reaction so a solution should be offered with the message to motivate positive decision making (McNeill and Stoltenberg 1989). The central route is very useful for
17


long-term attitude change rather than abrupt short-term impact (Petty 2013; Petty et al.
2002).
On the contrary, under the peripheral route, persuasion does not come from the main persuasive message but instead comes from secondary cues. The cues received by the recipients under the peripheral route are less related to the main message or arguments (Petty et al. 1983). The change results from the peripheral route are short-term like a simple compliance (Petty 2013; Petty et al. 2002). In a 1984 study by Petty and Cacioppo, they demonstrated the difference between the central route and peripheral route by using the number of arguments as a peripheral cue. The researchers found that a single message with a quality argument was enough to influence the highly involved respondents. While the exposure to multiple arguments made respondents with low involvement to develop an attitude change.
Based on Petty and Cacioppo (1984), when a patient is motivated and involved in his or her health condition, a focal alert will influence him or her to make a cognitive response in decision making regarding his or her condition. In some cases, when the patient is not highly motivated and involved regarding his or her condition he or she may require in addition to a focal alert secondary cues to be persuaded to change their attitude and behavior (Susan et al. 1998).
You can notice from the graph below, Figure 15, how the medical alerts page shows multiple alerts with some arguments. In our conceptual model we hypothesize that patients who are presented with multiple peripheral alerts will develop a greater change due to the existence of secondary cues, which aligns with the ELM theoretical framework.
18


Hypotheses Development
Alerts provide information cues concerning a possibility that patient will be afflicted with a disease, or may be already afflicted which needs to be maintained (McCoy et al.
2012). We argue that the informational cues contained in the alerts make a patient to access and digest more knowledge about the disease. Prior research notes that the benefits of an ambiance or cue to enhance a patients internal or intrinsic motivation to proceed towards the process of diagnosis and treatment for a disease partly due to the high involvement (McAllister et al. 2012). In a scenario where the alerts are not sent, the patient does not engage with such information based involvement process. On the contrary, with the alerts, the patient unravels the knowledge or issues involved with the disease. As a result, the patient understands why and what the test involves, and what may be the outcome of missing the test.
Every time a patient clicks on the alert page, the reinforcement of the test for the disease diagnosis becomes higher. The patient undergoes a series of encoding and interpretation process while he or she is trying to understand the alerts cause and effects, future consequences regarding the diseases progression and treatment options (Leveille et al. 2009). In other words, clicks help a time-extended understanding of the disease or state of health, as well as restructuring the mental representation of the disease attributes due to the information cues contained in the alerts.
According to the ELM, there are two routes to persuasion (Petty and Cacioppo 2012; Petty et al. 1983). The first route is known as the central processing, in this route, persuasion is based on considering thoughtful arguments and information. In the central processing, the receiver is more active in the persuasion process. The central processing route needs the
19


receiver to be motivated about the message her or she receives. If the receiver is not motivated by the message he receives, he will lack the motivation to do central processing (Petty and Cacioppo 1979).
The second route is known as the peripheral processing route. In the peripheral route, the receiver gets persuaded by a message not based on its focal argument but based on other cues. For example, a receiver may get persuaded by a message because it has many arguments but he or she lacks the motivation to think about each argument individually (Petty and Cacioppo 1979). The receiver in peripheral processing lacks the motivation and more passive. He or she will use the peripheral cues like many arguments in one message as a short-cut to make a decision (Petty and Cacioppo 2012).
Existing studies note that in the presence of informational cues and high involvement, consumers take a positive decision (Petty 2013; Petty et al. 1983), such as involvement in a product testing or advertisement campaign leads to purchasing a product (Michaelidou and Dibb 2006; Michaelidou and Dibb 2008). Similarly, in the health context, high involvement with the disease management process creates a shared responsibility in patients own health care (Wilson et al. 2014), and leads then to manage and derive care from the patient (Kamis et al. 2014; Sherer 2014). As much as each click leads a patient to feel the involvement with his or her health or disease situation, he or she will perceive that conducting the test will lead him to a sense of control over the management of health condition; and will lead for a faster closure time.
Thus, we argue that in the case of self-involvement and high motivation, each view of the alert page increases the information the patient is exposed to. That will yield to an increase in the level of elaboration of the persuasive arguments displayed in the alert
20


message. Such, can reinforce the patient attitude towards conducting the needed test. As much as high involvement results in a perception of empowerment to manage the own disease, the alert message will increase the motivation to conduct the test. High involvement and subsequent information will help a patient to overcome the situational and economic barriers to form a positive attitude to conduct the required test as soon as possible. Based on these arguments, we hypothesize:
Hypothesis HI: Number of alert clicks is negatively associated with test closure
time.
We argue for a quick decision for a patient to conduct the test when comorbid alerts are present. Comorbidity a distinct additional disease entity that has existed or may occur during the clinical course of a patient who has the index disease under study or simply put, the co-occurrence of multiple diseases in one person (Guralnik 1996; Van den Akker et al. 1998). Although measurement of comorbidity differs, a commonly accepted norm is to follow the evidence base around disease identification (such as international classification of disease codes) clusters to provide a count of comorbid diseases or symptoms (de Groot et al. 2003). We follow similar guidance to classify the comorbid alerts. For example, if a doctor is suspicious of any cardiovascular disease along with diabetes for a patient, then the doctor will order a complete blood, cholesterol and lipid profile test for the patient on the alert page. This alert indicates the patient has three comorbid alerts along with the focal diabetes alert.
The perception of comorbid alerts highly concerns to a patient than a single alert because it reflects that the patients health condition is not good, and he or she has multiple diseases. The state of disease affliction with two or more diseases would be a highly distressing situation for the patient (Bowman et al. 2006), and he or she will have a high
21


motivation to get clarity about the situation. The process views of decision making highlight the importance of mental restructuring and subsequent following up of compensatory or noncompensatory decision strategies by a patient. The stages of comorbidity or a set of comorbid alerts emphasize on a higher level of attributes for a patient (Valderas et al. 2009). He or she infers that the disease condition is at a serious or more aggravated stage and the patient needs to take some action. This changes the mental structuring associated with the disease condition from a casual or not-so-important one to a serious one. As a result, the evaluation and re-evaluation of the subsequent action plan to conduct the test becomes a central issue than a very trivial one. Research shows that with a high emotional situation in regards to health conditions, patients would seek a clear and reliable information (Anderson and Agarwal 2011). Because the tests will provide clarity to the patient on his or her multiple disease situation, irrespective of the relative compensatory or non-compensatory outcome of the test, the patient would want to collect more information. At least the test information will make him or her laden with information to take subsequent evaluations.
In addition to the seriousness associated with comorbid states, each view of the alerts page will expose the patient to more cues about the comorbid state. While the levels of interpretation regarding the test and disease-related changes with each view. Each view of the alerts page would expose the patient to more cues. For patients who are not highly involved, motivated, and expressed low elaboration of the focal alert, the existence of additional cues from comorbid alerts will be utilized to processes the information in a peripheral route. Therefore, we argue that the existence of comorbid alerts in the alerts page will generate more cues that will lead to a peripheral routes processing. Patients will seek more clarity to his or her condition, and furthermore, each view of the alert page will have a positive
22


reinforcement to persuade the patient to conduct the tests faster. Based on these arguments, we hypothesize:
Hypothesis H2: Number of comorbid alerts is negatively associated with test closure
time.
The route that the patient takes when seeing the alert message is important. In the central processing route, the persuasion is based on the argument provided in the alert and will have a higher impact on the long term in changing the patients behavior. Where in the peripheral processing the receiver in not highly motivated by the focal argument and he or she is persuaded by peripheral cues. Such type of persuasion is short-term in nature (Hawkins et al. 2008; Lustria et al. 2013). The providers need the focal alerts to include arguments that influence patients to be more engaged and influence them to follow the central processing route to make decisions regarding their conditions. This will make the impact of the focal alerts last longer and will have a higher influence on the patients long-term behavior (Noar et al. 2011; Ownby et al. 2012). However, even if messages include good arguments receivers many not be motivated to engage. Providers need also use additional cues to help to persuade the receiver using peripheral processing.
Calder et al. (1974) found that subjects who get exposed to a larger number of messages and arguments produce more favorable responses. The researchers found that greater number of arguments influenced the receivers cognitive responses. In addition to that multiple studies have found that messages that contain more arguments can create more chance in the receivers attitude (Calder et al. 1974; Chaiken 1980; Petty et al. 1983). The multiple messages or the message with multiple arguments will influence the receiver to peripheral processing. So even if the receiver is not engaged with the focal argument, the
23


multiple cues will manage to persuade him or her to for a desirable outcome. Thus, we hypothesize:
Hypothesis H3: Number of comorbid alerts complement number of clicks to reduce
the test closure time.
So far, we argued that the number of clicks and comorbid alerts influence quicker test times. Further, comorbid alerts complement the number of clicks to reduce the test closure time further. The underlying assumption under these arguments is that the probability to conduct the test is a somewhat a fixed event rather than a time-variant event. In other words, the likelihood of conducting a test on a particular day is fixed. While this assumption leads to explore the effect of clicks and comorbid stage on test closure, but it leaves an important aspect of patient behavior towards the decision to conduct the test. For example, a patient may change his or her perception to conduct a test as time passes on, with a collection of more information. When a doctor prescribes a test, a patient may view it unnecessary or not urgent. However, when the patient seeks family or friends opinions on the prescribed test, he or she may consider the test to be an important one to be conducted as soon as possible. Such external factors will motivate the patient to do the test with passage time. Alternatively, patients collection of information about the test will increase the probability of conducting the test with time.
In the context of alerts, the time variable captures such information effect that in reality may not be a linear projection. As time increases, early motivators (e.g., patients who are convinced to do the test early), will conduct the tests early. Some patients may be converted from late responders to early responders, and some may get an opposite information to delay the test forever or not do the test. In other words, we expect that as time
24


progresses, the probability of patients to conduct the test will increase, but until a threshold point, beyond which it will decrease. We explore the time variant effect on test closure time using survival analysis. In addition, such an exploration may provide a threshold window where the test closure probability is high than other times, and intervention at that window may lead to conduct the test earlier with a compounding effect.
Method
The methodology in this study follows two steps. In the first step, we test the hypotheses using a linear regression method. In the second step, we conduct survival analysis to explore the time variant effect of test closure time. Using the two steps analyses allows us to get over the limitations of the linear regression and take advantage of the survival analysis strength. In this section, we describe our data collection process, the variables used in this study, estimation models and approach.
Data and Variables
We used an archival data for diabetes and CKD patients. We treated these two different set of patients as two samples. The data was taken from diabetes and CKD registries for the Colorado region, captured and maintained by a providers electronic medical and patient portal management systems. The patients in our samples had at least one visit to a doctor from Jan 2014 to Feb 2014 to one out of many facilities of the provider. The patient data was already de-identified before being collected in the registry.
We used a structured data collection and sampling process as shown in Figure 8 and Figure 9. For the diabetes sample, we considered all the enrolled patients in the diabetes registry (e.g., suspected diagnosis of diabetes or pre-diabetes). The registry had 166,107 patients, out of which 152,219 patients were not given any lab test recommendations or lab
25


test alerts. That left us with 13,888 patients with some or other lab test alerts in the system. We had to exclude 54 patients who did not have complete records or had some missing data. Out of the remaining 13,834 patients, 11,670 were not given any diabetes-related alerts, and 2,164 patients were given diabetes-related alerts, including the focal or most prevalent A1C alert. These 2,164 patients form the sample for this study.
For the CKD sample (see the data sampling process flow chart in Figure 9), we considered all 51,343 patients who are suspected to have CKD, out of which only 11,691 patients were given any lab test alerts in the system. We had to exclude 17 patients who did not have complete records or had some missing data. After excluding 3,201 patients without any alerts in the sampling period, we had 8,473 CKD patients with 5349 patients closing the alerts within the stipulated time, and 3,124 patients not closing the alerts or doing the tests by the due date.
The dependent variable is the Test Closure Time (TCT) a continuous variable that measures the number of days that took a patient to conduct the focal A1C test or GFR test as applicable, from the day it was posted on the patient portal. Any doctor, nurse or lab technician in the providers network can access the system and place a flag that the test is done, with uploading of the results of the next steps. Once this is done, the system closes the alert, with a date and time imprint in the system. We took this date and time stamp as the alert or test closure day to calculate the Test Closure Time (TCT).
Amongst the independent variables, the Total Alert Clicks (CLICKS) variable captures the number of times the patient clicked and viewed (similar to the page views) the A1C alert or GFR alert in the patient portal before closure. This was also calculated through the digital health records system of the provider, using the time stamp for each click by the
26


patient that the patient used to view the alert. This provider is highly motivated to assess the impact of the portal to aid in further research and development, and the capturing of the time stamp of clicks is part of the initiative to assess the impact of the patient portal.
The variable Comorbid Alerts (COMBD) is a count of the total number of comorbid alerts given to the patient. The provider uses the comorbidity conditions as a practice based mechanism to track patients. Thus, the comorbid diseases for diabetes and CKD used in this study come from the evidence-based practice in health care. We provide the description of the comorbid diseases and related lists of tests for diabetes and CKD in Appendix A.
We include several controls in the models to account for the factors that may have influence for the time taken to close the alert. We included a variable Contiguous Alerts (CONT) with the count of the number of alerts for the same disease, i.e., diabetes or CKD.
We control the patients severity by controlling for their LACE index that accounts for the length of stay, acuity at the time of admission, comorbid diseases and emergency department visits for the past six months. This variable is used in existing practice and research literature to account for the patient severity (Au et al. 2012; van Walraven et al. 2012). We also added a variable indicating the maximum acuity stage of the patient during the last visit or stay for any disease in a hospital or clinic. We controlled for the age and gender of the patient. We included dummy variables as controls, indicating whether the patient is Medicare or Medicaid patient. We used the Ethno-Nationality (ETHN) variable (see Appendix C) for the groups in our sample. We used these groups to adjust for possible within group variations across groups by estimating and reporting standard errors adjusted for the clustering effect. We did not have income data for the patients, but to account for any income effect due to the geographical
27


location of the patient, we included the zip code of the patient as a control variable. We also included insurance dummies based on the patients insurance.
Table 1 provides the descriptive statistics of the variables. In the sample of 2,164 diabetes patients, 22 percent of patients had any of the Comorbid Alerts. In the diabetes sample, The Test Closure Time (TCT) vary from a day to 88 days, for which the CLICKS vary between 3 to 60 times. There are 52% male patients, with ages spanning 19 to 87 years; the average age in the sample was 65 years. Only 1% of the patients in the sample are Medicaid patients, while 47% belong to Medicare group. In the sample of 8,473 CKD patients, 67 percent of patients were not recommended any of the Comorbid Alerts. The Focal Alert Closure Time varied from 1 a day to 105 days, for which the CLICKS vary between 1 to 71 times. There are 51% male patients, with ages spanning 19 to 85 years; the average age in the sample was 60 years. Only 2% of the patients in the sample are Medicaid patients, while 49% belong to Medicare group. The majority (>85%) of the patients in both the samples have Health Maintenance insurance plans.
Table 2 provides correlation amongst variables for diabetes and CKD samples respectively. We observe that there is a high correlation between the Medicare and Age of the patients, indicating the Medicare patients are mostly elderly patients. Further, the variables Contiguous Alerts and Comorbid Alerts have correlated at above than 0.4 levels, indicating the presence of multiple tests for both focal and other diseases for patients. We compared the difference between both the patients who closed the alerts and those who did not in both the Diabetes and the CKD samples. The T-tests indicate no significant differences across the samples (see the results in Table 3).
28


Estimation Models and Specifications
The Test Closure Time (TCT) is a continuous variable. Therefore, we use ordinary least squares (OLS) estimation in our models. The main OLS specification is as follows:
Yi = p Xi+ s (1)
Where Yi is the dependent variable, Xi is a set of explanatory variables ft is a vector of parameters, and s are disturbances associated with each observation. Because we have both direct and interaction effects, we specified the following two equations:
TCT = fi10 + pn (CLICKS) + fi12 (COMBD) +filcv Controln +s} (2)
TCT = fi20 + P21 (CLICKS) + p22 (COMBD) + fi23 (CLICKS x COMBD) +fi2cv Controln +e2 (3)
We use survival analysis to investigate the change in the probability of test closure with time. The dependent variable in our survival analysis is a combination of time and event or censor. The time variable reflects on the duration until the patient went and conducted a diagnostic test. The event variable is when a patient conducted the test. We are interested in exit probability of the patient due to the event with time, i.e., calculating the hazard rate of conducting the test. Our dependent variable is considered to have a continuous probability distribution, where t is the time it takes a patient to take an action and conduct the focal test. For this analysis, we used the generic survival function of the form:
5(0 = 1 F(0 = Pr(T > 0 (4)
Where, 5(0 is the survivorship function reporting the probability of surviving beyond time, and which means the probability that there is no failure event prior to t. T is a random nonnegative variable denoting the time to the studied event. The survival function is equal to one at t equals to zero and decreases towards zero as t goes to infinity.
29


The hazard rate in our study is defined as the probability that the test closure will happen at time t. The hazard function, also known as the conditional failure rate, h(t) is the instantaneous rate of failure:
hit) = lim
At->0
Pr(t+At>T>t|T>t) /(t) At _ S(t)
(5)
Where, h(t) is the limiting probability that the failure event (closing the test) occurs in a given interval, and /(t) is the density function that is obtained from the first derivative of F(t) from the equation above, which equals to the negative first derivative of the Survival analysis.
m=d-7T=Tfi-m} <
The hazard rate range can vary from zero to infinity, which is interpreted not to risk at all (zero) to certain event (certainty of failure). The hazard rate can increase decrease or remain constant with time. The cumulative hazard function is derived from the probability density function and the survivor function:
-O" <7>
The cumulative hazard H(t)measure the total amount of risk that has been accumulated up to time t.
We also divide our sample into two distinct samples; the first contains patients that didnt have comorbid alerts shown on the web, and the second sample contains that patient that had comorbid alerts displayed on the web portal. We investigate the difference and compare the impact of the comorbid alerts on the hazard rate on both samples.
We extend our analysis further do draw more insights. The survival process is a stochastic process and the time index is one day, the process in our context governs whether
30


an alert disappears or closed in the patient portal. In a hazard context, we assess the impact of an explanatory variable on the hazard of exiting the patient portal rather than on the length of survival time. A proportional hazard model, on the other hand, would approximate the probability of a patient conducting a diagnostic test at a certain number of days. Although Cox proportional hazard measurement is non-parametric, a Weibull proportional hazard measurement uses a parametric form to estimate the hazard. We focus first on a possible change in alert survival due to viewing the alert on the patient portal. The model to be estimated is the proportional hazard model:
h(t) = h0(t) exp(X1tfH+ X2f2H + + Xp/3(7)
Where XL s are the set of explanatory variables which shift the hazard function proportionally, (3[H s are the parameters to be estimated, and h0(t) is the baseline hazard. In the Cox specification of the model above there is no assumption is regarding the distribution of h(t). In the second phase of the analysis, we examine the impact of comorbid alerts on a focal alert survival. We observe the number of comorbid alerts displayed with each focal alert during the observed time. We use this data to assess how the number of comorbid alerts can affect a focal alert survival. However, the change in survival may not be linear and may be moderated by the number of comorbid alerts. To be able to accommodate such possibilities, we interact the number comorbid alerts with the number of views.
Results
Results of Regression Analysis
Table 4 reports results of our estimation models. Columns 1 and 2 present the direct effect and interaction effects for the diabetes sample, and Columns 3 and 4 that of the CKD sample. The variable CLICKS has a negative and statistically significant association with
31


Test Closure Time for both diabetes patients sample (see Column 1, Table 4, ft = -0.090, p < 0.01) and CKD sample (see Column 1, Table 4,f} = -0.079, p < 0.01). These results support hypothesis HI. Further, the results translate to the findings that for diabetes patients, a patient will close the focal A1C alert a day earlier with approximately 11 clicks. Whereas, in the case of CKD patients, a patient will close the focal GFR alert earlier with approximately 13 clicks.
We find that Number of Comorbid Alerts (COMBD) has a negative and statistically significant association with Test Closure Time for both diabetes patients (see Column 1, Table 4, /? = -1.484, p < 0.05), and in the CKD sample (see Column 3, Table 4, /? = -1.189, p < 0.05). These results indicate that higher number of comorbid alerts results in the patient doing the test earlier. This result supports our hypothesis H2. We interpret these findings as that the diabetes patients close the A1C alert on average of 1.5 days earlier than in the presence of any additional comorbid alert, where CKD patients close the GFR alert on average of 1 day earlier in the presence of comorbid alerts.
In the interaction effects model, we find that the interaction term CLICKS x COMBD is negative and significant in diabetes sample (Column 2, Table 4, /? = -0.321, p < 0.01), as well as in CKD sample (Column 4, Table 4, /? = -0.191, p < 0.05).
These results provide support for H3 that higher number of comorbid alerts complement the number of clicks to decrease the test closure time further. We interpret these results as that the diabetes patients close the alerts a day earlier with approximately three clicks in the presence of one additional comorbid alert, which is otherwise higher (e.g., 11 clicks). Similarly, CKD patients close the alerts a day earlier with approximately five clicks in the presence of one additional comorbid alert. We plotted the interaction effect results and
32


presented it in Figure 2 and Figure 3. The difference in the negative slope of the lines showing patients with high comorbid alerts compared to low comorbid alerts is clearly discemable in the interaction plot.
Among other results in the estimation, we did not find any significant association of control variables except Medicaid. In the interaction effects model, Medicaid has a significant and positive effect on the Test Closure Time indicating that Medicaid patients may take a longer time to close the alerts.
We conducted several tests to assess the robustness of our findings. First, we introduced an interaction term consisting of the Contiguous Alerts and Clicks in our models to check if patients display similar concern for some alerts with the same disease. This interaction term was not significant, indicating no interaction effect of contiguous alerts on clicks. Second, we checked for multicollinearity by computing variance inflation factors (VIF) after regressions, which were less than 4 in all the models. These indicate that multicollinearity is not a serious concern in our analyses. Third, White (1980) tests for heteroscedasticity did not suggest the presence of heteroscedasticity. Nevertheless, as suggested (Greene 2003), we estimated robust standard errors in all our estimations, and our results remain unchanged whether we use robust or non-robust standard errors. Finally, the standard errors in the estimation were clustered by the ethnonationalism of the patient to adjust for possible within group variations, and we report standard errors adjusted for the clustering effect.
Results of Survival Analysis
The survival analysis results are presented in Table 6. We find that for both Diabetes and CKD, the Weibull and Cox proportional hazard models provide similar results with the
33


sign and significance levels being same for each focal variable. Much of the difference in the coefficient magnitudes comes from the variance differences.
First, in Column 1 of Table 6, the Weibull regression shows that the coefficient of CLICKS is positive and significant (Column 1, Table 6, /? = 1.247, p < 0.01). This finding shows that an increase in CLICKS will increase the hazards rate by 1.25 times. In other words, for diabetes sample, one additional click by a patient will increase the A1C test closure probability by 1.25 times in a day. Similarly, in the CKD sample (Column 3, Table 6, /? = 1.101, p < 0.01) one additional click will increase the GFR test closure probability by 1.14 times.
The results also show the number of comorbid alerts have positive and significant effect (See Columns 1 and 3, for diabetes sample ft = 1.152, p < 0.05, and for CKD sample,
P = 1.141, p < 0.05). In addition, we also find the CLICKS and COMBD interaction term is positive and significant (for diabetes sample ft = 1.332, p < 0.05, and for CKD sample, ft = 1.060, p < 0.05). We find similar results for the Cox proportional hazard models, i.e., CLICKS, COMBD and the interaction term CLICKS x COMBD are positive and significant for both diabetes and CKD samples (see Column 3 and 7 of Table 6).
We present the marginal effects of the Cox PH models in Columns 4, and 8 of Table 6 for interpretation purposes. At the mean values of all variables, a ten percentage increase in the clicks is predicted to reduce the probability of closure by 1.24 percentage (see Column 4 of Table 6), whereas ten percentage increase in the comorbid alerts is predicted to reduce the closure by 0.11 percentage. As a combined effect, ten percentage increase in both clicks and comorbid alerts is predicted to reduce TCT probability by 1.5 percentage for diabetes patients. In the case of CKD patients, a ten percentage increase in the clicks is predicted to
34


reduce the closure by 0.91 percentage, where ten percentage increase in the comorbid alerts is predicted to reduce the test closure by 0.09 percentage. Interpreting the complementary effect, ten percent increase in both clicks and comorbid alerts is predicted to reduce the closure by 1.15 Percentage.
We also present the survival analysis results using two main plots (Figures 4 and 5, Figures 6 and 7) and the hazard rate regression results in Table 6. As shown in Figure 4 and Figure 5, for the diabetes patients the hazard rate shows the probability of having the event (conducting A1C test) is going up for the 1st period to almost 1.4% then dropping after 60 days to reach about 0.03% after 80 days. So the chance of a patient to go and do the test increases for 60 days, then after that the chance of conducting the diagnostic test decreases with time. For CKD patients we observe an almost similar thing, the hazard rate shows the probability of having the event (conducting GFR test) is going up for the first period of time to about 1.2% then dropping after 70 days to reach as low as 0.08% after 100 days.
We also found that there is a somewhat a varying threshold of probability for the diabetes patients during the 40th to the 60th days, and for CKD patients, it is from the 60th to the 80th dayswhich we interpret as the ideal time for an intervention. The patients motivation for closing the alert (A 1C in the case of diabetes and GFR in the case of CKD patients) is at the highest level in these periods, with some minor variations. If another reminder through a phone call, text, email or any other medium is given to the patient, then that will increase the likelihood of closing the test. Such will increase the efficiency of intervention and make it more effective.
To aid in the interpretation of the comparison between the patients with comorbid and non-comorbid conditions, we plotted the Kaplan-Meier estimator of the survival function we
35


take the ratio of those without events over those at risk and multiply that over time, as the following equation:
Where nj is the number of observations at risk and dj is the number of events.
Figure 6 and Figure 7 show the patients with comorbid alerts (red line) vs. patients with no comorbid alerts (blue line). From the graph, we can see that patients with comorbid alerts will be associated with a lower survival rate at any point in time. This means that if someone has comorbid alerts, he/she is more likely to go and conduct the A1C test in less time comparing to someone with no comorbid alerts. The Chi-square test between the two groups (comorbid alerts vs. no-comorbid alerts) for the comparison between survival functions shows a significant difference with the Chi Square = 9.73 and significant at p=0.002 level (see results in Table C3, Appendix C). In addition to these differences, the Kaplan-Meier survival estimate diagram also shows that the survival curve starts from 1.00 at time period 0 and keep decreasing with each period of time (each day). After 90 days the curve approaches the probability of almost 40%, reflecting that the survival probabilities go down to 40% over 90 days. This means that 40% of the individual still would not have done that A1C after 90 days. This finding has practical implications for providers. They can infer from these results that the alerts should be renewed and fresh reminders should be given within 90-day period to reach to the 50% population who would not come to conduct the test within this interval.
Discussion
Findings
36


The goal of this study was to explore how alerts through patient portals influence a patients timeliness in conducting the necessary tests. Specifically, we investigated the complementary effects of a number of clicks and number comorbid alerts recommended through a patient portal, at the time of closure of a test. We find that the number of clicks is associated with an early closure of the focal test for diabetes patients. For diabetes patients, a patient will close the (A1C) alert a day earlier with approximately 11 clicks while in the case of CKD a patient will close the (GFR) alert earlier with approximately 13 clicks. This finding suggests that alerts in patient portals serve as effective reminders for timely conduct of tests. The second set of findings of this study is that patients take less time to close the test alerts in the presence of comorbid alerts, the diabetes patients close the A1C alert on average of 1.5 days earlier than in the absence of Comorbid Alerts, where CKD patients close the GFR alert on average of 1 day earlier. Third, in terms of the complementary effects, we found that diabetes patients close the alerts a day earlier with approximately 3 clicks in the presence of one additional comorbid alert, which is otherwise higher (e.g., 11 clicks), and CKD patients close the alerts a day earlier with approximately 5 clicks in the presence of one additional comorbid alert.
The survival analysis results show that in presence comorbid alerts, ten percentage increase in both clicks and comorbid alerts is predicted to reduce TCT probability by 1.5 percentage for diabetes patients; whereas similar increase may reduce the closure by 1.15 Percentage. Further, the survival plots show that a high probability threshold span exists between 40 to 60 days for diabetes, and between 60 to 80 days for CKD alerts. Furthermore, the difference in the threshold spans may be earlier and lesser for patients with comorbid
37


alerts than a patient without comorbid alerts, indicating the for comorbid patient the reminder should be earlier.
Implications
The implications of this research are three-fold. First, this study extends the existing practice discussion and motivation surrounding the impact of patient portals on patient care. The finding that each additional click on the alerts is associated with an early closure of the alerts has practical implications. It suggests that patient portals act as a reminder system for the patients. Specifically, for patients who have an affinity to get engaged with their own health management, and seek to be empowered, the portals are highly effective in reducing care gaps and timely interventions. The findings of this study should motivate practitioners to enroll as many patients as possible in their portals and motivate them to use the portals to deliver timely diagnosis and care.
Second, the study has implications towards the design of patient alert systems in a portal. While the practice of recommending multiple screening and diagnostic tests has been undergoing a debate in the current United States healthcare system due to cost implications (WSJ 2013), the findings of this study suggest that multiple test recommendation may be highly influential for timely intervention to curb diseases. Specifically, for diabetes and CKD, the symptoms at an early stage of the disease progression is not painful, and therefore, patients do not take the diagnosis seriously nor turn out for required tests. However, when diabetes or CKD test alerts are recommended along with other more serious disease symptoms and alerts, such as heart disease or epilepsy, perhaps patients turn in for more serious tests, and simultaneously conduct the required tests. In other words, even in the
38


presence of cost concerns, multiple comorbid tests may be beneficial to patients and providers in saving costs and providing effective care in the long run.
A third implication of the study is that alerts may not be effective as a one-shot recommendation. Renewal of alerts on a timely basis, more so within the high probability threshold span when the patients likelihood of considering the test is high. This study suggests that A1C alert repetition within 40 to 60 days of the first alert and GFR recommendation with 60 to 80 days of a first alert may increase the probability of more patients to conduct the tests.
Finally, the study indirectly suggests that patients weigh both compensatory and noncompensatory considerations for health. In other words, both financial and psychological conditioning of patients before prescribing tests is an important consideration. Patients would not be able to close test alerts unless their economic concerns associated with conducting the tests are manageable. Further, often many patients carry a stigma with the diagnosis of a disease, which poses a psychological barrier to come and do the tests. In the presence of these concerns, how a provider can motivate the patients to take tests and possibly curb any future, potentially serious, implications remains a relatively untouched topic of discussion in the current practice of healthcare in the United States. While this study highlights the importance of timely and repeated alerts, but plausibly, providers may discuss both financial issues with patients, and make alternative arrangements, such as self-tests, or low-cost tests available for economically disadvantaged patients. Similarly, creating social or peer groups around patient care (e.g., web-enabled social networks, such as patientslikeme.com) for some diseases may remove the psychological stigma associated with disease diagnosis and management concerns.
39


Contributions
This study has three theoretical contributions. First, the study extends the process view of decision making to health care context. Influence of information technology on patients decision making has been an important topic of discussion in information systems areas. Specifically, the time-extended effect of health IT on a patients decision to conduct tests has not been explored in the existing literature that is an important contribution to this study. Second, the study takes the view that information technology effectiveness in health context may have an effect on the seriousness of a disease or health condition. As much a single disease affliction is concerning, but multiple disease conditions make situations worse. In differentiating the role of alerts on patients decision making with and without comorbid conditions, this study highlights that health IT design may be highly consequential for serious patients with multiple diseases, a view that was not explored in existing health IT literature. Finally, in applying the survival analysis in a health IT context, the study explores a span of intervention that is highly effective to manage chronic diseases. The study suggests that by modifying alerts to be repetitive, one may influence some patients with chronic conditions to do a test. This inherently provides the view the health IT effects may not be constant with time, and such time-variant effects may need to be explored more.
Limitations and Suggestions for Future Research
The study has some limitations which future research can address. First, the data were collected only for one provider. Although this helps to enhance internal validity of the study and control of many cross-provider effects, it may limit the generalizability of the study. Future research can extend the analysis by incorporating studies that may include other providers. Similarly, this study is conducted considering two focal diseases that future
40


studies may replicate for other diseases in patient portals to consider generalizability of the findings.
Second, although we theorize the economic and psychological concerns associated with the alert closure decision process, and suggest a model to elaborate on the steps involved in the process, we do not empirically test the theoretical mechanisms underlying the phenomena being studied. Future research can empirically test such phase-mediating mechanisms potentially using alternate methods. Third, due to the cross-sectional nature of the study, similar to many other studies that use cross-sectional designs, the results may be interpreted more associational than establishing any causal linkages. However, the time variant survival analysis may mitigate some of these limitations. Future research can adopt additional empirical strategies that can use longitudinal data and designs for similar studies.
Conclusion
The goal of this study was to explore how alerts in a patient portal are effective. Specifically, we investigated the complementary effects of number of clicks, and the nature of the multiple alerts recommended through a patient portal, at the time of closure of a focal alert. We argued that with each additional click of the alert, a patient would have a higher concern and motivation to conduct the test and close the alert as soon as possible. Furthermore, we suggested with multiple alerts with comorbid conditions, the patient will have a higher concerning effect to lead to an early closure of the alert. We contextualize this study to two samples of diabetes and CKD patients, and empirically examined and found support for the hypothesized effects. In addition, we use survival analysis to explore the time-based effect of alert clicks on test closures and establish threshold spans periods for the alerts effect on closures. We recommend that alerts need to be repeated between 40 to 60
41


days for diabetes, and between 60 to 80 days for CKD. The study provides implications for the effectiveness of patient portals and suggests the repeated alert design aspects as a recommendation to motivate patients to conduct times tests. Further, we contribute to the existing information systems and healthcare research streams of research in exploring the impact of patient portals on reducing patient care gaps and highlight the importance of patient portal and alert systems as emerging patient-centered health information technologies.
42


Figures and Tables
Figure 1: Conceptual Model
Low Number of Comorbid Alerts
----High Number of Comorbid Alerts
Low Number of Alert Clicks High Number of Alert Clicks
Low Number of Comorbid Alerts
High Number of Comorbid Alerts
Low Number of Alert Clicks High Number of Alert Clicks
Figure 2: Interaction Plot Diabetes Sample
Figure 3: Interaction Plot for CKD Sample
Area of intervention
X 40 M *
Alert closure time in days
S
a -
s

st s
w
*
s
Afea of intervention
o eo so too
Alert closure time la days
Figure 4: The Smoothed Hazard Estimation Plot Diabetes Sample
Figure 5: The Smoothed Hazard Estimation Plot CKD Sample
43


Figure 6: The K-M Plot Showing with Comorbid vs. Non-Comorbid Diabetes Patients
Figure 7: The K-M Plot Showing with Comorbid vs. Non-Comorbid CKD Patients
44


Figure 8: Diabetes Sampling Process
45


Figure 9: CKD Sampling Process
46


No
The system keeps the alert posted on the Patient Web Portal
Figure 10: Alert Closure Flow Chart


Figure 11: The Alert Page of Portal Showing No Alerts
Patient name, and doctor's information appears here
Your Action plan oa/vtoi*
Figure 12: An Alert Page Showing a Childs Vaccination Alert
action pia
H*< IK*# will A*
(VX>
III* P*l>*nl to
Figure 13: An Alert Page Showing a Flu Vaccine Alert
48


Patient name, and doctor's information appears her*
Action What should I do next? Why i* this mpoitnnt?
You etc due for you choioctorol Mood last. P ease 90 la: for a hood :raw udj not nted :o fast UMICC your pnvlrter Iwrirud yin to d: co rou d: not need ar e ;po rtment 'xltis test. Aooisste-o test measures the fats in you' 0 tod and taipc accatt the htafth :f you' hear :ioo: vessels A'd rk frr raa-t ana'k O' ertwe D can a so chow if masication y :u are alin; is worksn^.
you are Our for a Idap VMdm. ask for a oilk-iu-tmrr nUattni v sit (weekdays cminq httlvx: ho. r) rr ,ri wngetr-k ~Tvun : cn etycur n-i sc'odjled Yttt IN irtap vamna srMart ajslncr tctjnu:, dpnrhcrla i'd v.hoofing luuyl
Your doctor -eco'D'-endr to (ondixt an A1C tert ?le>ey. IcMi fw a rtoo: dra*> >ou d< no: reed t: 'as: Otic:CO vo.t 'r-yfrtpr rctrurtart yen tn :o s:. You :o no: need :n 5jyjirnie t fit 1) b test. Tier Alt tot i>on.ei#jyu4 ,w t) juv: (bleed t.jsr io/ei over tha psit 2 :c 3 rriiilhj.Yu.i duilu My >jvi level at Icost Md:c a year.
Figure 14: An Alert Page Showing Multiple Alerts along with A1C Test Alert
What should I do next? Why Is this important'
You dexter rrtwrn* to conduct an A1C tart Sloes* y- Wlrli fin a tioo: draw vowdc nat eel e fas: UliUtS **.1 fr*vtrtr ritirwil yru tn U(interne IK* V r* uni. TierAUlenti>ieweaeuf tow dxm (b :od t.pir lemiori Too ere due for vo (VMMtml Mkm) tost. f ease o e la: for a hood :ra tcods not road 'sst UMISC your (un.lih- l>inurr<1 y-vi fa d: to rou d: not nrod ar :oo rtment *> (h test. A00- head and M pc Mtoct nw MaltSi :< iw Mart ;iee: retser X'diH. ftt root Util **# D on 1 sa stew V me:.w* > :u a-e -Jlin; is not tin:.
hood preocieo clr.k It Mswr/Mjin prune CO 101 la :e Pcrmaieete Primary Coe Owe Irseet tv 1 uas n to cost >iOd fw re dr* Nr rwwt tetJeJ. In ioutrerr aid Mortner MjnKunj your 5*co: pnwr r rspetan- tt our heart lea ti
Voa oiu fi fin a TCip vaccine. C* hi find Sofeu Ptmarerte medial cfice am as* fx a MNC-tr* imnunMtkn vlst (weekdays durirc bwsines: hours) vyau can gettn* iftwvnaitioi at oyr rext sOeOuloJ *hU. Tlie Tihy nukw rvkUi oyuticA teta'us. dichtnena and #ho r COUCh.
If ,ou ha-re already had this vaccine outtce of Kaiser fe-manents, phase brtig the immuMEatKr record toyojrnext kjcUuM vhrt.
Multiple peripheral alerts (secondary cues) with multiple arguments
Figure 15: Multiple Peripheral Alerts


Table 1: Descriptive Statistics
Diabetes Sample CKD Sample
Obs. Mean S.D. Min Vlax bs. Vlean S.D. Vlin Vlax
1 TCT 2,164 37.80 22.92 1 88 8,473 55.50 18.21 1 105
2 CLICKS 2,164 14.53 9.59 3 60 8,473 18.1 8.66 1 71
3 COMBD 2,164 0.22 0.86 0 4 8,473 0.67 0.67 0 3
4 CONT 2,164 0.20 0.78 0 3 8,473 0.60 0.55 0 3
5 LACE 2,164 8.63 4.03 0 19 8,473 8.63 4.03 0 19
6 LOS 2,164 2.20 1.01 0 4 8,473 0.87 0.67 0 4
7 ACUITY 2,164 1.60 0.76 0 3 8,473 1.60 0.76 0 3
9 AGE 2,164 64.56 12.15 19 87 8,473 60.1 10.45 19 85
10 GENDER 2,164 0.52 0.49 0 1 8,473 0.51 0.49 0 1
11 MEDICARE 2,164 0.47 0.50 0 1 8,473 0.49 0.50 0 1
12 MEDICAID 2,164 0.01 0.06 0 1 8,473 0.02 0.05 0 1
13 REC ADM 2,164 0.09 0.01 0 1 8,473 0.09 0.01 0 1
15 HMO 2,164 0.85 0.10 0 1 8,473 0.90 0.13 0 1
16 PPO 2,164 0.09 0.05 0 1 8,473 0.07 0.06 0 1
17 POS 2,164 0.06 0.05 0 1 8,473 0.03 0.02 0 1
50


Table 2: Correlation amongst Variables
Diabetes Sample
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 TCT 1
2 CLICKS -0.14 1
3 COMBD -0.08 0.32 1
4 CONT -0.02 0.06 0.43* 1
5 LACE 0.02 0.01 -0.02 -0.02 1
6 LOS 0.01 0.00 0.00 -0.01 0.68* 1
7 ACUITY 0.01 0.01 -0.02 -0.01 0.84 0.57* 1
8 AGE -0.01 0.04 -0.05 -0.02 0.02 0.02 0.02 1
9 GENDER 0.00 0.04 0.03 0.05 -0.01 -0.01 0.01 -0.04 1
10 MEDICARE -0.02 0.06 -0.04 -0.01 0.00 0.01 0.01 0.45* 0.00 1
11 MEDICAID 0.01 -0.01 0.02 0.01 -0.04 -0.02 -0.04 -0.09 0.04 0. 67* 1
12 RECADM -0.04 0.04 0.08 0.02 0.01 0.03 0.00 0.01 0.05 0.09 0.01 1
13 HMO 0.00 0.07 0.08 0.07 -0.04 0.04 0.08 0.01 0.03 0.00 0.00 0.09 1
14 PPO 0.09 0.06 0.06 0.05 0.03 0.07 -0.01 0.07 -0.03 0.02 0.04 0.06 0.01 1
15 POS 0.11 0.01 0.04 0.04 0.09 0.01 -0.04 0.08 0.02 0.01 0.03 0.01 0.09 0.08 1
CKD Sample
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 TCT 1
2 CLICKS -0.21 1
3 COMBD -0.11 0.35* 1
4 CONT 0.01 0.08 0.51 1
5 LACE 0.10 0.01 -0.02 -0.02 1
6 LOS 0.01 0.00 0.00 -0.01 0.45* 1
7 ACUITY 0.01 0.01 0.02 0.01 0.60* 0.34 1
8 AGE 0.01 0.04 0.05 0.02 0.02 0.02 0.02 1
9 GENDER 0.00 0.04 0.03 0.05 -0.01 -0.01 0.01 -0.04 1
10 MEDICARE 0.01 0.06 -0.04 -0.01 0.00 0.01 0.01 0.60* 0.00 1
11 MEDICAID 0.01 -0.01 0.01 0.01 0.01 0.02 0.03 0.00 0.03 0.74* 1
12 RECADM 0.00 0.09 0.00 0.00 0.02 0.01 0.03 0.01 0.05 0.09 0.01 1
13 HMO 0.00 0.07 0.08 0.07 0.05 0.09 0.02 0.19 0.09 0.04 0.07 0.04 1
14 PPO 0.09 0.00 0.06 0.05 -0.03 0.07 -0.01 0.07 0.03 0.02 0.04 0.06 0.01 1
15 POS 0.11 0.01 0.04 0.04 0.09 0.01 -0.04 0.05 0.09 0.02 0.05 0.09 0.07 3.08 1
* indicates significance at 1% level
51


Table 3: T-Test of Variables by Alert Closed for Diabetes and CKD Samples
Diabetes Sample 68.85% Closed Alerts)
Patients who closed alerts Patients who did not close alerts
N Mean SD N Mean SD P-value
Clicks 1,490 14.71 7.53 680 14.01 8.37 0.93 17
COMBD 1,490 0.21 0.09 680 0.19 0.11 0.57 a
CONT 1,490 0.21 0.11 680 0.20 0.10 0.74 a
LACE 1,490 8.21 5.89 680 8.58 5.70 0.2917
LOS 1,490 2.21 1.22 680 2.16 1.19 0.5217
ACUITY 1,490 1.53 1.15 680 1.56 1.20 0.7417
AGE 1,490 66.04 16.06 680 63.97 16.40 0.15 17
MEDICARE 1,490 0.45 0.40 680 0.48 0.41 0.4117
MEDICAID 1,490 0.01 0.00 680 0.01 000 0.8917
REC ADM 1,490 0.10 0.06 680 0.09 0.07 0.7917
HMO 1,490 0.88 0.32 680 0.86 0.35 0.2017
PPO 1,490 0.08 0.01 680 0.09 0.01 0.1817
POS 1,490 0.06 0.02 680 0.07 0.03 0.7117
Male gender (%) 775 52% 354 53% 0.92b
a Students t-test; b Chi-square values
CKD Sample (63.13% Closed Alerts)
Patients who closed alerts Patients who didnt close alerts
N Mean SD N Mean SD P value
Clicks 5,349 18.35 5.12 3,124 17.73 9.64 0.5217
COMBD 5,349 0.68 0.40 3,124 0.66 0.39 0.2017
CONT 5,349 0.62 0.50 3,124 0.68 0.56 0.2117
LACE 5,349 8.85 4.57 3,124 8.31 4.43 0.35 17
LOS 5,349 1.32 0.67 3,124 1.33 1.55 0.8917
ACUITY 5,349 1.63 1.25 3,124 1.66 1.05 0.55 17
AGE 5,349 61.52 20.43 3,124 56.39 20.36 0.93 17
MEDICARE 5,349 0.45 0.41 3,124 0.47 0.39 0.43 17
MEDICAID 5,349 0.02 0.1 3,124 0.01 0.00 0.15 17
REC ADM 5,349 0.10 0.09 3,124 0.11 0.08 0.85 17
HMO 5,349 0.87 0.49 3,124 0.93 0.43 0.2017
PPO 5,349 0.08 0.05 3,124 0.06 0.03 0.1817
POS 5,349 0.03 0.01 3,124 0.03 0.01 0.15 17
Male gender (%) 5,349 51% 3,124 51% 0.92b
17 Students t-test; bChi-square values
52


Table 4: Linear Estimation Models
Diabetes CKD
VARIABLES 1 2 3 4
OLS OLS OLS OLS
Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT)
Number of Alert Clicks (CLICKS) -0.090*** (0.040) -0.011*** (0.004) -0.079*** (0.002) -0.013*** (0.005)
Number of Comorbid Alerts (COMBD) -1.484** (0.770) -1.328*** (0.046) -1 189*** (0.046) -0.998** (0.009)
CLICKS x COMBD -0.321*** (0.020) -0.191** (0.003)
Number of Contiguous Alerts (CONT) 0.131 (0.097) 0.297 (0.161) 1.660 (0.161) 0.025 (0.097)
Observations 2,164 2,164 8,473 8,473
R-squared 0.424 0.433 0.011 0.011
Chi-squared 35.65*** 25.21*** 44.65*** 23.5***
F-test 19 78*** 22.24***
Note:
1. ***p<0.01, **p<0.05, *p<0.1
2. Robust standard errors in parentheses
3. Standard errors are adjusted for ethnonationalism groups: 60 clusters in diabetes sample, and for 54 clusters in CKD sample.
4. Models include all control variables.
5. Control variables are not significant except Medicaid, detailed results in Table 17
53


Table 5: Parametric and Semi-Parametric Estimation Models
Diabetes CKD
1 2 3 4 5 6 7 8
Weibull regression Cox proportional Weibull regression Cox proportional
Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT)
VARIABLES Coefficients Marginal effect Hazard rates Marginal effect Coefficients Marginal effect Hazard rates Marginal effect
Number of Alert Clicks (CLICKS) 1 247*** (0.137) 0.543*** (0.245) 0.237*** (0.030) -0.124*** (0.053) 1.101** (0.061) 0.493** (0.231) 0.201** (0.045) -0.091** (0.031)
Number of Comorbid Alerts (COMBD) 1.152** (0.166) 0.631*** (0.321) 0.021*** (0.057) -0.011*** (0.004) 1.141** (0.054) 0.701** (0.321) 0.019*** (0.089) -0.009*** (0.003)
CLICKS x COMBD 1.332** (0.145) 0.691*** (0.351) 0.032*** 0.011 -0.021*** (0.013) 1.060** (0.054) -0.517** (0.219) 0.023** (0.045) -0.015** (0.006)
Note: 1. ***p<0.01, **p<0.05, *p<0.1 2. Robust standard errors in parentheses 3. Standard errors are adjusted for ethnonationalism groups: 60 clusters in diabetes sample, and for 54 clusters in CKD sample. 4. Models include all control variables. 5. Control variables are not significant except Medicaid, detailed results in Appendix C, Table C2 6. The marginal effects are evaluated at the sample mean. 7. The marginal effects for the dummy variable are not reported.
54


Table 6: Description of Variables
Variable Definition
Test Closure Time (TCT) Number of days taken to close the focal test alert. For diabetes, the focal alert is glycated hemoglobin test (A1C) For CKD, the focal alert is glomerular filtration rate test (GFR)
Number of Alert Clicks (CLICKS) Number of times the patient clicked and saw the focal alert in the patient portal landing page prior to closure.
Number of Comorbid Alerts (COMBD) Total number of Comorbid Alerts (see list in Table 13 and 14).
Number of Contiguous Alerts (CONT) Number of test alert related to the focal alert. Diabetes: (e.g., Urine or Microalbumin tests for diabetes along with A1C), for the same disease diagnosis or treatment. CKD: (e.g., Creatinine or Lipid Profile along with GFR), for the same disease diagnosis or treatment.
GENDER Gender of the patient, Female=0, Male=l.
AGE Age of the patient in years
MEDICARE Dummy for Medicare patients
MEDICAID Dummy for Medicaid patients
ETHN Dummy variable indicating the patients ethnonationalism
LACE LACE Index scores for every patient on admission and discharge on the following parameters: length of stay, acuity of the admission, comorbid diseases, and emergency department visits in the previous six months
REC ADM Whether the patient was recently admitted to hospital, within last one month
LOS The length of stay; the duration of a single episode of the last hospitalization.
ACUITY The acuity of the patient at the point of admission during the last visit or stay in a hospital or clinic.
ZIP Zip code of the patients residence address
HMO Dummy for (Health Maintenance Organization) HMO insurance patients, 1= yes, 0=no.
PPO Dummy for (Preferred Provider Organization) PPO insurance patients organization, 1 = yes, 0 = no
POS Dummy for Point of Service health (POS) plans insurance patients, 1= yes, 0=no.
55


CHAPTER III
THE ROLE OF EFFECTIVENESS, EASE OF USE, AND FUNCTIONALITY ON
EVALUATION OF HEALTH APPS Abstract
Mobile health applications (apps) are a growing trend in healthcare. With so many options available and limited information regarding app performance patients, caregivers and healthcare professionals may be faced with the challenge to determine how effective health apps are. One way to assess the effectiveness of an app is by assessing the functionalities it offers. Most of the health apps available today offer just one or two functionalities. On the other hand, with the advancement in technology and the emergence of fierce competition between app developers we are seeing an increase in the number of apps that offer multiple functionalities. This research tracks a set of 185 health apps in the Android marketplace for a period of 14 weeks. We find that apps that offer instructive and integrative qualities are more efficient in providing healthcare services to patients; such positively impacts how patients evaluate the app. However despite providing good qualities, offering too many functionalities negatively impacts the app ease of use; such, negatively impacting how patients evaluate the health app. In the following, we discuss the managerial and research contributions of these findings.
Keywords: healthcare apps, mobile health applications, apps effectiveness, app functionalities
Introduction
Health apps are programmed applications that run on smartphones and tablets to provide healthcare services and improve a patients well-being (Phillips et al. 2010). There are many types of health apps. Some are designed to help the patient manage his or her
56


healthcare treatments; others help patients to live a healthier lifestyle through tracking diet and exercise patterns, still others, offer the ability for patients to communicate with medical providers and book appointments (Boulos et al. 2014; Fox and Duggan 2012). Recently, some apps have begun to offer integration with electronic medical records and personal health records. This allows patients to access their health records and enable providers to monitor patients and record their progress (Lobelo et al. 2016).
There are more than forty thousand mobile healthcare-related apps available to users on the iTunes platform, with almost equal numbers on the Android platform (Aitken and Gauntlett 2013). It is predicted that the number of users downloading health-related apps will climb to 1.7 billion by 2017 (Economist 2016), with a global revenue potential for mobile health app-based business of $21.5 billion by 2018 (BCC 2014). This degree of growth raises issues about the safety and efficacy of mobile health apps and suggests the need for empirical research to make determinations about such issues.
The category of health apps most downloaded and used are fitness and nutrition, Krebs and Duncan (2015) found among their respondents most used the apps at least daily. The researchers found the most common reasons from respondents who did not download an app to be a lack of interest, cost, and concern about apps collecting their personal data. Despite these concerns, the research found the users most likely to consistently use a health app were younger, of high socioeconomic status, educated, be Latino/Hispanic and have a body mass index (BMI) in the obese range.
Despite the popularity of health apps, some recent research indicates that they are not yet perceived as reliable models of healthcare service delivery. According to Dehzad et al. (2014), there is a lack of interest on the part of both patients and doctors to embrace apps due
57


to poor integration and concern over security issues (Dehzad et al. 2014). Luna et al. (2014) assert that wider acceptance of mobile health apps will require an overhaul of the current healthcare IT structure, which in turn will require investments in both public and private sectors geared towards increased integration of internal and external hardware and software systems. Without increased integration, mobile health apps will simply turn into information silos (Luna et al. 2014). With that, the successful development and deployment of mobile health apps will require continued investment and research. It will be necessary to consider both their success in helping patients achieve better outcomes and their integration and ease of use for both patients and healthcare providers. Although there are thousands of health apps available in the digital marketplace, only 600 are commonly used. Statista reports that 15% of the downloaded health apps are used only once during the first six months of ownership.
Health apps technology can impact patient care positively if they provide effective functionalities (Free et al. 2013a). Many health apps offer instructive functionalities such as health information, calorie counters, exercise instructions, drug instructions, and reminders. Instructive functions enable patients to self-manage medical conditions and help patients follow medical and health instructions. Instructive functionalities are passive and unidirectional.
New health apps are expanding functionality by offering integrative functionalities that enable apps to integrate with backend applications like electronic medical records and personal health records. Health apps can enhance efficiency by reducing the amount of direct interaction with healthcare professionals (Aitken and Gauntlett 2013). However, to increase the market for these apps, developers often add these functionalities before they are mature and fully developed, impacting the apps effectiveness. Functions that are not effective may
58


reduce the efficiency and lead to a negative impression and lower the potential for continual use by users (Tuch et al. 2012; Wu et al. 2011). The lack of integration and interoperability, or app performance, with electronic health records and backend systems are posing to be significant barriers to widespread mobile health usage and adoption (Dehzad et al. 2014).
The performance of apps is reflected in the ratings and reviews by the patients after app use (Irick 2008). Ratings and reviews impact the subsequent adoption and success of an application in the digital marketplace (Pagano and Maalej 2013). Apps that are poorly rated will eventually die off, while apps that have good ratings and reviews will have a better chance of attracting new users and receive a larger share of the market (Fu et al. 2013; Pagano and Maalej 2013). Thus, understanding how functionality designs and integration impact the evaluation of health apps remains important to both practice and research. The current study tries to address this gap.
Prior Literature
Previous research in health apps focused on an apps content rather than user reviews and comments (Fu et al. 2013). Analyzing reviews and comments help us to understand the relationship between users and apps (Finkelstein et al. 2014). Finkelstein et al. (2014) used data mining to investigate the impact of reviews on app downloads and found a positive relationship between the number of positive reviews and downloads. The researchers also found a correlation between app ratings and positive reviews. There have been several studies that examined reviews in the different areas like movies (Joshi et al. 2010), restaurants (Chahuneau et al. 2012), and retailers (Archak et al. 2011). On the other hand, there is little work in mining the digital health app market (Finkelstein et al. 2014; Fu et al. 2013).
59


Prior research found that health apps have the ability to increase the efficiency of healthcare service delivery and enable patients to interact remotely with healthcare professionals (Kumar et al. 2013). In addition, health apps can support in adopting a healthy lifestyle and managing chronic conditions (Free et al. 2013b). They accomplish this by enabling patients to self-monitor their medical conditions, diet and exercise targets, and other information (Ramanathan et al. 2013). Researchers have argued that apps can provide healthcare professionals with additional channels for intervention (Klasnja and Pratt 2012; Terry 2010). Today we can find many healthcare providers leveraging health apps for the treatment of chronic conditions and patient follow-up (Free et al. 2013b; Kumar et al. 2013; Milward et al. 2015).
A variety of research on mobile health apps is available in the extant literature, though little research focuses on the users experience. VonHoltz et al. (2015) surveyed patients to determine what features they desire in mobile health apps; their research was aimed to understand what drives patients to download an app rather than their experience with the app. Velsen Beaujean, and Genert-Pinjen (2013) state that users are confused and overwhelmed by the abundance of choices and functionalities available in mobile health apps. Their study focused primarily on the type of app features users looked for when making a decision to download the app rather than on user satisfaction with those specific features. In a review of patient responses to and satisfaction with mobile health apps, Zapata et al. (2015) found that users are commonly provided with specific apps for research studies and not given a choice. This type of research can help identify desirable app features, but it says little about what drives patients decisions about downloading and using apps.
60


The recent research reveals that health apps are often fragmented into information silos and suffer from poor integration with other apps or health information systems like electronic medical records (EMRs). A study of app design across multiple platforms suggested that apps often feature redundant features in closed proprietary systems that integrate poorly with other information systems (Paschou et al. 2013a; Paschou et al. 2013b). Wilhide, Peeples, and Kouyate (2016) posit a standardized framework for app development that integrates apps with electronic medical records. Fox et al. (2012) argue that app developers must embrace open-source technology and collaboration as a means of promoting wider use of apps.
Social influence is critical in the adoption of technology and the usage of digital services (Oh et al. 2015) (Cheung and Thadani 2012; Gupta and Harris 2010). Information Systems scholars argue that the decision to download a certain mobile app from the digital marketplace is influenced by the existing ratings and evaluations posted by other users (Huang and Korfiatis 2015; Senecal et al. 2005). Although the mobile application ratings and evaluations are subjective in nature, the existence of a large number of ratings and evaluations that are similar in tone increases their perceived objectivity (Flanagin et al.
2014). Health apps ratings and reviews are becoming increasingly important for healthcare providers to differentiate their apps as the industry continues to grow (Spriensma 2012).
In many studies IS scholars use user evaluation as surrogate measures of performance (Irick 2008); doing such, highlights the importance of achieving positive user evaluations and that healthcare providers should be aware of the factors impacting the users evaluation. Health IT scholars argue that technology that improves health outcomes is positively associated with patient satisfaction and evaluation (Delpierre et al. 2004; Jamal et al. 2009).
61


In the context of mobile apps, Huang and Korfiatis (2015) found that both users rational evaluation and emotional thinking are critical in formulating the users evaluation of an app. The abundance of available apps makes the users evaluation an important factor for users to compare a particular app to a competitor or a substitute.
Arnhold et al. (2014) found that very little research exists highlighting formative usability evaluations of health apps. In their research of diabetes applications, they were only able to find a single published article that connects app review with a formative usability evaluation. Users are flooded with application options once they open the App Store, evaluations and reviews help a user to understand each apps features. However, research is suggesting that users are still unaware of many apps that may contain specific features they desire (Krebs and Duncan 2015). Krebs and Duncan (2015) propose a way to mediate this process for users would be for a refereed clearinghouse that could simplify the help consumers understand the features and available apps.
Theoretical Framework
Task and Technology Fit and Information System Success
Task and Technology Fit (TTF) is a theoretical framework that is widely used within the field of information systems and has been applied to explain the different phenomena underlying the development and the usage of information technology (Tsiknakis and Kouroubali 2009). TTF was initially developed by Goodhue and Thompson (1995), later it was elaborated on by the researchers Zigurs and Buckland (1998), and Mathieson and Keil (1998). TTF theory addresses the importance of matching information systems functionality with the desired task (Lin 2014).
62


The theory of TTF emphasizes on how technology can be developed to support the user and enable completion of an underlying task (Lin 2014). When an individual uses technology to perform a given task, TTF predicts that if the task and technology are aligned, the user will be more effective at achieving his or her goal. Consequently, the user will form a positive opinion towards that technology and tend to use it more often (Liu et al. 2011). In addition to the technology and task fit, the TTF theory takes into consideration the complexity of the task. The theory examines how the interaction of technology and task requirements impacts the users performance under a different level of complexity (Ammenwerth et al. 2006).
Since the creation of the TTF model, it has been applied to a variety of contexts such as e-books adoption (D'Ambra et al. 2013), personal travel (D'Ambra and Wilson 2004), e-tourism (Usoro et al. 2010), and many others In order to use TTF to assess an information systems solution, the solution's value primarily depends on the task and the outcomes from using the technology. In addition, the user has to bear independently the responsibility of evaluating the technologies employed (Mathieson and Keil 1998; Zigurs and Buckland 1998). The Fit, according to TTF, illustrates how mediation operates as an intervention when one variable modulates one or more variables and is a key concept in the theory. This concept posits that matching technology capabilities and tasks characteristics significantly affects users perceived performance (Tsiknakis and Kouroubali 2009). Perceived Fit, on the other hand, describes a users perceptions relative to the extent that a technology solution appropriately meets targeted task requirements. As such, it is a construct that serves to bridge the relationships between the targeted task, technological features, performance and outcomes. Within the larger theoretical models of behavioral attitude and user perceptions
63


and in the context of health apps, the positive outcome of using an app to achieve a certain health goal can be viewed as the product of technological functionality satisfactorily meeting users needs (Ammenwerth et al. 2006).
Ammenwerth, Iller, and Mahler (2006) suggest that, in instances where the user anticipates that technology will offer the desired functionality required to perform needed tasks, then the user will progressively use the said app. This position implies that positive perceptions of enhanced fit between the task and the technology will ultimately increase the chance that users will continue to employ the technology to satisfy task requirements (Tsiknakis and Kouroubali 2009). Hence, the overall outcome will be improved app usage as well as better performance of the tasks that the technology is designed and expected to execute.
It is difficult to directly measure how well an information system technology can improve outcomes. This difficulty becomes the prevailing motivation for the decision by the majority of information system practitioners and researchers to rely on proxy measures to assess information systems success (Gebauer 2008). In the context of information systems applications, user evaluations are described as the qualitative or quantitative assessment users provide as feedback during the course of using a particular application (Amatriain et al.
2009). Indeed, user evaluations are the most commonly deployed forms of proxy measures (Forman et al. 2008). Another new proxy measure for information systems apps specifically is the app rank in the digital marketplace (Lee and Raghu 2014) which are directed by the user evaluations.
Task Technology Fit as a theory is related to the Information Systems Success (ISS) framework conceived by DeLone and McLean (1992). Although both models assess the
64


users opinion regarding an identified information system and the resultant influences modulating the individual capacity, the ISS framework provides a comprehensive understanding of the information systems success. On the same note, impact on performance can be applied as a proxy for information systems success. Impact on performance can be translatable to enhanced effectiveness, efficiency, or the quality of outcomes. Integrating specific features of each of the two models or theories and focusing on their broader implications for digital device-borne technologies can yield further revelations as to how TTF and ISS play similar and mutual roles in improving user outcomes and, eventually, technology adoption (Ammenwerth et al. 2006).
DeLone and McLean (1992) developed the original comprehensive and multidimensional IS Success Model as a response to the need to simplify and provide a means to conduct result comparison and knowledge accumulation around the different aspects of IS success. The researchers built the foundation of this model from the wealth of research around the aspects of IS Success, communication and information influence. This model consists of six dimensions that impact three levels of IS Success. The six dimensions are system and information qualities, use, satisfaction, and individual and organization impacts; the three levels are technical, semantic and effectiveness.
In the 2003 model update, DeLone and McLean (2003) intended to respond to their original model criticism by expanding its scope and clarify its terminology, to be able to be used more universally. Pitt et al. (1995) highlighted that DeLone and McLeans original model did not take into consideration the role of the IS department or IS personnel in working with the users on problems like installation and product education. Further discussion on the model asserted that the impacts of IS could affect entities beyond the user
65


and organization. DeLone and McLean updated the IS success model to more clearly define terminology and make it more comprehensive. These changes are identified by the inclusion of Variable Service Quality, redefining user to intention to use and finally combining individual impact and organizational impact into net benefits.
Although both the TTF and ISS models assess the users opinion regarding an identified information system and the resultant influences modulating the individual capacity, the ISS framework provides a comprehensive understanding of the IS success. On the same note, the model accommodates using impact on performance and user evaluation as a proxy for IS success. Integrating specific features of each of the two models and focusing on their broader implications for applications and technologies can yield further revelations as to how TTF and ISS play similar and mutual roles in improving user outcomes and performance (Ammenwerth et al. 2006).
Task and Technology Fit in Health Apps
It has been established in the IS literature that the TTF is a theoretical framework that examines the issues associated with the fit between technology, tasks, and how that impact the individuals performance (Ammenwerth et al. 2006; Del one and Mclean 2004b). The notion of fit in the TTF is applicable when discussing technology that allows users to perform tasks that often are completed in a traditional way (D'Ambra et al. 2013), like making a phone call to schedule an appointment or ask for a refill, or use paper diary to log calories consumption or sugar level. Investigating the intersection of the artifact functionalities for the appropriate tasks an individual want to perform is measured by the fit. TTF does not guarantee the usage of the artifact, but it helps to predict the utilization and the change of perception regarding it (D'Ambra and Wilson 2004; DAmbra et al. 2015). Such is
66


applicable in the health apps context. When an app provides a user with functionalities that enable him to perform desired tasks, the TTF predicts that if the task and technology is a good fit then the user will utilize the technology more and his or her perception will be enhanced.
A review of the literature shows that TTF explores the relationship between task and technology from different standpoints. For example, user perception of the technology, improved utilization, and enhanced performance (Yi et al. 2016; Yu and Yu 2010). Also, the literature shows that applying the TTF outside the organization context is still underresearched (DAmbra et al. 2015; Usoro et al. 2010). TTF can provide valuable insights for technology design and development (D'Ambra et al. 2013). Such makes make TTF an excellent framework to investigate health apps functionalities and their impact on the user evaluation.
Our research addresses this gap and tend to apply the TTF in context of health apps. Tasks are group of actions that an individual perform to achieve a certain goal. In our research we refer to tasks by functionalities. In our research functionalities and tasks are synonymous. An individual will download a health app that is equipped with one or more functionalities to perform one more tasks.
Assimilation and Contrast Effects
Assimilation and contrast effects reflect a positive or negative correlation between a persons initial evaluative judgements and contextual information. An assimilation effect occurs when the initial judgement and context are positively correlated, a contrast effect is when the two are negatively correlated (Johnson and Fomell 1991; Walker 1994). The theory explains how evaluative judgment is based on a mental process that is a culmination of
67


assimilation and contrast processes. These two processes are not mutually exclusive; they may happen separately from each other or overlap based on the context and the situation. Users often depend on some attributes of an object to make an overall evaluation (Forster et al. 2008; Hovland et al. 1957; Stapel et al. 1998).
The assimilation effect incorporates a new experience into an existing one to lower dissonance (Forster et al. 2008; Stewart and Malaga 2009). The assimilation occurs when the target shares common features and attributes with existing or previous experience (Forster et al. 2008; Stewart and Malaga 2009). As opposed to that, the contrast occurs when the experience falls outside the user knowledge or previous experience. The incorporation of the new experience and information can be uncomfortable to the individual. Generally speaking, the assimilation and contrast effects are silent processes, and it is hard to identify their direction, but they are often reflected in the reviews, ratings, and attitudes towards products or services (Oliver and Burke 1999). Assimilation effect is positive in nature where the contrast effect is negative (Stewart and Malaga 2009). Broadly, it is apparent that developers need to stimulate more assimilation effect than contrast effect in their apps design. Such will help in increasing the users positive attitude and gain good evaluations from users.
Conceptual Model and Hypotheses Development We suggest a conceptual model for this study; our model is presented in Figure 16. The research model leverages the TTF and ISS theories with the assimilation and contrast effects to better understand how users evaluate mobile health apps.
The previous literature shows that many ISS dimensions are applicable across several industries, whereas others are more or less specific to a particular service industry. In this study, we are focusing on two dimensions of the ISS model, instruction quality and
68


integrative quality. However, as stated by DeLone and McLean (2003) that while all aspects of the ISS represent different pieces to the ISS model, they are all interrelated and interdependent. Each affecting the overall success of the information system. The researchers explain that the construct information quality reflects both the content and quality of information provided by an information system. As in the previous section DeLone and McLeans updated model combines individual and organizational impact into net benefits. This allows for researchers to apply the model to a wider variety of contexts. In our study we define the net benefits as the overall user evaluation.
In studies following the DeLone and McLean (2003) updated model, many researchers utilized this opportunity to apply this model to a variety of research contexts within IS. For example, Lin (2008) on a study on the success of virtual communities, assert that the most accurate indicator of effectiveness is loyalty, defining for their study the net benefit as loyalty. They found information quality and system quality to affect satisfaction impacting user loyalty or as we are defining it, effectiveness. On the other hand, the TTF model describes that when the health app provides qualities and features aligned with the patient needs, then the patient will be more effective at achieving his or her goals or tasks. However, unlike the ISS model, TTF outcomes are independent of one another. That is to say that while there may show an increase in utilization, user performance effectiveness may not show an increase. This demonstrates to health care providers the need to focus on instruction quality to garner both increased utilization and performance effectiveness in an app. In our model, we work to capture the principle concept of the Technology Task Fit model in the construct app effectiveness.
69


We argue health apps that provide patients with instructions to confront their health conditions, and in some cases, information on how to cope with diseases will be more effective in managing medical and health conditions leading to higher app effectiveness. Thus, we hypothesize:
HYPOTHESIS 1 (HI): Instruction Quality will have a positive impact on app
effectiveness.
In the ISS model, DeLone and McLean (2003; 2004a) note that system quality captures desired technical characteristics of an information system. The researchers suggest that system quality can be defined by several sub-dimensions. In our study system quality is measured by Integrative Quality and Functionalities. Integrative Quality in our study is measured by the apps ability to connect to the patients health providers back end and workflow. An information system can be described as effective if it can achieve its objectives, of these, its ability to integrate.
The dominant number of current health apps have limited functionality and focus the user content to providing information and instruction. In their research, Aitken and Gauntlett (2013) were able to find only a limited number of apps that include more diverse functionalities to gather user data and send it back to the physicians and healthcare professionals. Due to the newness of this technology and the limited use, there is little research on its effectiveness and usability.
However, there is a rich abundance of literature on the perceived ease of use of health IT and apps using frameworks based on behavior theory in Psychology. Many of those studies pay specific attention to Self-Determination Theory (SDT). There are three main psychological needs outlined in SDT, (1) autonomy, (2) competence and (3) relatedness. The
70


studies above take these needs and examine what features should be included in mHealth apps to facilitate user satisfaction based on SDT. One specific aspect, relatedness, is the degree of connectedness that the person feels with others, this connectedness can be created within an mHealth app by connecting users to blogs, discussion boards, and communities that share the same diagnosis (Mendiola et al. 2015). With the continual surge in app use and mobile health, we anticipate seeing the value and importance of data transmission of health vitals to physicians and healthcare providers gain more momentum and utilization in future apps (Amhold et al. 2014). Based on that we hypothesize:
HYPOTHESIS 2 (H2): Integrative Quality will have a positive impact on app
effectiveness.
It is important to note that only recently mobile smart devices are capable of handling more functionality. That allowed healthcare app developers to create multifunctional apps (Amhold et al. 2014). Mendiola et al. (2015) conducted a study evaluating the impact of different app functions on user experience, with a specific focus on diabetes apps. The researchers found that although there were many apps available, few distinguished themselves from others by offering more than one or two functionalities. In a comparison of basic and more complex apps, Mendiola et al. (2015) noted that the more complex apps did not rate as high on their user evaluations as did the more basic apps. Arnhold et al. (2014) came to the same conclusion in their study. In addition, they found that the multifunctional apps evaluation in terms of ease of use was markedly worse. The decline in evaluations due to the increased number of features has been studied by product development scholars and been defined as feature fatigue.
71


Feature fatigue occurs when a user becomes overwhelmed with the complexity of a product or application and consequently chooses to quit using it (Rust et al. 2006). Research in product strategy shows that developers add more features to products to keep up with competition (Ellison 2003; Gottfredson and Aspinall 2005). Users will be more attracted to apps that offers many different features (Thompson et al. 2005). Once the user begins to use the multifunctional app they tend to become dissatisfied with the complexity of the app (Thompson et al. 2005; Wu et al. 2015b). To alleviate feature fatigue researchers have proposed making a simpler app that does fewer things really well (Wu et al. 2015a). From the previous arguments, the number of functionalities in a health app may potentially increase feature fatigue and thus negatively impact the ease of use. Based on that, we hypothesize: HYPOTHESIS 3 (H3): The increase in the number of App functionalities will negatively impact the App ease of use.
When individuals use a health app to obtain medical information, diagnosis of a condition or manage a condition, they will either judge the app based on the accuracy of the information, the accuracy of the diagnosis they provide, or the efficiency of the app in managing their condition. If the information or diagnosis attained is not accurate or the function is not efficient in managing their healthcare the result will be a negative evaluation (Metzger and Flanagin 2013).
When patients feel that an app has a positive influence or is an effective means to manage their health, they continue and increase utilization of the app (Price et al. 2014; Shin et al. 2012). Increased usage combined with effectiveness will yield to better performance and result in positive evaluations of the app (Direito et al. 2014). In this regard, although an app may have multiple features, patients may still not evaluate the app positively if the
72


functionalities are not effective (Azar et al. 2013). Health effectiveness means that the app is not only useful but also a good fit to provide or solve a specific health challenge (Laing et al. 2014).
A diabetes tracking app, for example, that just provides information and instructions to the patient may not engage and gamer the attention of patients; but when it helps the user monitor exercise and diabetes outcomes, providing an effective means to manage the disease, the patient will evaluate the app more favorably (Aitken and Gauntlett 2013). We argue that although developers may provide apps with multiple functionalities, for the apps to be favorable to patients they must be developed to efficiently manage health care. With a higher level of effectiveness, multi-functional apps will be able to engage patients more and subsequently get higher evaluations. Thus, we hypothesize:
HYPOTHESIS 4 (H4): App effectiveness will have a positive impact on the app evaluation.
McCurdie et al. (2012) argue that health app users are less engaged with apps that are not easy to use and do not return to them after the initial download. This continual pattern will hinder the potential effectiveness of any health interventions. Apps that are easy to use will engage users more quickly and thus result in the users evaluating them positively.
Mendiola et al. (2015) found that users placed a higher value on apps that were more simple than complex and intuitive to use. This is important to keep in mind if it becomes expected for patients to use mHealth apps as a regular part in their own care. Health apps are more able to be successful in assisting patients and serving the intended purpose with high ease of use. Apps with low ease of use, even if they have high effectiveness, the patients will
73


be less receptive of the application. Therefore, to ensure that the health app is successful in terms of evaluation, the ease of use should be a priority.
Based on the previous arguments we hypothesize:
HYPOTHESIS 5 (H5): App ease of use will have a positive impact on the app evaluation.
Method
Data and Variables
To test our hypotheses, we focused on health apps in the Android app store. We collected data through a consulting company for a span of 14 weeks from October 2014 to January 2015. Such a time span is sufficient to observe changes in the apps ratings and account for newly posted user reviews (Svedic 2015). The first week is the focal reference week for the health apps in this study. We found more than 3,203 health and medical applications in the focal week of the 13th October 2014 to 20th October 2014. We could not consider 127 apps for our analysis because they did not have any reviews in the marketplace in that week. In addition, we eliminated 96 which were not in in English, had unreadable names, or were duplicated in the market. We excluded 2,123 that were not related to patient information, disease instruction, treatment, diagnosis, or disease management. Finally, we excluded nine medical apps that are not available to the general browsing public and needs a special code to be able to use it. We also excluded 663 apps that were not directed to patients, rather directed towards providers, insurance professionals, coders, healthcare professionals, and medical students. A total of 185 apps from the Android app stores met the inclusion criteria. We tracked these 185 apps for the 14 weeks to have an unbalanced (minimal) panel data set of 2,215 observations.
74


Following the Institute for Healthcare Informatics report by Aitken and Gauntlett (2013), we have classified health apps according to their qualities into two categories, instructive and integrative apps. We also broke apps out according to the nature of their qualities into either integrative or instructive/informative apps. The instructive/informative qualities are reflected through the features of providing information in different formats to the patient or instruction on managing a condition, the prevention of diseases, healthy living, symptomatic, searching or finding a physician or facility, education on different procedures or diseases. The integrative qualities are reflected through the features of providing interactive services like following up with doctors, teleconferencing provisions, filling prescriptions, or post-hospitalization follow up with health care professionals and care givers. Table 7 provides a description of each quality and related features with some examples.
In Table 8, we provide a description of the variables we use in this study. The focal dependent variable in our model is the average weekly rating for each health app. The focal independent variable in this study is the number of health apps functionalities. The independent variable reflects the number of functionality offered by each app and was coded by coding the health apps in regard to six main functionalities. The second variable health app ease-of-use is a dependent variable it reflects the ease of navigation, how user-friendly the interface is, and whether the application was hard to learn or not. The third variable, health app effectiveness, was coded by mining the users text reviews of each health app, see Table 8 for more details.
We controlled for many variables such as the price of the app, the longevity of the app in the market, and the average download per week. We also coded dummy variables indicating the category of the app and included them as controls. We used text analytics to
75


measure both the app ease-of-use and the app effectiveness based on the reviews that were posted by the users in the Android digital market, a detailed description of the process is provided in Appendix E. Table 9 provides the descriptive statistics and correlations amongst key variables used in this study.
Empirical Analysis
Since we are tracking health apps across multiple weeks, we are using panel data for our analysis. We ran a Hausman test to determine which time of panel analyses we should follow; the result was significant and hence we used a fixed effect model. We applied the panel data estimation with fixed effect to model the patient evaluation because the change in patient evaluations is a continuous variable (Greene 2003).
Yn = pXit + a + uit + sit (1)
Where, Yn is the dependent variable, X; is a set of explanatory variables, P is a vector of parameters, u is the between-entity error and s are within entity error associated with each observation.
More specifically, to test the direct effects, we specified the following equations:
Effectivness = fin Instructive +fin Integrative + fin ControlVariahlel +a+Uit+£it (2) EaseOfUse = f2i NumberOfFunctionalities + fi2iC0ntr0lVa.ria.blei +a+Uit+£u (3) Evaluation fJ>;< / Instructive +fi 32 Integrative + [is 3 NumherOfEunctionalities + ft 3 4 Effectivnessi + (>33 Easel)ft Jse, + f2i ControlVariablei +a+uu+£u (4)
Our panel estimations employ Fixed-effects (FE) models which account for heterogeneity. A concern in our estimation might be the potential endogeneity of the variables. To address this concern, we estimated the models using the generalized method of
76


moments (GMM) Arellano-Bond estimator. This estimation method addresses situations where the regressors may be correlated with the error term (Arellano and Bond 1991).
To make sure that regressors are not exogenous, we used the GMM estimator as it is used for datasets with many panels and few periods. Using the time dimension (weeks), the Arellano and Bond (1991) derived a consistent GMM estimator for the parameters in our model. We also performed a Hansens J-test for over-identifying restrictions to test whether our instruments are valid or not. This test uses the moment conditions to test the validity of the instruments. Results (p-values) show that in our case, all the estimations were supported and we cannot reject the null hypothesis. The results obtained using the GMM models are qualitatively similar to those of the fixed-effects models. Our results are thus robust to controlling for endogeneity.
Results
We tested the direct relationship between instructive and integrative qualities and app effectiveness. The variable Instructive qualities have a positive and statistically significant association with App Effectiveness (see columns 1 of Table 10, p = 0.181, p < 0.01). These results support hypothesis HI. The variable Integrative qualities have also a positive and statistically significant association with App Effectiveness (see columns 1 of Table 10, p = 0.207, p < 0.01). These results support hypothesis H2. Further, the results translate to the finding that for apps with both Instructive and Integrative qualities that Effectiveness of the health app will increase by approximately 39%.
We find that Number of Functionalities has a negative and statistically significant association with App Ease-Of-Use (See Column 2 of Table 10, p = -0.209, p < 0.01). These
77


results indicate that App ease of use decreases with the increase of the app number of functionalities. This result supports our hypothesis H3.
We also tested for joint effects of all terms in our model (Column 3 of Table 10). The variable Effectiveness has a positive and statically significant association with App Evaluation (see column 3 of Table 10, P = 0.989, p < 0.01). This result supports hypothesis H4. We interpret this results as that if the effectiveness of the Health app increased by one fold the Evaluation of the app will increase by one star. We also find the variable Ease-Of-Use to have a positive and statically significant association with App Evaluation (see column 3 of Table 10, P = 0.189, p < 0.01). This result supports hypothesis H5. We interpret this last result as that users will evaluate health apps that are easy to use higher than the ones that are no.
We conducted several tests to assess the robustness of our findings. We tested for multicollinearity by computing variance inflation factors (VIFs) for all estimation models. The highest VIF was 2.0 in the direct-effect models, confirming that multicollinearity is not a serious concern. To reduce potential high multicollinearity issues due to the number of interaction terms in the models, all continuous variables were mean-centered by subtracting the corresponding variable mean from each value (Aiken et al. 1991). The VIF of any individual variable in any of the interaction effect models was less than 7.0. Furthermore, mean VIFs in all the models were less than 5.0. Thus, we find that multicollinearity is not a serious concern in the estimation.
To investigate how a change in functionality and appeal affects users evaluation, we employ a random-effects model to analyze our sample of the panel. To investigate how a change in functionalities and appeal affects the apps ratings, we employ a random-effects
78


model to analyze our sample of the panel. We also consider a more generic generalized-method-of-moments (GMM) approach. The GMM estimation results were similar to the panel estimation results. This validates the robustness of our results, specific to the sensitivity to endogeneity issues.
We plotted the interaction effect results and presented it in the interaction graphs in Appendix D. In the interaction plot, the difference in the positive slope of the lines in Figure 17 showing that apps with high health effectiveness compared to low health effectiveness is clearly discernable across multiple functionalities.
Discussion and Conclusion
With hundreds of health apps in the digital market, patients, caregivers, and healthcare providers need guidance on identifying apps that are effective, provide good information, provide good service, and are easy to use. This research can help researchers and developers better understand the strengths and weaknesses of existing health-related apps. With the growing demand and market for health apps, our research can help guide the development of the next generation of health apps as well.
From our previous results, we draw that adding more effective functionalities to health apps helps increase the average rating of the health app. Another finding is that the effectiveness and the ease-of-use of the Health apps will positively impact the overall rating of the health app in the marketplace. We also draw some managerial implications from this study. First, the apps qualities and functionalities play a valuable role in users evaluation. Hence, developers should pay attention toward what type of functionalities they provide in their applications. In addition, identifying users who value health apps and take their feedback is very important to enhance future releases. This study contributes to the literature
79


of mobile health applications, by identifying how technological and functional factors are associated with digital application and services success. Finally, tracking evaluations regularly is critical for a Health applications success. While review and feedback mechanisms are seen in other products and services such as games and entertainment, Health applications providers and developers need to implement such mechanisms.
In terms of research contributions, to our knowledge, this study will be the first one to explore the effects of factors like functionality, ease-of-use, and effectiveness on how users rate Health apps in the digital marketplace. These contributions will enrich the existing information systems literature and research associated with mobile Health applications. Future studies can look if Health apps were recommended or prescribed by providers and how that will impact consumers evaluation. In our study we only collected data about applications that came from the android digital market; future studies may include applications from other markets as well, like iOS and Microsoft.
There are a few limitations to this study. To begin, this study used just two pieces of data from the review process, review text and review rating, to assess the importance of the variables to the overall rating. It is likely that someone writing a positive review, indicating that the app is effective, would also give a high rating. Future research could get independent assessments of functionalities. Secondly, the constructs, integrative and instructive qualities, were measured as binary variables. For future research, researchers should look into creating a scale for measuring these constructs to provide a more insights. Finally, we used data from one digital marketplace, Google Android Store. Future research should work to replicate this research using datasets from the IOS store, Amazon and Microsoft.
80


In conclusion, as Mobile Health apps become more ubiquitous, Health app functionalities and features will have a stronger relationship with users short and long-term decision to use the app. In sum, our study shows that providers and healthcare professionals should pay attention not only to health apps functionalities but also if users find the app easy to user or not while designing and developing their health apps. Developers tend to add more functionalities to their apps to keep up with rivals and competition. That sometimes yields to increase the complexity of the app and reduce its usability. Developers that contradict what users find easy to use may trigger a bad first impression and damage users expectations, as functionality is important to establish a long-term relationship between the user and the health app. The usability and ease of use of the app is significant in leaving a good first impression to start using the app and increase their engagement. There is a need for a balance between providing more effective functionality and keep the apps simple and easy to use.
81


Figures and Tables
Figure 16: Conceptual Model
82


Table 7: Classification of Health App Functionalities and Features
Functionality* Definitions* Example(s)
Instructive/informative qualities
Prevention and Healthy Living Apps that focus on overall wellness like apps that promote healthy eating, help with weight management, fitness, tips for healthy living, smoking cessation, stress management, or promoting healthy sleep. 101 Natural Home; Remedies Cure; Relax Meditation
Symptomatic and Self-diagnosis Apps that focus on overall self-diagnosis like a reference for common symptoms, diagnosis based on data inputted and question answered. ADHD Quiz
Finding a Physician Apps that help with finding and locating medical professionals like locating a most appropriate physician or healthcare facility, find contact information, or provide rating and reviews of physicians and medical facilities. Amwell; Anthem Blue Cross
Education and post-diagnosis information Apps that focus on providing health material like drugs and medication information, emergency and first aid information, or condition management information Drug Guide for Consumers; Immunity Boosters
Integrative qua ities
Filling prescription Apps that focus on allowing the patient to request or refill a prescription like refilling of prescriptions and/or drug interactions and side effects. Anthem Blue Cross; MobileRx Pharmacy
Compliance and monitoring Apps that assist the patient to act within the prescribed interval and dose of a dosing regimen like, pill reminders, medication trackers, alert support network, enable healthcare professionals to remote monitoring of patients vitals. Allow patients to access their medical health records. Apps that integrate with the health providers workflow. Pill Reminder; Pill Organizer
* This table is adapted from the Institute for Healthcare Informatics report by Aitken and Gauntlett (2013)
83


Table 8: Key Variables Description
Variable Description and Operationalization Reference(s)
Evaluation Average weekly rating of the app in the app store (Mendiola et al. 2015; Tian et al. 2015)
Number of Functionalities The total number of instructive/informative and integrative functionalities. The four major instructive functionalities are display of health and medical information, providing medical and health instructions, search and explore for health and medical services, and providing general health education. The two integrative functionalities are connecting to healthcare providers back-end system and aligning with workflow and operational requirements. The integrative functionalities are reflected through the features of providing reminders, alerts, connecting or following up with doctors or providers, or with video or teleconferencing provisions, filling prescriptions, or compliance and adherence (e.g., medication, post-surgical). (Aitken and Gauntlett 2013; Arnhold et al. 2014)
Instructive and Informative Qualities The instructive functionalities are reflected through the features of providing information or instruction on healthy living, the prevention of diseases, self-diagnosis of diseases or a condition, searching or finding a physician or facility, medical and health education for different procedures or diseases. This variable captures the content the health app delivers in respect to correctness, completeness, ease of understanding, and relevance of the instructions or information. (Aitken and Gauntlett 2013; Delone and McLean 2003)
Integrative Quality Apps that provide interactive services like following up with doctors, teleconferencing provisions, filling prescriptions, or post-hospitalization follow up. This variable is coded as 1 if integrative functionalities are present, and 0 otherwise. (Aitken and Gauntlett 2013; Delone and McLean 2003)
App Effectiveness Effectiveness is the positive impact felt by the user that helped him or her to achieve specified health or medical goal. It can be determined by whether the users goal was successfully met or not. This variable is coded by mining the text reviews of each app in the weeks time. (Arellano and Bochniski 2012; Pham et al. 2016; Stoyanov et al. 2015)
App Ease of User This variable reflects ease of navigation, how user-friendly the interface is, and whether the application was hard to learn or not. (Arnhold et al. 2014; Mendiola et al. 2015)
84


Table 9: Descriptive Statistics amongst Key Variables
Variable Mean Std. Dev. Min Max
Evaluation 2.94 0.53 2.14 5
Ease-of-use .022 0.37 -0.89 0.95
Functionality 1.31 0.63 0 6
Effectiveness 0.35 0.23 0 1
Integrative 0.28 0.45 0 1
Instructive 0.48 0.49 0 1
Age 3.11 0.95 1.36 5.24
Price 1.07 2.62 0 24.99
Download 1,410 413 75 30,000
85


Table 10: Panel Estimation Fixed Effect Models
VARIABLES Fixed Effect Models
(1) (2) (3)
Effectiveness Ease of Use Evaluation
Instructive 0.181*** (0.014) 0.176*** (0.024)
Integrative 0.207*** (0.026) 0.137*** (0.034)
Functionalities -0.209*** (0.032) -0.089*** (0.026)
Effectiveness 0.989*** (0.037)
Ease of Use 0.189*** (0.016)
Age 0.038* (0.016) -0.036 (0.033) -0.054* (0.025)
Price -0.001 (0.002) -0.016** (0.004) 0.006** (0.003)
Download 0.001 (0.001) 0.001 (0.001) 0.001 (0.001)
App last update -0.009 (0.011) 0.571 (0.456) -0.011 (0.014)
Control variables Included Included Included
Constant 0.201*** (0.041) 0.422*** (0.087) 3.647*** (0.068)
Number of apps 185 185 185
Number of observations 2,215 2,215 2,215
R-squared 0.290 0.310 0.343
F Stat 24 92*** 15.86*** 15.91***
(1) Significance levels: ***/>< 0.01, **p < 0.05, *p < 0.10
(2) Standard errors in parentheses
(3) Models control for number of apps by same developer, developers average ratings and
reviews in the market, total rating of the developer, price of the app, when the app was last updated, when the app was introduced in the app store, number of downloads of the app, when the publisher released their first app in the app market___________________________
86


Table 11: Description of Variables
Variable Description and Operationalization Reference(s)
Evaluation Average weekly rating of the app in the app store (Mendiola et al. 2015; Tian et al. 2015)
Functionalities This variable is the total count of the four instructive functionalities and two integrative functionalities. The four major instructive functionalities of the apps: (1) display of information, (2) providing instructions, (3) search and explore functions, (4) providing education. The instructive functionalities are reflected through the features of providing information or instruction on the prevention of diseases, healthy living, symptomatic or self-diagnosis of the diseases, searching or finding a physician or facility, education on different procedures or diseases. The two integrative functionalities of the apps: (1) connecting to backend applications with the features, (2) aligning to workflow and operational requirements. The integrative functionalities are reflected through the features of providing reminders, alerts, connecting or following up with doctors or providers, or with video or teleconferencing provisions, filling prescriptions, or compliance and adherence (e.g., medication, post-surgical). Aitken and Gauntlett (2013)
Instructive/ Informative Apps that have the capability to provide information or instruction on the prevention of diseases, healthy living, symptomatic, searching or finding a physician or facility, education on different procedures or diseases. This variable is coded as 1 if instructive/informative functionalities are present, and 0 otherwise. (Aitken and Gauntlett 2013; Iribarren et al. 2016)
Integrative Apps that provide interactive services like following up with doctors, teleconferencing provisions, filling prescriptions, or post-hospitalization follow up. This variable is coded as 1 if integrative functionalities are present, and 0 otherwise. Aitken and Gauntlett (2013)
Health Effectiveness The positive impact felt or noticed by the user for using a health application. This variable is coded by mining the text reviews of each app in the weeks time. (Arellano and Bochniski 2012; Pham et al. 2016; Stoyanov et al. 2015)
Age of the app (Age) How long the app has existed in the Android market since its release. The variable was calculated by finding the difference in years between the focal week and the release date of the app. (Mendiola et al. 2015; Tian et al. 2015)
87


Variable Description and Operationalization Reference(s)
Price of the app (Price) The price of the health app in US dollars. (Mendiola et al. 2015; Tian et al. 2015)
Download Average downloads per week. (Mendiola et al. 2015; Tian et al. 2015)
Age Age of the app current version (Mendiola et al. 2015; Tian et al. 2015)
App last update Number of update versions (Mendiola et al. 2015; Tian et al. 2015)
In app purchase Availability of in-app-purchase option (Mendiola et al. 2015; Tian et al. 2015)
Category Dummy (C N) The category of the app, as pre-defined in the Android market. We coded dummy variables for each category to include in the analysis. (Mendiola et al. 2015; Tian et al. 2015)
Table 12: Correlation amongst Key Variables
1 2 3 4 5 6 7 8 9
1 Evaluation 1.00
2 Ease of use 0.100 1.00
3 Functionality -0.091 -0.180 1.00
4 Effectiveness 0.120 0.030 0.050 1.00
5 Integrative 0.061 -0.180 0.130 0.230 1
6 Informative 0.080 0.010 0.011 0.008 0.120 1.00
7 Age 0.120 0.110 0.040 0.020 0.44 0.010 1.00
8 Price 0.010 0.005 0.010 0.011 0.008 0.012 0.002 1.00
9 Download 0.010 0.010 0.110 0.091 0.051 0.010 0.005 0.074 1.00
88


REFERENCES
Adams, J., Giles, E. L., McColl, E., and Sniehotta, F. F. 2013. "Carrots, Sticks and Health Behaviours: A Framework for Documenting the Complexity of Financial Incentive Interventions to Change Health Behaviours," Health Psychology Review. Ahead-of-Print), pp. 1-10.
Agarwal, R., Gao, G., DesRoches, C., and Jha, A. K. 2010. "Research Commentary-the Digital Transformation of Healthcare: Current Status and the Road Ahead," Information Systems Research (21:4), pp. 796-809.
Ahern, D. K., Woods, S. S., Lightowler, M. C., Finley, S. W., and Houston, T. K. 2011. "Promise of and Potential for Patient-Facing Technologies to Enable Meaningful Use," American journal of preventive medicine (40:5), pp. S162-S172.
Aiken, L. S., West, S. G., and Reno, R. R. 1991. Multiple Regression: Testing and Interpreting Interactions. Sage.
Aitken, M., and Gauntlett, C. 2013. "Patient Apps for Improved Healthcare: From Novelty to Mainstream," Parsippany, NJ: IMS Institute for Healthcare Informatics).
Alpay, L., van der Boog, P., and Dumaij, A. 2011. "An Empowerment-Based Approach to Developing Innovative E-Health Tools for Self-Management," Health informatics journal (17:4), pp. 247-255.
Amante, D. J., Hogan, T. P., Pagoto, S. L., and English, T. M. 2014. "A Systematic Review of Electronic Portal Usage among Patients with Diabetes," Diabetes technology & therapeutics (16:11), pp. 784-793.
Amatriain, X., Pujol, J. M., and Oliver, N. 2009. "I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems," International Conference on User Modeling, Adaptation, and Personalization'. Springer, pp. 247-258.
American Diabetes Association. 2007. "Standards of Medical Care in Diabetes2007," Diabetes Care (30:suppl 1), pp. S4-S41.
American Diabetes Association. 2013. "Economic Costs of Diabetes in the Us in 2012," Diabetes Care (36:4), pp. 1033-1046.
American Diabetes Association. 2014. "Standards of Medical Care in Diabetes2014," Diabetes care (37), p. S14.
Ammenwerth, E., Iller, C., and Mahler, C. 2006. "It-Adoption and the Interaction of Task, Technology and Individuals: A Fit Framework and a Case Study," BMC medical informatics and decision making (6:1), p. 3.
89


Ammenwerth, E., Schnell-Inderst, P., and Hoerbst, A. 2012. "The Impact of Electronic
Patient Portals on Patient Care: A Systematic Review of Controlled Trials," Journal of Medical Internet Research (14:6), p. el 62.
Anderson, C. L., and Agarwal, R. 2011. "The Digitization of Healthcare: Boundary Risks, Emotion, and Consumer Willingness to Disclose Personal Health Information," Information Systems Research (22:3), pp. 469-490.
Anderson, J. R. 1982. "Acquisition of Cognitive Skill," Psychological review (89:4), p. 369.
Andrews, J. C., and Shimp, T. A. 1990. "Effects of Involvement, Argument Strength, and Source Characteristics on Central and Peripheral Processing of Advertising," Psychology & Marketing (7:3), pp. 195-214.
Arazy, O., and Woo, C. 2007. "Enhancing Information Retrieval through Statistical Natural Language Processign: A Study of Collocation Indexing," MIS quarterly), pp. 525-546.
Archak, N., Ghose, A., and Ipeirotis, P. G. 2011. "Deriving the Pricing Power of Product Features by Mining Consumer Reviews," Management Science (57:8), pp. 1485-1509.
Arellano, M., and Bond, S. 1991. "Some Tests of Specification for Panel Data: Monte Carlo Evidence and an Application to Employment Equations," The review of economic studies (58:2), pp. 277-297.
Arellano, P., and Bochniski, J. 2012. "Selecting a Mobile App: Evaluating the Usability of Medical Applications," Himss).
Arnhold, M., Quade, M., and Kirch, W. 2014. "Mobile Applications for Diabetics: A
Systematic Review and Expert-Based Usability Evaluation Considering the Special Requirements of Diabetes Patients Age 50 Years or Older," Journal of Medical Internet Research (16:4), p. el 04.
Au, A. G., McAlister, F. A., Bakal, J. A., Ezekowitz, J., Kaul, P., and van Walraven, C. 2012. "Predicting the Risk of Unplanned Readmission or Death within 30 Days of Discharge after a Heart Failure Hospitalization," American heart journal (164:3), pp. 365-372.
Azar, K. M., Lesser, L. I., Laing, B. Y., Stephens, J., Aurora, M. S., Burke, L. E., and
Palaniappan, L. P. 2013. "Mobile Applications for Weight Management: Theory-Based Content Analysis," American journal of preventive medicine (45:5), pp. 583-589.
Bai, X. 2011. "Predicting Consumer Sentiments from Online Text," Decision Support Systems (50:4), pp. 732-742.
90


Barth, K. S., Balliet, W., Pelic, C. M., Madan, A., Malcolm, R., Adams, D., Morgan, K.,
Owczarski, S., and Borckardt, J. J. 2014. "Screening for Current Opioid Misuse and Associated Risk Factors among Patients with Chronic Nonalcoholic Pancreatitis PainPain Medicine (Online Prior to Print: DOI: 10.1111/pme. 12403).
BCC. 2014. "Global Mhealth Technologies Market Projected to Reach Nearly $21.5 Billion in 2018; Europe Segment Growing at 61.6% Cagr." from
http://www.bccresearch.com/pressroom/hlc/global-mHealth-technologies-market-proj ected-to-reach-nearly-$21.5-billion-2018
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B.,
Procter, R., Majeed, A., and Sheikh, A. 2011. "The Impact of Ehealth on the Quality and Safety of Health Care: A Systematic Overview," PLoSMed (8:1), p. el000387.
Bloomgarden, Z. T. 2004. "Type 2 Diabetes in the Young the Evolving Epidemic," Diabetes Care (27:4), pp. 998-1010.
Boulos, M. N. K., Brewer, A. C., Karimkhani, C., Buller, D. B., and Dellavalle, R. P. 2014. "Mobile Medical and Health Apps: State of the Art, Concerns, Regulatory Control and Certification," Online journal of public health informatics (5:3).
Bowman, G., Watson, R., and Trotman-Beasty, A. 2006. "Primary Emotions in Patients after Myocardial Infarction," Journal of advanced nursing (53:6), pp. 636-645.
Britto, M. T., Jimison, H. B., Munafo, J. K., Wissman, J., Rogers, M. L., and Hersh, W.
2009. "Usability Testing Finds Problems for Novice Users of Pediatric Portals," Journal of the American Medical Informatics Association (16:5), pp. 660-669.
Buntin, M. B., Burke, M. F., Hoaglin, M. C., and Blumenthal, D. 2011. "The Benefits of Health Information Technology: A Review of the Recent Literature Shows Predominantly Positive Results," Health Affairs (30:3), pp. 464-471.
Byczkowski, T. L., Munafo, J. K., and Britto, M. T. 2011. "Variation in Use of Internet-Based Patient Portals by Parents of Children with Chronic Disease," Archives of pediatrics & adolescent medicine (165:5), pp. 405-411.
Calder, B. J., Insko, C. A., and Yandell, B. 1974. "The Relation of Cognitive and Memorial Processes to Persuasion in a Simulated Jury Trial," Journal of Applied Social Psychology (4:1), pp. 62-93.
Cao, Q., Duan, W., and Gan, Q. 2011. "Exploring Determinants of Voting for the
Helpfulness of Online User Reviews: A Text Mining Approach," Decision Support Systems (50:2), pp. 511-521.
Carletta, J. 1996. "Assessing Agreement on Classification Tasks: The Kappa Statistic," Computational linguistics (22:2), pp. 249-254.
91


Full Text

PAGE 1

HEALTHCARE EFFECTIVENESS USING INFORMATION TECHNOLOGY by YAZAN ALNSOUR B.S., University of Jordan, 2007 M.B.A., New York Institute of Technology, 2009 A thesis submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfilment o f the requirements for the degree of Doctor of Philosophy Computer Science and Information Systems Program 2016

PAGE 2

ii This thesis for the Doctor of Philosophy degree by Yazan Alnsour h as been approved for the Computer Science and Information Systems Program by Dawn Gregg, Chair Jiban Khuntia, Advisor Onook Oh Ilkyeun Ra Date: December 17, 2016

PAGE 3

iii Alnsour, Yazan ( Ph.D. Computer Science and Information Systems ) Healthcare Effectiveness Using Information Technology Thesis directed by Assistant Professor Jiban Khuntia ABSTRACT T he effectiveness of healthcare technology plays an important role as the predictor of its success or failure. This thesis proposal focuses on this phenomenon and is comprised of two essays that delve into the impact that information technology can have on healthcare delivery. The first essay explores the effect of medical aler ts and complementary comorbid alerts when viewed via a web portal, on the closure time of a focal medical alert. The objective of the first essay is to quantify the impact of the web based alerts on patients The second essay investigates what type functi onalit ies of mobile health applications (apps) have the greatest impact on patients and how this leads to better health apps evaluation. The combination of these two essays sets the foundation for the statement that the usage and subsequent effectiveness o f technology are dependent on the patient technology interaction characteristics and functionalities Data for the first essay comes from a national healthcare provider and includes of two datasets consisting of 2,164 diabetic patients and 8,473 chronic kidney disease patients. The data for the second essay comes from a dataset of 185 mobile health applications from the Android App Store, tracked for 14 weeks to form a panel dataset These two studies will contribute to the existing literature in exploring how health IT can positively impact healthcare and how particular health IT features and attributes enhance the efficiency of healthcare delivery. Additi onally, we highlight manager ial implications to s upport in the develop ment and design of health IT with the goal of providing more efficient and

PAGE 4

iv sustainable healthcare delivery and services This dissertation also includes research and practical implications that are paramount to gai n an understanding of the strategies needed when striving to enhance the design of health IT artifacts, which can potentially increase the overall effectiveness of healthcare delivery. The form and content of this abstract are approved. I recommend its pu blication. Approved: Jiban Khuntia

PAGE 5

v TABLE OF CONTENTS CHAPTER I. THESIS OVERVIEW ................................ ................................ ................................ ......................... 1 Objectives and Structures o f the Essays in the Thesis ................................ ................................ ........ 3 II. EFFECTIVENESS OF ALERTS ON TEST CLOSURE TIME FOR DIABETES AND KINDNEY PATIENTS ................................ ................................ ................................ ...................... 6 Abstract ................................ ................................ ................................ ................................ .............. 6 Introduction ................................ ................................ ................................ ................................ ........ 7 Prior Literature ................................ ................................ ................................ ................................ ... 9 Theoretical Framework ................................ ................................ ................................ .................... 12 Inform ation Cues and Decision Making ................................ ................................ ...................... 12 ELM Central and Peripheral Routes ................................ ................................ ............................. 17 Hypotheses Development ................................ ................................ ................................ ................. 19 Method ................................ ................................ ................................ ................................ .............. 25 Data and Variables ................................ ................................ ................................ ....................... 25 Estimation Models and Specifications ................................ ................................ ......................... 29 Results ................................ ................................ ................................ ................................ .............. 31 Results of Regressio n Analysis ................................ ................................ ................................ .... 31 Results of Survival Analysis ................................ ................................ ................................ ........ 33 Discussion ................................ ................................ ................................ ................................ ........ 36 Findings ................................ ................................ ................................ ................................ ........ 36

PAGE 6

vi Implications ................................ ................................ ................................ ................................ .. 38 Contributions ................................ ................................ ................................ ................................ 40 Limitations an d Suggestions for Future Research ................................ ................................ ........ 40 Conclusion ................................ ................................ ................................ ................................ ........ 41 Figures and Tables ................................ ................................ ................................ ............................ 43 III. THE R OLE OF EFFECTIVENESS, EASE OF USE, AND FUNCTIONALITY ON EVALUATION OF HEALTH APPS ................................ ................................ ............................. 56 Abstract ................................ ................................ ................................ ................................ ............ 56 Prior Literature ................................ ................................ ................................ ................................ 59 Theoretica l Framework ................................ ................................ ................................ .................... 62 Task and Technology Fit and Information System Success ................................ ......................... 62 Task and Technology Fit in Health Apps ................................ ................................ ..................... 66 Assimilation and Contrast Effects ................................ ................................ ................................ 67 Conceptual Model and Hypotheses Development ................................ ................................ ............ 68 Method ................................ ................................ ................................ ................................ .............. 74 Data and Variables ................................ ................................ ................................ ....................... 74 Empirical Analysis ................................ ................................ ................................ ....................... 76 Results ................................ ................................ ................................ ................................ .............. 77 Discussion and Conclusion ................................ ................................ ................................ ............... 79 Fi gures and Tables ................................ ................................ ................................ ............................ 82

PAGE 7

vii REFERENCES ................................ ................................ ................................ ................................ ..... 89 APPENDIX A. Focal and Comorbid Alerts for Diabetes and CKD ................................ ............................... 110 B. Detailed Estimation Results ................................ ................................ ................................ ... 113 C. GMM Estimation Approach ................................ ................................ ................................ .. 117 D. Interaction Graphs ................................ ................................ ................................ ................. 118 E. Text analytics ................................ ................................ ................................ ......................... 121 F. Literature review of apps effectiveness and impact studies in healthcare .............................. 126 G. Detailed Results of Panel Estimation Models ................................ ................................ ....... 129 H. Detection of Fake Reviews ................................ ................................ ................................ .... 131

PAGE 8

viii LIST OF TABLES TABLE 1: Descriptive Statistics ................................ ................................ ................................ ........... 50 2: Correlation amongst Variables ................................ ................................ ........................... 50 3: T Test of Variables by Alert Closed for Diabetes and CKD Samples ............................... 50 4: Linear Estimation Models ................................ ................................ ................................ ... 50 6: Parametric and Semi Parametric Estimation Models ................................ ......................... 50 5: Description of Variables ................................ ................................ ................................ ..... 55 7: Classification of Health App Functionalities and Features ................................ ................ 83 8: Key Variables Description ................................ ................................ ................................ .. 84 9: Descriptive Statistics amongst Key Variables ................................ ................................ .... 85 10: Panel Estimation Fixed Effect Models ................................ ................................ ............. 77 11: Description of Variables ................................ ................................ ................................ ... 87 12: Correlation amongst Key Variables ................................ ................................ .................. 88 13: Comorbid Alerts for Diabetes ................................ ................................ ......................... 112 14: Comorbid Alerts for CKD ................................ ................................ .............................. 112 15: Linear Estimation Models with Details Results ................................ ............................. 113 16: Parametric and Semi Parametric Estimation Models with Detailed Results .................. 114 17: Summary Statistics of Sample for Survivor Models ................................ ...................... 115 18: Log rank Test for Equality of Survivor Functions, Compared by Comorbid Alerts ...... 115 19: Ethno National ity Distribution for Diabetes Patients ................................ ..................... 11 5 20: Results of Estimation Models GMM ................................ ................................ ........... 117 21: Literature Review on App Functionalities/Effectiveness ................................ ............... 126 22: Results of Panel Estimation Models ................................ ................................ ............... 129

PAGE 9

ix LIST OF FIGURES FIGURE 1: Conceptual Model ................................ ................................ ................................ ............... 43 2: Interaction Plot Diabetes Sample ................................ ................................ ........................ 43 3: Interaction Plot for CKD Sample ................................ ................................ ........................ 43 4: The Smoothed Hazard Estimation Plot Diabetes Sample ................................ ................... 43 5: The Smoothed Hazard Estimation Plot CKD Sample ................................ ........................ 43 6: The K M Plot Showing with Comorbid vs. Non Comorbid Diabetes Patients .................. 44 7: The K M Plot Showing with Comorbid vs. Non Comorbid CKD Patients ....................... 44 8: Diabetes Sampling Process ................................ ................................ ................................ 45 9: CKD Sampling Process ................................ ................................ ................................ ...... 46 10: Alert Closure Flow Chart ................................ ................................ ................................ .. 47 11: The Alert Page of Portal Showing No Alerts ................................ ................................ ... 48 ................................ ....................... 48 13: An Alert Page Showing a Flu Vaccine Alert ................................ ................................ .... 48 14: An Alert Page Showing Multiple Alerts along with A1C Test Alert ............................... 49 15: Multiple Peripheral Alerts ................................ ................................ ................................ 49 16: Conceptual Model ................................ ................................ ................................ ............. 82 17: Plot for the interaction effect of functionalities and health effectiveness ....................... 118 18: Plot for the interaction effect of integrative qua lity and health effectiveness ................ 119 19: Plot for the interaction effect of instructive quality and health effectiveness ................ 120 20: Data Cleaning and Variables Coding ................................ ................................ .............. 122 21: Coding of Key Variables using Text Mining ................................ ................................ .. 125

PAGE 10

1 CHAPTER I THESIS OVERVIEW The evolution in the field of information technology (IT) has resulted in the various improvements in the healthcare industry (Goh et al. 2011; Iribarren et al. 2016; Lenz and Kuhn 2004; Yang et al. 2015) One of the notable contributions of information technology to healthcare is access to health information and services (Ahern et al. 2011) This access gave patients access to the health informatio n and electronic services that are available via the internet which was not possible in the past. Today, with just a few clicks patients can learn more about their diagnosis as well as self management and care by accessing the information through differen t Health IT artifacts (Krist and Woolf 2011) To this day, more and more people are opting to access health information and services through the internet (de Lusignan et al. 2014; Zach et al. 2012) Accessibility and connectivity are the main driving factors for the enablement of information, service access of the end users, and technological access to healthcare services such as teleconsultation (Kornak 2016; Vukovac et al. 2015) According to Thompson and Brailer (2004), with the increasing level of conn ectivity and utilization of the internet, patients are becoming more and more dependent and reliant on the internet to find healthcare information and health related services. This relianc e is especially true in a system where increasing demands for conne ctivity has resulted in the reduction of the cost of the internet related services such as data plans, and the emergence of competitive prices of smartphones With this the electronic physical barriers in accessing healthcare information and care (Wald et al. 2007). Using the i nternet and other methods of digital communication can enhance the efficiency of the delivery of healthcare and can empower patients to manage their health

PAGE 11

2 bett er and take ownership of their ailments (Black et al. 2011) Access to healthcare information and services are pivotal in to make better decisions regarding their health and conditions (Randeree 2009) This is the very reason why the electronic health related services that healthcare workers and professionals provide to their patients are critical in influencing positive hea lth outcomes from the patients (Goldzweig et al. 2013) (Demiris et al. 2008) Hence, the effectiveness of information technology can be measured by how patients are using it. Therefore, the d esign of a particular technology and its advanced features and functionalities can be pivotal for its success in enhancing the efficiency of healthcare delivery. Importance of the Topic As the cost of healthcare and the patient to doctor ratio continues to increase, the need for efficient and effective tools to deliver healthcare services has become a priority for the majority of healthcare providers (Douglas et al. 2011; Goldzweig et al. 2013; Jamal et al. 2009) Engaging patients in managing their diseases will reduce some of the major healthcare overheads and burdens related to communication and information dissemination (Oshima Lee and Emanuel 2013) T he use of HIT can empower the patients while at the same time allowing providers to deliver more efficient care (Graffigna et al. 2013) As functional advancements in H ealth IT have been made over the years, healthcare providers are working to identify specific characteristics of engagement that positively impact patients, empower them, and make them more involved in the healthcare process (Or and Karsh 2009) In recent years, patient web portals and health applications (apps) have become a popular topic amongst both researchers and healthcare communities, with the idea

PAGE 12

3 behind them being better to facilitate the communication between healthcare providers and patients and deliver care in a more efficient way Objectives and Structures of the Essays in the Thesis The essays in this thesis focus on two research questions and are held together by a unifying common theme, which is to explore how technology can have a positive impact on the efficiency of healthcare delivery. The first essay of this thesis asks the question: how do web b ased medical reminders impact the closure time of focal diagnostic tests. As health providers are using portals to alerts, it is proposed that increased enga gement and views of the test alerts have a positive effect on patients taking responsibility for the management of their health. In the first essay of this thesis we investigate the complementary effects of the number of focal alerts viewed, and the prese nce of multiple comorbid alerts within a portal, on the closure time of the focal decision making process and motivate him or her to proceed toward conducting th e needed critical diagnostic test. With every view of the alert, the information provided reinforces the earliest possible opportunity. The theoretical framework and hypotheses for this study are grounded in the decision making literature and positioned with the understanding that external information provided to the patient will influence the decision making process and existing studies regarding patient portal impact, this study includes a deeper investigation into portals and analyzes what particular

PAGE 13

4 features positively impact patient care. The study finds supporting evidence for using dat a from a national provider. The second essay addresses the research question: how do health application ( apps ) functionalities and effectiveness impact the apps evaluation. Patients evaluate applications based on a combination of criteria, which is formed functionality, appeal, and effectiveness of existing or previous experiences. The comparison experience draws on the theoretical foundation of the Task Technology Fit and the A ssimilation and C ontrast t heory D etermining the of health applications remains an unexplored question in the existing literature. 188 health applications were analyzed to explore the impact of functionalities and their appeal on a To test the hypotheses an empirical analysis is proposed to be conducted using data from Android market place fo r 188 applications, collected over a period of three months. The combination of the two essays sets the foundation for the argument that candid user reviews of the functionality and effectiveness of an application can greatly influence ns in adopting the application, such as a patient portal. It is paramount that users feel motivated to adopt the above prescribed patient portal, which in some cases is an application based channel of communication, in order to receive more personalized in formation regarding their health, which promotes greater self responsibility, and at the same time assists healthcare providers in delivering better and more efficient services. To sum up, the objective of this thesis is to explore the effectiveness of he alth information technologies. The first of alerts on test closure time for

PAGE 14

5 diabetes and kidney patients is already completed to its best, and is included in this thesis proposal. The second he Role of Effectiveness, Appeal and Functionality on E valuation of H ealth being proposed as work in progress as we continue to conduct our analysis.

PAGE 15

6 CHAPTER II EFFECTIVENESS OF ALERTS ON TEST CLOSURE TIME FOR DIABETES AND KINDNEY PATIENTS Abstract Health providers are using internet based patient portals to enable information access and communication to patients. Providers can initiate and inform patients about the necessary laboratory and diagnostic tests for treatment and disease management through the portals, ecision and time to conduct the test (i.e., test closure) is not explored in existing literature. This study investigates the complementary effects of some clicks, and multiple comorbid alerts in a portal, on the closure time of a focal alert. We argue tha t alerts provide cues to influence a decision making process to conduct a test, and every click on the alert reinforces the comorbid alerts intensify the influence of the cues and complement the effect of clicks to conduct a test earlier to reduce test closure time. We empirically examine and find support for the hypothesized relationships using two sample of archival data sets for 2,164 diabetes patients and 8,473 chronic kidney patient (CKD). Our additional exploration using survival analysis for the variant time effect of the alerts on the closure time suggests a span of high probability threshold for alert closures that we recommend as intervention pe riods. We discuss the practice implications and contributions of the findings. Keywords : patient portal, alert, diabetes, chronic kidney disease, comorbid alerts, focal alert, survival analysis.

PAGE 16

7 Introduction Patient portals enable information access and communication to patients from the lp to access and view information regarding disease and care management. Portals allow patients to communicate electronically and securely with their providers, request prescription refills, pay bills, review lab results and schedule medical appointments (Ammenwerth et al. 2012) Researchers note that portals have the potential to empower patients, improve care efficiency and effectiveness through information sharing, access and guiding patients for timely actions (Goldzweig et al. 2013) A distinct feature of a patient portal is the provision fo r doctors to recommend test alerts to patients. Providers initiate and inform patients about the necessary laboratory and alerts. Patients should follow the instructions, and conduct tests for proper diagnosis and management of diseases (Goldzweig et al 2013; Kruse et al. 2015) Once the tests are done patient portal, which is commonly known as test closure or alert closure (see F igures 11, 12, 13, and 14 for images on alert pages and Figure 10 for flow chart regarding alert placing and closure). Some examples of alerts are albumin limit tests, blood sugar tests, uric acid tests, ferritin levels test for iron deficiency etc Alerts serve as a mechanism to empower patients to take actions regarding their health condition and be aware of the severity of their condition. Alerts also help providers to Without a mechanism to inform the patients to conduct the tests, the disease management

PAGE 17

8 process often becomes reactive to a situation where the disease prog ression would have reached to an unmanageable stage (McCoy et al. 2012) For example, when a patient gets alerts and conducts regular cholesterol tests, he or she can take medi cation to control cholesterol levels, otherwise of which he or she may suffer from a severe heart attack. The patient alert feature is an important one to establish the portal effectiveness. Existing research has mainly explored the usability, usage an d adoption of portals (see reviews of studies on patient portals by Amante et al. 2014; Ammenwerth et al. 2012; Goldzweig et al. 2013; Kruse et al. 2015) Prior research suggests that patients are using portals frequently (Weingart et al. 2006) gradually overcoming the barriers that used to persist in the initial days of the introduction of portals in the healthcare system (Britto et al. 2009) However, Empirica l investigation of portal effectiveness is sparse in the extant literature. do a test? If so, how do patients differ this decision making? And, what is the threshol d period of the alert effect on closure? We contextualize this study to diabetes and chronic kidney disease (CKD) patients, two widespread chronic diseases. The economic burden of diabetes and CKD is huge in the United States (US), with 12% to 14% of the U nited States population being affected by diabetes, and 40% with pre diabetes (Menke et al. 2015) and 14 % of US population have some form of kidney disease (de Boer et al. 2011) These diseases have an estimated impact of more than $200 billion annual burden s (American Diabetes Association 2013) E arly diagnosis and inter vention for diabetes and CKD may aid in patients to adopt medication and lifestyle changes to return their lifestyle to normal (American Diabetes Association 2014; Drawz and Rahman 2015; Hussain et al. 2007) To

PAGE 18

9 aid in the diagnosis and management of diabetes and CKD routine tests are prescribed For diabetes, the diagnostic test is Glycated hemoglobin (e.g., A1C test) that tests the blood sugar level, and for CKD, the diagnostic test is Glomerular Filtration Rate (GFR) that checks the bold creatinine test. Because of the diagnostic importance t he two tests are considered as focal alerts in this study. Along with the A1C and GFR, providers also prescribe a set of associated test, when it is suspected that the patient may have other diseases along with diabetes or CKD. These associated test alert s are defined as comorbid alerts. We provide a detail description of the comorbid alerts taken for this study in Appendix A Theoretically, we argue that with every click on the alert, a patient derives a higher motivation to conduct the test through a cue influenced decision making process. We posit concern with multiple diseases, and therefore, the number of comorbid alerts will lead to an early closure of the a lert. Therefore, we further hypothesize for complementary effects of some alert clicks and number comorbid alerts on test closure time. We empirically examine and find support for the hypothesized relationships using an archival data set for two distinct s amples of 2,164 diabetes and 8,473 CKD patients. Also we use survival analysis to explore the time based effect of alert clicks on test closures and identify threshold periods with high alert closure probabilities. We suggest the threshold period as an intervention spans for the alerts to be more effective. We discuss the practice implications and contributions of the findings. Prior Literature P atient portals play a significant role for access to health information. A health care provider normally manages portals Technology enable s transforming data into information

PAGE 19

10 (Tallon et al. 2013) Patient web portals provide health information access, interactive services, decision support functionalities, and health reminders to patients (Carrell and Ralston 2006) Researchers suggest that patient portals have a positive impact on health outcomes. With more patients opting in to use the portals and participate in the data collection and monitoring process f ro m their home using devices, care managers receive (Byczkowski et al. 2011; Jones et al. 2015; Neuner et al. 2015) Mostly, with increasing use of portals, they are emerging as avenues for maintaining the continuing of care beyond hospitals and clinics and are helpful in managing diseases that need continuous monitoring and care (Ammenwerth et al. 2012) As a result, portals are enabling providers to achieve the goal of enhancing engagement with the patient so potential health problems are spotted and addressed early. There is an increase d interest among both petitioners and academic to better understand the impact of the healthcare information technology on healthcare delivery (Hah and Bharadwaj 2012) We ae seeing an increased reliance on healthcare providers on information (Agarwal et al. 2010; Wu et al. 2006) disease management with chronic conditions (Grant et al. 2008; Ross et al. 2004) and ambulatory care (Romano and Stafford 2011) Patient port als are emerging as plausible to provide a means to enhance the role of information technology in healthcare. Emerging information systems such as electronic health records, clinical information systems and patient health records are offering new opportu nities for efficient and high quality patient care (Wu et al. 2006) for managing his h ealth

PAGE 20

11 (Buntin et al. 2011; Holroyd Leduc et al. 2 011) In this process, patient portals are arguably emerging as an important means to involve patients in his or her treatment. Patient Portals can p lausibly provide multiple potential benefits such as conducting timely tests, compliance to medication and treatment regime; and emerge as an inherent tool for patient engagement (Alpay et al. 2011; Gianchandani 2011; Wilson et al. 2014) Existing studies suggest that W eb based interventions are effective in increasing patient empowerment compared with usual care or face to face interventions for diseases, such as increasing self efficacy or mastery with the disease management process (for a review of existing literature on web based interventions, see Samoocha et al. 2010) Emont (2011) explained how patient portals can help bridging care issues if it was linked correctly with the electroni c medical records and note that external factors like meeting the meaningful use requirements will expedite the adoption of patient web portals. Researchers suggest that the outcomes of patient empowerment using patient portals are derived from patient in sight into information and intervention of providers to use this mechanism for patient engagement (Otte Trojel et al. 2015; Otte Trojel et al. 2014) Interventions through patient portals will facilitate the instructions to reach to the patients irrespective of his or her visit schedule to the provider. In this regard, alerts to patients through the portals serve as a communication element that has the potential to work as a successful intervention (Neuner et al. 2015; Weingart et al. 2006) Irrespective of the value potential of patient portals, empirical evidence on the impact of portals on patient decision making and any efficiency or effectiveness o f the practice is limited. In a review, Goldzweig et al (2013) note that there is insufficient evidence about the effect of patient portals on health outcomes, with only a few st udies evaluating the

PAGE 21

12 outcome of care. Specific to the impact of alerts in a portal on health outcomes, a few studies outcomes, and has shown how clinical informati on systems with extra capabilities yield to more benefits. For example, multifunctional systems contributed toward increasing delivery of preventive care according to healthcare guidelines (Wu et al. 2006) and Ross et al (2004) found an improvement trend in guidelines adherence of an intervention group that used a n information reminder system over a control group. Specific to diabetes, a few studies show that there is the positive impact of patient portals on managi ng diabetes, such as patient provider communication, overall satisfaction, health management or patient outcomes (see review by Osborn et al. 2010) Overall, empirical evidence on how alerts in patient portals influence patient decision remains a gap in the existing literature that the current study tries to address using population wide disease based sampled datasets. Theoretical Framework Alerts contain a textual description of the prescribed tests with three parts. F igure 11 to Figure 14 in Appendix A show the alert pages as seen by a patient for several single and multiple alerts. In the alert views, the first column provides information about the action plan on the part of the patient in regards to the test. The second column provides information about what exactly the patient should do to conduct the test. The third column provides information on why the test is important. Information C ues and Decision M aking We posit the number of information bits in an alert work as informational cues for patients. The action plan mentioned in the first column of the alert page view contain the

PAGE 22

13 flu vaccine test This type of information makes a patient infer a state of disease affliction, i.e., the patient may interpret that he or she is sick, or his or her health condition needs attention and docto r intervention (McCoy et al. 2012) A state of disease affliction or indication of affliction is an unpleasant state of mind of the patient due to her health condition which induces emotional distress as well as physical pain or distress (Bowman et al. 2006) After the identification of the disease afflicti on, the patient would be thinking about the test and its future implications, i.e., what happens if the test becomes pos itive or negative, or the patient is diagnosed with an advanced stage of the disease. The second part in the alert recommendation indi cates or directs the patient on what doctor (with address) with an appointment the test This information prov ides the cue in regards to what need to be done and the logistical issues or challenges associated with the action to conduct the test. Further, it makes the patient think the logistics and scheduling plans involved with the test. The directions lead th e patient to weigh on a number of factors that are relevant to the test taking process itself, such as affordability of the test, his or her availability to spend time for the test, the accessibility of a testing lab near to him, and the flexibility of his time to go and really do the test (Adams et al. 2013; Rothman 2000) Often some tests need a substantial preparation regarding fasting or preparing for some hours before the test. The p atient may consider whether the tests are affordable. Purely economic factors such as insurance premium costs or patient co pays or deductibles, although may not be too large, may have a In other words the interpretation of the cues provided through the logistical direction and information may lead to a set of

PAGE 23

14 availability and accessibility of the tests. The third part contains informa tion on why the test is important, with some statements such as for ... Such information provides clarity to the disease regime and the role of the test concerning the disease treatment (Han et al. 2 006). At the same time, the interpretation of these cues may management process, such as how to deal with medication issues or treatment regime in the post test situat ions (Viswanathan et al. 2012; Weingarten et al. 2002) or subsequent health related behavior adjustments or lifestyle changes that may be needed to manage the diagnosis outcome (Koenigsberg et al. 2004) Thus, alert clicks provide patients a set of cues on which test needs to be completed why it is important and what logistical process need to be done to conduct the test. The question then remains how these cues influence a decision making process as a problem solving activity terminated by a solution deemed to be satisfactory to the decision maker (Kahneman and Tversky 1984) The involved process may be a reasoning or emotional process which can be rational or irrational, and can be based on an information based approach (e.g., tacit), or a cognitive approach (e.g., explicit), and may include a cost benefit analysis (Schacter et al. 2010) Some researchers describe a multiple phase process to the information processing and the subsequent decision making process, arguing that it is a step wise time taking cognitive and rational evaluation process. For example, studies in the contexts of acquisition of a cognitive skill (Anderson 1982) or taking decisions to adjusting to a social environment (Crick and Dodge 1994) suggest that

PAGE 24

15 information processing and decision making involves steps These steps include encoding and interpretation of the cues, generation, and evaluation of options, the decision to enact on a possible response and t he final phase of actually doing the needed action. Specifically, in researchers suggest that the information processing and decision making path is reliant on how the p roblem is presented, how an individual takes in information, processes that information and plans for solutions to personal problems, and carries out those plans (Heppner and Krauskopf 1987; Heppner and Lee 2009) We extend the prior research to the context informational cue test. Anchoring to the prior research on cue based information processing and decision making process (Hilligoss and Ri eh 2008; Kahneman and Tversky 1984; Sillence et al. 2007) we propose a conceptual model (see Figure 1) for this study. The conceptual model suggests that clicks on alerts and number of comorbid alerts have direct effects on test closure time. We argue that with each click, the i nformational cues intensify the momentum to take a decision, thereby decreasing the time to conduct the test. In addition we argue that to the details and more b its of information associated with multiple alerts. This concerning effect further reduces the time to conduc t the test. Also the model posits that comorbid conditions have a complementary role to decrease the effect of clicks on the test closure time. W e contextualize and elaborate on these arguments further do draw testable hypotheses. The theoretical framing of this study relies on the early work on process theory of decision making that proposes that decisions are outcomes of mental operations occur ring

PAGE 25

16 between the presentation of the decision problem and ultimate choice. The decision is dependent on the existing knowledge, experience, and perspective (Crozier and Ranyard 1997; Thomas 1999; Wagenaar et al. 1988) Applying to the context of health decisions, the process of making a decision involves tradeoffs between the quality of life and length of life, or, treatment with side effects and no treatment, depending on the individual preferences (Fraenkel 2013; Kassirer 1994; Pieterse et al. 2013) The process views of decision making highlight three points. First, decision making is a time extended process inclusive of a number of stages. Second, during the process, the decision makers understanding of the problem changes with the time to aid the decision maker to reach decisions (Bro wnstein, 2003). This mental restructuring depends on the evaluation and re evaluation of the problem attributes (Brounstein, Ostrove, & Mills, 1979; Brownstein, 2003; Mann, Janis, & Chaplin, 1969). Third, a decision maker follow s compensatory or non compen satory decision strategies (Pieterse et al. 2013) While compensatory decision strategy weighs information cues so that positive attributes can counterbalance negative attributes, the non compensatory decision strategy depends on the thresho ld value of options than information on the problem (Pieterse et al. 2013) For example, a patient may choose surgery as a decision for a disease (e.g., cancer, ovarian cysts) as a compensatory strategy to get well, although it involves side effects or further complications. A non compensatory strategy would be to monitor actively (or, surveillance) the disease progression that does not get rid of the disease but results in no side effects or complications with leading life. We apply the t hree points of the decision making process, i.e., the time extended stages, mental restructuring, and compensatory or non compensatory decision strategy to the

PAGE 26

17 context of alerts. We argue that the portal based alert context influences the three points through three different factors: (1) time spent on the portal providing a time extended understanding of the disease or state of health (2) mental representation and restructuring of the disease attributes due to the information cues contained in the alerts, and (3) compensatory or non compensatory decision strategy based on the comorbidity states. We draw our hypotheses mainly based on these three parameters associated with the proc ess of decision making. ELM Central and Peripheral Routes D eveloped by Richard E. Petty and John T. Cacioppo in the 1980s the Elaboration Likelihood M odel (ELM) works to explain a typical persuasion situation. A situation where an individual reads listen s or view s a message that presents an argument from another party in a certain setting (Petty et al. 2015; Petty et al. 1983) The ELM posits differ ent routes to and its impact on changing attitudes and behavior (Petty 2013) ELM focuses on attitudes which guide dec isions. The ELM proposes two routes to persuasion: the central route and the peripheral route (And rews and Shimp 1990) If the recipient follows the central route, persuasion will conscious thinking of the information presented to him or her Recipients who are highly motivated will be more likely to follow t he central route. Motivation can come from recipients who have a high level of attention and comprehension of the information carried in the message presented to them (Petty et al. 1983) Fear of consequences can also be a reason to increase the attention of the recipients. Strong fear can lead to the negative reaction so a solution should be offered with the message to motivate positive dec ision making (McNeill and Stoltenberg 1989) The central route is very useful for

PAGE 27

18 long term attitude change rather than abrupt short term impact (Petty 2013; Petty et al. 2002) On the contrary under the perip heral rout e, persuasion does no t come from the main persuasive message but instead comes from secondary cues. The cues received by the recipients under the peripheral route are less related to the main message or arguments (Petty et al. 1983) The change results from the peripheral route are short term like a simple compliance (Petty 2013; Petty et al. 2002) In a 1984 study by Petty and Cacioppo they demonstrated the difference between the central route and peripheral route by using the number of arguments as a peripheral cue. The researchers found that a single message with a quality argument was enough to influence the highly involved responde nts. While the exposure to multiple arguments made respondents with low involvement to develop an attitude change Based on Petty and Cacioppo (1984) w hen a patient is motivated and involved in his or her health condition a focal alert will influence him or her to make a cognitive response in decision making regarding his or her condition In some cases, when the patient is not highly motivated and involved regarding his or her condition he or she may require in addition to a focal alert secondary cues to be persuaded to change their attitude and behavior (Susan et al. 1998) You can notice from the graph below, Figure 1 5 how the medical alerts page shows multiple alerts with some arguments. In our conceptual model we hypothesize that patients who are presented with multiple peripheral alerts will develop a greater change due to t he existence of secondary cues, which aligns with the ELM theoretical framework.

PAGE 28

19 Hypotheses Development Alerts provide information cues concerning a possibility that patient will be afflicted with a disease, or may be already afflicted which needs to be ma intained (McCoy et al. 2012) We argue that the informational cues contained in the alerts make a patient to access and digest more knowledge about the disease. Prior research notes that the benefits of an ambi a nce process of diagnosis and treatment for a disease partly due to the high involvement (McAllister et al. 2012) In a scenario where the alerts are not sent the patient does not engage with such information based involvement process. On the contrary, with the alerts, the patient unravels the knowledge or issues involved with the disease. As a result, the patient understands why and what the test involves, and what may be the outcome of missing the test. Every time a patient clicks on the alert page, the reinforcement of the test for the disease diagnosis becomes higher. The patient undergoes a series of encoding and interpretation process while he or she is future consequences regarding (Leveille et al. 2009) In other words, clicks help a time extended understanding of the disease or state of health as well as restr ucturing the mental representation of the disease attributes due to the information cues contained in the alerts. According to the ELM there are two routes to persuasion (Petty and Cacioppo 2012; Petty et al. 1983) The first route is known as the central processing, in this route persuasion is based on considering thoughtful arguments and information. In the central processing the receiver is more active in the persuasion process. The central processing route needs the

PAGE 29

20 receiver to be motivated about the message her or she receives. If the receiver is not motivated by the message he receives, he will lack the motivation to do central processing (Petty and Cacioppo 1979) The second route is known as the peripheral processing route. In the peripheral route the receiver gets persuaded by a message not based on its focal argument but based on other cues. For example, a receiver may get persuaded by a message because it has many arguments but he or she lacks the motivation to think about each argument individually (Petty and Cacioppo 1979) The receiver in peripheral processing lack s the motivation and more passive. He or she will use the peripheral cues like many arguments in one message as a short cut to make a decision (Petty and Cacioppo 2012) Existing studies note that in the presence of informational cues and high involvement, consumers take a positive decision (Petty 2013; Petty et al. 1983) such as involvement in a product testing or advertisement campaign leads to purchasing a product (Michaelidou and Dibb 2006; Michaelidou a nd Dibb 2008) Similarly, in the health context, high involvement care (Wilson et al. 2014) and leads then to manage and derive care from the patient (Kamis et al. 2014; Sherer 2014) As much as each click leads a patient to feel the involvement with his or her health or disease situation, he or she will perceive that conducting the test will lead him to a sense of control over the management of health condition; and will lead for a faster closure time. Thus, we argue that in the case of self involvem ent and high motivation, each view of the alert page increases the information the patient is exposed to That will yield to an increase in the level of elaboration of the persuasive arguments displayed in the alert

PAGE 30

21 message. Such, can reinforce the patient attitude towards conducting the needed test As much as high involvement results in a perception of empowerment to manage the own disease, the alert message will increase the motivat ion to conduct the test. High involvement and subsequent information w ill help a patient to overcome the situational and economic barriers to form a positive attitude to conduct the required test as soon as possible. Based on these arguments, we hypothesize: Hypothesis H1 : Number of alert clicks is negatively associated with test closure time. We argue for a quick decision for a patient to conduct the test when comorbid alerts are present. Comorbidity a distinct additional disease entity that has existed or may occur during the clinical course of a patient who has the i ndex disease under study or simply put, the co occurrence of multiple diseases in one person (Guralnik 1996; Van den Akker et al. 1998) Although measurement of comorbidity differs, a commonly accepted norm is to follow the evidence base around disease iden tification (such as international classification of disease codes) clusters to provide a count of comorbid diseases or symptoms (de Groot et al. 2003) We follow similar guidance to classify the comorbid alerts. For example, if a doctor is suspicious of any cardiovascular disease along with diabetes for a patient, then the doctor will order a complete blood, cholesterol and lipid profile test for the patient on the alert page. This alert indicates the patient has three comorbid alerts along with the focal dia betes alert. The perception of comorbid alerts highly concerns to a patient than a single alert good and he or she has multiple diseases. The state of disease affliction with two or more dis eases would be a highly distressing situation for the patient (Bowman et al. 2006) and he or she will have a high

PAGE 31

22 motivation to get clarity about the situation. The process views of decision making highlight the importance of mental restructuring and subsequent following up of compensatory or non compensatory decision strategies by a patient. The stages of comorbidity or a set of comorbid a lerts emphasize on a higher level of attributes for a patient (Valderas et al. 2009) He or she infers that the disease condition is at a serious or more aggravated stage and the patient needs to take some action. This changes the mental structuring associated with the disease condition from a ca sual or not so important one to a serious one. As a result, the evaluation and re evaluation of the subsequent action plan to conduct the test becomes a central issue than a very trivial one. Research shows that with a high emotional situation in regards t o health conditions, patients would seek a clear and reliable information (Anderson and Agarwal 201 1) Because the tests will provide clarity to the patient on his or her multiple disease situation irrespective of the relative compensatory or non compensatory outcome of the test, the patient would want to collect more information. At least the test information will make him or her laden with information to take subsequent evaluations. In addition to the seriousness associated with comorbid states, each view of the alerts page will expose the patient to more cues about the comorbid state. While the levels of interpretation regarding the test and disease related changes with each view Each view of the alerts page would expose the patient to more cues. For patients who are not highly in volved motivated, and expressed low elaboration of the focal alert, the existence of additional cues from comorbid alerts will be utilized to processes the informatio n in a peripheral route. Therefore, we argue that the existence of comorbid alerts in the alerts page will generate more cues that will lead to a peripheral routes processing. Patients will seek more clarity to his or her condition, and furthermore, each v iew of the alert page will have a positive

PAGE 32

23 reinforcement to persuade the patient to conduct the tests faster. Based on these arguments, we hypothesize: Hypothesis H2 : Number of comorbid alerts is negatively associated with test closure time. The route that the patient takes when seeing the alert message is important. In the central processing route the persuasion is based on the argument provided in the alert and will have a higher impact on the long term in changing here in the peripheral processing the receiver in not highly motivated by the focal argument and he or she is persuaded by peripheral cues. Such type of persuasion is short term in nature (Hawkins et al. 2008; Lustria et al. 2013) The providers need the focal alerts to include arguments that influence patients to be more engaged and influence them to follow the central processing route to make decisions regarding their conditions This will make the impact of the focal alerts last long er and will have a higher influence long term behavior (Noar et al. 2 011; Ownby et al. 2012) However, even if messages include good arguments receivers many not be motivated to engage. Providers need also use additional cues to help to persuade the receiver using peripheral processing. Calder et al. (1974) found that subjects who get exposed to a larger number of messages and arguments produce more favorable responses. The researchers found that greater number of arguments influence d multiple studies h ave found that messages that contain more arguments can create more (Calder et al. 1974; Chaiken 1980; Petty et al. 1983) The multiple messages or the message with multiple arguments will influence the receiver to peripheral processing. So even if the receiver is not engaged with the focal argument, the

PAGE 33

24 multiple cues will manage to persuade him or her to for a d esirable outcome. Thus we hypothesize : Hypothesis H3: Number of comorbid alerts complement number of clicks to reduce the test closure time. So far, we argued that the number of clicks and comorbid alerts influence quicker test times. Further, comorbid alerts complement the number of clicks to reduce the test closure time further. The underlying assumption under these arguments is that the probability to variant event. In other words, the likelihood of conduct ing a test on a particular day is fixed While this assumption leads to explore the effect of clicks and comorbid stage on test closure, but it leaves an important aspect of patient behavior towards the decision to conduct the test. For example, a patient may change his or her perception to conduct a test as time passes on, with a collection of more information. When a doctor prescribes a test, a patient may view it unnecessary or not urgent. However, when the patient seeks family test, he or she may consider the test to be an important one to be conducted as soon as possible. Such external factors will motivate the patient to do the test with passage time. Alternatively of information about the test will increase the probability of conducting the test with time. In the context of alerts, the time variable captures such information effect that in reality may not be a linear projection. As time increases, early motivators (e.g., patients who are convinced to do the test early), will conduct the tests early. Some patients may be converted from late responders to early responders, and some may get an opposite information to delay the test forever or not do the test. In oth er words, we expect that as time

PAGE 34

25 progresses, the probability of patients to conduct the test will increase, but until a threshold point, beyond which it will decrease. We explore the time variant effect on test closure time using survival analysis. In ad dition such an exploration may provide a threshold window where the test closure probability is high than other times, and intervention at that window may lead to conduct the test earlier with a compounding effect. Method The methodology in this study follows two steps. In the first step, we test the hypotheses using a linear regression method. In the second step, we conduct survival analysis to explore the time variant effect of test closure time. Using the two steps analyses allows us to get over th e limitations of the linear regression and take advantage of the survival analysis strength. In this section, we describe our data collection process, the variables used in this study, estimation models and approach. Data and Variables We used an archival data for diabetes and CKD patients. We treated these two different set of patients as two samples. The data was taken from diabetes and CKD registries patient portal management systems. The patients in our samples had at least one visit to a doctor from Jan 2014 to Feb 2014 to one out of many facilities of the provider. The patient data was already de identified before be ing collected in the registry. We used a structured data collection and sampling process as shown in Figure 8 and Figure 9 For the diabetes sample, we considered all the enrolled patients in the diabetes registry (e.g., suspected diagnosis of diabetes or pre diabetes). The registry had 166,107 patients, out of which 152,219 patients were not given any lab test recommendations or lab

PAGE 35

26 test alerts. That left us with 13,888 patients with some or other lab test alerts in the system. We had to exclude 54 patients who did not have complete records or had some missing data. Out of the remaining 13,834 patients, 11,670 were not given any diabetes related alerts, and 2,164 patients were given diabetes related alerts, including the focal or most prevalent A1C alert. These 2,164 patients form the sample f or this study. For the CKD sample (see the data sampling process flow chart in Figure 9 ), we considered all 51,343 patients who are suspected to have CKD, out of which only 11,691 patients were given any lab test alerts in the system. We had to exclude 17 patients who did not have complete records or had some missing data. After excluding 3,201 patients without any alerts in the sampling period, we had 8,473 CKD patients with 5349 patients closing the alerts within the stipulated time, and 3,124 patients n ot closing the alerts or doing the tests by the due date. The dependent variable is the Test Closure Time (TCT) a continuous variable that measures the number of days that took a patient to conduct the focal A1C test or GFR test as applicable, from the d ay it was posted on the patient portal. Any doctor, nurse or lab done, with uploading of the results of the next steps. Once this is done the system closes the a lert, with a date and time imprint in the system. We took this date and time stamp as the alert or test closure day to calculate the Test Closure Time (TCT). Amongst the independent variables, the Total Alert Clicks (CLICKS) variable captures the number o f times the patient clicked and viewed (similar to the page views) the A1C alert or GFR alert in the patient portal before closure. This was also calculated through the digital health records system of the provider, using the time stamp for each click by the

PAGE 36

27 patient that the patient used to view the alert. This provider is highly motivated to assess the impact of the portal to aid in further research and development, and the capturing of the time stamp of clicks is part of the initiative to assess the impact of the patient portal. The variable Comorbid Alerts ( COMBD ) is a count of the total number of comorbid alerts given to the patient. The provider uses the comorbidity conditions as a practice based mechanism to track patients. Thus, the comorbid diseases for diabetes and CKD used in this study come from the evidence based practice in health care. We provide the description of the comorbi d diseases and related lists of tests for diabetes and CKD in Appendix A We include several controls in the models to account for the factors that may have influence for the time taken to close the alert. We included a variable Contiguous Alerts (CONT) with the count of the number of alerts for the same disease, i.e., diabetes or CKD. length of stay, acuity at the time of admission, comorbid diseases and emergen cy department visits for the past six months. This variable is used in existing practice and research literature to account for the patient severity (Au et al. 2012; van Wa lraven et al. 2012) We also added a variable indicating the maximum acuity stage of the patient during the last visit or stay for any disease in a hospital or clinic. We controlled for the age and gender of the patient. We included dummy variables as c ontrols, indicating whether the patient is Medicare or Medicaid patient. We used the Ethno Nationality (ETHN) variable (see Appendix C) for the groups in our sample. We used these groups to adjust for possible within group variations across groups by esti mating and reporting standard errors adjusted for the clustering effect. We did not have income data for the patients, but to account for any income effect due to the geographical

PAGE 37

28 location of the patient, we included the zip code of the patient as a contro l variable. We also Table 1 provides the descriptive statistics of the variables. In the sample of 2,164 diabetes patients, 22 percent of patients had any of the Comorbid Alerts. In the diabetes sample, The Test Closure Time (TCT) vary from a day to 88 days, for which the CLICKS var y between 3 to 60 times. There are 52% male patients, with age s spanning 19 to 87 years; th e average age in the sample was 65 years Only 1% of the patients in the sample are Medicaid patients, while 47% belong to Medicare group. In the sample of 8,473 C KD patients, 67 percent of patients were not recommended any of the Comorbid Alerts. The Focal Alert Closure Time varied from 1 a day to 105 days, for which the CLICKS vary between 1 to 71 times. There are 51% male patients, with age s spanning 19 to 85 y ears; the average age in the sample wa s 60 years Only 2% of the patients in the sample are Medicaid patients, while 49% belong to Medicare group. The m ajority (>85%) of the patients in both the samples have Health Maintenance insurance plans. Table 2 provides correlation amongst variables for diabetes and CKD samples respectively. We observe that there is a high correlation between the Medicare and Age of the patients, indicating the Medicare patients are mostly elderly patients. Further, the varia bles Contiguous Alerts and Comorbid Alerts hav e correlated at above than 0.4 levels, indicating the presence of multiple tests for both focal and other diseases for patients. We compared the difference between both the patients who closed the alerts and th ose who did not in both the Diabetes and the CKD samples. The T tests indicate no significant differences across the samples (see the results in Table 3)

PAGE 38

29 Estimation Models and Specifications The Test Closure Time (TCT) is a continuous variable. Therefore, we use ordinary least squares (OLS) estimation in our models. The main OLS specification is as follows: Y i i + (1) Where Yi is the dependent variable, Xi is a set of explanatory variables is a vector of parameters, and are disturbances associated with each observation. Because we have both direct and interaction effects, we specified the following two equations: 10 + 11 12 ( COMBD 1c Control + 1 (2) 20 + 21 22 ( COMBD 23 (CLICKS COMBD 2c Control + 2 (3) We use survival analysis to investigate the change in the probability of test closure with time. The dependent variable in our survival analysis is a combination of time and event or censor. T he time variable reflects on the duration until the patient went and conducted a diagnostic test. The event variable is when a patient conducted the test. We are interested in exit probability of the patient due to the event with time, i.e., calculating th e hazard rate of conducting the test. Our dependent variable is considered to have a continuous probability distribution, where is the time it takes a patient to take an action and conduct the focal test. For this analysis, we used the generic survival function of the form: (4) Where, is the survivorship function reporting the probability of surviving beyond time, and which means the probability that there is no failure event prior to i s a random nonnegative variable denoting the time to the studied event. The survival function is equal to one at equals to zero and decreases towards zero as goes to infinity.

PAGE 39

30 The hazard rate in our study is defined as the probability that the test closure will happen at time The hazard function, also known as the conditional failure rate, is the instantaneous rate of failure: (5) Where, he limiting probability that the failure event (closing the test) occurs in a given interval, and is the density function that is obtained from the first derivative of F(t) from the equation above, which equals to the negative first derivative of the Survival analysis. (6) The hazard rate range can vary from zero to infinity, which is interpreted not to risk at all (zero) to certain even t (certainty of failure). The hazard rate can increase decrease or remain constant with time. The cumulative hazard function is derived from the probability density function and the survivor function: (7) The cumulative hazard measure the total amount of risk that has been accumulated up to time We also divide our sample into two distinct samples ; the first contains patients that that patient that had comorbid alerts displayed on the web portal. We investigate the difference and compare the impact of the comorbid alerts on the hazard rate on both samples. We extend our analysis further do draw more insights. The survival process is a stochas tic process and the time index is one day, the process in our context governs whether

PAGE 40

31 an alert disappear s or closed in the patient portal. In a hazard context, we assess the impact of an explanatory variable on the hazard of exiting the patient portal rat her than on the length of survival time. A proportional hazard model on the other hand, would approximate the probability of a patient conducting a diagnostic test at a certain number of days. Although Cox proportional hazard measurement is non parametric a Weibull proportional hazard measurement uses a parametric form to estimate the hazard. We focus first on a possible change in alert survival due to viewing the alert on the patient portal. The model to be estimated is the proportional hazard model: (7) Where s are the set of explanatory variables which shift the hazard function proportionally, s are the parameters to be estimated, and is the baseline hazard. In the Cox specification of the model above there is no assumption is regarding the distribution of In the second phase of the analysis, we examine the impact of comorbid alerts on a focal alert survival. We observe the numbe r of comorbid alerts displayed with each focal alert during the observed time. We use this data to assess how the number of comorbid alerts can affect a focal alert survival. However, the change in survival may not be linear and may be moderated by the num ber of comorbid alerts. To be able to accommodate such possibilities, we interact the number comorbid alerts with the number of views. Results Results of Regression Analysis Table 4 reports results of our estimation models. Columns 1 and 2 present the direct effect and interaction effects for the diabetes sample, and Columns 3 and 4 that of the CKD sample. The variable CLICKS has a negative and statistically significant association with

PAGE 41

32 Test Closure Time for both diabetes patients sample ( see Column 1, Table 4, 0.090, p < 0.01 ) and CKD sample (see Column 1, Table 4, 0.079, p < 0.01 ). These results support hypothesis H1. Further, the results translate to the findings that f or diabetes patients, a patient will close the focal A1C a lert a day earlier with approximately 11 clicks. Whereas, in the case of CKD patients, a patient will close the focal GFR alert earlier with approximately 13 clicks. We find that Number of Comorbid Alerts ( COMBD ) has a negative and statistically signific ant association with Test Closure Time for both diabetes patients (see Column 1, Table 4, 1.484, p < 0.05 ), and in the CKD sample (see Column 3, Table 4, 1.189, p < 0.05 ). These results indicate that higher number of comorbid alerts results in t he patient doing the test earlier. This result supports our hypothesis H2. We interpret these findings as that the diabetes patients close the A1C alert on average of 1.5 days earlier than in the presence of any additional comorbid alert, where CKD patien ts close the GFR alert on average of 1 day earlier in the presence of comorbid alerts. In the interaction effects model, we find that the interaction term CLICKS COMBD is negative and significant in diabetes sample (Column 2, Table 4, 0.321, p < 0 .01 ), as well as in CKD sample (Column 4, Table 4, 0.191, p < 0.05 ). These results provide support for H3 that higher number of comorbid alerts complement the number of clicks to decrease the test closure time further We interpret these results as that the diabetes patients close the alerts a day earlier with approximately three clicks in the presence of one additional comorbid alert, which is otherwise higher (e.g., 11 clicks). Similarly, CKD patients close the aler ts a day earlier with approximately five clicks in the presence of one additional comorbid alert. We plotted the interaction effect results and

PAGE 42

33 present ed it in Figure 2 and Figure 3 The difference in the negative slope of the lines showing patients with high comorbid alerts compared to low comorbid alerts is clearly discernable in the interaction plot. Among other results in the estimation, we did not find any significant association of control variables except Medicaid. In the interaction effects mode l, Medicaid has a significant and positive effect on the Test Closure Time indicating that Medicaid patients may take a longer time to close the alerts. We conducted several tests to assess the robustness of our findings. First, we introduced an interactio n term consisting of the Contiguous Alerts and Clicks in our models to check if patients display similar concern for some alerts with the same disease. This interaction term was not significant, indicating no interaction effect of contiguous alerts on cli cks. Second, we checked for multicollinearity by computing variance inflation factors (VIF) after regressions, which were less than 4 in all the models. These indicate that multicollinearity is not a serious concern in our analyses. Third, White (1980) tests for heter oscedasticity did not suggest the presence of heteroscedasticity. Nevertheless, as suggested (Greene 2003) we estimated robust standard erro rs in all our estimations, and our results remain unchanged whether we use robust or non robust standard errors. Finally, the standard errors in the estimation w ere clustered by the ethno nationalism of the patient to adjust for possible within group variat ions, and we report standard errors adjusted for the clustering effect. Results of Survival Analysis The survival analysis results are presented in Table 6 We find that for both Diabetes and CKD, the Weibull and Cox proportional hazard models provide similar results with the

PAGE 43

34 sign and significance levels being same for each focal variable. Much of the difference in the coefficient magnitudes comes from the variance differences. First, in Column 1 of Table 6 the Weibull regression shows that the coefficient of CLICKS is positive and significant (Column 1, Table 6 ). This finding shows that an increase in CLICKS will increase the hazar ds rate by 1.25 times. In other words, for diabetes sample, one additional click by a patient will increase the A1C test closure probability by 1.25 times in a day. Similarly, in the CKD sample (Column 3, Table 6 ) one additional clic k will increase the GFR test closure probability by 1.14 times. The results also show the number of comorbid alerts have positive and significant effect (See Columns 1 and 3, for diabetes sample and for CKD sample 05). In addition, we also find the CLICKS and COMBD interaction term is positive and significant (for diabetes sample and for CKD sample 1.060, p < 0.05 ). We find similar results for the Cox proportional hazard models, i.e., CLI CKS, COMBD and the interaction term CLICKS COMBD are positive and significant for both diabetes and CKD sample s (see Column 3 and 7 of Table 6 ). We present the marginal effects of the Cox PH models in Columns 4, and 8 of Table 6 for interpretation purpos es. At the mean values of all variables, a ten percentage increase in the clicks is predicted to reduce the probability of closure by 1.24 percentage (see Column 4 of Table 6 ), whereas ten percentage increase in the comorbid alerts is predicted to reduce the closure by 0.11 percentage. As a combined effect, ten percentage increase in both clicks and comorbid alerts is predicted to reduce TCT probability by 1.5 percentage for diabetes patients. In the case of CKD patients, a ten percentage increase in the c licks is predicted to

PAGE 44

35 reduce the closure by 0.91 percentage, where ten percentage increase in the comorbid alerts is predicted to reduce the test closure by 0.09 percentage. Interpreting the complementary effect ten percent increase in both clicks and comorbid alerts is predicted to reduce the closure by 1.15 Percentage. We also present the survival analysis results using two main plots (Figures 4 and 5, Figures 6 and 7 ) and the hazard rate regression results in Table 6 As shown in Figure 4 and Figure 5 for the diabetes patients the hazard rate shows the probability of having the event (conducting A1C test) is going up for the 1st period to almost 1.4% then dropping after 60 days to reach about 0.03% after 80 days. So the chance of a patient to go and do the test increases for 60 days, then after that the chance of conducting the diagnostic test decreases with time. For CKD patients we observe an almost similar thing, the hazard rate shows the probability of having the even t (conducting GFR test) is going up for the first period of time to about 1.2% then dropping after 70 days to reach as low as 0.08% after 100 days. We also found that there is a somewhat a varying threshold of probability for the diabetes patients during t he 40 th to the 60th days, and for CKD patients it is from the 60 th to the 80 th days which we interpret as the ideal time for an intervention motivation for closing the alert (A1C in the case of diabetes and GFR in the case of CKD patients) is at the highest level in these periods, with some minor variations. If another reminder through a phone call, text, email or any other medium is given to the patient, then that will increase the likelihood of closing the test. Such wil l increase the efficiency of intervention and make it more effective. To aid in the interpretation of the comparison between the patients with comorbid and non comorbid conditions, we plotted the Kaplan Meier estimator of the survival function we

PAGE 45

36 take t he ratio of those without events over those at risk and multiply that over time, as the following equation: Where n j is the number of observations at risk and d j is the number of events. Figure 6 and Figure 7 sh ow the patients with comorbid alerts (red line) vs patients with no comorbid alerts (blue line). From the graph we can see that patients with comorbid alerts will be associated with a lower survival rate at any point in time. This means that if someone has comorbid alerts he / she is more likely to go and conduct the A1C test in less time comparing to someone with no comorbid alerts The Chi square test between the two groups (comorbid alerts vs no comorbid alerts) for the comparison bet ween survival functions shows a significant difference with the Chi Square = 9.73 and significant at p=0.002 level (see results in Table C3, Appendix C). In addition to these differences, t he Kaplan Meier survival estimate diagram also shows that the survi val curve starts from 1.00 at time period 0 and keep decreasing with each period of time (each day). After 90 days the curve approach es the probability of almost 40%, reflecting that the survival probabilities go down to 40% over 90 days. This means that 4 0% of the individual still would not have done that A1C after 90 days. This finding has practical implications for providers. They can infer from these results that the alerts should be renewed and fresh reminders should be given within 90 day period to reach to the 50% population who would not come to conduct the test within this interval. Discussion Findings

PAGE 46

37 The goal of this study was to explore how alerts through patient portal s influence a tim e liness in conducting the necessary tests. Specifically, we investigated the complementary effects of a number of clicks and number comorbid alerts recommended through a patient portal, at the time of closure of a test. We find that the numb er of clicks is associated with an early closure of the focal test for diabetes patients. For diabetes patients, a patient will close the (A1C) alert a day earlier with approximately 11 clicks while in the case of CKD a patient will close the (GFR) alert earlier with approximately 13 clicks. Th is finding suggests that alerts in patient portals serve as effective reminders for timely conduct of tests. The second set of findings of this study is that patients take less tim e to close the test alerts in the presence of comorbid alerts, the diabetes patients close the A1C alert on average of 1.5 days earlier than in the absence of Comorbid Alerts, where CKD patients close the GFR alert on average of 1 day earlier. Third, in t erms of the complementary effects, we found that diabetes patients close the alerts a day earlier with approximately 3 clicks in the presence of one additional comorbid alert, which is otherwise higher (e.g., 11 clicks), and CKD patients close the alerts a day earlier with approximately 5 clicks in the presence of one additional comorbid alert. The survival analysis results show that in presence comorbid alerts, ten percentage increase in both clicks and comorbid alerts is predicted to reduce TCT probability by 1.5 percentage for diabetes patients; whereas similar increase may reduce the closure by 1.15 Percentage. Further, the survival plots show that a high probability threshold span exists between 40 to 60 days for diabetes, and between 60 to 80 days for CKD alerts. Furthermore, the difference in the threshold spans may be earlier and lesser for patients with comorbid

PAGE 47

38 alerts than a patient without comorbid alerts, indicating the for comorbid patient the reminder should be earlier. Implications The implications of this research are three fold First, this study extends the existing practice discussion and motivation surrounding the impact of patient portals on patient care. The finding that each additional click on the alerts is associated with an early closure of the alerts has pract ical im plications. It suggests that patient portals act as a reminder system for the patients. Specifically, for patients who have an affinity to get engaged with their own health management, and seek to be empowered, the portals are highly effective in reducing care gaps and timely interventions. The findings of this study should motivate practitioners to enroll as many patients as possible in their portals and motivate them to use the portals to deliver timely diagnosis and care. Second, the study has implica tions towards the design of patient alert systems in a portal. While the practice of recommending multiple screening and diagnostic tests has been undergoing a debate in the current United States healthcare system due to cost implications (WSJ 2013) the findings of this study suggest that multiple test recommendation may be highly influential for timely intervention to curb diseases. Specifically, for diabetes and CKD, the symptoms at an early stage of the disease progression is not painful, and therefore patients do not take the diagnosis seriously nor turn out for required tests. However, when diabetes or CKD test alerts are recommended along with other more serious disease symptoms and alerts, such as heart disease or epilepsy, perhaps patients turn i n for more serious tests, and simultaneously conduct the required tests. In other words, even in the

PAGE 48

39 presence of cost concerns, multiple comorbid tests may be beneficial to patients and providers in saving costs and providing effective care in the long run A third implication of the study is that alerts may not be effective as a one shot recommendation. Renewal of alerts on a timely basis, more so within the high probability This study suggests that A1C alert repetition within 40 to 60 days of the first alert and GFR recommendation with 60 to 80 days of a first alert may increase the probability of more patients to conduct the tests. Finally, the study indi rectly suggests t hat patients weigh both compensatory and non compensatory considerations for health. In other words, both financial and psychological conditioning of patients before prescrib ing tests is an important consideration. Patients would not be able to close test alerts unless their economic concerns associated with conducting the tests are manageable. Further, often many patients carry a stigma with the diagnosis of a disease, which poses a psychological barrier to come and do the tests. In the presence of these c oncerns, how a provider can motivate the patients to take tests and possibly curb any future, potentially serious implications remain s a relatively untouched topic of discussion in the current practice of healthcare in the United States. While this study highlights the importance of timely and repeated alerts, but plausibly, providers may discuss both financial issues with patients, and make alternative arrangements, such as self tests, or low cost tests available for economically disadvantaged patients. Similarly, creating social or peer groups around patient care (e.g., web enabled social networks, such as patientslikeme.com) for some d iseases may remove the psychological stigma associated with disease diagnosis and management concerns.

PAGE 49

40 Contributions This study has three theoretical contributions. First, the study extends the process view of decision making to health care context. Influence of information technology on areas. Specifically, the time tests has not been explored in the existing literature that is an important contribution to this study. Second, the study takes the view that information technology effectiveness in health context may have an effect on the seriousness of a disease or health condition. As much a single disease affliction is concerning, but multiple disease conditions make situations worse. conditions, this study highlights that health IT design may be highly co nsequential for serious patients with multiple diseases, a view that was not explored in existing health IT literature. Finally, in applying the survival analysis in a health IT context, the study explores a span of intervention that is highly effective t o manage chronic diseases. The study suggests that by modifying alerts to be repetitive, one may influence some patients with chronic conditions to do a test. This inherently provides the view the health IT effects may not be constant with time, and such time variant effects may need to be explored more. Limitations and Suggestions for Future Research The study has some limitations which future research can address. First, the data were collected only for one provider. Although this helps to enhance internal validity of the study and control of many cross provider effects, it may limit the generalizability of the study. Future research can extend the analysis by incorporating studies that may include other providers. Similarly, this study is conducte d considering two focal diseases that future

PAGE 50

41 studies may replicate for other diseases in patient portals to consider generalizability of the findings. Second, although we theorize the economic and psychological concerns associated with the alert closure decision process, and suggest a model to elaborate on the steps involved in the process, we do not empirically test the theoretical mechanisms underlying the phenomena being studied Future research can empirically test such phase mediating mechanisms pote ntially using alternate methods. Third, due to the cross sectional nature of the study, similar to many other studies that use cross sectional designs, the results may be interpreted more associational than establishing any causal linkages. However, the time variant survival analysis may mitigate some of these limitations. Future research can adopt additional empirical strategies that can use longitudinal data and designs for similar studies. Conclusion The goal of this study was to explore how alerts i n a patient portal are effective. Specifically, we investigated the complementary effects of number of clicks, and the nature of the multiple alerts recommended through a patient portal, at the time of closure of a focal alert. We argued that with each additional click of the alert, a patient would have a higher concern and motivation to conduct the test and close the alert as soon as possible. Furthermore, we suggested with multiple alert s with comorbid conditions, the patient will have a higher concerning effect to lead to an early closure of the alert. We contextualize this study to two samples of diabetes and CKD patients, and empirically examined and found support for the hypothesized effects. In addition we use survival analysis to explore the time based effect of alert clicks on test closures and establish threshold spans periods for the be repeated between 40 to 60

PAGE 51

42 days fo r diabetes, and between 60 to 80 days for CKD. The study provides implications for the effectiveness of patient portals and suggests the repeated alert design aspects as a recommendation to motivate patients to conduct times tests. Further, we contribute t o the existing information systems and healthcare research streams of research in exploring the impact of patient portals on reducing patient care gaps and highlight the importance of patient portal and alert systems as emerging patient centered health inf ormation technologies.

PAGE 52

43 Figures and Tables Figure 1 : Conceptual Model Figure 2 : Interaction Plot Diabetes Sample Figure 3 : Interaction Plot for CKD Sample Figure 4 : The Smoothed Hazard Estimation Plot Diabetes Sample Figure 5 : The Smoothed Hazard Estimation Plot CKD Sample

PAGE 53

44 Figure 6 : The K M Plot Showing with Comorbid vs. Non Comorbid Diabetes Patients Figure 7 : The K M Plot Showing with Comorbid vs. Non Comorbid CKD Patients

PAGE 54

45 Figure 8 : Diabetes S ampling P rocess

PAGE 55

46 Figure 9 : CKD Sampling Process Exclusion criteria (N = 39,652) 51,343 p atients were not given any test alerts. Exclude 17 patients who did not have complete records or had some missing data 3 201 were not given any CKD related alerts 3,124 Patients had not closed the alert 5,349 Patients had closed GFR alert 11,691 Patients 8,473 Patients were given the CKD related alerts CKD registry had 51,343 active patients ( N = 51,343 ) 8,473 Patients used for Survival analysis

PAGE 56

47 Figure 10 : Alert Closure Flow Chart

PAGE 57

48 Figure 11 : The Alert Page of Portal Showing No Alerts Figure 12 Figure 13 : An Alert Page Showing a Flu Vaccine Alert

PAGE 58

49 Figure 14 : An Alert Page Showing Multiple Alerts along with A1C Test Alert Figure 15 : Multiple Peripheral Alerts

PAGE 59

50 Table 1 : Descriptive Statistics Diabetes Sample CKD Sample Obs. Mean S.D. Min Max Obs. Mean S.D. Min Max 1 TCT 2,164 37.80 22.92 1 88 8,473 55.50 18.21 1 105 2 CLICKS 2,164 14.53 9.59 3 60 8,473 18.1 8.66 1 71 3 COMBD 2,164 0.22 0.86 0 4 8,473 0.67 0.67 0 3 4 CONT 2,164 0.20 0.78 0 3 8,473 0.60 0.55 0 3 5 LACE 2,164 8.63 4.03 0 19 8,473 8.63 4.03 0 19 6 LOS 2,164 2.20 1.01 0 4 8,473 0.87 0.67 0 4 7 ACUITY 2,164 1.60 0.76 0 3 8,473 1.60 0.76 0 3 9 AGE 2,164 64.56 12.15 19 87 8,473 60.1 10.45 19 85 10 GENDER 2,164 0.52 0.49 0 1 8,473 0.51 0.49 0 1 11 MEDICARE 2,164 0.47 0.50 0 1 8,473 0.49 0.50 0 1 12 MEDICAID 2,164 0.01 0.06 0 1 8,473 0.02 0.05 0 1 13 REC_ADM 2,164 0.09 0.01 0 1 8,473 0.09 0.01 0 1 15 HMO 2,164 0.85 0.10 0 1 8,473 0.90 0.13 0 1 16 PPO 2,164 0.09 0.05 0 1 8,473 0.07 0.06 0 1 17 POS 2,164 0.06 0.05 0 1 8,473 0.03 0.02 0 1

PAGE 60

51 Table 2 : Correlation amongst Variables Diabetes Sample 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 TCT 1 2 CLICKS 0.14 1 3 COMBD 0.08 0.32 1 4 CONT 0.02 0.06 0.43* 1 5 LACE 0.02 0.01 0.02 0.02 1 6 LOS 0.01 0.00 0.00 0.01 0.68* 1 7 ACUITY 0.01 0.01 0.02 0.01 0.84 0.57* 1 8 AGE 0.01 0.04 0.05 0.02 0.02 0.02 0.02 1 9 GENDER 0.00 0.04 0.03 0.05 0.01 0.01 0.01 0.04 1 10 MEDICARE 0.02 0.06 0.04 0.01 0.00 0.01 0.01 0.45* 0.00 1 11 MEDICAID 0.01 0.01 0.02 0.01 0.04 0.02 0.04 0.09 0.04 0. 67* 1 12 REC_ADM 0.04 0.04 0.08 0.02 0.01 0.03 0.00 0.01 0.05 0.09 0.01 1 13 HMO 0.00 0.07 0.08 0.07 0.04 0.04 0.08 0.01 0.03 0.00 0.00 0.09 1 14 PPO 0.09 0.06 0.06 0.05 0.03 0.07 0.01 0.07 0.03 0.02 0.04 0.06 0.01 1 15 POS 0.11 0.01 0.04 0.04 0.09 0.01 0.04 0.08 0.02 0.01 0.03 0.01 0.09 0.08 1 CKD Sample 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 TCT 1 2 CLICKS 0.21 1 3 COMBD 0.11 0.35* 1 4 CONT 0.01 0.08 0.51 1 5 LACE 0.10 0.01 0.02 0.02 1 6 LOS 0.01 0.00 0.00 0.01 0.45* 1 7 ACUITY 0.01 0.01 0.02 0.01 0.60* 0.34 1 8 AGE 0.01 0.04 0.05 0.02 0.02 0.02 0.02 1 9 GENDER 0.00 0.04 0.03 0.05 0.01 0.01 0.01 0.04 1 10 MEDICARE 0.01 0.06 0.04 0.01 0.00 0.01 0.01 0.60* 0.00 1 11 MEDICAID 0.01 0.01 0.01 0.01 0.01 0.02 0.03 0.00 0.03 0.74* 1 12 REC_ADM 0.00 0.09 0.00 0.00 0.02 0.01 0.03 0.01 0.05 0.09 0.01 1 13 HMO 0.00 0.07 0.08 0.07 0.05 0.09 0.02 0.19 0.09 0.04 0.07 0.04 1 14 PPO 0.09 0.00 0.06 0.05 0.03 0.07 0.01 0.07 0.03 0.02 0.04 0.06 0.01 1 15 POS 0.11 0.01 0.04 0.04 0.09 0.01 0.04 0.05 0.09 0.02 0.05 0.09 0.07 0.08 1 indicates significance at 1% level

PAGE 61

52 Table 3 : T Test of Variables by Alert Closed for Diabetes and CKD Samples Diabetes Sample (68.85% Closed Alerts) Patients who closed alerts Patients who did no t close alerts N Mean SD N Mean SD P value Clicks 1,490 14.71 7.53 680 14.01 8.37 0.93 a COMBD 1,490 0.21 0.09 680 0.19 0.11 0.57 a CONT 1,490 0.21 0.11 680 0.20 0.10 0.74 a LACE 1,490 8.21 5.89 680 8.58 5.70 0.29 a LOS 1,490 2.21 1.22 680 2.16 1.19 0.52 a ACUITY 1,490 1.53 1.15 680 1.56 1.20 0.74 a AGE 1,490 66.04 16.06 680 63.97 16.40 0.15 a MEDICARE 1,490 0.45 0.40 680 0.48 0.41 0.41 a MEDICAID 1,490 0.01 0.00 680 0.01 000 0.89 a REC_ADM 1,490 0.10 0.06 680 0.09 0.07 0.79 a HMO 1,490 0.88 0.32 680 0.86 0.35 0.20 a PPO 1,490 0.08 0.01 680 0.09 0.01 0.18 a POS 1,490 0.06 0.02 680 0.07 0.03 0.71 a Male gender (%) 775 52% 354 53% 0.92 b a test; b Chi square values CKD Sample (63.13% Closed Alerts) Patients who closed alerts Patients who close alerts N Mean SD N Mean SD P value Clicks 5,349 18.35 5.12 3,124 17.73 9.64 0.52 a COMBD 5,349 0.68 0.40 3,124 0.66 0.39 0.20 a CONT 5,349 0.62 0.50 3,124 0.68 0.56 0.21 a LACE 5,349 8.85 4.57 3,124 8.31 4.43 0.35 a LOS 5,349 1.32 0.67 3,124 1.33 1.55 0.89 a ACUITY 5,349 1.63 1.25 3,124 1.66 1.05 0.55 a AGE 5,349 61.52 20.43 3,124 56.39 20.36 0.93 a MEDICARE 5,349 0.45 0.41 3,124 0.47 0.39 0.43 a MEDICAID 5,349 0.02 0.1 3,124 0.01 0.00 0.15 a REC_ADM 5,349 0.10 0.09 3,124 0.11 0.08 0.85 a HMO 5,349 0.87 0.49 3,124 0.93 0.43 0.20 a PPO 5,349 0.08 0.05 3,124 0.06 0.03 0.18 a POS 5,349 0.03 0.01 3,124 0.03 0.01 0.15 a Male gender (%) 5,349 51% 3,124 51% 0.92 b a test; b Chi square values

PAGE 62

53 Table 4 : Linear Estimation Models Diabetes CKD VARIABLES 1 2 3 4 OLS OLS OLS OLS Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Number of Alert Clicks (CLICKS) 0.090*** (0.040) 0.011*** (0.004) 0.079*** (0.002) 0.013*** (0.005) Number of Comorbid Alerts (COMBD) 1.484** (0.770) 1.328*** (0.046) 1.189*** (0.046) 0.998** (0.009) CLICKS COMBD 0.321*** (0.020) 0.191** (0.003) Number of Contiguous Alerts (CONT) 0.131 (0.097) 0.297 (0.161) 1.660 (0.161) 0.025 (0.097) Observations 2,164 2,164 8,473 8,473 R squared 0.424 0.433 0.011 0.011 Chi squared 35.65*** 25.21*** 44.65*** 23.5*** F test 19.78*** 22.24*** Note: 1. *** p<0.01, ** p<0.05, p<0.1 2. Robust standard errors in parentheses 3. Standard errors are adjusted for ethno nationalism groups: 60 clusters in diabetes sample, and for 54 clusters in CKD sample. 4. Models include all control variables. 5. Control variables are not significant except Medicaid, detailed results in Table 17

PAGE 63

54 Table 5 : Parametric and Semi Parametric Estimation Models Diabetes CKD 1 2 3 4 5 6 7 8 Weibull regression Cox proportional Weibull regression Cox proportional Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) VARIABLES Coefficients Marginal effect Hazard rates Marginal effect Coefficients Marginal effect Hazard rates Marginal effect Number of Alert Clicks (CLICKS) 1.247*** (0.137) 0.543*** (0.245) 0.237*** (0.030) 0.124*** (0.053) 1.101** (0.061) 0.493** (0.231) 0.201** (0.045) 0.091** (0.031) Number of Comorbid Alerts (COMBD) 1.152** (0.166) 0.631*** (0.321) 0.021*** (0.057) 0.011*** (0.004) 1.141** (0.054) 0.701** (0.321) 0.019*** (0.089) 0.009*** (0.003) CLICKS COMBD 1.332** (0.145) 0.691*** (0.351) 0.032*** 0.011 0.021*** (0.013) 1.060** (0.054) 0.517** (0.219) 0.023** (0.045) 0.015** (0.006) Note: 1. *** p<0.01, ** p<0.05, p<0.1 2. Robust standard errors in parentheses 3. Standard errors are adjusted for ethno nationalism groups: 60 clusters in diabetes sample, and for 54 clusters in CKD sample. 4. Models include all control variables. 5. Control variables are not significant except Medicaid, detailed results in Appendix C, Table C2 6. The marginal effects are evaluated at the sample mean. 7. The marginal effects for the dummy variable are not reported

PAGE 64

55 Table 6 : Description of Variables Variable Definition Test Closure Time (TCT) Number of days taken to close the focal test alert. For diabetes, the focal alert is glycated hemoglobin test (A1C) For CKD, the focal alert is glomerular filtration rate test (GFR) Number of Alert Clicks (CLICKS) Number of times the patient clicked and saw the focal alert in the patient portal landing page prior to closure. Number of Comorbid Alerts ( COMBD ) Total number of Comor bid Alerts (see list in Table 13 an d 14 ). Number of Contiguous Alerts (CONT) Number of t est alert related to the focal alert. Diabetes: (e.g., Urine or Microalbumin tests for diabetes along with A1C), for the same disease diagnosis or treatment. CKD: (e.g., Creatinine or Lipid Profile along with GFR), for the same disease diagnosis or treat ment. GENDER Gender of the patient, Female=0, Male=1. AGE Age of the patient in years MEDICARE Dummy for Medicare patients MEDICAID Dummy for Medicaid patients ETHN ethno nationalism LACE LACE Index scores for every patient on admission and discharge on the following parameters: length of stay, acuity of the admission, comorbid diseases, and emergency department visits in the previous six months REC_ADM Whether the patient was recently adm itted to hospital, within last one month LOS The l ength of stay; the duration of a single episode of the last hospitalization ACUITY The acuity of the patient at the point of admission during the last visit or stay in a hospital or clinic. ZIP Zip c HMO Dummy for (Health Maintenance Organization) HMO insurance patients, 1= yes, 0=no. PPO Dummy for (Preferred Provider Organization) PPO insurance POS Dummy for Point of Service health (POS) plans insurance patients, 1= yes, 0=no.

PAGE 65

56 CHAPTER III THE ROLE OF EFFECTIVENESS, EASE OF USE, AND FUNCTIONALITY ON EVALUATION OF HEALTH APPS Abstract Mobile health applications (apps) are a growing trend in healthcare With so many options available and limited information regarding app performance patients caregivers and healthcare professionals may be faced with the challenge to determin e how effective health apps are. One way to assess the effective ness of an app is by assessing the functionalities it offers Most of the health apps available today offer just one or two functionalities. On the other hand, with the advancement in technology and the emergence of fierce competition between app developers we are seeing an increase in the number of apps that offer multiple function ali t ies This research tracks a set of 185 health apps in the Android marketplace for a period of 14 weeks W e find that apps that offer instructive and integrative qualities are mo re effici e nt in providing healthcare services to patients; such positively impact s how patients evaluate the app However despite providing good qualities, offering too many functionalities negatively impact s the app ease of use ; s uch, negatively impact ing how patients evaluate the health app In the following we discuss the managerial and research contributions of the se findings Keywords : healthcare apps mobile health applications, apps effectiveness, app functionalities Introduction Health apps are programmed applications that run on smartphones and tablets to well being (Phillips et al. 2010) There are many types of health apps S ome are designed to help the p atient manage his or her

PAGE 66

57 health care treatments ; others help patients to live a healthier lifest yle through tracking diet and exercise patterns, still others offer the ability for patients to communicate with medical providers and book appointments (Boulos et al. 2014; Fox and Duggan 2012) Recently some apps have begun to offer integrat ion with electronic medical records and personal health records. This allow s patients to access their health records and enable providers to monitor patients and record their progress (Lobelo et al. 2016) T here are more than forty thousand mobile healthcare related apps available to users o n the iTunes platform, with almost equal numbers on the Android platform (Aitken and Gauntlett 2013) It is predicted that the number of users downloading h ealth related apps will climb to 1.7 billion by 2017 (Economist 2016) with a global revenue potential for mobile he alth app based business of $21.5 billion by 2018 (BCC 2014) This degree of growth raises issues about the safety and efficacy of mobile health apps and suggests the need for empirical research to make determinations about such issues. The category of health apps most downloaded and used are fitness and nutrition, Krebs and Duncan (2015) found among their respondents most used the apps at least daily. The researchers found the most common reasons from respondents who did not download an app to be a lack of interest, cost, and concern about apps collecting their personal data. Despite these concerns, the research found the users most likely to consistently use a health app were younger, of high socioeconomic status, educated, be Latino/Hispanic and have a body mass index (BMI) in the obese range. Despite the popularity of health apps, some recent research indicates that they are not yet perceived as reliable mode l s of h ealthcare service delivery. According to Dehzad et al (2014) t here is a lack of interest on the part of both patients and doctors to embrace apps due

PAGE 67

58 to poor integration and concern over security issues (Dehzad et al. 2014) Luna et al (2014) assert that wider acceptance of mobi le health apps will require an overhaul of the current health care IT structure, which in turn will require investments in both public and private sectors geared towards increased integration of internal and external hardware and software systems. Without i ncreased integration, mobile health apps will simply turn into information silos (Luna et al. 2014) With that, the successful development and deployment of mobile health apps will require continued investment and research. It will be necessary to consider both their success in helping patients achieve better outcomes and their integration and ease of use for both patients and healthcare providers Al though there are thousands of health apps available in the digital market place only 600 are commonly used Statista reports that 15% of the downloaded health apps are used only once during the first six months of ownership. Health apps technology can impact patient care positively if they provide effective functionalities (Free et al. 2013a) Many health apps offer instructive functionalities such as health information, calorie counters exercise instructions, drug instructions and reminders. Instructive functions enabl e patients to self manage medical conditions and help patients follow medical and health instructions Instructive functionalities are passive and unidirectional. N ew health apps are expanding functionality by offer ing integrative functionalities that enabl e apps to integrate with backend applications like electronic medical records and personal health records Health apps can enhance efficiency by reducing the amount of direct interaction with healthcare profess io nals (Aitken and Gauntlett 2013) However, to increase the market for these apps developers often add these functionalities before they are mature and fully developed, impacting the apps effectiveness. Functions that are n ot effective may

PAGE 68

59 reduce the effici enc y and lead to a negative impression and lower the potential for continual use by users (Tuch et al. 2012; Wu et al. 2011) The lack of integration and interoperability or app performance, with electronic health records and backend systems are posing to be significant barriers to widespread mobile health usage and adoption (Dehzad et al. 2014) The performance of apps is reflected in the ratings and reviews by the patients after app use (Irick 2008) Ratings an d reviews impact the subsequent adoption and success of an applicati on in the digital marketplace (Pagano and Maalej 2013) Apps that are poorly rated will eventually die off, while apps that have good ratings and reviews will have a better chance of attracting new users and receive a larger share of the m arket (Fu et al. 2013; Pagano and Maalej 2013) Thus, u nderstanding how functionality designs and integration impact the evaluation of health apps remains important to both practice and research. The current study tries to address this gap. Prior Literature Previous research in health apps focused on an app content rather t han user reviews and comments (Fu et al. 2013) Analyzing reviews and comments help us to understand the relationship between users and apps (Finkelstein et al. 2014) Finkelstein et al. (2014) used dat a mining to investigate the impact of reviews on app downloads and found a positive relationship between the number of positive reviews and downloads. The researchers also found a correlation between app ratings and positive reviews. There have been severa l studies that examined reviews in the different areas like movies (Joshi et al. 2010) restaurants (Chahuneau et al. 2012) and retailers (Archak et al. 2011) On the other hand, there is little work in mining the digital health app market (Finkelstein et al. 2014; Fu et al. 2013)

PAGE 69

60 Prior research found that h ealth apps have the ability to increase the efficiency of healthcare service delivery and enable patients to interact remotely with healthcare professionals (Kumar et al. 2013) In addition h ealth apps can support in adopt ing a healthy lifestyle and manag ing chronic conditions (Free et al. 2013b) T hey accomplish this by enabl ing patient s to self monitor their medical conditions, diet and exercise targets, and other information (Ramanathan et al. 2013) Researchers have argued that apps can provide healthcare professionals with additional channels for intervention (Klasnja and Pratt 2012; Terry 2010) Today we can find many healthcare providers leverag ing health apps for the treatment of chronic conditions and patient follow up (Free et al. 2013b; Kumar et al. 2013; Milward et al. 2015) A variety of research on mobile health apps is available in the extant literature, though little research focuses al (20 15) surveyed patients to determine what features they desire in mobile health apps ; their research was aimed to understand what drives patients to download an app rather than their experience with the app Velsen Beaujean, and Genert Pinjen (2013) state that users are confused and overwhelmed by the abundance of choices and functionalities available in mobile health apps. Their study focused primarily on the type of app features users looked for when making a decision to download the app rather than on user satisfaction with those specific features. In a review of patient responses to and satisfaction with mobile health apps, Zapata et al. (2015) found that users are commonly provided with specific apps for research studies and not given a cho ice. T his type of research can help identify desirable app features, but it says little about and using apps

PAGE 70

61 The recent research reveal s that health apps are often fragmented into information silos and su ffer from poor integration with other apps or health information systems like electronic medical records (EMRs). A study of app design across multiple platforms suggested that apps often feature redundant features in closed proprietary systems that integra te poorly with other information systems (Paschou et al. 2013a; Paschou et al. 2013b) Wilhide, Peeples, and Kouyate (2016) development that integrate s apps with electronic medical records. Fox et al (2012) argue that app developers must embrace open source technology and collaboration as a means of promoting wider use of apps. S ocial influenc e is critical in the adoption of technology and the usage of digital services ( Oh et al. 2015) (Cheung and Thadani 2012; Gupta and Harris 2010) Information Systems s cholars ar gue that the decision to download a certain mobile app from the digital market place is influenced by the existing ratings and evaluations posted by other users (Huang and Korfiatis 2015; Senecal et al. 2005) Although the mobile application rating s and evaluation s are subjective in nature, the existence of a large number of ratings and evaluations that are similar in tone increase s their perceived objecti vity (Flanagin e t al. 2014) Health apps ratings and reviews are becoming incre asingly important for healthcare providers to differentiate their apps as the industry continues to grow (Spriensma 2012) In many studies IS scholars use user evaluation as surrogate measure s of performance (Irick 2008) ; do ing such highlights the importance of achieving positive user evaluations and Health IT scholars argue that technology that improves heal th outcomes is po sitively associated with patient satisfaction and evaluation (Delpierre et al. 2004; Jamal et al. 2009 )

PAGE 71

62 In the context of mobile apps, Huang and Korfiatis (2015) ev a luation of an app. for users to compare a particular app to a competitor or a substitu t e Arnhold et al. (2014) found that very little research exists highlighting formative usability evaluations of health apps. In their research of diabetes applications, they were only able to find a single published article that connects app review with a formative usability evaluation Users are flooded with application options once they open the App Store, evaluations and reviews help a user to features However, research is suggesting that users are still unaware of many apps th at may contain specific feature s t hey desire (Krebs and Duncan 2015) Krebs and Duncan (2015) propose a way to mediate this process for users would be for a refereed clearinghouse that could simplify the help consumers understand the features and available apps. Theoretical Framework Task and Technology Fit and Information System Success Task and Technology Fit (TTF) is a theoretical framework that is widely used within the field of i nformation s ystems and has been applied to explain the different phenomena underlying t he development and the usage of i nformation t echnology (Tsiknakis and Kouroubali 2009 ) TTF was initially dev eloped by Goodhue and Thompson (1995) later it was elab orated on by the researchers Zigurs and Buckland (1998) and Mathieson and Keil (1998) TTF theory addresses the importance of matching i nformation s ystems functionality with the desired task (Lin 2014)

PAGE 72

63 The theory of TTF emphasize s on how technology can be developed to support the user and enable completion of an underlying task (Lin 2014) When an individual uses technology to perform a given task, TTF predicts that if the task and technology are aligned, the user will be more effective at achieving his or her goal. Consequently, the user will form a positive opinion towards that technology and tend to use it more often (Liu et al. 2011) In addition to the technology and task fit, the TTF theory takes into consideration the complexity of the task The theory examines how the interaction of technology and t ask performance under a different level of complexity (Ammenwerth et al. 2006) Since the creation of the TTF model, it has been applied to a variety of contexts such as e books adoption (D'Ambra et al. 2013) personal travel (D'Ambra and Wilson 2004) e tourism (Usoro et al. 2010), and many others In order to use TTF to assess an information systems solution, the solution's value primarily depend s on the task and the outcomes from using the technology In addition the user has to bear independently the responsibility of evalua ting the technologies employed (Mathieson and Keil 1998; Zigurs and Bucklan d 1998) The F it according to TTF, illustrates how mediation operates as an intervention when one variable modulates one or more variabl es and is a key concept in the theory T his concept posits that matching technology capabilities and tasks character istics significantly affe (Tsiknakis and Kouroubali 2009) Perceived Fit on the other hand, relative to the extent that a technology solution appropriately meets targeted task requirements. As such, it is a construct that serves to bridge th e relationships between the targete d task, technological features, performance and outcomes Within the larger theoretical models of behavioral attitude and user perceptions

PAGE 73

64 and in the context of health apps the positive outcome of us ing an app to achieve a certain health goal can be viewed as the product of technological functionality satisfactorily meeting (Ammenwerth et al. 2006) Ammenwerth, Iller and Mahler (2006) suggest that, in instances where the user anticipates that technology will offer the desired functionality required to p erform needed tasks, then the user will progressively us e the said ap p. This position implies that positive perceptions of enhanced fit between the task and the technology will ultimately increase the chance that users will continue to employ the technology to satis fy task requirements (Tsik nakis and Kouroubali 2009) Hence, the overall outcome will be improved app usage as well as better performance of the tasks that the technology is designed and expected to execute. It is difficult to directly measure how well an information system tech nology can improve outcomes. T his difficulty becomes the prevailing motivation for the decision by the majority of information system practitioners and researchers to rely on proxy measures to assess information systems success (Gebauer 2008) In the context of information systems applications, u ser evaluation s are described as the qualitative or quantitative assessment user s provide as feedback during the course of using a particular application (Amatriain et al. 2009) Indeed, user evaluations are the most commonly deployed forms of proxy measures (Forman et al. 200 8) Another new proxy measure for information systems apps specifically is the app rank in the digital marketplace (Lee and Raghu 2014) which are directed by the user evaluations. Task Technology Fit as a theory is related to the Information Systems Success (ISS) framework conceived by DeLone and McLean (1992) Although both models assess the

PAGE 74

65 opinion regarding an identified information system and the resultant influences modulating the individual capacity, the ISS framework provides a comprehensive understanding of the information systems success On the same note, impact on performance can be applied as a proxy for information systems s uccess. I mpact on performance can be translatable to enhanced effectiveness, efficiency or the quality of o utcomes. Integrating specific features of each of the two models or theories and focusing on their broader implications for digital device borne te chnologies can yield further revelations as to how TTF and ISS play similar and mutual roles in improving user outcomes and, eventually, technology adoption (Ammenwerth et al. 2006) De L one and McLean (1992) developed the original comprehensive and multidimensional IS Success Model as a response to the need to simplify and provide a means to conduct result comparison and knowledge accumulation around the different aspects of IS success. The rese archers built the foundation of this model from the wealth of research around the aspects of IS Success, communication and information influence. This model consists of six dimensions that impact three levels of IS Success. The six dimensions are system and information qualit ies use, satisfaction, and individual and organization impact s ; the three levels are technical, semantic and effectiveness. In the 2003 model update, DeLone and McLean (2003) intended to respond to their original model criticism by expanding its scope and clarify its terminology, to be able to be use d more univ ersally. Pitt et al. (1995) highlighted that DeL model did not take into consideration the r ole of the IS department or IS personnel in working with the users on problems like installation and product education. Further discussion on the model asserted that the impacts of IS could affect entities beyond the user

PAGE 75

66 and organization. DeLone and McLea n updated the IS success model to more clearly define terminology and make it more comprehensive. These changes are identified by the inclusion of V individual impact and Although both the T T identified information system and the resultant influences modulating the individual capacity, the ISS framework provides a comprehensiv e understanding of the IS success. On the same note, the model accommodates using impact on performance and user evaluation as a proxy for IS success. Integrating specific features of each of the two models and focusing on their broader implications for ap plications and technologies can yield further revelations as to how TTF and ISS play similar and mutual roles in improving user outco mes and performance (Ammenwerth et al. 2006) Task and Technology Fit in Health Apps It has been established in the IS liter ature that the TTF is a theoretical framework that examines the issues associated with the fit between technology, tasks, and how that impact performance (Ammenwerth et al. 2006; Delone and Mclean 2004b) The notion of fit in the TTF is applicable when discussing technology that allows users to perform tasks that ofte n are completed in a traditional way (D'Ambra et al. 2013) like making a phone call to schedule an appointment or ask for a refill, or use paper diary to log calories consumption or sugar level. Inves tigati ng the intersection of the artifact functionalities for the appropriate task s an individual want to perform is measured by the fit. TTF does no t guarantee the usage of the artifact but it helps to predict the utilization and the change of perception regarding it Such is

PAGE 76

67 applicable in the health apps context. When an app provides a user with functionalities that enable h im to perform desired tasks, the TTF predicts that if the task and technology is a good fit then the user will utilize the technology more and his or her perception will be enhanced. A review of the literature shows that TTF explore s the relationship betw een task and technology from different standpoints. For example, user perception of the technology, improved utilization, and enhanced performance (Yi et al. 2016; Yu and Yu 2010) Also, the literature shows that applying the TTF outside the organization context is still under researched TTF can provide valuable insights for technology design and development ( D'Ambra et al. 2013) Such makes make TTF a n excellen t framework to investigate health apps functionalities and their impact on the user evaluation. Our research addresses this gap and tend to apply the TTF in context of health apps. Tasks are group of actions that an individual perform to achieve a certain goal. In our research we refer to tasks by functionalities. In our research functionalities and tasks are synonymous. An individual will download a health app that is equipped with one or more functionalities to perform one more tasks. Assimilation and C ontrast E ffects Assimilation and contrast effects reflect a positive or negative correlation between a i nitial evaluative j udgements and contextual information. An assimilation effect occurs when the initial judgement and context are positively correlated, a contrast effect is when the two are negatively correlated (Johnson and Fornell 1991; Walker 1994) The theory explains how evaluative judgment is based on a mental process that is a culmination of

PAGE 77

68 assimilation and contrast processes. These two processes are not mutually exclusive; they may happen separately from each other or overlap based on the context and the situation. Users often depend on some attributes of an object to make an overall evaluat ion (Frster et al. 2008; Hovland et al. 1957; Stapel et al. 1998) The assimilation effect incorporates a new experience into an existing one to lower dissonance (Frster et al. 2008; Stewart and Malaga 2009) The assimilation occurs when the target shares common features and attributes with existing or previous experience (Frster et al. 2008; Stewart and Malaga 2009) As opposed to that, the contrast occurs when the experience falls outside the user knowledge or previous experience. The incorporation of the new experience and information can be uncomfortable to the individual. Generally speaking, the assimilation and contrast effects are silent processes, and it is hard to identify their direction, but they are often reflected in the reviews, ratings, and at titudes towar ds products or services (Oliver and Burke 1999) Assimilation effect is positive in nature where the contr ast eff ect is negative (Stewart and Malaga 2009) Broadly, it is apparent that developers need to stimulate more assimilation effect than contrast effect in their apps design. Such will help in increasing the positive attitude and gain good evaluations from users. Conceptual Model and Hypotheses D evelopment W e suggest a conceptual model for this study ; our model is presented in Figure 1 6 The research model leverage s the TTF and ISS theories with the a ssimilation and contrast effects to better understand how users evaluate mobile health apps. The previous literature shows that many ISS dimensions are applicable across several industries, whereas others are more or less specific to a particular service industry In this study, we are focusing on two dimensions of the ISS model, instruction quali ty and

PAGE 78

69 integrative quality. However, as stated by DeLone and McLean (2003) that while all aspects of the ISS represent different pieces to the ISS model, they are all interrel ated and interdependent. Each affecting the overall success of the information system. The researchers explain that the construct information quality reflects both the content and quality of information provided by an information system. As in the previous section DeLone and updated model combine s individual and This allows for researcher s to apply the model to a wider variety of contexts In our study we ation. In studies following the DeLone and McLean (2003) updated model, many researchers utilized this opportunity to apply this model to a variety of research contexts withi n IS. For example, Lin (2008) on a study on the success of virtual communities, assert that the mo st accurate indicator of effectiveness is loyalty, defining for their study the net benefit as loyalty. They found information quality and system quality to affect satisfaction i mpacting user loyalty or as we are defining it effectiveness. On the other hand, the TTF model describes that when the health app provides qualities and features aligned with the patient needs, then the patient will be more effective at achieving his or h er goals or tasks. However, unlike the ISS model, TTF outcomes are independent of one another. That is to say that while there may show an increase in utilization, user performance effectiveness may not show an increase. This demonstrates to health care providers the need to focus on instruction quality to garner both increased utilization and performance effectiveness in an app. In our model we work to capture the principle concept of the Technology Task Fit model in the construct app effectiveness

PAGE 79

70 We argue health apps that provide patients with instructions to confront their health conditions and in some cases information on how to cope with diseases will be more effective in managing medical and health conditions leading to higher app effectiveness Thus, we hypothesize: HYPOTHESIS 1 (H1): Instruction Quality will have a positive impact on app effectiveness. In the ISS model, DeLone and McLean (2003; 2004a) note that system quality captures desired technical characteristics of an information system. The researchers suggest that system quality can be defined by several sub dimensions In our study system quality is measured by Integrative Quality and Functionalities. Integrative Quality in our study is measured workflow An information system can be descri bed as effective if it can achieve its objectives, of these, its ability to integrate. The dominant number of current health apps have limited functionality and focus the user content to providing information and instruction. In their research, Aitken and Gauntlett (2013) were able to find only a limited number of apps that include more diverse functionalities to gather user data and send it back to the physicians and healthcar e professionals. Due to the newness of this technology and the limited use, there is little research on its effectiveness and usability. However, there is a rich abundance of literature on the perceived ease of use of health IT and apps using frameworks ba sed on behavior theory in Psychology. Many of those studies pay specific attention to Self Determination Theory (SDT). There are three main psychological needs outlined in SDT, (1) autonomy, (2) competence and (3) relatedness. The

PAGE 80

71 studies above take these needs and examine what features should be included in mHealth apps to facilitate user satisfaction based on SDT. One specific aspect, relatedness, is the degree of connectedness that the person feels with others, this connectedness can be created within an mHealth app by connecting users to blogs, discussion boards and communities that share the same diagnosis (Mendiola et al. 2015) With the continual surge in app use and mobile health, we anticipate seeing the value and importance of data transmission of health vitals to physicians and healthcare providers gain more momentum and utilization in future apps (Arnhold et al. 2014) Based on that we hypothesize: HYPOTHESIS 2 (H2): Integrative Q uality will have a positive impact on app effectiveness. It is important to note that only recently mobile smart devices are capable of handling more functionalit y. That allowed healthcare app developers to create multifunctional apps (Arnhold et al. 2014) Mendiola et al. (2015) conducted a study evaluating the impact of different app functions on user experience, with a specific focus on diabetes apps. The researchers found that although there were many apps available, few distinguished themselves from others by offering more than one or two functionalities. In a comparison of basic and more complex apps, Mendiola et al. (2015) noted that the more complex apps did not rate as h igh on their user evaluations as did the more basic apps. Arnhold et al. (2014) came to the same conclusion in their study. In addition they found th at the multifunctional in terms of ease of use was markedly worse. The decline in evaluations due to the increased number of features has been studied by product development scholars and been defined as feature fatigue.

PAGE 81

72 Feature fatigue occ urs when a user becomes overwhelmed with the complexity of a product or application and consequently chooses to quit using it (Rust et al. 2006) Research in product strategy show s that developers add more features to products to keep up with competition (Ellison 2003; Gottfredson and Aspinall 2005) Users will be more attracted to apps that offers many different features (Thompson et al. 2005) Once the user begins to use the multifunctional app they tend to become dissatisfied with the complexity of the app (Thompson et al. 2005; Wu et al. 2015b) To alleviate feature fatigue researchers have proposed making a simpler app that does fewer things really well (Wu et al. 2015a) From the previous arguments, the number of functionalities in a health app may potentially increase feature fatigue and thus negatively impact the ease of use. Based on that, we hypothesize: H YPOTHESIS 3 (H 3 ): The increase in the number of App functionalities will negatively impact the App ease of use. When individuals use a health app to obtain medical information, diagnosis of a condition or manage a condition, they will either judge the app based on the accuracy of the information, the accuracy of the diagnosis they provide, or the efficiency of the app in managing their condition. If the information or diagnosis attained is not accurate or the function is not efficient in mana ging their healthcare the result will be a negative evaluation (Metzger and Flanagi n 2013) When patients feel that an app has a positive influence or is an effective means to manage their health, they continue and increase utilization of the app (Price et al. 2014; Shin et al. 2012) Increased usage combined with effectiveness will yield to better performance and result in positive evaluation s of the app (Direito et al. 2014) In this regard, although an app may have multiple features, patients may s till not evaluate the app positively if the

PAGE 82

73 function alities are not effective (Azar et al. 2013) Health effectiveness means that the app is not only useful but also a good fit to provide or sol ve a s pecific health challenge (Laing et al. 2014) A diabetes tracking app, for example, that just provides information and instructions to the patient may not engage and garner the attention of p atients; but when it helps the user monitor exercise and diabetes outcomes, providing an effective means to manage the disease, the patient will evaluate the app more favorably (Aitken and Gauntlett 2013) We argue that although developers may provide apps with multiple functionalities, for the apps to be favorable to patients they must be developed to efficiently manage health care With a higher level of effectiveness, multi functional apps will be able t o engage patients more and subsequently get higher evaluations. Thus, we hypothesize: HYPOTHESIS 4 (H 4 ): App effectiveness will have a positive impact on the app evaluation. McCurdie et al. (2012) argue that health app users are less engage d with apps that are not easy to use and do not return to them after the initial download This continual pattern will hinder the potential effectiveness of any health interventions. A pp s that are easy to use will engage users more quickly and thus result in the user s evaluat ing them positively Mendiola et al. (2015) found that placed a higher value on apps that were more simple than complex and intuitive to use. This is important to keep in mind if it becomes expec ted for patients to use mHealth apps as a regular part in their own care. H ealth apps are more able to be successful in assisting patients and serving the intended purpose with high ease of use Apps with low ease of use, even i f they have high effectivene ss, the patients will

PAGE 83

74 be les s receptive of the application. Therefore to ensure that the health app is successful in terms of evaluation, the ease of use should be a priority Based on the previous arguments we hypothesize: HYPOTHESIS 5 (H 5 ): App ease of use will have a positive impact on the app evaluation. Method Data and Variables To test our hypotheses, we focused on health apps in the A ndroid app store. We collected data through a consulting company for a span of 14 weeks from October 2014 to January 2015. Such a time span is sufficient to observ e changes in the app account for newly posted user reviews (Svedic 2015) The first week is the focal reference week for the health apps in this study. We found more than 3 ,203 health and medical applications in the focal week of the 13th October 2014 to 20th October 2014. We could not consider 1 27 apps for our analysis beca use they did not have any reviews in the marketplace in that week. In addition, we eliminated 96 which were not in in English, had unreadable names, or were duplicated in the market. W e excluded 2 123 that were not related to patient information, disease i nstruction, treatment, diagnosis, or disease management. Finally, we ex c luded nine medical apps that are not available to the general browsing public and needs a special code to be able to use it. We also excluded 663 apps that were not directed to patient s, rather directed towards providers, insurance profess io nals coders, healthcare professionals, and medical students. A total of 185 apps from the Android app stores met the inclusion criteria. W e tracked these 185 apps for the 14 weeks to have an unbalanced (minimal) panel data set of 2,215 observations.

PAGE 84

75 Following the Institute for Healthcare Informatics report by Aitken and Gauntlett (2013) we have classified health apps according to their qualities into two categories, instructive and integrative apps We also broke apps out according to the nature of their qualities into either integrative or instructive /informative apps The instructive /informative qualities are reflected through the features of providing information in different formats to the patient or instruction on managing a condition, the prevention of diseases, healthy living, symptomatic searching or finding a physician or facility, educati on on different procedures or diseases. The integrative qualities are reflected through the features of providing interactive services like following up with doctors, teleconferencing provisions, filling prescriptions, or post hospitalization follow up wit h health care professionals and care givers Table 7 provides a description of each quality and related features with some examples. In Table 8 we provide a description of the variables we use in this study. The focal dependent variable in our model is th e average weekly rating for each health app. The focal independent variable in this study is the number of health apps functionalities The independent variable reflects the number of functionality offered by each app and was coded by coding the health apps in regard to six main functionalities The second variable health app ease of use is a dependent variable it reflect s the ease of navigation how user friendly the interface is, and w he ther the application was hard to learn or not. The third variable, health app effectiveness, was coded by mining the text reviews of each health app see Table 8 for more details We controlled for many variables such as the price of the app, the longevity of the app in the market, and the average download per week. We also coded dummy variables indicating the category of the app and included them as controls. We used text analytics to

PAGE 85

76 measure both the app ease of use and the app effectiveness based on the reviews that were posted by the users in the Android digital market, a detailed description of the process is provided in A ppendix E. T able 9 provides the descriptive statistics and correlations amongst key variables used in this study. Empirical Analysis Since we are tracking health apps across multiple weeks w e are using panel data for our analysis. We ran a Hausman test to determine which time of panel analyses we should follow ; the result was significant and hence we used a fixed effect model. We appli ed the panel data estimation with fixed effect to model the patient evaluation because the change in patient evaluations is a continuous variable (Greene 2003) Y n = it u it it (1) Where, Yn is the dependent variable, X i parameters, u is the between observation. More specifically, to test the direct effects, we specified the following equation s: Effectivness = 1 1 Instructive 1 2 Integrative + 1i ControlVariable i + it it ( 2 ) EaseOfUse = 21 NumberOfFunctionalities + 2i ControlVariable i + it it ( 3 ) Evaluation = 31 Instructive 32 Integrative + 33 NumberOfFunctionalities + 34 Effectivness i + 35 EaseOfUse i + 2i ControlVariable i + it it ( 4 ) Our panel estimations employ Fixed effects (FE) models which account for heterogeneity. A concern in our estimation might be the potential endogeneity of the variables. To address this concern, we estimated the models using the generalized method of

PAGE 86

77 moments (GMM) Arellano Bond estimator. This estimation method addresses situations where the regressors may be correlated with the error term (Arellano and Bond 1991) To make sure that regressors are not exogenous, we used the GMM estimator as it is used for datasets with many panels and few periods. Using the time dimension (weeks), the Arellano and Bond (1991) derived a consistent GMM estimator for the parameters in our test for over ident ifying restrictions to test whether our instruments are valid or not. This test uses the moment conditions to test the validity of the instruments. Results (p values) show that in our case, all the estimations were supported and we cannot reject the null h ypothesis. The results obtained using the GMM models are qualitatively similar to those of the fixed effects models. Our results are thus robust to controlling for endogeneity. Results We tested the direct relationship between instructive and integrative q ualities and app effectiveness. The variable Instructive qualities ha ve a positive and statistically significant association with App Effectiveness (see c olumn p < 0.01 ). Th ese results support hypothesis H1. The variable Integrative qualities have also a positive and statistically significant association with App Effectiveness (see c olumn 0. 207 p < 0.01 ). Th ese results support hypothesis H2. Further, the results translate to the finding that for apps w ith both Instructive and Integrative qualities that Effectiveness of the health app will increase by approximately 39%. We find that Number of Functionalities has a negative and statistically significant association with App Ease Of Use (See Column 2 of T 0.209 p < 0.01 ). These

PAGE 87

78 results indicate that App ease of use decreases with the increase of the app number of functionalities. This result supports our hypothesis H3. We also tested for joint effects of all terms in our model (Column 3 of T able 10 ). The variable Effectiveness has a positive and statically significant association with App Evaluation (see c olumn p < 0.01 ). This result support s hypothesis H4. We interpret th is results as tha t if the effectiveness of th e Health app increased by one fold the Evaluation of the app will increase by one star We also find the variable Ease Of Use to have a positive and statically significant association with App Evaluation (see c olumn 189 p < 0.01 ). Thi s result support s hypothesis H5. We interpret this last result as that users will evaluate health apps that are easy to use higher than the ones that are no. We conducted several tests to assess the robustness of our findings. We tested for multicollinearity by computing variance inflation factors (VIFs) for all estimation models. The highest VIF was 2.0 in the direct effect models, confirming that multicollinearity is not a serious concern. To reduce potential high multicolli nearity issues due to the number of interaction terms in the models all continuous variables were mean centered by subtracting the corresponding varia ble mean from each value (Aiken et al. 1991) The VIF of any individual variable in any of the interaction effect models was less than 7.0. Furthermore, mean VIFs in all the models were less than 5.0. Thus, we find that multicollinearity is not a serious concern in the estimation. To investigate how a change in employ a random effects model to analyze our sample of the panel To investigate how a change in functionalities and appeal affects the apps ratings, we employ a random effects

PAGE 88

79 model to analyze our sam ple of the panel We also consider a more generic generalized method of moments (GMM) approach. The GMM estimation results were similar to the panel estimation results. This validates the robustness of our results, specific to the sensitivity to endogeneit y issues. We plotted the interaction effect results and present ed it in the interaction graphs in A ppendix D I n the interaction plot, t he difference in the positive slope of the lines in F igure 17 showing that apps with high health effectiveness compared to low health effectiveness is clearly discernab le across multiple functionalities. Discussion and Conclusion With hundreds of health apps in the digital market patients, caregivers, and healthcar e providers need guidance on identifying apps that are effective, provide good information provide good service and are easy to use. This research can help researcher s and developers better understand the strengths and weaknesses of existing health relat ed apps With the growing demand and market for health apps, our research can help guide the development of the next generation of health apps as well From our previous results, we draw that adding more effective functionalities to health apps help s incre ase the average rating of the health app. Another finding is that the effectiveness and the ease of use of the Health apps will positively impact the overall rating of the health app in the marketplace. We also draw some managerial implications from this s tudy. First, the apps qualities and functionalities play Hence, developers should pay attention toward what type of functionalities they provide in their applications. In addition identifying users who value health apps and take their feedback is very important to enhance future releases. This study contributes to the literature

PAGE 89

80 of mobile health applications, by identifying how technological and functional factors are associated w ith digital application and services success. Finally, tracking evaluation s While review and feedback mechanisms are seen in other products and services such as games and entertainment, Health appli cations providers and developers need to implement such mechanism s In terms of research contributions, to our knowledge this study will be the first one to explore the effects of factors like functionality, ease of use and effectiveness on how users ra te Health apps in the digital marketplace. These contributions will enrich the existing information systems literature and research associated with mobile Health applications. Future studies can look if Health apps were recommended or prescribed by provide rs and how that will applications that came from the android digital market; future studies may include applications from other markets as well, like iOS and Microsoft. There are a fe w limitations to this study. To begin, this study used just two pieces of data from the review process, review text and review rating, to assess the importance of the variables to the overall rating. It is likely that someone writing a positive review, ind icating that the app is effective, would also give a high rating. Future research could get independent assessments of functionalities. Secondly, the constructs, integrative and instructive qualities, were measured as binary variables. For future research, researchers should look into creating a scale for measuring these constructs to provide a more insights. Finally, we used data from one digital marketplace, Google Android Store. Future research should work to replicate this research using datasets from t he IOS store, Amazon and Microsoft.

PAGE 90

81 In conclusion, as Mobile Health apps become more ubiquitous, Health app functionalities and features short and long term decision to use the app. In sum, our study shows that providers and healthcare professionals should pay attention not only to health app app easy to user or not while designing and developing their health apps. Developers tend to add more functionalities to their apps to keep up with rivals and competition That sometimes yields to increase the complexity of the app and reduce its usability Developers that contradict what users find easy to use may trigger a bad first impression s functionality is important to establish a long term relationship between the user and the health app The usability and ease of use of the app is significant in leaving a good first impression to start usi ng the app and increase their engagement There is a need for a balance between providing more effective functionality and keep the apps simple and easy to use

PAGE 91

82 Figures and Tables Figure 16 : Conceptual Model

PAGE 92

83 Table 7 : Classification of Health App Functionalities and Features Functionality* Definitions* Example(s) Instructive /informative qualities Prevention and Healthy Living Apps that focus on overall wellness like apps that promote h ealthy eating, help with w eight management f itness tips for healthy living, s moking cessation s tress management or promoting healthy s leep 101 Natural Home; Remedies Cure; Relax Meditation Symptomatic and Self diagnosis Apps that focus on overall self diagnosis like a reference for common symptoms diagnosis based on data inputted and question answered ADHD Quiz Finding a Physician Apps that help with finding and locating medical professionals like locat ing a most appropriate physician or healthcare facility, find contact information, or provide rating and review s of physicians and medical facilities. Amwell; Anthem Blue Cross Education and post diagnosis information Apps that focus on providing health material like d rugs and medication information e mergency and first aid information or c ondition management information Drug Guide for Consumers ; Immunity Boosters Integrative qualities Filling prescription Apps that focus on allowing the patient to request or refill a prescription like r efilling of prescriptions and/or d rug interactions and side effects Anthem Blue Cross ; MobileRx Pharmacy Compliance and monitoring Apps that assist the patient to act within the prescribed interva l and dose of a dosing regimen like, p ill reminders m edication trackers alert support network, enable healthcare vitals. Allow patients to access their medical health records. workflow Pill Reminder ; Pill Organizer *This table is adapted from the Institute for Healthcare Informatics report by Aitken and Gauntlett (2013)

PAGE 93

84 Table 8 : Key Variables Description Variable Description and Operati onalization Reference(s) Evaluation Average weekly rating of the app in the app store (Mendiola et al. 2015; Tian et al. 2015) Number of Functionalities T he total number of instructive/ informative and integrative functionalities. The four major instructive functionalitie s are display of health and medical information, providing medical and health instructions, search and explore for health and medical services and providing general health education The two integrative functionalities are connecting to healthcare provider back end system and aligning with workflow and operational requirements The integrative functionalities are reflected through the features of providing reminders, alerts, connecting or following up with doctors or providers, or with video or teleconferencing provisions, filling prescriptions, or compliance and adherence (e.g., medication, post surgical). (Aitken and Gauntlett 2013; Arnhold et al. 2014) Instructive and Informative Qualities The instructive functionalities are reflected through the features of providing information or instruction on healthy living, the prevention of diseases, self diagnosis of diseases or a condition searching or finding a physician or facility, medical and health education for different procedures or diseases. This variable captures the content the health app delivers in respect to cor rectness, completeness, ease of understanding, and relevance of the instructions or information. (Aitken and Gauntlett 2013; Delone and McLean 2003) Integrativ e Quality Apps that provid e interactive services like following up with doctors, teleconferencing provisions, filling prescriptions, or post hospitalization follow up. This variable is coded as 1 if integrative functionalities are present, and 0 otherwise. (Aitken and Gauntlett 2013; Delone and McLean 2003) App Effectiveness Effectiveness is the positive impact felt by the user that helped him or her to achiev e specified health or medical goal. It can be determined by was successfully met or not. This variable is coded by (Arellano and Bochniski 2012; Pham et al. 2016; Stoyanov et al. 2015) App Ease of User This variable reflect s ease of navigation how user friendly the interface is, and w he ther the application was hard to learn or not. (Arnhold et al. 2014; Mendiola et al. 2015)

PAGE 94

85 Table 9 : Descriptive Statistics amongst Key Variables Variable Mean Std. Dev. Min Max Evaluation 2 .9 4 0 .5 3 2.14 5 Ease of use .022 0 .37 0.89 0 .95 Functionality 1.31 0 .6 3 0 6 Effectiveness 0 .35 0 .23 0 1 Integrative 0 .28 0 .45 0 1 Instructive 0 .48 0 .49 0 1 Age 3.11 0 .95 1.36 5.24 Price 1.0 7 2.62 0 24.99 Download 1 41 0 413 75 30,000

PAGE 95

86 Table 10 : Panel Estimation Fixed Effect Models VARIABLES Fixed Effect Models (1) (2) (3) Effectiveness Ease of Use Evaluation Instructive 0.181*** (0.014) 0.176*** (0.024) Integrative 0.207*** (0.026) 0.137*** (0.034) Functionalities 0.209*** (0.032) 0.089*** (0.026) Effectiveness 0.989*** (0.037) Ease of Use 0.189*** (0.016) Age 0.038* (0.016) 0.036 (0.033) 0.054* (0.025) Price 0.001 (0.002) -0.016** (0.004) 0.006** (0.003) Download 0.001 (0.001) 0.001 (0.001) 0.001 (0.001) App last update 0.009 (0.011) 0.571 (0.456) 0.011 ( 0.014) Control variables Included Included Included Constant 0.201*** (0.041) 0.412*** (0.087) 3.647*** (0.068) Number of apps 185 185 185 Number of observations 2,215 2,215 2,215 R squared 0.29 0 0 .310 0.3 43 F Stat 1 4 91 *** 1 5 8 6*** 1 5 91 *** (1) Significance levels: ***p < 0.01, **p < 0.05, *p < 0.10 (2) Standard errors in parentheses reviews in the market, total rating of the developer, price of the app, when the app was last updated, when the app was introduced in the app store, number of downloads of the app, when the publisher released their first app in the app market

PAGE 96

87 Table 11 : Description of Variables Variable Description and Operationalization Reference(s) Evaluation Average weekly rating of the app in the app store (Mendiola et al. 2015; Tian et al. 2015) Functionalities This variable is the total count of the four instructive functionalities and two integrative functionalities. The four major instructive functionalities of the apps: (1) display of information, (2) providing instructions, (3) search and explore functions, (4) providing education The instructive functionalities are reflected through the features of providing information or instruction on the prevention of diseases, healthy living, sympto ma tic or self diagnosis of the diseases, searching or finding a physician or facility, education on different procedures or diseases. The two integrative functio nalities of the apps: (1) connecting to back end applications with the features, (2) aligning to workflow and operational requirements The integrative functionalities are reflected through the features of providing reminders, alerts, connecting or followi ng up with doctors or providers, or with video or teleconferencing provisions, filling prescriptions, or compliance and adherence (e.g., medication, post surgical). Aitken and Gauntlett (2013) Instructive/ Informative Apps that have the capability to provid e information or instruction on the prevention of diseases, healthy living, symptomatic, searching or finding a physician or facility, education on different procedures or diseases This varia ble is coded as 1 if instructive/informative functionalities are present, and 0 otherwise. (Aitken and Gauntlett 2013; Iribarren et al. 2016) Integra tive Apps that provid e interactive services like following up with doctors, teleconferencing provisions, filling prescriptions, or post hospitalization follow up. This variable is coded as 1 if integrative functionalities are present, and 0 otherwise. Aitk en and Gauntlett (2013) Health Effectiveness The positive impact felt or noticed by the user for using a health application. This variable is coded by mining the text reviews of each app in time. (Arellano and Bochniski 2012; Pham et al. 2016; Stoyanov et al. 2015) Age of the app (Age) How long the app has existed in the A ndroid market since its release. The variable was calculated by finding the difference in years between the focal week and the release date of the app. (Mendiola et al. 2015; Tian et al. 2015)

PAGE 97

88 Variable Description and Operationalization Reference(s) Price of the app (Price) The price of the health app in US dollars. (Mendiola et al. 2015; Tian et al. 2015) Download Average downloads per week. (Mendiola et al. 2015; Tian et al. 2015) A ge Age of the app current version (Mendiola et al. 2015; Tian et al. 2015) App last update Number of update versions (Mendiola et al. 2015; Tian et al. 2015) In app purchase Availability of in app purchase option (Mendiola et al. 2015; Tian et al. 2015) Category Dummy (C_N) The category of the app, as pre defined in the A ndroid market. We coded dummy variables for each category to include in the analysis. (Mendiola et al. 2015; Tian et al. 2015) Table 12 : Correlation amongst Key Variables 1 2 3 4 5 6 7 8 9 1 Evaluation 1 .00 2 Ease of use 0.10 0 1 .00 3 Functionality 0. 091 0. 1 8 0 1 .00 4 Effectiveness 0.12 0 0.03 0 0.05 0 1 .00 5 Integrative 0.0 61 0.18 0 0.13 0 0. 230 1 6 Informative 0.080 0.01 0 0.0 11 0.00 8 0.120 1.00 7 Age 0.12 0 0.11 0 0.04 0 0.02 0 0.44 0.010 1.00 8 Price 0.01 0 0.00 5 0.01 0 0.0 11 0.00 8 0.0 12 0.002 1 .00 9 Download 0.01 0 0.01 0 0.110 0. 091 0.0 51 0.01 0 0.005 0.0 74 1 .00

PAGE 98

89 REFERENCES Adams, J., Giles, E. L., McColl, E., and Sniehotta, F. F. 2013. "Carrots, Sticks and Health Behaviours: A Framework for Documenting the Complexity of Financial Incentive Interventions to Change Health Behaviours," Health Psychology Review :Ahead of Print), pp. 1 10. Agarwal, R., Gao, G., DesRoches, C., and Jha, A. K. 2010. "Research Commentary the Digital Transformation of Healthcare: Current Status and the Road Ahead," Information Systems Research (21:4), pp. 796 809. Ahern, D. K., Woods, S. S., Lightowle r, M. C., Finley, S. W., and Houston, T. K. 2011. "Promise of and Potential for Patient Facing Technologies to Enable Meaningful Use," American journal of preventive medicine (40:5), pp. S162 S172. Aiken, L. S., West, S. G., and Reno, R. R. 1991. Multiple Regression: Testing and Interpreting Interactions Sage. Aitken, M., and Gauntlett, C. 2013. "Patient Apps for Improved Healthcare: From Novelty to Mainstream," Parsippany, NJ: IMS Institute for Healthcare Informatics ). Alpay, L., van der Boog, P., and Dumaij, A. 2011. "An Empowerment Based Approach to Developing Innovative E Health Tools for Self Management," Health informatics journal (17:4), pp. 247 255. Amante, D. J., Hogan, T. P., Pagoto, S. L., and English, T. M. 2014. "A Systematic Review of Elec tronic Portal Usage among Patients with Diabetes," Diabetes technology & therapeutics (16:11), pp. 784 793. Amatriain, X., Pujol, J. M., and Oliver, N. 2009. "I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems," Internati onal Conference on User Modeling, Adaptation, and Personalization : Springer, pp. 247 258. American Diabetes Association. 2007. "Standards of Medical Care in Diabetes 2007," Diabetes Care (30:suppl 1), pp. S4 S41. American Diabetes Association. 2013. "Eco nomic Costs of Diabetes in the Us in 2012," Diabetes Care (36:4), pp. 1033 1046. American Diabetes Association. 2014. "Standards of Medical Care in Diabetes -2014," Diabetes care (37), p. S14. Ammenwerth, E., Iller, C., and Mahler, C. 2006. "It Adoption and the Interaction of Task, Technology and Individuals: A Fit Framework and a Case Study," BMC medical informatics and decision making (6:1), p. 3.

PAGE 99

90 Ammenwerth, E., Schnell Inderst, P., and Hoerbst, A. 2012. "The Impact of Electronic Patient Portals on Pa tient Care: A Systematic Review of Controlled Trials," Journal of Medical Internet Research (14:6), p. e162. Anderson, C. L., and Agarwal, R. 2011. "The Digitization of Healthcare: Boundary Risks, Emotion, and Consumer Willingness to Disclose Personal Hea lth Information," Information Systems Research (22:3), pp. 469 490. Anderson, J. R. 1982. "Acquisition of Cognitive Skill," Psychological review (89:4), p. 369. Andrews, J. C., and Shimp, T. A. 1990. "Effects of Involvement, Argument Strength, and Source Characteristics on Central and Peripheral Processing of Advertising," Psychology & Marketing (7:3), pp. 195 214. Arazy, O., and Woo, C. 2007. "Enhancing Infor mation Retrieval through Statistical Natural Language Processign: A Study of Collocation Indexing," MIS quarterly ), pp. 525 546. Archak, N., Ghose, A., and Ipeirotis, P. G. 2011. "Deriving the Pricing Power of Product Features by Mining Consumer Reviews," Management Science (57:8), pp. 1485 1509. Arellano, M., and Bond, S. 1991. "Some Tests of Specification for Panel Data: Monte Carlo Evidence and an Application to Employment Equations," The review of economic studies (58:2), pp. 277 297. Arellano, P., a nd Bochniski, J. 2012. "Selecting a Mobile App: Evaluating the Usability of Medical Applications," Himss ). Arnhold, M., Quade, M., and Kirch, W. 2014. "Mobile Applications for Diabetics: A Systematic Review and Expert Based Usability Evaluation Considerin g the Special Requirements of Diabetes Patients Age 50 Years or Older," Journal of Medical Internet Research (16:4), p. e104. Au, A. G., McAlister, F. A., Bakal, J. A., Ezekowitz, J., Kaul, P., and van Walraven, C. 2012. "Predicting the Risk of Unplanned Readmission or Death within 30 Days of Discharge after a Heart Failure Hospitalization," American heart journal (164:3), pp. 36 5 372. Azar, K. M., Lesser, L. I., Laing, B. Y., Stephens, J., Aurora, M. S., Burke, L. E., and Palaniappan, L. P. 2013. "Mobile Applications for Weight Management: Theory Based Content Analysis," American journal of preventive medicine (45:5), pp. 583 58 9. Bai, X. 2011. "Predicting Consumer Sentiments from Online Text," Decision Support Systems (50:4), pp. 732 742.

PAGE 100

91 Barth, K. S., Balliet, W., Pelic, C. M., Madan, A., Malcolm, R., Adams, D., Morgan, K., Owczarski, S., and Borckardt, J. J. 2014. "Screening for Current Opioid Misuse and Associated Risk Factors among Patients with Chronic Nonalcoholic Pancreatitis Pain," Pain Medicine (Online Prior to Print: DOI: 10.1111/pme.12403). BCC. 2014. "Global Mhealth Technologies Market Projected to Reach Nearly $21 .5 Billion in 2018; Europe Segment Growing at 61.6% Cagr." from http://www.bccresearch.com/pressroom/hlc/global mHealth technologies market projected to reach nearly $21.5 billion 2018 Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun T., McKinstry, B., Procter, R., Majeed, A., and Sheikh, A. 2011. "The Impact of Ehealth on the Quality and Safety of Health Care: A Systematic Overview," PLoS Med (8:1), p. e1000387. Bloomgarden, Z. T. 2004. "Type 2 Diabetes in the Young the Evolving Ep idemic," Diabetes Care (27:4), pp. 998 1010. Boulos, M. N. K., Brewer, A. C., Karimkhani, C., Buller, D. B., and Dellavalle, R. P. 2014. "Mobile Medical and Health Apps: State of the Art, Concerns, Regulatory Control and Certification," Online journal of public health informatics (5:3). Myocardial Infarction," Journal of advanced nursing (53:6), pp. 636 645. Britto, M. T., Jimison, H. B., Munafo, J. K., Wissman, J., Rogers, M. L., and Hersh, W. 2009. "Usability Testing Finds Problems for Novice Users of Pediatric Portals," Journal of the American Medical Informatics Association (16:5), pp. 660 669. Buntin, M. B., Burke, M. F., Hoaglin, M. C., and Blumenthal, D. 2011. "The Benefits of Health Information Technology: A Review of the Recent Literature Shows Predominantly Positive Results," Health Affairs (30:3), pp. 464 471. Byczkowski, T. L., Munafo, J. K., and Britto, M. T. 2011. "Variation in Use of Internet Based Patient Portals by Parents of Children with Chronic Disease," Archives of pediatrics & adolescent medicine (165:5), pp. 405 411. Calder, B. J., Insko, C. A., and Yandell, B. 1974. "The Relation o f Cognitive and Memorial Processes to Persuasion in a Simulated Jury Trial," Journal of Applied Social Psychology (4:1), pp. 62 93. Cao, Q., Duan, W., and Gan, Q. 2011. "Exploring Determinants of Voting for the Mining Approach," Decision Support Systems (50:2), pp. 511 521. Carletta, J. 1996. "Assessing Agreement on Classification Tasks: The Kappa Statistic," Computational linguistics (22:2), pp. 249 254.

PAGE 101

92 Carrell, D., and Ralston, J. D. 2006. "Variation in Adop tion Rates of a Patient Web Portal with a Shared Medical Record by Age, Gender, and Morbidity Level," AMIA Annual Symposium Proceedings : American Medical Informatics Association, p. 871. Casale, P., Devereux, R., Milner, M., Zullo, G., Harshfield, G., Pic kering, T., and Laragh, J. 1986. "Value of Echocardiographic Measurement of Left Ventricular Mass in Predicting Cardiovascular Morbid Events in Hypertensive Men," Annals of internal medicine (105:2), pp. 173 178. Chahuneau, V., Gimpel, K., Routledge, B. R ., Scherlis, L., and Smith, N. A. 2012. "Word Salad: Relating Food Prices and Descriptions," Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning : Association for Computati onal Linguistics, pp. 1357 1367. Chaiken, S. 1980. "Heuristic Versus Systematic Information Processing and the Use of Source Versus Message Cues in Persuasion," Journal of personality and social psychology (39:5), p. 752. Charpentier, G., Benhamou, P. Y., Dardari, D., Clergeot, A., Franc, S., Schaepelynck Belicar, P., Catargi, B., Melki, V., Chaillous, L., and Farret, A. 2011. "The Diabeo Software Enabling Individualized Insulin Dose Adjustments Combined with Telemedicine S upport Improves Hba1c in Poorly Controlled Type 1 Diabetic Patients a 6 Month, Randomized, Open Label, Parallel Group, Multicenter Trial (Telediab 1 Study)," Diabetes Care (34:3), pp. 533 539. Cheung, C. M., and Thadani, D. R. 2012. "The Impact of Electro nic Word of Mouth Communication: A Literature Analysis and Integrative Model," Decision Support Systems (54:1), pp. 461 470. Crick, N. R., and Dodge, K. A. 1994. "A Review and Reformulation of Social Information Processing Mechanisms in Children's Social Adjustment," Psychological bulletin (115:1), p. 74. Crozier, R., and Ranyard, R. 1997. "Cognitive Process Models and Explanations of Decision Making," Decision making: Cognitive models and explanations ), pp. 5 20. D'Ambra, J., and Wilson, C. S. 2004. "Use of the World Wide Web for International Travel: Technology Fit (Ttf) Model," Journal of the American Society for Information Science and Technology (55:8), pp. 731 742. Journal of the American Society for Information Science and Tech nology (64:1), pp. 48 64.

PAGE 102

93 de Boer, I. H., Rue, T. C., Hall, Y. N., Heagerty, P. J., Weiss, N. S., and Himmelfarb, J. 2011. "Temporal Trends in the Prevalence of Diabetic Kidney Disease in the United States," JAMA (305:24), pp. 2532 2539. de Groot, V., Bec kerman, H., Lankhorst, G. J., and Bouter, L. M. 2003. "How to Measure Comorbidity: A Critical Review of Available Methods," Journal of clinical epidemiology (56:3), pp. 221 229. de Lusignan, S., Mold, F., Sheikh, A., Majeed, A., Wyatt, J. C., Quinn, T., C avill, M., Electronic Health Records and Linked Online Services: A Systematic Interpretative Review," BMJ open (4:9), p. e006021. Dehzad, F., Hilhorst, C., de Bie, C., an Hindering Doctors and Patients?," Health (2014). DeLone, W. H., and McLean, E. R. 1992. "Information Systems Success: The Quest for the Dependent Variable," Information Systems Research (3:1), pp. 60 95. Delone, W. H., and McLean, E. R. 2003. "The Delone and Mclean Model of Information Systems Success: A Ten Year Update," Journal of Management Information Systems (19:4), pp. 9 30. Delone, W. H., and Mclean, E. R. 2004a. "Measuring E Commerce Success: App lying the Delone & Mclean Information Systems Success Model," International Journal of Electronic Commerce (9:1), pp. 31 47. Delone, W. H., and Mclean, E. R. 2004b. "Measuring E Commerce Success: Applying the Delone &Amp;Amp; Mclean Information Systems Su ccess Model," International Journal of Electronic Commerce (9:1), pp. 31 47. Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., and Lang, T. 2004. "A Systematic Review of Computer Based Patient Record Systems and Quality of Care: More Randomi zed Clinical Trials or a Broader Approach?," International Journal for Quality in Health Care (16:5), pp. 407 416. Demiris, G., Afrin, L. B., Speedie, S., Courtney, K. L., Sondhi, M., Vimarlund, V., Lovis, C., Goossen, W., and Lynch, C. 2008. "Patient Cen tered Applications: Use of Information Technology to Promote Disease Management and Wellness. A White Paper by the Amia Knowledge in Motion Working Group," Journal of the American Medical Informatics Association (15:1), pp. 8 13. Direito, A., Dale, L. P., Shields, E., Dobson, R., Whittaker, R., and Maddison, R. 2014. "Do Physical Activity and Dietary Smartphone Applications Incorporate Evidence Based Behaviour Change Techniques?," BMC public health (14:1), p. 1.

PAGE 103

94 Douglas, G. P., Landis Lewis, Z., and Hochh eiser, H. 2011. "Simplicity and Usability: Lessons from a Touchscreen Electronic Medical Record System in Malawi," interactions (18:6), pp. 50 53. Drawz, P., and Rahman, M. 2015. "Chronic Kidney Disease," Annals of internal medicine (162:11), pp. ITC1 ITC1. Books and Academic Tasks,"). Economist. 2016. "Mobile Health Apps Are Becoming More Capable and Potentially Rather Useful." from http://www.economist.com/news/business/21694523 mobile health apps are becoming more capable and potentially rather useful things are looking Ellison, G. 2003. "A Model of Add on Pricing," National Bureau of Economic Research. Emont, S. 2011. "Measuring the Impact of Patient Portals," California Healthcare Foundation ), pp. 1 20. Fauci, A. S. 2008. Harrison's Principles of Internal Medicine McGraw Hill Medical New York. Feng, S., Xing, L., Gogar, A., and Choi, Y. 2012. "Distributional Footprints of Decep tive Product Reviews," ICWSM (12), pp. 98 105. Finkelstein, A., Harman, M., Jia, Y., Martin, W., Sarro, F., and Zhang, Y. 2014. "App Store Analysis: Mining App Stores for Relationships between Customer, Business and Technical Characteristics," RN (14), p. 10. Flanagin, A. J., Metzger, M. J., Pure, R., Markov, A., and Hartsell, E. 2014. "Mitigating Risk in Ecommerce Transactions: Perceptions of Information Credibility and the Role of User Generated Ratings in Product Quality and Purchase Intention," Electr onic Commerce Research (14:1), pp. 1 23. Forman, C., Ghose, A., and Wiesenfeld, B. 2008. "Examining the Relationship between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets," Information Systems Research (19:3), pp. 291 3 13. Fox, S., and Duggan, M. 2012. "Mobile Health 2012," Washington, DC: Pew Internet & American Life Project ). Medical Care Research and Review (70:1 suppl), pp. 8 0S 93S.

PAGE 104

95 Free, C., Phillips, G., Galli, L., Watson, L., Felix, L., Edwards, P., Patel, V., and Haines, A. 2013a. "The Effectiveness of Mobile Health Technology Based Health Behaviour Change or Disease Management Interventions for Health Care Consumers: A Systematic Review," PLoS Med (10:1), p. e1001362. Free, C., Phillips, G., Watson, L., Galli, L., Felix, L., Edwards, P., Patel, V., and Haines, A. 2013b. "The Effectiveness of Mobile Health Technologies to Improve Health Care Service Delivery Processes: A Systematic Review and Meta Analysis," PLoS Med (10 :1), p. e1001363. Fu, B., Lin, J., Li, L., Faloutsos, C., Hong, J., and Sadeh, N. 2013. "Why People Hate Your App: Making Sense of User Feedback in a Mobile App Store," Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining : ACM, pp. 1276 1284. Fukuoka, Y., Vittinghoff, E., Jong, S. S., and Haskell, W. 2010. "Innovation to Motivation Pilot Study of a Mobile Phone Intervention to Increase Physical Activity among Sedentary Women," Preventive medicine (51:3), pp. 2 87 289. Frster, J., Liberman, N., and Kuschel, S. 2008. "The Effect of Global Versus Local Processing Styles on Assimilation Versus Contrast in Social Judgment," Journal of personality and social psychology (94:4), p. 579. Gebauer, J. 2008. "User Requirements of Mobile Technology: A Summary of Research Results," Information Knowledge Systems Management (7:1, 2), pp. 101 119. Gianchandani, E. P. 2011. "Toward Smarter Health and Well Being: An Implicit Role for Networking and Information Technology," Journal of Information Technology (26:2), pp. 120 128. Glynn, L. G., Hayes, P. S., Casey, M., Glynn, F., Alvarez Iglesias, A., Newell, J., "Laighin, G., Heaney, D., and Murphy, A. W. 2013. "Smart Move a Smartphone Based Intervent ion to Promote Physical Activity in Primary Care: Study Protocol for a Randomized Controlled Trial," Trials (14:1), p. 157. Goh, J. M., Gao, G., and Agarwal, R. 2011. "Evolving Work Routines: Adaptive Routinization of Information Technology in Healthcare, Information Systems Research (22:3), pp. 565 585. Goldzweig, C. L., Orshansky, G., Paige, N. M., Towfigh, A. A., Haggstrom, D. A., Miake Lye, I., Beroes, J. M., and Shekelle, P. G. 2013. "Electronic Patient Portals: Evidence on Health Outcomes, Satisfact ion, Efficiency, and Attitudesa Systematic Review," Annals of internal medicine (159:10), pp. 677 687. Goodhue, D. L., and Thompson, R. L. 1995. "Task Technology Fit and Individual Performance," MIS quarterly ), pp. 213 236.

PAGE 105

96 Gottfredson, M., and Aspinall, K. 2005. "Innovation Versus Complexity," Harvard business review ), pp. 62 71. Graffigna, G., Barello, S., and Riva, G. 2013. "How to Make Health Information Technology Effective: The Challenge of Patient Engagement," Archives of physical medicine and reh abilitation (94:10), pp. 2034 2035. Grant, R. W., Wald, J. S., Schnipper, J. L., Gandhi, T. K., Poon, E. G., Orav, E. J., Williams, D. H., Volk, L. A., and Middleton, B. 2008. "Practice Linked Online Personal Health Records for Type 2 Diabetes Mellitus: A Randomized Controlled Trial," Archives of Internal Medicine (168:16), pp. 1776 1782. Greene, W. 2003. "Econometric Analysis." Upper Saddle River, NJ: Prentice Hall. Gupta, P., and Harris, J. 2010. "How E Wom Recommendations Influence Product Considerati on and Quality of Choice: A Motivation to Process Information Perspective," Journal of Business Research (63:9), pp. 1041 1049. Guralnik, J. M. 1996. "Assessing the Impact of Comorbidity in the Older Population," Annals of epidemiology (6:5), pp. 376 380. Hah, H., and Bharadwaj, A. 2012. "A Multi Level Analysis of the Impact of Health Information Technology on Hospital Performance,"). Hawkins, R. P., Kreuter, M., Resnicow, K., Fishbein, M., and Dijkstra, A. 2008. "Understanding Tailoring in Communicating About Health," Health education research (23:3), pp. 454 466. Heit, H. A., and Gourlay, D. L. 2004. "Urine Drug Testing in Pain Medicine," Journal of pain and symptom management (27:3), pp. 260 267. Heppner, P. P., and Krauskopf, C. J. 1987. "An Informa tion Processing Approach to Personal Problem Solving," The Counseling Psychologist (15:3), pp. 371 447. Heppner, P. P., and Lee, D. g. 2009. Problem Solving Appraisal and Psychological Oxford University Press. Hilligoss, B., and Rieh, S. Y. 2008. "Devel oping a Unifying Framework of Credibility Assessment: Construct, Heuristics, and Interaction in Context," Information Processing & Management (44:4), pp. 1467 1484. Holmen, H., Torbjrnsen, A., Wahl, A. K., Jenum, A. K., Smstuen, M. C., rsand, E., a nd Ribu, L. 2014. "A Mobile Health Intervention for Self Management and Lifestyle Change for Persons with Type 2 Diabetes, Part 2: One Year Results from the Norwegian Randomized Controlled Trial Renewing Health," JMIR mHealth and uHealth (2:4).

PAGE 106

97 Holroyd Led uc, J. M., Lorenzetti, D., Straus, S. E., Sykes, L., and Quan, H. 2011. "The Impact of the Electronic Medical Record on Structure, Process, and Outcomes within Primary Care: A Systematic Review of the Evidence," Journal of the American Medical Informatics Association (18:6), pp. 732 737. Hovland, C. I., Harvey, O., and Sherif, M. 1957. "Assimilation and Contrast Effects in Reactions to Communication and Attitude Change," The Journal of Abnormal and Social Psychology (55:2), p. 244. Huang, G. H., and Korfia tis, N. 2015. "Trying before Buying: The Moderating Role of Online Reviews in Trial Attitude Formation toward Mobile Applications," International Journal of Electronic Commerce (19:4), pp. 77 111. Hussain, A., Claussen, B., Ramachandran, A., and Williams, R. 2007. "Prevention of Type 2 Diabetes: A Review," Diabetes research and clinical practice (76:3), pp. 317 326. Iribarren, S. J., Schnall, R., Stone, P. W., and Carballo Diguez, A. 2016. "Smartphone Applications to Support Tuberculosis Prevention and Treatment: Review and Evaluation," JMIR mHealth and uHealth (4:2). Irick, M. L. 2008. "Task Technology Fit and Info rmation Systems Effectiveness," Journal of Knowledge Management Practice (9:3), pp. 1 5. Jamal, A., McKenzie, K., and Clark, M. J. 2009. "The Impact of Health Information Technology on the Quality of Medical and Health Care: A Systematic Review," Health I nformation Management Journal (38:3), p. 26. Jindal, N., and Liu, B. 2008. "Opinion Spam and Analysis," Proceedings of the 2008 International Conference on Web Search and Data Mining : ACM, pp. 219 230. Johnson, M. D., and Fornell, C. 1991. "A Framework for Comparing Customer Satisfaction across Individuals and Product Categories," Journal of economic psychology (12:2), pp. 267 286. Johnston, J., and DiNardo, J. 1997. "Economctric Methods." McGraw Hill. Jones, J. B., Weiner, J. P., Shah, N. R., and Stewart, W. F. 2015. "The Wired Patient: Patterns of Electronic Patient Portal Use among Patients with Cardiac Disease or Diabetes," Journal of medical Internet research (17:2). Joshi, M., Das, D., Gimpel, K. and Smith, N. A. 2010. "Movie Reviews and Revenues: An Experiment in Text Regression," Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics : Association for Computational L inguistics, pp. 293 296. Kahneman, D., and Tversky, A. 1984. "Choices, Values, and Frames," American psychologist (39:4), p. 341.

PAGE 107

98 Kamis, A., Yao, Y., and Kim, S. 2014. "An Empirical Validation of the Patient Centered E Health Framework in Patient Focused Websites," Communications of the Association for Information Systems (34:25), pp. 477 492. Kassirer, J. P. 1994. "Incorporating Patients' Preferences into Medical Decisions," New England Journal of Medicine (330:26), pp. 1895 1896. Kirwan, M., Duncan, M. J., Vandelanotte, C., and Mummery, W. K. 2012. "Using Smartphone Technology to Monitor Physical Activity in the 10,000 Steps Program: A Matched Case Control Trial," Journal of Medical Internet Research (14:2), p. e55. Klasnja, P., and Pratt, W. 2012. "He althcare in the Pocket: Mapping the Space of Mobile Phone Health Interventions," Journal of biomedical informatics (45:1), pp. 184 198. Koenigsberg, M. R., Bartlett, D., and Cramer, J. S. 2004. "Facilitating Treatment Adherence with Lifestyle Changes in D iabetes," American Family Physician (69:2). Kornak, J. 2016. "System Requirements for Delivery of Telemedicine Services," in Telemanagement of Inflammatory Bowel Disease Springer, pp. 117 151. Kost, A. 2012. "Woman Paid to Post Five Star Google Feedback ," ABC7 News ). Krebs, P., and Duncan, D. T. 2015. "Health App Use among Us Mobile Phone Owners: A National Survey," JMIR mHealth and uHealth (3:4). Krippendorff, K. 1980. "Content Analysis: An Introduction to Its," Methodology, Sage Publications, Beverly Hills ). Krist, A. H., and Woolf, S. H. 2011. "A Vision for Patient Centered Health Information Systems," Jama (305:3), pp. 300 301. Kruse, C. S., Argueta, D. A., Lopez, L., and Nair, A. 2015. "Patient and Provider Attitudes toward the Use of Patient Por tals for the Management of Chronic Disease: A Systematic Review," Journal of medical Internet research (17:2), p. e40. Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W. T., Shar, A., Spring, B., and Spruijt Metz, D. 2 013. "Mobile Health Technology Evaluation: The Mhealth Evidence Workshop," American journal of preventive medicine (45:2), pp. 228 236. Laing, B. Y., Mangione, C. M., Tseng, C. H., Leng, M., Vaisberg, E., Mahida, M., Bholat, M., Glazier, E., Morisky, D. E., and Bell, D. S. 2014. "Effectiveness of a Smartphone Application for Weight Loss Compared with Usual Care in Overweight Primary Care P atients: A Randomized, Controlled Trial," Annals of internal medicine (161:10_Supplement), pp. S5 S12.

PAGE 108

99 Landis, J. R., and Koch, G. G. 1977a. "An Application of Hierarchical Kappa Type Statistics in the Assessment of Majority Agreement among Multiple Obser vers," Biometrics ), pp. 363 374. Landis, J. R., and Koch, G. G. 1977b. "The Measurement of Observer Agreement for Categorical Data," Biometrics ), pp. 159 174. Lavonas, E. J. 2013. "Management of the Critically Poisoned Patient," Emergency Department Crit ical Care ). Lee, G., and Raghu, T. 2014. "Determinants of Mobile Apps' Success: Evidence from the App Store Market," Journal of Management Information Systems (31:2), pp. 133 170. Lenz, R., and Kuhn, K. A. 2004. "Towards a Continuous Evolution and Adapta tion of Information Systems in Healthcare," International journal of medical informatics (73:1), pp. 75 89. Leveille, S. G., Huang, A., Tsai, S. B., Allen, M., Weingart, S. N., and Iezzoni, L. I. 2009. "Health Coaching Via an Internet Portal for Primary C are Patients with Chronic Conditions: A Randomized Controlled Trial," Medical care (47:1), pp. 41 47. Li, F., Huang, M., Yang, Y., and Zhu, X. 2011. "Learning to Identify Review Spam," IJCAI Proceedings International Joint Conference on Artificial Intelli gence p. 2488. Lim, E. P., Nguyen, V. A., Jindal, N., Liu, B., and Lauw, H. W. 2010. "Detecting Product Review Spammers Using Rating Behaviors," Proceedings of the 19th ACM international conference on Information and knowledge management : ACM, pp. 939 94 8. Lin, H. F. 2008. "Determinants of Successful Virtual Communities: Contributions from System Characteristics and Social Factors," Information & Management (45:8), pp. 522 527. Lin, T. C. 2014. "Mobile Nursing Information System Utilization: The Task Technology Fit Perspective," CIN: Computers, Informatics, Nursing (32:3), pp. 129 137. Liu, B. 2012. "Sentiment Analysis and Opinion Mining," Synthesis lectures on human language te chnologies (5:1), pp. 1 167. Liu, Y., Lee, Y., and Chen, A. N. 2011. "Evaluating the Effects of Task Individual Technology Fit in Multi Dss Models Context: A Two Phase View," Decision Support Systems (51:3), pp. 688 700.

PAGE 109

100 Lobelo, F., Kelli, H. M., Tejed or, S. C., Pratt, M., McConnell, M. V., Martin, S. S., and Welk, G. J. 2016. "The Wild Wild West: A Framework to Integrate Mhealth Software Applications and Wearables to Support Physical Activity Assessment, Counseling and Interventions for Cardiovascular Disease Risk Reduction," Progress in cardiovascular diseases ). Long, A. N., and Dagogo Jack, S. 2011. "Comorbidities of Diabetes and Hypertension: Mechanisms and Approach to Target Organ Protection," The Journal of Clinical Hypertension (13:4), pp. 244 25 1. Luna, D., Almerares, A., Mayan, J. C., Gonzalez Bernaldo de Quiros, F., and Otero, C. 2014. "Health Informatics in Developing Countries: Going Beyond Pilot Practices to Sustainable Implementations: A Review of the Current Challenges," Healthcare inform atics research (20:1), pp. 3 10. Lustria, M. L. A., Noar, S. M., Cortese, J., Van Stee, S. K., Glueckauf, R. L., and Lee, J. 2013. "A Meta Analysis of Web Delivered Tailored Health Behavior Change Interventions," Journal of health communication (18:9), pp 1039 1069. Marshall, A., Medvedev, O., and Antonov, A. 2008. "Use of a Smartphone for Improved Self Management of Pulmonary Rehabilitation," International journal of telemedicine and applications (2008), p. 2. Mathieson, K., and Keil, M. 1998. "Beyond the Interface: Ease of Use and Task/Technology Fit," Information & Management (34:4), pp. 221 230. McAllister, M., Dunn, G., Payne, K., Davies, L., and Todd, C. 2012. "Patient Empowerment: The Need to Consider It as a Measurable Patient Reported Outcome for Chronic Conditions," BMC health services research (12:1), p. 157. McCoy, A. B., Waitman, L. R., Lewis, J. B., Wright, J. A., Choma, D. P., Miller, R. A., and Peterson, J. F. 2012. "A Framework for Evaluating th e Appropriateness of Clinical Decision Support Alerts and Responses," Journal of the American Medical Informatics Association (19:3), pp. 346 352. McCurdie, T., Taneva, S., Casselman, M., Yeung, M., McDaniel, C., Ho, W., and Cafazzo, J. 2012. "Mhealth Con sumer Apps: The Case for User Centered Design," Biomedical Instrumentation & Technology (46:s2), pp. 49 56. McNeill, B. W., and Stoltenberg, C. D. 1989. "Reconceptualizing Social Influence in Counseling: The Elaboration Likelihood Model," Journal of Couns eling Psychology (36:1), p. 24. Mendiola, M. F., Kalnicki, M., and Lindenauer, S. 2015. "Valuable Features in Mobile Health Apps for Patients and Consumers: Content Analysis of Apps and User Ratings," JMIR mHealth and uHealth (3:2).

PAGE 110

101 Menke, A., Casagrande, S., Geiss, L., and Cowie, C. C. 2015. "Prevalence of and Trends in Diabetes among Adults in the United States, 1988 2012," JAMA (314:10), pp. 1021 1029. Metzger, M. J., and Flanagin, A. J. 2013. "Credibility and Trust of Information in Online Environments: The Use of Cognitive Heuristics," Journal of Pragmatics (59), pp. 210 220. Michaelidou, N., and Dibb, S. 2006. "Product Involvement: An Application in Clothing," Journal of Consumer Behaviour (5:5), pp. 442 453. Michaelidou, N., and Dibb, S. 2008. "Consumer Involvement: A New Perspective," The Marketing Review (8:1), pp. 83 99. Milward, J., Day, E., Wadsworth, E., Strang, J., and Lynskey, M. 2015. "Mobile Phone Ownership, Usage and Readiness to Use by Patients in Drug Treatment," Drug and alcohol dependence (146), pp. 111 115. Mukherjee, A., Liu, B., and Glance, N. 2012. "Spotting Fake Reviewer Groups in Consumer Reviews," Proceedings of the 21st international conference on World Wide Web : ACM, pp. 191 200. Neuner, J., Fedders, M., Caravella, M., Bradford, L., and Schapira, M. 2015. "Meaningful Use and the Patient Portal Patient Enrollment, Use, and Satisfaction with Patient Portals at a Later Adopting Center," American Journal of Medical Quality (30:2), pp. 105 113. Ngo Ye, T. L., and Sinha, A. P. 2014. "The Influence of Reviewer Engagement Characteristics on Online Review Helpfulness: A Text Regression Model," Decision Support Systems (61), pp. 47 58. Noar, S. M., Harrington, N. G., Van Stee, S. K., and Aldrich, R. S. 2011. "T ailored Health Communication to Change Lifestyle Behaviors," American Journal of Lifestyle Medicine (5:2), pp. 112 122. Oliver, R. L., and Burke, R. R. 1999. "Expectation Processes in Satisfaction Formation a Field Study," Journal of Service Research (1:3 ), pp. 196 214. Or, C. K., and Karsh, B. T. 2009. "A Systematic Review of Patient Acceptance of Consumer Health Information Technology," Journal of the American Medical Informatics Association (16:4), pp. 550 560. Osborn, C., Mayberry, L., Mulvaney, S., and Hess, R. 2010. "Patient Web Portals to Improve Diabetes Outcomes: A Systematic Review," Current Diabetes Reports (10:6), pp. 422 435.

PAGE 111

102 Oshima Lee, E., and Emanuel, E. J. 2013. "Shared Decision Making to Improve Care and Reduce Costs," New England Journ al of Medicine (368:1), pp. 6 8. Ott, M., Choi, Y., Cardie, C., and Hancock, J. T. 2011. "Finding Deceptive Opinion Spam by Any Stretch of the Imagination," Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lan guage Technologies Volume 1 : Association for Computational Linguistics, pp. 309 319. Otte Trojel, T., de Bont, A., Aspria, M., Adams, S., Rundall, T. G., van de Klundert, J., and de Mul, M. 2015. "Developing Patient Portals in a Fragmented Healthcare System," International journal of medical informatics (84:10), pp. 835 846. Otte Trojel, T., de Bont, A., Rundall, T. G., and van de Klundert, J. 2014. "How Outcomes Are Achieved through Patient Portals: A Realist Review," Journal of the American Medical Informatics Association (21:4), pp. 751 757. Ownby, R. L., Hertzog, C., and Czaja, S. J. 2012. "Tailored Information and Automated Reminding to Improve Medication Adherence in Spanish and English Speaking Elders Treated for Memory Impairment," Clinical gerontologist (35:3), pp. 221 238. Padwal, R. S., Hemmelgarn, B. R., Khan, N. A., Grover, S., McKay, D. W., Wilson, T., Penner, B., Burgess, E., McAlister, F. A., and Bolli, P. 2009. "The 2009 Canadian Hypertension Education Program Recommendations for the Management of Hypertension: Part 1 Blood Pressure Measurement, Diagnosis and Assessment of Risk," Canadian Journal of Cardiology (25:5), pp. 279 286. Pagano, D., and Maalej, W. 2013. "User Feedback in the Appstore: An Empirical Study," Requirements Engineering Conference (RE), 2013 21st IEEE International : IEEE, pp. 125 134. Pang, B., and Lee, L. 2008. "Opinion Mining and Sentiment Analysis," Foundations and trends in information retrieval (2:1 2), pp. 1 135. Paschou, M., Nodarakis, N., Tsakalidis, A., and Sakkopoulos, E. 2013a. "Mobile Healthcare Systems: Generating Dynamic Smart phone Apps to Serve Multiple Medical Specializations," Future (10), p. 2. Paschou, M., Sakkopoulos, E., and Tsakalidis, A. 2013b. "Easyhealthapps: E Health Apps Dynamic Generation for Smartphones & Tablets," Journal of medical systems (37:3), pp. 1 12. P ellegrini, C. A., Duncan, J. M., Moller, A. C., Buscemi, J., Sularz, A., DeMott, A., Pictor, A., Pagoto, S., Siddique, J., and Spring, B. 2012. "A Smartphone Supported Weight Loss Program: Design of the Engaged Randomized Controlled Trial," BMC public heal th (12:1), p. 1.

PAGE 112

103 Petty, R., and Cacioppo, J. T. 2012. Communication and Persuasion: Central and Peripheral Routes to Attitude Change Springer Science & Business Media. Petty, R. E. 2013. "Two Routes to Persuasion: State of the Art," International pe rspectives on psychological science (2), pp. 229 247. Petty, R. E., and Cacioppo, J. T. 1979. "Issue Involvement Can Increase or Decrease Persuasion by Enhancing Message Relevant Cognitive Responses," Journal of personality and social psychology (37:10), p. 1915. Petty, R. E., and Cacioppo, J. T. 1984. "The Effects of Involvement on Responses to Argument Quantity and Quality: Central and Peripheral Routes to Persuasion," Journal of personality and social psychology (46:1), p. 69. Petty, R. E., Cacioppo, J. T., and Kasmer, J. A. 2015. "The Role of Affect in the Elaboration Likelihood Model of Persuasion," Communication, Social Cognition, and Affect (PLE: Emotion) ), p. 117. Petty, R. E., Cacioppo, J. T., and Schumann, D. 1983. "Central and Peripheral Routes to Advertising Effectiveness: The Moderating Role of Involvement," Journal of consumer research ), pp. 135 146. Petty, R. E., Priester, J. R., and Brinol, P. 2002. "Mass M edia Attitude Change: Implications of the Elaboration Likelihood Model of Persuasion," Media effects: Advances in theory and research (2), pp. 155 198.2002. Pham, Q., Khatib, Y., Stansfeld, S., Fox, S., and Green, T. 2016. "Feasibility and Efficacy of an and Design Evaluation," Games for health journal (5:1), pp. 50 67. Phillips, G., Felix, L., Galli, L., Patel, V., and Edwards, P. 2010. "The Effectiveness of M Health Technologies for Improving Health and Health Services: A Systematic Review Protocol," BMC research notes (3:1), p. 250. Pickering, T. G., Hall, J. E., Appel, L. J., Falkner, B. E., Graves, J., Hill, M. N., Jones, D. W., Kurtz, T., Sheps, S. G., and Roccella, E. J. 20 05. "Recommendations for Blood Pressure Measurement in Humans and Experimental Animals Part 1: Blood Pressure Measurement in Humans: A Statement for Professionals from the Subcommittee of Professional and Public Education of the American Heart Association Council on High Blood Pressure Research," Hypertension (45:1), pp. 142 161. Pieterse, A. H., de Vries, M., Kunneman, M., Stiggelbout, A. M., and Feldman Stewart, D. 2013. "Theory Informed Design of Values Clarification Methods: A Cognitive Psychological P erspective on Patient Health Related Decision Making," Social science & medicine (77), pp. 156 163.

PAGE 113

104 Pitt, L. F., Watson, R. T., and Kavan, C. B. 1995. "Service Quality: A Measure of Information Systems Effectiveness," MIS quarterly ), pp. 173 187. Popken, B. 2010. "Ways You Can Spot Fake Online Reviews," The Consumerist ). Price, M., Yuen, E. K., Goetter, E. M., Herbert, J. D., Forman, E. M., Acierno, R., and Ruggiero, K. J. 2014. "Mhealth: A Mechanism to Deliver More Accessible, More Effective Ment al Health Care," Clinical psychology & psychotherapy (21:5), pp. 427 436. Ramanathan, N., Swendeman, D., Comulada, W. S., Estrin, D., and Rotheram Borus, M. J. 2013. "Identifying Preferences for Mobile Health Applications for Self Monitoring and Self Mana gement: Focus Group Findings from Hiv Positive Persons and Young Mothers," International journal of medical informatics (82:4), pp. e38 e46. Randeree, E. 2009. "Exploring Technology Impacts of Healthcare 2.0 Initiatives," Telemedicine and e Health (15:3), pp. 255 260. Romano, M. J., and Stafford, R. S. 2011. "Electronic Health Records and Clinical Decision Support Systems: Impact on National Ambulatory Care Quality," Archives of Internal Medicine (171:10), pp. 897 903. Ross, S. E., Moore, L. A., Earnest, M. A., Wittevrongel, L., and Lin, C. T. 2004. "Providing a Web Based Online Medical Record with Electronic Communication Capabilities to Patients with Congestive Heart Failure: Randomized Trial," Journal of Medical Internet Research (6:2), p. e12. Rothma n, A. J. 2000. "Toward a Theory Based Analysis of Behavioral Maintenance," Health Psychology (19:1S), p. 64. Rust, R. T., Thompson, D. V., and Hamilton, R. W. 2006. "Defeating Feature Fatigue," Harvard business review (84:2), pp. 37 47. Sacher, R., and McPherson, R. A. 2000. Wildmann's Clinical Interpretation of Laboratory Tests (11 ed.). F.A. Davis Company. Salton, G., and Buckley, C. 1988. "Term Weighting Approaches in Automatic Text Retrieval," Information processing & management (24 :5), pp. 513 523. Salton, G., Yang, C. S., and Yu, C. T. 1975. "A Theory of Term Importance in Automatic Text Analysis," Journal of the American society for Information Science (26:1), pp. 33 44. Samoocha, D., Bruinvels, D. J., Elbers, N. A., Anema, J. R ., and van der Beek, A. J. 2010. "Effectiveness of Web Based Interventions on Patient Empowerment: A Systematic Review and Meta Analysis," Journal of Medical Internet Research (12:2).

PAGE 114

105 Schacter, D. L., Gilbert, D. T., and Wegner, D. M. 2010. Psychology Wor th Publishers. Senecal, S., Kalczynski, P. J., and Nantel, J. 2005. "Consumers' Decision Making Process and Their Online Shopping Behavior: A Clickstream Analysis," Journal of Business Research (58:11), pp. 1599 1608. Sherer, S. A. 2014. "Patients Are No Communications of the Association for Information Systems (34:1), p. 17. Shin, C., Hong, J. H., and Dey, A. K. 2012. "Understanding and Prediction of Mobile Application Usa ge for Smart Phones," Proceedings of the 2012 ACM Conference on Ubiquitous Computing : ACM, pp. 173 182. Shorvon, S. D. 2010. Handbook of Epilepsy Treatment John Wiley & Sons. Sillence, E., Briggs, P., Harris, P. R., and Fishwick, L. 2007. "How Do Pat ients Evaluate and Make Use of Online Health Information?," Social science & medicine (64:9), pp. 1853 1862. Spriensma, G. J. 2012. "Year in Review," Retrieved July (10), p. 2013. Stapel, D. A., Koomen, W., and Velthuijsen, A. S. 1998. "Assimilation or C ontrast?: Comparison Relevance, Distinctness, and the Impact of Accessible Information on Consumer Judgments," Journal of Consumer Psychology (7:1), pp. 1 24. Starrels, J. L., Becker, W. C., Alford, D. P., Kapoor, A., Williams, A. R., and Turner, B. J. 2010. "Systematic Review: Treatment Agreements and Urine Drug Testing to Reduce Opioid Misuse in Patients with Chronic Pain," Annals of internal medicine (152:11) pp. 712 720. Stewart, K. J., and Malaga, R. A. 2009. "Contrast and Assimilation Effects on Consumers' Trust in Internet Companies," International Journal of Electronic Commerce (13:3), pp. 71 94. Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., and Mani, M. 2015. "Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps," JMIR mHealth and uHealth (3:1), p. e27. Streitfeld, D. 2012a. "Buy Revie ws on Yelp, Get Black Mark," New York Times. http://www nytimes. com/2012/10/18/technology/yelp tries to halt deceptive reviews. html ). Streitfeld, D. 2012b. "Fake Reviews, Real Problem," New York Times (10). Susan, D., KIRBY, J. R., UREDA RANDALL, L., and ROSE, J. H. 1998. "Peripheral Cues and Involvement Level: Influences on Acceptance of a Mammography Message," Journal of Health Communication (3:2), pp. 119 135.

PAGE 115

106 Svedic, Z. 2015. "The Effect of Informational Signals on Mobile Apps Sales Ranks across the Globe." Beedie School of Business Faculty: Segal Graduate School. Taboada, M., Brooke, J., Tofiloski, M., Voll, K., and Stede, M. 2011. "Lexicon Based Methods for Sentiment Analysis," Computational linguistics (37:2), pp. 267 307. Tallon, P. P., Ramirez, R. V., and Short, J. E. 2013. "The Information Artifact in It Governance: Toward a Theory of Information Governance," Journal of Management Information Systems (30:3), pp. 141 178. Terry, M. 2010. "Medical Apps for Smartphone s," Telemed JE Health (16:1), pp. 17 22. Thomas, N. J. 1999. "Are Theories of Imagery Theories of Imagination?: An Active Perception Approach to Conscious Mental Content," Cognitive science (23:2), pp. 207 245. Thompson, D. V., Hamilton, R. W., and Rust, R. T. 2005. "Feature Fatigue: When Product Capabilities Become Too Much of a Good Thing," Journal of marketing research (42:4), pp. 431 442. Tian, Y., Nagappan, M., Lo, D., and Hassan, A. E. 2015. "What Are the Characteristics of High Rated Apps? A Case Study on Free Android Applications," Software Maintenance and Evolution (ICSME), 2015 IEEE International Conference on : IEEE, pp. 301 310. Tsiknakis, M., and Kouroubali, A. 2009. "Organizational Factors Affecting Successful Adoption of Innovative Ehealth Services: A Case Study Employing the Fitt Framework," International journal of medical informatics (78:1), pp. 39 52. Tuch, A. N., Presslaber, E. E., StCklin, M., Opwis, K., and Bargas Avila, J. A. 2012. "The Role of Visual Complexity and Prototypicality Regarding First Impression of Websites: Working Towards Understanding Aesthetic Judgments," International Journal of Human Computer Studies (70:11), pp. 794 811. Tudur Smith, C., Marson, A., and Williamson, P. 2001. "Phenytoin Versus Valproate Monotherap y for Partial Onset Seizures and Generalized Onset Tonic Clonic Seizures," Cochrane Database Syst Rev (4). Usoro, A., Shoyelu, S., and Kuofie, M. 2010. "Task Technology Fit and Technology Acceptance Models Applicability to E Tourism," Journal of Economic Development, Management, IT, Finance, and Marketing (2:1), p. 1. Valderas, J. M., Starfield, B., Sibbald, B., Salisbury, C., and Roland, M. 2009. "Defining Comorbidity: Implications for Understanding Health and Health Services," The Annals of Family Medicine (7:4), pp. 357 363.

PAGE 116

107 Van den Akker, M., Buntinx, F., Metsemakers, J. F., Roos, S., and Knottnerus, J. A. 1998. "Multimorbidity in General Practice: Prevalence, Incidence, and Determinants of Co Occurring Chronic and Recurrent Diseases," Journal of clinical epidemiology (51:5), pp. 367 375. van Velsen, L., Beaujean, D. J., and van Gemert Pijnen, J. E. 2013. "Why Mobile Health App Overload Drives Us Crazy, and How to Restore the Sanity," BMC medical informatics and decision making (13:1), p. 1. van Walraven, C., Wong, J., and Forster, A. J. 2012. "Lace+ Index: Extension of a Validated Index to Predict Early Death or Urgent Readmission after Hospital Discharge Using Administrative Data," Open Medicine (6:3), p. e80. Viswanathan, M., Golin, C. E., Jo nes, C. D., Ashok, M., Blalock, S. J., Wines, R. C., Coker Schwimmer, E. J., Rosen, D. L., Sista, P., and Lohr, K. N. 2012. "Interventions to Improve Adherence to Self Administered Medications for Chronic Diseases in the United States: A Systematic Review, Annals of internal medicine (157:11), pp. 785 795. VonHoltz, L. A. H., Hypolite, K. A., Carr, B. G., Shofer, F. S., Winston, F. K., Hanson, C. Academic Emergency Medicine ( 22:6), pp. 765 768. Vukovac, D. P., Novosel Speech Language Pathology Services," 24th International Conference on Information Systems Development Wagenaar, W. A., Keren, G., and Lichtenste in, S. 1988. "Islanders and Hostages: Deep and Surface Structures of Decision Problems," Acta Psychologica (67:2), pp. 175 189. Walid, M. S., Newman, B. F., Yelverton, J. C., Nutter, J. P., Ajjan, M., and Robinson, J. S. 2010. "Prevalence of Previously Un known Elevation of Glycosylated Hemoglobin in Spine Surgery Patients and Impact on Length of Stay and Total Cost," Journal of Hospital Medicine (5:1), pp. E10 E14. Walker, J. L. 1994. Assimilation/Contrast Effects: The Effect of Expectations across the Go ods/Services Continuum Wayne, N., and Ritvo, P. 2014. "Smartphone Enabled Health Coach Intervention for People with Diabetes from a Modest Socioeconomic Strata Community: Single Arm Longitudinal Feasibility Study," Journal of Medical Internet Research ( 16:6), p. e149. Weingart, S. N., Rind, D., Tofias, Z., and Sands, D. Z. 2006. "Who Uses the Patient Internet Portal? The Patientsite Experience," Journal of the American Medical Informatics Association (13:1), pp. 91 95.

PAGE 117

108 Weingarten, S. R., Henning, J. M., Badamgarav, E., Knight, K., Hasselblad, V., Gano Jr, A., and Ofman, J. J. 2002. "Interventions Used in Disease Management Programmes for Patients with Chronic Illnesswhich Ones Work? Meta Analysis of Published Reports," Bmj (325:7370), p. 925. White, H. 1980. A Heteroskedasticity Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity Wilhide III, C. C., Peeples, M. M., and Kouyat, R. C. A. 2016. "Evidence Based Mhealth Chronic Disease Mobile App Intervention Design: Development of a Framework," JMIR research protocols (5:1). Wilson, E. V., Wang, W., and Sheetz, S. D. 2014. "Underpinn ing a Guiding Theory of Patient Centered E Health," Communications of the Association for Information Systems (34:1), p. 16. Wooldridge, J. M. 2001. "Applications of Generalized Method of Moments Estimation," The Journal of Economic Perspectives (15:4), p p. 87 100. Worringham, C., Rojek, A., and Stewart, I. 2011. "Development and Feasibility of a Smartphone, Ecg and Gps Based System for Remotely Monitoring Exercise in Cardiac Rehabilitation," PloS one (6:2), p. e14669. WSJ. 2013. "The Experts: Have We En tered a Period Where People Are Screening Too Much for Disease?," in: Wall Street Journal New York: Dow Jones & Company. Wu, L., Li, J. Y., and Fu, C. Y. 2011. "The Adoption of Mobile Healthcare by Hospital's Professionals: An Integrative Perspective ," Decision Support Systems (51:3), pp. 587 596. Wu, M., Wang, L., Li, M., and Long, H. 2015a. "Alleviating Feature Fatigue in Product Development Based on the Bass Model," Journal of Engineering, Design and Technology (13:3), pp. 350 368. Wu, M., Wang, L., Long, H., and Li, M. 2015b. "Feature Fatigue Analysis in Product Development," Total Quality Management & Business Excellence (26:1 2), pp. 218 232. Wu, S., Chaudhry, B., Wang, J., Maglione, M., Mojica, W., Roth, E., Morton, S. C., and S hekelle, P. G. 2006. "Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care," Annals of internal medicine (144:10), pp. 742 752.

PAGE 118

109 Xie, S., Wang, G., Lin, S., and Yu, P. S. 2012. "Review Spam Detection Via Temporal Pattern Discovery," Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining : ACM, pp. 823 831. Yang, J. J., Li, J., Mulder, J., Wang, Y., Chen, S., Wu, H., Wang, Q., and Pan, H. 2015. "Emerging Infor mation Technologies for Enhanced Healthcare," Computers in Industry (69), pp. 3 11. Yi, Y. J., You, S., and Bae, B. J. 2016. "The Influence of Smartphones on Academic Performance: The Development of the Technology to Performance Chain Model," Library Hi T ech (34:3), pp. 480 499. Online Learning Systems: An Empirical Study Combining the Task Technology Fit Model with the Theory of Planned Behaviour," British Journ al of Educational Technology (41:6), pp. 1003 1017. Access and Use in a Medically Underserved Population: Implications for Providing Enhanced Health Information Ser vices," Health Information & Libraries Journal (29:1), pp. 61 71. Zapata, B. C., Fernndez Alemn, J. L., Idri, A., and Toval, A. 2015. "Empirical Studies on Usability of Mhealth Apps: A Systematic Literature Review," Journal of medical systems (39:2), pp. 1 19. Zhang, T., Wang, J., Liu, P., and Hou, J. 2006. "Fall Detection by Embedding an Accelerometer in Cellphone and Using Kfd Algorithm," International Journal of Computer Science and Network Security (6:10), pp. 277 284. Zigurs, I., and Buc kland, B. K. 1998. "A Theory of Task/Technology Fit and Group Support Systems Effectiveness," MIS quarterly ), pp. 313 334.

PAGE 119

110 A PPENDIX A Focal and Comorbid Alerts for Diabetes and CKD Diagnosis of diabetes at an early stage is important, as the economic burden of diabetes is huge in the United States, with 7 % of the United States population being affected by diabetes, with an estimated $245 billion annual burden in 2012 (American Diabetes Association 2013) Similarly, CKD involves damage to the kidney characterized by a gradual loss of kidney function over time. The cause of CKD can be due to inflammation or damage s filtering units, cysts, immune deficiency, or high blood sugar and pressure. CKD may develop complications like high blood pressure, anemia (low blood count), weak bones, poor nutritional health and nerve damage. Also, kidney disease increases the risk o f having heart and blood vessel disease. These problems may happen slowly over a long period of time Detection, treatment and prolonged management can often keep chronic kidney disease from getting worse. When kidney disease progresses, it may eventually lead to kidney failure, which requires dialysis or a kidney transplant to maintain life. Early diagnosis and intervention for both diabetes and CKD may aid in patients to adopt medication and lifestyle changes to return their lifestyle to normal (Bloomgarden 2004; Hussain et al. 2007) We consider Glycated hemoglobin test (commonly known as A1C, but often referred as HbA1c, or Hb1c) conducted to diagnose a pre diabetic or diabetic condition as the focal alert for this study. Glycated hemoglobin is a form of hemoglobin measured to identify the average plasma glucose concentration over time. The t est is recommended for examination of blood sugar c ontrol in pre diabetic and patients with more elevated levels (American Diabetes Association 2007) High levels of A1C may also indicate risk of hypoglycemia or advanced stage of diabetes that may result in other health disease and issues, such as renal

PAGE 120

111 failure. Normal levels of glucose produce a normal amount of Glycated hemoglobin, and as the average amount of plasma glucose increases Glycated hemoglobin. This serves as a marker for average blood glucose levels over the previous months prior to the measurement. Existing studies show that there are a significant proportion of people who are unaware of their elevated Glycated hemoglobin level before they have any relevant blood lab tests (Walid et al. 2010) Although other tests may reveal the blood glucose content, such as a fasting blood glucose test, the A1C tests reveal better information on the glycemic behavior than other tests and are vital in making treatment choices The American Diabetes Associa tion (ADA) guidelines advi s e that the Glycated hemoglobin test is performed at least twice a year in patients with suspected pre diabetic or with diabetic conditions trying to maintain a stable glycemic level with other treatment goals. However, ADA prescribed A1C tests to be done quarterly for patients with diabetes whose treatment is not mee ting glycemic goals (American Diabe tes Association 2007) Therefore, providers often prescribe or advise the A1C tests to be done in 90 days and/or renew the alerts given again after 90 days. For Comorbid Alerts, we consider a set of alerts for four other diseases that are commonly pr escribed : cardiovascular and blood diseases, chronic pain and intoxication, epilepsy a nd thyroid diseases. In Table 13 we provide the alerts associated with these diseases and description of these diseases. A detailed description for these alerts are be yond the scope of this study and overall, these diseases reflect on a comorbid health condition of a patient along with diabetes. We consider Glomerular filtration rate test (commonly known as GFR, conducted to determine how much kidney function a person has as the focal alert for this study. The test is conducted by a calculation based on the patient creatinine level, age, race, ge nder and few

PAGE 121

112 other factors. The test is conducted throughout every stage of CKD. Changes in the GFR will give an indication on how fast or slow the CKD is progressing. Table 13 : Comorbid Alerts for Diabetes Table 14 : Comorbid Alerts for CKD Disease Alerts Description References 1 Cardiovascular and blood diseases Potassium, Lipid Profile, Complete blood count (CBC), Ferritin, Lipid Profile Diseases that involve the heart, blood, and the blood vessels (Fauci 2008; Lavonas 2013; Long and Dagogo Jack 2011; Padwal et al. 2009) 2 Chronic pain and intoxication Opioid dot or therapy monitoring, Urine tests for intoxication, Carbamazepine Addiction to prescription painkillers, alcohol, or other drugs (Barth et al. 2014; Heit and Gourlay 2004; Lavonas 2013; Starrels et al. 2010) 3 Epilepsy Phenytoin Level, Valproic Acid, Phenobarbital Brain disorder in which a person has multiple seizures (Shorvon 2010; Tudur Smith et al. 2001) 4 Thyroid disease Thyroid stimulating hormone (TSH), Carbamazepine Thyroid with impaired functionality (L avonas 2013; Sacher and McPherson 2000) Disease Alerts Description References 1 High blood pressure Blood pressure (BP), Electrocardiogram (ECG) ECG: tests that detect the electrical activity of the heart and records. (Casale et al. 1986; Pickering et al. 2005) 2 Cardiovascular disease (CVD) Lipid panel, Potassium Complete blood count (CBC), indicates the risk of having a heart attack or other heart disease (Fauci 2008; Lavonas 2013; Padwal et al. 2009) 3 Diabetes Glycated hemoglobin (A1C) Indicates average blood sugar level for the past two to th ree months. (Hussain et al. 2007; Long and Dagogo Jack 2011; Walid et al. 2010)

PAGE 122

113 A PPENDIX B Detailed Estimation Results Table 15 : Linear Estimation Models with Details Results VARIABLES Diabetes Sample CKD Sample 1 2 3 4 OLS OLS OLS OLS Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Number of Alert Clicks (CLICKS) 0.090*** (0.040) 0.011*** (0.004) 0.079*** (0.002) 0.013*** (0.005) Number of Comorbid Alerts (COMBD) 1.484** (0.770) 1.328*** (0.046) 1.189*** (0.046) 0.998** (0.009) CLICKS COMBD 0.321*** (0.020) 0.191** (0.003) Number of Contiguous Alerts (CONT) 0.131 (0.097) 0.297 (0.161) 1.660 (0.161) 0.025 (0.097) LACE 0.009 (0.005) 0.006 (0.006) 0.012 (0.005) 0.004 (0.002) LOS 0.020 (0.022) 0.024 (0.017) 0.051 (0.019) 0.035 (0.020) ACUITY 0.017 (0.014) 0.007 (0.005) 0.102 (0.071) 0.006 (0.004) ZIP 0.001 (0.000) 0.000 (0.000) 0.000 (0.000) 0.000 (0.000) REC_ADM 0.060 (0.055) 0.076 (0.044) 0.45 (0.039) 0.091 (0.067) ETHN 0.056 (0.005) 0.078 (0.076) 0.018 (0.005) 0.055 (0.002) HMO 0.020 (0.022) 0.024 (0.017) 0.051 (0.019) 0.035 (0.020) PPO 0.010 (0.008) 0.013 (0.005) 0.102 (0.071) 0.006 (0.004) POS 0.001 (0.000) 0.005 (0.001) 0.003 (0.001) 0.008 (0.004) AGE 0.015 (0.005) 0.001 (0.000) 0.010 (0.008) 0.065 (0.005) GENDER 0.131 (0.054) 0.102 (0.028) 0.167 (0.088) 0.189 (0.121) MEDICARE 0.019 (0.049) 0.023 (0.061) 0.005 (0.001) 0.009 (0.001) MEDICAID 0.118 (0.093) 0.300** (0.137) 0.203 (0.083) 0.156 (00.97) Constant 5.289*** (0.142) 5.946*** (0.212) 8.312*** (0.543) 9.510** (0614) Observations 2,164 2,164 8,473 8,473 R square 0.424 0.433 0.312 0.289 Adjusted R square 0.380 0.361 0.269 0.263 Chi squared 35.65*** 25.21*** 44.65*** 23.5*** F test 19.78*** 22.24*** *** p<0.01, ** p<0.05, p<0.1. Robust standard errors in parentheses ; Standard errors are adjusted for ethnonationalism groups: 60 clust ers in diabetes sample, and 54 clusters in CKD sample. Dropping contiguous alerts from models does not change the results.

PAGE 123

114 Table 16 : Parametric and Semi Parametric Estimation Models with Detailed Results Diabetes CKD 1 2 3 4 5 6 7 8 Weibull regression Cox proportional Weibull regression Cox proportional Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) Test Closure Time (TCT) VARIABLES Coefficients Marginal effect Hazard rates Marginal effect Coefficients Marginal effect Hazard rates Marginal effect Number of Alert Clicks( CLICKS) 1.247*** (0.137) 0.543*** (0.245) 0.237*** (0.030) 0.124*** (0.053) 1.101** (0.061) 0.493** (0.231) 0.201** (0.045) 0.091** (0.031) Number of Comorbid Alerts (COMBD) 1.152** (0.166) 0.631*** (0.321) 0.021*** (0.057) 0.011*** (0.004) 1.141** (0.054) 0.701** (0.321) 0.019*** (0.089) 0.009*** (0.003) CLICKS COMBD 1.332** (0.145) 0.691*** (0.351) 0.032*** 0.011 0.021*** (0.013) 1.060** (0.054) 0.517** (0.219) 0.023** (0.045) 0.015** (0.006) CONT 1.003 (0.057) 0.410 (0.110) 0.021 (0.018) 0.010 (0.004) 0.970 (0.091) 0.061 (0.033) 0.052 (0.038) 0.001 (0..000) LACE 1.016 (0.017) 0.210 (0.121) 0.018 (0.017) 0.006 (0.001) 1.560 (0.543) 0.315 (0.102) 0.010 (0.010) 0.004 (0.001) LOS 1.008 (0.048) 0.043 (0.021) 0.000 (0.000) 0.000 (0.000) 2.102 (0.087) 0.789 (0.623) 0.001 (0.000) 0.000 (0.000) ACUITY 0.954 (0.035) 0.321 (0.143) 0.053 (0.037) 0.001 (0.000) 1.002 (0.876) 0.090 (0.005) 0.950 (0.071) 0.069 (0.0178) REC_ADM 0.060 (0.055) 0.010 (0.001) 0.076 (0.044) 0.035 (0.019) 0.450 (0.039) 0.168 (0.059) 0.091 (0.067) 0.038 (0.017) HMO 0.122 (0.087) 0.021 (0.017) 0.213 (0.095) 0.056 (0.024) 0.176 (0.105) 0.032 (0.001) 0.166 (0.121) 0.087 (0.065) PPO 0.060 (0.047) 0.029 (0.013) 0.087 (0.005) 0.019 (0.009) 0.100 (0.064) 0.049 (0.014) 0.057 (0.034) 0.013 (0.004) POS 0.009 (0.005) 0.001 (0.000) 0.012 (0.001) 0.004 (0.001) 0.013 (0.001) 0.006 (0.002) 0.007 (0.004) 0.001 (0.000) AGE 1.003 (0.002) 0.041 (0.028) 0.003* (0.002) 0.001 (0.000) 1.765 (0.882) 0.801 (0.039) 0.010 (0.007) 0.007 (0.001) GENDER 0.501 (0.044) 0.027 (0.013) 0.056 (0.005) 0.019 (0.007) 2.994 (0.432) 0.678 (0.347) 0.091 (0.011) 0.037 (0.011) MEDICARE 0.992 (0.120) 0.032 (0.001) 0.017 (0.016) 0.008 (0.001) 1.671 (0.546) 0.498 (0.109) 0.005 (0.001) 0.001 (0.000) MEDICAID 0.801 (0.131) 0.129 (0.056) 0.010 (0.009) 0.003 (0.001) 1.090 (0.879) 0.399 (0.185) 0.057 (0.014) 0.018 (0.009) Frailty variance 3.12** (10.0) 3.51** (10.0) shape parameter) 3.33* (11.6) 3.61* (12.8) R square 0.398 0.356 0.298 0.287 Adj R square 0.390 0.350 0.295 0.275 Chi squared 26.15*** 24.29*** 23.98*** 21.17*** Observations 2,164 2,164 8,473 8,473 *** p<0.01, ** p<0.05, p<0.1 Robust standard errors in parentheses Standard errors are adjusted for ethnonationalism groups: 60 clusters in diabetes sample, and for 54 clusters in CKD sample. The marginal effects are evaluated at the sample mean. The marginal effects for the dummy variable are not reported

PAGE 124

115 Table 17 : Summary Statistics of Sample for Survivor Models Diabetes CKD Subjects tracked From 1 to 90 periods (days). From 1 to 105 periods (days). Event A1C test (event) or do not (censored) GFR test (event) or do not (censored) Number of subjects 2,164 8,473 Times at risk (periods summed over the subjects) 130,282.89 213,005.61 Number of failures 1,490 5,349 Incidence rate is 1% (The number of failures divided by the time at risk). 2.5% (The number of failures divided by the time at risk). Table 18 : Log rank Test for Equality of Survivor Functions, Compared by Comorbid Alerts Diabetes CKD Comorbid Alerts Observed Event Expected Events Observed Event Expected Events No 1,359 1,333.51 5,214 4,921.27 Yes 805 830.49 3,259 3,551.73 Total 2,164 2,164 8,473 8,473 Table 19 : Ethno Nationality Distribution for Diabetes P atients Ethnicity/Nationa lity Percentage Diabetes Sample Percentage in CKD Sample Ethnicity/Nationa lity Percentage Diabetes Sample Percentag e in CKD Sample 1 Afghanistan 0.00% 0.06% 42 Hispanic/Latino Mexican 6.10% 10.87% 2 United States 60.49% 72.09% 43 Hispanic/Latino Other 7.67% 0.00% 3 Argentinean 0.05% 0.00% 44 Hispanic/Latino South Am 4.71% 0.20% 4 Argentinian 0.00% 0.67% 45 Hungarian 0.09% 0.06% 5 Australian 0.05% 0.00% 46 Indian 0.23% 4.07% 6 Austrian 0.00% 0.05% 47 Iranian 0.00% 0.09% 7 Belgian 0.05% 0.00% 48 Irish 0.88% 0.72% 8 Belgium 0.00% 0.19% 49 Israel 0.05% 0.09% 9 Bosnian 0.00% 0.08% 50 Italian 0.28% 0.40% 10 Bosnian/Herzegovi nian 0.05% 0.00% 51 Jamaican 0.00% 0.35% 11 Brazilian 0.00% 0.99% 52 Japanese 0.14% 0.17% 12 British 0.00% 0.47% 53 Korean 0.23% 0.09% 13 British Isles/British Isle 0.00% 0.02% 54 Latin American Indian 0.05% 0.00% 14 Bulgarian 0.00% 0.13% 55 Lithuanian 0.05% 0.00% 15 Cambodian 0.09% 0.00% 56 Mexican American 0.14% 0.00%

PAGE 125

116 Ethnicity/Nationa lity Percentage Diabetes Sample Percentage in CKD Sample Ethnicity/Nationa lity Percentage Diabetes Sample Percentag e in CKD Sample Indian 16 Cameroonian/Cam eroon 0.05% 0.00% 57 New Zealand 0.00% 0.59% 17 Canadian 0.05% 0.22% 58 Nigerian 0.09% 0.40% 18 Canadian American Indian 0.00% 0.18% 59 Norwegian 0.18% 0.15% 19 Celtic 0.05% 0.00% 60 Pakistani 0.60% 0.00% 20 Cherokee 0.14% 0.00% 61 Peruvian 0.79% 0.05% 21 Chinese 0.14% 0.17% 62 Pole/Polish 0.32% 0.28% 22 Costa Rican 0.00% 0.02% 63 Portuguese 0.00% 0.11% 23 Croatian 0.05% 0.00% 64 Puerto Rican 0.37% 0.12% 24 Cuban 0.00% 0.13% 65 Russian 0.23% 0.11% 25 Czech 0.00% 0.07% 66 Scandinavian 0.05% 0.00% 26 Czechoslovakian 0.00% 0.08% 67 Scotch Irish 0.18% 0.00% 27 Dane/Danish 0.05% 0.04% 68 Scottish 0.23% 0.09% 28 Dutch 0.09% 0.00% 69 Slovenian 0.05% 0.00% 29 English 0.83% 0.78% 70 Spanish 1.52% 0.00% 30 Eritrean 0.05% 0.00% 71 Spanish American Indian 0.28% 0.00% 31 Ethiopian 0.00% 0.02% 72 Sudanese 0.00% 0.02% 32 Philippine 0.09% 0.09% 73 Swedish 0.14% 0.00% 33 Finnish 0.00% 0.07% 74 Swiss 0.00% 0.04% 34 French 0.18% 0.08% 75 Syrian 0.28% 0.00% 35 French Canadian 0.09% 0.00% 76 Tagalog 0.23% 0.00% 36 German 1.94% 1.77% 77 Taiwanese 0.05% 0.00% 37 Ghanaian 0.09% 0.09% 78 Tanzanian 0.00% 0.09% 38 Greek 0.42% 0.11% 79 Thai 0.42% 0.00% 39 Hawaiian/Native Hawaiian 1.25% 0.00% 80 Turkish 0.51% 0.00% 40 Hispanic/Latino Caribbean Latino 2.31% 0.07% 81 Ukrainian 0.18% 0.00% 41 Hispanic/Latino Central American 3.93% 0.15% 82 Vietnamese 0.14% 0.00% 42 Hispanic/Latino Mexican 6.10% 10.87% 83 Welsh 0.00% 0.12%

PAGE 126

117 A PPENDIX C GMM Estimation Approach Table 20 : Results of Estimation Models GMM VARIABLES GMM Estimation (1) (2) (3) Effectiveness Ease of Use Evaluation Instructive 0.18 0 *** (0.013) 0.17 1 *** (0.02 3 ) Integrative 0.20 1 *** (0.02 4 ) 0.1 27 *** (0.03 0 ) Functionalities 0.20 1 *** (0.03 0 ) 0.08 3 *** (0.02 1 ) Effectiveness 0.9 70 *** (0.03 1 ) Ease of Use 0.180 *** (0.01 5 ) Age 0.031 (0.015) 0.0 29 (0.0 20 ) 0.047 (0.021) Price 0.001 (0.001) 0.014 (0.00 3 ) 0.006 (0.002) Download 0.001 (0.001) 0.001 (0.001) 0.001 (0.001) App last update 0.007 (0.005) 0.024 (0.016) 0.011 ( 0.0 05 ) Control variables Included Included Included Constant 0.2 00 *** (0.041) 0. 5 1 0 *** (0.087) 2.512 *** (0.068) Number of apps 185 185 185 Number of observations 2,215 2,215 2,215 (1) Significance levels: ***p < 0.01, **p < 0.05, *p < 0.10 (2) Standard errors in parentheses (3) reviews in the market, total rating of the developer, price of the app, when the app was last updated when the app was introduced in the app store, number of downloads of the app, when the publisher released their first app in the app market

PAGE 127

11 8 A PPENDIX D Interaction Graphs Figure 17 : Plot for the interaction effect of functionalities and health effectiveness Explanation: At high levels of health effectiveness the association between app evaluation and number of functionalities has a positive slope. Furthermore, at low levels of h ealth e ffectiveness, the association between app evaluation and number of functionalities has a less steep slope. Th is plot show s that the effect of functionalities on app evaluation is higher for apps with high effectiveness

PAGE 128

119 Figure 18 : Plot for the interaction effec t of integrative quality and health effectiveness Explanation: At high levels of health effectiveness the association between app evaluation and integrative qualities has a positive slope. Furthermore, at low levels of h ealth e ffectiveness, the association between app evaluation and integrative qualities has a negative slope. Th is plot show s that the effect of integrative qualities on app evaluation is higher for apps with high effectiveness

PAGE 129

120 Figure 19 : Plot for the inter action effect of instructive quality and health effectiveness Explanation: At high levels of health effectiveness the association between app evaluation and instructive qualities has a positive slope. Furthermore, at low levels of h ealth e ffectiveness, the association between app evaluation and number of functionalities has a less steep slope. Th is plot show s that the effect of instructive qualities on app evaluation is higher for apps with high Effectiveness

PAGE 130

121 A PPENDIX E Text analytics Text analytics i ncludes applications and algorithms for turning unstructured data in to a text format into a structured data to analyz e it using various statistical methods (Bai 2011) T ext analytics can s ometimes be confused with text data mining, but in the recent years the discipline ev olved and grew in a distinctive way to include natural language processing (NLP) techniques to enable researchers to extract topics and summarizing the content of unstruct ured data (Liu 2012; Taboada et al. 2011) Generally, there are two main approaches to th e problem of extracting sentiment from reviews The first approach is t he lexicon based (unsupervised) approach involves analyzing reviews based on the words and phrases in the text using natural language processing (NLP) and computational linguistic appro aches (CLA) (Pang and Lee 2008) In contrast, in the supervised approach, researchers train classifier algorithms based on predefined labels and synonyms of the review sentiment (Liu 2012; Taboada et al. 2011) Sentiment mining i s usually done using a lexicon based approach by software because of its efficiency and scalability (Bai 2011) In this study, we used SAS E nterprise which has also been used in past studies. Our data cleaning and variables coding followed the diagram below. W e ran the text analytic nodes before conducting any data cleaning action. Based on the result of Text Topic nodes, we created a Stop List for each week of data W e Used Interactive Filter Viewer to drop the synonymy manually according to its weight, frequency, and relevance We started our analysis by exploring data. We set the names of the apps as an identifier and combined both the reviews and the titles of the reviews as a text field. In the Text parsing step, we considered entities to be proper nouns and we excluded numbers and punctuation attributes.

PAGE 131

122 We also set SAS to ignore parts of speech like auxiliary, conjunction, determiner, interjection, part, preposition, and pronoun We set the text filter t o include a minimum number of five reviews ; we set the weighting frequency to be log and used entropy term weight, since the reviews are less than one paragraph. Then we added the stop and synonyms words to SAS. Figure 20 : Data Cleaning and Variables Coding

PAGE 132

123 Coding of Key Variables using Text Mining We applied a machine learning technique (a supervised approach) by training classifier algorithms. SAS E nterprise is a machine learning framework that includes rich libraries for data transformation and manipulation Before training the classifiers, the unwanted characters and punctuations were removed from the reviews. In order to use the by somewhat by columns. While the Term by classifie algorithms in past studies (Ngo Ye and Sinha 2014) only using the term frequency cannot distinguis h the reviews well (Cao et al. 2011) This is because one term that appears frequently in one review may also appear in other s For instance, in reviews, the health effectiveness of the app and some reviews that does not. To address this problem the term frequencies were transferred by TF IDF weighting (Term Frequency Inverse Document Frequency). TF IDF surges comparabl e to the number of times a word appears in a review but is counterweight by the occurrence of the word in all the reviews (Arazy and Woo 2007; Cao et al. 2011; Salton et al. 1975) The TF IDF has been reported to increase the information retrieval precision up to 70 percent when compared to the T erm by Frequency Matrix (Salton and Buckley 1988) Equation 1 shows th e standard TF IDF weighting (Arazy and Woo 2007; Cao et al. 2011; Salton et al. 1975) (1)

PAGE 133

124 In the above equation, is the weighted frequency of term i in document j is the frequency of term I in document j, = where N is the number of the documents, and is the frequency of term i in the documents. In the next step, 30 % of the total reviews were randomly selected to train and test the performance of the classifiers. Two researchers, well versed in healthcare research, were asked to read the reviews and examine the consistency between the text and the associated appeal and health effec believe the information provided in the review text is consistent with what the reviews imply. T he Cohen's kappa (Carletta 1996) inter rater reliability measure exceeds 0.7, indicating the probability of agreed understanding between coders is significantly high (Krippendorff 1980; Landis and Koch 1977a; Landis and Koch 1977b) Thus, the coders met to resolve the conflicting ratings un til the overall agreements converged. Further, the recommendation consistency label provided through the coding process were used for training and test the performance of classifiers in the machine learning the step We evaluate the performance of support vector machines and random forests classifiers to predict the recommendation consistency labels. These classifiers have been used to study the helpfulness of reviews (Ghose et al., 2010). To predict the recommendation consistency labels the reviews were r andomly split into training and test sets by using 30 % of the data for training the classifiers and 70 % to validate their performance. The TF IDF matrix w as used as the set of features to train the classifiers based on the training set. Further, the accura cy metric was used to evaluate the performance of the classifiers based on the test set. Our results show that the random forest algorithm, with 81 % accuracy, outperforms the SVM classifier (with 61% accuracy), which is consistent with results of the past study (Ghose et al., 2010). Finally, the

PAGE 134

125 model built upon random forest were used to predict the unknown recommendation consistency labels in the remaining reviews. Those reviews were later used to empirically test the research model using panel data analy sis Reviews example: Construct Low score High score Health App Effecivness weight like it said helped me very useful and handy works for me Health App Ease of Use waste twenty minutes per day super simple to easy to use Figure 21 : Coding of Key Variables using Text Mining

PAGE 135

126 A PPENDIX F Literature review of apps effectiveness and impact studies in healthcare Table 21 : Literature Review on App Functionalities/Effectiveness Citation Meth odology/ Design Independent Variables Dependent Variable/Mod erating Variable Relevant Finding (s) Disease/ Condition (Wayne and Ritvo 2014) This single arm pilot study; Smartphone based interventions over 6 month trial Using custom smartphone app with Health Coach functionalities glycosylated hemoglobin (HbA1c) Reduce the risks of type 2 diabetes, support for the feasibility of smartphone health coaching. T ype 2 D iabetes (Holmen et al. 2014) 3 arm prospective randomized trial M obile phone self ma nagement system HbA1c level A total of 151 participants were randomized, HbA1c level decreased in all groups T ype 2 D iabetes (Kirwan et al. 2012) Matched Case control trial; Intervention group and control group Using an App as a physical activity intervention Usefulness of the Apps Use of the Apps a ssociated with an increased physical activity during intervention Physical activity (Charpen tier et al. 2011) Patient trial; adult patients with type 1 diabetes in parallel groups at multicenter. Using smartphones for recommendin g insulin doses. Using mobile health app with logbook, Using mobile health app with logbook and teleconsultatio n Change in the HbA1C Over six months, the system gave a 0.91% improvement for test groups over control groups. However, there was a reduction o f 0.67% when used without teleconsultation Diabetes (Worring ham et al. 2011) Patient trial ; mo nitored exercise assessment in cardiac patients. Pre and post intervention Using smartphone based App that manage walking based cardiac rehabilitation Cardiac D epression and Quality of Life Participants walking significantly fu rther on the post intervention. Rehabilitat ion

PAGE 136

127 Citation Meth odology/ Design Independent Variables Dependent Variable/Mod erating Variable Relevant Finding (s) Disease/ Condition using smartphone application. (Fukuok a et al. 2010) Pre post study: Using a pedomet er app Health status, self efficacy and BMI Daily prompts were delivered to participants during the intervention by a mobile app The intervention appeared to motivate to increase physical activity. Average daily total steps increased by 15% over three weeks, resulting in lower BMI. Physical activity (Zhang et al. 2006) Experiments; 32 volunteers, 12 elders (age 60 80) and 20 younger (age 20 39) Microsensors with an ambulatory device (tri axial accelerometer in a cell phone with an application for preprocessing) Fall detection Sy stem can detect the falls effect ively and make less disturbance Senior citizens in independe nt living (Glynn et al. 2013) A two group, parallel randomized controlled trial. Daily step count s between baseline and follow up over eight weeks. Using a smartphone application Results are not reported Physical activity (Pellegri ni et al. 2012) Randomized controlled trial; designed to examine the efficiently of smartphone supported weight loss program. Weight loss. We ight loss intervention with mobile technologies. Results are not reported Obesity (Marshal l et al. 2 008) Patient trial Using smartphone based application Adherence and self management during Results are not reported chronic respiratory

PAGE 137

128 Citation Meth odology/ Design Independent Variables Dependent Variable/Mod erating Variable Relevant Finding (s) Disease/ Condition that manage pulmonary rehabilitation

PAGE 138

129 A PPENDIX G Detailed Results of Panel Estimation Models Table 22 : Results of Panel Estimation Models VARIABLES Fixed Effect Models (1) (2) (3) Effect Ease Evaluation Integrative 0.181*** (0.014) 0.176*** 0.024 Instructive 0.207*** 0.026 0.137*** 0.034 Fun 0.209*** 0.032 0.089*** (0.026) Effect 0.989*** 0.037 Ease 0.189*** 0.016 Age 0.038** (0.016) 0.036 0.033 0.054** (0.025) Last updates 0.009 (0.011) 0.024 (0.023) 0.011 (0.017) App price 0.001 (0.002) 0.016*** 0.004 0.006** (0.003) Free 0.009 (0.008) 0.193*** (0.021) 0.050*** (0.015) Reviews 0.000 (0.000) 0.000*** (0.000) 0.000** (0.000) Download 0.000 (0.000) 0.000 (0.000) 0.000 (0.000) pub_no_of_apps 0.005* (0.003) 0.015** (0.006) 0.011** (0.005) pub_sumofallratings 0.000 (0.000) 0.000 (0.000) 0.000*** (0.000) pub_avg_ratings 0.000 (0.000) 0.000 (0.000) 0.000*** (0.000) pub_sum_of_def_ratings 0.000 (0.000) 0.000 (0.000) 0.000 (0.000) pub_sum_of_reviews 0.000 (0.000) 0.000 (0.000) 0.000* (0.000) pub_avg_reviews 0.000 (0.000) 0.000 (0.000) 0.000 (0.000) pub_sum_def_reviews 0.000 (0.000) 0.000 (0.001) 0.000 (0.000) pub_tools_cat 0.006 0.011 0.073

PAGE 139

130 VARIABLES Fixed Effect Models (1) (2) (3) Effect Ease Evaluation (0.048) (0.120) (0.082) pub_productivity_cat 0.097*** (0.021) 0.009 (0.052) 0.079** (0.036) pub_photo_cat 0.039 (0.059) 0.034 (0.126) 0.022 (0.092) pub_business_cat 0.109*** (0.029) 0.040 (0.071) 0.143*** (0.049) pub_health_fit_cat 0.001 (0.026) 0.012 (0.061) 0.025 (0.043) pub_lifestyle_cat 0.091* (0.054) 0.050 (0.115) 0.077 (0.083) pub_travel_cat 0.022 (0.039) 0.029 (0.098) 0.000 (0.067) pub_entertain_cat 0.030 (0.023) 0.017 (0.058) 0.057 (0.040) pub_finance_cat 0.133*** (0.036) 0.084 (0.092) 0.120* (0.063) pub_total_free_apps 0.013* (0.007) 0.018 (0.016) 0.020* (0.012) publisher_debut 0.004 (0.012) 0.002 (0.026) 0.004 (0.019) pub_last_release_date 0.001 (0.010) 0.016 (0.024) 0.039** (0.017) pubs_last_update_date 0.013 (0.013) 0.026 (0.031) 0.026 (0.021) pub_def_avg_rating 0.042 (0.029) 0.073 (0.074) 0.436*** (0.051) pub_percentage_free_apps 0.108*** (0.022) 0.064 (0.050) 0.209*** (0.036) pub_avg_apps_price 0.004 (0.004) 0.025*** (0.008) 0.022*** (0.006) Constant 0.201*** 0.041 0.412*** 0.087 3.647*** 0.068 Observations 2,215 2,215 2,215 Number of app s 185 185 185 R squared 0.310 0.347 0.381 F Stat 15.85*** 16.36*** 16.48*** (1) Significance levels: ***p < 0.01, **p < 0.05, *p < 0.10 (2) Standard errors in parentheses (3) reviews in the market, total rating of the developer, price of the app, when the app was last updated, when the app was introduced in the app store, number of downloads of the app, when the publisher released their first app in the app market

PAGE 140

131 A PPENDIX H Detection of Fake Reviews Many internet and online users are using o nline reviews to ma k e a purchase choose a restaurant and other decisions. More and more patients are relying on online reviews to choose a physician, a hospital, or even off the shelf medication. It is not a secret any more that the digital footprint and the digital image of any healthcare provider or professional is very important. Positive reviews can yield to significant financial gains and increase any business popularity Such gives incentives for pretenders to mani pulate the system by posting fake reviews to endorse or to disrepute some target ed organizations, individuals and services. These types of reviews are categorized as opinion spamming, and those who write them are called opinion spammers (Jindal and Liu 2008) The concern and issue with fake reviews and opinion spamming continues to grow each day. This is evident in the several cases reported in the news, such as Amazon reviews (Streitfeld 2012b) We are seei ng more and more reporting and investigating on how consumers and people can spot fake reviews (Popken 2010) Evidence has also shown that many fake reviewers admit to having received financial compensation to write their fake review (Kost 2012) Yelp.com has since d ecided to act against these fake reviews and engage in this unethical behavior (Streitfeld 2012a) Deceptive opinion spam was introduced a few years ago by Jindal and Liu (2008) Since then researchers have explored several avenues for detection; Lim et al. (2010) worked to id entify individual fake reviews, Mukherjee et al. (2012) identifying group spammers, Xi e

PAGE 141

132 et al. (2012) applied time ser ies analysis to detect fake reviews, and Feng et al. (2012) applied distributional analysis to address the same issue One of the more common detection tech nique s is supervised learning this uses linguistic and/or behavioral features in detection Existing works have made important progress in this area However, they mostly rely on specified fake and non fake labels for model building. Jindal and Liu (2008) assumed duplicate and near duplicate reviews to be fake reviews in model building for their study An AUC (Area Under the ROC Curve) of 0.78 was reported using logistic regression (Jindal and Liu 2008) Their assumption of fake reviews is too narrow for detecting the generi c fake reviews. To expand on the previous rese arch, Li et al. (2011) tried applying a co training method on a manually labeled dataset of fake and non fak e reviews and getting an F1 score of 0.63. However, t his result also may not be reliable as human labeling of fake reviews has been shown to be quite poor and unreliable (Ott et al. 2011) Even though developing ways to detect fake reviews is outside of the scope of this study, we took precautions while performing the sentiment mining of the reviews collected from the app store. We made sure that the reviews collected were not fake and w ere posted by actual users of the app. We collected a total of more than 25,000 reviews of 218 medical and health apps from the Android store Then we selected a random sample of around 5, 000 reviews and used the software called Review Skeptic, which is de veloped at Cornell University. Review Skeptic uses machine learning to identify fake reviews with 90% accuracy. We found that less than 1% of the reviews were identified by the software as fake reviews. Considering the large number of reviews, we collecte d to measure our key variables our assumption in this study is that fake reviews are not a concern in our study and will not impact our results.