Citation
Design, implementation, and evaluation of Emerge2Maturity, an innovative serious game that uses simulation to depict the data warehouse development

Material Information

Title:
Design, implementation, and evaluation of Emerge2Maturity, an innovative serious game that uses simulation to depict the data warehouse development
Creator:
Khojah, Mohammed
Place of Publication:
Denver, CO
Publisher:
University of Colorado Denver
Publication Date:
Language:
English

Thesis/Dissertation Information

Degree:
Doctorate ( Doctor of Philosophy)
Degree Grantor:
University of Colorado Denver
Degree Divisions:
Department of Computer Science and Engineering, CU Denver
Degree Disciplines:
Computer Science and Information Systems
Committee Chair:
Gregg, Dawn
Committee Members:
Mannino, Michael
Gerlach, James
Choi, Min-Hyung

Notes

Abstract:
Data warehouse development is a complex process involving several related factors and extended time periods to reach a stable solution. Learners face challenges to observe changes, determine key success factors, and understand project relationships involving costs and benefits. This research describes the design, implementation, and evaluation of Emerge2Maturity, an innovative, serious game that provides simulated experiential learning for information technology students and professionals about data warehouse development challenges. Emerge2Maturity addresses learning challenges faced by students to experience data warehouse development over time, determine capabilities to balance costs and benefits for consistency with an organization’s strategy, observe organizational learning effects on costs and benefits, and gain awareness of the impact of external events on strategy. The research uses design science processes (Peffers et al. 2006) for development and evaluation of artifacts as a methodology to guide the development and evaluation of Emerge2Maturity.

Record Information

Source Institution:
University of Colorado Denver
Holding Location:
Auraria Library
Rights Management:
Copyright Mohammed Khojah. Permission granted to University of Colorado Denver to digitize and display this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.

Downloads

This item is only available as the following downloads:


Full Text

PAGE 1

DESIGN, IMPLEMENTATI ON, AND EVALUATION O F EMERGE2MATURITY, AN INNOVATIVE SERIOU S GAME THAT USES SIM ULATION TO DEPICT TH E DATA WAREHOUSE DEVEL OPMENT B y MOHAMMED KHOJAH B.S., King Abdulaziz University, 2005 M.S., University of Southampton, 2010 A d isser tation submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfillment of the requirements for the degree of Doctor of Philosophy Computer Science and Information Systems Program 2018

PAGE 2

ii © 2018 MO HAMMED KHOJAH ALL RIGHTS RESERVED

PAGE 3

iii Th is dissertation for the Doctor of Philosophy degree by Mohammed Khojah h as been approved for the Computer Science and Information Systems Program by Dawn Gregg, Chair Michael Mannino , Advisor James Gerlach Min H yun g Choi Date: May 12 2018

PAGE 4

iv Khojah, Mohammed (Ph.D., Computer Science and Information Systems Program ) Design, Implementation, and Evaluation of Emerge2Maturity, an Innovative Serious Game that Uses Simulation to Depict the Data Warehouse Development Di ssertation directed by Associate Professor Michael Mannino ABSTRACT Data warehouse development is a complex process involving several related factors and extended time periods to reach a stable solution. Learners face challenges to observe changes, determi ne key success factors, and understand project relationships involving costs and benefits. This research describes the design, implementation, and evaluation of Emerge2Maturity, an innovative, serious game that provides simulated experiential learning for information technology students and professionals about data warehouse development challenges. Emerge2Maturity addresses learning challenges faced by students to experience data warehouse development over time, determine capabilities to balance costs and b organizational learning effects on costs and benefits, and gain awareness of the impact of external events on strategy. The research uses design science processes (Peffers et al. 2006) for de velopment and evaluation of artifacts as a methodology to guide the development and evaluation of Emerge2Maturity. The game framework combines evolution of data warehouse architecture and assessment of related capabilities over some decision making periods or phases. Emerge2Maturity uses two novel models to support decision making by players, the Capability Assessment Model (CAM) for decisions about extraction, transformation, and integration levels of

PAGE 5

v data sources and the Configuration Model (CM) for trans ition among decision making phases involving constraint levels, learning effects, and random events. Although data warehouse development is complex, the game is designed to capture important tradeoffs. We developed a prototype implementation using Microsof t Excel to understand complexities of the CAM and CM. A full implementation of Emerge2Maturity uses JavaScript libraries, an Oracle database model, and a simple interface to show game progress and decision details. Emerge2Maturity is assessed in two stage s to establish the content validity of the game. instructors from the information systems field. The second stage involves graduate students who want to learn about da ta warehouse development. The Results from both also shows great alignment between the game design and the proposed learning objectives. Participants provided valuable fee dback on how to advance the development of the game. The feedback should bring more enhancements to the game, which require additional developments. The game will continue to evolve and further analysis could be applied. The research contributes to the bo dy of knowledge by providing a framework to combine business strategy and organizations capability that supports the development of data warehouses. The framework is used as a base to build Emerge2Maturity, the first reported business strategy game to depi ct and decompose the complexity of data warehouse projects. The research also builds an evaluation framework for serious games

PAGE 6

vi and use emerge2Maturity to validate the framework. The process of designing and implementing emerge2Maturity adheres to the desig n science research. The form and content of this abstract are approved. I recommend its publication. Approved: Michael Mannino

PAGE 7

vii DEDICATION This dissertation is dedicated to the soul of my father who would be so proud of me, to my mother the source of m y power, to my wife the origin of my success , to my children the cause of my motivation. the thankful for having them in my life. my friends who always supported me and encouraged me to continue and thrive.

PAGE 8

viii ACKNOWLEDGEMENTS In the name of Allah, the most merciful, the most compassionate. All praises are to Allah, the Lord of the universe and prayers and peace be upon Muhammad (PBU H), the servant of Allah and His last messenger. I acknowledge that all my strength, knowledge, and power are from Allah and His blessings. I would like to thank my advisor for his help and support during my years in the Ph.D. program. I would like to than k the chair committee for her active role, encouragement, and guidance. I would like to thank all my committee members for their dedicated time and effort to make this dissertation a success. I would like to thank the government of Saudi Arabia represented by the ministry of higher education for their generous scholarship and funding. I would like to thank the Center for Faculty Development (CFD) for their grant. Special recognition to Jeff Rynhart for his help in the development of the Emerge2Maturity COMI RB Protocol 17 1712

PAGE 9

ix TABLE OF CONTENTS CHAPTER I . OPENING ................................ ................................ ................................ .......................... 1 II . LITERATURE REVIEW ................................ ................................ ................................ . 6 Introduction ................................ ................................ ................................ ............. 6 Data Warehouse Development ................................ ................................ ............... 6 Data Warehouse A rchitecture ................................ ................................ ................. 7 Maturity Models ................................ ................................ .......................... 9 Data Warehouse Maturity Model ................................ .............................. 12 Architecture Selection ................................ ................................ ............... 19 Cost Benefit Analysis Models ................................ ................................ .............. 22 Games in Education ................................ ................................ .............................. 26 Serious Game Evaluation ................................ ................................ ...................... 27 Game Characteristics ................................ ................................ ................ 29 Learning M echanics ................................ ................................ .................. 31 Learner Motivation ................................ ................................ ................... 32 Outcomes ................................ ................................ ................................ .. 33 Methodology ................................ ................................ ............................. 34 Design Science Research ................................ ................................ ...................... 35 Literature Summary ................................ ................................ .............................. 37 III . EMERGE2MATURITY DESI GN ................................ ................................ ............... 40 Introduction ................................ ................................ ................................ ........... 40 Problem Identification ................................ ................................ .......................... 40

PAGE 10

x Objective of Solution ................................ ................................ ............................ 42 Game Design ................................ ................................ ................................ ......... 44 Game Design Decisions ................................ ................................ ............ 47 Game Flow ................................ ................................ ................................ 50 Capability Assessment Model (C AM) ................................ ...................... 51 Data Source Categories ................................ ................................ ............. 53 Decision Variables ................................ ................................ .................... 55 Coefficients ................................ ................................ ............................... 56 Model Functions ................................ ................................ ....................... 57 Objective Function and Constraints ................................ .......................... 58 Config uration Model ................................ ................................ ................. 60 Weights for Costs and Benefits ................................ ................................ . 60 Events ................................ ................................ ................................ ........ 62 Score and Rank ................................ ................................ ......................... 63 Game Experiences ................................ ................................ ................................ 64 Design Summary ................................ ................................ ................................ ... 66 IV . EME RGE2MATURITY IMPLEME NTATION ................................ .......................... 67 Introduction ................................ ................................ ................................ ........... 67 Prototype Game Development ................................ ................................ .............. 67 Actual Game Development ................................ ................................ ................... 72 Game Controller and Database Model ................................ ...................... 72 Implementation Details ................................ ................................ ............. 74 Game Demonstration ................................ ................................ ............................ 75

PAGE 11

xi Implementation Summary ................................ ................................ ..................... 79 V . EMERGE2MATURITY EVAL UATION ................................ ................................ ..... 80 Introduction ................................ ................................ ................................ ........... 80 Proposed Evaluation Framework ................................ ................................ .......... 81 Design Evaluation Metho dology ................................ ................................ .......... 87 ................................ ................................ .......................... 92 Results ................................ ................................ ................................ ....... 92 Discussion ................................ ................................ ................................ . 95 Classroom Evaluation ................................ ................................ ........................... 97 Results ................................ ................................ ................................ ....... 98 Discussion ................................ ................................ ............................... 115 Combined Discussion ................................ ................................ ......................... 118 Effect Evaluation ................................ ................................ ................................ 120 Hypotheses ................................ ................................ .............................. 121 Experiment Design ................................ ................................ .................. 123 Constructs and Measurement Items ................................ ........................ 125 Results ................................ ................................ ................................ ..... 126 Evaluation Summary ................................ ................................ ........................... 127 VI . CLOSING ................................ ................................ ................................ ................... 128 REFERENCES ................................ ................................ ................................ ........................... 132 APPENDIX A GAME NARRATIVES ................................ ................................ ............................... 147 B INSTRUCTOR'S SURVEY ................................ ................................ ......................... 157

PAGE 12

xii C ST ................................ ................................ ............................... 170 D CONSTRUCTS AND MEAS UREMENT ITEMS ................................ ...................... 182 E CODED FEEDBACK ................................ ................................ ................................ .. 184

PAGE 13

xiii LIST OF FIGURES FIGURE 1. Market share for selected data warehouse architectures ................................ ............... 9 2. Capability Maturity Model (CMM). Adapted from Pau lk et al. 1993 ........................ 11 3. Data warehouse maturity model. Adapted from Sen et al. (2012) .............................. 14 4. Data Warehouse Maturity Model (Eckerson W, 2007 ) ................................ .............. 16 5. Combined view of the factors that affect the architecture selection ........................... 22 6. Design Science Processes by Peffers et, al. (2006) ................................ .................... 37 7. Interaction between Strategy and Capability ................................ .............................. 45 8. Overview of Decisions in Emerge2Maturity ................................ .............................. 51 9. Elements of the Capability Assessment Model (CAM) ................................ .............. 52 10. The cost weight ................................ ................................ ................................ ........... 61 11. The benefit weight ................................ ................................ ................................ ...... 62 12. T he Interface Worksheet ................................ ................................ ............................. 68 13. First Configuration Sheet ................................ ................................ ............................ 68 14. Coefficients Table ................................ ................................ ................................ ....... 69 15. Second Configuration Sheet ................................ ................................ ....................... 70 16. An Expected Worksheet for Calculations ................................ ................................ ... 71 17. A Solver Worksheet for Calculations ................................ ................................ ......... 71

PAGE 14

xiv 18. Data Model for the Game Controller ................................ ................................ .......... 73 19. Emerge2Maturity Game Phase 1 Interface ................................ ................................ . 76 20. Phase Simulation for Extraction Decisions in Phase 1 ................................ ............... 77 21. Game Score and Leaderboard Ranks ................................ ................................ .......... 78 22. Proposed Evaluation Framework ................................ ................................ ................ 83 23. Gne1 Descriptive Analysis ................................ ................................ ......................... 99 24. Gne2 Descriptive Analysis ................................ ................................ ......................... 99 25. Gne3 Descriptive Analysis ................................ ................................ ....................... 100 26. Gne4 1 Descriptive Analysis ................................ ................................ .................... 101 27. Gne4 2 Descriptive Analysi s ................................ ................................ .................... 101 28. Gne4 3 Descriptive Analysis ................................ ................................ .................... 102 29. Gne4 4 Descriptive Analysis ................................ ................................ .................... 102 30. Gne4 5 Descriptive Analysis ................................ ................................ .................... 103 31. Gne4 6 Descriptive Analysis ................................ ................................ .................... 104 32. Gne4 7 Descriptive Analysis ................................ ................................ .................... 104 33. Gne4 8 Descriptive Analysis ................................ ................................ .................... 105 34. Gne4 9 Descriptive Analysis ................................ ................................ .................... 106 35. Gne5 Descriptive Analysis ................................ ................................ ....................... 107 36. Gne6 Descriptive Analysis ................................ ................................ ....................... 107

PAGE 15

xv 37. Gne7 Descriptive Analysis ................................ ................................ ....................... 108 38. Efc6 Descri ptive Analysis ................................ ................................ ......................... 108 39. Efc7 Descriptive Analysis ................................ ................................ ......................... 109 40. Gnl2 Descriptive Analysis ................................ ................................ ........................ 110 41. Gnl3 Descriptive Analysis ................................ ................................ ........................ 11 1 42. Emerge2Maturity Evaluation Model ................................ ................................ ........ 123 43. Experiment Design ................................ ................................ ................................ .... 124

PAGE 16

xvi LIST OF TABLES TABLE 1. 20 2. Factors that affect architectur e selection (Ariyachandra and Watson, 2010) ............. 21 3. List of factors that affect the strategic view ................................ ................................ 46 4. Summary of Design Decisions ................................ ................................ .................... 47 5. Relationship of Category Features to Model Components ................................ ......... 54 6. Decision variables for extraction, transformation, and integration ............................. 55 7. List of coefficients ................................ ....................... Error! Bookmark not defined. 8. Capability Assessment Model Functions ................................ ................................ .... 58 9. Other D efinitions in the CAM ................................ ................................ .................... 58 10. Impact of events ................................ ................................ ................................ .......... 63 11. Information provided to players in each phase ................................ ........................... 65 12. List of E2M Learning Objectives ................................ ................................ ............... 86 13. Gagne Events and Their Corresponding Events in E2M Game ................................ . 87 14. vents Items (without learning objectives) ................................ .................. 89 15. Learning Objectives Items ................................ ................................ .......................... 89 16. Game Efficacy Items ................................ ................................ ................................ ... 90 17. General Items ................................ ................................ ................................ .............. 90

PAGE 17

xvii 18. T ................................ ................................ ................ 113 19. Mean values for the Two Groups ................................ ................................ .............. 114 20. T test results for Game Efficacy Items ................................ ................................ ..... 114 21. T test results for General items ................................ ................................ ................. 115 22. Findings Summary from Both Data Collections ................................ ....................... 119 23. Minimum Number of Sample Size (Soper, 2017) ................................ .................... 125 24. Game Narratives for the Welcome Page ................................ ................................ ... 147 25. Game Narratives for the Instructions Page ................................ ............................... 148 26. Game Narratives for the Game Concepts Page ................................ ......................... 149 27. Game Narratives for the Game Preparation Page ................................ ..................... 152 28. Game Narratives for the Phase Preparation Page ................................ ..................... 153 29. Game Narratives for the Phase Simulation Page Extraction ................................ . 154 30. Game Narratives for the Phase Simulation Page Transformation ......................... 154 31. Game Narratives for the Phase Simulation Page Integration ................................ . 155 32. Game Narratives for the Phase Summary Page ................................ ........................ 155 33. Game Narrativ es for the Game Summary Page ................................ ........................ 156 34. Game Narratives for the Game Score Page ................................ .............................. 156 35. Constructs and Items for Game Mechanics ................................ .............................. 182 36. Constructs and Items for Motivation ................................ ................................ ........ 182

PAGE 18

xviii 37. Game Efficacy Construct and Items ................................ ................................ ......... 182 38. Learning Outcomes Constructs and Items ................................ ................................ 183 39. Coded Text for Gne1 Item ................................ ................................ ........................ 184 40. Coded Test for Gne2 Items ................................ ................................ ....................... 184 41. Coded Test for Gne3 Item ................................ ................................ ......................... 185 42. Coded Text for Gne4 Item ................................ ................................ ........................ 185 43. Coded Text for Gne5 Item ................................ ................................ ........................ 186 44. Coded Text for Gne6 Item ................................ ................................ ........................ 186 45. Coded Text for Gne7 Item ................................ ................................ ........................ 187 46. Coded Text for Grl4 Item ................................ ................................ ......................... 187

PAGE 19

1 CHAPTER I OPENING Using serious games to facilitate learning through simulating real life events is a fundamental goal of design science in the information systems discipline (Kankanhalli et al. 2012; Zich ermann et al. 2011). It involves creation of new knowledge through the design of innovative artifacts and the analysis of the artifact. A ttention to computer based serious game research is growing. Serious games have a long history in business education. Since t he Beer Game (Anderson & Morrice, 2000), scholars and students observed the potential benefits of using games to deliver knowledge and skills to participants. Using serious games is effective in teaching students about business processes and impact of info rmation technology (IT) (Monk and Lycett 2011) . Serious games facilitate learning about strategy, collaboration, integration, and development maturity (Leger, 2006), processes difficult to grasp using traditional learn ing practices without practice and experience. Clearly, there is a need to build more serious games and expand the research into areas where serious games have not been used yet. In business education, these areas include impact of information technology ( Monk and Lycett 2011), strategy, collaboration, integration, and development maturity (Leger, 2006). More than 70% of data warehouse projects fail to achieve the desired outcomes (Inmon, 2001). Research indicates that user training is an important success factor in data warehouse projects (Wixom and Watson, 2001). With complexity and ambiguity in designing and implementing data warehouses, students and IT professionals struggle to understand complexities of data warehouse projects , such as intangible benefi ts, relationships between strategy and capability assessment in an organizational setting , and

PAGE 20

2 learning effects curve when dealing with capability assessment . It is no surprise that many data warehouse projects fail (Shin, 2003). Although many university c ourses cover data warehouse design and implementation, traditional learning approaches fail to capture complexity and challenges that occur in real situations. Researchers have acknowledged the gap between knowledge learned in school and skills gained thro ugh experience (Boyle & Strong, 2006; Kim, Hsu, & Stern, 2006; Mackrell, 2009). Traditional university courses do not reveal ambiguity in real work (Lee et al., 2002). Clearly, there is a strong demand for innovative learning approaches to help students ex perience complex relationships involving technology and organizational structures. In this research, we present the design, implementation, and ev aluation of a serious game, Emerge2Maturity. The game facilitates learning by IT students and professionals a bout data warehouse architecture selection and capability assessment. The game provides a dynamic learning experience to help learners acquire capabilities to support an evolving architecture in an organizational setting. The game design can be applied to other areas of IT strategy selection and capability assessment. As a business intelligence strategy with its data warehouse capabilities. Players manipulate capabilities to max imize expected benefits subject to organizational constraints on budget and resources. Emerge2Maturity features two novel decision models and simulation of player choices to provide a serious game experience. The first model is the Capability Assessment M odel (CAM), a novel decision model that evaluate cost benefit tradeoffs among player choices for resources. The second model is the Configuration Model, a

PAGE 21

3 model to revise constraint levels and resource coefficients based on architecture evolution and occur rence of events in each decision phase. In each decision phase, the Capability Assessment Model (CAM) calculates expected, simulated, and optimal results of player choices for the available resource choices. To transition between decision phases, the Conf iguration Model (CM) revises cost and benefit levels based on organizational learning rates and constraint levels based on occurrence of events. The educational value involves experiential learning about data source categories, constraints on budgets and r esource limits, balancing costs and benefits, decisions across multiple phases, and changes across phases. The player performance depends on balancing costs and benefits to maximize profit as compared to optimal play. The prototype of the CAM and CM was bu ilt in Microsoft Excel to evaluate the analytical engine of Emerge2Maturity. The game is fully implemented using Javascript, lp solver, Node.js, Aurelia.js, Express.js, and an O racle database. E2M will be evaluated in two main stages. To evaluate the conte nt validity of the game, instructors in the IS field with knowledge and skills in data warehousing were framework (Gagne, 1970) . Then, the effectiveness of the game on the learning outcomes of students will be tested . The purpose is to evaluate the effectiveness of E2M in delivering the proposed learning outcomes to students. Participants will be asked to join a learning session experiment where they are divided randomly int o two groups: control and treatment group. This research contributes on both academic and practice. T his project integrates two research streams about data warehouse development into a framework for game

PAGE 22

4 development and instruction. Architecture selection and maturity models have been shown important to management of data warehouse development, but a framework to integrate them for instruction does not exist. This research develops a framework for game development to facilitate experiential learning about business strategy and maturity models. The framework can also be generalized to other areas of information technology management. Emerge2Maturity employs two novel analytical engines (CAM and CM) to help players evaluate tradeoffs among resource levels as an organization evolves to a mature state. Emerge2Maturity game is the first reported business strategy game for data warehouse development. In order to evaluate E2M, this research provide s a comprehensive framework for serious games evaluation. The frame work combines design and effect evaluations into a single framework. Planned experimental evaluation of Emerge2Maturity will combine outcomes of player engagement, perceived learning, and task performance, a comprehensive approach not typically used to eva luate business strategy games. Both game development and evaluation adhere to design science principles. This approach corresponds with the view of Benbasat and Zmud (2003) regarding the need for more IT artifacts in the information systems discipline. This research also contributes to the practice of data warehouse instructions. Emerge2Maturity is the first serious game develope d for data warehouse strategy and capability assessment, an important yet difficult learning area. Existing approaches to maturity and capability assessment for data warehouses lack descriptive precision and the ability for students to manipulate outcomes . Emerge2Maturity allows players to experience the simulated evolution of a data warehouse infrastructure , and thus increase s understanding of the relationship among architecture and capability assessment. Applying

PAGE 23

5 skills learned from simulated environments into real world has been demonstrated from experiments involving other simulation games (Lainema and Makkonen, 2003). The remaining chapters of the research are organized as follows. Chapter 2 presents a review of the literature about data warehouse devel opments, cost benefit analysi s model, games in education, game evaluation , and design science research . Chapter 3 presents the game design featuring topics of proposed framework, capability assessment model CAM, and configuration model CM. Chapter 4 presen ts the development of a prototype using Microsoft Excel application and actual game implementation using various technologies such as nodejs and O racle D B. Chapter 5 presents an evaluation framework and the evaluation of Emerge2Maturity . Chapter 6 presents a summary of the research indicating the research limitations, contributions, and possible future studies.

PAGE 24

6 CHAPTER II LITERATURE REVIEW Introduction This research involves topics from two main areas, serious games for education and management of data warehouse de velopment. The review about serious games provides details about game development and evaluation. The review of management of data warehouse development covers data warehouse architectures, maturity models, architecture selection, and cost benefit models f or decision making. The methodology of this research follows the design science guidelines by Peffers et al. (2006). The review concludes with a summary linking related work to the development of the proposed business strategy game. In addition , the conclu sion describes the fit of this research into design science, the research methodology guiding development and evaluation of the proposed business strategy game. Data Warehouse Development Over the last 20 years, large and medium sized companies have adopte d the idea of a data ware house as a strategic solution. A d ata warehouse is a structural design that stores a large amount of secondary data. The data originally come from operational databases along with external data sources. Then, the data is processed through an operation called Extract, Transform and Load (ETL) and saved into data repositories. Bill integrated, non volatile, and time variant collection of data in sup port of management's and timely data to top management to get reliable decisions. Integration is a key aspect of

PAGE 25

7 a data warehouse to combine data from different data sources into unified data with a common schema. The level of integration is one of the key factors that determine data warehouse design and a key success factor for data warehouse projects. Data W a rehouse A rchitecture Although many data warehouse architect ures have been designed and developed over the years, none is considered to be the ultimate solution (Ariya chandra and Watson 2010) . The structure of the data warehouse determines the level of scalability and performance (Strange, 2003). Moreover, the architecture is a combination of hardware and software (Septoff and Simmons 1996) . Hardware includes storage, processors and communications and software includes database management systems (DBMS) and data in tegration tools. The early beginnings of data warehouse s started the need for central database apart from the operational databases to feed decision support systems (Sprague and Carlson, 1982). The i ndependent data marts concept was introduced as a resp0ns e but it suffered from inconsistent data (Hackney, 2000). Working toward more consistent and integrated data sources , two main concepts have emerged: Data mart bus architecture and enterprise data warehouse. Data mart Bus Architecture (DBA) was introduced by Ralph Kimball all , 2003). The concept was to create independent data marts based on the business process. These data marts share common standards that guarantee consistency. This approach is kn own as bottom up approach since it is driven by the business needs . The development of data mart bus architecture stars with building one data mart for specific business process. Predefined standards and rules are used in the initial development. When buil ding other

PAGE 26

8 marts, these standards and rules are applied. The result is more concise and unified view of the data. On the other hand, Enterprise Data Warehouse (EDW) is known as a top down approach because it requires high involvement from top management du e to its higher strategic value and higher costs. The concept is to have single data warehouse to feed the organization. Separate data marts can be created from this single data warehouse based on organizational needs. This concept has been introduced by B ill Inmon who has been known as the father of data warehouse. The architecture type is important in the data warehouse design because it determines the level of scalability and performance. In fact, selecting the appropriate architecture for the organizati on is a key success factor (Laney, 2000). It is also one of the challenges that face any data warehouse project (Strange, 2003). Other architectures were mentioned in the literature such as Hub and Spoke, Federated, and distributed data warehouse. Figure 1 shows the market share for some data warehouse architectures according to a survey conducted by Alsqour et al. ( 2012) . Selecting among these architectures is not an easy task. Issues related to ownership, end user and technical requirements must be take n into account (Septoff and Simmons 1996) . The literature shows two approaches to determine the p rocess of adopting data warehouse architecture, maturity models and architecture selection. Research in architecture selection approach defined some factors that influence the selection of particular data warehouse architecture among the other architecture s. This approach assumes that organizations have the option to select the design and implement it as if they are buying a new solution. On the other hand, research for the other approach, maturity model, assumes that the development of data warehouse is co ntinues process that evolves from a stage to another over periods of time. It tells that organizations start with simple

PAGE 27

9 architecture and keep improving it as they mature over time. More details about maturity models and how they are related to data wareho use developments are in the next section. Figure 1 : Market share for selected data warehouse architectures Maturity Models Maturity models are commonly used as a roadmap to evaluate and understand an time especially for technology capabilities and deployment. A maturity model consists of several stages. Each stage has its unique objectives, requirements and measurements. Becker et al. ( 2009) a sequence of ma turity levels for a cla ss of objects. It represents an anticipated, desired, or typical evolution path of these objects shaped as discrete stages (Becker et al. 2009 ; P213 ) . It is a description of a linear progress or en hancement in the capability of the organization (Russell et al. 2010) state of being complete, perfect or ready sure to evaluate the capabilities of an organization (Bruin et al. 2005 ; P 1). In general, immature organizations exceed budgets and deadlines, focus on immediate problems rather than strategic problems, do not have rigorous criteria to evaluate quality and do not enforce the use of developed systems (Paulk et al. 1993) . Similarly, immature data warehouse is designed to react to certain issues, lack proper

PAGE 28

10 measurements for data quality and the budget and the delivery dates are frequently exceeded (Sen et al. 2006) . Most maturity models have their bases in the Capability Maturity Model (CMM), from a process maturity framework, a method to understand and evaluate the progress of sof tware developments. The model ( Figure 2 ) consists of five main maturity levels each have its own key processes areas, common features and key practices (Paulk et al. 1993) . Each level is unique in its characteristics and can be evaluated by using assessments or questionnaires. Here is a description of each level. (1) Initial level: Organizations at this level encounter many operational and managerial problems. There are no explicit rules or policies for the development process. Projects are always run out of budgets and pass over their timelines. (2) Repeatable: In level 2, organizations start to benefit from the accumulated experiences from previous projects. Organizations at this level s tart to establish policies and procedures. (3) Defined: Documentation is introduced in this level. Companies become more mature to standardize their processes and apply strict rules on how developers should write codes or document their work. (4) Managed: Quantitative measurements are used in the 4 th level to assess situations and evaluate the quality of the products. Thus, results become more predictable because organizations depend on reliable measures. (5) Optimizing: This is the highest maturity level i n the model. In this level, organizations perform continuous improvements in all their projects. Organizations may find their own ways to accom plish results more efficiently, which might lead to innovative development processes.

PAGE 29

11 Figure 2 : Capability Maturity Model (CMM). Adapted from Paulk et al. 1993

PAGE 30

12 Data Warehouse Maturity Model Capability maturity model has been applied to many areas other than software process developments. The literature shows several models that have been p ublished including management control systems (Marx et al. 2012) , IT management (Becker et al. 2009) , enterprise systems (Ma thrani et al. 2009) , organizational knowledge (Freeze et al. 2003) and data warehouse development (Sen et al. 2012 ; Dinter 2012 ; Lahrmann et al. 2011 and Eckerson 2007) . In the data warehouse area, the literature usually does not differentiate bet ween a data warehouse and its applications such as business intelligence (BI). technologies, applications, and processes for gathering, storing, accessing, and analyzing data This statement implicitly refers to the activities performed in the data warehouse side. Data warehouse s are commonly described as the back end of many applications and systems. Maturity models for data warehouse a nd BI gained attention from both academics as well as practitioners (Raber et al. 2013) . Maturity models share three main features based on the capability maturity model: maturity l evels, key process areas and activities. The next section will focus more on the effort taken to develop maturity models for the data warehouse development process. The literature shows many research efforts to compare different maturity models and evalua te their effectiveness. For example, Lahrmann and Marx ( 2010) compared 10 different maturity models that have been developed in the area of data warehouse s . Despite the number of models, two maturity models seem to dominate. Sen et al. ( 2006) argued that the data warehouse development must be addressed and evaluated as series of

PAGE 31

13 processes similar to the software process development. Their efforts were to build a maturity model that imitates the capabili ty maturity model. The five levels from the capability maturity model have been applied but different key process areas (KPAs) and features were used . These KPAs and features are unique for the data warehouse development process. See Figure 3 for more details about the maturity levels and the KPAs. In a further research, Sen et al. ( 2012) differentiated between operations and developments in data warehouse projects. They suggested separate key process area for each section although both may share some similar activities.

PAGE 32

14 Figure 3 : Data warehouse maturity model. Adapted from Sen et al. (2012)

PAGE 33

15 In the data warehouse maturity model, six out of nine features were found to be significant: alignment of architecture, data warehouse size, data quality, organizational readiness, organizational slack and change management. Here is a summary of findings about significant features. Alignment of architecture: The architecture of a data warehouse by itself is not enough to determine the lev capability require organizations to have higher alignments between their DW architectures and their business strategies. The strategy determines the integration level of the data in the data warehouse. Dat a size: as the capability of an organization increases, organizations tend to process and store larger amounts of data. Data quality: the quality of the data becomes a big issue in larger data warehouses. Thus, data quality feature correlates with the dat a size feature. Data transformation process e nsures that the data has the right quality level. Organizational readiness: This feature refers to the perception of the organization about the value and the benefits of the data warehouse. Organizational slac k: Data warehouse projects require resources such as financial resources and workforce resources. As organizations cruse toward higher levels of maturity, the required resources to leverage their capabilities become more accessible and available. Change management: Maturity models assume continues improvements and changes over time. Thus, tracking these changes is crucial for successful projects.

PAGE 34

16 Changes in higher capability levels go through some change policies and procedures. The second dominant data w arehouse maturity model was developed by The Data Warehouse Institute (TDWI). The TDWI provides this model as part of their solutions to evaluate the current status of organizations along with an assessment tool. The TDWI maturity model emphasizes maturit y in business value instead of maturity in capability. The model consists of six distinct stages, Parent, Infant, Child, Teenager, Adult and Sage ( Figure 4 ). These levels are evaluated by eight dimensions (Cardoso et al. 2013) , scope, sponsorship, funding, value, architecture, data, development and delivery. Figure 4 : Data Wa rehouse Maturity Model (Eckerson W, 2007 ) Similar to the Capability Maturity Model, each stage in the TDWI model has its own characteristics based on the eight dimensions. Here is a summary of each stage is as follow. Prenatal: The organization is highly dependent on rigid and inflexible reports. These reports come from traditional management report systems. The information

PAGE 35

17 systems structure may involve some legacy systems, which makes customized reports difficult to produce. Infant : Organizations rely on third party software such as spreadsheets for more flexible reports. Spreadsheets are easy to use with low cost to own and maintain. The result is huge divers ity in standards and versions of data among departments. Child: Independe nt data marts are introduced in this level. Departments start to realize the hassle of dealing with data without using database. Data marts are used to store data extracted and transformed directly from the source systems. However, each department has its own standards and multiple versions of data are stored in each data mart. Teenager: After building few independent data marts, integration among these data marts becomes a big concern. IT departments resolve this issue by providing standards and rules for the ETL process. However, each department will still have its own data marts. This structure is called Data mart Bus Architecture (DBA). Adult: By this stage, organizations have invested so much so far and start to see the benefits of more consistent and integrated data. Thus, Organizations may want to build an Enterprise Data Warehouse (EDW) either by continue the development on what they have accomplished so far or rebuild it from scratch. This structure will bring single view of truth concept to the or ganization. Since all the data are stored in one enterprise repository with unified standards and rules, organizations will start benefiting from new uses for the data they already have. As a result, the return on investments will exceed the costs.

PAGE 36

18 Sage: Organizations already have highly perception about the strategic value of their data and aware of potential benefits from opening and sharing their data with others and get the same in return. Data can be available to external customers, suppliers and BI u sers over the Internet. Also, organizations may connect to other sources of data and scale data mining and BI analytical services into a different level. Maturity models are not just artifacts used to solve certain problems, but also have roots to scientif ic theory called stages of growth. This theory says that any process that evolves over time goes through certain stages. Two main features describe the theory (Kuznets, 1965). First, each stage has defined characteristics that make it possible to identify each stage. Second, the relationships between any two consecutive stages are well defined. Nolan ( 1973) used this theory in his study to show the process of adopting computers as a resource and how this process develops over a period of time . The capability maturity model used the same concept to identify stages that organizations passé over during their journey in the software development process. In 2001, Watson et al. introduced their model that describes the development of data warehouses in organizations using the concept of stages of growth theory. The model had only three distinct stages and nine features to describe each stage. Since then, r esearchers brought more insights to the model. Many efforts have been put from both academic and practitioners to develop a coherent reliable model that helps developers and managers to evaluate their situations and determine the next targeted goal. There are many benefits for using data warehouse maturity model in data warehouse implementation. Maturity models help identifying weak practices and provide

PAGE 37

19 guidelines for enhancements (Paulk et al. 1993) . It also can be used as benchmark to measure the benefits ( Hribar 2010) . Higher maturity level is an indication for higher capability for the organization to take its processes to the next level (Russell et al. 2010) . It reduces the risk of project failure (Chuah and Wong 2011) . Architecture Selection The other approach for data warehouse adoption is architecture selection. Many researchers inves tigated the factors that affect architecture selection. This section focuses on two main studies. The first study (Choudhary 2010) exam ined 11 factors that hypothesized to affect the architecture selection. Only seven were found significant. The factors are: resource constraints, perceived IT skills, need for integration, level of sponsorship, strategic view, urgency and need for informat ion flow between organization units. The study came to a conclusion on when different architectures are more likely to be selected. Independent data mart architecture is more likely to be selected if the organization has high constrain t s on resources and t he perceived IT skill among its staff is low. Moreover, the organization would select the data mart bus if the organization has low constraints on resources, high need for data integration and high sponsorship level. Finally, enterprise data warehouse woul d be selected if the organization perceived the data as a strategic resource. The study also tells when organizations favor data mart bus over the enterprise data warehouse. It happens when there is high urgency for data warehouse, organizations have limit ed scope and the need for information flow between organization units is high. Table 1 summarizes these factors.

PAGE 38

20 Table 1 : Factors that affect architecture selection based on Choudhary (2010) Factor Descrip tion Resource constraints Slack resource that includes financial resources and people resources required for system implementation. Perceived IT skills How organizations perceive the computer efficacy or their staff Need for integration Reports require data from different sources or units. Level of sponsorship Individuals or groups who have control over the data. Perceived strategic view How organizations perceive their IT and data sources. Urgency Organization requires fast implementation and deliver y for the system. Need for information High demand on data between certain departments or units. The other study (Ariyachandra and Watson 2010) introduced seven factors that affects architecture selection but only six were significant. They are: perceived IT skills, constraints on resources, the perceived strategic view of the data warehouse, information i nterdependence, task routineness and the level of sponsorship. See Table 2 for more description about these factors.

PAGE 39

21 Table 2 : Factors that affect architecture selection ( Ariyachan dra and Watson , 2010) Factor Description Perceived IT skills How organizations perceive the computer efficacy or their staff Resource constraints Slack resource that includes financial resources and people resources required for system implementation. P erceived strategic view How organizations perceive their IT and data sources. Information interdependence Describes how departments within an organization rely on data from each other Task routineness Very structured data that are automated need less hum an interference Level of sponsorship Individuals or groups who have control over the data. Later, they found that information interdependence, task routineness and the level of sponsorship are affecting the architecture selection through the perceived s trategic IT skills, the strategic view about more advanced data warehouse structure from the top management is low and the resource constraints are high. Data mart Bu s Architecture (DBA) on the other hand, is favored if the organization has high perception about its employee IT skills, the strategic view is also high and the resource constraints are low.

PAGE 40

22 The study concluded that organizations would jump into enterprise data warehouse only if the strategic perception about their data is exceptionally high that they are willing to take the integration into that level. Fi gure 5 summarizes the combined view of the two studies mentioned above. Fi gure 5 : Combined view of the factors that affect the architecture selection Cost Benefit Analysis Models A cost benefit analysis CBA, or benefit cost analysis BCA, is a simple economic model to evaluate projects based on cost and b enefit. Several research efforts have been conducted using cost benefit analysis. In data warehouse, Rao and Osei Bryson ( 2008) suggested a cost benefit model using 0 1 integer progra m ming . The model maps data sources (DS) to decision support views (DSV). The DSV has estimated benefits to the firm and measurable quality levels. Higher quality level gives more benefits but also

PAGE 41

23 requ ires additional cost. This model determines data sources that provide the maximum value. Ballou and Tayi (1999) also used integer progra m ming approach to determine the quality level that provide maximum value. Kalfus, Ronen and Spiegler (2004) used selecti ve retention approach to determine retention polic i es that saves costs related to data sources. For development and operation of a data warehouse, cost is tangible and easily measured, but uncertain. On the other hand, benefits are largely intangible and difficult to measure directly. Costs can be divided into two main parts: fixed cost to build the data warehouse, and variable cost to maintain and operate the data warehouse. Dimensions of these costs involve hardware, software, labor, and time. Processes that involve cost are acquiring, extracting, transforming, integrating, and securing the data. Understanding the sources and tracking the amount of these costs are important for the cost benefit analysis model. Once the cost is calculated , the estimated be nefits should be, at least, equal to the total cost. However, benefits are mostly intangible and difficult to quantify. The main the decision making process (Watson et al. 2002) . Although some organizations id measures to evaluate this benefit. Some useful measures can be increased sales, increased market share, user satisfaction, and confidence in decisions (DeLone and McLean, 1992). Benefits from data warehouse s come from queries and reports that are gener ated using data sources. Two approaches determine which sources to include in the data warehouse are the demand driven approach and supply driven approach (Winter and Strauch 2002) . The demand driven approach anticipates demand first and then selects

PAGE 42

24 related data sources. This approach analyzes the requirements of decisions made by various roles in the organization. Then, the data sources are analyzed and only sources that match the requirements are selected. This approach minimizes the risk of including data sources that might not be b eneficial to the decision makers but also increase the chances of missing opportunities. On the other hand, supply driven approach makes all the data sources available for better return. Costs in this approach might be higher but there are great chances th at decision makers will benefit from unexpected relationships and patterns that are revealed from the data. Benefits from queries and reports are affected by factors from three processes, extraction, transformation, and integration. Each factor adds more v alue to the queries and reports, which will be reflected by the quality of the decision making. Data sources can take many forms from operational databases, files, web contents or even external data sources. These sources have potential value for decision making. The first step toward benefiting from these sources is extracting these data and stores them into the data warehouse. Kalfus et al. (2004) suggested that organizations should always extract all their data sources and make them available in the data warehouse. Ballou and Tayi (1999) however, prefer analyzing the data sources and select what is more relevant and appropriate for the decision makers. Although none of these approaches are necessarily true, data warehouse designers must be aware of the co st and benefit trade off when it comes to data sources selection. The second factor to increase the value of queries and reports is data transformation. Transformation enhances the quality of the data inaccuracy , com pleteness, and timelines (Ballou and Ta yi, 1999). Transformation also resolves

PAGE 43

25 inconsistencies, applying business rules, and summarization (W atson et al. 2002) . The quality of the data is reflected on the quality of the decision making process. Jarke et al. (1999) introduced three main quality dimensions with sub categories : design and administration dimension, data usage dimension, and data d imension. Design and administration dimension includes correctness, completeness, minimality, traceability, interoperability, and metadata evaluation. Data usage dimension includes responsiveness, timeliness, interpretability, security, and availability. D ata dimension includes accuracy, completeness, consistency, credibility, and data interpretability. Rao and Osei Bryson (2008) used two quality dimensions, system and information. The system dimension includes measures like response time, reliability, and ease of access, while the information dimension includes accuracy. The third factor, data integration, is one of the most important decision factors because of its influence on the architecture of a data warehouse (Ramamurthy et al. 2008) . Integration of data mean s that relevant data are processed under the same rules and stored once forming a single point of truth (Gulle dge 2006) . From the previous explanations of both maturity models and architecture selection, it becomes clear that the objective is to build or select the appropriate architecture depending on the level of data integration. Each data mart in independent data marts architecture processes data from single or small number of data sources. On the other hand, data marts in the enterprise architecture will process data from a higher integrated repository. Thus, the integration level of the data in the data ware house is reflected on the type of the data warehouse architecture. Data integration can uncover hidden relationships among data that has not

PAGE 44

26 been possible to capture using traditional reporting tools. Higher integration level can also reduce the total cost of ownership by managing data. Games in Education Osatuyi, Osatuyi, and de la Rosa ( 2018 ) c onducted systematic review of published papers in to IS journals and conferences between 2008 and 2017 resulted in 41 papers in gamification area. These papers use based The Game Based Learning paradigm focus es on achieving the particular objectives of given educational content through gameplay 2009: p. 801). Typically, s erious games imitate real life events and simplify relationships among constructs. Serious games developed through this paradigm provide be nefits other than entertainment (Michael & Chen, 2006 ). Serious games can be used to teac h concepts or provide experience to participants. In higher education, serious games support interactive lea r ning and e ngagement through entertainment (Prensky, 2007) with advantage s over trad itional teaching approaches (Pivec, 2004). Also, s erious games h ave been found effective in teaching college level students about the business processes and integration of IT goals and business strategy (Monk and Lycett 2011) . Computer games help students in four main characteristi cs (Leger, 2006) that are difficult to capture using traditional learning practices and require time in practice and experience: Combine information systems design with business strategy. Conceptualize the holistic view of the enterprise system in an orga nization. Provide insights on required technical skills. Show the effectiveness of collaborative work.

PAGE 45

27 Research in serious games has been growing since the last two decades. From around 100 papers in research published in 2004, the number exceeded 1200 paper s in 2013 (Laamarti et al. 2014) . Communications of the ACM in 2003 released a special or use simulation games in many areas. Games have a long history in research in general and business education in specific. Game applications applied in areas such as military, healthcare, management, IT, and education ( Robertson and Miller, 2008 , Smith, 2007 and Susi, Johannesson and Backlund, 2007). Examples of games used for military purpose are WarCraft and Close Combat (Michael and Chen, 2006). Healthcare games examples are like Hungry Red Planet develope d by the support of The National Institutes of Health (NIH) in the United States and Free Dive game developed by BreakAway Games (www.breakawaygames.com). The Beer Game (Anderson & Morrice, 2000) has been used to deliver supply chain knowledge and skills t o participants. The HEC Montreal Enterprise Resource Planning Simulation Game has been used to teach students about Enterprise Resource Planner (ERP) (Hopkins and Foster 2011) . A simulation game also has been used to teach about decision support systems (Ben zvi 2009) . Thus, we can conclude that games could be applied to many other disciplines too. Serious Game Evaluation Serious games have the potential to become one of the effective ed ucational instruments for instruction . Their ability to engage students provides the necessary motivation for continuous learning. However, there are still some challenges. First, empirical evidence of the effectiveness of these games is dearth in literatu re (All,

PAGE 46

28 Castellar, and Looy, 2014). Although empirical evaluation of serious games is very important, only 55 papers out of 7392 had empirical evaluation between 1996 and 2009 (Boyle et al. 2012) . The literature shows some e fforts based on individual contexts to create and evaluate serious games. However, researchers admit that high quality empirical research is scarce (De Freitas, 2006; Wouters et al, 2009). Game developers and educators need more empirical evidence of the i mpact of these games in order to support the creation and usage of serious games in education. The main reason for the lack of empirical evidence is that there is no solid framework for evaluation of serious games (Connolly, Stansfield, and Hainey, 2007a; De Freitas, 2006) . Items in game evaluation vary based on stakeholders which makes it difficult for a single evaluation framework to accommodate the interests of all parties (Mayer et al. 2014) . This makes it another challenge in the serious games literature. For example, game designers are interested in improving the functionality of the game while educators are interested in the effectiveness of the game in the teaching context. Evaluation frameworks are based on individual games. It is difficult to generalize from individual aspects of a specific game into a broad game category (Don di and Moretti, 2007). The literature shows two types of frameworks related to serious games: design frame works and evaluation frameworks . Design frameworks decompose the components of a game so they become clear for developers. Evaluation frameworks, on t he other hand are used to measure the effectiveness of these games. Evaluation is two types: evaluating the design and evaluating the effect (Ainsworth, 2003). Both types should be included in any evaluation framework (Arnab, et al, 2015). Design evaluatio n helps refining the

PAGE 47

29 game (Ogle, 2002). Forms of design evaluation could include expert reviews, one to one review, focused group review, or beta test. It is better performed during the implementation of the game and not at the end (Braden, 1992). Effect e valuation helps in validating the game or justifying the cost (Harpel, 1978). This literature review will describe design and evaluation framework components and theories. Game Characteristics Serious games are a special type of digital games like video g ames and simulation games. Digital games have been around for so much time with many design frameworks. These frameworks were adopted in the design of serious games and thus several frameworks and guidelines are now available. This allowed the development of serious games in many areas. The unified theory of digital games (Ralph and Monu, 2015) indicates two important characteristics in serious games design: game mechanics and narrative mechanics. Evaluation framework must be able to evaluate both. Game me chanics are elements that used to either create the game environment such as spaces, rules, and decision options or create challenges and competitions such as puzzles and quests. Gamification elements are also part of game characteristics. Gamification is defined as based mechanics, aesthetics and game thinking to engage people, motivate 2012 identified 15 elements that can be used as gamification elements. The y are Achievement, Avatar, Badges, Boss Fights, Collections, Combat, Content Unlocking, Gifting, Leader Board, Levels, Points, Quest, Social Graphs, Teams, and Virtual Goods. However, not all of them were found in literature. These elements are used to inc rease

PAGE 48

30 intrinsic motivation, which improves learning experience (Putz, and Treiblmaier, 2015) and affects other outcomes in behavioral and knowledge context (Hamari, 2013). Self Determination Theory SDT (Deci and Ryan, 1985) has been widely used in the eva luation of video games. It explains the motivation behind playing video games in general and serious games as well (Ryan, Rigby, and Przybylski, 2006; Tamborini, Bowman, Eden, Grizzard, and Organ, 2010). The theory suggests that human are motivated to cont inue in an activity if these three psychological needs are satisfied: autonomy, competence, and relatedness. These three needs were also used under the name Player Experience of Need Satisfaction PENS ( Ryan, Rigby, and Przybylski, 2006 ). Autonomy is the fr eedom to select choices in the activity. Competence is the ability to complete the activity. Relatedness is to show social connections with others during or after the activity. However, not all these variables were equal. Research shows more impact of Auto nomy and Competence on intrinsic motivation and less impact of relatedness (Deci and Rya n, 2002). Peng et al. (201 2 ) studied the effectiveness of autonomy and competence in a study that manipulates these factors. Results comply with the previous finding th Only few researches used relatedness and showed significant impact on motivation Narrative mechanics are elements that used to deliver mess ages mostly in a story format. Narratives could be embedded in the game, emerge during gameplay , or interpreted b y players (Ralp h and Monu, 2015). While game mechanics are the essential part of digital games, narratives are used to advance the game. We arg ue that serious

PAGE 49

31 games should use narrative mechanics to convey learning material to users. More about this is in the next section. Learning M echanics is usually associated with fun and excitement. The word and skills being taught by the game (Laamarti et al., 2014). An important part of any serious game is the leaning content. Game characteristics such as game mechanics and narrative mechanics should be utilized to convey learning materials to players. The GM LM model (Arnab et al., 2015) introduced learning operation of learning, that we typically model relying on learning theories and 5: P. 393). Similarly, Carvalho et al. (2015) also proposed a conceptual design that includes learning components and gaming components. To evaluate the learning mechanics , several theories and frameworks were used . The Constructivism view, for example has been widely used to explain game based learning (Li and Tsai, 2013). This view suggests that users build knowledge through experience. This theory has been used in business and simulation games (Lainema, 2009), (Tao, Cheng, and Sun, 2009), and (Thavikulwa t and Pillutla, 2010). Framew orks such as Gagne (Sreelakshmi et, al. 2015), Kirk (Schumann et al, 2001), or Blooms (Carvalho et al, 2015) were also used to evaluate the learning mechanics in serious games. The most detailed framework is Gagne framework. G agne et al (1970) proposed nine instructional events that are supposed to engage participants in the learning process. These events were highly used in education research

PAGE 50

32 to evaluate lectures, presentation, online courses, virtual reality based learning, a nd serious games too. First event is gaining audience attention. This can be done by providing a story or using sound/video effects. Second, participants should be informed about the learning outcomes. These outcomes should also serve as performance measur ements when evaluating the learning process. Third, after each phase, participants should receive a brief summary of previous accomplishments. This will keep students on track and keep the learning material connected. Fourth, players experience the actual presentation of the content, participating in the learning process. In the proposed game, the students will face several decisions and need to carefully evaluate each option and select the appropriate one . Fifth, provide some hints and guidelines. More add itional information will be provided to participants as they progress in the decision making process. Sixth, provide indications about the performance. Students should be given some signs that they are in the correct path. Seventh, give feedbacks. This wil l be in terms of explanations on why certain decisions are either correct or wrong. The remaining two events can be achieved through the game evaluation (in this case the experiment). The ough answering questions to evaluate their learning outcomes. However, the ninth event, which is apply learning in real job, can only be determined if the participant is actually applied the learning outcomes in a real situation. Learner Motivation Student s need to be motivated in order to participate in academic activities (Hess and Gunter, 2013). Serious games create the necessary motivation to play and interact with learning material (Swartout and Van Lent, 2003). Games satisfy the needs of people

PAGE 51

33 for en tertainment (Schramm, Lyle and Parker, 1961). In addition , games satisfy the needs for competence, autonomy and relatedness (Deci and Ryan, 1985). Thus, by playing users are motivated to learn. Self determination theory addr esses the motivation in games (D eci and Ryan, 2002) and explains when individuals are motivated to engage and enjoy participating in activities ( Ryan, Rigby, and Przybylski , 2006). Serious games may fail to deliver their learning outcomes if they fail to engage players (Monu and Ralph, 2 016). Motivation can be measured by engagement and enjoyment . Surveys have been used in literature to evaluate engagement and enjoyment. Connolly et al. (2012) identified 129 papers evaluating serious games by engagement. Outcomes When it comes to the objective of evaluation, the focus on evaluating video games (mostly commercial games) is to measure their success. These games are designed to amuse players. The motivation to keep pl aying is to get entertained but the outcomes are coming in other forms like buying the game, give higher rates, tell others about the game, and future play . Although serious games use entertainment to motivate players, the previous outcomes are not the ess ential focus of evaluation. Iten and Petko (2016 ) argue that serious games should focus on delivering the educational content rather than trying to entertain the player. Thus, the learning outcomes should be the central focus on evaluating the effectivenes s of serious games. The difficulty of coming with a general framework to evaluate serious games comes from the variation in learning outcomes. Serious Games are designed to deliver different learning material by using different combinations of game element s and learning elements to different audiences.

PAGE 52

34 In general, the learning outcomes can be classified into three main categories: Knowledge transfer, skill acquisition, and attitude change (All, Castellar, and Looy, 2014). They can be measured directly or i ndirectly. Direct effect is the evaluation of the change in knowledge, skill, or behavioral by using pretest/ posttest technique. These could be perceived effects or actual effects. An example of perceived effects is efficacy (Cheng and Su, 2011) and an ex ample of actual effect is declarative knowledge (Santhanam, Sasidharan and Webster, 2008). Indirect effect is the evaluation of transfer, which is measured by a second posttest usually in related environment (Mayer et al., 2013; Korteling et al., 2011). M ethodology The most two common methodologies used in evaluation frameworks are pre test/post test and questionnaire. Using multiple methods to avoid bias and incorrect assumptions is recommended (Wilson et al, 2016). Pre test is needed to compare the diffe rences in learning outcomes (Clark, 2007). Also, pre test and post test can be used to measure the effect on motivation and competence (Grace and Cohen, 2016). Experiments are usually used in pre test post test methodology. Experiments should also be used to measure items that are difficult to evaluate using surveys such as emotions and learning outcomes ( Becker et al., 2005) . Connolly et al. (2012) reviewed 129 papers that evaluate serious games and found that Quasi experiment is used when measuring knowledge and skills gains. Moreover, Hainey et al. (2011) used pretest / post test and treatment group/ control group to evaluate the effectiveness of serious games on knowledge acquisition. test/ post test while the learning outcomes were assessed using the exper iment. In experiment settings,

PAGE 53

35 participants usually are assigned randomly to control groups and treatment groups. The control group uses traditional teaching method while the treatment group plays the serious game. Questionnaire is also a method in serious games evaluation. In fact, surveys were found to be the dominant way to evaluate seri ous games (Connolly et al., 2009 ). However, some researchers avoid surveys because of their weaknesses in methodological design and dependence on wording and presentation style (Saari, Johnson, McLaughlin, and Zimmerle, 1988). Design Science Research The i nformation systems discipline is an applied science that aims to find solutions to organizational problems by utilizing cutting edge technologies. It falls between two ma in types of sciences: computer and social science. Computer science is also an applied science where it is governed by the laws of math . Organizational behavior follows the laws of social science. Thus, information systems research adheres to the laws from natural, computer and social sciences. That explains why IS normally adopts theories from other disciplines. Some researchers have called for more theory building, suggesting that information systems should develop its own theories (Benbasat and Zmud, 200 3). Other researchers are satisfied with the idea of borrowing the ories from different fields (De Sanctis, 2003; Agarwal and L ucas , 2005). IS research is also interested in the idea of developing artifacts. Benbasat and Zmud (2003) believe that the unit of study in IS research should be the technology. This creates a conflict in the IS discipline between researchers that are interested in building theory (rigor) and that focus on more practical artifacts (relevance). As a result, researchers in the IS disci pline have proposed different epistemologies on how to resolve

PAGE 54

36 this conflict. Bertelsen (2000) and Iivari (2004) suggested a reverse correlation where theory can be seen as a category of an artifact. Other researchers classified theories based on their pur pose. This allowed a special type of theory called design theory to emerge (Walls et all , 1992; Gregor, 2002; Markus et al , 2002 and Hooker, 2004). Under design science, artifacts are classified into four main categories: constructs models, methods and in stantiations. These types of artifacts are created and later evaluated according to their context. An important part of design science research is theorizing around the outcomes of implementing a specific artifact and justifying the reasons for building an d using specific types of artifacts (March and Smith, 1995). These concepts were essential for the IS discipline to add rigorous to design science research. Hevner et al (2004) summarized the relevant issues of design research and proposed guidelines to in crease the rigor of design science research. IS researchers have widely adopted those guideline s . Peffers et , al . (2006) built onto these guidelines and developed a coherent research process to conduct design research in information systems ( Figure 6 ) . The steps proposed by Peffers et , al . (2006) cover issues of theory, artifact, relevance, design, evaluation, rigorous and communicating the outcomes to the correspondence audiences. In addition to following specific steps, design s cience research should also utilize other non academic resources like conference papers, trainings and practitioner papers (Osterle et al 2010). Researchers in information systems must come together to enrich the content of design research by adopting this methodology. Producing more research will help in refining the methodology and push the development of similar methodologies forward.

PAGE 55

37 Figure 6 : Design Science Processes by Peffers et, al. (2006) Design science research evolved a s a methodology to carry out studies concerning IT artifacts and their impact on organizations. It has similarities with another common research method that has long been used in IS research, action research. Action researc h suggests that introducing chang es to an existing social environment and then observing th e se reaction i s an effective way to understand reality in its complex settings. The introduction of the artifact into an organizational structure is the planned changed in action research in IS. Ac tion research tends to be more of a qualitative study than design science (Baskerville, 1999). The data collection typically requires interviews, observations and case studies. Literature Summary The review of the literature on management of data warehous e development provides key points for development of a business strategy game. The literature shows that warehouse projects involve several architectures with important variables to determine an appropriate architecture. This research focuses on three main architectures: Independent Data Mart (IDM), Data Mart Bus Architecture (DBA) and Enterprise Data Warehouse (EDW). These architectures comprise the majority in a survey by Alsqour et al. (2012). Two approaches for adopting data warehouse architectures are chosen ,

PAGE 56

38 architecture selection and maturity models. Three factors were found to have indirect effect on architecture type through the strategic view: information i n ter dependence, t ask routineness , and level of s ponsorship . The strategic view indicates the perception of an organization on its IT and data sources. The maturity model for data warehouse development consists of several stages and features to evaluate these stages. Three features, data size, data quality, and level of integration are selected bec ause of their importance and ease of measurement. Serious games are proper method to explain complicated information by using entertainment as motivation. There are several elements in data warehouse development that make it more applicable to simulate the development process in a game environment. These elements include the alignment between strategy and capability, evolution of data warehouse architecture, and phases where learners experience the development process. This paper presents the design of a st rategy game that teaches topics related to data warehouse by using cost benefit analysis model concept. The cost benefit analysis model can be used to explain tradeoffs among resource acquisition for data warehouse development. In data warehouse developmen t, costs are associated with data extraction, data transformation, and data integration processes. Cost categories are hardware, software, and labor. In contrast, benefits are assessed by the value of queries and reports needed for the decision making proc ess. The game provides an analytical model to evaluate tradeoffs between costs and benefits to achieve a business strategy. The main objective of design science is to build an artifact to solve particular problems. Artifacts can be a form of constructs, m odels, methods, or implementations (March and Smith 1995). However, Gregg et al. (2001) see the artifact as an

PAGE 57

39 implementation of software. We argue that serious games are legitimate artifacts. Design science provides a methodological foundation for develop ment and evaluation of business strategy games such as Emerge2Maturity. The d esign science approach suggests that game implementation should have two main parts: game creation and game evaluation (March and Smith 1995) . The first part, game creation, aims to develop a prototype of a fully functional game. The second part, game evaluation, aims to apply the game in a relevant environment and measure the effectiveness of it to deli ver the required knowledge and skills. By considering the design research processes (Peffers et al. 2006) and the design research guidelines (Hevner et al. 2004) , the development of the game adheres to both the rigorous and the relevan t principles of design science research.

PAGE 58

40 CHAPTER III EMERGE2MATURITY DESI GN Introduction In a r ole pla ying game, participants make decis ions in a realistic simulation. Participants can observe the effect of their current decisions and either commit their choices or revise their choices . The game continues over a number of periods or phases with the environment of the game progressing over the phases. This section presents the design of Emerge2Maturity, a substantial artifact in the design science research paradigm. This section begins with the motivation of the game about learning difficulties and objectives. The second part provides a broa d framework of the game design involving interaction between strategy and capabilities and overview of decision models used in the game. The third part presents details of the decisions models for capabilities and phases. The fourth part presents details p rovided to players of the game and details of a spreadsheet prototype for the decision models. The final part presents an outline of evaluation of the game involving both perception of players and objective learning outcomes. Problem Identification As crit ical infrastructure for business intelligence, data warehouse projects involve large expenditures and high risk. A typical data warehouse project involves a large capital investment, typically more than $1 million in just the first year (AbuAli and Abu Add ose, 2010). Several studies indicated that data warehouse projects have more failures than success es (Shin, 2003; Wixom and Watson, 2001; Power, 1998; Ballou and Tayi, 1999). Inmon (2001) reported a failure rate of 70 to 80 percent, while Conning

PAGE 59

41 (2000) re ported a failure rate of 90 percent. Projects fail because they exceed the allocated budget, exceed the project deadline, or companies end up not using the system. Several factors affect the success of data warehouse projects. These factors include user tr aining (Shin, 2003) and the skill s of the team (Wixom and Watson, 2001). Despite this , the re has been little research on ways to train users to work on data warehouse projects. We claim that teaching users about what to expect when dealing with a data ware house project is part of the training that can contribute to the success of the project. Identifying a comprehensive list of difficulties in a data warehouse project is not a simple task. Data warehouse development is a complex process involving several re lated factors and extended time periods to reach a stable state. Data warehouse projects are investment in a costs in order to justify financing such a costly project. However, it is dif ficult to measure the benefits from a data warehouse project because they are often intangible , especially during initial periods of usage. Benefits may become tangible and increase as organizational units increase usage. The common evaluation tools for in vestments are Return on Investment ROI, Net Present Value NPV, and cash flow. These tools need defined cost and benefits values. In data warehouse projects, costs are uncertain and benefits are mainly intangible (Power, 1998) . Thus, evaluation of data ware house projects tend to use qualitative approach such as improving competitive advantage, productivity, and enhancing the decision making process (Power, 1998). These evaluation method s are not standard ized and could be biased by subjective perspectives . A systematic review of syllabi of data warehouse courses indicates a focus on conceptual aspects of data warehouse development . The courses typically explain topics

PAGE 60

42 like intangible benefits. However, there is no practical tool that can help learners understa nd the dynamicity aspect of data warehouse implementation project s . For example , courses lack a practical instrument to evaluate a data warehouse project in the presence of intangible benefits. Instructors have reported that students have difficulty obtai n ing the types of insights that are gained from experiencing project relationships involving costs and benefits over extended time periods. Benefits of data warehouse deployment are often intangible especially during initial periods of usage. Benefits bec ome tangible and increase as organizational units increase usage. In contrast, costs are tangible and high during data warehouse development especially with uncertain levels of data quality. Costs decline as benefits increase during use of a data warehouse over time. Learners need to gain experience from balancing costs and benefits as organizations acquire capabilities to support an understand the risk associated with data wareh ouse development. Risks are tangible and high during data warehouse development especially with uncertain levels of data quality. Risk declines as benefits increase during use of a data warehouse over time. Th is suggests th ere is a strong need to develop a new tool to support the current educational material on data warehouse development . Objective of Solution Serious games are tools that have been used in education for decades. A serious game can provide experiential learning to common educational material (Prensky, 2007). Serious games have been used in military to train soldiers (Michael and Chen, 2006) but, they have also been used in education in general (Johannesson and Backlund, 2007). The

PAGE 61

43 beer production distribution game, commonly known as the beer game, illustrates the bullwhip effect which is one of the supply chain problems that students typically have difficult y understand ing . The two causes of the problem come from demand forecasting and replenishment policy (Lee et al. 1997a). The game has been used in teaching supply chain management courses to simplify the problem to students (Boute and Lambrecht, 2009) The game started as board game Sterman (1989). The first introduction of an electronic version is made by Jaconbs (2000). The game was further developed for multiplayer use by Samure et al. (2005). Other researchers replicated the mathematical model in Excel worksheet (Boute and Lambrecht, 2009). The spreadsheet is used as part of a course. The length of the course is about 90 minutes. Another d evelopment was in R language (Edali, and Yasarcan, 2014). The study provides a mathematical model that replicates the board game. The mathematical model included assumptions, explanations, explanations, units for par ameters and variables. Continues attenti on to the game development can be seen as an indication of the successfulness of the game . In this research, we propose the development of a serious game, later named as Emerge2Maturity or E2M. The design of Emerge2Maturity addresses previous problems in u nderstanding data warehouse development. The game decomposes the complexity of data warehouse development into a sequence of standard steps. To increase the focus of learners , the game provides common factors across organizations such as da ta sources, budg et, and ex traction, transformation, and integration decisio ns . Learners are not distracted by other elements related to specific situations, remaining focused on the important aspects of data warehouse development. The game combines aspects of

PAGE 62

44 strategy and capability to help learners understand the relationship between them. The game simulates the development process to show trends, costs, and benefits with increased profits and decreased risks over time. Simulation provides a real like situation where lear ners can observe results of their decisions before implementing them. The simulation uses models to quantify costs and benefits related to choices. These models depict the relationship between the costs of acquiring capabilities to develop a data warehouse and benefits for using a data warehouse. We claim that the game will bring insights about data warehouse development and can be used as a supplementary tool in business intelligence education . Game Design The data warehouse capability maturity model propo sed by Sen et al. (2012) indicated several organizational capabilities that can be improved over the period of data warehouse implementation phases. Three of these capabilities are directly associated with the data sources: size, quality, and architecture. As the capability of an organization increases, organizations tend to extract and process larger amount of data, achieve better data quality through transformation, and upgrade their system architecture to store and process highly integrated data from var ious sources. As organizations mature over the period of the data warehouse project, the alignment between the business strategy of the organization and their capability of managing data warehouse system becomes stronger. Three components affect the percei ved strategic view of the organization: information independence, task routines, and level of sponsorship (Ariyachandra and Watson, 2010). The game framework, as shown in Figure 7 , combines the strategic view of a data warehouse w ith maturity of capabilities. The combination of information

PAGE 63

45 interdependence, task routineness, and sponsorship level determines the strategic view. As the strategic view increases to higher levels, organizations should increase capabilities to support the strategic view. Capabilities involve three important variables of a data warehouse, extraction size in the number of data sources, transformation level in the data quality processing for individual data sources, and integration level in data quality proce ssing to unify data sources. In some situations, events occur with an impact on strategy or capabilities. Events can be internal to an organization such as a merger or external such as a recession. Figure 7 : Interaction between S trategy and Capability Three factors determine the strategic view of the organization: information interdependence, task routineness, and level of sponsorship. Each factor can be measured on a three point scale of low, medium, and high. The strategic view starts at the lowest level (low for all factors) and increases over phases until it reaches the highest level. The strategic view in a phase influences resource requirements and relative importance of costs and benefits of capabilities. Table 3 shows examples of strategic view factors for each level. In Emerge2Maturity game, these factors are represented in terms of data source constraints for each phase in the project.

PAGE 64

46 Table 3 : List of factors that affect t he strategic view Levels Factors Descriptive statement Low Information interdependence Staff in each department requires data only from their operational data source. They don't see any benefits from getting data from other departments. T his is mainly bec ause they are busy in getting, cleaning and organizing the data. Task routineness The data are not automated. Employee needs to spend hours to structure the data in a readable format. Level of sponsorship Each department is overprotective when it comes to share data with other departments. Request forms are needed and almost always rejected. Med Information interdependence Majority of employee predicts that combining some data sources from different categories would benefit their reports and decision m aking. Many requests ordering data from other departments have been made since the last phase. Task routineness There is some sort of data automation since the last phase. However, it is not significant. Level of sponsorship Since the last phase, head of departments have observed the benefits of sharing the data with other departments. They are willing to share some of their data. High Information interdependence The previous developments of the data warehouse got the attention of the executive board. They believe that the integration of the data from all the sources will bring a competitive advantage to the organization. Task Routineness There is a growing need for data automation and improve the quality of the data. Level of Sponsorship The top ma nagement claims the sponsorship of the organizational data. There will be no boundaries or restrictions from head of departments.

PAGE 65

47 Game Design Decisions The foundation of Emerge2Maturity involves decisions in four design areas as summarized in Table 4 . Decisions in these design areas constitute the search process in the guidelines for design science research proposed by Hevner et al. (2004). The design of Emerge2Maturity followed an iterative process with simplification and decomp osition of design alternatives. Table 4 : Summary of Design Decisions Design Area Design Decision Organization specificity Organization independent using data source categories Player decisions Allocate resources for capabilities and observe strategy Game duration Multiple periods with learning effects and events Model development Cost benefit with demand driven decisions about extraction, transformation, and integration Serious games in business can organization specific or in dependent. Organization specific games typically occupy a large part of a course, while organization independent games support a unit or two of a course. Organization specific games typically involve much complexity, while organization independent games em phasize simplicity. Organization specific games, such as the FinGame (Brooks 2007), involve a hypothetical company with simulation and game features extending a detailed case study. FinGame covers a range of academic skills in financial management and anal ysis of financial statements. In contrast, the Beer Game (Sterman 1989) is organization independent, focusing on inventory decisions in a supply chain. The Beer Game emphasizes the BullWhip effect (Croson and Donohue 2006), a symptom of coordination proble ms in managing a supply chain.

PAGE 66

48 Since instruction about management of data warehouse development typically involves one or two units in a data warehouse course, Emerge2Maturity was developed as an organization independent game. Emerge2Maturity provides a s implified representation of data sources in categories to support usage in a limited part of a data warehouse course. Data source categories involve common features with implications about costs and benefits of utilizing data sources in a data warehouse. P layers focus on key aspects of data source categories, making resource decisions with economic consequences for an organization. To simplify player choices and model development, the initial version of Emerge2Maturity involves capability assessment in reso urce decisions made by players. Players observe strategy elements related to capabilities as a game progresses. Strategy elements in Emerge2Maturity involve the number of phases in a game and progression of constraints on budgets and resources as an organi zation matures in its deployment of a data warehouse. Because Emerge2Maturity involves both capability assessment and strategy, game play involves multiple decision making periods or phases. In transition among phases, Emerge2Maturity allows players to obs erve impacts of learning difficulties and events. The trade literature contains anecdotal evidence about learning difficulties for data warehouse projects. Kimball and Ross (2013 ) indicates that data warehouse projects have a steep learning curve. Merrick (2014) and Frolick and Lindsey (2003) provide several reasons that building a data warehouse may involve learning challenges for an organization. High reported rates of failure for data warehouse projects provide evidence about learning difficulties that o rganizations face. Because data warehouses mature

PAGE 67

49 typically over a long period, events (both internal and external) can affect budgets and resource limits. Emerge2Maturity uses cost benefit analysis, a simple economic model used in previous research about data sources . Rao and Osei Bryson (2008) proposed a cost benefit model that maps data sources to decision support views with estimated benefits to a firm and measurable quality levels. Higher quality levels give more benefits but also require additional co st. Ballou and Tayi (1999) developed a cost benefit model to determine the quality level that provid e maximum value. Emerge2Maturity uses a demand driven approach that anticipates demand first and then selects related data sources (Winter and Strauch 2002) . This approach minimizes the risk of including data sources that might not be beneficial to decision makers but a lso increases chances of missed opportunities. In Emerge2Maturity, cost benefit analysis applies to decisions about extraction, transformation , and integration of data sources. Each capability adds more value to queries and reports, but also involves fixed and variable costs. The first step toward benefiting from these sources is extracting data and storing them into a data warehouse. Ballou and Tayi (1999) prefer analyzing the data sources and select what is more relevant and appropriate for the decision makers. The second capability to increase the value of queries and reports is data transformation. Transformation enhances the quality of the d ata in accuracy, completeness, and timelines (Balloe and Tayi, 1999). Transformation also resolves inconsistencies, applying business rules, and summarization (Watson et al. 2002) . The third capability, data integration, is one of the most important decision factors because of its influence on the architecture of a data warehouse (Ramamurthy et al.

PAGE 68

50 2008) . Integrat ion of data means that relevant data are processed under the same rules and stored once forming a single point of truth (Gulledge 2006) . Game Flow Emerge2Maturity involves decisions about capabilities over phases as depicted in Figure 8 . In each phase, players make sequential decisions about capabilities for extraction, t ransformation, and integration. Players attempt to maximize net benefits using details about costs, benefits, and constraints. The demand for information assets provided by the capabilities is stochastic so players deal with uncertainty in assessing capabi lities. The game evolves over a number of phases representing budgeting or decision making periods. The strategic view in terms of data sources and budget constraints progresses over the phases, impacting coefficients for costs and benefits. Events influen ce coefficients and constraints on capabilities. The game terminates after a specified number of phases when the organization reaches its highest maturity level. Emerge2Maturity involves two models: Capability Assessment Model in each phase (CAM) and Confi guration Model (CM) for transition among decision phases. The following sections will elaborate more on these two models.

PAGE 69

51 Figure 8 : Overview of Decisions in Emerge2Maturity Capability Assessment Model (CAM) The Capability Asse ssment Model (CAM) provides the foundation for decision making in Emerge2Maturity. The CAM is an educational model to demonstrate relationships among important variables of data warehouse capabilities. It is not designed as a model for decision makers in a n organization. Figure 9 shows components of the CAM with decision variables, functions, and coefficients. The CAM manipulates three decision variables (data size X , transformation level Y , and integration level Z ) used in process es for extraction, transformation, and Extraction involves selecting data sources to include in a data warehouse. Transformation involves increasing the data quality through operations on individual dat a sources. Integration involves combining data from different sources, matching, and consolidating common data. For each decision variable, represents the incremental capabilities added in a phase.

PAGE 70

52 Figure 9 : Elements of the Capab ility Assessment Model (CAM) To indicate the contribution of decision variables for costs and benefits, the CAM uses functions for production, fixed costs, variable costs, demand, and benefits. In each function, the index i ranges over the number of data source categories. The phase index ( j ) represents a period for budgeting or resource decisions. Due to the CAM embedded in the Configuration Model (CM), each function uses incremental change in phase j for decision variables. The delta notation indicates the incremental level of a decision variable in a phase. The CM determines the values for weights applied to alter base coefficients used in the functions of the CAM. Thus, coefficients use subscripts for data source category ( i ) and phase ( j ) such as p ij for production. The CAM uses a cost benefit, demand driven approach, maximizing profit from capabilities, subject to constraints on capabilities and budget. Cost benefit model has been used to quantify cost and benefit values in data warehouse development research (Rao and Osei Bryson, 2008). The CAM provides three sub models to manipulate decision variables for extraction, transformation, and integration. The sub models can be used either sequentially or jointly depending on game design. The sub models de pend on

PAGE 71

53 data sources within the organization to determine the cost of inclusion and the value of use. The next section will elaborate more on how the game handle the data sources. Data Source Categories There is no doubt about the potintial value of data to enhance the decision making in organizations. Data is considered a competitive advantage. Organizations tend to store all kind of data that they own or can obtain (Kalfus et al. 2004). Data can come from various sources like operational databse, files, and exteral sources that are bought of gained. Data sources vary in their value and their cost to be included in data warehouse. The data must be extracted, transformed and integrated to become valuabel for top management decisions. However, several resear ch struggle to evalaute the benefits from information technolgoy projects(Gibson, Arnott, and Jageilska, 1990) or projects that involve business strategy (Irani and Love, 2001). The reason is that the benefits are mostly intangible and difficult to measure (Counnihan, Finnegan and Sammon, 2002). Tayyari and Kroll (1990) proposed a methodology of quantifying benefits based on subjective factors. This approach requires some justifications but it is still an appropriate method to quantify intangible benefits ( Hares and Royle, 1994). We adopt this approach and suggests three categories that should descrip and justify the cost and benefits of data sources. To manage complexity from a large number of data sources, data sources in the game are grouped into categori es. Categories facilitate determination of cost and value of individual data sources as all data sources in a category share features. The CAM classifies data sources using features related to costs and benefits. Features support

PAGE 72

54 grouping on model componen t from the cost benefit model such as fixed costs, variable costs, production, benefits, and risk. Emerge2Maturity uses features for technology, complexity, and size to define data source categories. Technology ranges from legacy systems to modern systems . Technology level can be accessed by features in programming language, database management system, operating system, and hardware platform. Complexity involves difficulty to transform diverse data for decision making. More complex data will need extensive time and effort to analyze. Size involves processing effort for data such as the number of rows. Larger data will require additional storage and maintenance. The value of each feature is represented on a three point scale, low, medium, and high, resulting in 27 different possible categories. Categories determine the coefficients of production, cost, benefit, and risk. Table 5 explains how the relationship of features to model components. The complexity and the size of the data s ource determine the amount of production, the variable cost, the benefit, and the risk. Technology and data size determine the fixed cost. Table 5 : Relationship of Category Features to Model Components Production (P) Fixed Cost (FC ) Variable Cost (VC) Benefit (B) Risk (R) Technology Complexity Size The CAM uses stochastic demand, common in models in operations management ( Schmitt, Snyder, and Shen, 2010; and Miranda, and Garrido, 2004) and econometric s

PAGE 73

55 ( Ben Daya, and Hariga, 2004; De Castro, Tabucanon, and Nagarur, 1997; and Browne, and Zipkin, 1991) Demand is a function of production plus risk. Expected demand is the production level determined by values for decision variables and the uncertain risk o r error term. Risk is modeled as a Normal distribution with mean of 0 and standard deviation of r , a function of features of a data source category. Decision Variables Decision variables represent activities to increase capabilities so that capabilities be come aligned with the strategic view of an organization. The CAM contains three decision variables for data size, level of transformation, and level of integration. Table 6 summarizes the three decision variables . Table 6 : Decision variables for extraction, transformation, and integration Decision Variable Value Extraction size X Number of data sources extracted Transformation level Y Level between 0 100 Integration level Z Level between 0 100 Here are descriptions of these three variables: The first variable, data size, involves data extraction from potential data sources that can be added to a data warehouse. Data size represents the capacity of the data warehouse, measured by the number of data s ources from each category used in the data extraction process. Thus, the data size decision variable in a phase is the number of data sources from each category to extract into the data warehouse.

PAGE 74

56 The second variable, transformation level, involves data q uality improvements applied to individual data sources. The transformation level represents processing to standardize data into common formats, units of measures, abbreviations, date representations, and so on. As transformation level increases, demand, fi xed costs, variable costs, and benefits also increase. The third variable, data integration level, involves data quality improvements concurrently applied to multiple data sources. The data integration level represents processing to match entities across data sources and reconcile differences among common entities. As integration level increases, demand, fixed costs, variable costs, and benefits also increase. Coefficients The CAM uses base coefficients for production ( p ), fixed costs ( fc ), variable costs ( vc ), and benefits ( b ). Table 7 describes each coefficient. Base coefficients are multiplied by the values from feature levels related to CAM components. For example, the production coefficient for a data source from high complex ity and medium size is p * 2.5, where high=1.5 and medium=1. The coefficients are then applied to decision variables .

PAGE 75

57 Table 7 : List of coefficients C oefficient Name Description p Production Determines the amount of queries that a re generated . Depends on decisions about the number of data sources, transformation level, or integration level. fc Fixed Cost Determines the cost of software and hardware regardless of the production. This coefficient is affected by the extraction proces s (data size). vc Variable Cost Determines the cost of the operation. This coefficient is affected by production. b Benefit Determines the value of a query to the decision makers M odel Functions To indicate the contribution of decision variables for cos ts and benefits, the CAM uses functions for production, fixed costs, variable costs, demand, and benefits. In each function, the phase index j , represents a time period for budgeting or resource decisions. Due to the Capability Assessment Model used inside of the Phase Model, each function uses incremental change in phase j for decision variables. represents the incremental capabilities added in phase j . The Phase Model determines the values for coefficients and weights used in the functions of the Capabi lity Decision model. Data size has a separate impact in each function as well as combining with transformation level and integration level. Because of these dependencies, the CAM is not separable on data size. Table 8 lists all functions in the CAM and Table 9 provides definitions for the incremental change and the normal distribution.

PAGE 76

58 Table 8 : Capability Assessment Model Functions Function Definition Production P( X ij , Y ij , Z ij ) p ij ij + p ij ij ij + p ij ij ij Fixed Cost FC( X ij , Y ij , Z ij ) fc ij ij + f c ij ij ij + fc ij ij ij Variable Cost VC( X ij , Y ij , Z ij ) vc ij p ij ij + vc ij p ij ij ij + vc ij p ij ij ij Demand D( X ij , Y ij , Z ij ) p ij ij + N(0, r 2 i ) + p ij ij ij + N(0, r 2 i ) + p ij ij ij + N(0, r 2 i ) Benefit B( X ij , Y ij , Z ij ) b ij p ij ij + N(0, r 2 i ) + b ij p ij ij ij + N(0, r 2 i ) + b ij p ij ij ij + N(0, r 2 i ) Table 9 : Other Definitions in the CAM Term Definition X ij X ij X ij 1 Y ij Y ij Y ij 1 Z ij Z ij Z ij 1 N( 0, r i 2 ) Normal distribution with mean of 0 and variance of r i 2 Objective Function and Constraints The objective function involves maximization of expected profit for decision variable choices for the number of data sources ( X ) from each categor y i ( X i ), the level of transformation for each category i ( Y i ), and the level of integration for each category i ( Z i ). Profit is revenue (R) minus total costs ( TC ), the sum of functions for fixed costs ( FC ) and variable costs ( VC ). Total costs ( TC ) for the number of data sources are a summation of costs for each data source category. TC j i ( FC(X i j , Y i j , Z i j ) + VC(P(X i j ) , Y i j , Z i j )) Revenue ( R ) involves a stochastic demand and production function for each data source category. Demand ( D ) represents the es timated queries that an organization uses. P roduction ( P ) is the number of queries that each data source can support. R j i (B(X i j , Y i j , Z i j ))

PAGE 77

59 Profit ( Pr ) is calculated by the revenue minus total costs : Pr j = R j TC j The optimization model is profit maximization for each data source category subject to constraints on the budget ( Bd j ) for total costs, minimum capability levels (data size, transformation level, and integration level) for each data source category, dependency of integration on transforma tion for each data source category, and maximum capability levels (data size, transformation level, and integration level) for each data source category . The model is solved for the expected demand without the risk term. subject to Budget limit: Data source requirements: (Minimum Data Size) Transformation requirements: (Minimum Transformation Level) Integrat ion requirements: (Minimum Integration Level) Integration dependency constraints: Data source demand limits: where MaxDS ij is the number of data sources for category i Transformation demand limit: Integration demand limit: The dependency on data size ( X ) in the profit function adds considerable Emerge2Maturity supports sequential choices for extraction, transformation, and integration. Initially, a player chooses X satisfying relevant constraints. After selecting X , a player selects

PAGE 78

60 transformation level ( Y ) satisfying relevant constraints using t he selected X value. After selecting X and Y , a player selects integration level (Z) satisfying relevant constraints using the selected X and Y values. Configuration Model Phases indicate the maturity of data warehouse development in an organization. A pha se represents a decision making period for budgeting and capability assessment. The emerging strategic view, cumulative capabilities acquired in previous phases, and events influence the configuration of a phase. Configuration of a phase involves weights a pplied to coefficients for costs and benefits and constraints for budgets, capabilities, and demand. Weights for Costs and Benefits Coefficients in the capability assessment model have base values. However, cost and benefit coefficients may change during t he game. Weights are applied for capability costs and benefits in order to reflect learning effects. As an organization acquires capabilities, it becomes more efficient with decreasing costs for deploying resources and effective with increasing benefits. T he early documented used of learning curve was by Wright (1936). He described that while workers in an aircraft assembly facility gained experience, the time and effort to build airplanes have decreased and the performance has increased. A l earning curve h as been used to explain the relationship between relative efforts and cost reduction and increased performance in several disciplines like software development (Pendharkar and Subramanian, 2004) and help desk support (Deng, 2005). The following function is used as a weight and multiplied by the cost and benefit coefficients.

PAGE 79

61 W = a * where a and b are parameters. The value of W starts at 100% when zero effort is taken . The a and b parameters will determine the decrease or the increase of the function and they are part of the game configuration. The following two sections will des cribe how this function in used for both cost and benefit coefficients. The fixed cost and variable cost coefficients have weights for every decision in each category i in each phase j ( W c Xij , W c Yij , and W c Zij ). The weights for t he costs will decrease ove r time ( Figure 9 ). Cost for each decision variable decreases as the game progresses contingent to efforts. The cost starts at a 100% rate when zero effort is taken and decreases up to a certain value when maximum efforts in extrac tion, transformation and integration are reached . The maximum effort is 100 when the data warehouse is fully developed. The efforts are represented in the x axis. The following function is used as a weight applied to the cost coefficients. Wc X, Y, Z = a * where a and b are parameters. The value of Wc starts at 100% when 0 effort is taken. Relative efforts are related to decision variables. Figure 10 : The cost weight Weights for the data size: Wc Xij = (a * X i j 1 b ) / 100 Weight s for the transformation: Wc Yij = (a * Y i j 1 b ) / 100

PAGE 80

62 Weights for the integration: Wc Zij = (a * Z i j 1 b ) / 100 The benefit coefficient has weight for every decision in each category i in each phase j (W X ij, W Y ij, W Z ij ). The weights for t he benefit will incre ase as an organization matures ( Figure 11 ). Benefit for each decision variable increases as the game progresses contingent on efforts. The benefit starts at a 100% rate when zero effort is taken and increases up to maximum value w hen maximum efforts in extraction, transformation and integration are reached . The maximum effort is 100 when the data warehouse is fully developed. The efforts are represented in the x axis. The following function is used as a weight applied to the benefi t coefficients. Wb X, Y, Z = a * ; where a and b are parameters. The value of W starts at 100% when 0 effort is taken. Relative efforts are related to decision variables. Figure 11 : The benefit weight Weights for the data size: Wb Xij = (a * X i j 1 b ) / 100 Weig hts for the transformation: Wb Yij = (a * Y i j 1 b ) / 100 Weights for the integration: Wb Zij = (a * Z i j 1 b ) / 100 Events Events are occurrences of actions with long term consequences initiated externally or internally by an organization. An i nternal event is an occurrence of actions

PAGE 81

63 within an organization such as a merger or divestment. An e xternal event is an occurrence of actions that organizations have no control such as a recession, regulation, or litigation. An organization reacts to events by adjusting t heir strategic view and/or capabilities. Emerge2Maturity uses a small set of events with a probability of occurrence. If an event occurs, phase configuration randomly adjusts data sources, data source categories, or budget constraint. ( Table 10 ). Table 10 : Impact of events Type Event Impact Internal Minor acquisition Increase number of data sources in selected categories Major merger Add one or more data source categories Minor divestment Drop data sou rces from one or more categories. Major divestment Drop a data source category and one or more data sources from some other categories External Recession Decrease budget constraint Expansion Increase budget constraint Minor regulation Add a category with low benefits and a small number of required data sources Major regulation Add a category with low benefits and a larger number of required data sources Score and Rank The configuration model also gives a score, stars and a rank for each player a t the end of each game. Scores, stars and ranks are calculated separately for each game. There are two versions of the game: educational and competitive. In the educational version, players make their own decisions each phase by simulating and committing d ecisions

PAGE 82

64 values. The game calculates and compare the expected and optimal for each phase the present that to players in phase summaries. However, players receive assistance at the beginning of each phase by adjusting the decisions to the ones that represen t the optimal decision values from the previous phase. So, players have the chance to start fresh at the beginning of each new phase. All players in this type of the game receive full score and all will be ranked first place. On the other hand, a competiti ve game allows players to build on their previous decisions. At the end of the game, the configuration model calculates the difference between the optimal profit value and the expected profit value. Players receive a score based on the following formula: S coreValue = 100 i ((GPOptProfit GPExpProfit) / GPOptProfit) * 100) The score is always out of 100 and the ranking will be based on the score. Values in the data model determine the number of stars for each score. For example, scores that are 99% or abov e are given five stars. Game Experiences Emerge2Maturity allows players to interact with the CAM through a sequence of configured phases. In each phase, Emerge2Maturity provides players with background about the strategic view, features of data source cate gories, cumulative capabilities in data source categories, impact of events, and available capabilities for allocation. Players receive qualitative measurement of costs and benefits before engaging in capability assessment decisions. Table 11 summarizes background provided to players in each phase.

PAGE 83

65 Table 11 : Information provided to players in each phase Item Details Budget Amount of budget available for the phase and each decision. Events Type, description , and impact Data source categories Features and impact on costs, benefits, and risk Capabilities Cumulative capabilities (data size, transformation level, and integration level) acquired and remaining capabilities to allocate for each data source catego ry Constraints Quantitative values of constraints Costs/Benefits Relative maturity adjustment based on capabilities acquired Emerge2Maturity can provide a variety of experiences based on the initial configurat ion. Number of phases can vary , providing a long term progression from an initial strategic view into the highest strategic view. Emerge2Maturity provides a mapping from an initial strategic view in the first phase into the highest strategic view in the final phase. In the Capability Assessment Mo del, Emerge2Maturity can provide a variety of experiences. In the simpler approach, the CAM can be presented in sequential form with players making decisions sequentially for all data source categories in the order of data size, transformation level, and i ntegration level. Players can be provided several simulation attempts to revise capability choices for each decision variable. In the more complex approach, players make decisions jointly for each category with several simulation attempts to revise all cap ability choices. In a mixed approach, players can provide capability decisions in a sequential order but then be allowed to revised

PAGE 84

66 capabilities for previous decisions. Essentially, all capability choices become tentative until finalized. Game scoring and ranking are used to evaluate and sort players based on their performance. Higher scores are given to players who meet or are close to meet ing optimal profits at the end of the game. Both score and rank encourage players to play again to beat their previous scores or compete against other players. During the game, players may simulate their data warehouse design. However, when they commit their decision, demand is sampled from a demand distribution. Both the profit from simulated reality and the profit from expected demand are shown and compared. In the educational version of the game, players receive assistance at the beginning of each phase. Thus, the game will always give the full score at the end. In the competitive game, however, players continue from th eir previous decisions and the score is carried over each phase. Design Summary This chapter identified a problem in teaching graduate students about the development of data warehouse. It suggests using simulation to depict the challenges in data warehouse projects. Data warehouse maturity model and architecture selection model show the potential capability to explain these difficult learning outcomes if they are used in a simulation game environment. The chapter presents the design of Emerge2Maturity, a se rious game for strategy and capability assessment of business strategy about data warehouse development . The game simulates the capability decisions for data warehouse development using the Capability Assessment Model and Configuration Model. Next chapter presents the development of the actual game.

PAGE 85

67 CHAPTER IV EMERGE2MATURITY IMPL EMENTATION Introduction The capability assessment model is implemented using Excel worksheet in order to validate its functionality. The Excel Solver and macros are used to facilitate th e creation of the prototype. Later various technologies such as JavaScript based website and Oracle database are used in the production implementation. This chapter presents both the prototype and the actual implementation of Emerge2Maturity. The chapter p resents the development and demonstration steps from Peffers et al. (2006) framework of design science research. Prototype Game Development We have created a prototype of the Emerge2Maturity game using Microsoft Excel software for the capability assessment model (CAM) and the configuration model (CM). There are 33 worksheets depicting a game with 3 phases and three categories for data sources. A documentation worksheet is used to track changes. An interface worksheet is used as an inp ut screen for players ( Figure 12 ). The blue sections indicate the input fields by the player. For each phase, the player is required to select the number of data sources, transformation level and integration level for each category. Other sheets are do ing the required calculatio ns (m ore on the calculations is described later in this section ) . The player may go back and adjust the inputs and see the effect of their changes. Then, the player moves to the next phase and decide on inputs for the next phase. In the real developed game , a limited on the numbers of attempts will keep players from simulating their decisions infinitely. Also, required restrictions and constraints are enforced .

PAGE 86

68 Examples include minimum number of data sources and next phase luck until previous phase is compl ete. Figure 12 : The Interface Worksheet Beside these two worksheets, the prototype included configuration sheets, Solver sheets, expected sheets, and simulation sheets. There are separate sheets for each phase. These sheets are h idden from the player. Figur e 13 shows how Categories are assigned in the first configuration sheet. This is performed once at the beginning of the game as the categories remain constant for the duration of the game. Figure 13 : First Configuration Sheet

PAGE 87

69 Features have three levels each has a numeric value: low = 0.5, medium = 0.75, and high = 1.0. In this example, categories A, B, and C are randomly assigned feature levels. Two macro buttons are used to allow o r block the change of the values. The values are then applied to the table in figure 14 . The table shows the coefficients and the base value for the production, fixed cost, variable cost, benefit, and risk. The base value of each coefficient is multiplied by the sum of the l evel values of the related features. For example, the related features for the production are complexity and size. For category A, the complexity and size values are both 0.75. Thus, the adjusted production coefficient for category A is 1000 * (0.75 + 0.75 ) = 1500. The risk uses sigma value as the variance. Figure 14 : Coefficients Table For each phase, a configuration worksheet determines the budget, constraints, and the effect of learning curve on coefficients. Figure 15 shows ho w the configuration for the second phase.

PAGE 88

70 Figure 15 : Second Configuration Sheet When the player enters values i n the interface sheet (Figure 12 ), the values go into the tables in the calculation sheets for ex pected and simulation (Figure 16 ). The tables have the CAM functions and they are used for the calculations. The stochastic demand uses the variance to generate random demands each time the player simulate the decisions.

PAGE 89

71 Figure 16 : An Expected Works heet for Calculations The solver worksheets do not depend on interface values. Instead, they calculate the optimal decisio ns using Excel solver (Figure 17 ). They also calculate the coeffici en ts for the Java LP solver. Figure 17 : A Solver Worksheet for Calculations There are similar sheets for each phase. The categories mostly remain consistent. However, coefficients are affected by previous efforts. Thus, weights are applied to coefficients at the beginning of the following phases .

PAGE 90

72 Actual Game Development This section describes the game controller, the database model, and game scoring details. Then game implementation details using JavaScript and Oracle database are summarized. Finally, the section demonstrates the game interface showing results from an actual gameplay . Game Controller and Database Model The game controller uses a database with static configuration details and dynamic tracking of gameplay . The data model in F igure 18 shows the internal database structure of the gam e. The database has configuration tables and operation tables. The game controller can provide various experiences based on the configuration tables. The results from a game are stored in the operation tables. The configuration tables are Game Configuratio n, Game Phase Constraint, Category Feature, and Category Constraint. The Game Configuration table specifies variables for game type, phase number, and number of simulation attempts. In a simple game type, players can provide capability decisions in a seque ntial order for all data source categories in the order of data size, transformation level, and integration level. Players get the correct solution and continue to build on it for the next phase. In the more complex game type, players make decisions jointl y for each category with the chance to continue on committed decisions. Phases can vary providing a short or long term progression from an initial strategic view into the highest strategic view. Players are provided several simulation attempts to revise ca pability choices for each decision variable.

PAGE 91

73 Figure 18 : Data Model for the Game Controller The operation tables are Player Game Configuration, GamePlay , Game Phase Decision Variable, Game Phase Play, Phase Simulation Decision Var iable, and Phase Simulation. The game controller obtains game parameters from the Game Configuration table. Then, the game controller saves values in related operation tables. For example, Game Play table will record values for date, attempt number, and ex pected profit. Players can play the game several times. Each time, the game controller saves the expected profit value for each game play.

PAGE 92

74 During gameplay , the game controller uses recorded values to calculate scores. Game scoring is used to track the pr ogress of players across phases. It also encourages players to play again to beat their previous scores. During the game, players may simulate their data warehouse design. However, when they commit their decision, demand is sampled from a demand distributi on. Both the profit from reality and the profit from expected demand are shown and compared. Players may achieve lower or near the exact value. The difference is subtracted from a starting point. In more advanced version of the game, scores may be used to compare efforts among competitors and players may play against each other or collaboratively to achieve goals. Implementation Details Emerge2Maturity implementation uses the MEAN stack architecture, a common architecture for JavaScript based web applicatio ns. The MEAN stack is a preferred architecture for web application development due to its light overhead, ease of use and customization, and a large and evolving library of packages that provide a wide range of functionalities to the developer. This stac k consists of Node.js, Aurelia.js, Express.js, JavaScript lp solver, and Oracle DB. Node.js is a server side JavaScript framework that provides the game functions and services via a REST API. Aurelia.js is a client side JavaScript web application framewor k, which consists of the game's views and controls. Express.js is a routing framework used to connect the Aurelia client to the Node server. The lp solver is a JavaScript base used to find the decision variables that optimize the profit based on phase cons traints. Emerge2Maturity uses an Oracle database to initiate, save and retrieve game related data. The game can be deployed on a Linux or Windows

PAGE 93

75 server, and does not require an external web server such as Apache, as a bundled Node.js server handles all AP I requests from clients. To extend the implementation of Emerge2Maturity game , two Gamification elements were added to the game. Playing games is a daily routine for many people. Enjoyment is the main motivation to play games. Gamification elements, if com bined with educational materials, can also bring enjoyment to the learning process and help elements in non 011). According to Werbach (2012 ), there are 15 game elements that are responsible for creating the enjoyment in games. Emerge2Maturity uses points to reward players for their accomplishments and a leaderboard to show the name and the score of the highest ranked players. The game narrati ves were carefully designed and written to deliver the learning objectives. A table in the appendix shows these narratives and their corresponding locations in the game. These narratives were inserted into the game database and can be updated as needed. Ga me Demonstration The game interface utilizes the Aurelia.js framework. It allows game interactions between a player and the game. The game is available online using any Internet browser. Initially, a player provides some demographic questions and obtains l ogin credentials. Then, a player chooses a game and phase 1 starts. At the beginning of a phase, the game provides the player with some qualitative information about the d a ta sources categories (Figure 19 ). Then, the player selects the best combination of data sources that maximizes the ). Players simulate their decisions and see potential

PAGE 94

76 results from a stochastic demand. Players have specified number of simulation attempts before they must commit one answer. After committin g an answer, Emerge2Maturity uses the lp solver to determine the optimal answers and show them to the player. The player continues to the transformation decisions and integration decisions. Figure 19 : Emerge2Maturity Game Phase 1 Interface

PAGE 95

77 Figure 20 : Phase Simulation for Extraction Decisions in Phase 1 outcomes. The game controller initiates the next phase and the player cont inues the game. At the end, the game controller calculates the total score and shows it to the player. It also compares the score to the previous scores and rank players in the leaderboard (Figure 21 ).

PAGE 96

78 Figure 21 : Game Score and L eaderboard Ranks As this demonstration indicates, Emerge2Mature provides a simulated, educational experience about management of data warehouse development. Players focus on data sources grouped by important features for technology, complexity, and size. F or data source categories, players manipulate capabilities for three related decisions in data warehouse development (extraction, transformation, and integration). Simulation allows players to observe impacts of a limited number of choices. Phase results c ompare player choices for capabilities with optimal choices. In transition among phases, players observe a learning effect, strategy changes for capability and budget constraints, and impact of external events. A simple point system and leaderboard provide incentives to improve and compete with other players.

PAGE 97

79 Implementation Summary Emerge2Maturity has two main models: capability assessment model and configuration model. The capability assessment model is implemented using Excel to validate the concept of d ecisions needed in data warehouse developments. Both the capability assessment model and game configuration were implemented using MEAN stack architecture, a common architecture for JavaScript based website. This stack also utilizes Node.js, Aurelia.js, Ex press.js, JavaScript lp solver, and Oracle database. Next chapter will highlight the evaluation process of Emerge2Maturity.

PAGE 98

80 CHAPTER V EMERGE2MATURITY EVAL UATION Introduction Rigorous evaluation of serious games is essential because it brings empirical evidence of the effectiveness of serious games. Despite the importance of the evaluation phase of game development, there is little research focusing on the game evaluation. This chapter begins by developing a serious game evaluation framework that can be used by ser ious game developers and users to evaluate the design of the game and the effectiveness of the game, the two distinct evaluation types identified in the literature. It includes a two phased design evaluation that involves both instructors with experience i n teaching about data warehousing as well as students interested in learning about data warehousing. Instructors, with experience in teaching IS courses and at least some level of understanding of data warehouse development, and students, who want to learn about event of learning. Does the game provide adequate learning experience about data warehouse development f rom It also proposes a large scale effect evaluation methodology that can be used to evaluate the game effectiveness in a large scale deployment . Participants from massive open online course MOOC will be invited to an e xperiment setting. The objective is to establish the effectiveness of using Emerge2Maturity as a supplement material for education. The effect evaluation is

PAGE 99

81 development using Emerge2Maturity game as a supplement tool compared to using a This chapter is organized as follows. The next section will draw on literature to form a comprehensive evaluation framework. The design evaluation section will eval uate the design of Emerge2Maturity using convenient samples from instructor and student populations. The effectiveness evaluation section will explain the experiment settings and measurement items of the effectiveness evaluation. The chapter end by a summa ry of Emerge2Maturity evaluation phase. Proposed Evaluation Framework Serious games can provide necessary motivation to students to increase course engagement and learning outcomes. However, it is important to evaluate the game before introducing it to par ticipants. The empirical evidence of the effectiveness of serious games is dearth in the literature (All, Castellar, and Looy, 2014). Few research papers involving development of serious games have empirical evaluation (Boyle et al. 2012). This is due to t he lack of generic evaluation frameworks that can be applied to any serious game. The literature show several research studies with serious games evaluation , but most of them are built for evaluating specific games (Mayer, 2006). A detailed evaluation fram ework that can be applied to any serious game is a key for improving the quality of research papers involving evaluation of serious games. Serious games consist of two main components: game mechanics (Ralph and Monu, 2015) and learning mechanics (Arnab et al., 2015). Game mechanics are elements that are needed to create and manage the game environment such as space, rules, and competition. These elements motivate individuals to play the game. Learning mechanics,

PAGE 100

82 on the other hand, are elements that are need ed to convey learning outcomes such as a story or an operation. These elements promote knowledge, skills, or behavioral change of game mechanics and learning mechanics p romoting necessary motivation for users to frameworks should evaluate serious games in both game mechanics and learning mechanics components (Arnab, et al, 2015). Evaluation frameworks should also evaluate the level of motivation that a game brings to players and whether the game successfully delivers the proposed outcomes. The literature differentiates between the design evaluation and the effect evaluation (Ainsworth, 2003). The design evaluation measures whether the combination of the game mechanics and the learning mechanics presents the proposed learning objectives. In other word, it establishes the content validity of the game (Straub, 1989). Expert reviews and beta testi ng are among the common methods for this evaluation (Ogle, 2002). Alpha and beta testing are common in computer based games and simulation games (Peng et al, 2012 and Rigby and Ryan, 2007). The literature shows several guidelines for design evaluation but learning (Gagne 1970). Other research also use the game efficacy (Freitas et al, 2010; Schumann et al, 2001 ) and self efficacy (Santhanam et al, 2008). On the other hand, the effect evaluation measures the effecti veness of the game in motivating players and the effectiveness of the game delivering the proposed outcomes. The most dominant model for this evaluation comes from the self determination theory ( SDT ) . The theory lists three human needs to motivate players to use the game:

PAGE 101

83 autonomy, competence, and relatedness (Deci and Ryan, 1985). Motivation can be measured directly (Backlund P. et al., 2011) or by using Engagement ( Kiili et al., 2014) and enjoyment (Wrzesien et al., 2010) constructs. Serious games should focus on delivering three main outcomes: knowledge, skills, or behavioral change (All, Castellar, and Looy, 2014). Common methods in this evaluation are experiment and survey (Connolly et al., 2009). From 102 papers identified in a systematic review conduc ted by Calderón and Ruiz (2015), about 38 % used experiment and about 90% used questionnaires. Based on previous literature, we propose an evaluation framework that combines the design and effect evaluation. The framework is validated using Emerge2Maturit y. We also suggest that this framework can be used to evaluate other serious games. The evaluation framework (Figure 22 ) considers categories of evaluation, measurements, and empirical research design as suggested by Connolly et al. (2009). Figure 22 : Proposed Evaluation Framework

PAGE 102

84 The main artifact is the game, which consists of game mechanics and learning mechanics. The design evaluation could take a form of expert reviews or beta version testing and could also use Gagne framew ork or efficacy construct. The effect evaluation could take a form of experiment or survey. Player Experience of Need Satisfaction PENS ( Ryan, Rigby, and Przybylski, 2006 ) measurements such as autonomy, competence and relatedness can be used in the process of evaluating the motivation behind playing games as explained by the self determination theory. Engagement and enjoyment are used to measure motivation. Learning outcomes such as knowledge acquisition or skills gain can also be measured. The proposed fra mework will be used to evaluate Emerge2Maturity. The framework is designed to provide guidance to serious game developers on how to evaluate their serious games. Its allows serious game developers to choose between different evaluation options depending o n the nature of the game and the types of outcomes it is designed to support Design evaluation Emerge2Maturity is designed to teach students several learning objectives about data warehouse development. The game design involves several components to convey the learning objectives to students. The purpose of the design evaluation is to determine if the game supports the learning objectives. Data warehouse development is commonly taught in a graduate level course in universities. Traditional teaching methods including lecture notes and presentations use the following topics as their main learning objectives: Understand the definition and data warehouse characteristics. Describe common examples of data warehouse architecture. Understand data warehouse maturity model.

PAGE 103

85 Understand the learning effects on cost of development and benefit of use. The learning objectives of the E2M game emphasize decision making across phases, decision variables for extraction, transformation, and integration, cost benefit trade offs, and organizational learning. In specific, E2M promote s learning about the following learning objectives: Describe some important features that can explain costs and benefits of data sources. Explain grouping of data sources into categories using common fea tures. Explain the complexity of data warehouse development. Understand common strategy factors that are needed in data warehouse development. Understand common capability decisions that are needed in data warehouse development. Explain the relationship be tween strategy and capability. Understand intangible benefits in data warehouse development. Explain learning effects with increased benefit rates and decreased cost rates over time. Understand the impact of events. To facilitate learning, Emerge2Maturity uses three components. First, Emerge2Maturity uses simulation to predict decision outcomes. Second, it uses categories to group data sources (use common features like technology, complexity, and size). Finally, it uses gaming environment with multiple leve ls, attempts, feedback about choices, and convenient performance summaries. To create challenges, Emerge2Maturity

PAGE 104

86 uses random demand for queries and events that can affect budget constraints. To promote player engagement, Emerge2Maturity uses p oints and a leaderboard. Table 12 lists E2M learning objectives and how they are addressed in the game. Table 12 : List of E2M Learning Objectives List of Learning Objectives Addressed in E2M by Describe some important features that can explain costs and benefits of data sources. introducing three examples of features which are technology, complexity, and size Explain grouping of data sources into categories using common features. grouping data sources into categories based on levels of featur es. Explain the complexity of data warehouse development. decomposing the project into standard phases Understand common strategy factors that are needed in data warehouse development. introducing common factors such as budget, phases, and constraints on resources. Understand common capability decisions that are needed in data warehouse development. introducing three main decisions which are extraction, transformation, and integration. Explain the relationship between strategy and capability. providing the business strategy and allowing players to manipulate the capability decisions. Understand intangible benefits in data warehouse development. quantifying the benefits as a total profit made by the organization. Explain learning effects with increased benefit rates and decreased cost rates over time. associating Reduction of costs and increasing of benefits with efforts made in previous phases. Understand the impact of events. allowing random events such as change in economy to affect the budget. Th e learning objectives are supported by using a combination of game mechanics and learning mechanics. The game mechanics are controlled by the capability assessment model and the configuration model. The learning mechanics, which are the game narratives, we re carefully written to deliver the learning objectives. A complete list of these narratives and their corresponding locations in the game is in appendix A . These narratives were inserted into the game database and ca n be updated as needed. Table 13

PAGE 105

87 shows section will evaluate the first seven events only. The a ssess performance event will evaluate the differences in knowledge acquisition and sk ills gain in a separate study. The r etain learning outcome event requires a follow up assessment and also will not be address ed . The objective of design evaluation is to establish the content validity of the game. Content validity is about measuring the representation of certain instrument l ike a serious game. It is established through subjective evaluations from reviewers and experts (Straub, 1989). Table 13 : Gagne Events and Their Corresponding Events in E2M Game Gagne Element Game equivalent Provide objectives Pro vide text based objectives at the beginning of the game. Gain attention Present the story of the company; create challenges to build a better data warehouse. Link to previous Provide summaries at the end of each phase and also at the end of the game. P resent content Read about terminology while playing. See the effect of decisions. Assess benefits and risks. Give guidance Provide a help document to play the game; give instructions on what need to be done . Practice opportunity Players can simulate thei r decisions before committing. Feedback Show the results of simulation attempts and committed decisions after phases and at the game summary. Assess performance Assess knowledge acquisition and skills gain by using pre test/post test (in a separate stud y) Retain learning outcomes Follow up assessment (not applicable) Design Evaluation Methodology The design evaluation involves collecting data to provide evidence that the game design addresses the proposed learning objectives. Also, it involves collecti ng feedback on how to advance the development of the game. Two populations are conc erned:

PAGE 106

88 instructors who teach about data warehouse developments and students who are interested in learning about data warehouse development. Validating the game from instruc them using the game as an educational tool benefits of using the game in class settings. The data is collected by a questionnaire. Participants are asked to use the game and answer questions regarding their experience. Some qu estions evaluate certain aspects of the game using a Likert scale while other questions ask for detailed feedback. The survey instrument has three items, game efficacy using 7 point Likert scale. The seven anchor points are 1 Extremely , 5 is 15 with several open ended questions (Table 14 and 15 ). Examples of these items are to help you gain for game efficacy is 7 statements (Table 1 6). Examples of these s questions related to the whole experience of playing the game (Table 1 7 ). This section has general quations about the game and an open ended question.

PAGE 107

89 Table 14 Item Code Item Sta tement Gne1 Emerge2Maturity provides clear learning objectives in the Welcome page Gne2 In the Game Preparation page, Emerge2Maturity presents a realistic business situation and creates a challenge to gain learner's attention Gne3 Emerge2Maturity provid es clear summaries at the end of each phase to help you gain insight about activities performed in the phase Gne5 Emerge2Maturity provides useful help documentation and adequate instructions about playing the game Gne6 Emerge2Maturity provides a useful s imulation feature showing the effect of capability decisions before committing actual decisions Gne7 Emerge2Maturity provides a useful summary of simulation attempts before committing to a capability decision Table 15 : Learning O bjectives Items Item Code Item Statement Gne4 1 Emerge2Maturity provides some important features that can explain costs and benefits of data sources such as Technology, Complexity, and Size Gne4 2 Emerge2Maturity depicts grouping of data sources into cat egories using common features Gne4 3 Emerge2Maturity decomposes the complexity of data warehouse development into a sequence of standard phases Gne4 4 Emerge2Maturity provides common strategy factors that are needed in data warehouse development such as budget, phases, and constraints on resources Gne4 5 Emerge2Maturity provides common capability decisions that are needed in data warehouse development such as levels of extraction, transformation, and integration of data sources

PAGE 108

90 Gne4 6 E merge2Maturity combines aspects of strategy and capability to help learners understand the relationship between them Gne4 7 Emerge2Maturity makes benefits tangible by calculating profits after each capability decision Gne4 8 Emerge2Maturity shows learnin g effects with increased benefit rates and decreased cost rates over time Gne4 9 Emerge2Maturity shows the impact of events, such as change in economy, on data warehouse development Table 16 : Game Efficacy Items Item Code Item Sta tement Efc1 I could successfully use E2M. Efc 2 I would be comfortable using E2M. Efc 3 It would be easy for me to become skillful at tasks learned from E2M Efc 4 Efc 5 I could apply new concepts th at I learned from E2M Efc 6 Using Emerge2Maturity is an efficient way for me to learn about data warehouse development Efc 7 From my experience of learning about data warehouse development concepts and playing Emerge2Maturity, I feel more confident about making strategy and capability decisions about data warehouse development in an organization Table 17 : General Items Item Code Item Statement Gnl2 What do you think about the length of time it took you to play one game? Gnl3 By considering everything, what is your final thought about Emerge2Maturity?

PAGE 109

91 Collected data consists of mainly quantitative results but also includes qualitative feedback. Mixed methods research type is used in this research. This type combines quantitative and qualitative approaches. Although there are no clear boundaries that separate these two approaches, the research will distinguish between them in terms of the research assumptions, data collection methods, and data interpretations and analysis. The m ix ed methods approach adds more value to the research quality as appose to a single approach (Creswell and Plano Clerk, 20 0 7). The evaluation seeks understanding of the subjective experience observed by participants from interacting with Emerge2Maturity game . Assumptions in subjectivism research paradigm includes (Crotty, 1998): Participants construct subjective meaning of their experience with an artifact. interpretation. The broad re the Emerge2Maturity game in providing an experiential learning about data warehouse quantitative and qualitative question s (Creswell, 2009). Quantitative questions are Qualitative questions, on the other hand, are open ended questions that aim to collect more detailed feedback on particul ar aspect of the game. The results from both types of To ensure reliability of the qualitative data, two procedures were considered. The first is document the process of th e data collection (Yin, 2003). The second is to provide clear definitions of codes used in the data analysis and to ensure that data are correctly

PAGE 110

92 related to their corresponding code (Gibbs, 2007). To ensure validity of the data, two procedures were consid ered. The first is to triangulate the quantitative and qualitative data (Olsen, 2004). The second is to use external auditor to examine the inferred information beyond the raw data (Creswell, 2009). ms to establish the content validity of the game education such as professors, PhD candidates, and instructors in the IS field are asked to evaluate the game in gen eral and the learning objectives in specific. The evaluation is Participants play ed th e game first. Then, they fill ed out a short survey with some open ended questions to advance the development of the game. Ten participants were invited. However, only six individuals completed the evaluation and provided feedback. All instructors hold at least a Results Data collection period t ook about two weeks. Only 6 participants completed the full survey. This section provides descriptive information categories by the items of the survey.

PAGE 111

93 o One reviewer disagrees that the list of objectives at the beginning of the ga me is sufficient. The feedback suggests that indicating the learning objectives relative to game views. Another suggestion says that players should play at least one game with learning objectives in mind then play other games. o Two re viewers indicated that the story told at the beginning of the game is not enough to gain the attention of players. Suggestions included adding more graphics effects and reduce the amount of text. o All reviewers agree that the game p rovides good summaries. One suggestion says that summaries should be in bullet points. o reviewers: quantifying the intangible benefit and complexity decompositi on. o from all reviewers: learning effect, combination of strategy and capability, common capability decisions, and using categories. o Two impact of events and example of category features.

PAGE 112

94 o igher: providing common strategy factors. o All reviewers agree that the game provides good help and instructions. One suggestion says that instructions should be in bullet points. o All reviewers strongly agree that the game allows practice before making a decision. This is because the simulator is a core part of the game. o All reviewers agree that the feedback from the simulator is sufficient. However, suggested improvements include showing resu lts in graphs and detailed elaborations on the limitation feedback. o o Comfortable use of E2M: Two reviews i o

PAGE 113

95 o Learn from E2M or lower. o . o o Players expr essed various answers regarding the length of the game. The answers range from adequate to normal to too long. All instructors agree that the game is suitable for teaching data warehouse development. Almost all reviewers indicated that the amount of text i n the game should be reduced . They suggested that some text are better represented as bullet points. Finally, additional graphs are recommended in sections like the phase summary and game summary pages. Discussion In the survey, there is one item that des cribes each of the seven elements of 9 items. Almost all items related to This is an indication that the design of Emerge2Maturity foll ows the standards of good practice related to educational material. The only exception is the gaining attention part of the game. Two reviewers felt that the story presente d at the beginning of the game was not sufficient. F eedback indicates using graphic effects and reduction in the amount of text

PAGE 114

96 would improve the overall game . Revisions of text in the game , in general , and the story specific ally is feasible and will be considered. However, graphical effects are currently out of the develo pment scope. Fut ure development will include enhancements to the game mechanics part of Emerge2Maturity including effects such as graphics, sounds, and some other gamification elements. Items related to learning objectives received different opinions from reviewers. Four major issues were raised . First, reviewers suggested using bullet points to list the objectives of the game at the beginning of the game. Second, reviewers recommended indicating each learning objective in its related part of the game. Third, reviewers su ggested creating a specific version of the game to explicitly indicate learning objectives while playing the game . Forth, reviewers indicate that the length of the game inhibits the focus on the learning objectives. These issues can be easily be resolved b y using the game as supplement material in conjunction with other learning materials like lectures and notes. The game was developed to be used as a stand alone teaching method. Players should receive additional information regarding the teaching topic and instructions on how to play the game prior using Emerge2Maturity. This will reduce the amount of time players spend on reading the text and allow them to focus on the experience. Also, the game allows creating multiple sessions with variations in the numb er of phases, allowing control over the time spent on each game. Game efficacy also received contrasting opinions. By calculating the means for each reviewer, four reviewers gave the game 5.5 out of 7 and two reviewers had game efficacy scores above 3.0 . T he average of all means is 5, indicating that the game , in general , ha s the potential to be effective learning material. Concerns from reviewers

PAGE 115

97 focus on the amount of text that players need to read before and during the game session. Moreover, reviewers w ere also concern about the length of the game. There is also a concert that p layers might use the game as a way of competing with others or trying to achieve better score as oppose to learn from it. However, the game will allow for shorter games and longer games as options to players. In addition , the game should be used as a secondary educational tool not as a stand alone method. Instructors who teach related courses should explain the topics before players engage in playing the game. In conclusion, the ga me received positive feedback from the instructor reviewers. All instructors indicated that the game is an interesting and effective teaching tool that can be used to teach about challenges in data warehouse development. The results and feedback from this study will help to enhance the development of the game. Classroom Evaluation Classroom evaluation of E2M aims to establish the content validity of the game from the stud . Graduate students who are taking a data warehouse course at Universi ty of Colorado Denver are asked to evaluate the game in general and the learning objectives specifically . This evaluation is also based framework (1970). The evaluation also uses two game efficacy items. There are two main types of partic ipants: students who attend the class face to face and students who take the class online. The class is recorded so online students can watch the lectures. The process starts by providing a concepts lecture about data warehouse and data warehouse developme nt. The lecture took about an hour and followed by a game concepts lecture which lasted about twenty minutes. Participants got a short break for ten minutes. Then, students received a game demonstration session for ten minutes. Participants had the

PAGE 116

98 chance to play at least one game by themselves. Then, they are asked to fill a short survey with some open ended questions to advance the development of the game. T he t otal number of students in the class is 54. Two students did not take the survey providing com plete responses from 52 students. The survey was an assignment in the course and completed surveys will receive full mark regardless of the answers. This evaluation will wa Results We received 52 completed responses. There are 29 (55.8%) students who attend the class, listened to the lectures, and complete the survey. The other 23 (46.2%) students viewed the lectures onlin e. This section presents descriptive information about each item in the survey. Also, it shows results from T test analysis comparing the results from the two student groups in order to check if there is a significant difference between them. (Figure 23) The majority of the participants agree that the objectives of Emerge2Maturity game is clear. Only two respondents indicated a disagreement opinion.

PAGE 117

99 Figure 23 : Gne1 Descriptive Analysis (Figure 24) The majority of the participants agree that the business situation is realistic and gained their attention to play the game. Three participants were undecided and only 2 respondents disagree with this statement. Figure 24 : Gne2 Descriptive Analysis (Figure 25)

PAGE 118

100 The majority of the participants agree that the summaries provided at the end of each phase were clear. Only three respondents disagree to this statement. Figure 25 : Gne3 Descriptive Analysis (Figure 26) There are nine proposed learning objectives in the evaluation. Students are asked to evaluate these objectives individually. The results will also present the mean across all objecti ves for each participant. o Gne4 : the majority of the participants agree that the provided examples of features can explain costs and benefits in data warehouse projects. Two participants were undecided and only one respondent disagree .

PAGE 119

101 Figure 26 : Gne4 1 Descriptive Analysis o Gne4 (Figure 27) : The majority of the participants agree that Emerge2Maturity illustrates grouping of data sources into categories using examples of common features. Thr ee participants were undecided and only one respondent disagree. Figure 27 : Gne4 2 Descriptive Analysis o Gne4 (Figure 28) : The majority of the participants agree that Emerge2Maturity breaks down the co mplexity of

PAGE 120

102 data warehouse development to few standard phases. Three participants were undecided and none of the respondents disagreed with this statement. Figure 28 : Gne4 3 Descriptive Analysis o Gne4 (Figure 29) : The majority of the participants agree that Emerge2Maturity gives examples of some common strategy factors related to data warehouse development. One participant was undecided and only one respondent disagree. Figure 29 : Gne 4 4 Descriptive Analysis

PAGE 121

103 o Gne4 (Figure 30) : The majority of the participants agree that Emerge2Maturity gives examples of some common capability decisions related to data warehouse development. One participant was undecided and only o ne respondent disagree. Figure 30 : Gne4 5 Descriptive Analysis o Gne4 (Figure 31) : The majority of the participants agree that Emerge2Maturity illustrated the relationship between business strategy and capability. One participant was undecided and only one respondent disagree.

PAGE 122

104 Figure 31 : Gne4 6 Descriptive Analysis o Gne4 (Figure 32) : The majority of the participants agree that Emerge2Maturit y quantifies the intangible benefits in data warehouse projects. One participant was undecided and only one respondent disagree. Figure 32 : Gne4 7 Descriptive Analysis o Gne4 (Figure 33) : The majority of the par ticipants agree that Emerge2Maturity explains how learning effect occurs in data

PAGE 123

105 warehouse projects. However, six participants were undecided. None of the participants indicated any of the disagree options. Figure 33 : Gne4 8 Des criptive Analysis o Gne4 (Figure 34) : The majority of the participants agree that Emerge2Maturity illustrated how some certain events can affect data warehouse projects. Five participants were undecided and six respondents disagree. Eve nts in the game are random. A player need to play several games in order to observe a change in the Economy. We anticipate some students did not play enough games and thus, could not see the impact of events.

PAGE 124

106 Figure 34 : Gne4 9 De scriptive Analysis among scale categories. The result shows that, on average, all participants indicated that emerge2Maturity addresses all proposed learning objectives (7[13%] S trongly agree, 34 [65%] Agree, and 11 [21%] Somewhat agree). (Figure 35) The majority of the participants agree that the help information provided in the game is useful. One participant was undecided and three respondents disagree to t his statement.

PAGE 125

107 Figure 35 : Gne5 Descriptive Analysis (Figure 36) The majority of the participants agree that the simulator is useful. One participant was undecided. None of the participants indicated any of the disagreement options for this statement. Figure 36 : Gne6 Descriptive Analysis (Figure 37) The majority of the participants agree that the results from the simulator is useful. Only one respondent disagr ees with this statement.

PAGE 126

108 Figure 37 : Gne7 Descriptive Analysis Two items were used to evaluate this variable. This section presents each item separately. Also, it presents the mean of the two items for all par ticipants. o (Figure 38) : The majority of the participants agree that playing emerge2Maturity is an efficient method to experience the data warehouse development. Only three participants were undecided. None of the participants indicated an y of the disagree options. Figure 38 : Efc6 Descriptive Analysis

PAGE 127

109 o (Figure 39) : The majority of the participants agree that playing Emerge2Maturity make them more confidence in making decisions about data warehous e development. Eight participants were undecided and only two respondents disagree. Figure 39 : Efc7 Descriptive Analysis among scale categories. The r esult shows that, on average, almost all participants indicated that emerge2Maturity is a good tool to teach about data warehouse development (7[13%] Strongly agree, 26 [50%] Agree, and 17 [33%] Somewhat agree). Only two responses, on average, fall into th e undecided category (2 [4%] Neither agree nor disagree). (Figure 40) provides various levels ranging from beginner to advance. Participants were asked to pl ay an intermediate game as much as they want. Intermediate games consist of three standard phases. There are five maximum simulation attempts for each

PAGE 128

110 decision in each phase. The majority of participants indicated that the length of the game is just right. Seven participants indicated that the game is long compared to six participants who indicated short. One participant indicated that the game is very long and another participant indicated that the game is very short. Figure 40 : Gnl2 Descriptive Analysis (Figure 41) We asked participants about their overall evaluation of the game. The majority of the participants believe that the game is a good fit for teaching about data warehouse development but with so me additional changes. Few participants indicated that the game is fit without any changes. Only one respondent indicated that the game is not suitable at all.

PAGE 129

111 Figure 41 : Gnl3 Descriptive Analysis To check if there is any signi ficant difference between the two student groups (on campus and online), we used two tailed independent samples T test with 95% confidence level. IBM SPSS version 4 is used in the analysis. The null and alternative hypotheses for the test are: H0: There is no significant difference between the students who attended the class and the students who did not attend the class H1: There is a significant difference between the students who attended the class and the students who did not attend the class. Almost all 0.05. Thus, we can assume equal variance in responses. The only exception value is responses for t his item from online students . Table 18 test for evidence to reject the null hypotheses. The only exceptio ns are for items Gne5 and Gne7

PAGE 130

112 which have values of (p=0.042 and p=0.012) respectively. These values provide evidence that there is a significant difference in responses between the two groups with online students demonstrating decreased agreement when com pared to campus students . Thus, we can reject the null hypothes is for these two items. Table 19 shows the means for the two groups.

PAGE 131

113 Table 18 : T test results

PAGE 132

114 Table 19 : Mean values for the T wo Groups Table 20 test values for Game efficacy items. T test t enough evidence to reject the null hypotheses. Table 20 : T test results for Game Efficacy Items Table 21 test values for the gene ral items. T test for the two t enough evidence to reject the null hypotheses.

PAGE 133

115 Table 21 : T test results for General items Discussion Participants also provided detailed feedback for each item. A systematic approach is used to analyze the statements. First, statements are organized by their corresponding item in one document. Second, similar texts are grouped under one code. Color coded text is used to highlight the text in the document. Third, each code is described under a theme. Finally, themes are integrated into a general conclusion. This section will describe the detailed feedback and the results of their analysis. After reading the detailed feedback from participant s , we were able to organize In this category, subjects show positive attitud e toward the evaluated items. The second code is In this category, subjects propose enhancements that can statements In this category, subje cts expressed negative feelings or attitude toward certain items or aspects of the game. Appendix E lists codes and their corresponding texts form the feedback. This study is not interested in the first code, compliment statements, as it does not help with enhancing the development of the game. Instead, this study focus

PAGE 134

116 on examining the critical statements and using suggestion statements to advance the development of the game. Three main themes emerged by combining the codes from each item. The first theme describes general aspects of the game. Statements, although some are provided under one item, can also be generalized to cover other parts of the game. The majority of criticism statements under this theme describe two main aspects of the game: the amount of text content and the lack of visual content. Many participants are frustrated by reading through the learning objective and the game concept pages. They say that there is too much to read about or the presentation of the information is not appealing. S uggestions say that bullet points, colored texts, figures, and graphs should be used to list or depict important topics. Some information are better presented in figures like instructions of how to play the game supported by some pictures from the game. Ot her information like definitions and extra details are better presented by a link to an external resource page, pop up windows when hovering over the text, or an expand/collapse trigger. On the other hand, participants criticized the lack of detailed infor mation in other parts of the game. They indicated that they would like to read more about what is happening during the game. Suggestions say that there should be more details about the business and the industry of the company presented in the game preparat ion page. In addition , participants want to learn more about data sources, costs and benefits, and the difference between simulated and expected profits with some real examples. Some statistical analysis such as return on investment (ROI) could also help i n the comparison between committed and optimal decisions.

PAGE 135

117 One participant indicated that the game should use engaging language compared Another suggested that any theory or a cademic knowledge should be separated from the game. Several indicated that the presentation of the game should be enhanced. In general, students who played the game were expecting to see more elements from video games such as animations or pictures. Some participants also suggested a video that shows how to play the game with some emphasis on the aspects of the game that highlight the learning objectives. Alternatively, the game should show a pop up message to indicate what to learn from certain parts whi le players are engaged in the game. According to the feedback, participants sometimes struggle to connect between what they are doing in the game and what the game proposes to teach. Some participants had to play several games and reread the learning objec tives in order to form a link between a concept and its practical example. Statements in the specific aspects theme criticize and provide suggestions that can enhance particular traits of the game. Participants are interested in reading more about the exa mple company used in the game. They also want to read about the industry in general and how data warehouse development affects the business in specific and the industry in general. Almost all individual learning objective received some criticism by not bei ng clearly addressed in the game. It is acknowledged that some learning objectives are difficult to observe. For example, the effect of events on business strategy happens randomly in the game. Players need to play the game several times to observe the eff ect.

PAGE 136

118 However, suggestions indicate that more elaborations of learning objectives are needed during the game not only in the game concept page . While the previous suggestions are feasible, some suggestions are outside the scope of the current development bu dget . These statements are categorized under outside the scope theme. One says budget should vary for each game play. Although the current development fixes the budget for each game, it also provides several levels with different budgets. Another says that players should decide on the number of simulation attempts. The number of simulation attempts is set, so players have limited trials before they commit a decision. Open trials will allow players to simulate all possible decisions. The purpose of the simul ator is to infer the cost and benefits of each category by using a combination of qualitative information with the simulation attempts. However, different game levels can provide different attempts based on the level of difficulty. Finally, the game should game presents general text with customized tables showing the result values for each player. Players should examine the table and compare their values with optimum values. The sugge stion would like to see customized summary indicating, for example, where the player should have invested more. Addressing the above issues requires significant modifications to the data model of the game and unlikely to be part of the current development plan. Combined Discussion This section combines views from both the instructors and the students and also summaries important fi ndings and suggestions. Table 22 learning with summaries from both types of participants. The common sim ilarities

PAGE 137

119 between the two groups are the need to reduce the amount of text in the game, use more graphics, and provide more details in certain parts of the game. Both groups indicated the importance of linking the learning objectives from the beginning of the game with the practical actions in the game. In contrast, instructors want to reduce text for example in the game preparation view by using bullet points while students want to learn more about the industry and the business. The feedback will be consid ered for future development. Table 22 : Findings Summary from Both Data Collections Event Gne1 5 / 6 Agree Indicate learning objectives during the play. 50 / 52 Agree Separate educational material from the game material. Reduce text and use bullet points. 4 / 6 Agree Use graphics and reduce text. 50 / 52 Agree Provide more details about the company and the industry. to 6 / 6 Agree Use bullet points. 49 / 52 Agree Provide more details and statistical tools. Gne4 5 (on average) / 6 Agree Establish clear link between learning objectives and practice in the game. 48 ( on average) / 52 Agree Provide more details that links between actions in the game the learning objectives. Indicate learning objectives in their corresponding part of the game. 6 / 6 Agree Use bullet points. 48 / 52 Agree Provide detailed instructions and short instructions. Gne6 6 / 6 Agree Give more details. 51 / 52 Agree Provide more details on cost, benefits, simulated, expected, and optimal values. Gne7 6 /6 Agree Use graphs and give more details. 51 / 52 Agr ee Provide more details in form of graphs or additional text and statistical tools for analysis.

PAGE 138

120 I t is interesting to notice that although participants are frust rated from reading certain text in the game , like concepts and instructions , they also want to read more in other parts of the game , like details about the business and the results of their decisions. A video demonstrating the game concept and showing instructions on how to play the game is a preferred choice to present at the beginning of the g ame. Detailed explanations regarding expanded text and comparison tools such as ROI are preferred choices to summaries efforts in the game. In addition , the game should be more explicit in demonstrating its learning objectives. More elaborations during the game should indicate the learning objective that is expected to be observed and learned. The indication can appear as a pop up text or to be included in the text. Finally, participants want the game to look and feel like a video game by adding special eff ects with more visual contents and less text. The project h as several avenues for continual improvements. From the design side, the game will continue to develop by incorporating the suggestions made by participants from the evaluation studies. The game wi ll also be integrated in a data warehouse course with a tutorial explaining the game concepts and learning objectives. The course will provide assignments that allow students to use the game to find answers. Effect Evaluation Recent years have showed inte rests in educational games in both development and evaluation. Serious games in general , and Emerge2Maturity in specific , are meant to be used for teaching. Before we can claim that E2M is a good educational instrument, we need to evaluate the effectivenes s of the game in delivering the proposed learning objectives . The purpose of the effect evaluation is to measure the learning outcomes of

PAGE 139

121 learners. For years, professors and instructors have struggled explaining the complexity of data warehouse development s by using traditional learning material such as PowerPoint slides and lectures. Emerge2Maturity can be used as a supporting material to teaching about data warehouse dev elopment using E2M compared to using a the experiment where the effectiveness of E2M is evaluated . Hypotheses Autonomy and competence are two psychological needs that if satisfied, players are motivated to play games. Games should allow players to feel they are in control of their actions. Several studies have found direct effect between these needs and motivation to play the game. Researchers such as Peng et al. (2012), R yan, Rigby, and Przybylski (2006), and Przybylski, Ryan, and Rigby (2009) have studied the dire ct effect between a utonomy and competence on enjoyment. Other researchers such as Niemiec and Ryan (2009), Gagne (2003), and Reeve, Jang, Carrell, Jeon, and Barc h (2004) stu died the direct effect between a utonomy and c ompetence on engagement. Based on these findings, we posit the following hypotheses: H1a. Higher perceived Autonomy leads to an increased perceived Engagement. H1b. Higher perceived Autonomy leads t o an increased perceived Enjoyment. H2a. Higher perceived Competence leads to an increased perceived Engagement. H2b. Higher Perceived Competence leads to an increased perceived Enjoyment. Serous games are pedagogical tools similar to other educational to ols such as lecture notes and presentation. However, not all learners are capable of using computers

PAGE 140

122 or games. Studies have suggested evaluating the efficacy of ser i ous games (Backlund, Engstr m, Johannesson, Lebram, and Sj d n, 2008) and efficacy of pl ayers ( De Freitas and Jervis, 2006). Some researches show influence of efficacy on motivation and learning outcomes (Schunk, 1995 and Pajares, 1996). Based on these suggestions and results , we posit the following hypotheses : H3a. Higher perceived efficacy leads to an increased perceived Skill. H3b. Higher perceived efficacy leads to an increased perceived Knowledge . The Constructivism theory suggests that users build knowledge through experience. This theory has been used in business and simulation games (L ainema, 2009), (Tao, Cheng, and Sun, 2009), and (Thavikulwat and Pillutla, 2010). Players who are engaged in games are more receptive to learn. Santhanam et al. (2008) used both perceived skills and knowledge to evaluate the learning outcomes. In fact, som e research show s that serious games are more effective in delivering learning out comes than motivating players (W outers, van Nimwegen, van Oostendrop and van der Spek, 2013). Thus, we posit the following hypothese s : H4a. Higher perceived Engagement leads t o an increased perceived Skill. H4b. Higher perceived Engagement leads to an increased perceived Knowledge. H5a. Higher perceived Enjoyment leads to an increased perceived Skill. H5b. Higher perceived Enjoyment leads to an increased perceived Knowledge. Th e res earch model is shown in figure 42 and is drawn from the prop osed evaluation framework. There are four types of variables in the model. The learning outcomes (dependent variables) are perceived skill and perceived knowledge. Motivation (mediator variab le) is measured by engagement and enjoyment . Game characteristics

PAGE 141

123 (independent variables) will be measured by autonomy, competence and game efficacy . Previous knowledge and experience in database and data warehouse development are used as control variable s . Figure 42 : Emerge2Maturity Evaluation Model Experiment Design Participants from Massive Open Online Course (MOOC) specialized in data warehouse development will be invited to join a learning session. They will be randomly assi gned to one of two groups: the control group and the treatment group. Demographic information such as gender, age, income level, and educational level will be collected . Both groups will be given a pre test before the experiment and a post test after the e xperiment. The pre test is to evaluate prior perceived confidence in knowledge and s k ills. The control group will be given access to a traditional learning material. The treatment group will be given access to play Emerge2Maturity in addition to the tradit ional learning material . Post test will assess the perceived confidence in knowledge and skills after the experiment. Participants may be compensated for their participation. The experiment is designed to measure the effect of using Emerge2Maturity game wi th

PAGE 142

124 conventio nal learning material (Figure 43 ). The game should bring more insights and depicts deeper contents that are hard to explain in conventional material. Figure 43 : Experiment Design The data will be analyzed using a t t est and Structural Equation Modeling SEM method s. T test is used to assess whether the differences of the mean between two groups is significant. The sample size is an important factor when using t tests. Although there is no minimum sample size for the te st, low number of participants will result in low statistical power. SEM allows more complex analysis such using m ediated factors (Bag ozzi and Yi, 2012). However, sample size is an important factor when using SEM. In order to see effects, minimum sample si ze must be cons idered (Westland, 2010). Table 23 shows recommended minimum sample size based on parameter values such as effect size, power level, and probability level. The sample frame is expected to be 10,000 students who have completed the essential da ta warehouse course on Coursera.org. With response rate of 3%, t he expected number of participants should not be less than 300 total .

PAGE 143

125 Table 23 : Minimum Number of Sample Size (Soper, 2017) Variables: 7 Items: 35 Probability: 0.01 Probability: 0.05 Probability: 0.1 Power Level Power Level Power Level Effect Size 0.1 0.3 0.5 0.1 0.3 0.5 0.1 0.3 0.5 0.99 4081 381 100 3459 323 100 3176 297 100 0.90 2747 257 100 2242 210 100 2015 189 100 0.85 2473 231 100 1994 187 100 1781 167 10 0 Due to the length of the experiment, the study will reduce the actual data collection to only one motivation variable (enjoyment) and the two learning outcomes (knowledge and skills). It is important to establish the effectiveness of the game in delive ring the proposed learning outcomes first before evaluating other components of the game. In addition , the game will continue to develop by enhancing the game interface based on future planned designs and based on feedback. In order to evaluate the game el ements, another study using only a survey is possible. Constructs and Measurement Items There are three main constructs in this experiment: perceived confidence in knowledge, perceived confidence in skills, and enjoyment. The measurement items for both con structs will be evaluated using the 7 point L ikert scale. The 7 anchor points perceived knowledge and perceived skills are 1 3 Perceived knowledge is measuring the knowledge level in various topics taught by the notes/ game. Items fall in

PAGE 144

126 I can define data warehouse characteristics I can explain impact of events on budget various topics taught by the notes/ I can are based on the learn ing objectives of both conventional and the game teaching methods. In addition to the learning outcome constructs, the experiment will measure the perceived enjoyment of participants in both groups to determine motivation. The 7 point Likert scale points a re 1 Ext reme ly Neither agree nor disagree Ext reme ly An example of c omplete list of measurement items for these constructs is in the appendix. User demographic information is also measured by simple questions asking for gender, age group, income level, education level, gaming level, and previous data warehouse background. None of the questions can or will be used to identify participants. Full survey questions are in the appendix. Results The measurement model and the structural model will be evaluated . The measurement model will evaluate construct alpha and composite reliability), indicator reliability (loadings of measurement items), convergent validity (Average Variance Extracted AVE), and discriminant validity (Cross loadings). The structural model will evaluate collinearly (Variance Inflation Fa ctor VIF), path coefficients (Bootstrapping), total effect significance, and coefficients of determination (R 2 ). Thresholds will be used as suggested by previous research such as Fornell and

PAGE 145

127 Larcker (1981); Ringle, Wende, and Will, 2005; Cronbac h , 1951; Ha ir, Hult, Ringle, and Sarstedt, 2014; Gefen and Straub, 2005. Evaluation Summary Serious games evaluation is important because it provides evidence of the effectiveness of this type of games. Emerge2Maturity creates a dynamic environment to provide experie ntial learning experience to students who want to learn about data warehouse development. This chapter proposed a comprehensive evaluation framework that can be used to evaluate serious games in two stages. The design evaluation help developers to align th e components of the game with the proposed learning objectives. We used participants from instructor and students populations to validate this part of the framework and establish the content validity of Emerge2Maturity game. The result shows positive attit ude toward the game. Most of the responses indicate that the game actually addresses the learning objectives. The responses also show that the game adhere to the constructive c riticism and feedback that can help in advancing the development of the game. On the other hand, the effect evaluation help in assessing the effectiveness of the game by measuring the motivation and the learning outcomes. This chapter proposed the researc h methodology for the effect evaluation but, there was no data collection. The game need to incorporate the suggestions from the design evaluation before conducting this part of the research.

PAGE 146

128 CHAPTER VI CLOSING Data warehouse projects tend to fail in many ways. Examples are incomplete implementation, exceeding the budget, or not using the system by users. On reason to these projects to fail is the lack of user trainings. A quick review of curriculum in universities reveal that students learn about theoretical asp ects of data warehousing leaving the experience part until they graduate and join a company that goes through a data warehouse project. Several learning objectives are missed in these courses that might doubt the value of a data warehouse in organizations. Examples are explaining the intangible benefits of a data warehouse and understanding the learning effects. In this research, we developed a business strategy serious game named Emerge2Maturity or E2M. The name and the process of the game are inspired by the data warehouse maturity model. The foundation of the game is that companies should base the development on increasing their capabilities over distinct periods of time. The game also combines business strategy components like budget and resource constr aints. The alignment between the business strategy and the organizations capability is an important factor in data warehouse development. Several serious games have been develop for several years. However, evidence of rigorous evaluation is scarce in the l iterature. There is little research focusing on constructing evaluation frameworks for serious games. In response, we developed a comprehensive evaluation framework that combines evaluation of the design and evaluation of the effect of serious games. These two evaluation types are distinct but the literature typically does not distinguish between them. The design evaluation help s

PAGE 147

129 developers to align the game components with the proposed learning objectives. The effect evaluation provide s empirical evidence of the effectiveness of the game in motivating participants and delivering the learning outcomes. We validated the design evaluation part of the framework by using it to evaluate Emerge2Maturity. We collected quantitative and qualitative data from instruct ors and students who used the game and attitude toward the game indicating the potential benefits of using it in teaching about data warehouse development. The extended qu alitative feedback also furnish the road for continues development and improvements to the game design. We used the rapid development approach in the implementation of the game. This approach promotes continues evaluation during the development cycle. Cur rently, the game has reached a stable state where the mathematical functions are accurate and run without errors . It is a big leap in the development from an Excel workbook to a fully functional javascript application. However, there are three main limitat ions. The first is time. The project is part of a dissertation for a PhD degree and as such needed to be completed within the time frame to complete the degree . The second is the lack of funding. The third is the lack of expertise which can only be develop ed through experience . The suggested future developments require skills that need to be developed or acquired . Serious games can take decades to improve . F or example, the beer game was first introduced in the 19 80s as a board game (Sterman, 1989) and conti nue to evolve until recent years. This project h as several avenues for continued improvements. T wo main areas of improvement were identified: gamification and Strategy Assessment Model (SAM).

PAGE 148

130 Playing games is one of the daily routines for many people. En joyment is the main motivation to play games. The game elements, if combined with educational materials, can also bring enjoyment to the learning process and help accomplish learning outcomes. non (Deterding et al., 2011). According to Werbach (2015), there are 15 game elements that are responsible for creating the enjoyment in games. Future research will add suitable gamification elements to Emerge2Matruity. After adding gamific ation features, future research will need to evaluate Emerge2Maturity using a combination of survey and experiment. Most serious games are evaluated by engagement and learning outcomes ( Connolly et al. 2012) . To evaluate game engagement, the questionnaire method is commonly used. In contrast, experiment is used to measure items that are difficult to evaluate using surveys such as learning outcomes. Hainey et al. ( 2011) used a combination of pre test/ post test and experiment using the pre test/post test while the learning outcomes were assessed using the experiment. Pa rticipants were randomly assigned to control groups and treatment groups. Future development of Emerge2Maturity will develop the Strategy Assessment Model (SAM), a novel strategy model to decide the data warehouse development strategy. The current version of the Emerge2Maturity determines the development strategy by specifying the number of phases and constraints for each phase. SAM will allow players to determine the strategy and provide players the freedom of specifying constraints and number of phases. T his is a higher level version of the game that requires not only capability assessment but also the strategy assessment using factors identified in

PAGE 149

131 data warehouse research including information interdependence, task routineness, and level of sponsorship Th is research contributes to the body of knowledge and practice related to serious games and data warehouse education . The research follows the design science approach suggested by Peffers etl al. (2006). It describes a problem affecting the successfulness o f the data warehouse development and traces the problem back to educational material in universities. Then, it suggests a practical solution to explain and resolve the problem by providing a supporting tool to help students better understand several learni ng objectives. The research continues by designing, implementing, and evaluating the artifact (Emerge2Maturty). Researchers indicate that the IS discipline will benefit from more research in design science and developments of artifacts ( Benbasat and Zmud, 2003) . Emerge2Maturity game is the first business strategy game addressing complex issues related to data warehouse developme nt. It can be used by both students and professionals to gain insights regarding some concepts that are difficult to explain using conventional learning material. It provides an experiential learning to teach about intangible benefits of data sources and l eaning effects occurring in projects that deal with capability improvements over time. The research also proposes a comprehensive approach to evaluate serious games by building an evaluation framework. The framework combines two distinct parts of evaluatio ns, yet hardly addressed in literature. The design evaluation help developers to assess the alignment between the game functionality and the learning objectives. The effect evaluation part helps in measuring the effectiveness of the game in motivating part icipants and increasing their leering outcomes.

PAGE 150

132 REFERENCES Agarwal, R., and risis: Focusing on High Visibility and High Impact R esearch , MIS Quarte rly (29:3), pp. 381 398. A tutorial for the 11th International Conference on Artificial Intelligence Education . Amsterdam. ng Effectiveness in Digital Game International Journal of Serious Games (1:2), pp. 3 21. Alsqour Survey of Data Warehouse Architectures Preliminary Results Federated Confere nce on Computer Science and Information Systems , pp. 1121 1126. Simulation Game For Teaching Service Oriented Supply Chain Management: Does Information Sharing Help Managers With Service Capacity Decisions Produ ction and Operations Management (9: 1), pp. 40 55. Organizational Factors In Data Warehouse Architecture Selection Decision Support Systems (49:2), pp. 200 212. Arnab, S., Lim, T., Carvalho, M. B., Bellotti, F., De Freitas, S., Louchart, S., Suttie, N., British Journal of Educational Technology (46:2), pp. 391 411. Backlund, P., Engström, H., Johannesson, M., Lebr am , M., & Sjödén, B. 2008. "Designing For Self Efficacy In A Game Based Simulator An Experimental Study And Its Implications For Serious Games Design ". In Visualization , 2008 international conference , pp. 106 113. Backlund, P., Taylor, A.S.A., Engström, H. , Johannesson, M., Lebram, M., Slijper, A., Svensson, K., Poucette, J. and Sunnerhagen, K.S., 2011 . " Evaluation of usefulness of the Elinor console for h ome based stroke rehabilitation," In Games and Virtual Worlds for Serious Applications (VS GAMES), 2011 Third International Conference on , pp. 98 103 . IEEE. Journal of the Academy of Marketing Science (40:1), pp. 8 34. Ballou, D.P. and Tayi, G.K. 1999. "Enhancing Data Quality in Data Warehouse Communications of the ACM (42: 1), pp. 73 78.

PAGE 151

133 Employee Motivation Improvement a Railway Case Study Based on the Plac ebo in Proceedings of the 50th Hawaii International Conference on System Sciences , pp. 1278 1287. Baskerville, R., 1999. " Action R esearch for Information Systems," AMCIS 1999 Proceedings , p. 288. Becker, C., Prendinger, H., Ishizuka, M., and Wachs Affective Feedback Of The 3D Agent Max In A Competitive Cards Game In International Conference on Affective Computing and Intelligent Interaction , pp. 466 473. urity Models for IT Business & Information Systems Engineering (1:3), pp. 213 222. Ben International Journal of Product ion Economics (92:1), pp. 75 80. Ben zvi AMCIS 2009 Proceedings , p. 549. The Identity Crisis within the Is Discipline: Defining and Communic MIS Quarterly (27:2), pp. 183 194. Design Artifacts towards A Design Scandinavian Journal of Information Systems (12), pp. 15 27. Boute, R.N. and Lambrecht, M.R., 2009 9. In Digital Entertainment Games: A Systematic Review Comp uters in Human Behavior (28:3) , pp. 771 780. Boyle., T. A., and Strong, S. E. 2006. "Skill Requirements of ERP Graduates, " Journal of Information Systems Education (17: 4), pp. 403 412. Braden, R.A., 1992. "Formative Evaluation: A Revised Descriptive Theory and a Prescriptive Model , " Association for Educational Communications and Technology . McGraw Hill Education , 5th edition, ISBN 978 0077219888.

PAGE 152

134 nventory Models With Continuous, Stochastic Demands The Annals of Applied Probability (1:3), pp. 419 435. Bruin, T. De, Freeze, R., Kaulkarni The Main Phases of Developing A Maturity Assessment Model 16th Aust ralasian Conference on Information Systems . Computers & Education (87), pp. 396 422. Cardoso, E., Alcolea, J. J., Rieger, B., Schulze, S., Rivera, M., Leone, A., and Brighi, E. of the Maturity Level of BI Initiatives in European Higher Education In EUNIS 2013 Congress Proceedings (1:1). Carvalho, M. B., Bellotti, F., Berta, R., De Glor ia, A., Sedano, C. I., Hauge, J. B., Hu, J., Activity Theory Based Model for Serious Games Analysis Computers and Education (87), pp. 166 181. Cheng, C. H., and Su, C. Based Learning Sy stem for Improving Procedia Social and Behavioral Sciences (31:2011), pp. 669 675. Organizational Factors in Data Warehouse Architecture Selection Vivekanada Journ al of Research (24), pp. 24 32. Review of Business Intelligence and Its Maturity Models African Journal of Business Management (5:9), pp. 3424 3428. Clark, R. 2007. "Learning From Serious Games? Arguments , Evidence, and R esearch Suggestions ". Educational Technology . (47:3), p. p 56 59. Publications, 2000. Con nolly, T. M., Stansfield, M. H., & Hainey, T. 2007a. "An Application of Games Based Learning within Software Engineering ". British Journal of Educational Technology (38: 3), pp. 416 428. Connolly, T., Stansfield, M. and Hainey, T., 2009. "Towards The Develo pment of a Games Based Learning Evaluation Framework ". Games based learning advancements for multisensory human computer interfaces: Techniques and effective practices , Hershey PA: IGI Global.

PAGE 153

135 Connolly, T. M., Boyle, E. A., Macarthur, E., Hainey, T., and B Systematic Literature Review of Empirical Evidence o n Computer Games a nd Serious Games Computers & Education (59:2) , pp. 661 686. investments in dat Information Systems Journal (12:4), pp. 321 338. Creswell, J. W. 2009. " Research Design: Qualitative , Quantitative, And Mixed Methods Approaches , " (Vol. 3rd). Sage. Creswel l, J. W., & Plano Clark, V. L. 2007 . " Designing and Conducting Mixed Methods Research," Thousand Oaks, CA: Sage. Cronbach, L. J. 1951. "Coefficient Alpha and the Internal Structure of Tests ". Psychometrika (16: 3), pp. 297 334. observed value 336. Crotty, M. 1998 . " The Foundations Of Social Research: Meaning and Perspective in The Research Process," London: Sage. Csíkszentmihályi, M. 1990. "Flow : The Psychology of Optimal Experience, " N ew York: Harper and Row. Davis, F.D., Bagozzi, R.P. and Warshaw, P.R., 1992. "Extrinsic and Intrinsic Motivation to Use Computers in the Workplace, " Journal of applied social psychology (22: 14), pp.1111 1132. End Users: An Experimental Investigation of the Roles of the Computer Interface and Training Methods MIS quarterly , pp. 61 85. Production Order Quantity Model with Stochastic Demand for A Chocolate Milk Manufacturer International Journal of Production Economics (49:2), pp. 145 156. De Freitas, S. and Jarvis, S., 2006. "A Framework For Developing Serious Games To Meet Learner Needs, ". Deci, E. L., & Ryan, R. M. 1985. "Intrinsic Motivation and Self Determ ination in Human Behavior, " New York: Plenum. Deci, E.L., and Ryan, R. 2002. "Overview Of Self Determination Theory : An Organi smic Dialectical Perspective," I n Handbook of Self Determination Research , University of Rochester Press, pp. 3 33.

PAGE 154

136 DeLone, W. H., Systems Success: The Quest For Information systems research (3: 1), pp. 60 95. The Effect Of Cumulative Experience And Learning A Field Study Of Help Desk Support AMCIS Proceedings (1:341) , pp. 1775 1782. Journal of the Association for Info rm ation Systems (4:7), pp. 360 376. Deterding, S., Dixon, D., Khaled, R. and Nacke, L., 2011, "September . From Game Design Elements to Gamefulness: Defining Gamification ". In Proceedings of the 15th international academic MindTrek conference : Envisio n ing future media environments, pp. 9 15. Dinter AMCIS Proceedings . Dondi, C., & Moretti, M., 2007. "A Methodological Proposal for Learning Games Selection and Quality Assessment," British Journal of Educational Technology ( 38 : 3) , pp. 502 512. . Journal of Artificial Societies and Social Simulation (7:4), p.2. Freitas, S. De, Rebolledo mendez, G., Liarokapis, F., Magoulas, G., and Poulovassilis, A. Dimensional Framework for Designing and Evaluating Immersive Learning Experiences in a Virtual Wo British Journal of Educational Technology (41:1), pp. 69 85. Fornell, C., & Larcker, D. 1981 "Evaluating S tructural Equation Models with Unobservable Variables and Measurement Error," Journal of Marketing Research (18: 1), pp. 39 50. Freeze, R., Kulka and Complementari AMCIS Proceedings . Gagné, M. 2003. "The Role of Autonomy Support and Autonomy Orientation in Prosocial Behavior Engagement," Motivation and E motion ( 27 ) , 199 223. (2nd ed.). Oxford, England: Holt, Rinehart & Winston. viii 407 .

PAGE 155

137 Gefen, D., & Straub, D., 2005. "A Practical Guide to Factorial Validity Using PLS Graph : Tutorial and Annotated Examp le," Communications of the Association for Information Systems (16: 1), pp. 91 109. Gibbs, G. R. 2007 . " Analyzing qualitative data ," In U. Flick (Ed.) , The Sage qualitative research kit. London: Sage. Gibson, M., Arnott, D., Jageilska, I. and Melbourne, A., Proceedings of the 2004 IFIP International Conference on Decision Support Systems (DSS2004): Decision Support in an Uncertain and Complex World , pp. 295 305. Prato, Italy. 2002 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems Communication of the Association for Information Systems (11 ) , . Game Based Le arning Indicate Submission Type Twenty Second Americas Conference on Information Systems , San Diego, pp. 1 10. Gregg, D.G., Kulkarni, U.R., Vinz , A.S. 2001. "Understanding the Philosophical Underpinnings of Software Engineering Research in Information Systems." Information Systems Frontiers ( 3 ) , pp. 169 183 . Design Theory in Information Systems Australian Journal of Informati on Systems (December), pp. 14 22. Gulledge, T. Industrial Management & Data Systems (106:1), pp. 5 20. The Journal of t he International Digital Media and Arts Association (3:1), pp. 1 19, 93 105. Anarchy And How To Survive It: God Save the Enterprise Systems Journal (15:4) ,. Hainey, T., Connolly, T. M., Stansfield, M., and Boyle, E. of A Game To Teach Requirements Collection And Analysis In Software Engineering At Tertiary Education Level Computers and Education (56:1), pp. 21 35. Hair Jr., J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. 2014. "A Primer on Partial Least Squares Structural Equation Modeling ( PLS SEM )," Los Angeles, California: SAGE Publications.

PAGE 156

138 Hamari, J., 2013. "Transforming Homo Economicus into Homo Ludens: A Field Experiment on Gamification in a Utilitarian Peer To Peer Trading Service ". Electronic commerc e research and applications (12: 4), pp.236 245. Chichester, Wiley. Harpel, R. L. 1978 . " Evaluating from a Management Perspective," New Directions for Student Services (1), pp. 19 33. Courses: Learning Experiences and Outcomes," British journal of educational technology (44:3), pp. 372 385. Design Scie nce in Information Systems Research MIS Quarterly (28:1), pp. 75 105. Is Design Theory Possible Journal of Information Technology Theory and Application (6:2), pp. 73 83. Hopkins, J. L., and Foster, S. 201 lation: a Unique Learning E AMCIS Proceedings , . "Overview of B usine ss Intelligence Maturity Models," Management: Journal of Contemporary Management Issues (15: 1), pp. 47 67. Information Systems Developm ent , pp. 15 27. Inmon, W.H., 1996. "Building the Data Warehouse," New York: Wiley Computer Publishing, Second Edition . Irani, Z. and Love, P. E. D., 2001. "The Propagation of Technology Management Taxonomies for Evaluating Investments in Information System s." Journal of Management Information Systems (17:3), pp. 161 177. Iten, N., & Pet ko, D. 2016 . " Learning With Serious Games: Is Fun Playing The Game A Predictor of Learning Success ? ," British Journal of Educational Technology (47: 1), pp. 151 163. Jacobs, F Production and Operations Management (9:1), pp 31 39. Jarke, M., Jeusfeld, M., Quix, C., & Vassiliadis, P. 1999. "Architecture And Quality In Data Warehouses: An Extended Repository Approac h, " Information Systems (24: 3), pp. 229 253.

PAGE 157

139 Kalfus, O., Ronen, B., & Spiegler, I. 2004. "A Selective Data Retention Approach In Massive Databases," Omega ( 32 ) , pp. 87 95. Kankanhalli, A., Taher, M., Cavusoglu, H., and Kim, S. H. 2012. "Gamification: A New Paradigm for Online User Engagement," Thirty Third International Conference on Information Systems : Orlando, FL. Kapp, K.M. 2012 "The Gamification of Learning and Instruction: Game based Methods and Strateg ies for Training and Education," John Wiley & So ns, New York. Kiili, K., Perttula, A., Lindstedt, A., Arnab, S. and Suominen, M., 2014. " Flow experience as a quality measure in evaluating physically activat ing collaborative serious games," International Journal of Serious Games (1:3) , pp. 35 49 Kim, B., Cognitive Strategies in Game Computers and Education (52 :4), pp. 800 810. Kim, Y., Hsu, J. & Stern, M. 2006. "An Update On the IS / IT Skills Gap, " Journal of In formation Systems Education (17:4), pp. 396 402 Data Warehouse Lifecycle Toolkit: Expert Methods for Designing, Developing, and Deploying Data Warehouses, . Kimball, R. and Ross , M. 2013. The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, Wiley, July, ISBN 13: 978 1118530801. Korteling, J.E., Helsdingen, A.S., Sluimer, R.R., Van Emmerik, M.L. and Kappé, B., 2011. " Transfer Of Gaming: Transfe r Of Training in Serious Gaming," Soesterberg: TNO. Kuznets, Simon. 1965. "Economic Growth and Structure: Selected Essays, " W. W. Norton, New York, pp. 213 216 Inte r national Journal of Compute r Games Technology (2014) , . Intelligence Maturity Models: An itAIS 2010 , pp. 1 12. Maturity: Development and Evaluati 2011 44th Hawaii International Conference on System Sciences Ieee , pp. 1 10.

PAGE 158

140 Lainema, T., 2009. "Perspective Making: Constructivism As A Meaning Making Structure For Simulation Gaming," Simulation & Gaming (40: 1), pp. 48 67. Lai nema, T., and Makkonen, P. 2003. "Applying Constructivist Approach To Educational Business Games: Case REALGAME ," Simulation & Gaming: An Interdisciplinary Journal ( 34 : 1 ) , pp. 131 149. Factors To Address for Success onal (14: 5) . Sloan Management Rev (38:3), pp. 93 102. Lee, S., Koh, S., Yen, D., and Tang, H L. 2002. "Perception Gaps between IS Academics and IS Practitioners: An Explor atory Study," Information and Management ( 40 ) , pp. 51 61. Leger, P. M. 2006. "Using a Simulation Game Approach To Teach Enterprise Resource Planning Concepts, " Journal of Information Systems Education (17:4), pp. 441 447. Li, M. C., & Tsai, C. C. 2013. "Ga me Based Learning in Science Education: A Review Of Relevant Research," Journal of Science Education and Technology ( 226), pp. 877 898. of Data Warehousing, 8(1), 48 5 4. Mackrell, D. 2009. "The work readiness of Master of Information Systems International students at an Austr alian University: A pilot study, " Issues in Informing Science and Information Technology ( 6 ) , pp. 179 191. sign and Natural Science Research On Information Technology Decision Support Systems (15:4), pp. 251 266. Support Emergent Knowledge Pro MIS Quarterly (26:3) , pp . 179 212 Business & Information Systems Engineering (4:4), pp. 193 207. Practitioners Pacific Asia Journal of the Association for Information Systems (1:3),. Mayer, I., Bekebrede, G., Harteveld, C., Warmelink, H., Zhou, Q., Van Ruijven, T., Lo, Research And Evaluation Of Serious

PAGE 159

141 Games: Toward A Comprehensive Methodology British Journal of Educational Technology (45:3), pp. 502 527. Mayer, I., Warmelink, H., and Bekebrede, G. 2013. "Learning in a Game Based Virtual Environment: A Comparative Evaluation in Higher Educa tion," European Journal of Engineering Education (38:1), pp. 85 106. https://blog.rjmetrics.com/2014/12/04/10 common mistakes when building a data warehouse. Michael, D. & Chen, S. 2006. "Serious Games: Games That Educate, Train, and Inform," Boston, MA. Thomson Course Technology. Inventory Control Decisions into A Strategic Distribution Network Design Model with Stochastic Demand Transportation Research Part E: Logistics and Transportation Review (40:3), pp. 183 207 Effectiveness of Enterprise Resource Planning Education on Busines s Process Co ICIS , pp. 1 10. AMCIS 2016 Proceedings , pp. 1 9. School Field (7:2), pp . 133 144. The Computer Resource: A Stage Hypothesis Communications of the ACM (16:7). Ogle, G. J. 2002. "Towards a Formative Evaluation Tool. PhD Thesis submitted to Virginia Polytechnic Institute and State University , ". Olsen , W., 2004. "Triangulation In Social Research: Qualitative And Quantitative Methods Can Really Be Mixed," Developments in sociology ( 20 ) , pp.103 118. Research in IS Educa tion: A Multi Association for Information Systems (42:1), p.5. Österle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar, H., Loos, P., Mertens, P., Oberweis, A. and Sinz, E.J., 2011. "Memorandum on Desig n Oriented Information Systems Research," European Journal of Information Systems (20:1), pp. 7 10.

PAGE 160

142 ns in Self Advances in Motivation and Achievement . Greenwich, CT: JAI Press. pp . 221 251. Paulk, M. C Maturity Model, Version 1.1 IEEE Software (10:4), pp. 18 27. Peffers, K., Tuunanen, T., Gengler, C.E., Rossi, M., Hui, W., Virtanen, V. and Bragge , J., 2006. "The Design Science Research Process: A Model for Producing and Presenting Information Systems Research," In Proceedings of the first international conference on design science research in information systems and technology , pp. 83 106 irical Study of Learning Curve Experience Relationship in Software Engineering and Project Management An Empirical Stu dy of Learning Curve AMCIS 2004 Proceedings , . Peng, W., Game Features as Motivational Determinants: An Experimental Study of a Self Determination Theory Guided Media Psychology (15), pp. 175 196. Peng, W., Pfeiffer, K. A., Win n, B., and Lin, J. 2015. "A Pilot Randomized, Controlled Trial of an Active Video Game Physical Activity Intervention," Health Psychology (34), pp. 1229 1239. : Play British Journal of Educational Technology (38:3), pp. 387 393. Power, D.J. The On Line Executive Journal for Data Intensive Decision Support (2: 5). Prensky, M. 2007 . "Digital Game Based Learning, " Paragon House, U.S.A. Przybylski, A. K., Ryan, R. M., & Rigby, C. S. 2009. "The motivating role of violence in video games , " Personality and Social Psychology Bulletin ( 35 ) , pp. 243 259. Putz, L. M., and Based Research Agenda for AMCIS 2015 Proceedings (1), pp. 1 13. iness ECIS , p. 95 . Ralph, P., and The Computer Games Journal (4:1 2), Springer New York, pp. 81 100.

PAGE 161

143 Empirical Investigation of the Key Determinants of Data Warehouse Adopti on Decision Support Systems (44:4), pp. 817 841. Rao, L., and Osei Bryson, K. Approach for Incorporating Quality Based Cost Benefit Analysis in Data Warehouse Design Information Systems Frontiers (10:3), pp. 361 373. Reeve, J., Jang, H., C arrell, D., Jeon, S., & Barch, J. 2004. "Enhancing Students' Engagement by Increasing Teachers Autonomy Support," Motivation and Emotion (28: 2), pp. 147 169. Immersyv e Inc . Ringle, C. M., Wende, S., & Will , S. 2005. "SmartPLS (M3) Beta," Hamburg, http://www.smartpls.com . Brain Training in 22 Primary Classrooms: A Randomised Controlled Study, " A summary for the BBC . Scotland: Learning and Teaching Scotland). Theory Proceedings of the 3rd European Con ference on Game Based Learning , pp. 332 344. Russ Capability Maturi AMCIS Proceedings , . Games: A Self Determinat Motivation and Emotion (30:4), pp. 347 363. Saari, L. M., Johnson, T. R., McLaughlin, S. D., & Zimmerle, D. M. 1988. "A Survey of Management Training and Education Practices in US Companies ". Personnel Psychology ( 41 ) , pp. 731 743. Sa Game and Decision System Dynamics Conference. Boston, MA. Santhanam, R., Sasidharan, S. and Webster, J., 2008. " Using Self Regulatory Learning To Enhance E Learning Based Information Technology Training," Information Systems Research (19: 1), pp. 26 47.

PAGE 162

144 Systems With Stochastic Demand And Supply: Properties and Ap proximations European Journal of Operational Research (206:2), pp. 313 328 . Schramm, W., Lyle, J., & Parker, E. B. 1961. "Television in the Lives of Our Children, " Palo Alto, CA: Stanford University Press. Schumann, P. L., Anderson, P. H., Thomas, S., Sc Developments in Business Simulation and Experiential Learning (28), pp. 215 220. Schunk, D.H., 1995. "Self efficacy and education and instruction". In Self effi cacy, adaptation, and adjustment , pp. 281 303 . Springer, Boston, MA. Sen, A., Ramamurthy, K., and Sinha, IEEE Transactions on Software Engineering (38:2), pp. 336 353. Sen, A., Sinha, A. P., and R IEEE Transactions on Engineering Management (53:3), pp. 440 455. a Data Warehouse Architect . Shin, B. 2003. " An exploratory investigation of system succ ess factors in data warehousing," Journal of the Association for Information Systems (4: 1), p.6. UK: Oxfor d University Press Smith, R. D. 2009. "Military Simulation & Serious Games: Where We C ame From and Where We Are Going, " Modelbenders LLC. Prentice Hall, Inc., Englewood Cli ffs, NJ, 339. the key issues, LE 19 4691, Gartner Group S reelakshmi, R., McLain, M. L., Jayakrishnan, R., Rajeshwaran, A., Rao, B., and Bijlani, 6th International Conference on Computing, Communications and Networking Technologies ,.

PAGE 163

145 Str aub, D., Boudreau, M. C., and Gefen, D. 2004 "Validation Guidelines for Is Positivist Research," The Communications of the Association for Information Systems, (13:1), pp. 63. E learning (73:10), p. 28. Sw artout, W. & van Lent, M. 2003 . "Making a Game of System Design," Communications of the ACM (46: 7), pp. 32 39 . Tamborini, R., Bowman, N. D., Eden, A., Grizzard, M., & Organ, A. 2010. "Defining Media Enjoyment as the S atisfacti on of Intrinsic Needs," Journal of Communication ( 60 ) , pp. 758 402. Tao, Y.H., Cheng, C.J. and Sun, S.Y., 2009. "What Influences College Students to Continue Using Business Simulation Games? The Taiwan Experience ". Computers & Education (53: 3), pp. 929 939 . Justification Methods for Computer Integrated Manufacturing Systems . New York, New York, Elsevier Science Publishers, pp. 234 241. Thavikulwat, P. and Pillutla, S., 2 010. "A Constructivist Approach to Designing Business Simulations for Strategic Management," Simulation & Gaming ( 41 : 2), pp. 208 230. I nformation Systems Research (3:1) , pp. 36 59. Information Systems Management (18:3), pp. 42 50. Benefits of Data Warehousing: Why Some Organizations Realize Exceptional Payoffs Information and Management (39:6), pp. 491 502. Webster, J. and Ahuja, J.S., 2006. "Enhancing the Design of Web Navigation Systems: The Influence of User Disorientation On Engagement And Performance," MIS Quarterly , pp. 661 678. Werbach, K. and Hunter, D., 2012. "For The Win: How Game Thinking Can Revolutionize Your Business," Wharton Digital Press. on S Electronic Commerce Research and Applications (9:6), Elsevier B.V. , pp. 476 487.

PAGE 164

146 Wilson, D. W., Jenkins, J., Twyman, N., Jensen, M., Valacich, J., Dunbar, N., Wilson, S., Miller, C., Adame, B., Lee, Y. H., Burgo Proceedings of the Annual Hawaii International Conference on System Sciences (2016 March), pp. 638 647. Method For Demand drive n Information Requirements Analysis in Data Warehousing Project Proceedings of the 36th Hawaii International Conference on System Sciences , p. 9. Wixom, B.H. and Watson, H.J. 2001. " An empirical investigation of the factors aff ecting data warehousing suc cess," MIS quarterly , pp.17 41. International Journal of Business Intelligence Research (1:1), pp. 13 28 . Meta Analysis of the Cognitive and Motiva tional Effects of Serious Games Journal of Educational Psychology (105:2), pp. 249 265. Wright, T. 1936. "Factors Affecting the Costs of Airplanes," Journal of Aeronautical Science (3), pp. 122 128. Wrzesien, M., López , D.P. and Raya, M.A., 2010. " Learning ecology issues of the Mediterranean Sea in a Virtual aquatic World pilot study ," Journal of Cyber Therapy & rehabilitation (3:3), p. 255. Zichermann, G., and Cunningham, C. 2011. "Gamification by Design: Implementing Game Mech anics in Web and Mobile Apps," O'Reilly Media: Sebastopol, CA. Yin, R. K. 2003 . " Case study research: Design and methods ," (2nd ed.). Thousand Oaks, CA: Sage.

PAGE 165

147 APPENDIX A GAME NARRATIVES Table 24 : Game Narratives for th e Welcome Page PTPageName PTPageTitle PTPosition PTText Welcome Abstract 1 Emerge2Maturity helps players understand the process of data warehouse development on both strategy and capability sides. The game simulates the maturity model approach. This appr proper data warehouse architecture. A player has a role in an organization to implement the business strategy by making decisions about data extraction, t ransformation, and integration. profit within limited budget and resource constraints. Emerge 2 Maturity depicts the complexity of the maturity process and related capability assessment for data warehouse d evelopment. However, Emerge2Maturity supports educational purposes not actual prediction. capabilities as a data warehouse matures over a number of development phases. Learning Objectives 2 Students as well as professionals struggle to understand development of data warehouses in organizations over time. Data warehouse development is a complex process involving several related factors and extended time periods to reach a stable solution. Learners face challenges to observe changes and determine key success factors in data warehouse projects. Learners have difficulty understanding project relationships involving benefits and risks. Benefits of data warehouse deployment are often intangible especially during initial periods of usage. Benefits become tangible and increase as organizational units increase usage. In contrast, costs are tangible and high during data warehouse development especially with uncertain levels of data quality. Risk declines as be nefits increase

PAGE 166

148 during use of a data warehouse over time. Learners need to balance benefits and risks as organizations acquire capabilities to support an The design of Emerge2Maturity addresses these difficulti es. The game decomposes complexity of data warehouse development into a sequence of standard steps. To help focus learners on key factors , the game provides common factors across organizations. Learners are not distracted by other elements related to spe cific situations, remaining focused on the important aspects of data warehouse development. The game combines aspects of strategy and capability to help learners understand the relationship between them. The game simulates the development process to show trends, costs, and benefits with increased profits and decreased costs over time. Simulation provides a real like situation where learners can observe results of their decisions before implementing them. The simulation uses models to quantify costs and b enefits related to choices. These models depict the relationship between the costs of acquiring capabilities to develop a data warehouse and benefits for using a data warehouse . Related Courses 3 Emerge2Maturity game helps students who are taking course s related to data warehouse developments. Table 25 : Game Narratives for the Instructions Page PTPageName PTPageTitle PTPosition PTText Instructions Instructions 1 Playing the game is simple. You need to read the game concept sect ion to gain basic knowledge about data warehouse development. Then, click login and register. Provide basic information for registration. Before starting game play, you select a game with a specified level of difficulty and strategy. To maximize your lea rning outcome, you should play all games beginning with a beginner game and progressing to more advanced games.

PAGE 167

149 Educational games facilitate the learning process by providing the optimal decision values for the previous phase at the beg in ning of new phase s allowing players to catch up. Competitive games facilitate co m petence by allowing results of players decisions to accumulate until the end of the game. For each game, you allocate resources over a specified number of decision making periods to maximize profit. Your decisions of extraction, transformation, and integration implement a strategy for the development of a data warehouse. The strategy contains constraints on the number of phases, budget for each phase, and resource levels for a phase. You wil l have limited resources and budgets through the phases. At the end of each phase, you will receive a summary of your performance and a score. You need to finish all phases to complete a game. Some phases may involve a twist like introduction of events. Events are temporary situations in which you may need to adjust your decision approach. At the end of the game, you will receive a summary of your performance, a relative score, and a ranking among other players. Please do not click the back button in you r browser. The game is designed to flow through the provided buttons. We recommend you leave the instructions and the game concept pages open in other tabs so you can refer to them while you are playing the game. Table 26 : Game Narratives for the Game Concepts Page PTPageName PTPageTitle PTPosition PTText Game Concepts Data Sources 1 Benefits from data warehouse come from queries and reports that are generated using data sources. Data sources can take many forms from operationa l databases, files, web contents and external data sources. These sources have potential value for decision making. To manage complexity from a large number of data

PAGE 168

150 sources , data sources are grouped into categories. Categories facilitate determination of costs and benefits of individual data sources as all data sources in a category share features. The game classifies data sources using features related to costs and benefits. Features support grouping on model component for fixed costs, variable costs, p roduction, benefits, and risk. The features are Technology, Complexity, and Size. Each feature has a three point scale, low, medium, and high. Features 2 Technology ranges from legacy systems to modern systems. Technology level can be assessed by featu res in programming language, database management system, operating system, and hardware platform. Complexity involves difficulty to transform diverse data for decision making. More complex data will need extensive time and effort to analyze. Size involve s processing effort for data such as the number of rows. Larger data will require additional storage and maintenance. Decisions 3 Extraction: The first step toward benefiting from these sources is extracting these data and storing them into the data ware house. There are two common methods for data extractions. Method one suggests that organizations should always extract all their data sources and makes them available in the data warehouse. The other method suggests analyzing the data sources and select what is more relevant and appropriate for decision makers. Although neither of these approaches is necessarily true, data warehouse designers must be aware of the cost and benefit trade off when it comes to data sources selection. Transformation: Transfo rmation enhances the quality of the data inaccuracy , completeness, and timelines. Transformation also resolves inconsistencies, applying business rules, and summarization. The quality of data is reflected on the quality of the decision making process. The re are three main quality dimensions with sub categories : design and administration dimension, data usage dimension, and data dimension. Design and administration dimension includes correctness, completeness, minimality, traceability, interoperability,

PAGE 169

151 a nd metadata evaluation. Data usage dimension includes responsiveness, timeliness, interpretability, security, and availability. Data dimension includes accuracy, completeness, consistency, credibility, and data interpretability. Integration: Integration of data means that relevant data are processed under the same rules and stored once forming a single point of truth. Integration determines architecture with high levels of integration associated with an enterprise data warehouse covering an entire organi zation and lower levels of integration associated with independent data marts associated with different parts of an organization. Data integration can uncover hidden relationships not possible to capture without combining data across parts of an organiza tion. Higher integration level can also reduce the total cost of ownership in managing data. Maturity 4 As an organization acquires capabilities, it becomes more efficient with decreasing costs for deploying resources and effective with increasing benef its. The early documented used of learning curve was in aircraft assembly. While workers in an aircraft assembly facility gained experience, the time and effort to build airplanes decreased and performance increased. Learning curves have been used to expl ain the relationship between relative efforts and cost reduction and increased performance in several disciplines like software development and help desk support. Maturity has three main components: Strategy, Capability, and Architecture. The strategy is defined by: information interdependence: describes how departments within an organization rely on data from each other task routineness: very structured data that are automated need less human interference level of sponsorship: individuals or groups who have control over the data. The capability is defined by: Data size: as the capability of an organization increases, organizations tend to process and store larger amounts of data. Data quality: the quality of the data becomes a big issue in larger data wa rehouses. Thus, data quality feature

PAGE 170

152 correlates with the data size feature. Data transformation process e nsures that the data has the right quality level. Integration level: Higher levels of capability require organizations to have higher alignments betwee n their DW architectures and their business strategies. The strategy determines the integration level of the data in the data warehouse. There are several data warehouse architectures. Maturity model suggests that the development should emerge from low le vel architecture to a higher level architecture for a data warehouse project to succeed. Examples of data warehouse architectures are: Independent data mart: independent units of data from operational databases. Data Mart Bus: independent data marts based on the business process Enterprise: a single data unit to feed the organization Events 5 Events are occurrences of actions with long term consequences initiated externally or internally by an organization. An internal event is an occurrence of actions w ithin an organization such as a merger or divestment. An external event is an occurrence of actions that organizations have no control such as a recession, regulation, or litigation. An organization reacts to events by adjusting their strategy and/or capa bilities. Emerge2Maturity uses a small set of events with a probability of occurrence. If an event occurs, Emerge2Maturity randomly adjusts budget constraints. Table 27 : Game Narratives for the Game Preparation Page PTPageName P TPageTitle PTPosition PTText Game Preparation Game Preparation 1 Hello %PlNickName% and welcome to our implementation team. The CIO has decided that our company may benefit from building a data warehouse. Since the organization lacks experience with data warehouse development, you have been hired to help manage the development process. The planning team has decided on the development

PAGE 171

153 strategy to ensure alignment with our business strategy. The project is divided into %GCNumPhases% phases and the allocated budget for the entire project is $%GCGameBudget%. The available data sources are grouped into several categories as shown in the table below. Features of these categories have impact on cost and benefits of data sources. As the data warehouse d evelopment lead, you should implement this strategy by making decisions about extraction, transformation, and integration levels. You will receive additional information at the begging of each phase. If you are ready for your mission, click Next to continue. Table 28 : Game Narratives for the Phase Preparation Page PTPageName PTPageTitle PTPosition PTText Phase Preparation Phase Preparation 1 Okay %PlNickName%, this is phase number %GPCPhaseNumber%. As you make resource de cisions, you will find the limits for extraction, transformation, and integration for this phase for each category. These limits were determined based on the data warehouse. Because the economy is predi cted to be in %Economy% mode, the budget is affected by %change% percent. You have a budget of %budget% to spend on this phase. Depending on your efforts in previous phases (after phase 1), the factor that affects the benefit from data sources has changed by %benefit% percent and the factor that affects the cost of operation has changed by %variablecost% percent. Click Next to continue.

PAGE 172

154 Table 29 : Game Narratives for the Phase Simulation Page Extrac tion PTPageName PTPageTitle PTPosition PTText Phase Simulation Extraction Phase Simulation for Extraction 1 You need to determine the extraction level for each data source category. Enter the number of data sources from which you want to extract data. Th en, select Check Feasibility to ensure your decisions are within the constraints. If your allocation is feasible, select Simulate to see the impact of your choice on the profitability of the organization. The simulation is only a prediction. The predicted results of your decisions are shown in the graphs. You may revise your decision up to %GCMaxSim% times. After reaching the maximum attempts, you must select a choice from the simulation attempts table and then select Commit t o finalize your decision. After you commit your decision, you will see the expected profit and the optimal cost and benefit values from the optimal decision. Click Next to continue. Table 30 : Game Narratives for the Phase Simulation Page Transformation PTPageName PTPageTitle PTPosition PTText Phase Simulation Transformation Phase Simulation for Transformation 1 You need to decide on the transformation level for each category. Enter the percentage of data transformation you want to apply on extracted data for each category. Then, select on Check Feasibility to ensure that your decisions are within the constraints. If your allocation is feasible, select Simulate to see the impact of your choice on the profit ability of the organization. You may revise your decision up to %GCMaxSim% times. After reaching the maximum attempts, you need to select a choice from the simulation attempts table and then select Commit to finalize your decision. After yo u commit your decision, you will see the

PAGE 173

155 expected profit and the optimal cost and benefit values from the optimal decision. Click Next to continue. Table 31 : Game Narratives for the Phase Simulation Page Integration PTPag eName PTPageTitle PTPosition PTText Phase Simulation Integration Phase Simulation for Integration 1 You need to decide on the integration level for each category. Enter the percentage of data integration you want to apply on transformed data for each cat egory. Then, select on Check Feasibility to ensure that your decisions are within the constraints. If your allocation is feasible, select Simulate to see the impact of your choice on the profitability of the organization. You may revise your decision up to %GCMaxSim% times. After reaching the maximum attempts, you need to select a choice from the simulation attempts table and then select Commit to finalize your decision. After you commit your decision, you will see the expecte d profit and the optimal cost and benefit values from the optimal decision. Click Next to continue. Table 32 : Game Narratives for the Phase Summary Page PTPageName PTPageTitle PTPosition PTText Phase Summary Phase Summary 1 Here is a phase summery to show how you did in this phase. You can see your committed decisions and the results from your committed decisions and the results from the optimal decisions. We gained some experience implementing this phase in our data ware house development. Also, our employees and management are noticing the value of this data warehouse project. As you start a new phase (if any), notice how your effort in this phase is going to affect costs and benefits for the next phase. Click Next to continue.

PAGE 174

156 Table 33 : Game Narratives for the Game Summary Page PTPageName PTPageTitle PTPosition PTText Game Summary Game Summary 1 Good work %NickName%. Here is a summary of this project. Review the table below to see how m uch profit we made implementing the data warehouse project using your action plan. Compare your results with the optimal results. Click Next to continue to see your score in this game. Table 34 : Game Narratives for the Gam e Score Page PTPageName PTPageTitle PTPosition PTText Game Score Game Score and Leader Board 1 Good job %PlNickName%. You have completed the project. Check out your final score and rank. The rank shows the top 10 scores from all players. Play again to im prove your score and climb up the rank. See you next time. Click Play Again to go to the game selection page or End to go to the welcome page .

PAGE 175

157 APPENDIX B INSTR UCTOR' S SURVEY

PAGE 176

158

PAGE 177

159

PAGE 178

160

PAGE 179

161

PAGE 180

162

PAGE 181

163

PAGE 182

164

PAGE 183

165

PAGE 184

166

PAGE 185

167

PAGE 186

168

PAGE 187

169

PAGE 188

170 APPENDIX C

PAGE 189

171

PAGE 190

172

PAGE 191

173

PAGE 192

174

PAGE 193

175

PAGE 194

176

PAGE 195

177

PAGE 196

17 8

PAGE 197

179

PAGE 198

180

PAGE 199

181

PAGE 200

182 APPENDIX D CONSTRUCTS AND MEASU REMENT ITEMS Table 35 : Constructs and Items for Game Mechanics Construct Adopted From Items Autonomy Ryan, Rigby, and Przybylski ( 2006 ) 1. I did things in the game becaus e they interested me. 2. I felt controlled and pressured to be a certain way. 3. I experienced a lot of freedom in the game. 4. The game provides me with interesting options and choices. Competence Ryan, Rigby, and Przybylski ( 2006 ) 1. I felt very capable and effective. 2. The game kept me on my toes but did not overwhelm me. 3. I felt competent at the game. 4. I felt capable and effective while playing. Table 36 : Constructs and Items for Motivation Construct Adopted From Items Enjoyment Davis, Bagozzi, and Warshaw (1992) 1. Interacting with the game is enjoyable. 2. Interacting with the game is exciting. 3. Interacting with the game is pleasant. 4. Interacting with the game is interesting. 5. Interacting with the game is fun. E ngagement Webster, Jane, and Ahuja (2006) 1. The game held my attention. 2. The game kept me totally absorbed. 3. The game excited my curiosity. 4. The game aroused my imagination. 5. The game was engaging.

PAGE 201

183 Table 37 : Game Effic acy Construct and Items Construct Adopted From Items Serious game self efficacy Santhanam, Radhika , Sharath, and Webster, 2008) 1. I feel confident using E2M to learn about and apply new concepts. 2. Using E2M is an efficient way for me to learn new thin gs. 3. I could apply new concepts that I learned from E2M. E2M. 5. It would be easy for me to become skillful at tasks learned from E2M. 6. I would be comfortable using E2M. 7. I could successfully use E2M . Table 38 : Learning Outcomes Constructs and Items Construct Adopted From Items DW Lecture (knowledge) Learning Objectives 1. I can describe what data warehouse is 2. I can define data warehouse characteristics 3. I can explain d ata warehouse architectures 4. I can describe how an organization matures in it usage of a data warehouse 5. I can explain learning effects on costs of developing a data warehouse 6. I can explain learning effects on benefits of using a data warehouse E2M (knowledge) Learning Objectives 1. I can list features affecting costs and benefits of using data sources 2. I can explain impact of events on budget 3. I can explain impact of acquiring capabilities on costs and benefits E2M (Skills) Learning Objectives 1. I can apply features of a collection of data sources to choosing capability levels 2. I can choose capability levels within constraints 3. I can analyze capability choice among alternatives to improve profit objective

PAGE 202

184 APPENDIX E CODED FEEDBACK Table 39 : Coded Text for Gne1 Item Codes Text Compliments Learning objectives are presented with good quality. Objectives are clearly described . The welcome page is extremely detailed. Content provided is very informative. Suggestions Present examples of previous data warehouse development. Reduce the amount of text or use bullet points Use some graphs to explain the objectives and the game process More explanations on cost and learning curves with examples Explain how things appear in reality Vs the game Use colorful texts to distinguish important parts Make a video presentation Separate educational content in a link outside the game Use animation Increase the size of text box. No scroll bar Criticism Difficulties understanding the objectives. Creates doubts for players who still need more information. Too much details make it hard to read and follow Text presentation is not appealing Theory reading make the game not fun to play Table 40 : Coded Test for Gne2 Items Codes Text Compliments Good overview with engaging story. The story is interesting The game is attractive Suggestions Add more details of the business Add more details of the industry Details on how DW af fect the business and industry More explanations about the data sources Graphical and animation objects. More explanations on the strategy side like why 3 phases? Multiple games should be based on multiple businesses. List processes by steps Add demo video in this stage Criticism The page design is not attractive.

PAGE 203

185 Table 41 : Coded Test for Gne3 Item Codes Text Compliments Summaries encourage players to make better decisions in the next phases. The summaries are described pretty goo d. Phase summaries are really clear. Suggestions Provide more details on variance of choices Elaborate more on the numbers Add more statistic values like ratio Hover over decisions to see more details. Use colors to highlight simulated values and expected values Criticism It is somewhat complex and hard to understand Creates some doubts rather than explaining what happened. es are calculated . Table 42 : Coded Text for Gne4 Item Codes Text Compliments Budget and other factors were well explained . Learning outcomes are well presented throughout the game. It is good to use categories to group data sourc es. The game clearly showed increased benefit and decreased cost rates. Providing common strategy factors was very good. Suggestions Consider data quality as a feature Consider importance of data (is it part of benefits?) More elaborations on the decompos ition of the development into phases More elaborations on the relationship between cost and benefits from the start of the game More elaborations on the impact of change in budget (because of events) More elaboration on the separation of ETL processes More elaboration on the link between features and categories Show real data source examples from each category Learning effects need more elaboration and examples performance. Criticism Categories can be confusi ng The integration between strategy and capability is not clear

PAGE 204

186 Table 43 : Coded Text for Gne5 Item Codes Text Compliments The instructions are clear and sufficient. Game playing instructions is straightforward . It was easy to fol low. Suggestions Use simpler terms for explanations. Give more details Provide detailed instructions for beginners and one for frequent players. GIF image to show how to select values or a help mark to provide more instructions. Criticism Text based inst ructions are boring. Should add more graphs, Language was in high level. Help instructions are not clear Table 44 : Coded Text for Gne6 Item Codes Text Compliments Simulation is a great tool. Simulation cle arly helps to understand the estimated values for cost and profit. Suggestions Provide more details on cost and benefits Manual decision inputs beside s the sliding bar Change the scale in graphs Increase simulation attempts Explain the differences between simulated and optimal Provide comparison values between simulated and optimal Criticism Graphs are small and do not show close differences between simulated and optimal. Need to compare the values by looking.

PAGE 205

187 Table 45 : Coded Text for Gne7 Item Codes Text Compliments Summaries are straightforward . Summaries are useful Suggestions Allow players to select the number of simulation attempts Add graphs Provide more details Allow organizations of simulation attempts (by profit) Sum mary by category for more feedback Criticism Table 46 : Coded Text for Grl4 Item Codes Text Compliments It is good approach to learn about data warehouse development. User interface is very good and easy to follow. Overall game experience was quit e good. Suggestions Examples at the beginning of the game Make random budget Enhance the game interface. Make it more of a game environment with animation, pictures, and more colors Demo video Precise game instructions with tutorial. Us e real case example Criticism Using the same budget for each game is boring Too much text is lengthy Descriptions and instructions are a bit complex Difficulties in understanding what are feasible options.