Citation |

- Permanent Link:
- http://digital.auraria.edu/AA00003717/00001
## Material Information- Title:
- Predicting undergraduate mathematics success an analysis of UCDHSC placement methods and the accuplacer exam
- Creator:
- Olson, Gary A
- Publication Date:
- 2005
- Language:
- English
- Physical Description:
- xii, 63 leaves : ; 28 cm
## Subjects- Subjects / Keywords:
- Accuplacer (Computer program) ( lcsh )
Prediction of scholastic success -- Colorado -- Denver ( lcsh ) Mathematical readiness ( lcsh ) Mathematics -- Study and teaching (Higher) -- Colorado -- Denver ( lcsh ) - Genre:
- bibliography ( marcgt )
theses ( marcgt ) non-fiction ( marcgt )
## Notes- Bibliography:
- Includes bibliographical references (leaves 61-63).
- General Note:
- Department of Mathematical and Social Sciences
- Statement of Responsibility:
- by Gary A. Olson.
## Record Information- Source Institution:
- |University of Colorado Denver
- Holding Location:
- |Auraria Library
- Rights Management:
- All applicable rights reserved by the source institution and holding location.
- Resource Identifier:
- 62879140 ( OCLC )
ocm62879140 - Classification:
- LD1193.L622 2005m O57 ( lcc )
## Auraria Membership |

Full Text |

PREDICTING UNDERGRADUATE MATHEMATICS SUCCESS: AN
ANALYSIS OF UCDHSC PLACEMENT METHODS AND THE ACCUPLACER EXAM by Gary A. Olson B.A., Carroll College, 2003 A thesis submitted to the University of Colorado at Denver in partial fulfillment of the requirements for the degree of Master of Science Applied Mathematics 2005 This thesis for the Master of Science degree by Gary A. Olson has been approved Midge Cozzens Olson, Gary A. (M.S., Applied Mathematics) Predicting Undergraduate Mathematics Success: An Analysis of UCDHSC Placement Methods and the Accuplacer Exam Thesis directed by Professor Michael Jacobson ABSTRACT Predicting student success has long been a question of interest for admis- sion counselors throughout the United States. With the changing face of the nations post-secondary population, a need has also developed for mathematics departments to address this question. Existing research examines the validity of several methods designed for pre- dicting undergraduate success. High school record, standardized test scores, extra-curricular activities and combinations of all three have historically been successful predictors. However, limited research has been conducted on predict- ing success in lower-level mathematics courses. In recent years standardized test scores have become less valuable for plac- ing students in lower level mathematics courses; placement exams such as ACCUPLACER have taken their place. Validity studies for these exams are crucial for institutions to analyze cut-off scores and ensure students are appro- priately placed in courses that match their skill-level. In June 2003, the Colorado Commission of Higher Education (CCHE) ap- proved three tests for the purpose of entry and secondary-level assessment: the m ACT Assessment Test, the College Board Scholastic Aptitude Test, and the College Board ACCUPLACER. In Spring 2004, the University of Colorado at Denver and Health Sciences Center (UCDHSC) began using ACCUPLACER scores as mandatory placement criteria for all mathematics students enrolling in three courses, Analytical Geometry & Calculus I, College Algebra, and Alge- bra for Social Sciences and Business. Two of the initial goals of the placement procedure were to lower withdrawal rates and increase passing (A/B/C) rates for the three courses. This thesis begins with a literature search and analysis of past research devoted to predicting college success. Major studies from 1980 to 2004 are pre- sented and compared. Research on the prediction of lower-level undergraduate mathematics success is also discussed as well as two current studies of mathe- matics placement validity. The ACCUPLACER Exam is then analyzed along with a comparison of placement techniques for different post-secondary institutions throughout Col- orado. UCDHSC course data are then analyzed and predictors of mathematics success are discussed. Finally, statistical analysis is performed for all three classes and conclusions are drawn about the impact of ACCUPLACER on both withdrawal and passing rates. This abstract accurately represents the content of the candidates thesis. I recommend its publication. v DEDICATION This thesis is dedicated to my parents and three sisters for all of their love and support during my academic pursuits. ! I ACKNOWLEDGMENT There are many people that I would like to thank for their help during my thesis experience. First, I would like to thank my two readers and committee members, Dr. William Briggs and Dr. Midge Cozzens for all of their help and support. I would also like to thank my advisor Dr. Michael Jacobson for all of his help during the last year. I am grateful for the opportunity and support he gave me to perform research in the area of mathematics education. I would also like to thank Jennifer Thurston for her help in the extraction of my data set and all of the statistics professors and students at UCD for their insight. A special thanks also goes to Leslie Varys for her SPSS insight and statistical genius. CONTENTS Figures ............................................................. x Tables.............................................................. xi Chapter 1. Introduction...................................................... 1 2. Classical Methods for Predicting Long-Term College Success.... 3 2.1 SAT Verbal Scores............................................... 4 2.2 SAT Math Scores................................................. 5 2.3 SAT Verbal and SAT Math Scores ...................... 6 2.4 High School Record.............................................. 7 2.5 SAT Verbal Score, SAT Math Score, and High School Record ... 9 2.6 Extra-Curricular Activities.................................... 10 2.7 Problems Associated with Predictors of College Success......... 12 3. Placement Techniques for Lower-Level Undergraduate Courses .... 14 3.1 High School Coursework......................................... 15 3.2 SAT/ACT Scores................................................. 15 3.3 Placement Tests................................................ 16 3.4 Placement Exam Validity ......................... 17 4. The ACCUPLACER Exam ............................................. 20 5. Current Placement Techniques for Colorado Colleges .............. 23 6. UCDHSC Placement Techniques for Spring 2004 ..................... 30 6.1 Data Description.................................................. 31 6.2 Correlation Among Variables ...................................... 31 6.3 Analysis of the Results........................................... 34 6.4 Limitations of the Data .......................................... 34 6.5 Analyzing College Algebra Cut-Off Scores.......................... 35 6.6 MA 1110 Fall 2004 Statistics...................................... 38 7. Current ACCUPLACER Results.......................................... 40 7.1 Analysis of MA 1401............................................... 41 7.2 Analysis of MA 1110............................................... 44 7.3 Analysis of MA 1070 .............................................. 46 8. Discussion.......................................................... 49 9. Conclusions......................................................... 51 9.1 Future Analysis................................................... 52 Appendix A. Descriptive Statistics Tables....................................... 54 References............................................................. 61 FIGURES Figure 6.1 Aecuplacer Versus Final Grade Spring 2004 ....................... 35 6.2 Distribution of Course Grades for Students Scoring Between 72-85 . 36 6.3 Aecuplacer Versus Final Grade Fall 2004 ......................... 39 x TABLES Table 2.1 Predicting Cumulative GPA From SAT Verbal Scores.............. 4 2.2 Predicting Cumulative GPA from SAT Math Scores................ 5 2.3 Predicting Cumulative GPA From SAT Verbal & Math Scores ... 7 2.4 Predicting Cumulative GPA from High School Record ................. 8 2.5 Predicting GPA From SAT Verbal, Math, & High School Record . 10 6.1 Numerical Equivalent of Letter Grades......................... 32 6.2 Correlations for College Algebra Spring 2004 ..................... 33 6.3 Correlations for College Algebra Fall 2004 ....................... 38 7.1 Independent-Samples T-Test for MA 1401 Withdrawals............ 42 7.2 Independent-Samples T-Test for MA 1401 Mean Student Grades . . 43 7.3 Calculus Comparisons.......................................... 43 7.4 Independent-Samples T-Test for MA 1110 Withdrawals............ 45 7.5 Independent-Samples T-Test for MA 1110 Mean Student Grades . . 45 7.6 College Algebra Comparisons................................... 46 7.7 Independent-Samples T-Test for MA 1070 Withdrawals............ 47 7.8 Independent-Samples T-Test for MA 1070 Mean Student Grades . . 48 7.9 Business Algebra Comparisons ..................................... 48 A.l Descriptive Statistics for MA 1401 Withdrawals................ 55 A.2 Descriptive Statistics for MA 1110 Withdrawals................ 56 A.3 Descriptive Statistics for MA 1070 Withdrawals....................... 57 A.4 Descriptive Statistics for MA 1401 Student Grades.................... 58 A.5 Descriptive Statistics for MA 1110 Student Grades.................... 59 A.6 Descriptive Statistics for MA 1070 Student Grades.................... 60 1. Introduction Post-secondary mathematics departments currently use a variety of methods to place mathematics students in courses commensurate with their abilities. Each institution develops placement techniques based upon curriculum, student population, methodology, and placement test accessibility. Present mathematics placement methods include the use of ACT and SAT standardized test scores, high school record, a combination of standardized test scores and high school record, and placement exams like ACCUPLACER. Since 2002, the University of Colorado at Denver and Health Sciences Center (UCDHSC) has required all freshmen without AP credits or completed college level courses in mathematics to take the math ACCUPLACER. Beginning with spring semester 2004, the UCDHSC mathematics department began using math ACCUPLACER scores for placement decisions in three different mathematics classes: MA 1070-Algebra for Social Sciences and Business, MA 1110-College Algebra, and MA 1401-Analytical Geometry and Calculus I. After three semesters of using ACCUPLACER scores, it is now relevant to examine the validity and success of the current methods. This paper first explores the classical methods used by admission officers to predict academic success throughout college. Placement techniques for lower-level undergraduate classes are then discussed and placement validity is examined for two cases in the literature. Next the ACCUPLACER exam itself is analyzed along with current placement procedures at four-year institutions throughout Colorado. Finally, 1 UCDHSC mathematics placement techniques are discussed and correlations are developed to examine predictors of success in UCDHSC College Algebra courses. Statistical analysis is then used to analyze withdrawal rates, passing rates, and mean student grades for ACCUPLACER required classes. Initial results show that ACCUPLACER is a statistically significant predic- tor of both final exam score and final course grade for MA 1110 students. Also, withdrawal rates for all three courses has statistically decreased with the use of ACCUPLACER. Mean student grades for MA 1401 students after ACCU- PLACER have increased, but analysis shows no statistical difference for mean student grades of MA 1110 and MA 1070. Finally, passing rates have increased for MA 1401, but no statistical change is observed for MA 1110 and MA 1070. 2 2. Classical Methods for Predicting Long-Term College Success Predicting student success has long been a question of interest for admission decisions. Counselors aim to admit students who will be successful in their chosen program and eventually attain their degree. Research suggests that the best predictor of college graduation is first year college grade point average [27]. In addition, first year grades are the single most revealing indicator of successful adjustment to the intellectual demands of a particular colleges course of study [10]. To aid counselors who seek predictions of first year GPA from just an application packet, researchers have analyzed the best methods for estimating first year college GPA. Since 1980, several major studies have analyzed predictors of college success. Throughout most of the studies, five main methods are analyzed: SAT Verbal score, SAT Math Score, a combination of SAT Verbal and SAT Math score, High School Record, and a combination of SAT Verbal score, SAT Math score, and High School Record. In addition to these five cognitive predictors, non- cognitive factors such as extra-curricular activities, admission essays, and college interviews have also been researched. In a 2001 College Board research report by Burton and Ramist, the major collegiate success studies since 1980 were grouped and analyzed. The results will be summarized according to the five main cognitive predictors of college suc- cess. Correlations for each individual study were used to determine a weighted correlation for each of the five main predictors. The results are detailed in the 3 sections that follow. 2.1 SAT Verbal Scores Table 2.1 summarizes the results of nine major studies conducted between 1980 and 1998. Each study examined the correlation between SAT Verbal score and undergraduate cumulative GPA [4], Predictor Paper Year # of Students Correlation Young & Barrett 1992 91 .17 Shoemaker 1986 296 .21 Shoemaker 1986 238 .23 SAT Crews 1993 336 .37 Verbal Elliot & Strenta 1988 927 .38 Moffat t 1993 28 .42 Ra 1989 170 .42 Young 1991 1,564 .46 Total Students 4,155 .40 Table 2.1: Predicting Cumulative GPA From SAT Verbal Scores A weighted average (the average of the reported correlations weighted by the number of students included in each study) gives an overall correlation of r = .40 between SAT Verbal scores and first year undergraduate GPA. Notice that this weighted average is highly dependent upon Youngs 1991 study and the 1988 study by Elliott and Strenta. These were large studies which contributed 4 significantly to the overall weighted average. Also of interest is the particularly low correlation of r = 17 associated with the Young and Barrett study of 1992. This low correlation could be attributed to the fact that only 91 students were analyzed, of which several were outliers [28]. 2.2 SAT Math Scores Table 2.2 summarizes the results of nine studies conducted between 1986 and 1993. In each study, SAT Math scores were analyzed as predictors of un- dergraduate GPA. Predictor Paper Year # of Students Correlation Ra 1989 170 .28 Crews 1993 336 .31 Elliot & Strenta 1988 927 .34 SAT Moffat t 1993 28 .35 Math Shoemaker 1986 238 .35 Young & Barrett 1992 91 .41 Shoemaker 1986 296 .43 Young 1991 1,564 .46 Moffatt, 1993 505 .49 Total Students 4,155 .41 Table 2.2: Predicting Cumulative GPA from SAT Math Scores 5 A weighted average shows that for 4,155 students there is an overall corre- lation of r = .41 between SAT Math scores and GPA. Comparing these results with that of SAT Verbal scores, one can conclude that either measure will result in predictions with similar accuracy. In recent years the ACT exam has also been extensively used, particularly in the Western United States. Preliminary analysis shows that ACT Verbal and Math scores produce similar results and correlations as the SAT [4]. 2.3 SAT Verbal and SAT Math Scores Table 2.3 summarizes results of ten major studies which analyze the com- bination of SAT Verbal and SAT Math scores as a predictor of college GPA. In each study, multiple regression analysis was used to determine the combina- tion of S AT Verbal and Math scores which would produce the best predictor of student performance at each respective institution. Notice that the weighted average produces a correlation of r = .36 between the SAT Verbal and Math combination and college GPA. This correlation is lower than either SAT Verbal or SAT Math alone. This occurs because the correlations are based on different samples of students and institutions. In par- ticular, the 1992 study by Baron and Frank examined 3,816 students and only found a predictor correlation of r = .20. This is perhaps due to the combination of Verbal and Math score that they chose to use [2]. If the samples were compa- rable, the correlation for the combination of SAT Verbal and Math scores would be higher [4] and prove to be a slightly better predictor for college success than either SAT Verbal or SAT Math alone. 6 Predictor Paper Year # of Students Correlation Baron & Frank 1992 3,816 .20 Nettles & Thoeny 1986 4,094 .31 SAT Moffatt 1993 28 .34 Verbal Wolf & Johnson 1995 201 .34 and Ra 1989 170 .39 SAT Tracey & Sedlacek 1985 1,339 .40 Math Willingham 1985 3,442 .41 Elliot & Strenta 1988 927 .43 Ragosta & Braun 1991 2,473 .52 Moffatt 1993 505 .56 Total Students 16,995 .36 Table 2.3: Predicting Cumulative GPA From SAT Verbal & Math Scores 2.4 High School Record Table 2.4 summarizes results from twelve major studies which compare high school record to college GPA. Each study used either cumulative high school GPA, high school class rank, or a combination of the two in order to define high school record. From the table it can be seen that high school record has been the most studied predictor of college success. These twelve studies involved 25,175 partic- 7 Predictor Paper Year # of Students Correlation Baron & Frank 1992 3,816 .30 Young & Barrett 1992 91 .31 Young 1991 1,564 .35 Wolf & Johnson 1995 201 .40 High Elliot & Strenta 1988 927 .41 School Nettles & Thoeny 1986 4,094 .41 Record Shoemaker 1986 238 .41 Ra 1989 170 .44 Willingham 1985 3,442 .45 Leonard & Jiang 1995 10,000 .46 Shoemaker 1986 296 .48 Crews 1993 336 .59 Total Students 25,175 .42 Table 2.4: Predicting Cumulative GPA from High School Record 8 ipants from different ethnicities, backgrounds, geographical location, and college institution. Overall, the weighted average signifies a correlation of r = .42 between high school record and undergraduate success. Notice this correlation is slightly higher than either SAT verbal or SAT math scores. Past studies have consis- tently shown this to be the case. With the rise of grade inflation, the validity of using high school record alone has become a concern [23]. High school GPA is highly dependent upon which region, state, school district, and even school a student came from. As the meaning of GPAs vary between schools and more students are being categorized in the A and B range, admissions counselors need to combine GPA with other criteria in order to make accurate predictions. 2.5 SAT Verbal Score, SAT Math Score, and High School Record As a result of grade inflation, most post-secondary schools have instituted a combination of high school record, SAT Verbal, and SAT Math scores for ad- mission decisions. Table 2.5 summarizes results from five major studies in which predictors were developed from a combination of all three measures. Multiple regression was used to determine the best combination of SAT Verbal, SAT Math, and high school record at each institution. The weighted average shows a correlation of r -- .52 between college GPA and the combination of SAT Verbal, SAT Math, and high school record. Also of interest is that every major study had a correlation higher than any of the weighted averages for the other predictors. The results from these studies illus- trate that the combination of SAT results and high school record provide the 9 best prediction of college success. Predictor Paper Year # of Students Correlation SAT Verbal, Leonard & Jiang 1995 10,000 .49 Math, and Willingham 1985 3,442 .53 High Ra 1989 170 .58 School Young 1991 1,564 .58 Record Ragosta & Braun 1991 2,473 .62 Total Students 17,649 .52 Table 2.5: Predicting GPA From SAT Verbal, Math, & High School Record 2.6 Extra-Curricular Activities Another factor that aids the admissions process, is the participation in extra- curricular activities. High school students are encouraged to join and partici- pate in several extra-curricular activities to strengthen their college applications. However, research has consistently shown that participation in these activities is not a stand-alone predictor of college success. In Willinghams study from 1985, twenty-five thousand applications were received for nine different colleges. All of the applications were examined and admissions counselors made decisions on which students would be successful at their institution. Several factors went into determining admission, including the use of high school GPA and extra-curricular activities [27], 10 After admission was granted, 4,814 of tlie students were followed through four years of college and their accomplishments were recorded. Although extra- curricular activities played a large role in determining admission, only high school GPA was determined to be a predictor of college success [27]. Likewise, in a 1994 study by Wade and Walker, honors students were tracked for two years of college. Admissions at the small southern university were dependent upon GPA, class rank, ACT scores, extra-curricular activities, and selection of honors courses. Again, high school GPA proved to be the only consistent predictor of success [24], Finally, the Standard Research Service of the ACT program conducted a study of 10,758 college freshmen in which multiple criteria for determining col- lege success were examined. In an attempt to predict English success, each student was evaluated based upon ACT scores, GPA, and yearbook and high school newspaper staff participation [14]. Both the ACT English score and the students high school GPA were determined to be valid predictors of English success. However, extra-curricular activities were shown to have no real positive correlation to success in English courses [7]. Although grade point average is the most consistent predictor for success, Rice and Darke contend that extra-curricular activities are an indirect predictor of success [20]. In their 2000 study, participants were randomly selected from Leadership or Academic Scholarship students at a public university. While the researchers agreed that GPA is an accepted predictor of success, they also con- tend that, the strongest addition to the traditional predictors of college success was a measure of persistent and successful extra-curricular accomplishment in 11 high school. 2.7 Problems Associated with Predictors of College Success Recent concern has been raised that predicting college success based upon standardized test scores and high school record alone may only provide accurate predictions for majority groups. Research indicates that traditional methods of prediction have decreased validity when applied to minority and non-traditional students and other factors need to be considered when examining their applica- tions. Ramist et al. [19], conducted a study on the predictive validity of SAT scores for Blacks in 11 different mainly White colleges. The researchers found a correlation of r = .09 between SAT and Black students grades while there was a correlation of r = .176 for SAT and White students grades. This study is one of many that have led critics to argue that the assessment provided by standardized tests is culturally and educationally inappropriate for students from racial and ethnic minority groups [4]. A difference has also been found for the predictive validity of the SAT be- tween males and females. Burton and Ramist [4], point out that the SAT his- torically under-predicts the first year college performance of females and slightly over-predicts the performance of males. Many factors have been considered as possible explanations. One contribution to this difference is that males enroll in more advanced placement math and science classes in high school, leading to slightly higher SAT scores [12]. Other studies have shown that non-cognitive predictors have more influence on predictor validity for females than their male counterparts. Areas such as self-esteem, motivation, and student support were 12 deemed more important additions to the use of standardized tests for female students and could potentially explain the SATs historical under-prediction of female success [11]. 13 3. Placement Techniques for Lower-Level Undergraduate Courses In todays post-secondary environment, many students enter without the proper background necessary for successful completion of mathematics courses required for their field of study. According to the Colorado Commission of Higher Education (CCHE), 26.6% of Colorado public high school graduates (7,507 stu- dents) entering Colorado public higher education in 2002-2003 were assigned to remediation. Of the students assigned to remediation, 85.3% were assigned remediation in mathematics [6]. Therefore, proper placement and recommenda- tion techniques play a crucial role in ensuring that students are placed in the appropriate course commensurate with their current skill level. Currently about 90% of post-secondary institutions utilize some form of placement and devel- opmental instruction [22], College administrations seek placement procedures that are simple, economical and defensible; while faculty seek a procedure that matches appropriate student skill with course objectives. Although much research has been conducted on predicting college success, fewer studies analyze predictors of lower-level undergraduate mathematics suc- cess. Traditional predictors of math success have been the number of high school mathematics classes and scores on standardized tests (SAT/ACT exam). In re- cent years, the use of placement exams such as ACCUPLACER have also been used to determine appropriate student placement. The following sections examine the literature on methods for predicting un- dergraduate mathematics success and developing predictive validity for place- 14 inent techniques. 3.1 High School Coursework The effect of high school courses on lower-level undergraduate mathematics success was studied by Roth et. al [21] in 2001. The study found that taking more higher level math courses in high school is an accurate predictor of scoring well on aptitude tests commonly required for admission into four-year baccalau- reate institutions. Even the students with average or poor grades in these high level courses were better prepared for college level work and less likely to need non-credit or remedial math. The exposure to higher level coursework helped students recognize concepts covered on the placement exams and achieve pass- ing scores. Students who took four years of high school math were also found to be better prepared for immediate entrance into post-secondary education. The fourth year of mathematics was deemed as a bridge which links the material from high school to college [15]. 3.2 SAT/ACT Scores Standardized test scores are often used as one option for determining which students require remedial mathematics. For instance, a student with an ACT Math score of 19 or above, or an SAT Math score of 460 or above would fulfill remediation requirements set by CCHE. If post-secondary institutions use these standardized test scores for specific course placement, they are usually accompa- nied by an alternative cut-off score on a placement examination. This is due to research suggesting that standardized test scores are good predictors of mathe- matics exam scores, but are not overall predictors of success in a mathematics 15 class. Intangible and non-cognitive factors such as desire, motivation, and peer- study have been shown to have a large effect on overall performance in the class and offset much of the correlation due to test-taking skills. These tests are still considered predictors of college success, but are becoming outdated measures to predict mathematics success; as a result standardized tests are often replaced by new placement techniques [25]. 3.3 Placement Tests Many community colleges and four-year institutions are beginning to imple- ment placement tests as either the sole means of placement, or as an important part of the placement decision [12]. Several placement tests are available and common tests used within the U.S. include: 1. Assessment of Skills for Successful Entry and Transfer (ASSET) 2. New Jersey College Basic Skills Placement Test (NJCBSPT) 3. Mathematical Association of America Placement Test Program (MAA) 4. Descriptive Tests of Mathematical Skills (DTMS) 5. ACCUPLACER Test 6. ACT COMPASS Tests The majority of these placement exams are actually sets of tests which include one or more components in mathematics [3]. These tests can be admin- istered in a variety of ways. In an effort to best prepare students for college, many high schools are now administering placement exams to their graduating 16 students the last month of school. Other placement exams are administered to students during required orientation periods before the first semester of classes. Placement exams can also be administered prior to specific courses and deemed as prerequisite requirements for successful enrollment. 3.4 Placement Exam Validity According to the American Mathematical Association of Two-Year Colleges (AMATYC), regardless of how the placement test is administered, the main goal of testing is to determine the highest level of mathematics appropriate to students educational goals, at which they have the prerequisite knowledge to be successful [18]. AMATYC goes on to state: placement tests should provide a measure of students abilities not only to show mastery of algorithmic skills, but also to think critically and solve problems. This leads to one of the main drawbacks of placement testing; the perceived notion that capable students may be denied access to higher-level mathematics instruction at which they could have been successful. Independent studies by Armstrong [1], Jenkins [9], and Isbell [8] have all determined that a large portion of students actually succeed in mathematics classes even though their test scores labeled them as unprepared. This notion has driven research on placement validity and the development of placement techniques that avoid unfairly holding back potentially successful students. Two recent studies have addressed the issue of developing better predictive placement techniques. Judith Marwick [12] recently completed a study at a Mid- western University which compared three alternative mathematics placement methods with the rigid placement by exam score alone. All students who took 17 the ACCUPLACER exam during the summer of 2001 were randomly assigned to one of four placement methods: ACCUPLACER exam score alone, high school preparation alone, a balance of the two measures, or student choice constrained by the two measures [12]. The study found that overall the students performed equally well in their mathematics course regardless of which method was used for placement. When comparing placement test scores with high school record, it was determined that placement of the students would have been approximately the same regardless of which technique was used. Final recommendations show that placement by ACCUPLACER score alone denied many students access to the appropriate course in which they could have been successful. Marwick con- cluded that placement techniques should involve a combination of high school record and placement test score with students placed in the higher-level course recommendation. Another ongoing study of placement technique validity has been conducted at Indiana University Purdue University Indianapolis (IUPUI). At IUPUI a computerized mathematics placement exam is used which consists of 40 objective items ranging from arithmetic operations to introductory calculus. A students raw score is determined by the number of items answered correctly. Based on the score students are given recommendations for placement in one of three different remedial courses, or allowed to take the corresponding college course of their choice dependent on their academic major. As students were allowed to choose whether to follow the placement rec- ommendation, they naturally formed two separate groups for comparison, com- pliant and noil-compliant. Predictive validity of the mathematics placement 18 exam was then studied by analyzing the mathematics results of the two differ- ent groups. For the compliant students who followed the placement test course rec- ommendation, the correlation between placement score and mathematics exam score was .23. For students who did not follow placement recommendations (i.e. chose to take a more difficult course), the correlation between placement score and mathematics exam scores was .25. Also of interest is the correlation between placement test score and final course grade. For compliant students the average correlation coefficient was r = .19 and for non-compliant students r .17. As a result of this study, IUPUI concluded that its current placement tech- niques needed revision. Students who were not following placement recom- mendations were succeeding in higher level coursework and there was no sta- tistical difference in the correlation between compliance and performance and non-compliance and performance. Following the study, the mathematics depart- ment switched to a Computerized Adaptive Testing (CAT) procedure much like the ACCUPLACER Exam. Preliminary results show an improvement in cor- relation between placement score and mathematics success; however, the non- compliant students still performed as well as the compliant students indicating that placement test scores still prevented capable students from enrolling in suitable coursework. 19 4. The ACCUPLACER Exam With the help of committees of college faculty, The College Board developed the ACCUPLACER Computerized Placement Tests. These are adaptive tests that choose the difficulty of questions based on responses to previous items. There are ten different tests available which measure reading, writing, English, and mathematics ability. Within the mathematics category, three different tests are available to measure varying skill levels: the Arithmetic Test, the Elementary Algebra Test, and the College-Level Mathematics Test. Each of these tests is multiple choice with no enforced time limit. The College-Level Mathematics Test assesses proficiency from intermediate algebra through pre-calculus. Twenty questions are asked and the topics covered include algebraic operations, solutions of equations and inequalities, coordinate geometry, applications and other algebra topics, and functions and trigonometry. The Elementary Algebra Test is twelve questions long and covers topics includ- ing operations with integers and rational numbers, operations with algebraic expressions, and solving equations, inequalities, and word problems. Finally, the Arithmetic Test consists of sixteen questions covering operations with whole numbers and fractions, operations with decimals and percents, and applications and problem solving. ACCUPLACER utilizes Item Response Theory (1RT) to create a more ef- ficient, and individualized assessment of student mathematical skills. fRT grew out of the need to individualize assessment and adapt questions based on the 20 skill level the student exhibits. The IRT technique that ACCUPLACER uses first creates a latent ability, 8, that can be assessed given items of known dif- ficulty, b. IRT also includes parameters to account for item discrimination, a, and guessing, c [26]. These are combined into a 3-parameter model: PW = C+ P(8) probability of respondent with ability 6 to answer a question of difficulty b correctly. a = item discrimination. Proportional to the slope: 0.425a(l c). b = item difficulty c = guessing parameter 8 = respondents ability This logistic model is referred to as an Item Characteristic Curve. Ability discrimination is theoretically maximized when P(8) is close to the inflection point for a bank of items. If there is no guessing, this point would be around 50%. In order to ensure maximum discrimination for each item, the set of questions should differ for those of varying ability levels. In the Computerized Adaptive Test format, answering a question correctly will raise the difficulty of the next question and an incorrect response lowers the difficulty of the question that follows. This process keeps P(8) near the curves inflection point until 8 can be estimated within an acceptable error range [26]. This procedure requires 21 fewer questions than non-adaptive tests and can reduce both time and facilities needed to administer the exams. There are also drawbacks of the ACCUPLACER, particularly in the Elemen- tary Algebra Exam. The adaptive nature of the test results in large penalties for incorrect answers chosen early in the exam. If a student answers initial questions incorrectly, future questions are worth fewer points. This can lead to difficulty building up to questions of higher point value. Also, the difference between an ACCUPLACER score of 72-85 and above 85 is due to one additional correct answer. Therefore, one correct guess can effect a students chances of achieving a cut-off score. 22 i i ! i j i 5. Current Placement Techniques for Colorado Colleges Colorado law (CRS 23-1-113) requires all incoming freshmen and transfer- students to be assessed for their basic skills in reading, writing, and mathe- matics. In order for a student to show proficiency in mathematics they must meet one of four different requirements set by CCHE. A student who achieves a score of 19 or higher on the ACT assessment mathematics exam, a score of 460 or higher on the SAT Math section, or a score of 85 or higher on the ACCU- PLACER elementary algebra exam does not require remediation. In addition, if a student has successfully completed a college-level mathematics course or a remedial mathematics course, no additional assessment is necessary. Following this May 2000 legislation, most schools throughout Colorado have implemented placement procedures for lower level mathematics courses based loosely upon remediation assessment guidelines set out by CCHE. The following gives a brief summary of placement techniques used at various colleges and universities throughout Colorado. Adams State College Students entering Adams State College with an ACT mathematics score of less than 19 (470 SAT mathematics score) are required to take the ACCUPLACER Elementary Algebra Test. Based on the results of that exam students are advised as follows: 1. Score of 85 or above: Student may enroll in Finite Mathematics or College Algebra. 23 2. Score of 55-84: Student may enroll in Intermediate Algebra. 3. Score of 40-54: Student may enroll in Basic Algebra Skills Class. 4. Score of 39 or lower: Student may enroll in Arithmetic Skills Course. 5. ACT Score of 26 or higher: Student may enroll in Calculus I as their first college math course. Oth- erwise the student must first take College Algebra before enrolling in Cal- culus. Colorado Christian University Students entering Colorado Christian University with appropriate ACT or SAT scores may enroll in the course of their choice. Students who have not taken either the ACT or SAT exam must take the math placement exam COMPASS. Based on the results of this exam the student will be advised to take remedial math or proceed with the appropriate college level course. Colorado School of Mines (CSM) No exams are required for students enrolled at CSM. The entry-level mathemat- ics class for all students at CSM is Calculus I. Colorado State University Students who have received a score of three or above on the Advanced Placement 24 Calculus Exam (either AB or BC), or have transfer credit in a math course at the level of college algebra or above are not required to take any placement examinations. All other students must take the Mathematics Placement Exam (MPE). This exam consists of 50 multiple choice-multiple response questions and covers topics in College Algebra, Numerical and Analytical Trigonometry, and Logarithms and Exponential Functions. In addition, students may take the Entry Level Mathematics (ELM) Exam. This exam covers intermediate algebra topics and helps guide student choices for the entry-level mathematics courses offered at Colorado State University. Colorado State University-Pueblo All students are placed according to their ACT or SAT math score. If students would like to challenge the placement, they may take the ACCUPLACER Exam and attempt to receive a qualifying score. Fort Lewis College All students are required to take the ACCUPLACER placement exam. Each student begins with the elementary algebra exam. If a student correctly an- swers a certain proportion of the questions on this exam, they will then take the college-level mathematics exam. If a student does not successfully answer enough questions on the elementary algebra exam, the student is then directed to take the arithmetic exam. Based on the results of these tests, the student is advised on which course to enroll. 25 Metropolitan State College of Denver Students with an ACT Math score of 19 or above or an SAT Math score of 460 or above are exempt from placement exams. All other students are required to take the ACCUPLACER Exam and must take the course corresponding to their score. The placements are as follows: 1. Score of 0-56 on the Arithmetic Exam: Students may take Fundamentals of Mathematics. 2. Score of 57-120 on the Arithmetic Exam: Students may take Pre-Algebra. 3. Score of 45-60 on the Elementary Algebra Exam: Students may take Introductory Algebra (a Community College of Denver (CCD) course). 4. Score of 61-84 on the Elementary Algebra Exam: Students may take Survey of Algebra (a CCD course). 5. Score of 85-99 on the Elementary Algebra Exam: Student may take any of the following: Integrated Mathematics I, Math- ematical Modes of Thought, Introduction to Statistics, College Algebra with peer study, or Finite Math with peer study. 6. Score of 100-120 on the Elementary Algebra Exam: Student may take either College Algebra or Finite Math. 7. Score of 65-79 on the College Level Math Exam: Student may take Pre-Calculus. 26 8. Score of 80-120 on the College Level Math Exam: Student may take Calculus I. University of Colorado at Colorado Springs (UCCS) Two math placement exams are offered at UCCS. The first is the Algebra Di- agnostic which measures intermediate algebra skills. Students scoring above 20 on this exam are also invited to take the Calculus Readiness Test. The scores are used for placement as follows: 1. Score of 0-8 on the Algebra Diagnostic: Student is required to take Fundamentals of College Algebra. 2. Score of 9-16 on the Algebra Diagnostic: Student may take College Algebra 3. Score of 17-19 on the Algebra Diagnostic: Student may take Elementary Functions of Calculus, Topics in Linear Algebra, or Calculus for Business and Economics. 4. Score of 20-32 on the Algebra Diagnostic: Student may take Elementary Functions of Calculus, Topics in Linear Algebra, or Calculus for Business and Economics. In addition, the student may take the Calculus Readiness Exam. 5. Score of 0-9 on the Calculus Readiness Exam: Student may take Elementary Functions of Calculus as preparation for Calculus I. 27 6. Score of 10-12 on the Calculus Readiness Exam: Student may take Calculus I along with the corresponding pre-calculus review offered in the math learning center. 7. Score of 13-25 on the Calculus Readiness Exam: Student may take Calculus I. University of Northern Colorado (UNC) Current placement methods at UNC are based upon orientation advising. At scheduled orientations, graduate students hold 5-10 minute advising sessions with incoming students. The students are asked a series of questions regarding their ACT/SAT math score, the mathematics courses which they took in high school, the length of time since their last mathematics course, and their intended major. Based on the information provided, the graduate students make course recommendations, however, students may register for the class of their choice. Western State College of Colorado (WSCC) WSCC uses a combination of ACT scores, SAT scores, and a placement exam to assess incoming students. Students are placed in the courses according to the following criteria: 1. Basic Algebra Review: Students may take this class regardless of their performance on the three placement techniques. 2. Mathematics for the Liberal Arts, Mathematics for the Managerial Sci- ences, Algebraic Functions, or Theory of Arithmetic and Geometry I: 28 In order to enroll in any of the above courses a student must attain a score of 19 or above on the ACT Math exam, score a 460 or above on the SAT Math exam, pass the ACCUPLACER elementary algebra test with a score of 85 or above, or pass the Basic Algebra Review class. 3. Transcendental Functions: A student must pass the ACCUPLACER college-level mathematics test with a score of 75 or above. 4. Probability and Statistics: A student must pass the ACCUPLACER college-level mathematics test with a score of 85 or above. 5. Calculus I: A student must pass the ACCUPLACER college-level mathematics test with a score of 95 or above. The above sections detail placement procedures for several Colorado Col- leges. It is worth noting that some colleges do not have placement requirements for all courses. This leads to an alternate way many students choose to fulfill their remediation requirement; enroll in a lower-level college course that does not mandate placement procedures. At UCDHSC there are currently no placement procedures for MA 1010-Mathematics for the Liberal Arts Student, enabling any student to enroll. A student who is in need of meeting remediation requirements during the first 30 credit hours, may take MA 1010 and fulfill the requirement. This option enables the student to gain college credit while fulfilling remedia- tion. 29 6. UCDHSC Placement Techniques for Spring 2004 In Spring 2004, UCDHSC began mathematics placement procedures for all sections of MA 1110-College Algebra and MA 1070-Algebra for Social Sciences and Business. Mandatory placements were made based upon the students per- formance on the ACCUPLACER Elementary Algebra Exam. For the Spring semester a passing score of 72 was required in order to continue enrollment in each class. All students were given two opportunities to pass the ACCUPLACER Exam, with a one-day grace period required between sittings. Students who failed the exam twice could petition the Associate Chair of the Mathematics Department for a third opportunity at attaining a passing score. Elementary Algebra review sessions were available in the Math Education Resource Center (MERC) and utilized by many students. If a student did not obtain a passing score on the exam by the second week of class they were administratively dropped from the course and advised to take proper remediation to prepare them for the course. Detailed data were recorded for all Spring 2004 College Algebra students. A database was created with the following information for each individual stu- dent: ACCUPLACER score, ACT score, SAT score, College Algebra instructor, common final exam score, and final course grade. Statistical analysis on these data had two goals: 1. Determine if statistically significant correlations exist between ACCU- PLACER score, ACT score, SAT score, score on the common final exam, 30 and final course grade. 2. Analyze the effect of raising the cutoff score in MA 1110 from 72 to 85. 6.1 Data Description A total of 151 students initially enrolled in College Algebra for Spring Semester 2004. Of the 151 enrolled, twelve students never took the ACCU- PLACER and were administratively dropped. In addition ten students were unable to attain a score of 72 or above and were also administratively dropped. A total of 19 students achieved a passing score on the ACCUPLACER Exam, but dropped the class. Thus, the study had a sample size of 110 students who passed the ACCUPLACER exam and also finished the course. By limitations of the data collection, ACCUPLACER scores of four stu- dents who took the course were unattainable. Also, as the majority of students in Colorado take the ACT instead of the SAT exam, the data includes ACT information for 84 students who finished the class and SAT information for only 32. Lastly, final exam scores were only available for 101 students; this indicates that nine students either did not take the common final exam or their score was unattainable in the data collection. 6.2 Correlation Among Variables For purposes of attaining correlations, all letter grades were converted to the UCDHSC numerical equivalent shown in Table 6.1: 31 > II o A-=3.7 B+-3.3 B=3.0 B-=2.7 C+=2.3 C=2.0 0 1 II I1 D+=1.3 D=1.0 o II i Q o o II eL Table 6.1: Numerical Equivalent of Letter Grades SPSS Software was used to calculate Pearson bivariate correlations among the five variables and analyze the statistical significance of each. The correlation matrix in Table 6.2 displays the results of the analysis. From the correlation analysis, it is evident that the highest correlation oc- curred between ACT Math score and SAT Math score. This is not a surprising statistic for the 32 students who took both exams; research has consistently shown the two standardized tests to be highly correlated. As expected grade on the final exam and final grade in the class were also highly correlated. One interesting aspect of this correlation analysis is that ACT Math score was determined to be a statistically significant predictor of ACCUPLACER score at the 0.01 level with a correlation of r = .473. However, the correlation between SAT Math score and ACCUPLACER score was not statistically significant with r = .293. This could be related to the smaller sample size for SAT scores which potentially caused error in the analysis. Also, both ACT math (at the 0.01 level) and SAT math (at the 0.05 level) were considered statistically significant predictors of final exam score, but not 32 ACT MATH SAT MATH ACCUP Grade Final ACT MATH Pearson Corr. 1 .742** .473** .191 .328** Sig. (2-tailed) .000 .000 .082 .004 N 106 32 99 84 77 SAT MATH Pearson Corr. 1 .293 .159 .404* Sig. (2-tailed) .063 .392 .030 N 42 41 31 29 ACCUP Pearson Corr. 1 .390** .418** Sig. (2-tailed) .000 .000 N 135 106 98 Grade Pearson Corr. 1 .691** Sig. (2-tailed) .000 N 110 101 Final Exam Pearson Corr. 1 Sig. (2-tailed) N 101 ** Indicates the Correlation is significant at the 0.01 level (2-tailed) * Indicates the Correlation is significant at the 0.05 level (2-tailed) Table 6.2: Correlations for College Algebra Spring 2004 33 final grade in the class. Finally, there is a correlation of r = .418 and r = .390 between ACCUPLACER and final exam and final grade respectively. Both correlations are significant at the 0.01 level. 6.3 Analysis of the Results Several conclusions can be drawn from the correlation analysis. The re- sults support current research which attests that standardized test scores alone should not be used to predict student success in mathematics courses. Both standardized test scores were statistically significant predictors of final exam score, indicating a relationship between test-taking abilities. However, as the research suggests, standardized tests scores are not a significant predictor of final grade. Also of interest, ACCUPLACER is a statistically significant predictor of both final exam and final grade. Therefore, course data validate the selection of ACCUPLACER as a placement technique for College Algebra students over the use of standardized tests. 6.4 Limitations of the Data Aspects of the data set limit the strength of some results. Correlations involving SAT Math data should be interpreted with caution. Only 32 of the 110 students who took this course had SAT Math scores available, therefore the data could be biased. In addition, correlations involving ACT Math scores may also reflect bias. Although ACT information was available for 84 of the 110 students, the missing information may not be due to random effects. Often students who do not report ACT or SAT information are low-achieving students who did not plan on attending college while in high school. This could result in 34 biased results where the excluded cases are heavily weighted with low-achieving students. 6.5 Analyzing College Algebra Cut-Off Scores The next question of interest is whether raising the required ACCUPLACER score to 85 would significantly effect students in College Algebra. Figure 6.1 provides a graphical representation of MA 1110 Spring 2004 course data. The scatter-plot displays ACCUPLACER score as the independent variable and final grade as the dependent variable. 4.00 3.00 a> TJ as 2.00 U. O 1.00 0.00 o O 00 o oo o o o oo o o 00 o ooo o o o o o cooooo o oooooo o ooo o o ooo 00 o ooo ocooa >0000 00 00 o ooo o o o o o oo O O OO 003 o o o I -------1------I------------1----------1 I I 60.00 70.00 80.00 90.00 100.00 110.00 120.00 ACCUPLACER Figure 6.1: Accuplacer Versus Final Grade Spring 2004 From the graph, first note that one student took the ACCUPLACER, scored a 64, and was still able to finish the class. Following up on this student, the 35 i i I I 1 I t f I I t I i I j I 1 i Numerical Grade Figure 6.2: Distribution of Course Grades for Students Scoring Between 72-85 mathematics department associate chair confirmed that approximately one stu- dent slips through the cracks every other semester and is allowed to take the class without a passing ACCUPLACER score. A total of 39 students qualified for College Algebra with an ACCUPLACER score between 72-85. Figure 6.2 shows the distribution of grades for these stu- dents. While eleven students in this range received a D or F and did not pass the course, 28 students received a passing grade of C or above. These results can be interpreted in two ways. By examination, 72% of students scoring in the 72-85 36 I j | range passed the course. This seems to indicate that the cut-off score should not be raised because capable students would be prevented from taking coursework in which they could be successful. However, many of these students achieved a score in that range on their first ACCUPLACER attempt. With attendance at an Elementary Algebra review session and pointed study, these scores could conceivably be raised to 85 or above. Also, out of the twelve students who received a failing grade of F in the course, nine fell in the 72-85 range. In order to make accurate conclusions, students should also be tracked on whether they achieved their passing ACCUPLACER score on the first, second, or third attempt. If the majority of the students who received scores in the 72-85 range achieved their score on the first attempt, then the evidence would support raising the cut-off score to 85. After teaching Elementary Algebra review sessions and working in the MERC lab, it became apparent that students scoring above a 72 on the first attempt would most likely raise their score to the appropriate level on the next attempt. Recall that improving ones score from a 72-85 to above an 85 requires the student to answer one more question correctly on the twelve question ACCUPLACER exam. Another factor influencing the cut-off score for College Algebra is the state requirement on remediation. As mentioned earlier, the Colorado Commission of Higher Education requires an elementary algebra score of 85 or above as one criteria for exemption from remediation. Since College Algebra is a non-remedial course, UCDHSC does not want to set its placement score lower than CCHEs remediation requirement. 37 6.6 MA 1110 Fall 2004 Statistics Beginning in Fall 2004, the required ACCUPLACER score for College Al- gebra was raised to 85 or above. Limited data from this class providing each students ACCUPLACER score and final course grade was available. The cor- relation matrix in Table 6.3 provides predictor information for ACCUPLACER versus final grade. ACCUPLACER Final Grade ACCUPLACER Pearson Corr. 1 .181* Sig. (2-tailed) .017 N 200 172 Final Grade Pearson Correlation 1 Sig. (2-tailed) N 173 * Indicates the Correlation is significant at the 0.05 level (2-tailed) Table 6.3: Correlations for College Algebra Fall 2004 It is interesting to note that ACCUPLACER is a significant predictor of College Algebra course grades at the 0.05 level. However, the correlation coeffi- cient of r = .181 is lower than any of the ACCUPLACER correlation coefficients for courses with a cut-off score of 72. Figure 6.3 illustrates a scatter-plot for the data with ACCUPLACER score as the independent variable and final course grade as the dependent variable. 38 Grade The graphical display illustrates no apparent patterns for the data collected with a cut-off score of 85 or above. 4.00- 3.00- 2.00- 1.00- o.oo- o o ooo o oo oo o oo o oo o ooo o o o o o o o oo oo oooooooooo ooooooooo o oo o
oo oooo o o |