Hearing on
“H.R. 4283, the College Access & Opportunity Act: 
Does Accreditation Provide Students and Parents Accountability and Quality?”

June 22, 2004

Thank you for the opportunity to discuss college student outcome assessment. As background about my own institution, I started and oversee the Center for Assessment and Research Studies at James Madison University in the Shenandoah Valley of Virginia. James Madison is a publicly supported institution of about 16,000 students.

Our Assessment Center is probably the largest campus based assessment center in the country and has been in operation since 1986. We have nine doctoral level assessment faculty and focus on assessment of general education, the major, and out-of-class activities in student affairs.

Since 1987, we have annually tested all entering freshmen, about 3600 students, just prior to matriculation, and then retest them about two academic years later in general education. The general education areas in which we regularly assess are: technology and information literacy, oral communication, written communication, critical thinking, arts and humanities, quantitative reasoning, scientific reasoning, government, and wellness.

We utilize several analytical strategies for analyzing these data:

1. competency—in technology, in information literacy and in oral communication—how many students reach a standard?

3. course impact—compare students in a given area of general education who have not yet complete any courses versus who have had one course versus who have had two or more courses—student should do better on the related outcome measure the greater number of courses completed.

We use the assessment data to improve our curriculum, to improve our degree requirements, and to evaluate the effectiveness of instructional delivery approaches such as a particular instructional software package.

For example, in 1996, we completely reworked our general education program, which is approximately one-third of our undergraduate curriculum, largely because we had mixed assessment results. In the new curriculum, we established policies that every general education course sequence must regularly demonstrate positive student learning for those courses to remain in the curriculum. Each academic department’s annual report features their assessment results of their respective major programs and if applicable their part of general education. Each academic department unit lead is evaluated on the viability of their department’s assessment efforts.

In general, higher education institutions have been slow and reluctant to systematically measure student learning in a programmatic way. It is a hard job, can be threatening to faculty and presidents alike, and involves much complexity. Faculty, of course, give course examinations, but most institutions do not use common measures particularly in general education across an institution. General education is the coursework that all students take regardless of major. I view general education as the primary focus for accountability purposes because it is the imprint of every graduate of the institution.

According to Gaff of the Association for American Colleges and Universities, over 90% of the colleges and universities deploy the distribution system of general education whereby students select one to three courses from a given distribution area such as social science where many courses are listed. Generally, the more courses listed in a given distribution area for students to chose from, the more difficult it is to have commonalities among all the courses listed. Assessing a given distribution area becomes problematic then when there is little instructional content or skills in common.

The problem of defining and therefore assessing general education is magnified if one goes beyond the institution and tries to develop commonalities among several institutions such as at a state level. Most institutions have not developed measurable, specific learning objectives, sometimes referred to as "content standards" in general education. Common "performance standards" are more rare when an institution sets a single cut-off score on an assessment instrument in any given area of general education.

Most states have some kind of policy or mandate regarding assessment. These policies vary from statewide tests such as Georgia’s Regents Exam to the majority of states that allow for institutions to choose their own ways of assessing. Some states such as Virginia designate the areas that are to be assessed: technology, written communication, scientific reasoning, quantitative reasoning, oral communication, and critical thinking.

Too often institutions over utilize self-report surveys as the primary measures of student learning. What students think they learn can be different than what they actually learned.

In the past few years, states have been experimenting with linking "quality" and funding but struggle with defining and measuring quality. Often states collect only what data are available, but these data usually do not include about student learning. For example, outputs such as graduation rates are available, but these do not describe how or what students have learned. I believe many states would benefit from assistance in improving their current policies and procedures.

The current state of assessment practice still makes it improbable to generalize beyond a single institution, and in many cases difficult to generalize within a single institution.

As you have noted, the consumer is hungry for information about quality. Prospective students, their parents, and employers desire information about what students acquire from a particular institution and what the value of that knowledge and skills has. For instance, look at the popularity of ranking systems such as the US News and World Report rankings. Unfortunately, those rankings are determined largely by reputation, resources, outputs, admissions selectivity, and alumni giving rates. No comparable student learning data are available for the public to view.

As you probably know, accrediting organizations, whether regional or professional, are increasingly emphasizing assessment of student learning. Before 1985, few institutions reported any results about student learning in accreditation reports. In my opinion, the accrediting organization’s role has been a very positive influence helping institutions focus on educational results not just description of resources. But the acceptance of including student learning data in institutional reviews has been slow.

In my opinion, an increased role by the federal government would be welcome. The US Department of Education’s involvement has been very positive at the K-12 level, but there seems to be a void of leadership when it comes to assessment of collegiate learning. No single entity seems to be coordinating collegiate assessment. I believe some time and effort could be saved if practices and instruments could be shared, for example. The issues are so complex and the job so big that a single state can feel overwhelmed.

Here are some current advances that could be spotlighted at the national level that could raise the sophistication of all of collegiate assessment: highlighting the advancements of automated computer scoring of writing, using advanced measurement techniques such as item response theory to reduce bias in assessment instruments, featuring computer based testing that utilizes multimedia capabilities allowing us to formulate better test questions, using advancements in cognitive psychology to assess where a student’s misconceptions lie so remediation can be better planned, and encouraging new assessment instruments to encourage greater sophistication of our measuring tools. Regarding the last point, a colleague and I are designing an instrument called the Curiosity Index to measure one’s intrinsic motivation to learn that is central to lifelong learning.

Similar to the food labels listing the ingredients, wouldn’t it be helpful to know what knowledge, skills, and personal characteristics college graduates possess? Such information would benefit both individuals and society. I applaud this committee for looking at the issue of collegiate outcome assessment, and I hope you can help us move the current state of affairs forward.

At my institution, we have designed our own assessment instruments in general education, and are just establishing standards or cut-off scores in each area of general education. Competence is then achieved by an individual student reaching a particular cutoff score; again our current areas of competency are in technology (Word, Excel, Powerpoint), in information literacy (the ability to find, access, and find credible information usually stored in electronic form), and in oral communication. All of these instruments are computer-based tests.

As you know, there is much attention on the delivery of instruction via technology, but there is little attention to the delivery of assessment and testing via technology. We can ask better and more sophisticated test questions by incorporating multimedia components such as music, speeches from national leaders for government test questions, and video vignettes of speeches for public speaking courses.

Testing, as you may know, also has a very technical side. Reliability, or the precision at which we measure a student’s ability, is critical and is validity, or the match between the learning objectives/content standards and the given test.

In general, most available collegiate assessment instruments have not used the most advanced measurement techniques. When we reviewed the problem solving, critical thinking, and writing collegiate assessment instruments for the National Postsecondary Education Cooperative (which receives financial support from NCES), very few testing instruments were designed with advanced techniques such as item response theory, generalizable measurement theory, or structural equation modeling.

For example, very few proprietary collegiate assessment instruments examine test item bias. This procedure is very common in college admissions tests and in K-12 tests. The shortage of reliable and valid assessment instruments for our institution is one reason our Assessment Center has the staff it does.

There are some exciting advances in the automated computer scoring of writing. Research has shown that the computer is more consistent in its ratings than people. And the relationship between the computer generated ratings and human ratings is moderately high.

I applaud Sec. 1025, "Study of Student Learning Outcomes and Public Accountability" (p. 201-203). I would offer two suggestions for additions. First, under point #3, p. 202, regarding instruments, please add "and sophistication." For example, our most advanced thinking in cognitive psychology and psychometrics should be brought to bear on existing practice. And second, I would also submit for your consideration that you request the report to offer recommendations for further steps. It is certainly useful to have summaries of the current status of state efforts, overlap in institutional effectiveness requirements among the accrediting organizations, and a review of existing instruments; but what is needed, in my opinion, is a series of action-oriented steps that outline how to achieve that an assessment process of student learning exists at each institution. This process is outlined to some extent in the regional accrediting associations, but more guidance could be established. Each college and university has learning as part of its mission, but not every college and university evaluates student learning in a programmatic way.

I am not advocating selection of a particular assessment instrument, nor am I advocating that you dictate what is to be measured. However, I do believe it is important to have aggregated information about student learning in general education that is available to the public.

Thank you again for the opportunity to share some ideas.