Student Course Evaluation Project:
CoPE Final Report (May 2004) (PDF)
Mandate from CoPE/Steering Committee (May 2004)
------------
Proposed New SCS Form (March 2005)
Updated/Edited New SCS Form (April 2005)
Proof Draft of New SCS Form (April 2006) (PDF)
Proposed New SCS Analysis System (March 2005) (PDF)
Report of the Committee on Pedagogy and its Evaluation (CoPE)
and the Steering Committee
on the Student Course Survey Form
The Steering Committee has charged the Committee on Pedagogy and its Evaluation (CoPE) to consider how to approach a revision of the Student Course Survey (SCS) form -- and of the manner in which the data compiled from responses to the form are analyzed. This report presents seven recommendations that are the outcome of the committee's discussions and of the discussions of the steering committee:
- The role of the SCS, to provide a consistent College-wide standard for evaluating teaching, is important and should be conserved. Changes to the form which might make it more effective for teaching development should be considered.
- Additional questions should be introduced that ask students to assess their learning without reference to the instructor's performance. At the same time, some questions focussing on faculty "knowledge" should be eliminated. The addition of a few narrative questions to the form should be considered.
- The length of the form should not increase substantially (the form should occupy no more than one sheet of paper).
- Should Williams use an on-line form? There are important advantages and disadvantages to consider.
- The analysis of the form should be adjusted to avoid artificially sharp boundaries such as those between quintiles.
- The committee designing changes in the SCS form, its analysis, and the reporting of results would benefit from including two advisory members: David Brodigan (Director of Institutional Research), and a representative from OIT (to advise on technical matters).
- The best use of the "Blue sheet" portion of the teaching evaluations should be reconsidered.
The rationale for these recommendations is presented below.
- The role of the SCS, to provide a consistent College-wide standard for evaluating teaching, is important and should be conserved. Changes to the form which might make it more effective for teaching development should be considered.
The SCS, as a form of evaluation, was standardized and began to be administered to all classes in the late 1980s. It replaced a longer form that had questions that were somewhat more diagnostic than evaluative in their emphasis, and limited the questions to a) those that asked the students to describe themselves (gender, effort, etc.) and b) those that asked the students to evaluate characteristics of the course and the instructor. In the early 1990s, the questions were revised and verbal descriptors added to the numbers on the evaluation scale to try to increase consistency in the students' interpetation of the numerical scores; these changes were generally successful in accomplishing their purpose. CoPE considered the use of similar forms by several peer institutions as reported by David Brodigan of the Provost's office, and found that the Williams College SCS form, its use, and its analysis are generally more comprehensive than similar evaluation procedures at peer institutions. Our survey is used more widely (by all instructors), is more uniformly used (the same form is used by all departments except the Athletics department, which has its own forms), generally includes a larger set of questions, uses a broader scale of responses (1-7 instead of 1-4 or 1-5), and is analyzed more comprehensively than at comparable colleges. Some of our peer institutions (Bowdoin, Mt. Holyoke) have forms similar to those used at Williams; some use open-ended, unquantified questionnaires for evaluation (Carleton, Smith, Bryn Mawr), and others use a combination of open-ended questions and questionnaires that are numerically scored and analyzed college-wide (Wesleyan, Middlebury, Wellesley, Mt. Holyoke). Still other institutions have no centralized form of evaluation (Amherst, Pomona, Oberlin, Trinity, Swarthmore). As this brief summary points out, Williams College can reasonably be said to be a leader in the use of quantitative, campus-wide surveys for evaluation.
CoPE considers that Williams College's universal use of the SCS form, though somewhat unusual within our group of peer institutions, is an important tool for evaluation and serves the College and the untenured faculty well. The ability to provide information that can be compared across departments and across cohorts is important to the College, because it allows standards that are consistent from year to year and from department to department to be applied in the evaluation of teaching. The information gathered from the forms also serves to protect untenured faculty members from potential inconsistencies or biases within individual departments. We strongly recommend that no changes in the form or in the way it is analyzed should be made that would compromise these important functions.
At present the SCS form is of limited use in helping new teachers develop pedagogical strength. Factors contributing to this problem include the following:
- Uniformity of the form. At present the form can be tailored to particular course styles or disciplines to only a very limited extent. Thus, questions relevant to particular modes of learning (e.g. labs, language learning, studio experiences) are largely absent. If the forms were switched to an online format they could be more specifically tailored and analyzed, and as a result might give more useful information to teachers.
- Absence of context. At present the statistical information is received in aggregate. Thus, it is possible to know that some students rated the course higher for "educational value," and that some rated it higher for "previous expectations" or "difficulty." However, it is not possible to know whether the students who got most out of the course were also those who found it most difficult, or were those who did most (or least!) work. Possible solutions (again most easily incorporated online) include short narrative responses to complement the statistical data, as well as correlated data that gives at least some impression of the connections between students' responses to various key questions on the forms.
- Focus on instructor performance, rather than course effectiveness. This issue is discussed in the next section.
In summary, we urge that changes to the SCS forms be considered in light of teacher development as well as effective evaluation. We expect that many changes that will aid one cause will also benefit the other.
return to list of recommendations
- Additional questions should be introduced that ask students to assess their learning without reference to the instructor's performance. At the same time, some questions focussing on faculty "knowledge" should be eliminated. The addition of a few narrative questions to the form should be considered.
The questions in the Williams College SCS are generally couched in terms that place the instructor in the foreground. The sense of the committee is that it would be useful to include some additional questions that ask the students to assess how their own learning was affected by the course. Such questions could also address whether courses are contributing to college-wide initiatives to improve students' writing and quantitative reasoning skills. Some possible questions might include:
- How valuable was this course in improving your writing skills?
- How valuable was this course in improving your quantitative reasoning skills?
- How well did this course promote your learning about the subject?
- How strongly would you recommend this course to other students?
Questions of this type would be useful in two ways. First, they would allow the instructor and the College to determine how courses are contributing to an instructor's goals for the course as well as to the curricular goals of the institution. Second, such questions would ask students to assess how successful a course was in promoting learning without specifically tying the response to the instructor's performance.
Some of the current instructor focussed questions, particularly those that ask about the instructor's expertise in the field, are useful neither for evaluation nor for teaching development. Students are generally not in a position to assess how expert faculty are, and have been known to assess faculty expertise most harshly when faculty are teaching in their precise specialty. (Perhaps the raising of more open questions gives the students a sense of unease.) We recommend that such questions be removed from the SCS forms.
return to list of recommendations
- The length of the form should not increase substantially (the form should occupy no more than one sheet of paper).
The Committee strongly suggests that the addition of any questions (such as those described in our second recommendation) be balanced by deleting some questions from the existing form. The SCS should fit, without the use of small type, on two sides of a single sheet of standard letter-sized paper. The reason for this recommendation is the concern that a long form decreases the probability that students will consider questions carefully and thoughtfully and answer each question independently (instead of, for example, checking off "5" all the way down the sheet). Two sides of a single sheet of paper is, we feel, the right length.
Which question or questions should be removed? Our first candidate for elimination is the question on the instructorıs "Knowledge of subject." Many of us have experienced the phenomenon of receiving low scores on this question when teaching in areas central to our own expertise (where we are careful to convey all the nuances of particular issues) while receiving high scores in areas where we are less knowledgeable (and so are less likely to qualify any statements we make). It is possible that there are other questions that could be combined or eliminated. In addition, there are very few courses that require answers to all of the existing questions: many courses have neither a foreign language nor a laboratory/studio component, and few if any have both. If forms generated for a specific course only included questions relevant to that course the issue of space on the sheet of paper that presents the SCS would be less problematic.
return to list of recommendations
- Should Williams use an on-line form? There are important advantages and disadvantages to consider.
Using an on-line form would make several aspects of administering the SCS and compiling results easier, and practical considerations might favor this option. However, there are equally important disadvantages, primarily in the area of how the evaluation process presents itself to the students.
There are several characteristics of on-line forms that would make them attractive. Our third recommendation pointed out the advantages of being able to "tailor" forms to the type and format of individual courses. This would be very easy to do with an on-line form, and less easy to do with a paper form. With an on-line form, limiting questions to what can fit on one sheet of paper would become less problematic. Collecting responses into a database is easy to do with an on-line form, more difficult with a paper form, and still more complicated if there are different versions of the paper form. An on-line form would also allow the instructor to add questions specific to their own courses, which could be tabulated and communicated to the instructor without being passed on to the instructor's department or used in evaluation. Thus the ease of gathering and collating data and the flexibility of on-line forms are clear advantages that would favor their use.
The committee unanimously agreed that there are important disadvantages to on-line forms. If forms are administered outside class hours, as would be the norm for on-line forms, the effects on responses must be considered. Some of the questions raised by the potential use of on-line forms include:
- Should students who do not bother to come to class be given the opportunity to evaluate the course? Having students fill out forms in the classroom associates the forms with a certain seriousness of purpose and an academic context. Would on-line forms promote a more cavalier attitude as students fill out forms while engaged in other activities (for example, during a vigorous beer-pong match)?
- How do we ensure that enough students respond to make the analysis of the forms useful? If filling out the form is optional, the number of students responding is likely to decrease dramatically. If students are required to fill out the form, perhaps by making the completion of all forms a condition for the release of grades, will quickly-formulated, box-checking, unthinking responses be promoted?
- How would we ensure that the students' privacy is maintained, that data collected are not handled and used inappropriately, and that security is sufficient to prevent invalid responses?
We find these issues troubling enough to feel that they potentially outweigh the administrative advantages of on-line forms. CoPE recommends that the advantages and disadvantages of on-line forms be considered further before any decision about using on-line forms is made.
return to list of recommendations
- The analysis of the SCS form should be adjusted to avoid artificially sharp boundaries such as those between quintiles.
The use of SCS scores in evaluation has, for understandable reasons, come to be centered primarily upon quintiles. The volume of paper dedicated to the analysis of the SCS responses for each course and each instructor can be somewhat overwhelming for the recipient and for the Department and the CAP to consider. Thus the quintiles, a simple integer score with only five possible values, are attractive as a summary of the overall student response. However, the emphasis on quintiles does not serve the College or the untenured faculty well. Each quintile range covers 20% of the responses, and the boundaries are sharply drawn. Thus quintiles artificially separate, for example, an average response with a percentile rank of 39.75 (second quintile) from an average response with a percentile rank of 40.00 (third quintile). Likewise, the boundary between the third quintile and the fourth quintile artificially distinguishes between average responses with percentile ranks of 59.75 and 60.00. Yet these distinctions (second vs. third or third vs. fourth quintiles) may play a very large role in the evaluation of teaching, and thus in the tenure process.
Using statistical significance to define differences from college averages in the evaluation of SCS scores raises similar issues. The standard for statistical significance is that there is a 5% (or smaller) chance that the distribution of scores for that course could be due to some anomaly of the sampling process. Again, a sharply drawn boundary draws a line between sets of responses at the 5% line and those with a value of 5.25%. It is our sense that such sharply drawn lines may create artificial distinctions between two courses or instructors whose performances are essentially identical.
Thus CoPE recommends that the analysis of SCS scores move away from any system that draws sharp boundaries within a set of responses that are actually continuous. The changes instituted by the Steering Committee after the report by Tiku Majumder's subcommittee -- placing the instructor's scores on a graph of the entire set of responses emphasizes the position of the scores within a continuous context -- were a good first step in this direction.
Two possible methods for analyses that do not draw arbitrary boundaries along a continuous scale have occurred to us. One method is related to that originally proposed by David Zimmerman: reporting the difference between the college averages and the scores for an individual or course in terms of the number of standard deviations (with a plus or minus sign used to denote above and below the average. Another possibility would be to give the percentiles for the instructor's or the course's scores, with the lowest score falling at zero and the highest at 100. The advantage of the percentile method is that it is easily understood and can readily be converted into quintiles for those who wish to relate results to the existing system. The disadvantage of percentiles is that they represent a ranking, in order, of all instructors or courses, with no indication of actual intervals between the ranks. For example, the difference in actual scores between the 45th and 55th percentiles might be smaller than the difference between the 15th and 25th percentiles. Information about the nature of the intervals between scores could be provided by including, along with the percentile ranks, the range of absolute scores with 5 percentage points on either side of a given score. In contrast, the standard deviation method for reporting SCS scores would more accurately reflect a score's distance from the college average, but would be less intuitive for many users and is harder to relate to the existing quintile system. Both of these systems, whether used separately or in concert, would have the important advantage of avoiding arbitrary and often meaningless boundaries that draw sharp distinctions between faculty members who are performing at essentially the same level.
Revision of the statistical analyses of SCS data is long overdue. The original analyses were instituted in the early 1980s, and the programs that perform the analysis are written in Fortran, a now obsolete programming language, and are very difficult to maintain, much less to upgrade. In addition, the accretion of changes in forms, analyses, and reporting have strained the existing system to its limits and make it very inefficient (in terms of human resources devoted to the analysis and reports). Finally, the existing program's limitations and antiquity place limits on the data analyses that can be performed. For these reasons, CoPE recommends that an entirely new analysis program be implemented.
In revising the analysis of SCS scores, it is important that the relationship between the existing system and any new system is clear and can be easily understood. The new system and the existing system should run in parallel for an interim period so that a smooth transition between the two can be accomplished. It is also important that the analysis of all student evaluation forms, including those from Athletic Department forms, should be conducted using the same methods, and all reports should have the same format.
return to list of recommendations
- The committee designing changes in the SCS form, its analysis, and the reporting of results would benefit from including two advisory members: David Brodigan (Director of Institutional Research), and a representative from OIT (to advise on technical matters).
This report recommends substantive changes in the SCS forms and in their analysis and reporting. Even if such changes are not made, the programs (and computers that can run those programs) that manage the SCS form data, perform the statistical analyses and generate reports are near or at the end of their useful life span and will need major revision or complete rewriting in the very near future. Thus CoPE's recommendation that the statistical analyses of SCS data be revised coincides with the need to implement new software (and hardware) for performing the analyses. Next year's committee will have the task of defining the specific implementation of these general recommendations and of ensuring an orderly transition between systems. It is our sense that David Brodigan, who oversees the analysis and reporting of SCS results and is familiar with other approaches at other institutions, would provide valuable insights and advice to the future committee. Because the new committee will need to address issues with implications for software design, and because it would be best to ensure that any new system makes appropriate use of existing expertise and resources in programming and data management, we further recommend that a representative from OIT also be named to the committee to advise on technical matters. We suggest that the Steering Committee consult with Dinny Taylor (Chief Technology Officer) in naming the OIT representative. This recommendation arises from our sense that the design and implementation of a new SCS form and revised analyses will be substantially easier and smoother with the participation of the two advisory members.
return to list of recommendations
- The best use of the "Blue sheet" portion of the teaching evaluations should be considered.
Blue sheets are most commonly used as private and general communication to the instructor regarding the course. However, some individuals and some departments use forms with set questions to draw out student opinion on specific issues of course effectiveness. Given the current limitations of the SCS, it would be useful for departments to discuss how best to make use of blue sheets for teacher development. A resource sheet or web site on developing useful blue sheet questions would likely be helpful to both new and experienced faculty.
return to list of recommendations
Committee on Pedagogy and its Evaluation
Ralph Bradburd
Stephen Freund
John Gerry
Julie Greenwood
Marjorie Hirsch
Cornelius Kubler
Karen Merrill
Heather Williams, Chair
Steering Committee
Denise Buell, Chair
Sarah Bolton
Julie Cassiday
Antonia Foias
Soledad Fox
Tom Smith