College logo
Outcome Assessment at College of DuPage

Report on the CIL English Assessment Project Part I: English 101

February 14, 2000

To:       Dr. Irene Kovala, Dean, Alternative Learning
            Dr. Annette Haggray, Associate Dean, Alternative Learning Director, Center for Independent Learning
            Jan Geesaman, Interim Associate Dean, Liberal Arts
            Joanne Parke, Coordinating Manager, Off-Campus Centers for Independent Learning
            CIL English Instructors

From:  Deborah E. Ryel, Associate Professor, English -- Professional Assistant,
                Center for Independent Learning

Regarding: Report on the CIL English Assessment Project Part I: English 101

The first part of the CIL English Composition Assessment Project is designed to study 25 English 101 student papers from Fall Quarter CIL students, 1999.

I would like to thank Irene Kovala, Dean of Alternative Learning and Annette Haggray, Associate Dean of Alternative Learning and Director of the Center for Independent Learning, for their support for this project. I would also like to thank Ida Hagman, former Professional Assistant for English Composition in the CIL for her participation in the planning of this project and our study of the results. It was Ida who first saw the need for an assessment project along these lines, and she has continued to support the CIL program through her commitment to flexible learning and her expert knowledge of the program.

Please share your response to this report with Ida and me. We would appreciate hearing your observations or receiving a note with your comments.

The CIL English Composition Assessment Project

Part I: English 101

Purpose:

The purpose of this assessment project is to determine if successful students actually demonstrate adequate writing that conforms to the objectives of the course. To learn more about successful students in particular, we have chosen to study only the work of students who receive A or B in English 101. This assessment is an informal study on a small scale, and is not intended as a statistical study. Rather it is a sampling to provide food for thought and discussion among us. This assessment project is the first to target CIL composition students only, so we are hoping to gain some useful information about our students’ performance. We also hope that the assessment will help us evaluate the effectiveness of our syllabi.

Procedure:

Selection of English 101 Papers:

At the end of fall quarter, 1999, 25 papers were selected from the fall quarter English 101 student folders: 1 paper from each of the off campus sites, and 21 papers from the on campus student folders. All the papers were selected from the folders of students who had received a B or A in the course. In addition, the papers were selected from those papers completed after midquarter, so that they would represent work from the second half of the quarter. An effort was made to select papers from the files of as many different instructors as possible, both full and part time.

Preparation of the 25 English 101 Papers:

The names of the students and instructors were removed from the selected papers. Grades and annotations on the papers were also whited out. Copies of the papers were made. Copies of the criteria sheets, which had been tested in the summer, were also made. See the Criteria Sheet for English 101, page 8.

Holistic Grading Session

Four CIL English composition instructors participated as graders for this holistic grading session in addition to Debby Ryel and Ida Hagman: Radha Avadhani, Denise Cantrell, Carol Kropp, and Tess Fawell. We met Friday evening, January 21, in the CIL. Three hours had been allotted for the task. The graders were paid for their time.

We began by reading one sample essay as practice to discuss the criteria and compare evaluations. Then we split up into two groups to read the essays. Two graders read each essay. For each student paper, the grader filled out a criteria sheet, indicating strengths and weaknesses, and then gave an overall evaluation. Graders were asked to rate each paper overall as

+ a strong paper ( A or B)

/ an adequate paper ( C or C+)

        an inadequate paper (D or F)

After each paper had been read by 2 graders, the criteria sheets were compared. If the overall evaluations on the criteria sheets were not in agreement, the paper was assigned to a third grader.

After completing the grading, we discussed our reactions to the process, to the papers, and to the criteria sheets we were using. Some useful observations emerged from our discussions in addition to the information provided by a study of the criteria sheets.

Compiling the Results

The results were tallied by totaling the evaluations on the criteria sheets in terms of three categories (strong +, adequate /, inadequate --) wherever two graders were in agreement. The overall evaluations were tallied and also the results by individual criteria.

 

Overall Findings:

The papers in the study demonstrate that on the whole these students are satisfying the objectives of the English 101 course. Two criteria were in strong evidence in the papers: having a clear purpose and choosing a topic appropriate to the assignment. Almost half the papers were also cited for strong paragraph development. Organization, sentence structure, diction and mechanics were other areas where student performance was adequate or better.

Sentence structure, grammar, and mechanical problems were not significant weaknesses in the papers. Instead, thought, the thesis statement, and paragraph organization were the weaknesses which lowered the overall evaluation of a paper. One third of the papers showed weaknesses in regard to two criteria: maturity of thought and focusing the topic by means of a clear thesis statement

A review of the grading also demonstrates that we do not have agreement in our grading practices on the level of success demonstrated by the papers. In particular, there seems to be uncertainty or differing standards for how to distinguish between an adequate paper (C grade) and a strong paper (B or A).

Numerical Summary of Findings:

The Overall Evaluations, out of 25 English 101 papers (at least 2 graders concur):

The Overall Evaluations by Types of Essays:

7 Argument Essays:

6 Essays in Response to an Essay:

4 Comparison/ Contrast Essays:

4 Cause/Effect Essays:

3 Personal Experience Essays:

1 Definition Essay:

Strengths of the Papers (out of 25 papers):

At least 2 graders concur in citing the following strengths from the criteria:

Demonstrate a strong controlling purpose

Had chosen an appropriate topic for the college setting

Incorporated a strong thesis statement

Demonstrate strongly a coherent, unified organization

Demonstrate strength in sentence structure and word choice

Avoid errors effectively in grammar, usage, punctuation, or spelling.

Weaknesses of the Papers (out of 25 papers):

At least 2 graders concur in citing the following weaknesses from the criteria:

Grading

The original instructors’ grades ranged from B- to A.

The holistic graders’ evaluations ranged from inadequate to strong;

In the holistic grading session, 13 papers out of 25 were read by a third grader.

Discussion:

Student Success: 21 out of 25 Successful Papers:

Out of the 25 English 101 student papers in this study, only 2 papers were considered inadequate in their efforts to meet the objectives on the criteria sheet. Of the remaining papers, 2 papers received inconclusive evaluations. As a result, 21 papers were judged to be meeting the criteria. Therefore, the writing of most of the students in the study satisfies the objectives of English 101. See the Numerical Summary of the Findings, page 7.

Strengths:

Purpose, Topics, Organization, Sentence Structure, Diction, Mechanics:

Sixteen papers were cited for demonstrating a strong controlling purpose and 20 papers were cited for an especially appropriate topic for a college paper. No papers were considered weak in these areas. These two criteria were the highest results cited among the strengths of the papers. The next highest ranked strength was strong paragraph development (10) which represents slightly less than half the papers. These findings may suggest that our syllabi, texts, and instructors were effectively helping these students in these areas and that our expectations in these areas are clear. See the graph, A Sample of Student Performance, page 6.

Organization is another area where we see success in the student papers. Seven papers were mentioned because the organization was a particular strength, while only two papers were cited for weakness in organization. Organization is probably an area to which our texts and instructors give special attention and an area also stressed in high school preparation, judging by the state tests for public school students.

Surprisingly, sentence structure and mechanics emerge as areas of strength, not weakness. Seven papers were cited for strong sentence structure and word choice, and no papers were considered weak in this area. Six papers were cited for effectively avoiding mechanical errors, and only three papers were considered weak in this area. We seem to be satisfied with these students’ progress in the areas of sentence structure, diction, and mechanics. Probably we are in agreement that proficiency in these areas is required of successful students.

Weaknesses:

Thought, Thesis Statement:

Sentence skills and mechanics were not weak areas for these students. Instead, the two papers considered inadequate showed serious weaknesses in thought, thesis statement, and organization. Overall, eight papers failed to demonstrate a mature level of thought and seven papers failed to focus the topic adequately with an acceptable thesis statement. Only five papers demonstrated strengths in each of these areas. In other words, about a third of these students are not showing an adequate mastery of these objectives, in the judgment of the graders. What’s more, the weaknesses in regard to thought and thesis statement are emerging in the papers of students we otherwise consider successful on the basis of their grades in the course. This finding raises the question of whether our syllabi and texts for English 101 provide enough useful guidance for the instructors and the students in these areas. It also raises the question of whether we instructors have a common understanding of what we want students to accomplish in these areas. Could this finding prove true for a larger number of our students in addition to this sample?

Differing Grading Standards:

The holistic graders considered only eight of the papers to be strong papers (of B or A quality), while the original instructors’ grades ranged from B- to A. This finding suggests that we have differing grading standards, a conclusion also reinforced by the fact that half of the papers in the study needed a third grader because of disagreement in the overall evaluation. Probably, our graders were representative of our instructors as graders, since we did not take time in the grading session for the kind of training usual for holistic grading. On the other hand, two graders were in agreement for twenty-three of the twenty-five papers. We are encouraged by that level of agreement; however, we may need to review our grading standards.

Problems with the Criteria:

What Makes an Above-Average Paper?

The criteria as presently stated do not include paragraph organization, which we found ourselves wanting to note in several cases as a marked weakness. The holistic graders also noted no guide to the relative importance of the criteria. How important should mechanics be, for instance, in relation to thought or organization?

More importantly, the criteria sheet does not set any standards in its articulation of the objectives. How should one distinguish between an adequate paper and an above-average paper? This question clearly emerged from our discussion during the grading session. What is the "something more" we expect for students whose papers are above the norm?

Some Possible Recommendations:

Additional Study.

We could do a more extensive study targeted to determine if any specific findings in this study emerge similarly with a larger number of student papers.

We could survey the faculty on their response to the assessment project. Our

survey could ask if the instructors’ teaching experiences reinforce or contradict any of the findings. Instructors could also be asked what follow-up approaches they would favor.

Improving Instruction.

We could review the texts and syllabi that we are presently using in English 10l to see how they address the two weakest areas: maturity of thought and thesis statements.

A discussion of the faculty about how to define maturity of thought, perhaps in terms of critical thinking skills, might be a first step toward articulating a more useful explanation of these criteria. This discussion could be followed by a brainstorming session on how instructors presently address these criteria.

In relation to thesis statement, a supplemental handout geared to the kind of writing done in English101 (and English 102) might prove useful. The handout we presently have is probably most relevant to English 103.

A criteria sheet for grading English 101 papers could be developed, which would help instructors easily give detailed feedback to students on the strengths and weaknesses of their papers. The criteria sheet could also be used by students as a self-assessment tool. We currently have such a criteria sheet for English 103.

A series of small group grading workshops for the CIL instructors is one of the best ways to explore grading standards, especially to address the question of what distinguishes an above-average paper from an average paper.