FRAMES Critical Thinking Project
Foothill's Rubric Assessment Model for Evaluating SLOs
The purpose of FRAMES is to improve student learning on campus. Starting with one of Foothill's four institutional student learning outcomes (SLOs), a team of faculty designed an assessment rubric that they will now test on over 150 student artifacts for evidence of Critical Thinking. More teams of faculty have volunteered to head up the development of tools to assess Computation, Communication, and Community/Global Consciousness SLOs.
FRAMES: Critical Thinking Phase I (completed)
The resources, methods, and results of 24 other education institutions were identified and researched by the team based on their student learning outcomes and assessments for evaluating critical thinking. A list of over 100 statements evaluative of critical thinking was created. Over the next few months, the team worked within the parameters of local knowledge and relevancy and filtered the list of statements based on clarity, purpose, and semantics. When the filtering process was complete, a draft rubric was formed and ready for initial testing.
Initially, the team applied the draft rubric to 10 assignments: 5 essays from the History department and 5 letters from the Adaptive Learning department. Modifications were made to several of the statements, but overall the rubric worked for both sets of assignments.
The committee tested the rubric on 11 more assignments: 6 essays from the English department and 5 short research reviews from the Biology department. The rubric continued to work well for both departments, however the scoring system that was being used was altered to allow more flexibility in choosing which assessment statements to evaluate assignments. Finally, the team analyzed 9 more assignments using the rubric: 6 written artifacts from the Computer Information Systems department and 3 recorded performances from the Drama department. The rubric continued to work well for both sets of assignments and one seemingly repetitive statement was removed from the rubric.
In summary, the participating faculty agreed that they were better able to plan critical thinking assignments for their students when working backwards from the rubric. In addition, the team agreed that the rubric statements and scoring system were useful tools for assessment.
FRAMES: Critical Thinking Phase II (completed)
The FRAMES Critical Thinking group started the second phase of the implementation process by planning how to collect and assess 150 student artifacts using the Critical Thinking rubric. After defining the parameters by which to randomly select student artifacts, the team sent a letter the second week of the quarter to the instructors of selected students requesting they choose an assignment that requires critical thinking and is due no later than Week 6 of Spring Quarter 2007. The random student artifacts came from students with at least 60 quarter units acquired from Foothill College. When the instructor had the students’ work in hand, they forwarded it to the team. All identifiers were removed from the students’ work before evaluation, and all assignments were handled confidentially within the Critical Thinking team. Only 64 artifacts were collected during Spring 2007, so the project was extended to Fall 2007 in order to collect a larger sample for the benchmark year.
During Fall Quarter 2007, another 100 artifacts were collected using the same parameters. During the second week of Winter Quarter 2008, each artifact was labeled and again distributed to two members of the Critical Thinking team for evaluation. The Critical Thinking team evaluated the artifacts during Winter Quarter 2008 and examined the results along with other FRAMES teams during Spring Quarter 2008.
FRAMES: Critical Thinking Results
The scoring scales that the FRAMES Critical Thinking team used were as follows:
- Excellent: 14-16 points
- Good: 11-13 points
- Minimally Competent: 8-10
- Deficient: 0-7
Based on preliminary research from the Instruction and Institutional Research Office, we knew that critical thinking was a challenge for many of our students. Using a sample size of 157 critical thinking assignments from across the campus, the team found that the mean score was 9.2 and the median score was 9.5. On average, our students are minimally competent in critical thinking based on the FRAMES Critical Thinking rubric. More importantly, what do we do with this revealing benchmark data? Total FRAMES Critical Thinking results:
- Sample size (N=157)
- Mean = 9.2, Median = 9.5
- 35 of 157 (22%) Excellent
- 36 of 157 (23%) Good
- 39 of 157 (25%) Minimally Competent
- 47 of 157 (29%) Deficient
FRAMES: Critical Thinking Going Forward
The FRAMES Critical Thinking team, consisting of faculty from each division felt strongly that the campus needed to address the critical thinking deficit in the classroom in order to meet the college's institutional critical thinking outcome. During Fall Quarter 2008, there will be a series of workshops in which the FRAMES Critical Thinking faculty will work with individual faculty to redesign their assignments in order to include stronger critical thinking skills found in the rubric.
