Deepening Mathematics Understanding of Middle School Teachers
Teachers' work was examined not only for accuracy of final answers, but also for processes in obtaining answers and a subsequent verification process involving articulating sound reasoning for the answers obtained. The following measures were used to show changes in teachers' understanding.
Objective Assessment of Content Knowledge. The Learning Mathematics for Teaching (LMT) project at the University of Michigan developed and made available several sets of items designed to assess content knowledge for teaching mathematics. GBMP used a variation of the Content Knowledge for Teaching Mathematics - Patterns (CKTM-Patterns) as an objective measure of changes in content knowledge as a result of participation in the Patterns course. The CKTM-Patterns was administered to middle school teachers at the beginning and end of the course. The pretests and posttests for individuals were then paired so that individual progress could be examined. For further evidence of reliability, a Cronbach alpha was calculated to determine the internal consistency of the test. The test showed good internal consistency (alpha = .82).
Performance Assessment. During the summer courses, instructors administered a performance assessment at the beginning of the course and again at the end of the course. Evaluators used the Oregon Department of Education Mathematics Problem Solving Official Scoring Guide as their rubric to assess four domains (conceptual understanding, processes & strategies, communication, and accuracy). Only those assessments for which both the pre-task and the post-task were available were scored.
Although the rubric has been used successfully in Oregon, the evaluators took precautions to ensure that there was consensus on the performance assessment scores. Three scorers applied the rubric individually to each of the assessments then met to discuss their scores and to resolve any discrepancies. The individual results were in good agreement in most cases, and in all cases, the consensus scores were easily determined.
Course Portfolio.As part of the summer courses each participant kept a portfolio of completed tasks, assignments, and reflections. The portfolios were assessed using a rubric designed by the evaluation team that was based on the project definition of "Challenging Courses and Curriculum." Three members of the evaluation team piloted the rubric on five portfolios, and in January 2006, the rubric was reviewed by the rest of the GBMP team. Slight modifications were made as a result of both the pilot exercise and input from the team.
Five portfolios from middle school teachers were randomly selected from each of the three summer course sessions, resulting in a total of fifteen portfolios to be scored. The same process of reaching consensus on scores that was applied to the performance assessments was used with the portfolios.
Behavioral Checklist. Researchers developed a behavioral checklist for use in observing teachers as they participated in the GBMP courses. The checklist contained four dimensions-deepening mathematics understanding, productive disposition, inquiry and reflection, and communication. These four dimensions are the same dimensions that form the GBMP's definition of Challenging Courses and Curriculum. For each dimension, there were four observable behaviors that were determined to be evidence of each dimension by the evaluation team. Three participants from each course section were selected randomly to be observed on three occasions throughout the 2-weeks.
Course Satisfaction Survey. The last day of course sessions was used to collect data from the participants. One of the instruments used was a simple course satisfaction survey that was comprised of seven Likert-type items and four open-ended items. The items addressed participants' perceptions of how the course may have benefited them and how well the course was presented. All participants were asked to complete this instrument. Responses were reported using frequencies.
Classroom Observations.The Reformed Teaching Observation Protocol (RTOP), specifically designed for mathematics and science classroom observations, was used as the classroom observation tool. The instrument focused on five areas: (1) lesson design and implementation, (2) propositional knowledge, (3) procedural knowledge, (4) communicative interactions, and (5) student/teacher relationships. When possible, teachers were observed across project years.
