Rising Above a Gathering Storm of Data: Triangulating Findings Through Formative and Summative Evaluation Methodologies
Author: Sonya Martin

Contents
    Project & Author
    Document
3. Design, Data & Analysis
Print Poster
« Back to Poster Hall
3. Design, Data & Analysis
Next »

Internal evaluators used the Science Lesson Plan Analysis Instrument (SLPAI) to analyze unit lesson plans across four subscales: alignment with endorsed practices, cognitive and metagcognitive issues, sociocultural and affective issues, and portrayal and use of the practices of science. Each subscale contains two to eight items (a total of 21 items), which are individually scored and weighted in computing the subscale score and an overall score. The items contained in each subscale include:

  • Alignment with endorsed practices: alignment with standards and awareness of science;
  • Cognitive and metacognitive issues: education research, goal orientation, content accuracy, content presentation, pre-assessment, meaningful application, student reflection, and assessment;
  • Sociocultural and affective issues: equity, student attitudes about science, student engagement, student participation, classroom discourse/fostering community of learners, appropriate use of technology, adaptability, and variety of instructional techniques;
  • Portrayal and use of the practices of science: hands-on exploration, nature of science, student practitioners of scientific inquiry, and analytical skills development.

Baseline and Thesis/Capstone data are provided in Table 1 for all teachers completing both baseline and final lesson plans. When more than one reviewer rated lesson plans the scores were averaged. The following data is excerpted from an external evaluation report on this program (Scantlebury, Kahle, & Yue,, 2008).


Table 1. MISEP and MCEP Cohort A Teachers' Baseline 2005 and Final 2007 Scores on Unit Lesson Plans

Score Category

Baseline
# of MISEP
teachers

Capstone
# of MISEP teachers

Baseline
# of MCEP teachers

Thesis
# of MCEP teachers

Below 35%

1

1

2

3

35-49%

2

0

3

2

50-59%

5

4

2

1

60-69%

3

6

0

1

70% and higher

6

8

1

5

Table 2 shows the average percent for each subscale and overall average percent for MISEP baseline and Capstone unit lesson plans. There were significant differences between the baseline and Capstone lesson plans for the overall SLPAI score, as well as for each subscale score. All findings indicate improvement in the development of lesson plans.

Table 2.SLPAI Overall and Subscale Means for the MISEP Teachers' Unit Plan Analysis: Cohort A, Baseline 2005 and Capstone 2007

SLPAI Subscales

Base/
Capstone

n

Mean

SD

t-value

p-value

Alignment with endorsed practices

Base

15

68

18

3.6

0.0037

 

Capstone

15

87

10

 

 

Cognitive and metacognitive issues

Base

15

65

20

7.2

0.0001

 

Capstone

15

78

9

 

 

Sociocultural & affective issues

Base

15

65

18

2.3

0.0159

 

Capstone

15

78

20

 

 

Portrayal and uses of practices of science

Base

15

50

20

2.3

0.035

 

Capstone

15

65

15

 

 

Overall average percentage

Base

15

58

14

4.2

0.0003

 

Capstone

15

76

9

 

 

Table 3 compares average percent for each subscale and overall average percent for MCEP baseline and Thesis unit lesson plans. There were no significant differences for teachers' overall mean score for SLPAI and for the subscale, cognitive and metacognitive issues. There were significant differences between MCEP teachers' baseline and Thesis mean scores for the subscales alignment with endorsed practices, sociocultural and affective issues, and portrayal and uses of practices of science. These findings suggest that MCEP, compared to MISEP teachers were more reluctant to change the style and/or substance of their lesson planning.

Table 3. SLPAI Overall and Subscale Means for the MCEP Teachers' Unit Plan Analysis: Cohort A, Baseline 2005 and Thesis 2007

SLPAI Subscales

Base/
Thesis

n

Mean

SD

t-value

Alignment with endorsed practices

Base

7

46

28

2.3*

 

Thesis

7

75

19

 

Cognitive and metacognitive issues

Base

7

52

16

1.3

 

Thesis

7

68

31

 

Sociocultural & affective issues

Base

7

51

20

2.1**

 

Thesis

7

78

28

 

Portrayal and uses of practices of science

Base

7

35

19

3.2***

 

Thesis

7

69

21

 

Overall average percentage

Base

15

47

17

2.1

 

Thesis

15

70

24

 

* p < .05; ** p<.01; ***p<0.001

Six SLPAI items were identified by the internal evaluators as ones that were a concern for one or both MCEP and MISEP teachers. Pre/post analysis of these six items was completed for MISEP and MCEP teachers using Baseline Portfolio (pre) and Thesis/Capstone (post) data. MISEP teachers showed improvement on four of the six items, while MCEP teacher means improved for all six items, as shown in Table 4.

Table 4. Comparison of SLPAI Subscale Means by Project: Cohort A, Baseline and Capstone/Thesis 2007


Item

Weight

Baseline
MISEP Average

Capstone
MISEP Average

Baseline
MCEP
Average

Thesis MCEP
Average

Meaningful Application

2

1.3

1.9

0.6

1.4

Student Reflection

2

1.2

1.8

0.9

1.4

Assessment

3

1.3

1.6

1.0

1.1

Student Engagement

2

1.5

1.5

0.9

1.6

Nature of Science

3

0.7

0.7

0.0

0.9

Students as Practitioners of Scientific Inquiry

3

1.2

1.6

0.6

1.3

In summary, the higher scores in MISEP Capstone lesson plans may be attributable to the greater specificity of guidelines, scaffolding around curricular design models, longer duration of their lesson plan unit (MISEP, two weeks; MCEP, two to five days), and/or more extensive support by project staff. It is difficult to ascertain if teachers' low scores on the "Nature of science" subscale is due to a lack of implementation of this topic or to teachers failing to document their understanding and teaching of those issues in their lesson plans. Results from the "Views about science" subscale on the Teacher Questionnaire suggest that teachers have a solid understanding of the "Nature of science."