Does Sustained Science Content Based Professional Development Make a Difference to Students' Science Achievement?

Authors: Kathryn Scantlebury, Yue Li, Jane Butler Kahle, Constance Blasie

MISEP Teachers' Science Achievement

Table 1 shows the results of the middle school teachers' pre and final mean scores on the content test. There was a highly significant increase in the teachers' scores after, compared to before, they completed the project. A Rasch measure of 0 indicates a respondent answered all questions of this test incorrectly, while a measure of 100 indicates a respondent answered all questions correctly. Teachers' Rasch mean scores were calculated using 53 items. Reliability for the middle school teacher science test is 0.69.

Table 1. Rasch Mean Scores of Cohort A MISEP Teachers' Content Tests: Pre (Spring 2005) and Final (Fall 2007)

Year	n	Mean	SD	t-value	df	p-value
Pre (2005)	21	43.96	5.76	-12.80	20	< 0.001
Final (2007)	21	58.24	4.75

Science Achievement Middle School Students - The Discovery Inquiry Test (DIT)
The DIT was constructed using public-released items from the 1990 and 1992 National Assessment of Educational Progress (NAEP) science tests. The test measures a student's ability to analyze and interpret data, extrapolate from one situation to another, and utilize conceptual understanding. The test includes 29 items across four science content areas (Kahle, Meece, & Scantlebury, 2000).  It has been used with over 20,000 middle school students.  In this case, the DIT had a Cronbach alpha value of 0.66. Rasch scores were computed on student scores. Table 2 shows the students' mean scores for the first cohort of middle school science teachers.

Table 2. Comparison of DIT Rasch Mean Scores by Baseline (Pre), Post Year 1, and Post Year 2 for Cohort A

Group	n	Mean	SD	F-value	df	p-value
Year 2005 (Pre)	555	54.93	12.10	12.19	2, 1142	< 0.001
Year 2006 (Post1)	327	56.47	10.62
Year 2007 (Post2)	263	59.14 *	10.75

*Mean score for year 2007 is significantly higher than mean scores for year 2005 and year 2006.

Multiple regression analyses were conducted to examine whether gender, student group, and/or Student Questionnaire subscale mean scores predicted achievement test scores for 5th, 6th, 7th, or 8th grade students of MISEP teachers. These results (standardized coefficient estimates) are shown in Table 3.

Table 3. Multiple Regression Standard Coefficients for the DIT by Grade Level: Cohort A, Post Year 2, Spring 2007

	5th	6th	7th	8th
	n = 33	n =75	n = 73	n = 73
Female	0.01	0.17	0.05	-0.21
White	-0.04	0.31 **	0.43 ***	0.06
What I do in class	-0.20	0.06	0.26	0.10
What my teacher does in class	0.07	0.30 *	0.02	-0.11
What my friends do	-0.04	-0.16	-0.29	0.00
What parents/adults do	0.19	0.16	-0.18	-0.15
My attitude towards science	0.75 ***	-0.11	0.28 *	0.23
Adj R2	0.50	0.30	0.19	0.07
* p < 0.05, ** p < 0.01, *** p < 0.001 Estimates presented are standardized coefficient estimates (bi).

The combination of the variables explained approximately 50% of the test variance for 5th graders, 30% for 6th graders, 19% for 7th graders, and 7% for 8th graders. For students in the 8th grade, none of the demographic variables (gender and student group) nor behavioral and attitudinal variables were statistically significant predictors of test scores. For 5th and 7th graders, their subscale score "Attitudes towards science" was a significant predictor of achievement. This finding is supported by other research (Kahle, Meece, & Scantlebury, 2000) and is particularly evident among middle school students.

MCEP Teacher Chemistry Achievement

In 2005, Cohort A MCEP teachers completed the high school chemistry teacher test, composed of 62 items with a reliability of 0.85. Table 4 shows the pre/post Rasch mean scores for MCEP chemistry test. There was a highly significant increase in the teachers' chemistry content knowledge after, compared to before, they completed the Penn STI project.

Table 4. Rasch Mean Scores of Cohort A MCEP Teachers' Content Tests: Pre (Spring 2005) and Final (Fall 2007)

Year	n	Mean	SD	t-value	df	p-value
Pre (2005)	17	45.16	6.17	-7.67	16	< 0.001
Final (2007)	17	56.55	7.07

High School Chemistry Conceptual Content Test (HSCCCT) - Cohort A Students

Using t-tests, comparisons were made on Rasch mean scores for the achievement test by gender and by student group for Cohort A high school chemistry students, as shown in Table 5. Comparisons were made between boys and girls as well as between White and Non-White students. Significant differences were found by gender, favoring boys. There were no significant differences between White and Non-White students.

Table 5. Comparisons of HSCCCT Rasch Mean Scores by Gender and by Student Group: Cohort A, Post Year 2, Spring 2007

Group	n	Mean	SD	t-value	df	p-value
Female	76	40.94	10.21	-2.30	152.00	0.023
Male	78	45.07	12.00
Non-White	64	42.72	14.88	-0.40	89.63	0.689
White	91	43.54	8.16

Multiple regression analyses were conducted to examine whether gender, student group, or Student Questionnaire Rasch mean subscale scores predicted achievement scores for 10th and 11th grade students of Cohort A high school chemistry teachers. Regression results (standardized coefficient estimates) are displayed in Table 6.

Table 6. Multiple Regression Standard Coefficients for the HSCCCT by Grade Level: Cohort A, Post Year 2, Spring 2007

	10th n = 63	11th n = 82
Female	0.23 *	0.16
White	-0.01	0.15
What I do in class	-0.17	0.06
What my teacher does in class	-0.07	0.04
What my friends do	0.22	-0.07
What parents/adults do	-0.39 **	-0.18 *
My attitude towards science	0.46 **	0.68 ***
Adj R2	0.31	0.42
* p < 0.05, ** p < 0.01, *** p < 0.001 Estimates presented are standardized coefficient estimates (bi).

The combination of the variables explained approximately 31% of the variance for 10th graders and 42% of the variance for 11th graders. For students in the 10th grade, gender and attitude were significant predictors of test scores. The other significant predictor, "What parents/adults do" was a negative, indicating that the more help a student received the lower his/her test score was. With the exception of gender, similar predictors were found for 11th graders.