It was a great day of professional development and friendly school PR chat last week in Shippensburg at our annual Symposium! For those of you who could not attend or follow live on twitter, or would just like notes, we will summarize the presentations in the next few issues of e-Comm. The three presentations are also posted on the members’ only page of our website. As always, if you have trouble logging into this page, please contact Karen.
Our first presenter was Dr. Ed Fuller. Dr. Fuller is an Associate Professor at PSU and is a data expert; hIs topic was “Understanding and Using Education Research.” Dr. Fuller began by explaining that many educators are not well trained to identify good and bad research, and that simple research is always wrong.
"Chances are there is something significantly wrong with most research you see. That’s why you need a PhD to do good research, but you don’t need a PhD to understand research," Dr. Fuller explained.
Here is a quick checklist of common errors in research you can use to review research:
- Reliance on one study
- Reliance on non-peer reviewed studies (such as research from think tanks)
- Reliance on a study or studies that have unique samples/Ignoring the role of context
- Acceptance of studies that fit one’s existing world view
- Poor sampling and methods (Should be random sample)
- Low response rates on surveys (Look for 70% or higher)
- Look for selective entrance & disappearance
- Beware no disaggregation by subgroups
- Peer & selection effects play a huge role in school success
Dr. Fuller then went on to demonstrate examples of how data can be misinterpreted. We are all aware how many schools and districts use test scores to indicate student achievement. Yet, this scatter plot of statewide achievement in third grade shows the relationship between poverty & scores. The percent of students scoring proficient and advanced measures the percent of students in poverty and not much else. Poverty is the greatest indicator of achievement.
Dr. Fuller went on to discuss why scaled scores result in a more accurate representation of achievement. A scaled score is the total number of correct questions that have been converted onto a consistent and standardized scale. Change in percent proficient or advanced is not accurate. Scaled scores are currently not available from PDE, but Dr. Fuller is working with PDE to change this.
Next, Dr. Fuller discussed why using a test like the ACT or SAT, where not all students take the test, should not be used to compare schools or districts. The only way to compare states accurately is by using NAEP scores. NAEP testing is randomly sampled to be accurate research and as a result can compare achievement across districts.
Dr. Fuller reviewed our state scores here in PA too. While SPP scores do have a great deal of importance, all these scores actually tell you is what kinds of kids are demographically in your district. He demonstrated how one district ranked in the top ten by scores ended up in the bottom 5 when re-sorted by demographics. (Subgroups)
Another interesting point is in regards to what is on standardized tests. We all know the tests are quite long, but did you know they are actually not nearly long enough to provides an accurate result? For example, there are 3 questions on measurement on the math test, but you can't make a valid conclusion on whether kids know measurement with 3 questions. You would need about 20!
Recent education research has also demonstrated with an accurate study of the progress of cyber students that the students made absolutely no progress over the school year. It was as if the students did not go to school at all that year.
Dr. Fuller recommends the book "Measuring Up" by Daniel Koret for all you ever want to know on evaluating research for accuracy. Also, he invited us to send research to him to check if you see research that doesn't seem right to you. He can review it and tell you if it is accurate or not. Email him here
Increasing our understanding of accurate and valuable education research can help us better tell the story of public education to our audiences. Follow Dr. Fuller on Twitter at @EdFuller_PSU.