Put People to the Test

Only after testing the volunteers did Gideon know who was best qualified to bring victory.

How often do we use foolish tests instead of wise ones?

Ex Post Grade-o

Have you ever heard of ex post facto?

This term describes a law which makes a person a criminal for committing an act before the legislation was passed.

It is typically prohibited by the US Constitution.

However, high-stakes testing frequently go through a technical, psychometric process called equating. Conceptually, this seems not only moral, but necessary.

When tests are high-stakes, the public deserves reassurance that exams from year to year will be of similar difficulty. Despite test-writers’ best efforts, some exam forms will be a bit more difficult than others.

However, in reality there is a more powerful force in play when No Child Left Behind testing is in place: the public’s perception of fairness and accountability.

If too many people pass, then the tests are too easy. If too many people fail, then the tests are too difficult.

Florida’s state board of education recently lowered cut scores because too many students failed the FCAT.

I’d like to call this an example of “Ex Post Grade-o.” The numbers didn’t tell the right story, so the state changed its standards so more kids could pass.

Pearson is in the midst of scoring and rating not only students but also teachers throughout New York State.

It would appear, on the surface, that they are highly qualified to make these decisions. Pearson employees publish in the area of psychometrics, or the science of fair testing and best practices.

But with just a little bit of reflection and an examination of the facts, it is clear that Pearson also plays the “Ex Post Grade-o” game.

Their goal is not to assure the public that tests are scored fairly from year to year.

Rather, their goal is to tell the story Governor Cuomo wants to tell the public about New York’s public schools.

We’ve seen this happen since 2006, when McGraw-Hill started writing and scaling the ELA and math exams in New York.

The New York Education Commissioner had a story to tell each year, and cut scores were adjusted “Ex Post Grade-o” to fit the commissioner’s script.

Results on these tests are determined not by an objective measure of relative difficulty from year to year, but rather by political needs and a convenient script.

Note that this post is being written on June 12, 2012. Here are my predictions for the next four years:

1) New York will have about a 75% overall pass percentage this year. Not too high, not too low.

2) Next year, for the 2013 round of grades 3-8 ELA and math testing, New York’s pass rates will fall significantly as the Common Core Standards are implemented for the first time and it’s OK for kids to struggle on the new tests.

3) Then, from 2014-2016, New York students will show steady growth, as the next presidential election cycle nears and Governor Cuomo demonstrates his effectiveness as an educational leader.

Bonus Prediction: The Board of Regents and the Governor’s office will not implement Common Core exams for Regents tests.

Regents exams need to be more stable because they represent credit-bearing instruments and can’t be manipulated for political purposes in the same way. Families sue if a high school exam is unfair. They don’t bother when an exam is just used to generate teacher evaluation scores and a four-point rating for each child.

Results on No Child Left Behind high-stakes tests are not an objective measure of reality. They are a political tool used for political purposes.

Ex-Post Grade-o is not going away anytime soon. Not until a bold, revolutionary leader can expose the divorce between valid psychometrics and what actually happens at Pearson, McGraw-Hill,and other big testing businesses.

I’d love to see state departments of education increase their levels of transparency and require testing companies to open their equating processes for public review.

What goes on behind closed doors is untrustworthy. States and testing companies say they are saving money by keeping test items secured. That’s a weak argument.

Call your state department of education and testing companies and hold them accountable! Ask them to publish details of their equating processes. Or better yet, include members of the public in these discussions.

Governor Cuomo, I call on you to include members of your blue-ribbon education panel to observe and report on Pearson’s equating and teacher growth formula creation. Add transparency to this process.

Addicted to Data

Politicians are always looking for simple solutions and quick fixes.

I believe the influence of Race to the Top and No Child Left Behind is being driven by the Hawthorne effect.

The idea is that if people know their performance is being measured, they’ll show a brief uptick in performance and productivity: “Whatever is measured will improve.”

The trouble is that once the process of measuring performance is concluded, there is a corresponding slump in performance and productivity.

Even worse, if the measures are unreliable, this can generate frustration and rebellion and even gaming the system.

I’d like to share a quick example.

Education Week reported yesterday that there were errors in the rankings of American High Schools in US News and World Report.

These rankings were based upon federal NCLB data. The same faulty data sets compiled to sanction schools and districts.

However, Thomas High School ranked #687 in the US, 87th in New York.

So despite the ranking’s questionable reliability, it is being celebrated.

Interestingly, Newsweek also ran a ratings system of its own, here.

As listed on the websterschools.org web site, Thomas and Schroeder High Schools are both nationally-ranked.

However, an important reason both schools are rated more highly in Newsweek is because only districts who returned a survey form are listed in the Newsweek rankings.

There was a significantly smaller sampling of high schools included in the Newsweek rankings.

I don’t deny the amazing work that Webster students, faculty, and staff are doing. But chasing after the latest ranking reinforces the importance of educational data, whether or not is it valid and reliable data.

Before celebrating your school’s ranking, dig into the methodology behind the statistics.

Appreciate your local schools, but appreciate them for the real things going on, not the artificial data points produced by media outlets and political institutions looking to sell more magazines and skim away tax dollars.

Pearson Damage Control

Pearson has already started its damage control machine. An excellent example of this effort can be found here: “Pineapplegate” Ignites Testing Debate.

By blaming New York State’s response to the testing debacle, Pearson deflects its own responsibility for implementing flawed exams.

It’s a predictable reaction by a bully. Blame the system. Blame the implementation. Blame the teachers. Blame the students.

Anything but taking responsibility for messing up.

Peter DeWitt is seeing red

Check out this brilliant analysis of the testing situation in New York here.

Red Background

Our background here at Schooleration.com is red because education is under attack.

Until this unhealthy emphasis on testing eases up, we will continue to stay red.

When kids are throwing up because they’re so nervous about their tests, we’re red.

When teachers are pressured to get test scores up, no matter the price, we’re red.

Cuomo, King, Pearson, and the Rotten Pineapple: An American Parable of Power

On this year’s 8th grade English Language Arts (ELA) New Youk State exam, there was a story about a pineapple.

That’s all I knew before reading the papers, because teachers are prohibited from discussing test items:

“To minimize the number of testing irregularities, principals should conduct a review of the test administration procedures prior to each test administration with all faculty and staff that will be involved in the test administration and scoring. In addition, to preserve the integrity of the test materials, advise all staff that they are not to discuss test questions or other specific test content with each other, with others online via e-mail or listserv, or through any other electronic means.” [Accessed 4/21/2012, on page 5 of the English Language Arts and Mathematics Tests School Administrator’s Manual 2012]

In other words, Pearson and the Department of Education collaborated to ensure that this pineapple story, and other controversial questions like it, would never see the light of day. Not only that, teachers were directed not to discuss the questions with each other, even casually.

Anyone who has ever taught or coached or done other forms of instruction knows that one of the best ways to become a better facilitator is to debrief on student responses to particular tasks and test items. But with secret test questions, engaging in this kind of professional development is criminalized!

Why would New York State criminalize professional development?

On March 20, 2012, Dr. John King announced that he was hiring “Tina Sciocchetti as Executive Director of Test Security and Educator Integrity.” [Accessed 4/21/2012 from the SED web site here.] Further down in this memo, we read, “In her new role, Sciocchetti will also be responsible for overseeing teacher and administrator discipline, including the Department’s enforcement of moral character regulatory provisions that are applied when certified educators are found to have engaged in misconduct, ranging from test integrity violations to inappropriate relationships with students” (emphasis added).

In other words, the person who leaked this exam to the press is being investigated right now and–without whistleblower protection–is in danger of losing his or her job.

This makes me more wary than ever about publicizing teacher results in newspapers and online for everyone to see and interpret. If these are the kinds of questions my students are expected to answer, how can I be confident that Pearson and the NYS Department of Education will be able to draw valid conclusions about my effectiveness as a teacher?

As I prepared to criticize Governor Cuomo for supporting the publication of teacher results in newspapers and online as recently as last month, I was shocked to learn that even he has changed his perspective on this issue.

Cuomo said, “I believe in the case of teachers, the parents’ right to know outweighs the teachers’ right to privacy,” the governor said. “After that, it’s less clear to me. And that’s why I think it warrants conversation.” He’s changed his view that teacher evaluations must be available for anyone and everyone to review and interpret. Now, he’s hedging his bets and leaving open the possibilty that he might be willing to give parents limited access to this data instead.

Anyone who has dealt with Governor Cuomo knows how shocking it is to hear him agree to a “conversation” about anything. He is an effective and ruthless negotiator, and he knows how to get his way.

Why would Cuomo, Arne Duncan, and Bill Gates all agree that it may not be best to make all teacher evaluations public?

Maybe it’s just that they’ve developed a compassionate, collective conscience. Maybe they can’t stomach the abuse leveled at teachers in LA and New York City after test scores were published.

But when there are tens of millions of dollars at stake, maybe there’s something else going on.

David Abrams was the director of assessment in New York for many years. He was released abruptly after releasing an “unauthorized memo” describing longer tests a few months ago. But these longer tests, 90 minutes each, for three days each in math and ELA, are now reality.

Why would Mr. Abrams, long-time director of testing in New York, be forced to resign for telling the truth? I suspect that he has access to information that many, many people in power don’t want going public.

I believe the preponderance of the evidence indicates that something at the (pineapple) root of our No Child Left Behind and Race to the Top’s testing system is rotten. If we can figure out the real reason that David Abrams was driven out, we’ll be closer to uncovering the source of the spoilage.