Greater Greater Education

DCPS gets all its teacher evaluators on the same page

In an effort to build confidence in its teacher evaluation system, DCPS has launched a program that uses videos to ensure that its various evaluators are scoring teachers consistently.


Photo by David Stroble on Flickr

As part of the IMPACT evaluation system introduced by former Chancellor Michelle Rhee, teachers are observed in their classrooms up to 5 times a year. Sometimes a principal or assistant principal observes the teachers, while other times it's one of a team of outside evaluators called Master Educators.

As detailed in the guidebooks that explain the evaluation system, a teacher's classroom performance is measured against the district's "Teaching and Learning Framework," and observation scores can make up as much as 75% of a teacher's overall IMPACT score. (For some teachers it's 40%; the remainder of the score is largely determined by student test scores, which is a whole other source of controversy.)

A teacher's IMPACT score has significant consequences. teachers rated "highly effective" are eligible for bonuses and salary increases, while those at the opposite end of the spectrum can lose their jobs. And teachers have had complaints about the IMPACT system since its inception.

One problem has been that two evaluators observing the same lesson may come up with two significantly divergent scores. For the past year and a half, DCPS has been working to rectify that situation through a project called Align, funded by a $1.5 million grant from the Bill and Melinda Gates Foundation. (In a recent Washington Post op-ed, Bill Gates argued that an evaluation system that uses multiple measures, including classroom observations, is fairer than a system that relies solely on students' test scores.)

Classroom teachers received cameras and microphones and tips on how to film themselves teaching. Then, groups of Master Educators and principals watched those videos together, discussed their reactions, and built consensus on "anchor ratings."

It's been a long and painstaking process. According to Michelle Hudacsko, Deputy Chief of IMPACT, it takes many hours of filming to produce one usable video. And getting multiple viewers to agree on ratings across the 9 different "teaching standards" in the Teaching and Learning Framework rubric can take many more hours.

Among the 9 teaching standards are "Provide Students Multiple Ways to Move Toward Mastery" and "Develop Higher-Level Understanding Through Effective Questioning." Although the criteria for evaluating a teacher's performance against the standards are spelled out in detail, there's still a lot of room for subjectivity.

In the process of aligning their scores with one another, evaluators say they begin to recognize the different approaches and orientations each of them bring to their observations. The idea is that once evaluators are aware of where they tend to diverge from the norm, they'll be better able to control for their own particular biases.

Now that the anchor ratings are in place, DCPS will soon unveil an online platform that it can use to train other evaluators. An evaluator will be able to watch a 3- to 4-minute video clip that illustrates a particular teaching standard, come up with a score, and then compare it to the anchor score. DCPS will then provide in-person training for evaluators whose scores in certain areas vary significantly from the anchor ratings. The goal is to have all Master Educators and principals trained in the Align system by the beginning of the next school year.

DCPS hopes that standardizing evaluations will lead teachers to place more trust in the evaluation process. "Observations used to be a threatening thing in my building," says Tenia Pritchard, the principal at Whittier Education Campus, who was a pilot participant in Align training. "Now the teachers know we're using the framework to help them improve their instruction."

Under the new system the emphasis in post-evaluation conferences will shift from providing a score to providing feedback based on evidence, says Eric Reese of the DC Public Education Fund, an organization that funnels private philanthropic funds to DCPS. "The big change is that we're using the master educators as coaches," he says, "especially in the lowest-performing 40 schools."

It remains to be seen how many DCPS teachers will embrace IMPACT even after the Align system is in place. Teachers have had other complaints about observations aside from the lack of consensus in scoring. For example, while one "informal" observation is announced in advance, the others (the ones that count) are unannounced. The surprise nature of those visits can be a source of stress. And some teachers feel under pressure to demonstrate proficiency across all 9 teaching standards in the space of 30 minutes.

Beyond that, there's a basic question about whether it's fair to compare a teacher who has a class of high-performing students at School Without Walls or Wilson against a teacher at a non-selective neighborhood school in Ward 7 or 8. If a class has a high proportion of students with learning disabilities or behavior problems, some teachers say, the teacher is far more likely to be rated poorly.

But since DC adopted IMPACT, teaching quality (at least as measured by the system itself) has gone up: according to DCPS, the proportion of "minimally effective" teachers whose rating improved went up to 73% last year, as compared to 58% the year before.

IMPACT isn't perfect, but some kind of systematic evaluation system for teachers is necessary. We can't go back to the old, pre-Rhee days when principals had a lot of leeway in doing evaluations and nearly all teachers got high ratings, even though student performance was dismal. It may be impossible to design a comprehensive evaluation system that pleases everyone, but bringing consistency to evaluators' ratings is certainly a step in the right direction.

Natalie Wexler is the editor of Greater Greater Education and a member of the boards of DC Scholars Public Charter School and The Writing Revolution, an organization that promotes the teaching of analytical writing. She has been a lawyer, a historian, and a journalist, and is the author of three novels. 

Comments

Add a comment »

Hopefully this will lead to a better understanding of where teachers are at skill-wise and result in more effective feedback for them in their classrooms. This seems like a much better way of evaluating them strictly (or majority) by test scores.

by Jessica Christy on Apr 8, 2013 10:21 am • linkreport

Yes, on first approach this certainly seems a bit complex. But it is also an attempt to address common concerns about teacher evaluation rubrics. I wish something like this was in place when my daughter had consecutive years in elementary grades with INEFFECTIVE teachers. Everyone is so concerned about giving second changes, support and professional development to improve effectiveness of teachers who are rated low in the system. What about the KIDS who only have ONE CHANCE in second and third grades? Don't they deserve EFFECTIVE teaching??

by Tom M on Apr 8, 2013 3:09 pm • linkreport

@Tom:

I was in the same position last year. Our kid was in a class with a teacher who every parent in the school tried to avoid, every teacher knew was completely ineffective and bordering on cruel, and who the principal had been trying to get rid of for years. Of course, that didn't happen because, whatever the impact on year after year of kids, the important thing was to make sure this teacher didn't have to find another line of work.

Whatever damage anti-union zealots on the far right may have done to the reputation of teacher's unions, it pales in comparison to the damage a bad, infireable teacher can do. On the other hand, most parents I know think that the teachers who are effective and hard-working deserve whatever we can give them.

by oboe on Apr 8, 2013 3:20 pm • linkreport

Another bias in the IMPACT observation evaluation is that the observations are not standardized. That is, Teacher A may be observed on the first Monday after winter vacation, Teacher B late in the afternoon the Wed before Thanksgiving and Teacher C mid-morning any typical Wed at least a month into the school year. Time of day has an impact too; mid-morning on a typical Tuesday vs the last hour before winter break.

Day of the week and whether or not a vacation has just completed/is imminent, and whether or not there are other disruptions in the teaching flow (e.g. the cafeteria lady had a heart attack at work that morning) are all variables that affect the conditions under which a teacher is evaluated. Even 'highly effective' fantastic teachers have days that are better than others.

Day of the week including separation from vacations and time of day are elements in data collection that could be standardized to eliminate some bias in the observations.

Scheduling observations purposefully at more difficult times/days could be useful too. However that data collection can not take place without acknowledgement that there is bias in the times/days that observations are currently scheduled.

by Tina on Apr 8, 2013 6:56 pm • linkreport

@oboe- thats an example of a bad principal! The only reason I can think of for not removing that teacher was that there was no replacement. A teacher one was rated 'ineffective' can lose job w/o notice -its something teachers live with that causes some stress -I don't understand why that one was kept on. The contract w/union does not support a teacher who is evaluated 'ineffective". Any teacher receiving that grade can be sent packing on very short notice.

by Tina on Apr 8, 2013 7:02 pm • linkreport

@oboe-also-if the principal knew the teacher was that bad-even cruel -why wasn't he/she in that classroom everyday to make sure that the students knew they would be supported if they had a complaint about the teacher and to tell the teacher where the boundary was? IMO thats what a great principal would have done. The other day you wrote about how much power the principal had and said it was a good thing -here your write about a principal that seemingly has little power (ineffective IMO) yet you do not criticize the principal. Why? To me this seems like a contradiction.

by Tina on Apr 8, 2013 7:09 pm • linkreport

@Tina,

Prior to IMPACT, it was extremely difficult to get rid of ineffective teachers. So the opportunity to get rid of this teacher only really arose in the last couple of years. This principal has been at the school for 3-4 years, and has been building out the staff. Rather than come in and purge staff, she had been (and continues to be) working closely with the few teachers who weren't up to snuff. The teacher in question took early retirement this year. (She'd been at the school for a long time, and was quite likable outside of the classroom--just in the Wong profession).

Bottom line is, any system of accountability has minor flaws, but if a reasonable system of evaluation had been in place for the last ten years, there wouldn't be so much drama and heartache going on now.

by oboe on Apr 9, 2013 7:47 am • linkreport

@Tina

"The other day you wrote about how much power the principal had and said it was a good thing -here your write about a principal that seemingly has little power (ineffective IMO) yet you do not criticize the principal. Why?"

I wasn't saying that principals have a lot of power over staffing decisions, but rather arguing that they *should* have that power since they are ultimately accountable. The reforms of the last few years have to some extent changed the dynamic. For whatever tweaks need to be made to IMPACT, I think that's an improvement over the previous way of doing things, and don't want to see us go backwards.

by oboe on Apr 9, 2013 7:55 am • linkreport

@Tina
Another bias in the IMPACT observation evaluation is that the observations are not standardized. That is, Teacher A may be observed on the first Monday after winter vacation, Teacher B late in the afternoon the Wed before Thanksgiving and Teacher C mid-morning any typical Wed at least a month into the school year. Time of day has an impact too; mid-morning on a typical Tuesday vs the last hour before winter break.

Except there are several evaluations throughout the year (6?) and they are scheduled for random times. If every teacher has a random sample of 6 then do the observations need to be standardized by time of day? And how would you even do that?

Beyond that, I question the assumption behind this criticism, which is that there are good teachers out there who are being rated ineffective and fired. I don't think that's the case.

if the principal knew the teacher was that bad-even cruel -why wasn't he/she in that classroom everyday to make sure that the students knew they would be supported if they had a complaint about the teacher and to tell the teacher where the boundary was? IMO thats what a great principal would have done.

While this teacher may have needed important focus, surely you're aware that principals have many job duties beyond babysitting teachers.

by MLD on Apr 9, 2013 8:23 am • linkreport

@oboe-I agree its better (the system compared to before), but I see lots of places where it can be improved.

@MLD - surely you're aware that principals have many job duties beyond babysitting teachers.

1) the principals first and only priority is protecting the kids and seeing to their education. You think the principal should be in his/her office pushing papers around? A good principal is hardly ever in his/her office; in a school w/ a good principal all the students know the principal and s/he them, by name. Your perspective is amiss if you think its the teacher I think whom needs babysitting in the example above. Its the students who need the principal. And in my view its the principals job to step in when a teacher isn't doing it. its not for the teachers benefit, its for the kids.

Also, in the text of mine that you quoted and then commented, I question the assumption behind this criticism, which is that there are good teachers out there who are being rated ineffective and fired.
--that assumption is not in that paragraph. I expressed strictly what I think a good principal does in a situation like that.

Regarding IMPACT and whether or not good teachers get rated "ineffective", the evaluation based on test scores is designed to be relative so that its always the case that 50% of the teachers will be rated "ineffective".

do the observations need to be standardized by time of day? And how would you even do that?

Currently its 5 observations 3 from the principal. Here's how it can be standardized
'3-4 observations are made mid-morning mid-week (Tu/W/Th) on weeks that are not following or preceding a vacation/student day off. 1-2 observations are made during known "challenge" days; the first day back after a vacation/the day before a vacation/other'. Its not that difficult.

If the system is going to aim toward objective measure then they need to do the basic thing to get as close to objective as possible. Day of week/time of day/season of year are standard variables used for all kinds of data collection that show patterns. Its basic! Anyone with a psych or marketing 101 class learns this. And 5 observations is not enough of a data set to see any kind of pattern.

I agree this evaluation system may be better then before and may be better than nothing but its far from good.

by Tina on Apr 9, 2013 10:55 am • linkreport

I agree this evaluation system may be better then before and may be better than nothing but its far from good.

Fair enough. Let's hope it continues to improve!

by oboe on Apr 9, 2013 11:51 am • linkreport

I would love to know if teachers will have access to these videos. It would only make sense for teachers to also be in the know for what is considered effective, highly effective, ect...

by liz on Apr 9, 2013 12:24 pm • linkreport

Liz -- teachers may not have access to the videos used by the evaluators, but they do have access to another set of videos designed to help them figure out what effective teaching looks like (these videos have also been funded by the Gates Foundation). You can see a description of them here: http://dcps.dc.gov/DCPS/realitypd

by Natalie on Apr 9, 2013 7:30 pm • linkreport

It would be optimal if the teachers could see the video of themselves that got graded so they can see what they themselves did to earn the grade they got, in the same way a gymnast, or any athlete, views video of him/herself. Its great to see someone else executing perfectly, but the learning is in seeing what you yourself did. People often make mistakes they aren't aware they are making and its much easier to correct it if you see yourself doing it rather than having someone tell you "you turned you back to the classroom" or whatever.

by Tina on Apr 10, 2013 10:45 am • linkreport

Tina, I see what you mean. The teachers who videotaped themselves for purposes of developing the anchor scores did get a chance to see those videos themselves. But these videos were really used for purposes of standardizing the evaluations. But just to be clear, I'm not sure the scores became part of the videotaped teachers' actual IMPACT scores. Those observations are still being done live rather than via video.

by Natalie on Apr 10, 2013 1:47 pm • linkreport

@Natalie, thanks

by Tina on Apr 10, 2013 2:51 pm • linkreport

IMPACTplus was put in place to retain the best teachers in the district. I'm a Title 1 ECE teacher (and a WTU member) who has been rated highly-effective through IMPCATplus evaluations two out of my three years with DCPS. My school was recently closed and I was excessed. So not only did I lose my job, I no longer qualify for the IMPACTplus bonus that I earned through my hard work in the classroom with some of the city's most at-risk youth. And DCPS let go of the "type" of teacher they try so desperately hard to hold on to. It's a bit ironic. This excerpt was taken directly from DCPS's website:

"We feel it is essential to demonstrate – in the boldest way possible – how much we value the work teachers do. We are proud that outstanding DCPS educators are now being paid what they deserve. We recognize that teachers do not go into the profession for the money. But at DCPS, we believe that great teachers deserve to be compensated like the professionals they are.

Any WTU member who earns an IMPACT rating of Highly Effective is eligible for IMPACTplus. IMPACTplus has two parts: an annual bonus after one year of being rated Highly Effective and an increase in base salary after two consecutive years of being rated Highly Effective."

I'm heartbroken by DCPS.

by annon on Jul 15, 2013 4:11 pm • linkreport

Add a Comment

Name: (will be displayed on the comments page)

Email: (must be your real address, but will be kept private)

URL: (optional, will be displayed)

Your comment:

By submitting a comment, you agree to abide by our comment policy.
Notify me of followup comments via email. (You can also subscribe without commenting.)
Save my name and email address on this computer so I don't have to enter it next time, and so I don't have to answer the anti-spam map challenge question in the future.

or

Support Us