On The Value Of Testing

One of my former students contacted me via LinkedIn last night. Hearing from former students is one of the great joys in my life. I guess the first part of that is ego – I am thrilled to death that years after leaving high school they even remember me. Then to have them want to make some sort of contact and tell me what they are doing just feels great. Looking at this student’s resume I was struck by his success. Oh sure he was obviously a bright student but clearly he’s developed and grown since high school. He wasn’t the greatest test taker in the world though. In fact many of my best and brightest students were not great test takers.

Well perhaps that is too general a statement. It depends on the nature of the test. For example when I look back on the students I have stayed in touch with (ok admittedly this is not a scientific sample/study) I see a lot of great success in the computer science field on the part of students who did not even pass the Advanced Placement Computer Science exam. The students who did get 4s and 5s on the exam have also done well. So what does the APCS exam tell me about my students future success? Nothing.

I gave tests and quizzes and projects to all my students. My philosophy though was that the purposes of the tests and quizzes was to gauge how much of what I was teaching the students were learning. How well were we (the students and I) working together to get them the knowledge they needed. Projects were as much, no they were more, learning exercises than evaluative tools. Tests and quizzes take valuable learning time so for me they have to be directly useful. They have to be navigation instruments and help us to make course corrections to help us all reach the end goals. High stakes, standardized tests are fairly useless for this purpose unless teachers have enough detailed information about the results to change what they are doing. One does not get that from the APCS exam – or any AP exam for that matter.

So what the AP exams do is sort of a pass/fail for those involved but it has no diagnostic value. I used to ask my students what was on the exam (in general terms since they are not allowed to share specific questions). I asked what did I cover too much or too little. The balance changed from year to year though so this was helpful but only minimally so. I don’t feel like I ever really taught the course the “right way” to make sure all the students earned their 4s and 5s. Though honestly that seemed to have bothered the guidance department more than it did my students. Those who went on to college and or careers in computer science seemed to do well. And I haven’t heard much back saying “you taught me poorly and I played catch up in college.” Now there may well be some students who feel that way but they haven’t come back and yelled at me. (Which they are certainly entitled to do if they feel that way.)

I’ve talked to a number of teachers who would rather not teach an APCS course. They’d rather teach an advanced CS course their own way with their own priorities. I actually asked my headmaster about that when I was teaching and he told me “no” in no uncertain terms. The value of the AP exam to students is in college admissions and for a college prep high school to not offer as many AP courses as possible is pretty much a requirement.

Now I completely buy the value of teaching a college level course to students who are ready for it. And aside from articulation agreements with local colleges the AP exam is the single most accepted way to show that one is doing that. There is value in AP curriculum in many subjects. But is the test itself useful? That’s less clear to me.

I have many friends who have served on the AP CS test development committee over the years. I’ve met and got along well with the Chief Readers for the last 10+ years. Besides knowing many people who have helped grade the APCS exam I helped grade it myself. So I believe that the APCS exam is in many ways a great exam. It is graded as fairly and professionally as anything I have ever seen. I learned a lot from being an AP Reader (I need to teach a course like the AP course somewhere so I can get them to let me read again – it was that valuable to me.) But for the teachers who are not readers and for the teachers who worry about the multiple choice questions I’m not sure they get a lot of value from their students taking the exam. And there is that nagging problem of “teaching to the test” that gets to some of us.

I’ll leave you with one more thought. Real life is an open book test. I strongly believe that. It is one of the great lessons I have learned in my life. Some people never do well on the “read and regurgitate” sort of test that makes up so much of standardized testing. It is just not the way their minds work. They learn well. They know how to find things out. They are willing to work hard to find a way. They’re just not test takers. On the other hand the kids who do well on standardized tests so what? If the real world is really an open book tests how do standardized closed book tests reflect how the test takers will do in real life?

[Note: a related post by Doug Holton is at Problems with Test Prep, Related to "Disaggregating Education"? ]