Disjointed thoughts on test construction

I'm giving the first test in my new "Honors Physics I" course, the course that's intended to foreshadow the future AP Physics I.  I've also been helping to write and prepare tests in conceptual and general (Regents) level physics.  Thus, I've been reflecting a bunch lately on methods of test construction.

Lyle Roelofs, who (perhaps tied with Walter Smith) was simply the best teaching physics professor ever, emphasized repeatedly to anyone who might teach physics: "The only time a teacher can be sure of a student's full attention is on a test.  So use tests to your advantage."  Thus the origin of my test corrections, the test-question-writing exercise, serious exam review,  and more.  But, if I expect my students to take the tests seriously as study tools, I have to take serious care in the construction of the test.

That care starts with a professional-looking test.  There's nothing wrong with handing out a nightly problem set via a sloppy email or via a handwritten slip of paper.  Practice multiple choice problems sometimes consist of faded xeroxes from 30-year-old master copies.  No problem, 'cause no one is expecting every night's problems to be beautiful.  However, a TEST should include clean, nicely-formatted proofread copies.  Mistakes should me not just minimized, but eliminated -- how do you justify docking a student's test grade for a lack of units when you yourself misprinted the units of acceleration?  Sure, stuff happens, but if more than one test in a year contains a major typo or a substantive error, you need to proofread better.

The format of each test, I think, should be generally consistent throughout the year.  Students are taught not to read directions on the SAT.  Why?  Because the directions for each section are the same on every test, every year; and because these directions are available ahead of time for preparation purposes.  A physics test is supposed to be a measure of a student's content knowledge.  Sure, careful reading of individual problems is essential to a student's successful performance... but we shouldn't surprise anyone with a different kind of question than they're used to.  

In an AP class, I give tests in a format identical to the AP exam:  Multiple choice, followed by free response.  In my general, honors, and conceptual classes, the format is always free response, short answer, multiple choice.  I hand out the instruction sheet and any reference tables before the test, so that students know what to expect.  

In Honors Physics I have control over test design, since we're not yet formally teaching to an AP test.  So I combine free response, short answer, and multiple choice into a single time period:  2 hours for the end-of-year cumulative, national exam, and 80 minutes for the monthly in-class tests.  The rule of thumb for timing:  about a minute and a half per mulitple choice item, about three minutes per short answer item.  For AP-style free response, give a bit longer than one minute per point -- for example, in an hour I expect students to be able to solve five 10-point problems, or two 15-pointers and two 10-pointers.  (As a comparison, the AP Physics B exam allows 90 minutes for 80 points of free response; the AP Physics C exams allow 45 minutes for 45 points of free response.)

If you're not teaching AP, consider switching the traditional order of the test.  I put the free response questions at the beginning of the test, and multiple choice at the end.  Why?  Because I've too often seen students get captivated by a one-point multiple choice question, leaving no time even to make a reasonable guess at the 15-point free response question.  If, on the other hand, someone gets hung up on a free response question, there might be time to make reasonable guesses at the multiple choice questions at the end.

The content of each test should be transparent, even if that means "everything we've ever covered."  I thoroughly approve of cumulative tests; why should I bother teaching in September if everyone's allowed to forget what we learned?  But I also approve of a clear course outline, indicating the general topics that have been covered in class and that will show up on the test.  A cumulative test is not a licence to play "gotcha!"  If you're consistent all year in what you expect students to understand in each unit, and if every test includes something from each previous unit, the class will recognize and meet your expectation that they learn physics for the long term.  The nice side effect is that final exam preparation becomes a piece of cake if all tests are cumulative.

Someone stopped me in the hall yesterday after the first test, and said, "Mr. Jacobs!  That was like an EXAM, not just a test!"  I smiled at him... imagine how seriously he'll take my actual trimester exam, now that he knows what my monthly tests are like.  And imagine how comfortable he'll be in May on a cumulative, year-long, national exam like the AP or the SAT II.

GCJ

The OJ Simpson Question: Magnitude and Direction of Accleration

Knowing what an acceleration vector means about motion is perhaps the biggest conceptual challenge in first semester physics.  No matter how many times I say "the direction of motion has nothing to do with the direction of acceleration," this misconception (among many others) remains.

I ask students to memorize:  

* Speeding up means acceleration is in the direction of motion.

* Slowing down means acceleration in the opposite direction of motion.

However, put these facts in the context of a velocity-time graph, or in the context of specific motion north and south, and heads explode.  And that's really all we can do -- ask about the meaning of acceleration in as many different contexts as possible until the class is sick of such questions.

In the first several kinematics assignments, I've displayed a position-time or velocity-time graph and asked for a description, in everyday language, of the represented motion.  In order to tease out the physical meaning of an acceleration vector, I switch up:  I present a description of motion, and ask students to make a velocity-time graph.

1.    In an alternate universe that still obeys our laws of physics, O.J. Simpson leaves a tollbooth in his white Bronco the morning after killing his wife.  Soon after, he sees a police officer flash his lights.  Hoping to get away, he slams the gas pedal to the floor, but then O.J. hits a concrete barrier and crashes.  (a) On the axes below, sketch a velocity-time graph of OJ’s motion.[1] 

---

[1] Sketch, according to the College Board’s course description, means to “draw a graph that illustrates key trends in a particular relationship, such as slope, curvature, intercept(s), or asymptote(s).  Numerical scaling or specific data points are not required in a sketch.”

Most of the class gets this essentially right on the first attempt; the rest get it after a quick conversation with a friend.  The real point of the problem comes next:

2. Describe in words the magnitude[1] and direction of O.J.’s acceleration as O.J. is leaving the tollbooth.

3. Describe in words the magnitude and direction of O.J.’s acceleration as O.J. is traveling along the road unmolested.

4. Describe in words the magnitude and direction of O.J.’s acceleration just after O.J. sees the officer.

5. Describe in words the magnitude and direction of O.J.’s acceleration while the Bronco slams into the wall.

---

[1] “Magnitude” in this context means, how much acceleration does OJ have?  Answer relative to his acceleration at other parts of his motion.  No numbers are required, though you are welcome to make calculations if you so desire.

I can tell almost immediately upon reading these responses who understands acceleration, and who does not.  The ones who truly don't get it come in for a consultation, where we work on these concepts.  How can I tell?

Well, the response I'm expecting to parts (c) and (d) refers explicitly to the v-t graph and/or to definitions that we've learned:  "(c) When OJ sees the officer, the slope of the v-t graph is positive (a frontslash), so the acceleration is forward.  The slope of the v-t graph is steeper than when OJ calmly sped up from the tollbooth, so the acceleration has a larger magnitude here.  (d) When OJ crashes, the slope of the v-t graph is much steeper than anywhere else; so the magnitude of the acceleration is highest of all parts of the motion.  OJ's acceleration is backwards, because the car is still moving forward, but is slowing down."


The most common mistake is to state the direction of MOTION rather than of acceleration:  "(c) When OJ sees the officer, he speeds up rapidly.  So his acceleration is moving forward.  (d) When OJ crashes, he bounces back off the wall, so his acceleration is moving backward."  Anytime a student says that the acceleration is "moving," I know that he is conflating acceleration and velocity, so the answer is marked wrong -- yes, verbal skills are part of physics.


A less common mistake is to think that the acceleration must change if velocity changes.  "(c) After OJ sees the officer, his acceleration must change rapidly, because the accelerator pedal is on the floor.  (d) When he crashes, OJ's acceleration changes from a high value to zero."  No, constant acceleration means speeding up or slowing down; this student thinks acceleration must change in order for speed to change.


I do get about half the class writing clear, concise, and specific explanations that refer to the v-t graph or to the definition of acceleration.  I will show a fellow student's good explanation to someone who's struggling, to show the difference in the style of prose.  I'm teaching writing as much as I'm teaching physics, sure.  But the time I spend now demanding clear writing pays off tremendously later in the year, when an AP-style free response test requires one-minute justifications.


GCJ


P.S. Only about half of my class had ever heard of OJ Simpson.  That says something about pop culture in the post-internet era.  What it says, I have no idea.

Assignment: Write your OWN dang test.

Writing a physics (open response) test question is not in any way an easy task.  It's hard enough to think of a new situation that provides interesting physics, and figure out solvable questions regarding that situation.  It's even harder to write the question so that it's solvable in 10-15 minutes and clear, with no possibility of misinterpretation.

Students get so caught up in the solving of these problems that they fail to appreciate the elegance of their design.  All they see are tough problems to which they have to get the right answers.  It doesn't necessarily occur that, while the details and values are different every time, each problem fits a narrow range of overall themes.  Johnny might be fully capable of dealing with an inelastic collision problem; he might be downright good at projectile problems.  Yet, put him under the pressure of a test, give him an inelastic collision that occurs on the end of a cliff, and ask him how far from the cliff the objects land?  Johnny is likely to be overwhelmed, and freeze like a Microsoft OS with more than one Window open.

One of the many ways I try to debug Johnny is the activity I'm finishing up this week in my general physics course:  the Student-Written Test.  Each student spends lots of time in and out of class writing 4-5 carefully prepared problems for possible use on a test.  I collect these problems, grade them for correctness of solution and quality of presentation, and give them all back.  A day or two later, I give the class a test consisting of three authenticly student-written problems.  (Sure, I adjust wording for clarity, but I use these problems as much verbatim as is reasonable.)

By the end of this one-two week process, my class has developed a serious appreciation for what it takes to write a physics problem... but more importantly, by writing problems of their own, they've become more comfortable dealing with extended multi-topic problems.

Below is the assignment as I handed it out last year, complete with a detailed schedule.  Feel free to use this... I find that I don't usually have time for this sort of thing in AP physics, but the general class can afford to slow the pace.  Besides, what better way to review the topics from the first four months of school?

Problem-writing assignment – instructions and schedule

Your assignment is to write 5 test-quality physics problems, using the concepts we’ve studied so far this year:

Kinematics

Motion graphs

Kinematics in 1-d

Kinematics in 2-d (includes projectile motion)

Newton’s second law

1-D

Newton’s second law in 2-d (includes inclined planes)

Momentum

Impulse-momentum theorem

Law of conservation of momentum

Energy

Definition of work

Work-energy theorem

You don’t have to use all the concepts, but those are all available to you.

A test-quality physics problem is one that involves more than a simple plug-in to an equation. Often, more than one of the above concepts will be required to solve the problem. Looking over past test problems should give you a good idea of the level of complexity you are going for.Writing a test-quality problem is not a simple exercise. As you may have noticed, test problems involve more than simply plugging into an equation. Yet, a test problem must be solvable in 10-15 minutes, so you can’t ask anything horrendously complicated. Finding the balance between solvability and complexity, and meanwhile making every problem interesting, is your challenge.

The structure of a problem will usually follow a standard form. First, you’ll draw and describe a physical situation; then, you’ll ask several different questions about that situation in parts (a), (b), etc. To determine whether your problem has the proper scope and level of difficulty, consider how many of the above physics concepts must be used in the solution. Each problem should involve two or three concepts – no more, no less.

Writing a problem will involve several steps:

1. Think of a situation that provides fertile ground for asking physics questions about it. The situation should be able to be depicted in some sort of picture or diagram, which you’ll need to make (you may draw this freehand or with the aid of a computer drawing program). We’ll call this picture and short, written description of the situation your problem sketch.

2. Next, you’ll want to identify what your problem is asking the solver to find. You’ll also want to come up with reasonable values for whatever given information you are providing the solver, and you’ll want to determine how the question(s) will be asked. This is your draft.

3. Finally, you’ll take your draft and put it into finished form. To do this, you’ll have to check over the draft to make sure you have described the situation adequately, provided all the necessary information, and asked the question(s) clearly and unambiguously. Also, to be absolutely sure you have included everything you need to include, you’ll need to solve the problem yourself. This carefully proofread version, including your written-out solution, is your final problem.

Schedule:

The items due here are in addition to the homework problems assigned for this Monday, Wednesday, and Thursday.

Monday, 1/12: 2 sketches due

Wednesday, 1/14: 1 draft, 1 more sketch (3 total turned in)

Thursday, 1/15: 2 more drafts (3 total turned in), 2 more sketches (5 total)

Friday: 2 more drafts (5 total)

Saturday: 5 final problems

Wednesday, January 21: 45-minute test made from student-written problems

After you hand your final problems in on Thursday, I will look at all of your problems. I will grade each problem on a 5-point scale and return them to you on Monday.

On Wednesday, there will be a three-question test composed from questions you have written. The questions we choose could come from either class. We may do some rewording of the questions for the sake of clarity, but the physics content will be exactly what you wrote, with the solutions you devised.