This past spring I taught a standard data structures course (stacks, queues, binary trees, heaps, asymptotic analysis, that kind of thing). Inspired by a group I participated in exploring pedagogy and course design—led by the wonderful Betsy Burris—I decided to give an oral final exam instead of a typical written exam. For whatever reason, oral exams don’t seem very popular in the US these days, but overall I think my experiment was a success and I definitely hope to repeat it in the future. This post reports on how I organized things, what went well and poorly, and what I learned overall. I’d love to hear from anyone else who tries something like this in their own course. I’m also happy to answer questions, either in a comment on this post or via email.
Exam content
I prepared three questions for the exam. The first was fairly simple (“explain algorithm X and analyze its time complexity”) and I actually told the students ahead of time what it would be—to help them feel more comfortable and prepared. The other questions were a bit more open-ended:
-
The second question was of the form “I want to store X information and do operations Y and Z on it. What sorts of data structure(s) might you use, and what would be the tradeoffs?” There were then a couple rounds of “OK, now I want to add another operation W. How does that change your analysis?” In answering this I expected them to deploy metrics like code complexity, time and memory usage etc. to compare different data structures. I wanted to see them think about a lot of the different data structures we had discussed over the course of the semester and their advantages and disadvantages at a high level.
-
The final question was of the form “Here is some code. What does it do? What is its time complexity? Now please design a more efficient version that does the same thing.” With some students there was enough time to have them actually write code, with other students I just had them talk through the design of an algorithm. This question got more at their ability to design and analyze appropriate algorithms on data structures. The algorithm I asked them to develop was not something they had seen before, but it was similar to other things they had seen, put together in a new way.
Overall I was happy with the questions and the quality of the responses they elicited. If I do this again I would use similar sorts of questions.
Time
You might well be wondering how long all of this took. I had about 30 students. I planned for the exam to take 30 minutes, and blocked out 45-minute chunks of time (to allow time for transitioning and for the exam to go a bit over 30 minutes if necessary; in practice the exams always went at least 40 minutes and I was scrambling at the end to jot down final thoughts before the next students showed up). I allowed them to choose whether to come in by themselves or with a partner (more on this later). As seems typical, about 1/3 of them chose to come by themselves, and the other 2/3 in pairs, for a total of about 20 exam slots. 20 slots at 45 minutes per slot comes out to 15 hours, or 3 hours per day for a week. This might sound like a lot, but if you compare it to the time required for a traditional written exam it compares quite favorably. First of all, I spent only two or three hours preparing the exam, whereas I estimate I would easily spend 5 or 10 hours preparing a written exam—a written exam has to be very precise in explaining what is wanted and in trying to anticipate potential questions and confusions. When you are asking the questions in person, it is easy to just clear up these confusions as they arise. Second, I was mostly grading students during their exam (more on this in the next section) so that by about five minutes after the end of their slot I had their exam completely graded. With a written exam, I could easily have spent at least 15 hours just grading all the exams.
So overall, the oral exam took up less of my time, and I can tell you, hands down, that my time was spent much more enjoyably than it would have been with a written exam. It was really fun to have each student come into my office, to get a chance to talk with them individually (or as a pair) and see what they had learned. It felt like a fitting end to the semester.
Assessment
In order to assess the students, I prepared a detailed rubric beforehand, which was really critical. With a written exam you can just give the exam and then later come up with a rubric when you go to grade them (although I think even written exams are usually improved by coming up with a rubric beforehand, as part of the exam design process—it helps you to analyze whether your exam is really assessing the things you want it to). For an oral exam, this is impossible: there is no way to remember all of the responses that each student gives, and even if you write down a bunch of notes during or after each exam, you would probably find later that you didn’t write down everything that you should have.
In any case, it worked pretty well to have a rubric in front of me, where I could check things off or jot down quick notes in real time.
Social aspects
People are often surprised when I say that I allowed the students to come in pairs. My reasons were as follows:
- For many students, being able to come with a partner may help ease their anxiety about the exam. I was especially sensitive to this since for many of them an oral exam was a new experience. (I asked beforehand, and only two or three of them had ever had an oral exam, and those were in foreign language classes.)
- Computer science, like most disciplines, is often collaborative. No matter whether a student ends up going into industry or academia, it is very likely that any data structure design or analysis that they end up doing in “real life” will be in collaboration with others. So in this sense a collaborative oral exam is fairly authentic—much more so than a written exam. That is, the skills, activities, and modes of thinking and communicating required by a collaborative oral exam are actually fairly similar to what might be required of the students in the “real world”.
- There was also, of course, the simple reason of expedience: it cut down on the amount of time I had to spend administering the exam.
Overall I was really happy with the result. Many of the students had been working with a particular partner on their labs for the whole semester and came to the exam with that same partner. For quite a few pairs this obviously worked well for them: it was really fun to watch the back-and-forth between them as they suggested different ideas, debated, corrected each other, and occasionally even seemed to forget that I was in the room.
One might worry about mismatched pairs, where one person does all of the talking and the other is just along for the ride. I only had this happen to some extent with one or two pairs. I told all the students up front that I would take points off in this sort of situation (I ended up taking off 10%). In the end this almost certainly meant that one member of the pair still ended up with a higher grade than they would have had they taken the exam individually. I decided I just didn’t care. I imagine I might rethink this for an individual class where there were many of these sorts of pairings going on during the semester—but in that case I would also try to do something about it before the final exam.
Another interesting social aspect of the process was figuring out what to do when students were floundering. One explicit thing one can do is to offer a hint in exchange for a certain number of points off, but I only ended up using this explicit option a few times. More often, after the right amount of time, I simply guided them on to the next part, either by suggesting that we move on in the interest of time, or by giving them whatever part of the answer they needed to move on to the next part of the question. I then took off points appropriately in my grading.
It was difficult figuring out how to verbally respond to students: on the one hand, stony-faced silence would be unnatural and unnerving; on the other hand, responding enthusiastically when they said something correct would give too much away (i.e. by the absence of such a response when they said something incorrect). As the exams went on I got better (I think) at giving interested-yet-non-committal sorts of responses that encouraged the students but didn’t give too much away. But I still found this to be one of the most perplexing challenges of the whole process.
Coverage
One might wonder how much of the material from an entire semester can really be covered in a 30-minute conversation. Of course, you most certainly cannot cover every single detail. But you can actually cover quite a lot of the important ideas, along with enough details to get a sense for whether a student understands the details or not. In the end, after all, I don’t care whether a student remembers all the details from my course. Heck, I don’t even remember all the details from my course. But I care a great deal about whether they remember the big ideas, how the details fit, and how to re-derive or look up the details that they have forgotten. Overall, I am happy with the way the exam was able to cover the high points of the syllabus and to test students’ grasp of its breadth.
My one regret, content-wise, is that with only 30 minutes, it’s not really possible to put truly difficult questions on the exam—the sorts of questions that students might have to wrestle with for ten or twenty minutes before getting a handle on them.
Next time
Would I do this again? Absolutely, given the right circumstances. But there are probably a few things I would change or experiment with. Here are a few off the top of my head:
- This time, I only decided to do an oral final exam quite late in the semester. Next time, with more planning, I would like to think carefully about how to explicitly prepare students for this type of assessment, i.e. to give them chances to practice the necessary skills and to receive feedback. I am not yet sure what that would look like.
- I might try something like a system for e-mailing students a harder question a certain amount of time before their slot—the idea being that they would have that time to think about and prepare their answer to the question, which they could then present during the slot (in addition to answering some questions on the spot). However, a potential major downside is that students who struggle with the hard question might come in feeling demoralized and do even worse on the rest of the exam.
- Next time I will definitely be more careful and intentional about writing down a script for exactly what I want to say at the start of each exam. There were a lot of things I wanted to remember to say, and specific ways I wanted to frame the time, and at times it seemed a bit haphazard and inconsistent.
Again, I’m happy to answer questions in the comments or by email. If you are inspired to also try giving an oral exam, let me know how it goes!
blog :: Brent -> [String] 
This has a lot of similarities with what I used to do when I gave whiteboard programming problems in technical interviews. It is currently fashionable to deride this process but I think I did it sympathetically and it worked really well for us.
The first thing that jumped out at me was that you tried to cover the whole thing in 30 minutes, and I guess you found it quickly that that was an aggressive schedule. I allowed 30 minutes for a single question (which was simple enough that on-the-ball people could do it in 5) and often hit the limit. Of course your students had just learned the material so it was fresh in their heads and they were specifically preparing to be tested (I hope).
I did a fair amount of guiding, perhaps more proactively than you did, since my goal was to assess their skill and time spent floundering was wasted. It might be more difficult to do that in a scenario like an exam where you expect more precise grading criteria, as with your rubric.
I never had an oral exam in school but I expect I would have liked it more than written ones (and I was a good test-taker).
First, I want to echo dfan. This is very similar to giving a technical interview. There is a good amount of experience and ideas from that area, though they are rarely systematically studied or applied. Frankly, patterning the oral exam after technical interviews is probably a good idea as it is likely to be most relevant to them. Semi-facetiously, you could save a lot of time by just looking up appropriate job positions and just having them apply. I would usually expect about an hour would be needed for the interview, even for a “no”, and would have usually an extra hour between if I was doing more than one in a day (rare). Usually, we had two people giving the interview, though we didn’t necessarily have both people there the whole time.
I’m curious on what your thoughts are on the various biases that would come up with such an arrangement. Since the exam was spread over several days, it’s quite likely that those examined at the end had talked to those examined before. You undoubtedly have a variety of biases that would not come in to play for a written exam. There are no doubt students you like more than others, there are students who are better at speaking and speaking on their feet than others. Even with a rubric, you almost certainly compared different exams. This suggests, for example, that the very first exam is likely to get an “average” score and the rest would be compared against it. (Incidentally, if your students were clever and had control over the order they were examined, it would be best for all of them to sort themselves in order of aptitude.) A similar situation happens if a student does particularly poorly (for whatever reason) on an early question. That’s likely to color your evaluation of their later answers, even though it may just happened to be a topic where they were weak.
The above is not meant to criticize, though I suspect the above concerns are some of the reasons why oral exams are rare except for individualistic/project-based work like a PhD defense or a science fair project. Frankly, with my industry hat on, some of the issues above are just reality. Being a poor speaker is a problem and will hurt you throughout your career, from interview to leadership.
Turning back to my experience in interviews: the approach I typically took was to let people flounder a bit with a flat effect to give them time to come to a solution on their own (or ask for more information). I would definitely watch how they were floundering. Were they trying to “feel out” an answer from me, were they brainstorming, were they making things up? I would explicitly provide hints if I thought it would get over a particular hurdle/confusion into an area where they might be more comfortable. Generally, though I didn’t give too many hints before just providing an answer (there was usually several I’d accept and I was open to anything that would work), and moving on. I started from easier questions to harder questions and from what their claimed strong areas were to areas they were less familiar. Usually I would ask a single, fairly open-ended “representative” question from each topic area I wanted to cover. This meant if I asked a question about JavaScript first, the next question may well be about database design. It’s possible this mitigated some of the consequences (on both sides) of how an earlier question was handled. I was generally looking for my expectations to be met (or exceeded) rather than some absolute level of skill/knowledge, and my expectations could vary wildly between and within an interview. For example, if an interview wasn’t already going poorly, I’d usually ask a question out of the interviewees claimed area of expertise (and be explicit about it, it wasn’t a “gotcha” question) to see how they would handle being asked about something they didn’t know much about and how they would handle learning what was needed.
We started having a sizable pre-interview project later on which would be more realistic insofar as developers tend not to have to solve problems on the spot purely from memory. It also provided something concrete to talk about. We didn’t usually require it to be completed to hold the interview, you would just bring whatever you had done. (It was definitely something that could be completed in an evening or a weekend.) I was less involved with interviewing at the time, but I’d generally say that that would improve the value/quality of the interview (and mitigate some of the earlier mentioned factors). For industry, there’s an issue with people being turned-off and not applying due to such a pre-interview project, but that’s not a problem for an exam.
On a different topic, though aimed more at K-12, have you seen the work of Kieran Egan? “An Imaginative Approach to Learning” is probably a good place to start if you don’t mind buying a book.
At my university (fau in germany) we do oral exams all the time. There are always 3 people involved:
* the student (obviously)
* the professor, asking the questions
* an observer, writing a protocol
After the exam itself, the student leaves the room and after a short discussion the grade will be reported.
This might be an idea for you, too. You would have an easier time interacting with the student(s) and there would be a protokoll never the less.
Thanks, that is a good idea! Where do you find the people to serve as observers? Are they other professors?