Hacking Assessment: Redesigning the Numbers Game

photo credit: davidfg via photo pin cc

In a recent post, I outlined some ideas that I have about integrating principles of game design into the FYC course. As I pointed out, I’m not all-out gung-ho about the idea of the gamification of education. It turns out that many of my reservations about this latest trend in reforming education are shared by game designers themselves. In her post “Everything Is Game Design,” game designer Elizabeth Sampat makes clear that the assumption that any group of practitioners can co-opt and apply the extremely complex and abstract principles at play in a successfully engaging (to some) game to any other domain is over-reaching:

Gamification” assumes all games share the same mechanics, which means everything that’s gamified is basically the same shitty game. Using badges and leaderboards and offering toothless points for clearly-commercial activities isn’t a magic formula that will engage anyone at any time. Demographics are different, behavior is different . . .

These are the same issues with gamifying the classroom that keep me from wholly embracing the concept. For one, the whole point of a game is that it is . . . well, a game. Games are voluntary. As soon as you force someone to join in a game, it stops being a game for them. It becomes a compulsory activity devoid of intrinsic value and all of the extrinsic rewards you can throw at them, while perhaps artificially increasing their motivation to play the game, cannot turn it back into a game, unless it’s in the negative sense. Even when we gamify a class, we’re still making the learning that takes place within that game compulsory and effectively negating any positive characteristics of gaming that we are attempting to channel. And, as Sampat points out, the characteristics that make any game engaging cannot be standardized. What works for one gamer doesn’t work for another. So, in many ways, game designers face the same kinds of issues and challenges that educators face.

Another point that I think has been largely overlooked in this debate is that, for the large majority of students (if not all), school is already a game. We have goals (behavioral or learning objectives), challenges (in-class activities, homework, exams, and standardized tests), and rewards (grades). We’ve got levels (grade levels based on age in K12 and hours-earned status in college) and leaderboards (A/B honor roll in K12 and President’s and Dean’s lists in college). And we have clearly defined roles (teacher as locus of power and expertise, student as powerless and largely silent novitiate). Some students figure out pretty early how to play the game. In college, these are the students whose identity is inextricably intertwined with their grades. “But I’m an ‘A student,'” they insist when faced with anything other than. Other students learn early on how to game the game. These are the students who know how to manipulate the system and those in charge of it and can often be just as successful at winning the game as their overachieving counterparts. But some students never learn how to play the game according to our rules. Others don’t want to play it because they see it for what it is.

Whether we realize it or not, we’re already playing games with our students. And it’s a numbers game. Play the game according to our rules and we’ll reward you with a high GPA and a diploma, with the promise that these things are the badges you need in order to level up to the American Dream. This kind of game is both irrelevant and counterproductive in a culture that is becoming increasingly participatory, rather than competitive, in nature (just read Share or Die: Voices of the Get Lost Generation in the Age of Crisis to get an idea of how important cooperation and collaboration is becoming for those graduating into the current economy). While many educators are fighting to reform the standardized, hierarchical forms of assessment that have been in place since the industrialization of education, until they are successful at effecting a wholesale paradigm shift and not just applying a false facade and calling it reform, we are forced (much like our students) to try to figure out ways to hack the game. As Sampet argues:

Finding the reward structures and the rules that are already in place, and figuring out how to make them more effective, is the key to making life better for everyone— not adding an additional layer of uninspiring mechanics that push us to engage with mechanics that already suck.

Just as games are not one-size-fits-all, assessment shouldn’t be one-size-fits-all, neither in terms of standardized criteria applied to all students nor evaluative formats used for all courses/disciplines. Just as each course has its own unique set of learning objectives, each course should have a different method for assessing how students go about achieving those objectives. I think it important to explore various assessment methods in an effort to find which is the most effective for a particular course. For example, I have found that a portfolio method is exceptionally well-suited for my composition courses, as it allows for the abstract nature of the writing process and the subjectiveness that characterizes the act of evaluating and valuing a piece of writing. But in trying to incorporate a portfolio system into my speech courses (both an introductory oral communication class and an advanced argumentation and debate class), I have had less success, though for different reasons (perhaps due to the differences among the students: freshman and upper level secondary-education majors, respectively). As much as the portfolio method places value on each student’s individual learning needs, goals, and achievements, within the current grades-based system, students in certain courses need to be able to visualize their learning at both a qualitative and quantitative level. So, what are the alternatives?

Peer Assessment
One option that is gaining ground is peer assessment. Cathy Davidson has successfully explored this method in her “This Is Your Brain on the Internet” class (read “How to Crowdsource Grading” for her description of the process and the thought-provoking debate that followed and “How to Crowdsource Grading: A Report Card” for an overview of her students’ responses to the method). Many MOOCs utilize peer assessment out of necessity. According to Debbie Morrison, within the MOOC environment, peer assessment results in an enhanced learning experience for the student, as grading their peers’ work requires a deeper engagement with course content.

I’ve utilized peer assessment in both of my speech classes to varying degrees and with varying levels of success. In my introductory speech class, the students work together at the beginning of the term to develop a checklist for an effective speech (I don’t use rubrics because, in my experience, they become just another hierarchical form of grading that allows students to retain many of the gaming habits they adopted in K12). They do this by watching several speeches on YouTube and creating individual lists of do’s and don’ts, which we then collate into a master list. For each speech, students are evaluated by five randomly selected anonymous peers, who use the checklist to assess the speech. The students are also filmed and they must use both the video and their peers’ checklists to compose an assessment of their speech that they post to an e-portfolio, along with all artifacts associated with the speech (outlines, bibliographies, slideshows, photos of visual aids, the video of the speech, etc.). For this particular class, I have found that a combination of self and peer assessment has been much more effective than a solely self-based assessment (which tended to be superficial) or even an instructor-based assessment (in which students received only one assessment, as opposed to five, and tended to focus more on improving their “grade” than becoming a more effective speaker). With the peer assessment method, students’ speeches are being evaluated by their audience and their focus becomes oriented towards improving their audience’s response to subsequent speeches.

I have tried this kind of peer assessment in my debate class with far less success. For one, the class is much smaller, and consists, for the most part, of a cohort of sophomore and junior-level secondary education majors. These students tend to be very cliquish and ironically conservative in terms of the practices they expect in the class; they tend to be “A-gamers” obsessed with acing the course and uncomfortable with the level of abstractness and improvisation involved in debate. As a result, they tend to assess their peers over-generously and resist critiquing one another (one class even admitted to giving each other positive assessments across the board because they didn’t want to “hurt someone’s grade”). They look to me as the expert, so their portfolio reflections tend to be focused on flattering me and the course and highlighting aspects of their performances from my point of view (“If I were the instructor, I would give this speech a [insert grade here]”). Despite my best efforts, these students are resistant to assessment formats that are not instructor-based. So what’s a disruptive pedagogue to do?

Contract Grading
While I was at first dismissive of contract grading based on the distaste I harbor for the artificially hierarchical nature of any type of grades-based assessment (and the name’s implications of a kind of capitalistic supply and demand relationship between student and teacher), I have become less dismissive of the method in terms of its ability to bridge the gap between my students’ need for a quantitative value to be placed on their learning and my own objective of encouraging them to recognize and become complicit in the qualitative value of that learning.

For one, I’m hoping that it will eliminate the specter of grades that haunts the course by directly addressing the students’ anxiety regarding their status in a course that has no exams or other easily quantifiable activities. Students will decide what grade they wish to work towards and will have a specific, objective set of criteria that they must achieve in order to earn that grade (yes, I know this sounds just like a syllabus with a traditional grading schema, but contract grading makes the implicit aspects of the traditional schema explicit and, in many ways, mimics the game design principle of starting at zero and gaining points as you go). Once the question of grades is out of the way, perhaps the students will be more willing to focus on learning and improving.

Secondly, contract grading requires student input in regards to the challenges that must be met in order to level-up (yes, I know I’m wading back into gaming territory, but, as I’ve argued, our goal should be figuring out what works for a particular course and cohort of students rather than a wholesale dismissal or acceptance of any one method or theory). Often, in order to earn an A or a B, students must complete additional learning tasks, sometimes choosing between several options, which they can be invited to develop. This aspect of contract grading is the one that I find most promising in terms of encouraging student investment in the learning environment. While I have long preached to students that, in the words of Lennon and McCartney, “in the end, the love you take is equal to the love you make,” contract grading makes student-centered initiative an explicitly integral component of the course.

Thirdly, contract grading will allow me to both address the students’ insistence that I fulfill the role of expert assessor and my wish for them to fulfill the role of deliberate and reflective practitioner. Different grades require different levels of mastery, so students who contract for a certain grade must revise and/or re-attempt assignments that don’t demonstrate mastery. While my debate students can’t re-do a live debate, they can complete a video re-enactment that improves upon their live performance or record a play-by-play self-critique using Voice Thread or screencasting software. In addition, some of the optional assignments can require peer or self-assessment or other types of reflective learning practices.

While I’m not completely comfortable with contract grading (just as I am not completely comfortable with gamification), I also recognize that other assessment methods are not working for my upperclassman and, as a result, are interfering with my efforts to push them beyond a superficial engagement with their learning in the course. I believe firmly that we must recognize our students’ needs, values, and histories; but we can’t pick and choose which of those we take into consideration when designing their learning environments. Sampet makes a point that I think is important for us to keep in mind in the process:

The core principle to remember is that game design is everywhere. Instead of trying to stick a crappy, half-formed game onto real life, the real challenge— the one that’s tough, the one that will bring the greatest results— is to fix the bad game design that’s all around us.

Students won’t be open to assessment that values quality over quantity or process over product until we recognize that our current assessment paradigm is a badly designed game that needs to be torn down and redesigned. Sampet suggests two questions to ask when considering whether or not something is badly designed:

  • What’s supposed to be the goal here?
  • Is this experience set up to help or hinder my ability to reach that goal?
I’m game.
Resources on Contract Grading
These are the sources that I consulted to help me to better understand the possibilities afforded by contract grading:


Advertisements

Building a Better Blogging Assignment Redux

photo credit: Mike Licht, NotionsCapital.com via photo pin cc

One of the sessions at last week’s THATCamp dealt with the issue of designing a better model of student blogging. You can view my Storify of the session here.

I thought that I would add some of my own ideas and discuss how I address some of the issues raised during the session (since, unfortunately, I couldn’t be there).

As noted on the session’s Google Doc, a major problem with requiring students to blog is that the large majority of them are unfamiliar with blogs, so we need to identify effective methods for acculturating them to the genre. Since I’m an advocate of immersive learning, I’ve found that many students begin to “get” blogging by spending a good deal of time actually doing it. But I’ve developed a few orientation assignments that help them get off to a good start.

  • Require students to locate, deconstruct, assess, and subscribe to blogs on topics that interest them: As homework during the first week of class, I have students locate several blogs on a topic that they’re interested in. They pick the best three and subscribe to them. While exploring blogs on their topic, they create a list of criteria for an effective blog. We use a class meeting to collate their criteria into a master list that they can then use as a checklist for their own blogs. Next term I’m planning to expand this assignment by having students work together to deconstruct a blog.
  • Teach them how to comment: This is something that I still struggle with. I provide students with several resources on commenting, including those mentioned at the session; nonetheless, many of them provide largely superficial comments. Next term I plan to have students read and assess comments on the blogs they’ve subscribed to and add their own comments. Similarly to the assignment above, students will work together to establish criteria for effective commenting.

A second, and equally important issue, is the logistics of blog management, both for yourself and the students: controlling pacing (so that you don’t have to deal with an influx of posts and comments at the last minute), encouraging engagement with the blogs (both their own and their peers’), and assessing the blogs.

  • Establish submission guidelines (and stick to them): I establish strict deadlines for post submissions and stick to them from the very first post. I generally make the deadline the night before class in the case of totally face-to-face courses. For my hybrid courses, the deadline is on the day that we do not meet. Either way, I set the deadline for a time well before I and other students need to access the blogs.
  • Encourage engagement with peers’ blogs: I require that students subscribe to each others’ blogs and read and comment on a certain number of them each week. I’ve tried to encourage more depth to their comments by staggering the due dates for posts and comments (generally they have 12-24 hours after the blog post deadline to read and respond to peers’ posts). I’ve had even better success this past term with combining this with rotating students’ roles between posters and readers/commenters. This allows them to fully focus on and engage in their role. This method requires reducing the number and frequency of posts for each student, but I think that the pay-off will be worth it, especially by placing as much emphasis on their comments on others’ blogs as on their own blog posts (which means that I’ll have to invest more time into assessing their comments somehow).
  • Make the blogs an integral component of the course: I try to immerse students in their blogs as much as possible because I’ve found that the more they blog, the better bloggers they become. I now require that all of their writing be done on their blog and I ask them to blog and comment on blogs as frequently as possible (at least once a week). I think that it’s a major mistake to have students blog but then not integrate the blogs into the classroom interactions in some way; this encourages students to view the blogs as secondary to the other class work. In my literature courses, the students’ blogs become the fulcrum for the class discussions. I encourage students to pick the most thought-provoking for us to look at together in class. In my FYC courses, I pick one model post each week for us to critique as a class, asking students to assess the post in small groups, looking for reasons why I selected the post as being a good model. Since the class uses Google+ as a virtual learning space, I also “plus 1” those posts that are especially thought-provoking, well written, and/or visually appealing (I encourage students to do this, as well); this provides students with almost instantaneous feedback and encourages those who might not have read and/or commented on the posts to do so. This also results in a type of gamification of the blogs, as some students begin to work to earn “plus 1’s” from me and their peers. Next term, I plan to also encourage students to use other social media to promote and “like” their peers’ posts.
  • Involve students in the assessment of their blogs: In a previous post, I outlined how I require students to self-assess their writing. I have been happy with the way I’ve asked students to create a portfolio of their blog posts to submit to me at the end of term, rather than assigning a grade to each individual blog post (I’ve tried to eliminate traditional grades as much as possible in my classes). Normally, I have students do this via a final assessment form that they fill in and submit to me via email, hyperlinking to specific posts that they want to include in their assessment, and discussing in detail why they selected them and how they demonstrate what they’ve learned about writing. But I’m considering remixing Mark Sample’s idea of a blog audit; I think that making their reflections public on their blogs will encourage an even deeper consideration of who they are as writers and what they’ve done as bloggers over the course of the term, mirroring the way that many bloggers use their blogs as reflective spaces. I also like his idea of having students revisit and revise some of their old posts, which is something I used to encourage students to do with their writing before I switched to blogs, and would like to re-incorporate into their portfolio creation.
  • Utilize formative and peer assessment: This is still something that I’m tweaking. So far, I’ve found my method for providing formative assessment effective (and students have indicated the same). What I haven’t been able to integrate as effectively is peer assessment. I would love to use a badge system, like Mozilla’s Open Badges, but I haven’t had the time to figure out the best way to do so (or if it’s even possible, since I don’t know how to code or if it’s necessary to know how to do so to use the program, two issues I’m hoping to remedy soon). In the meantime, I’ll encourage the use of readily available social media feedback systems such as Facebook’s “like” and Google’s “plus 1” buttons.

A third issue that seems to have been prevalent during the session is that of how to allow for disruption and alternatives within the blogging domain.

  • Allow/encourage alternative uses for blogs: Since I require that students publish all of their writings for the class to their blog, this means that sometimes their blog posts contain nontraditional material (although I always try to help students understand that, with the advent of photoblogs, vlogs, and podcasting, there is no longer such a thing as traditional blog content). For example, this term I’m requiring my FYC students to use Storify to create their annotated bibliographies and then embed their stories into their blogs for comment by me and their peers. Last term, my students participated in DS 106, which meant that their blogs became populated with memes, mashups, animated gifs, and sound clouds.
  • Disrupt the digital environment: Interestingly enough, as participants were discussing Mills Kelly’s ideas about disruptive pedagogies and then subsequently considering ways to disrupt student blogging, I was blogging about Paul Fyfe’s theory of teaching naked and considering how to disrupt the digital environments within which I ask my students to work. One idea that I blogged about that serendipitously showed up on the blogging session Google Doc is that of requiring students to engage with and use their blog posts in non-digital ways. I think that this is an aspect of student blogging that needs more attention and I hope that a conversation can develop around it.

These are just a few of the blogging methods that I have found effective and, as indicated, I’m still working at improving some of them. I encourage those who require their students to blog or who are thinking of doing so to help continue the conversation here, on my Storify of the THATCamp session, on Mark Sample’s THATCamp blog post, or on Twitter (use the #thatcamp hashtag).

This Is What a Final Exam Should Look Like

image by freeimageslive.co.uk – gratuit

I used to give traditional final exams, even in my First-Year Composition course. Every semester during finals week, my FYC students would sit in the classroom for two hours writing an essay. Supposedly, this was an exercise in assessment: by composing a full-length essay in class, students were demonstrating what they had learned about writing that term. But I began, a few years ago, to question just what this exercise proved–how much did writing an in-class essay show about student learning?

First of all, it seemed to be a contradictory assignment. I had spent an entire semester trying to convince students to spend time developing their essays–to let their drafts rest for a while before revising them; to proofread carefully, looking for one type of error at a time; to let others read their writing and provide feedback on it to help them see it through their readers’ eyes. And then, for their final, heavily-weighted piece of writing, I asked them to throw all of that advice out of the window and write an essay, from start to finish, in two hours, with no peer review and little time to revise or proofread. Did I have a legitimate pedagogical reason for doing this? To be honest, the answer is No. I did it because it was what everyone else was doing and it was what I was told I should do, as well. As recently pointed out by David Jaffe in “Stop Telling Students to Study for Exams,” traditional final exams are grounded more in tradition than they are in good pedagogy or the realities of how we want students to learn and what we claim we want students to take away from our classes (and college).

So, I began to re-think how best to use the two hour final exam meeting and to try out different ways of capping off the term with a demonstration of student learning. One thing that always bugged me about classes that required a research project (which, as an English student, was all of them) was the fact that no one, other than the professor, ever got to see the results of all my efforts. And I was always curious about what my classmates were researching and what the results were. Sometimes we’d talk about our projects in process, but the paper itself was usually due on the day of the final, so there was never any kind of whole-class plenary discussion of the topics and issues we had all been immersed in all term. What happened in our research papers stayed in our research papers. Using this experience as an example of how not to make the research project relevant to the course and to the students, I’ve experimented over the years with various ways for students to use the final exam meeting to share their research projects and what they have learned over the course of the semester with the class.

One semester, we had our own mini-symposium (mimicking our university’s annual student research symposium, which I had asked my students to attend that term), with students simply reading their research essays out loud at the podium. And as I’ve discussed in another post, I recently asked my first hybrid FYC class to turn their research projects into multi-media presentations, requiring them to articulate their written ideas in multi-modal rhetorical mediums. I think that both of these were fairly successful methods of asking students to demonstrate their learning while making that learning relevant to the course as a whole and encouraging students to take pride in the work they had done. But I wasn’t completely happy with either, as they both encouraged a kind of passivity on the part of the student’s audience, including myself.

Then, the other day this video popped up in my Twitter stream and, for me, it was an epiphanic moment:

I couldn’t help but think about what classrooms look and sound like during a traditional final exam. And even what my own final exams look and sound like, i.e., a sage on a stage (even though I’ve shifted the locus of power slightly by placing the students on the stage) with a glassy-eyed audience who typically respond with silence when asked if they have any questions. And I couldn’t help but to compare those classrooms to the one in the video (you can read more about the research slam at “The Unconference Strikes Back”).

What if final exams looked more like this? What if students shared their learning with one another in the kind of interactive, experiential, small-group method encouraged by the research slam? And what if I could join those moving from group to group, listening to (and perhaps even videoing) them engage in conversations about their learning? What if I asked them to post those videos on their blogs so that anyone could see them sharing and answering questions about their learning?

How powerful would that be?

Pretty powerful, I think. I’ll let you know how it turns out.