Revisiting “Peer Review” in the Post-Pandemic Era

In WAC, we consider writing as a process that promotes critical thinking. One of its implications is to emphasize revision. The revision process is built into the scaffoldings of big projects during the semester. In revising, we make new connections and discover new ways to understand and express thoughts. Perhaps one support for this argument is to observe the differences between the second edition (2011) of John C. Bean’s Engaging Ideas, one of the key texts for WAC, and its third edition (2021) (in which Bean co-authored with Dan Melzer). Of course, much can be discussed regarding the differences between the two, as we have new developments of writing pedagogy in the past ten years (Preface ix-xi), face new challenges (e.g., covid-19), and adapt ourselves to new ways of learning and teaching. In the preface of the third edition, Bean and Melzer list major changes in the new edition (xi-xii). In this essay, I want to focus on the “peer review,” one of the key strategies for revision. I’ll trace and discuss the changes in Bean’s book and reflect on my own experience in teaching. I’ll also compare the revision process with the one from the design industry and show what I think can be improved in our teaching.

In the second edition of Bean’s book, guidance for peer review is in Chapter 15 “Coaching the Writing Process and Handling the Paper Load,” a chapter focusing on how to minimize teachers’ workload while at the same time encouraging students to write. Peer review is considered one of the key strategies to save time (295-302). Although it is still included as a strategy for saving time in the same chapter in Bean’s third edition, “peer review” is singled out and reframed into a new chapter “Helping Students Use Self-Assessment and Peer Review to Promote Revision and Reflection.” In this newly added chapter, not only does it include all the useful advice in the second edition, but also lays out recent research findings for the benefits of peer review (trying to advocate its value; e.g., research shows that “peer review from at least three peer readers results in feedback that is strikingly similar to instructor feedback and leads to as much or more global revision” (252)), admits the problems with peer review (244-245), and adds a new digital platform (Eli review), in addition to the original two (namely, Calibrated Peer Review and Peerceptiv). I like the shifting from a teacher-centered view to a student-centered one; in other words, the third edition emphasizes more how peer review benefits students.

Including the three digital platforms raises an interesting question: how should we use media to facilitate peer review in a post-pandemic era? Before the pandemic, I used to ask students to print out hard copies and bring them to class (this causes some problems: some students tend to work on their drafts at the last minute and cannot find a place to print their work in an early morning class). In class, they exchange drafts, write comments, and give them back to each other. Transitioning to online teaching, I used Zoom breakout rooms and Google Doc to do the work, with guided instructions. Coming back to the classroom again this semester, I find that students are more likely to bring their laptops to class and feel more comfortable with digital tools, so I continue using online platforms such as Google Doc and Google Form. To me, a digital copy can be accessed more easily than a hard copy, considering sometimes the print copy gets lost. Sometimes handwriting is not easy to recognize compared to the regularized typography. Of course, there are disadvantages. Different platforms can be overwhelming as students need to learn how to manage them. Students also lose the freedom of writing on a piece of paper, and of arranging and rearranging ideas in a non-linear way. Additionally, they are limited by a small screen, instead of a wider view of multiple pages. Handwriting is also more intimate than the standard font. We should also consider students’ needs and accessibility: some students prefer physical copies, while others like the digital version better.

We may continue discussing the pros and cons of learning through different media, but I want to return to the topic of how to do peer review. I believe that we should guide students what we instructors practice when we comment on their papers and lay out some guided instructions and templates for their reference. Without guidance, I find students tend to focus on editing sentences such as syntax, grammar, etc. In the prompt for peer review, I usually ask students to write marginal comments of only higher-order concerns, and an end-comment or a one-paragraph comment (312-313), which consists of comments on the strong and weak points of the draft, as well as the suggestions for revision. In this way, reviewers can focus more on the higher-order elements. I also consider it important to ask students to write specific and descriptive comments and support their comments with evidence and analysis, in line with Bean’s proposal for “descriptive questions” instead of “judgment questions” (247).

Although I previously asked students to talk/communicate with each other about the comments they make, to clarify any confusions and make a to-do list for revision, I found it hard for students to translate the feedback into revisions. I also find Bean and Melzer spend most of their time addressing how to do peer review and give good feedback, but not so much how to integrate comments into revision. Indeed, there is a short section on “Revision Plans” (239), which includes a sheet to ask students to reflect on if the reviewer can correctly identify the important elements (such as thesis, analysis, etc.) and think about how to revise. The set of questions are helpful to guide students to reflect on some important writing elements, but sometimes this top-down approach may ignore certain aspects that are not listed in the guidance but exist in the feedback. Hence, I would propose a bottom-up approach.

Peer review is not only limited to academic practices. It is also a critical process in designing an app or a website. During the summer, I took a series of courses on Google UX Design Professional Certificate, in which they discuss peer review in detail. Some suggestions are also valuable for my teaching.

I think perhaps one missing piece in Bean’s book is how to read feedback. The first tip would be to “stay open minded.” Reading feedback sometimes can trigger mixed and even defensive feelings. It is important to see the feedback as something to expand one’s own limited view, and to “think about where that feedback is coming from,” how it is different from one’s own expectations. Secondly, keep in mind one’s goals. For a writing assignment, what are the goals and rubrics? Could applying the feedback improve the writing to better meet the goals? If not, think about why the feedback is not effective and document it for later consideration. Thirdly, keep one’s own agency. There will be moments when one disagrees with the feedback, and that’s okay. Everyone has their own opinions about the work. When we disagree, we should not completely ignore the feedback. Instead, we need to think about where the disagreement comes from, and whether and how the different thought is supported by evidence and reasonable analysis. We then need to judge if the feedback would fit the writing goals.

In addition to how to read feedback, I suggest arranging the feedback in an “affinity diagram.” The diagram asks one to pull the comments from different sources to one single place (with sticky notes; online platforms such as Miro, Jamboard, Notely, Mural, Padlet can do something similar.) Sometimes marginal comments are spread out through the entire paper, and can be confusing if one receives feedback from more than one reviewer. Writers may not notice some common patterns without stepping back a little bit. The affinity diagram helps with synthesizing the feedback into groups. One may rearrange/reorganize the comments into big categories such as thesis, transitions, structure, support, evidence, citation, etc. The categories/related ideas/themes can also be customized to fit one’s needs. Think: if there are any connections or relations between feedback from different reviewers? Do any common patterns/themes stand out particularly? For example, if one notices that a few comments from different reviewers are about the transitions in different paragraphs of the essay, one may need to revisit some common strategies for making transitions and go back to examine the transitions again. Gathering all the comments together in one place and organizing them according to themes offer a global view on the feedback from bottom up and allows one to identify patterns that may not be easily recognized when the feedback is from different sources.

After organizing and thinking about feedback in the affinity diagram, the next step is to come up with “insights for each theme”/pattern. The “insight” is to write out a specific “next step” for how to improve the writing. The specific insights should be based on the feedback, evidence and analysis, and the goals and criteria of the writing assignment. They also should lead to direct actions to revise. When writing insights, one may follow a template like this: It was observed that xx out of xx reviewers think that ______________ (description of the feedback). This means that ______________ (analysis of the feedback). An insight based on the theme/pattern is: ______________ (what to do next). One should also prioritize key elements in revising. Revising thesis should be prior to revising transitions, as transitions may need to change according to the revised thesis. Grammar and sentence editing should be the last step because high-order concerns will lead to sentence-level changes. Some insights “should be considered a Priority Zero,” meaning “they must be fixed” for your essay, such as having an argument, not a descriptive statement for writing an argumentative essay.

In this article, I focus on peer review, one of the teaching strategies for revision by tracking the changes in two editions of Bean’s book, and by reflecting on my own teaching experience. I’m hoping the above tips from another industry that practices peer review—reading feedback, creating affinity diagrams, drawing insights from feedback—can offer some practical steps for students to translate the feedback into effective revision and complement Bean’s wonderful suggestions for peer review.

Works Cited

Bean, John C. Engaging Ideas. 2nd Edition, Jossey Bass, 2011.

Bean, John C., and Dan Melzer. Engaging Ideas. 3rd Edition, Jossey Bass, 2021.

Google UX Design Professional Certificate. Coursera.

Recapping: Avoiding Plagiarism Workshop

Last Thursday’s workshop on “Avoiding Plagiarism” brought out a fantastic showing of professors, for one of our most attended workshops yet! Thank you to all those who were able to make it, for those of you who weren’t, here’s a little recap:

No professor wants to deal with plagiarism (the disappointment! The bureaucracy! The uncomfortable conversations with a student!), this workshop takes as its premise that it is possible for professors to take steps to prevent plagiarism before it occurs! In particular, here at WAC we believe that often plagiarism occurs because a student hasn’t fully understood what counts as plagiarism (and we saw during our workshop that there is a lot of gray area that even professors can disagree on!).

City Tech has a particularly notable policy on academic misconduct, that emphasizes the professor’s responsibility in informing students about plagiarism. It states:

“Students and all others who work with information, ideas, texts, images, music, inventions, and other intellectual property owe their audience and sources accuracy and honesty in using, crediting, and citing sources. As a community of intellectual and professional workers, the College recognizes its responsibility for providing instruction in information literacy and academic integrity, offering models of good practice, and responding vigilantly and appropriately to infractions of academic integrity.” – NYCCT statement on academic integrity (emphasis added)

With this responsibility in mind, the first part of the workshop included a number of activities and handouts that professors can use to assist them in raising awareness about plagiarism in their classrooms. Many of us have used these strategies in our own classes and have found them particularly helpful.

Crafting Assignments to Avoid Temptation to Plagarize

Student plagiarism can have many different causes. Another prominent one we’ve found- that can easily be targeted!- is a lack of confidence, or difficulty with time management. The pedagogical tool of scaffolding can be an invaluable resource for creating assignments that develop students’ confidence and encouraging time management skills. Scaffolding, as many of you know, emphasizes building towards larger projects, step by step. This graduated nature of scaffolded assignments helps students from feeling overwhelmed by a large term paper, and feeling tempted to go online and download a preexisting one.

In the workshop we examined how a scaffolded assignment schedule helps both develop students confidence and promotes working in increments rather than leaving everything for the night before.

In addition to scaffolded assignments, designing assignments with a unique or contemporary twist can help students develop an interest in the work, and also mitigates the temptation to hand in something they found online.

beyonce

For example- one sociology professor has her students write an analysis of Marx’s notion of estranged labor, but asks students to argue whether or not Beyonce could be considered “alienated”. An English professor teaches The Crucible and has students create a podcast in the style of the extremely popular “Serial”.

CrucibleHanging

Both Marx’s notion of “estranged labor” and The Crucible are certainly topics which students could find a wealth of prefabricated, rote essays to pilfer from on the internet, but these alternative assignments seek to engage students’ interests, and avoid the temptation to hand in a preexisting essay by shaking things up a bit. As an extra bonus for professors , these types of assignments can be more interesting to read and grade as students really can let their passions shine through!!

serial

Professors in attendance were encouraged to think up some different and unique assignments they could design to get students thinking through the core concepts of their class. One electrical engineering professor designed an assignment where students would have to calculate the amount of electricity needed to power a Beyonce concert!

concert 2

Do you have any unique assignments that have been particularly successful? We’d love to hear see in the comments below!

Of course, not all assignments have to be unique and scaffolded, we encourage professors to try out a variety of different tactics that might work best for their needs.

Be sure not to miss our next workshop:

  • Effective Grading and Minimal Marking
    • Thursday, November 19, 2015
      • 1:00-2:15pm
    • Room: Namm 1005
    • Free lunch and coffee!

 

*If you would like to see the full workshop, slides are available for download here

The Benefits of Peer Review

As I finalize and submit my class grades, I tend to reflect on class components that worked well and didn’t work so well. As I reflect, I often create a list of strengths and weaknesses for the course and note the chapters and concepts that students had the most difficulty with.

When reading student exam responses, I often find myself frustrated with the fact that a large number of students still had difficulties grasping certain core concepts, even though I felt that I had covered the topic adequately in my lectures and assignments. Over the years I realized that in order to understand certain complex concepts students need something that I can’t provide myself: their critical engagement. I have previously discussed the benefits of in-class exercises to promote critical thinking, and these types of exercises (as well as writing assignments) can be further expanded to include a helpful peer-review component.

As professors and academics scholars we learn so much from our peers. Peer reviews can provide us with some of the most insightful feedback, and help us develop stronger work. The American Psychological Association (APA), for example, reports that a majority of peer-reviewed articles are accepted with contingencies. This means that papers are accepted with the agreement that the authors improve or clarify several aspects of their work based on feedback from peers. So why is it that we, who benefit so greatly from the peer-review process ourselves, don’t utilize this resource more when helping our students grow as professionals?

There are several benefits that students may gain. It can be helpful to communicate these to students as well, so that they know why they are being asked to review their peer’s work.

  1. Students often learn more from people at their level of learning.

Professors feel responsible for their students’ learning, which is great! However, it is okay to step back and have students learn more independently; allowing the student to discover knowledge for him or herself can be very powerful. And one way that many students learn well, is from one another (Boud, Cohen, & Sampson, 2014).

  1. Peer-review can build comfort and normalcy around receiving constructive feedback.

Being able to listen to others and utilize feedback effectively is important to future career success. When writing recommendation letters for students, I’ve noticed that many graduate programs ask that we discuss the student’s openness to feedback, as this is central to student success. To better serve our students, it is thus important that we help them develop their ability to effectively work with constructive criticism early on. With this, it is also important to monitor that feedback remains constructive. The teacher can assist in this by developing a guided peer-review worksheet and by discussing acceptable feedback in class.

  1. Providing peer feedback can strengthen students’ own work.

By providing feedback to peers, students often begin to think more flexibly about their own writing. For example, by taking the grader’s perspective, a student might start to better understand that the writer isn’t always successful in communicating something clearly. This experience may then promote the student’s ability to take the grader’s perspective when they review their own work before submitting it for a grade.

Additionally, by having students review each other’s writing assignments, they have to divide the paper writing process up into at least two stages: the draft and final paper. Scaffolding assignments in this way is known to lead to more critical engagement and learning (Bean, 2011).

  1. Peer review can save grading time.

This can be a nice added benefit! However, implementing a peer-review component may not immediately save you time. It is important to think about the design of the peer-review activity, so that it is designed to integrate well with your current grading system. If you feel that you need assistance with this, don’t hesitate to contact one of our writing fellows for guidance.

How do you develop a strong peer-review exercise?

It is important to lead the students through their own discoveries. This means that you as the teacher want to think about the cognitive steps students need to take in order to come to the appropriate conclusions about the assignment they are responding to. This will facilitate their ability to provide constructive feedback and accurate peer grades.

Here is an example of a peer-review exercise for an annotated bibliography assignment. In this exercise, the teacher uses specific questions to help the student focus on the most important aspects of the assignment: the peer’s clarity in communicating ideas and the quality of the research methods they used.

As you update your class syllabi this summer and think about improving coverage of certain topics, consider developing a peer-review component!

References

Bean, J. C. (2011). Engaging ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom. John Wiley & Sons.

Boud, D., Cohen, R., & Sampson, J. (Eds.). (2014). Peer learning in higher education: Learning from and with each other. Routledge.

Tailoring Expectations

One useful perspective-realignment I’ve found useful raising to faculty, particularly those who don’t teach strictly “English,” is the that many assignments have implicit writing assumptions which must be made explicit.  It is difficult sometimes to see the necessity of writing underlying even ostensibly non-“expressive,” or technical, assignments.  This sounds like an easy, or superficial suggestion, but consider, for instance, courses which integrate design and writing in an integrative and mutually-informing manner — in order to produce any sort of finished, visually appealing document, the writing present within must be coherent and “finished;” yet, this expectation is often only alluded to tacitly.  Further, even if one is actively grading “writing,” it is often difficult to break down this “writing” requirement into constitutive units the students can follow, or knowingly deal with on an individual, then total, basis. As an added benefit, when students are made more conscious about articulation, even in a small way regarding a tangible quality of writing, it makes them more aware of the total flow and logic of their work.  (These tangible qualities are then able to compound, and inform one another.)

One possible suggestion:  Perhaps (even as a sort of pedagogical thought experiment), try outlining one or two explicit qualities of writing to be graded, or paid attention to, in a non explicitly English or even humanities assignment.  As we often discuss at WAC, try to scaffold, or otherwise anticipate the exact skill you would like them to exercise by introducing it earlier than the exact moment you wish them to recall or produce it.  Then, see if, for example, should you ask them to pay attention to something like topic sentences, or even choosing neutral, or discipline-specific jargon for the assignment, whether the overall clarity of thought, and quality of product produced, improves.

This means of “tailoring” expectations, or honing in on required, but implicit, qualities of writing in assignments, is also transferable to other areas, such as peer review.  Rather than asking students to holistically grade entire documents for “quality” or “followability,” try to hone in on two or three qualities (perhaps even breaking a “thesis” question down into a subcategory or two), and set firmly-defined timelines for how long students spend on each portion.  This means of narrowing the scope of the students’ attention will likely improve the sharpness and nuance of the skills paid attention to, and overall improve the logic, thinking, and argument of the writing, and writing-reliant aptitudes, required.

Writing to Learn

As the fall semester of 2013 draws to a close, it is useful to reflect on what we have accomplished over the course of the semester. We the Writing Across the Curriculum fellows have led three main faculty workshops since September: Effective Assignment Design, Peer Review, and Effective Grading. Despite the three varied topics of these workshops, they share a common thread, which is the WAC philosophy of “writing to learn,” and in addition, their content overlaps nicely.

In order to highlight WAC principles, I wish to focus on one particular aspect of the effective grading strategies that Jake Cohen and I discussed in our workshop on Tuesday, December 12 (the last of the semester). We went over some techniques to improve student writing and work, most of which also incidentally result in reduced grading time, which is always welcome, especially at this end-of-semester crunch grading time. To view our workshop slides, please click here, and check out the handout. (You can also visit this page to download documents from all of our workshops.) We discussed minimal marking, supportive responding when writing comments on student papers, rubrics, and planning assignments ahead of time to make grading more efficient. This last category is closely related to the two previous workshops from this semester: assignment design, clearly, and also peer review, in that having students assess each others’ work can save time, and greatly improve student writing.

This assignment design category is also the “one particular aspect” that I choose to elaborate on for this post. Among the several techniques we suggested for planning ahead to make assignments more “gradable,” one sticks out as being particularly WAC-esque: the uncollected writing assignment. The value of this notion, which is generally under-utilized by faculty in all departments, is two-fold: It is easy to see how uncollected assignments decrease the overall amount of time we spend grading work, of course, but why assign them at all? The answer lies in the foundation of WAC philosophy, which is that people learn by doing—and more specifically, by writing. So, what kind of uncollected writing do we recommend you assign, how do you enforce such assignments without collecting them, and, finally, how do students “learn by writing”?

One of the best illustrations of this concept is provided eloquently by Toby Fulwiler in “Why We Teach Writing in the First Place”: “Writing the thought on paper objectifie[s] the thought in the world… [which] even happens when I write out a grocery list—when I write down ‘eggs’ I quickly see that I also need ‘bacon.’ And so on” (127). This concept works well for professors across the curricula: Think about assigning a five-minute, in-class free-write asking students to describe course content covered in the past month/week/hour, by way of ensuring that they can articulate it well for whatever type of exam they have coming up, and by way of allowing them to discover holes in their understanding of what you have covered so far. If you are concerned that they won’t oblige the assignment without the potential for reward, then you can choose, for example, to select three at random to read aloud in class, or to be posted on your Blackboard/OpenLab page that same evening.

We hope that those who incorporate this technique will ultimately find that the grading process of the final papers you assign will be ameliorated, in that the students have now had a chance to “practice” or “train” for the final writing process, something akin to athletes who could never run a marathon without similar training, without you having been required to grade an intermediary draft. Ideally, as students come across “holes” in their own comprehension of your course content, they may come to you with more questions, or make better use of your office hours. I know that they will arrive at a deeper understanding of your course material in the same way that I have done regarding WAC philosophy, in the process of writing out this blog post.

Happy Holidays!

Workshop Recap: Utilizing Peer Review in the Class Curriculum, November 12, 2013

Earlier this year, WAC fellow Zak Aidala wrote an excellent blog post about using peer review in the college classroom. On November 12, WAC fellows Melanie Lorek and Heather Zuber led a full workshop expanding on peer review, introducing us to their “eight great” strategies to make peer review work for faculty in the classroom. The PowerPoint and handouts from the workshop are provided below.

Workshop participants also got a chance to role play a peer review ourselves, letting us see firsthand the benefits of using peer review. Heather and Melanie also helped to dispel a variety of myths and misconceptions about peer review.

After acknowledging some common misconceptions about peer review, the participants brainstormed a list of benefits and advantages to peer review. Most groups came up with a similar list: students feel less pressure when being reviewed by their peers, students are forced to reflect on their writing, students are encouraged to feel autonomy as writers, it saves time for the professor when assessing writing, and it allows shy students to participate.

Melanie and Heather then introduced their “eight great strategies” for effective peer review. While you can look in detail at all of them in the PowerPoint presentation below, their first and most important aspect was to focus on only a single feature of a draft, such as the thesis, the supporting evidence, topic sentences, etc., rather than having reviewers look at global revisions. This helps to avoid aimless peer reviewing, or confused questions of “what am I supposed to do?” by the students. Using a handout with specific instructions and questions aids this process.

We then put this theory into practice, breaking into groups and peer reviewing only the thesis of a few different student papers. While each paper had a variety of higher- and lower-order concerns, by focusing on just one aspect, the thesis, we were able to give good recommendations to the hypothetical writer. Heather and Melanie also provided us all with samples of handouts that we might use for a variety of different peer review activities.

Click here for the PowerPoint, and click here for the handouts.

Did you attend the workshop? What worked for you? What would you have found more useful? Feel free to comment below!

Our next workshop is on Effective Grading Techniques, and will be held on December 10, 2013, at 1pm in Namm 226. Lunch will be provided!