Short Version: The GRE writing rubric is a hugely influential source in defining what good writing is. The rubric is good, but sometimes tough to use because it’s a holistic rubric with undefined terms. To get a better understanding of those terms, it’s useful to examine the studies the GRE uses to establish the significance of the test. In the study here, we see a list of 29 things that a ton of faculty from across the country say are important for writing well in grad school. They are shown to match up well with the GRE rubric elements. Anyone looking to help others understand what good writing looks like should take note.
There are three big sources for defining good writing: style guides, readability formulas, and rubrics. Let’s look at one very influential rubric–the GRE Writing Rubric–and one study used to establish its validity.
The GRE Writing Rubric
The GRE is the test you take if you want to go to grad school. More than half a million people take it each year and they all begin with the writing section. Teachers and students around the world examine the rubric and try to follow its lessons.
So, yeah, kind of a big deal to a lot of people. Style guides and readability formulas have a big impact, but really only the rubrics for a few other tests rival the impact of the GRE writing rubric.
Here’s how ETS–the makers of the GRE–describe the top two scores for the argument task:
In holistic rubrics like this, scorers consider all the elements together. This can cause problems. Sometimes an element of the essay is drastically worse/better than other elements. Like, you can imagine an essay that is well organized and uses standard English, but is terribly reasoned. (See the editorial section of any newspaper.) Scorers need to feel their way through such a situation. We can contrast holistic rubrics with analytic rubrics in which each criterion is scored and then you add up the points. (My Clear Writing Rubric is like that.)
Another problematic part of the rubric is that some of the terms aren’t clearly defined. Reasonable people could debate what they mean.
For example: “develops ideas cogently, organizes them logically and connects them with clear transitions”.
“Cogently” means in a convincing or clear way. But what makes something convincing? And how about “clear”? If you’ve ever understood something that someone else didn’t get, then you know that clear isn’t always clear.
And which ways count as logical for organizing? Which are illogical? Is there middle ground? And for “clear transitions”, do people need to use transitions or do things just need to flow well?
I could go on, but I do think there is are reasonable answers to all those questions. Some of those answers lie in style guides. For particularly nuanced questions, the answer is eventually: “You know it when you see it.” That’s not very helpful if you don’t know it when you see it. In the end, opinions will vary. So, it matters a great deal who’s doing the seeing.
The test makers have published several studies that let us know who’s doing the seeing and what they’re thinking. The studies help us understand the rubric, establish its significance, and flesh out its undefined terms.
Study On The Significance Of The GRE Writing Section
Here’s a link to one study. In it the GRE people justify the rubric by showing that the skills described on the GRE writing rubric match those needed for success as a grad student.
The goal was to figure out whether the rubric elements match up with the writing skills grad students need. They used writing experts and college faculty to come up with a list of 39 things that seemed important for grad student writing.
Then they asked 1529 faculty members at a diverse set of schools and in a diverse set of departments if those things really matter. Averaging the responses, 29 of the 39 items do matter. The ten that got thrown out either weren’t broadly important or were rather unimportant in a specific field of study. (Like if all the biologists said something didn’t matter, then it got thrown out.)
These are the 29 things college profs say are important for success as a grad student:
(The ratings are out of 5.)
That was step one: figuring out what mattered to college faculty. Next, they wanted to find out if the rubric reflected those things.
They asked some experts if those 29 things match up with the 9 rubric elements. For the most part, the experts said they did. Between 4 and 21 of of the 29 task statements matched up with each of the 9 rubric elements.
Here are the same rubric elements from above as they’re presented in this study:
So, the GRE rubric is influential and it’s backed up by matching the skills faculty across America say is important.
Last point: For any teacher of writing, I think this list is an invaluable source on what matters in writing. We’re seeing what academic America thinks matters in writing.