What makes a Cars.com comparison test run

If you’ve ever shopped for a car in the United States, chances are you’ve read or seen a comparison test. They’re the mainstays of many automotive publications, both in print and online, and just as vehicles have grown more complicated, so too have the tests.

We asked Joe Wiesenfelter, the executive editor of Cars.com, one of America’s most popular automotive sites, to explain what goes into their process. (And yes, the fact that the 2018 Volkswagen Tiguan and the 2018 Volkswagen Atlas recently came out on top in two Cars.com SUV comparisons did catch our attention.)

Q: How many comparisons does Cars.com conduct every year?

Joe Wiesenfelter: We’re currently at a rate of 6-8 per year including pickuptrucks.com, a subsidiary.

Q:  Who do you use as evaluators?

JW: We usually use three of our editors plus a guest consumer judge. Unfortunately the consumer was a no-show for the Compact SUV Challenge in which the Tiguan competed, but we typically find someone who’s in market for the vehicle type or at least owns or has an interest in a similar vehicle. The guest judge for our Luxury Sport Sedan Challenge, which the 2018 Audi A4 won, was the owner of an older BMW 3 Series.

Our editors have an average of 12 years’ experience as car reviewers; actually, most have 11 or 12 years. The least tenured has been at it for seven years, and the most more than 20. Since 1997 we have built a staff of journalists that are also car fans and experts, not the other way around. The fairness and ability to communicate well is foundational; it’s who they are.

Usually we’re joined by a producer and videographer from PBS’ MotorWeek, a longtime partner of ours. Though they don’t serve as judges for the Cars.com Challenge, they take advantage of the vehicles and typically ask us to share impressions on camera for their broadcast.

Q: How do you determine the criteria? What do you tell the judges to look for?

JW: To borrow from the published results: Three judges individually awarded points in 12 categories: interior quality, front-seat comfort, backseat comfort, cargo storage, in-cabin storage, handling, powertrain, ride quality, noise, visibility, worth the money and multimedia — the latter a category that accounts for the touchscreen-based interfaces that are, more than ever, the means to activate and adjust fundamental features of the vehicle itself, not simply ways of controlling audio sources and navigation systems.

Each model is also awarded points for the advanced active safety features with which the test vehicle is equipped as well as for its grades in our Cars.com Car Seat Check, which gauges the accommodation of various child-safety seats.

To elaborate, the above categories are pretty common across our Challenges, but there is variation based on the vehicle type. We don’t do zero-to-60 testing for SUVs, but we have for sport sedans, hot hatches, muscle cars and the like, sometimes with a road course as well. Fuel economy testing also comes and goes depending on the vehicle type and consumer mood.

We’re always trying to whittle our conclusions down to what matters most, so if the scoring from a predetermined category proves inconsequential, it doesn’t get mentioned in our reporting. Likewise, if we find on location that a truly distinguishing characteristic or feature isn’t accounted for in our scoresheets, we’ll add it.

We always attempt to test a vehicle as it’s meant to be used, which is why we use track testing when called for, as mentioned above, we stuff cargo areas full of luggage, bicycles or widescreen TV boxes.1 We install a variety of child-safety seats to see how accommodating the vehicles are. We use pickup trucks to tow trailers of varying weights and often include an off-road component.

What we try to do is share the information consumers can’t get by looking at specifications or feature lists. Just because a feature is present doesn’t mean it’s well executed, so a lot of our effort goes toward revealing how well features that look the same on paper actually perform in the real world, from transmissions to multimedia systems or driver aids like lane-centering steering.

We do comment on interior quality, because it plays into a vehicle’s value, but we tend to leave most aesthetic issues off the table and the scoresheet. A consumer can draw his or her own conclusion about how a vehicle looks. We use our opportunity to rate it on what’s less obvious.

Q: How long does one take to conduct? What kind of work goes into it?

JW: Apart from planning and all the writing and production that follows, the Challenge usually takes four to seven days, depending on the vehicle type(s) and location. Pickups tend to take the whole week. When winter hits Chicago, we hold them elsewhere. We may have a dozen people on location at a given time – sometimes more, sometimes less, depending on photography, video and testing schedules. It’s a heavy lift, sometimes literally, because the sacks of rock salt used to weigh down pickup truck beds for testing aren’t going to move themselves.

Suffice it to say we’re extremely busy and the days are long, but we’re trying to squeeze as many impressions and as much content as possible out of having closely matched competing models all in the same place.

Q: What value do such tests have for your readers?

JW: Competitive comparisons are part of much of our content. Even our model-specific reviews address the competitive set and attempt to compare different aspects. It helps shoppers determine what’s most important to them and can lead them to cross-shop a vehicle they wouldn’t have considered. Our Challenges are an extreme example of this approach. Nothing is as instructive as driving vehicles back to back, or sitting in one after the other. Throw in the relatively close pricing and equipment we require for Challenges, and it’s extraordinarily illuminating. This is what we attempt to share with consumers. Apart from an auto mall, dealerships are designed for the opposite: to show you one brand to the exclusion of others. For consumers, our Challenges are the next best thing to an auto show.

The value is clear in the demand for the Challenges, which is high. Site visitors gobble them up. It’s clear in the traffic and engagement.