|
|
|
|
moondog
Jan 16, 2006, 6:02 PM
Post #51 of 60
(4603 views)
Shortcut
Registered: Nov 25, 2002
Posts: 196
|
In reply to: I'm not going to make any snap decisions, but I've begun to consider some changes to the featured review process. Here's what I'm currently considering: 1. Adding a disclosure statement atop every editorial review.
- How the gear was obtained, who paid for it, and what became of it after the review period
- How it was tested and by whom, under what conditions, with what ropes, for how long, etc.
- Whether other equipment in the same category was tested along with this gear.
- Other stuff I haven't thought of for the moment...
The standard for the entire review process must be published and easily found/linked on every gear review.
In reply to: 2. Eliminating the "Users review their stuff" section. (I could use the FP space, anyway.)
If the users were held to some minimum standards, this section could gain some relevance. Some reviewers disclose similar products they have used, and others don't. Without a published standard governing the review process, this section is of limited value.
In reply to: 3. Eliminating the "No negative features" policy.
Excellent move - please make it. All gear reviews should be treated equally - regardless of the opinions rendered. A few years ago, a major climbing gear manufacturer pulled its ads from a major climbing pub as a result of "less than favorable" product reviews.
In reply to: (Bear in mind, I'm not willing at the moment to eliminate the free gear, because we have no other way to get the gear and I have no other way to compensate reviewers. Ceasing that practice would be the functional equivalent to eliminating editorial reviews entirely.)
That's fine. As long as folks are fully informed of how the process works. I think the only way the appearance of (mild) corruption in the process can be mitigated is by paying full retail for the gear and publishing a minimum standard of how the reviews are conducted.
OTOH...
..."Pay per view" reviews could be funded in various ways, all of which involve soliciting funds directly from the users of the site (à la CU). It would be interesting to offer this as a trial - bet we would soon see a marked difference in the tone of the reviews. The "paid" reviewers should be different from the unpaid to avoid apparent conflict. All gear reviewed "pay per view" would be tested on a comparison basis by multiple reviewers with all major offerings in the category being tested. Without comparison, the relevance of the review is significantly reduced.
|
|
|
|
|
crotch
Jan 16, 2006, 6:16 PM
Post #52 of 60
(4603 views)
Shortcut
Registered: Jan 16, 2003
Posts: 1277
|
In reply to: Here's what I'm currently considering: 1. Adding a disclosure statement atop every editorial review. 2. Eliminating the "Users review their stuff" section. (I could use the FP space, anyway.) 3. Eliminating the "No negative features" policy. What are your thoughts on these? If you were me, what else might you consider?
#1 - Good on you.
#3 - Jay, I've read those reviews you linked earlier and sure enough you guys do put out negative reviews. I stand corrected. I think featured reviews should be about significant/hyped/awaited products, and the decision to feature should be made before the product is received and tested. Good or suck, stick with that decision.
Do any of the gear reviewers ever worry about being "cut off"? Do you ever think that if you slam a product, that same company won't send you another freebie to test in the future? I'm just extrapolating from how I think I might feel under the circumstances.
Let's say that Yvon Salathe sends me a haulbag to test. Turns out I think it's just average, but I know he's going to be putting out a new portaledge this summer that I'd love to get my hands on. I think I'd be tempted to take some of the sting out of the haulbag review (you know, emphasize the good points) in the hope that I'd get a free portaledge down the line. Can y'all honestly say this never crosses your minds?
Jay, I'm impressed with your openness and willingness to adopt change. Ithink the job you guys are doing is a cut above the mags. I'd like to see things get even better-more like consumer reports and less like Rock and Ice.
I also agree with you that most of the stuff that comes to market is pretty good. With that in mind, what is the true range of the rating scale? Let's use carabiners as an example. Should the average carabiner get a 3-star rating? What about among the different classes of carabiners? Should the average ultralight carabiner get 3 stars? How far above and beyond should a 5-star product be? How much of a piece of crap must something be to get 1 star? What about 2 stars?
|
|
|
|
|
moondog
Jan 16, 2006, 6:30 PM
Post #53 of 60
(4603 views)
Shortcut
Registered: Nov 25, 2002
Posts: 196
|
In reply to: Jay, I'm impressed with your openness and willingness to adopt change. Ithink the job you guys are doing is a cut above the mags. I'd like to see things get even better-more like consumer reports and less like Rock and Ice.
Ditto. I'm excited to see this thread get so much attention and feedback.
|
|
|
|
|
dingus
Jan 16, 2006, 6:52 PM
Post #54 of 60
(4603 views)
Shortcut
Registered: Dec 16, 2002
Posts: 17398
|
When you (or RC.COM) starts paying for these words, then you get to say what's what. Until then you are subject to the whim of the reviewer.
I don't see how it could be more simple.
If you want a structured environment with a strong editorial presence... get out the checkbook.
DMT
|
|
|
|
|
dingus
Jan 16, 2006, 7:00 PM
Post #55 of 60
(4603 views)
Shortcut
Registered: Dec 16, 2002
Posts: 17398
|
In reply to: If I decide, right here in this thread, to incorporate a change, it'll get done.
I reckon I'm starting to take this whole thread personally.
DMT
|
|
|
|
|
brent_e
Jan 16, 2006, 7:19 PM
Post #56 of 60
(4603 views)
Shortcut
Registered: Dec 15, 2004
Posts: 5111
|
In reply to: In the spirit of openness, should gear reviewers voluntarily disclose whether they paid out of their pocket for the gear they are reviewing, or whether they were given the gear by the manufacturer? Is there a difference between a gear review from someone who: 1) Purchased the gear with their own money 2) Was given the gear for free by the manufacturer I'd argue that a review from someone who was given free gear is an advertisement in sheep's clothing. Can anyone point to a bad gear review from a reviewer who got the gear for free? Are gear reviews in mags anything other than ads? What about the stuff posted here?
i suppose, Crotch, that you could try and read between the lines. Maybe compare the review to another by the same person. If they were really happy about one, and only so so about another perhaps that will shed some light.
I think that's the nice thing about the gear section on here. most of the stuff that people are talking about they have purchased themselves. Doesn't mean that they know what they're talking about, i guess.
Brent
|
|
|
|
|
vegastradguy
Jan 17, 2006, 12:55 AM
Post #57 of 60
(4603 views)
Shortcut
Registered: Aug 28, 2002
Posts: 5919
|
man, this thread would come into existence right as my gear review for the atc-guide is on deck! yowser!
well, i'm in agreement with much of what has been posted so far- the idea of full disclosure makes me feel better about the reviews.
i'm on the fence about the positive/negative featured review. on the one hand, it seems like the site is playing favorites, but on the other, a featured review seems like it should be something worth displaying on the front page. obviously if its the big new thing and it sucks- that should go up front and center, because folks will be looking for the review on it anyway....
i'll also say (as others have), i've never been told one way or the other what to say about a piece of gear i've received from rc.com- jay just tells me when he needs it by and that's about all the direction i get. the rest is up to me.
|
|
|
|
|
j_ung
Jan 17, 2006, 4:56 PM
Post #58 of 60
(4603 views)
Shortcut
Registered: Nov 21, 2003
Posts: 18690
|
In reply to: In reply to: Here's what I'm currently considering: 1. Adding a disclosure statement atop every editorial review. 2. Eliminating the "Users review their stuff" section. (I could use the FP space, anyway.) 3. Eliminating the "No negative features" policy. What are your thoughts on these? If you were me, what else might you consider? #3 - I think featured reviews should be about significant/hyped/awaited products, and the decision to feature should be made before the product is received and tested. Good or suck, stick with that decision.
That's exactly what I'm considering with #3. Make the decision in advance. I think there's enough info available -- even before a product hits the market -- to do just that.
In reply to: Do any of the gear reviewers ever worry about being "cut off"? Do you ever think that if you slam a product, that same company won't send you another freebie to test in the future? I'm just extrapolating from how I think I might feel under the circumstances... Can y'all honestly say this never crosses your minds?
That's a good question. I won't speak for the others on this, but I have worried about that, yes, although more for the site's benefit than my own. I have other options and relationships that don't involve RC.com at all, and even though I get some gear free, I almost always make a point of paying retail at my local shop for almost everything else.
In reply to: Jay, I'm impressed with your openness and willingness to adopt change.
Thanks crotch. I'll take all the compliments I can get.
(Hmmm... that sounds funny when read aloud. "Thanks crotch!" :lol: )
In reply to: Ditto. I'm excited to see this thread get so much attention and feedback.
And that's the best compliment of all. Thanks moondog.
In reply to: I also agree with you that most of the stuff that comes to market is pretty good. With that in mind, what is the true range of the rating scale? Let's use carabiners as an example. Should the average carabiner get a 3-star rating? What about among the different classes of carabiners? Should the average ultralight carabiner get 3 stars? How far above and beyond should a 5-star product be? How much of a piece of crap must something be to get 1 star? What about 2 stars?
It's pretty subjective, isn't it? That little biner scale works best on an average, rather than one person's opinion. To boulderer, a Camelot is a $75 keychain. Personally, I don't like the biner scale so much, since, as I mentioned before, every product has good and bad points. Perhaps another suggestion might be to eliminate that scale for editorial reviews and let the user average take complete control.
In reply to: I reckon I'm starting to take this whole thread personally.
Don't dingus! (Hmmm... not as funny as "Thanks crotch," but still...) We're getting valuable feedback that we can use to make our reviews even better. Yours, BTW, are top notch. I picked all you guys because I value your opinions and style -- that hasn't changed -- and a lot of people, including here in this thread, agree with me.
In reply to: there should be a standard for the disclosure statement. For example, it should include: - How the gear was obtained, who paid for it, and what became of it after the review period - How it was tested and by whom, under what conditions, with what ropes, for how long, etc. - Whether other equipment in the same category was tested along with this gear.
Wellll... I agree that all of the above should be included, but I like the second two suggestions in the body of the review. I'm sure we've missed a detail or two from time to time along those lines, but we do make an effort to include that info. The first of your list, I absolutely agree -- it should be in the disclosure statement. In fact, I've pretty much made that decision; it will be there from now on and once I write it, I'll start adding it to previous editorial reviews, too.
|
|
|
|
|
|
|
|
j_ung
Jan 20, 2006, 5:39 PM
Post #60 of 60
(4603 views)
Shortcut
Registered: Nov 21, 2003
Posts: 18690
|
j_ung has locked this thread.
|
|
|
|
|
|
Re: Honesty in Gear Reviews
[
