okay, we are in sga, therefore, we know of the scientific method

Thought. Random thought. Actually fairly undoable thought.

In another lj, there's a fascinating meta about correlation between comment number and how good a story is. I really wont' rehash that here, because it will only make me cite the stories I hate most that had high comment counts and send me into blind rages. It's a problem. I deal with it.

So I'm trying to figure out how would a true double blind work in fandom as it stands. The only way I can see that would level all playing fields--and even then, I'm talking a severe difference in level, but close enough--would be a double blind. Anonymous authors, screened comments--and a single writing prompt. Because while I buy that quality of fic has something to do with quantity of feedback--I think it's not as much as we--and I mean, me, the writer--always hopes it will be.

Okay, just thinking. A double blind, if you wanted to test the hypothesis -- a fic with a lot of feedback is (usually) better or at least far more publicly accessible than one that has a lower one. What would be the constants?

Comments

[that_mireille]

My first question would be, how would you define "better"? Because you'd need an objective, quantifiable measure of goodness in fiction, wouldn't you?

[seperis.livejournal.com]

Better as defined in this particular instance is writing, stripped of author name, in single pairing or gen format, where the entirety of the story rests on the authors' ability to tell the story well.

Now in that, we'd be left with style preferences, pov preferences, and even, with a single writing prompt, kink preferences.

See, that's why I'm not like, opening this as a challenge or anything. I can't even out all all the variables. For the purposes of this, it would have to come down to the ability of this story to appeal almost entirely on the basis of how the writer writes.

[ladycat777.livejournal.com]

Would something like the anonymous_sga (I have no idea of the exact journals name, sorry) work as a test-case? The stories are posted anonymously, the authors revealed a few weeks later.

Also -- a double-blind involving fandom? I wish you so much luck in trying to achieve that because I have no idea. I think we have too many ways of communicating, not just lj but through chat programs and email and the phone... there are too many avenues to invalidate it.

[seperis.livejournal.com]

It *might*--but we can still see the number of commetns, and that's anotehr form of influence on readership. For purity, it would have to also be entirely screened.

And exactly. I'd have to leave at least part of it purely up to author honesty in not communicating to anyone which story they were doing.

[nopejr.livejournal.com]

What would be the constants?

Fixed length, third person limited, past tense, genfic in a fixed fandom with pre-specified characters, using authors who have both written that and had good feedback before.

Although, how do you define better?

[seperis.livejournal.com]

In this particular instance--adn only in this particular instnace, not generalized--the story appealing to the largest number of people totally resting on the way the author wrote the story.

[amalthia]

the sga_santa is almost like a double blind in a way because no one knows who is posting what and you can kind of see which stories appealed to the most readers based on the feedback.

[seperis.livejournal.com]

Right. And that's a variable--does the number of comments already received influence readership? So it would have to remain screened.

[blade-girl.livejournal.com]

Hmm. I agree with previous posters that if your goal is to see whether more comments = better fic, you would need to define what you mean by "better," and that achieving double-blind is pretty much impossible, because there would be no way at all of knowing whether one or more authors had told other people which story was theirs, and if those people had told people, etc., etc.

I like the idea of what you're trying to accomplish, though. It would be really nice to definitively determine how much impact such things as name recognition, popularity, hot-buttons in fic content, etc., really have. But how can you ever be certain that you've filtered out the name recognition and popularity?

[seperis.livejournal.com]

In this case only--and for the purposes of my imaginary double blind--it would totally rest on the author's ability to tell the story and the widest number of people who enjoyed it. But only based on that.

[suzvoy.livejournal.com]

because it will only make me cite the stories I hate most that had high comment counts and send me into blind rages. It's a problem. I deal with it.

*nods verrrrrrrrrrrrrrry enthusiastically*

[seperis.livejournal.com]

*grins*

p.s.

[amalthia]

also, good is kind of subjective. Even among people who read a lot there are sometimes disagreements on the quality of the story.

Re: p.s.

[seperis.livejournal.com]

yes, it is.

but for teh purposes of this particular exercise, removing all other variables, and assumign if someone liked the story, they'd comment--and this is a lot of assumptions--the 'success' of the story would rest solely on teh authors' ability to tell it. Basically, on *them*. So better--in this case, and only this case--would be based on how broadly it appealed.

[kageygirl.livejournal.com]

Here's a problem--in order to approximate the typical behavior of readers/commenters, they would have to not know that there was any kind of comment-count thing going on, or they might comment/not comment more often than they typically would.

And, yeah, who would judge "better"? You couldn't go by comment count, of course.

[seperis.livejournal.com]

Actually, in my imaginary double blind, comments *would*.

Okay, the base idea is this--do higher comment counts equal to a better story y/n? We don't know because of the variables--author name, author following, pairing, heck, plotline or kink, the number of comments already received, who recced it. All of these things are influence. So on an ideal double blind--and God knows it would have to be ideal--all of these things are stripped out. No author name, a single story prompt, and no way to see the comments. What would be left is the authors' ability to tell that particualr story, whatever it might be. So--and this is hugely theoretical--people would comment more favorably the broder the appeal of that writer's story, or--and only for the purposes of this particular setup, not in any way in general--and the broader the appeal, theoretically more comments. So we could take from that at very least that a particular author is better *to more people* than to others, which for the purposes of this, would be a better story.

[tacittype.livejournal.com]

Oh, good idea! You're gonna run this, right? I want to play!

*casts mind back to setting up studies...*

The order that the fics are in should be randomised and different for every refresh. And they should probably be stripped of all identifying info until after you hit the link - including title and number of comments so far. So that every piece gets a similar hit count.

The fics should be posted in one go - multi chapter wips seem to get more support, and I'm not going to get into the tactics thereof.

Ideally they should be the same length and equally easy-on-the-eyes, but how would you enforce that? It could be a fun extra variable to play with in the stats. No, I do not get any kind of sick pleasure from playing with stats.

Okay. Maybe once.

Also, ideally the commenters shouldn't know they're participating in a study. But again, how?!?

As an interesting aside, if you do run it, can you set up hit counters (that the reader can't see or know about) so we can see whether there's a difference in hit-rate and comment-rate and how that varies with the story quality? That'd be kinda fun to know.

And, uh, obviously people should be able to rate the stories so there's something to analyse there.

Wow, I just came over as a complete stats weirdo, didn't I? Am actually a bit alarmed by how interested I am in this. Really, I'm alarmed. I have almost no idea how you'd institute it. Leave me alone.


but, um, I'm available for brainstorming

[tacittype.livejournal.com]

I suppose I'm suggesting you need a programmer on board to make up a proper database, fully equipped with randomising and secret comments and counters, and where you're asked to rate the story when you cross off its window.
I kinda wish I a) could program and b) had a spare week!

Als, I agree with other commenters who suggest the pairing/prompt should be constants, but if it's a ratio of hit:comment that you're really wondering about then it's not strictly neccessary. If it's the relationship between comments:quality rating or quality:comment then the actual number of hits shouldn't matter.

[justbreathe80.livejournal.com]

I think about this A LOT, and it's hard, because I really want to write for myself and not care what other people think, or at least not what the masses think, per se, but I'm not always successful. I mean, on some level, we are writing for other people's enjoyment, on some level, and it kind of sucks to spend time on something and not know if people liked it. And comments, in some ways unfortunately, are the only way to TELL if people read and liked your fic.

I will admit to looking at comment counts, especially in fandoms like SGA, because there is SO much fic, and the quality, unless you know the author, can be somewhat hit or miss, so I sometimes look to see if it's been commented on a lot, if I don't have a rec, to know whether or not I should read it. I kind of hate doing that, but I don't have the time to read everything and decide for myself. I know I'm probably missing out on some good stuff as a result, but I don't know what else to do.

On the flip side, I have had a fic in SGA that has gotten LOTS of comments. My very first fic, actually, but subsequently, the comments have been sparser, for fics that *I* think are much, much better than the first one. I'm not sure what to make of that. Is there any way past that? Or around it? I try not to get too worked up about comments, but it bums me out that someone could miss something that I'm really proud of because, for whatever reason, it didn't get a lot of comments.

I think you're right - the only way around this issue is to get rid of comments entirely, and I'm not sure that's the best thing. I mean, I know that I treasure all of the feedback I get on my writing. So I wouldn't want to stop that. Maybe the solution is for people who DO read a lot to rec fics that are outside of the mainstream. I mean, there are certain authors in every fandom who, when they post, will get LOADS of comments and praise and recs. Without fail. I generally don't comment to those people, but try to comment to those who I know aren't going to get a lot. And I try to rec those people too. Maybe that's the answer. I don't know.

*ponders*

[justbreathe80.livejournal.com]

I admit to being intrigued by something where the pairing was constant, maybe, but everything else was up for interpretation, so it couldn't be written off as a pairing preference.

And then anonymous and, yes, screen comments, and see what happens.

I think it might fall out like it does now, but there would be some surprises.

[brownbetty]

Tangentially, I was talking with someone about the problem of concrit: no way to separate those who sincerely want it from those who want to hear, "no, don't change a thing!" and consequently, concrit is basically only given by those with some sort of social disability or people with an axe to grind. I was wondering if some sort of double blind set-up might be the solution there, too.

[seperis.livejournal.com]

Now *that* would be interesting.

[mecurtin.livejournal.com]

I am just here to laugh myself silly at your subject line.

[seperis.livejournal.com]

We can all find it in wikipedia, anyway. *g*

[akacat]

They'd have to all be approximately the same length, and short enough to be read immediately.

Same fandom, same primary character, approximately the same rating.

It would probably help if there was a small team of betas (who are not allowed to comment) who beta'd all the stories. This can be in addition to the author's preferred beta(s). The official team would be there to make sure all the authors have a minimum standard of spelling, grammar, and tense/pov consistency. They can also help the authors tone down any serious 'tells' in their stories.

[wickedwords.livejournal.com]

The story would have to have several basic things the same: mckay/sheppard, first time, 7K-10K words, non-earthside, non-AU. That way there would be room for some plot (and all stories would have to have some plot), world building would be limited to alien cultures and would not be domestic culture. I think that you might get something like what you are interested looking at last year's Back-to-Basics challenge:

http://community.livejournal.com/atlantisbasics/3812.html

[seperis.livejournal.com]

Mmm. See, I like that. That would actually be perfect.

[ellixis.livejournal.com]

There's also the consideration of who's posting it and where. You'll get a different group of readers/commenters/lurkers depending on your source.

[seperis.livejournal.com]

True. So it'd have to be defintiely posted somewhere easily accessible for all of lj part of sga fandom.

*thoughtful* Hmm.

[vonknibble.livejournal.com]

What makes a story good, bad or indifferent will undoubtably vary for each reader, which poses a whole set of variables rigth there. There is also the issue of commenting, in that not everyone who reads a fic will comment, regardless of whether they liked or disliked it, and some actual comments may even be negative and/or unrelated.

Perhaps rather than using comments as results, a poll (with results/votes hidden) per story would be better? get people to grade either the entire story, or mutiple factors - for example, 'score 1-5 on spelling/grammar, plot, chacterisation,' etc etc, possibly with option to comment on each/overall (on the poll, not as reply).

[seperis.livejournal.com]

Oh god, so awesome--I am very seriously enchanted with your poll idea here, so *much*--but would also baseline have ot trust that everyone who read would leave feedback on what they liked.

seriously, I love this poll idea. that would narrow it down into exactly what worked and waht didn't.

God, if my mind-control powers would *jsut come on already*....

[rivier.livejournal.com]

I can see the interest in this from a theoretical / hypothetical point of view, as in, can it actually be done? But not only do I think it's nigh-on impossible, for the various reasons pointed out above, and a few others - but I think the fact that the old 'feedback and what it means in qualitative terms' thing is going the rounds again makes me feel a bit disheartened.

Any form of feedback is only and ever going to be a subjective morsel. People leave fb because the story 'works' for them in some subjective way, which can range from 'superlative piece of creative writing' to 'you pressed all my shameful little kinky sex buttons with this piece of otherwise disagreeable writing', or 'thank God, at last! someone is writing my SUV / Myfanwy OTP, at last!'

And some fb (a fair bit?) is to do with knowing the author, as a friend or as a BNF who is perceived as a good thing to laud, to show that the feedbacker is 'doing the right thing' and fitting in properly to the community hierarchy.

I feel like the more pressing question - for me at least - is, why does everyone get so almightily hung up on feedback? It can never tell any writer anything genuinely objective about their writing. It can tell you only a range of information correlated against how popular your theme and (for fanfic) fandom is in demographic terms, how popular or well-known 'you' are, how well or widely you publicised your story, how easily accessible you make it, and so on.

What it can't do is tell you whether you have written something with which you can, by your own standards, feel a legitimate sense of satisfaction. Only you can do that, though you can get some of that from beta readers, maybe - depending on your beta and their relationship with you.

But trying to figure out a way in which feedback can somehow be stripped of all the contaminating factors, and made into a kind of objectively pure commodity? Wheeeee, good luck there! I honestly think you'll go bananas trying. Which takes me back to, why knock yourself out bothering to try? When you could be filling that time with the writing of some more nice satisfying fanfic!

[mmmchelle.livejournal.com]

What it can't do is tell you whether you have written something with which you can, by your own standards, feel a legitimate sense of satisfaction.

Yes, exactly.

[cathexys.livejournal.com]

You've gotta love a fandom where a random near-ranty post can result in *this*!!! :)

I love your idea and hope you'll organize it somehow. Except...I'm still not clear on who decides which story was better...

[seperis.livejournal.com]

it was an awesome topic and got me all contrary adn thinky. I'm *still* thinking about it.

[mahaliem.livejournal.com]

I'm not sure if it could be done.

One thing that I'm not sure how you might control is the order in which the stories are presented.

There was an anonymous story challenge in the Smallville fandom. The stories were posted alphabetically so "Your Next Bold Move" by svmadelyn (which was absolutely wonderful) was at the bottom of the list. The story didn't get nearly as many comments as I thought it should have and I wonder if it might have been partly because of its location and the tendency of people to start at the top of the list and work their way downward.

[mmmchelle.livejournal.com]

it will only make me cite the stories I hate most that had high comment counts and send me into blind rages

You have those too?

I am not alone. /g/

[cathexys.livejournal.com]

I'll third you!

Only thing that comforts me is if i go through the comments and i don't know or think too highly of the commenters. But then there are the stories that ate fandom, and when you're not in sync with it and don't like those particular genres/tropes/styles/whathaveyou...

And really, it should be no skin off our backs...so I wonder why that is such an anger inducing experience. It's not a zero sum game necessarily, and readers liking X may not mean thy won't read or like Y. I think for me it's a feeling of not fitting in, of having weird taste, of doubting everyone around me :D

...And I bet those stories are very different ones for each of us too, aren't they?

[elandrialore.livejournal.com]

Not to take all the fun out of it, but that would kind of suck for the authors of the study, especially if they didn't know about it.

I mean, even if you didn't tell them specific results, if you found there was a correlation between feedback and how good a story is, then even if an author typically got a lot of comments, then they might start to doubt themselves. "Well, sure people say they like it, but that doesn't actually mean anything, they're just following the crowd." And if you found that there wasn't a correlation, then the writers who don't typically get a lot of feedback will be all, "Well I was holding out hope that I was good at this despite limited feedback, but maybe I'm deluding myself."

I can tell you that as a reader and strictly speaking for myself, yes, sometimes the amount of comments on a story will induce me to read something. If I'm skimming a community or just lazy or there's not much of a summary or if I don't know the author all that well, I may glance down and think, "Hey, lots of other people liked it, I might too," but I'd say most of the time, whether I read or not depends on a) what type of story it is, b) whether I've liked anything else the author has written and/or c)if the author/story has been been recced by someone whose opinion I trust.

On the other hand, there are plenty of stories out there that I didn't like for one reason or another that have gotten loads of comments, accolades, and recs and yet I still don't like them and don't read them and wonder what the hell about the story has captured so many people's attention.
*shrug*
It's possible that I don't actually want to know because I'm afraid of the answer:)

[seperis.livejournal.com]

You know, my biggest draw on fic right now is rec first, author recognition second. It really is. But that has less to do with quality and more to do with the fact I have figured out that a lot of people write a John Sheppard that's anathema to me, and that's my break point on fic.

I mean, even if you didn't tell them specific results, if you found there was a correlation between feedback and how good a story is, then even if an author typically got a lot of comments, then they might start to doubt themselves. "Well, sure people say they like it, but that doesn't actually mean anything, they're just following the crowd." And if you found that there wasn't a correlation, then the writers who don't typically get a lot of feedback will be all, "Well I was holding out hope that I was good at this despite limited feedback, but maybe I'm deluding myself."

*thoughtful* The thing is, reversed--the authors that do get a lot of comments on a fic will think, oh god, so people realyl *do* like this and not due to my dS, Sentinel, SV, etc work. And if it went the other way, people who get less feedback would have the warmth of knowing that they don't suck, they just dind't hit whatever weird fandom frenzy there was going on at the time. It's totally a glass half-full sitch.

Basically, even if I could pull this off, nothing would actually change in terms of what people will and will not comment on, will or will not read, will or will not write, or will or will not do. I mean, if I honestly believed constructive crit could make people stop writing, I'd have been doing it a lot more than I do.

But it does make me stop and think on what variables every story has, what makes it good, great, or merely mediocre, and how much those variables depend on the author and something that is out of her control--the timing to the audience--that makes the differnece between wildly popular and acceptable.

[veejane.livejournal.com]

A true double-blind study would have to control for quality -- I can think of a couple of quantitative measures of quality you could use, like spelling and grammar and number of mixed metaphors -- and would have to control for as many other circumstances as possible. So, say you establish two false LJ identities, each name innocuous, both on the same day. They would need comparable user-histories, interactions with people in fandom, similar levels of regard and well-spokenness. Both User1 and User2 would have to be on the same comms, and have roughly the same flists. They would have to publish fic in the same fandom, to the same comms, on the same topic, with the same kinds of appeals, at the same time on the same day, such that the main difference between them would be the pre-defined quality-markers within the stories.

You know, we should be able to gank experimental results from all those sockpuppets roaming the earth, don't you think?? I mean, aren't they gathering our data for us, every day? (Okay, that is observational study, which is not the same.)

I think it's not technically feasible, unless you're willing to knock yourself out or expose yourself by having your two identities act like Doublemint twins. The closest I have seen was a secret cabal of writers, in XF back in 1999-00, who collectively created a new identity for the purpose of writing outside of the genres each was known for. From outside the cabal itself, all could tell that this new person on the scene was a pseud, but nobody knew who it was (and it took a while to twig to the fact it was more than one person). Later, when I talked it over with some of the participants, I got the sense that the cabal members tended to get feedback very different from what they would have expected to get if they had posted under their own identities, but of course the whole point was that they were publishing outside their comfort zones, so take from that what anecdotal evidence you wish.

Any way you slice it, an institutional review board would have unkind things to say about the informed consent of the participants, although, writing a crappy story isn't exactly reproducing the Stanford Prison Experiment!

[tacittype.livejournal.com]

iirc from *cough* psych degree you only need informed consent if your actions alter someone's behaviour or compromises their privacy in some way - public behaviours in a public place are technically fair game for observation. And the Internet is pretty darned public. It'd be admittedly rude, though, much like any observer with a clipboard; the authors should know about it if the data's going to be shown around the net.

I have some thoughts on Seperis' study up-stream and think we could get what she's interested in without having to worry about all the variables, provided the number of stories was high enough. I'm getting a bit fidgety waiting for someone to implement it. This idea is fascinating!

Ahh, the prison study. I especially liked the part where people were abducted from their homes and given nervous breakdowns. And then caused nervous breakdowns in their fellow ppts. But I read another study where the psychologists accidentally enduced life-long eating disorders in previously healthy. And one of the participants got so depressed he chopped off one of his own fingers. Wow, we humans are creepy.

[eleveninches.livejournal.com]

I think it'd be interesting to do a double-blind challenge or something, but the problem is, certain authors are easy to distinguish because of their style. A lot of people guessed correctly on Madelyn's anonymous schmoop challenge.

[seperis.livejournal.com]

*bites lip*

well. yes. That's part of the variables that aren't controllable. So we'd never know if tehy liked the *story*, or liked you adn so liked the story. Etc.

It's fun to imagine, though.

[tacittype.livejournal.com]

...Is anyone else wondering what's defining which of the comments upstream are getting their own comments? And whether that makes them better comments?

Just me? Okay, then.

[seperis.livejournal.com]

*begins to laugh hysterically*

[copracat]

1. okay, we are in sga, therefore, we know of the scientific method

Sarcasm, right?

2. So I'm trying to figure out how would a true double blind work in fandom as it stands.

Too many people are good at working out a writer's identity from their style and content for this to work. We can see this in Yuletide, Secret Santas and guess the writer fiction communities.

In the end, if it matters to you no amount of experiment is going to stop you banging on about it and being some level of anguished. If it doesn't matter to you then you've got more time in your life to write fiction. Or watch TV or pet puppies.

[seperis.livejournal.com]

Hey, we all have heard of it and know where wikipedia is. That totally counts.

And yeah, on teh second, I figured as much. But as a theoretical exercise, it's immensely fun.