Prolific Academic Survey Alerts

DontBringBertie

olliebean said:

I've just looked at the guidelines for researchers to set attention checks. I think these fail on a few counts:

• "It should check whether a participant has paid attention to the question, not so much to the instructions above it."

The question was "what percentage of the dots are orange?" The instructions, apparently, changed. (Actually quite a lot of attention checks fail this one, with questions designed to trick people who haven't read the preceding instructions telling them to ignore the question.)

• "You should only use the check if, without it, the task couldn’t be completed properly"

This is certainly not the case, unless the purpose of the task was to see how many people failed the attention checks - in which case, those who failed are still contributing to the data, and should be paid. Otherwise, given the time limit, I would argue that the attention checks actually impeded the proper completion of the task.

• "They cannot: 
• Be in repeated, unchanging text"

That's an absolute fail, then. Apart from the pages where it contained attention checks, the text was repeated and unchanging on every page.

There's also a link to this article: https://researcher-help.prolific.co/hc/en-gb/articles/360009501033, the first couple of paragraphs of which could be precisely describing the attention checks in this study.

In summary, I acknowledge that I failed attention checks, but not that I failed fair attention checks.

I did a survey earlier which had the below, surely that’s against the rules too?

Sports Participation

 

Most modern theories of decision making recognize the fact that decisions do not take place in a vacuum. Individual preferences and knowledge, along with situational variables can greatly impact the decision process. In order to facilitate our research on decision making we are interested in knowing certain factors about you, the decision maker. Specifically, we are interested in whether you actually take the time to read the directions; if not, then some of our manipulations that rely on changes in the instructions will be ineffective. So, in order to demonstrate that you have read the instructions, please ignore the sports items below and do not click on any of the sports options. Instead, simply proceed to the next page.

 

Which of these activities do you engage in regularly?
(Click on all that apply)

olliebean

I'd have thought so, and I did one yesterday which had two attention checks where there was a question printed in large, bold text, and instructions in smaller text essentially telling me to select the wrong answer - also against the rules, as I understand them. I think there are a number of survey setters who believe the point of an attention check is to try to trick you into getting it wrong, rather than to ensure you are paying as much attention as you would reasonably be expected to.

Crabby

Anyone have problems with this one?  It was quite tough, took longer than the alotted time and made no mention of the need for accuracy.

Story Study

Hosted by S4HRI IIT

£3.3030 mins £6.60/hr 1 place remaining

Welcome to our experiment and thank you for participating! This experiment consists of two parts. The first part will be a reading task and the second will be a numeric task.  In the reading task you will read short stories about a person and answer YES or NO questions about the stories. In the numeric task we will show you some sequences of numbers. At the end of each sequence you have to answer two questions about them.

Dear participant, Unfortunatly you did not achIeve the accuracy need. We can only use data with accuracy higher than .65, which it means a moderate level of attention. We kindly ask you to return your submission so we don't have to reject it. Thanks for your collaboration. S4HRI

The numeric task was 6 single digit numbers flashed up at 3 second intervals, then few seconds to answer 2 questions like; what is the 3rd digit in the sequence or the sum total of digit 2 and 4.

olliebean

Crabby said:

The numeric task was 6 single digit numbers flashed up at 3 second intervals, then few seconds to answer 2 questions like; what is the 3rd digit in the sequence or the sum total of digit 2 and 4.

That sounds unfair to me as an attention check; the guidelines explicitly state that a fair attention check cannot rely on memory recall. If the study itself involves remembering information, that's fine, but in that case a failure to remember correctly is still data and should be compensated. The basic payment is for the time spent, regardless of how good you are at the task, and while a researcher is free to give a bonus payment to anyone who performs particularly well, I don't think they're allowed to dock payment for not performing well enough.

Crabby

That wasn't an attention check, it was the numeric task, 12 sets of 6 random single digits and 2 questions after each.

If the study itself involves remembering information, that's fine, but in that case a failure to remember correctly is still data and should be compensated. The basic payment is for the time spent, regardless of how good you are at the task,

I totally agree and have told them to pay up. If they don't I'll escalate it with Prolific.

DontBringBertie

olliebean said:

I'd have thought so, and I did one yesterday which had two attention checks where there was a question printed in large, bold text, and instructions in smaller text essentially telling me to select the wrong answer - also against the rules, as I understand them. I think there are a number of survey setters who believe the point of an attention check is to try to trick you into getting it wrong, rather than to ensure you are paying as much attention as you would reasonably be expected to.

Yeah - I think the ‘select strongly agree for this row’ is fair enough obviously.

But not a list of countries and saying ‘select China from the list below’ when naturally you’d choose where you live.

Luckily after doing surveys for years you can spot an attention check question like that easily.

ndf9876

olliebean said:

Crabby said:

The numeric task was 6 single digit numbers flashed up at 3 second intervals, then few seconds to answer 2 questions like; what is the 3rd digit in the sequence or the sum total of digit 2 and 4.

That sounds unfair to me as an attention check; the guidelines explicitly state that a fair attention check cannot rely on memory recall. If the study itself involves remembering information, that's fine, but in that case a failure to remember correctly is still data and should be compensated. The basic payment is for the time spent, regardless of how good you are at the task, and while a researcher is free to give a bonus payment to anyone who performs particularly well, I don't think they're allowed to dock payment for not performing well enough.

If they told you up-front that there was a minimum percentage required, or some kind of established benchmark, then what they are asking is fair enough. However, if they didn't tell you, I believe this would fall under "Unfair reasons for rejection" because you failed an arbitrary threshold that wasn't made known to you. See here : https://researcher-help.prolific.co/hc/en-gb/articles/360009092394-Approvals-rejections-returns#heading-2

Crabby

If they told you up-front that there was a minimum percentage required, or some kind of established benchmark, then what they are asking is fair enough.

If they did, it was lost in a lot of very small print. They used the score to suggest I wasn't paying full attention, when all it means is that I didn't meet their expectations. I'd be surprised if many did.

Also spot the difference;

1) Story Study

Hosted by S4HRI IIT

£3.30 30 mins £6.60/hr 1 place remaining

2) Story Study

Hosted by S4HRI IIT

£3.30 39 mins £4.96/hr 0 places remaining

It  took me 46 minutes, which actually equates to £4.30/hr.

olliebean

I just got a £3 bonus for a study that I returned, which I'm guessing must have been a mistake, but I'm not complaining!

BarryT

I've read that Prolific only regards certain ISPs as providing the required level of security for its surveys and rejects the others. I'm thinking of joining Prolific and as I'm coming to the end of my broadband contract with The Post Office thought I'd try to find out which ISPs it accepts. Apparently Prolific do not provide a list of these. At the moment I'm looking at Plusnet, John Lewis, Vodafone, TalkTalk.  Who are you guys with or any info on this?
Thanks