Prolific Academic Survey Alerts

olliebean

I never saw any pictures of people. It sounds like I missed the checks that the previous two posts have described. But I do think this was unreasonable. The instructions were clearly set out at the start of the task, with no indication that they might change during the task, and furthermore there was a time limit imposed which encouraged completing the tasks as quickly as possible rather than taking time to re-read the instructions on every page. This seems like a check not designed to test whether people are paying attention, but explicitly to catch people out who had a good faith belief that the study wouldn't suddenly change the rules without making it very clear they were doing so (or having explicitly stated at the start that that might happen).

I hope the fact that a good number of people were apparently caught out will lead to a realisation that this was not a reasonable attention check. (Or perhaps it will turn out the purpose of the study was to see how people respond to being unfairly tricked into failing in this way.)

MollyR

The instructions at the start of the study made it clear that there would be attention checks, although not specifying what they were;  they also said that anybody failing those checks would be immediately disqualified.  I found them pretty obvious;  the balance of the dots was so one-sided, compared to the previous images, that I automatically stopped to think and wonder why, and then saw the instruction to select the "wrong" one.

olliebean

Well, I wasn't immediately disqualified; they let me finish the study first. And I still think "there may be attention checks" isn't the same thing as "these are the rules that you must follow throughout, oh and by the way we might subtly change them a couple of times so make sure you keep re-reading them on every page even though we've presented them at the start as fixed rules that you must follow and given you a time limit that doesn't allow time to re-read them."

elsien

MollyR said:

The instructions at the start of the study made it clear that there would be attention checks, although not specifying what they were;  they also said that anybody failing those checks would be immediately disqualified.  I found them pretty obvious;  the balance of the dots was so one-sided, compared to the previous images, that I automatically stopped to think and wonder why, and then saw the instruction to select the "wrong" one.

Nothing obvious about the picture ones.
We were asked to use a sliding scale to rate pictures for attractiveness and we had 15 seconds to do so for each picture, whilst also looking at a rating that other people were alleged to have given. So no I didn't read the instructions each time for each picture because we'd already been told what we need to do. 

MollyR

It certainly seems that there is more than one study which people have taken;  one with blue and orange dots and one with pictures to be rated for attractiveness.  My comments only relate to the blue and orange dots, since I have never seen the other study.  And I found the attention checks very little different from other ones;  being told "For this image, select Orange" is very much the same as being told "For this row, select Somewhat Disagree.  But maybe there are additional variations apart from the main structure of the study - who knows?

Value

Both studies were the same length, same payment and same researcher so I think they were variations on his theme.

I wasn't immediately disqualified either but I reckoned it would be rejected once I realised what was going on which is why I returned it midway through.

olliebean

I've just looked at the guidelines for researchers to set attention checks. I think these fail on a few counts:

• "It should check whether a participant has paid attention to the question, not so much to the instructions above it."

The question was "what percentage of the dots are orange?" The instructions, apparently, changed. (Actually quite a lot of attention checks fail this one, with questions designed to trick people who haven't read the preceding instructions telling them to ignore the question.)

• "You should only use the check if, without it, the task couldn’t be completed properly"

This is certainly not the case, unless the purpose of the task was to see how many people failed the attention checks - in which case, those who failed are still contributing to the data, and should be paid. Otherwise, given the time limit, I would argue that the attention checks actually impeded the proper completion of the task.

• "They cannot: 
• Be in repeated, unchanging text"

That's an absolute fail, then. Apart from the pages where it contained attention checks, the text was repeated and unchanging on every page.

There's also a link to this article: https://researcher-help.prolific.co/hc/en-gb/articles/360009501033, the first couple of paragraphs of which could be precisely describing the attention checks in this study.

In summary, I acknowledge that I failed attention checks, but not that I failed fair attention checks.

elsien

Just emailed the researcher pointing this out.

On a positive note, just got a nice £6.20 bonus on a 25 minute £2 ish survey. Happy bunny.  

albion22

Online challange

Hosted by Ricardo Santiago Araujo

£0.88 6 mins £7.68/hr 0 places remaining

Yes I had the same issue. The attention check was grossly unfair when not clarified beforehand, particularly when only given 5 seconds I think to focus on the photo attractiveness. I complained to the researcher who unsurprisingly hasn't replied so will make a formal complaint to Prolific.

MollyR

It is very strange how many variations there were of this study, all under the same heading.  On looking at it again, it does say "In this study, you will be randomly assigned to participant in an online challenge."  So it seems that not only were there two entirely separate tasks, one of photo attractiveness and one of coloured dots, but maybe the in-task instructions have differed.  As I said earlier, when I took the study it made it clear in the instructions at the start that there would be attention checks, and that failing one would lead to instant disqualification;  but in my case the attention checks were clearly marked and the coloured dots which related to it were very different to the regular ones.  But others seem to have had their DQ at the end rather than instantly, or have not had the attention checks clearly marked.  It doees seem likely that the basis of the study is how people coped with the differences, which makes it particularly unfair to DQ people.  I must have been in a very lucky minority, to have been able to sail through it without facing any problems!