Artificial Analysis - Prolific Researcher Reviews

This is the second study today from a researcher (albeit a different one) where I got into the study, completed the preliminaries such as consent form, reading the instructions, and mic and audio connection, but I couldn't proceed to the active part of the study because of an error message to the effect that there are no places available and to come back later. The error message for this one was, "Could not start - 429: {"detail":"The study is at capacity right now; please try again later."}". I'm not sure if something is out-of-sync with Prolific and researchers' servers today, but it seems to me that when you are able to accept a task, that means there *are* places available. Disappointing and a waste of my time.

Video Output Evaluation study, $4 for 30 minutes is far from fair remuneration.

This study appeared in a sea of studies by my favourite AI researcher and I clicked on it by mistake thinking it was one of theirs. Returned it after giving it a good go because it is an endless timesuck. They want you to do a quality review of 50 video prompts in 30 mins for $4! When there are 3 x 10sec videos to watch for each prompt. hahaha. Have blocked this researcher to make sure I don't click on one of theirs by mistake again.

Fairly simple to do study. You simply have a short conversation (~2-3 minutes) with one AI model, then do it again with a different AI model, and then rate which you preferred on things like instruction following, tone, etc. You then repeat for 4 total trials of that.

Quite a long study to do correctly as its by number of videos (They state 15 seconds per video, but each consists of 2, 10 second snips), rather than time taken, so it can (WILL) take longer than stated. There is no middle ground, forcing you to like one video or the other, as some are both good or both bad this can be difficult to select the 'right' one. Not sure that they obtain good results this way as some will simply rush.

Absolute insanity. The Mandarin Audio Preference Evaluation I was given had 2500 audio samples for me to evaluate - each one taking AT LEAST 6 seconds to listen to. And given the allotted 37 minutes, I do not see or understand how anyone could complete this at all.

Gives you a fixed quantity as opposed to a fixed time like the $5 dollar studies you may used to, As others have said, this vastly understates the time taken, and ends up paying half the amount as you will not get through any more than two of these in an hour. Interface is quite good, shame that they don't want to pay us a reasonable amount.

Completely ridiculous that we are expected to do 80 of these in 20 minutes when some of them have very long prompts we have to read. Absurd.

Estimated completion time was listed as 20 mins but it took me 40 mins to complete. The description said it should take a maximum of 10 seconds to evaluate each prompt, but it honestly takes much longer than that to make a good evaluation. If I took a maximum of 10 seconds then my evaluation would be sloppy and carelessly, unless they're looking for you to evaluate things at a glance.. (but honestly without taking a closer look it's hard to tell which one is better than the others a good portion of the time)

Audio study had constant technical issues and kept giving me "request timed out" and error messages, so it took me 10 minutes more to complete it than it should have. I noticed if you can't get it to work, if you go back to Prolific and re-open the study from there, it would usually start working again.