Artificial Analysis - Prolific Researcher Reviews

study 6acd969eeb54ba891194076f This is a complete scam. "By agreeing to these terms, you give your specific, informed, written consent for us and our licensees (including the developers of the AI models evaluated) to use your recordings to train, fine-tune, test and evaluate AI models, including speech and voice models. This may include creating synthetic, computer-generated speech or voice models that resemble your voice (a "digital voice replica"), which could say things, in any language, that you never actually said, and which may be difficult to distinguish from your real speech." "You grant us a worldwide, royalty-free, perpetual, irrevocable (to the maximum extent permitted by law), sublicensable and transferable license to use, reproduce, store, modify, analyse, create derivative works from and share the recordings and other study data you provide, including to train AI models and create digital voice replicas as described in Section 6, subject to the protections in Section 6. Where applicable law treats any right in your voice as personal and non-assignable, this operates as a license of that right to the same extent." They will literally own your voice and can profit from it forever and always. All for the low, low price of $10?

mic needed, 8 short ai conversations

You've got to watch these guys. Instead of the usual 50 prompts, this one had 75. While they claimed: This task consists of 75 prompts with a pair of videos for each. You will be tasked with voting for which video is a higher quality generation than the other. Each prompt takes about 20 seconds to evaluate. They are full of it. It took me almost 35 minutes because, guess what? 25 more prompts are going to take longer because the videos are about the same length as those in the 50-prompt studies.

great studies pay within few hours

This is the second study today from a researcher (albeit a different one) where I got into the study, completed the preliminaries such as consent form, reading the instructions, and mic and audio connection, but I couldn't proceed to the active part of the study because of an error message to the effect that there are no places available and to come back later. The error message for this one was, "Could not start - 429: {"detail":"The study is at capacity right now; please try again later."}". I'm not sure if something is out-of-sync with Prolific and researchers' servers today, but it seems to me that when you are able to accept a task, that means there *are* places available. Disappointing and a waste of my time.

Video Output Evaluation study, $4 for 30 minutes is far from fair remuneration.

This study appeared in a sea of studies by my favourite AI researcher and I clicked on it by mistake thinking it was one of theirs. Returned it after giving it a good go because it is an endless timesuck. They want you to do a quality review of 50 video prompts in 30 mins for $4! When there are 3 x 10sec videos to watch for each prompt. hahaha. Have blocked this researcher to make sure I don't click on one of theirs by mistake again.

Fairly simple to do study. You simply have a short conversation (~2-3 minutes) with one AI model, then do it again with a different AI model, and then rate which you preferred on things like instruction following, tone, etc. You then repeat for 4 total trials of that.

Quite a long study to do correctly as its by number of videos (They state 15 seconds per video, but each consists of 2, 10 second snips), rather than time taken, so it can (WILL) take longer than stated. There is no middle ground, forcing you to like one video or the other, as some are both good or both bad this can be difficult to select the 'right' one. Not sure that they obtain good results this way as some will simply rush.

Absolute insanity. The Mandarin Audio Preference Evaluation I was given had 2500 audio samples for me to evaluate - each one taking AT LEAST 6 seconds to listen to. And given the allotted 37 minutes, I do not see or understand how anyone could complete this at all.

Gives you a fixed quantity as opposed to a fixed time like the $5 dollar studies you may used to, As others have said, this vastly understates the time taken, and ends up paying half the amount as you will not get through any more than two of these in an hour. Interface is quite good, shame that they don't want to pay us a reasonable amount.

Completely ridiculous that we are expected to do 80 of these in 20 minutes when some of them have very long prompts we have to read. Absurd.

Estimated completion time was listed as 20 mins but it took me 40 mins to complete. The description said it should take a maximum of 10 seconds to evaluate each prompt, but it honestly takes much longer than that to make a good evaluation. If I took a maximum of 10 seconds then my evaluation would be sloppy and carelessly, unless they're looking for you to evaluate things at a glance.. (but honestly without taking a closer look it's hard to tell which one is better than the others a good portion of the time)

Audio study had constant technical issues and kept giving me "request timed out" and error messages, so it took me 10 minutes more to complete it than it should have. I noticed if you can't get it to work, if you go back to Prolific and re-open the study from there, it would usually start working again.

Grossly underestimates the time it would take to do the study. Claims that each prompt should take a maximum of 15 seconds to evaluate, but the videos in the first set exceeded that time limit, let alone accounting for reading the prompt itself. Requires 50 comparisons in 20 minutes. Not a reasonable time limit for the amount of work required, regardless of the pay (which is low). I returned after seeing the first prompt.

Unfortunately after trying in 3 different browsers without success, I had to return this one. I kept getting an error when submitting my first answer, and I attempted to message the researcher, but never got a response. I am not sure why the errors occurred, but I was bummed to have to miss out.

80 image questions are supposed to take only 12 minutes to complete? And 40 video questions are supposed to take 14 minutes? No way. Super underpaid and actual time is double if not triple the time advertised.

Nice and easy. Not an auto-approved, but I submitted around 6:00pm Friday, by 8:00am Saturday it was approved.

Intended completion was listed as 30 mins but it took me 25 mins to complete. The study was fun and easy to get through :> I honestly spent some extra listening to the clips.

Pays okay, feels like I have to rush a bit to get through in the intended time though.

Pretty fun and decent pay

Pretty easy and great pay but I experienced technical issues. For around 80% of the videos, my selection wouldn’t register and I had to click the button like 50 times and/or wait an extra 30 seconds to proceed. The study ended up taking 10 minutes longer than expected because of this.

Similar work to the Alphabet studies. Would highly recommend these especially for those without a 1080p screen. Almost as good pay as well. No complaints. I’d argue their platform is more intuitive than Populous Studio.