Amazon Mechanical Turk Pays Less Than 40% of US Minimum Wage, Research Suggests

The results of a study by researchers in the US and Mexico suggest that crowdsourced workers in the Amazon Mechanical Turk marketplace, with a 75% majority based in the United States, are earning as little as a median $2.83 per hour – about 39.5% of the mandated federal minimum wage of $7.25 per hour, since the remote workers spend a third of their working time scanning lists of requests for work, managing payments, and performing other kinds of ‘invisible labor’.

The findings come in the wake of a recent report from the University of Massachusetts Amherst, which criticized the low level of pay and benefits involved in being a ‘turker’, and which characterizes the AMT ecostructure effectively as a low-quality sweatshop for under-funded academic researchers.

Launched in 2005, Amazon Mechanical Turk (AMT) supplies crowdworkers with ad hoc Human Intelligence Tasks (HITs), frequently leveraged by machine learning researchers in the context of ‘affordable’ study groups for tasks such as image recognition, Natural Language Processing (NLP) evaluation, and other systematic, programmatic tasks designed to help formulate or test algorithms (though the service is also extensively used to generate spam).

Invisible Labor on MTurk

The new study, titled Quantifying the Invisible Labor in Crowd Work, was carried out by researchers from Northeastern University at Boston, Microsoft Research, and the Universidad Nacional Autonoma de Mexico (UNAM).

The extent of invisible labor performed by 100 workers on over 40,000 HIT tasks over a one-week period. Source: https://arxiv.org/pdf/2110.00169.pdf

The paper details the extent of ‘invisible labor’ that constitutes 33% of a turker’s work-time, as evidenced in a field study undertaken for the work. The greatest amount of ‘lost’ time was represented by the amount of time AMT workers spend examining their payment details to ensure that they received payment for a task. The report states that ‘97% of the workers in our study practiced invisible labor around visiting the earnings section on their workers’ dashboard (perhaps to ensure they had gotten paid fairly)’.

The second-biggest time drain for turkers was the ‘hyper-vigilance’ necessary for actually obtaining work, where workers refresh the pages of requesters’ profiles and searched for new tasks advertised in the AMT HIT listings sections.

Monitoring Turks

The researchers developed a Chrome extension, which they have since made available at GitHub, based on prior work that studied the attention habits of AMT workers, but which did not address the evaluation of invisible labor. The extension was installed on the systems of 100 participating Mechanical Turk workers, who collectively undertook 40,903 HITs over a one-week period.

The main task of the extension is to evaluate the actual time a worker spends on a HIT task, and to this end it monitors the state of AMT-related pages by scraping and parsing the HTML content of a page that the turker is currently focused on, in order to understand the worker’s interactions with it. The extension also features a time-driven background process that logs browser events on the MTurk domain, and the status of each page (i.e. active, inactive, closed).

Time-consuming facets monitored under this system include workers checking the ‘qualifications’ status (to see if recently-completed work has ‘upgraded’ them, giving them access to better-quality HITs); ‘wading and sorting’ through an avalanche of HIT postings with search filters to identify a possible next job, and exclude undesired work; and filtering out ‘fraudulent’ HITS and generally managing/invigilating queued HITs.

Delays and Drags

The system also recognizes ‘delaying’ events and ‘false starts’, such as when a worker returns a previously-accepted task to the HIT pool because it turned out to be different from their expectations; the time taken to ‘preview’ HITs, or to read information about the HITs; to send messages to the HIT originator in order to better understand the demands of the task; and reading general platform help from the Mechanical Turk help pages.

In the very time-consuming category of ‘payments’, the research found that AMT workers were frequently at their dashboard to check if they had been paid, since it’s possible for completed work to be rejected, unpaid, for reasons that the worker will never be told; viewing their earnings, which have a dedicated section of the MTurk dashboard, as opposed to announcements of a new payment; and performing HITs that ‘time out’ – where a worker fails to complete a HIT in the allotted time, and receives no payment for any work done on it (a category that the researchers feel should be in the ‘payments’ section).

Besides the report’s findings that invisible labor lowers the previously estimated median wage ($3.76) of an AMT worker by 39.5% when factored into a working session, it also notes that workers’ perception of how much time they have spent on unpaid work during a session is notably lower than the monitoring sessions reveal.

The authors note that they have rounded down all estimates in their methodological findings in order to ensure a rigorously ‘conservative’ approach to the work, and to avoid accusations of exaggeration of findings. They conclude:

‘Considering that the median hourly wage of workers is just $3.76 (without considering invisible labor), it is clear that crowd workers still need a dramatic increase in their wages before we can consider this labor fair. However, this is not only something for requesters to consider, but also something for platforms, workers, and even policy makers.’

The State of Mechanical Turk Workers

Other authors and organizations estimate that the average AMT worker actually earns less even than is estimated in the new report – as little as $1 an hour.

Proponents of the benefits of AMT argue, with relative frequency, that controversies around poor remuneration of AMT workers is overblown. One study from 2020 surveyed a section of United States MTurk workers, finding that ‘the financial situation of people on MTurk mirrors the general population‘, and that turkers ‘would not trade the flexibility of MTurk for less than $25 per hour’.

That work was conducted by employees at CloudResearch (formerly TurkPrime), whose services include ‘tools that allow researchers to run studies on Mechanical Turk’.

Since AMT workers are considered to be independent contractors, they are not commonly covered by labor laws in the US. In a 2016 survey by the Pew Research center, more than half of AMT workers (52%) reported making less than $4.99 an hour, which – assuming full-time work with no holidays or sick leave – is equivalent to an income of $10,379.20 a year.

For many workers, the survey found, Mechanical Turk represented ‘all or most’ of their income, and this category of turkers was characteristically younger, less educated, and lived in households with lower incomes.

However, turkers were, according to the survey, generally better-educated than average American workers, with 51% reporting that they have at least a college degree, against 36% of adult workers in the US – with only 12% of AMT workers claiming to only have a high school degree or less.

The Right Stuff?

The Pew report also indicated that typical AMT workers may be the exact opposite of the ‘average person’ that most academic studies are seeking to leverage, with implications for the extent to which their unusual responses could potentially bias or degrade the quality of machine learning projects that rely on turkers.

The findings of the Pew report state that turkers ‘are not representative of Americans overall – or even of working adults’.

It also notes research from an earlier (2014) study, which indicated that AMT workers are subject to so many experiments and surveys that the quantity of this type of work ‘could condition the workers to certain types of answers or have them become overly familiar with the types of questions they will be asked’ – effectively, that the grind of AMT work will incline turkers to become experts at ‘pleasing the system’ rather than providing authentic and academically (or even clinically) useful input.