‘Never again’: Inside the chaotic world of the people making AI seem human

Jonathan, who lives in Washington D.C., was laid off in January this year. While searching for jobs on LinkedIn, he received a message offering him $40 per hour to train AI systems, with the possibility of full-time work. Jonathan—who, like everyone in this article, asked to go by a pseudonym or first name to speak openly about their experiences—saw the job as a nice side hustle while figuring out his next career move.

The application process, with a company called Outlier, was quick. Jonathan had a brief Zoom with a recruiter, completed some writing tests, and was added to a large Slack channel. The work involved writing essays on unusual topics, like how mountains make good parents, or rating AI-generated content.

But problems quickly accumulated. Despite promises of 40 hours a week, Jonathan quickly discovered there was only work for about half that. Sometimes, mid-task, he’d have to stop for 30 to 60 minutes of unpaid retraining.

Pay discrepancies also became an issue. Once, Jonathan used a stopwatch while working, clocking in four hours, but Outlier recorded just shy of two hours. He only got paid $49, even less than expected. When he sought help from managers on Slack, they directed him to submit a ticket, which went unanswered.

“It was really frustrating figuring out what counts and what doesn’t,” he says.

Eventually, Jonathan was assigned to a new group, but found no work for days at a time. Unsure if he was being penalized or if there simply wasn’t any work, he has since given up checking for assignments, discouraged by the experience.

Now, he says, “I don’t even know what hoops to jump through.”

Teaching the new generation of powerful AI models requires massive amounts of data and armies of humans, whose feedback to the models helps improve their outputs. That happens through techniques including supervised learning, which involves teaching the AI from human-generated writing, and reinforcement-learning from human feedback, or RLHF, in which the AI learns from how humans evaluate their responses.

Outlier is one of the largest employers of this new AI-training workforce, promising applicants that through its platform they can “get paid training cutting-edge AI on your own schedule” and “shape the next generation of AI with your expertise.” Outlier’s parent company, San Francisco-based Scale AI, says it’s building out the “data foundry” needed for AI. Cofounded in 2016 by CEO Alexandr Wang, Scale AI has raised $1.6 billion dollars from investors like Meta, Amazon, and Nvidia, and provided critical data to companies like OpenAI and Google. Big Tech contracts have also boosted other data contractors, including DataAnnotation Tech, xAI, Surge AI, and SuperAnnotate, all of which have been recruiting humans across North America to teach computer systems how to seem more human.

For the workers, however, the rewards of training AI can be elusive.

Interviews with 10 current and former Outlier contractors across the United States and Canada reveal a knowledge-worker gig economy plagued by a dizzying tangle of problems, including technical and communication snafus, inconsistent rates and unpredictable schedules, and unpaid wages.

This invisibility of the outsourced workforce is particularly concerning. While headlines trumpet layoffs at major tech companies, the impact on the vast network of contractors and gig workers often goes unnoticed and unrecorded. As companies like Scale AI rapidly adjust their operations in response to market pressures, the ripple effects on this shadow workforce could be substantial, yet largely hidden from public view.

“These workers are extremely vulnerable,” says Jonas Valente, a postdoctoral researcher with the Fair Work Project at the Oxford Internet Institute.

Travis, from Ontario, experienced firsthand the shifting pay scales and inconsistent work opportunities at Outlier. He was initially hired through a recruiting agency at $55 per hour. But he says his rate was quickly adjusted to $41 to align with Latin American rates, since he was living in the Dominican Republic at the time.

His experience took a turn when identity verification issues arose, forcing him to return to Canada to re-upload his documentation after two months. Upon resolving this, he found his rate had dropped again to $35 per hour.

Despite his efforts to resolve the issue internally, including submitting claims and reaching out to team leaders, Travis struggled to receive his owed wages, which he says totaled approximately $3,600. The missing funds mean he’s relying on his credit cards and struggles to pay for his son’s epilepsy medication.

The disruption intensified when Scale AI abruptly terminated its relationship with the recruiting agency that had hired Travis, leaving many workers in limbo. While some, including Travis, were offered direct positions with Outlier, the transition was far from smooth. He now checks Outlier “for entertainment purposes only.”

“I would never work with them ever again,” he says. “My life is way too stressful.”

Scale AI declined to comment on the individual workers’ experiences in this story. But a Scale AI spokesperson said in a statement that they “are continuously working to improve worker experience.” The number of tasks available, they said, “fluctuates with the needs of our customers, and most workers choose to work on tasks for 5-10 hours a week.” Delays or interruptions to payments “are rare and occur primarily in instances of incorrect payment information (such as not providing it or submitting incorrect PayPal account information), and/or evidence of fraudulent activity,” the spokesperson said.

Additionally, “workers have multiple channels for questions and support, including 24/7 support teams, community discussion channels with specially trained moderators, and a ‘speak up’ hotline where contractors can report concerns anonymously.”

Two sources in this story had turned to the support teams for help, to no avail. When asked for more details on Outlier’s hotline, the spokesperson apologized for “the confusion on the nomenclature of hotline, as not all hotlines are the traditional phone number.” To report concerns, they said, “contributors are directed to submit support tickets through their Outlier account, or reach out to the various support emails that are offered, which can be done anonymously if desired.”

The costs of scale: lower wages, abrupt shut-downs

Founded in 2016, Scale AI started by hiring overseas workers, in Africa, India, and the Philippines to label images from autonomous driving and machine vision companies, for about $1 to $2 per hour. The generative AI revolution that kicked off in November 2022 with the launch of ChatGPT has expanded the company’s work, helping clients like OpenAI and Microsoft to refine and test various generative AI systems, including models being developed by the Pentagon. This newer work often involves fine-tuning the models, which requires workers to provide high-quality, context-specific feedback on AI outputs, rate the relevance and accuracy of responses, and sometimes generate human-written content for comparison.

Unlike earlier data annotation, this work requires Scale to hire more highly educated workers based in the U.S. and Canada, as it demands a nuanced understanding of cultural context, idioms, and subtle language use that native English speakers are best equipped to provide.

This new labor force—often with advanced degrees—is more expensive, earning rates between about $20 and $40. This has contributed to a drop in the company’s gross margins, from 59% to 49% between 2022 and 2023, according to financial data obtained by The Information.

To rein in costs, the outlet reported in June, the company has told investors it has been using tools to automatically identify more “efficient experts” to train models, and relying on computer-generated data to augment human work.

It’s also been cutting wages: In April, many Outlier contractors got an email announcing it was cutting wages for the mandatory time they spent training for tasks; one worker told The Information their new rate was $17 per hour. A month later, the company announced its latest funding round, which raised $1 billion from investors including Amazon and Meta, bringing its valuation to $14 billion.

The impact of Scale’s cost-saving efforts appears to be far-reaching. In early March, RemoteTasks, another Scale AI subsidiary, abruptly shut down in Kenya, Pakistan, and Nigeria, and halted new sign-ups in Thailand, Vietnam, and Poland, Rest of World reported. The company offered workers no explanation, but later told Rest of World that the closures were due to “enhanced security protocols,” and blamed its failure to notify workers on “operational errors.”

The shift follows years of complaints and accusations by Remotasks workers about inconsistent and unpredictable work for low and sometimes unpaid wages. Last September, Scale AI addressed “misunderstandings and mischaracterizations” around Remotasks, after widespread accusations of the company over low and unpaid wages.

“We partner with the Global Living Wage Coalition, and our economists conduct quarterly pay analyses that take a number of factors into consideration, including local costs of rent, healthcare, and transportation, in order to ensure fair and competitive compensation,” the statement said.

For Scale’s newer contractors in the U.S. and Canada, the problems sound familiar. Even when work remains and payments arrive on time, contractors voice a litany of complaints on Reddit, LinkedIn, Glassdoor, and Medium posts. The issues that come up again and again are around rate reductions, unpredictable work, and a lack of guidance from supervisors.

Work for some

Not everyone is so frustrated that they are leaving or have left the platform. Ashley, who lives in Texas, began working for Outlier at $40 an hour as a second job (she’s a middle school teacher). While she enjoyed the work and found it “fairly simple,” the platform changes have made the side gig feel unsteady. When the company reduced pay, she says many workers grew frustrated.

“For some, this is their sole income, so they were more upset than others,” she says. “I still have a full-time job and a second income from my husband. For me, it’s just extra money to pay down our credit card debts and add to our savings account.”

Jane, who lives in the Midwest, started working for Remotasks about a year ago as a part-time gig while she worked on an advanced degree. Initially, she was able to work around 20 hours a week easily, but now it’s more challenging to get consistent work.

Now at Outlier, Jane highlighted several issues, including frequent changes in project standards and rules, sudden cuts to positions, and a lack of clear communication from the company. Workers often feel undervalued and that their concerns go unheard, she says.

Despite these challenges, Jane continues to work for Outlier, though she’s now looking for other opportunities. She emphasized the need for more transparency and better treatment of workers, noting how the company’s rapid growth and constant hiring have led to internal issues and high turnover.

“It’s really hard, especially when you’re not even considered an employee,” says Jane. “They’re expecting a lot of employee requirements from their independent contractors.”

The gig economy switcheroo

Benjamin Shestakofsky, an assistant professor of sociology at UPenn, sees the emergence of onshore AI training labor as fitting into broader trends in data enrichment work. He notes that as AI has advanced, companies are moving away from undifferentiated crowd work platforms toward more structured labor arrangements.

“More broadly, we’ve seen that this data enrichment work has been increasingly organized by traditional outsourcing companies,” Shestakofsky explains. These companies provide workers with training and oversight, creating what he calls “more deeply embedded employment relationships.”

While these arrangements can offer some benefits to workers, such as potentially higher and more consistent pay, Shestakofsky cautions that the jobs often remain precarious. He points out that demand for work can be unreliable, and the multilayered structure of outsourcing can create opacity, making it difficult for workers to address problems.

Shestakofsky also sees a parallel to other platform-based gig economy jobs, noting how venture capital funding cycles can impact worker conditions. As companies move from proving their concept to showing profitability, there’s often pressure to cut costs. “The focus shifts from this proof of concept to showing us you can get these workers to show us how you can monetize this,” he explains.

Regarding the relationship between AI companies like Scale AI and their major clients, such as OpenAI and Facebook, Shestakofsky says that this dynamic can lead to downward pressure on wages as large clients expect discounts at scale.

“If Scale is dependent on those clients, that gives those companies a lot of leverage,” he says.

Those clients and the rest of the tech industry might be experiencing their own pressure to reduce costs. OpenAI could lose as much as $5 billion this year due to the towering costs of training its AI systems, according to The Information. Recently, Alphabet saw its stock tumble after lackluster AI performance in its latest earnings. After the past two years of tech layoffs, a downturn in AI could continue to impact jobs, including for these types of digital workers, according to Valente. Companies are able to end contracts with companies like Scale AI and move to other platforms and use a different pool of workers.

“Those companies say workers have flexibility,” he says. “But at the end of the day, it’s the companies that have the flexibility to manage the workforce.”

That includes workers like Stephen, a 2021 English major graduate from New York, who struggled to break into a job that would let him have some sort of writing career. In March, he saw an opportunity with Outlier and decided to quit his existing service job to take it.

“I found it had been fun to interact with ChatGPT,” Stephen says. “When I got hired as a senior writer, I thought, ‘That sounds awesome, I needed a real stepping stone. This is my way in.'”

The promise of $35 to $40 per hour as a senior writer seemed like the perfect chance. However, reality quickly set in. The pay dwindled to between $17 and $25 per hour, depending on the project. Training pay was cut, and he found himself frequently reassigned.

“You don’t have someone to go to,” he explains. “I just felt like I was annoying people with questions because I had little to no guidance. I was promised a certain amount of money and kept getting moved around.”

Despite initially liking the work, Stephen faced inconsistent grading from reviewers and long periods without assignments. He now found himself earning less than at his previous job, struggling to support himself and help care for his parents.

“Outlier got my hopes up,” he says. “I feel like I was promised something and it was ripped away.”

Despite his disappointment, Stephen remains interested in AI. After all, he says, “AI needs human intervention. It can’t teach itself.”