There’s been a lot of fear, uncertainty, and doubt (FUD) about the potential for generative AI to take people’s jobs. The capability of large language models (LLMs) to answer questions and handle digital tasks when prompted has caught people’s attention, for better and for worse. But what are the odds that LLMs actually will replace human workers? A new study from Indeed sheds some light on that question.

The digital job board Indeed recently conducted a test to determine the efficacy of LLMs at handling basic work skills. The Indeed Hiring Lab signed up for GPT-4o, the latest LLM from OpenAI, and asked it to perform more than 2,800 job skills tracked in the Indeed database, from office jobs like account management and insurance claims to more physically demanding jobs, like bus driver and cook.

For each job skill, the Indeed Hiring Lab set up a way to measure how successful the LLM completed the task. They created sophisticated, 1,000-word prompts for each task, which took a lot of trial and error. After finally settling on the best prompt, the Hiring Lab workers ran the prompt by GPT-4o 15 times, and then aggregated the result. GPT-4o was asked to assess its own capability at each prompt, and the results were validated by human researchers.

The Hiring Lab focused on three main areas with the experiment, including the capability of GenAI to provide theoretical knowledge related to the skill; the capability of GenAI to solve problems using the skill; and GenAI’s determination of the importance of physical presence in utilizing that skill. GPT-4o analyzed its own capability to utilize these attributes with a given job on a five-point scale. The researchers tabulated the results and published them last week in a paper titled “AI at Work: Why GenAI is More Likely to Support Workers Than Replace Them,” which you can download here.

The title is a big hint at Indeed’s findings with the GenAI experiment. The report’s authors, Annina Hering and Arcenis Rojas, write that none of the 2,800 work skills are “very likely” to be replaced by GPT-4o or any other LLM. In fact, Indeed found nearly 69% of the skills are either “very likely” or “unlikely” to be replaced by GenAI.

Clearly, no jobs that require hands-on execution or the application of physical force, such as bus driver or emergency room nurse, are going to be replaced by GenAI, which is just software at the end of the day (self-driving buses and robot-assisted surgery are real, but they also require a lot more tech than just GenAI). Considering that more than half of jobs involved in this report required some type of physical execution, the prospects of full GenAI replacement look pretty bleak.

But that’s not to say there will be no benefit. Indeed says that, even for jobs like bus driver or nurse, GenAI could help with repetitive tasks, such as documentation, which will “allow workers to refocus on the core skills necessary in these roles,” Hering and Rojas write.

The researchers concluded that about 29% of jobs could “potentially” be replaced by GenAI “as it continues to improve and if certain changes to workplaces and/or working norms occur going forward,” the researchers write. The jobs that GenAI will have the biggest impact are “more stereotypical office jobs,” the researchers write.

Across the three measures at the heart of the study–theoretical knowledge; problem solving; and physical job skills–GenAI excels the most with theoretical knowledge, followed closely by problem solving. In fact, theoretical knowledge was the only attribute that GenAI gave itself a 5, the top score, thanks to the extensive training of LLMs on large amounts of information on the Web, and the capability to use search engines.

GPT-4o also scored decently on problem-solving. It rated itself a 3 for 70% of the skills it assessed, and for 28% of those tasks, it said it was “possible” that it could replace a human. It also received several 4s, and rated itself “likely” that could replace a human for 3% of the tasks.

GenAI is most likely to replace humans at office jobs and jobs that are done predominantly on the computer. For instance, researchers concluded that it was “possible” or “likely” that GenAI could replace a human at more than 71% of skills commonly found in job postings for software development. Similarly, GenAI was “possible” or “likely” to replace humans for 78% of skills commonly found in a typical accounting occupation, the report says.

GenAI is less likely to replace humans at jobs that require more problem-solving than theoretical knowledge. This is an area where GenAI developers and data scientists may want to focus their efforts.

“If GenAI models improve their problem-solving abilities for more skills within more jobs,” Hering and Rojas write, “it’s likely that the share of skills that may eventually be replaced in those jobs will also rise

There are things that companies can do to help them prepare for GenAI. In the accounting field, for example, investments in electronic record-keeping and digitization will go a long way towards preparing a firm to successfully use GenAI.

Fine-tuning (no pun intended) one’s interaction with GenAI can also yield better results. For instance, a loose prompt can be interpreted any number of ways by an LLM, which is likely to give different answers every time it’s asked. More advanced tasks will require better prompt writing and prompt engineering skills to get the most out of GenAI, the authors write.

At the end of the day, it seems likely that GenAI will replace at least some of the tasks that human workers are doing now, with lots of variation by industry and position. However, Indeed’s researchers don’t see a time in the near future when GenAI will replace humans en masse, simply because GenAI, as it exists today, can’t function without humans.

“Even as GenAI evolves and learns to complete demanding tasks,” Hering and Rojas write, “humans that oversee, guide, and correct GenAI-derived output will not easily be replaced.”