A recent article in the New York Times points out that sophisticated data analytics software is doing the kinds of jobs once reserved for highly paid specialists. Specifically, it talks about data mining-type software applied to document discovery for lawsuits. In this realm, these applications are taking the place of expensive teams of lawyers and paralegals.
Basically it works by performing deep analysis of text to find documents pertinent to the case at hand. It’s not just a dumb keyword search; the software is smart enough to find relevant text even in the absence of specific terms. One application was able to analyze 1.5 million documents for less than $100,000 — a fraction of the cost of a legal team, and performed in a fraction of the time.
Mike Lynch, founder of Autonomy (a UK-based e-discovery company), thinks this will lead to a shrinking legal workforce in the years ahead. From the article:
He estimated that the shift from manual document discovery to e-discovery would lead to a manpower reduction in which one lawyer would suffice for work that once required 500 and that the newest generation of software, which can detect duplicates and find clusters of important documents on a particular topic, could cut the head count by another 50 percent.
Such software can also be used to connect chains of events mined from a variety of sources: e-mail, instant messages, telephone calls, and so on. Used in this manner, it can be used to sift out digital anomalies to track various types of criminal behavior. Criminals, of course, are one workforce we’d like to reduce. But what about the detectives that used to perform this kind of work?
The broader point the NYT article illuminates is that software like this actually targets mid-level white collar jobs, rather than low-end labor jobs we usually think of as threatened by computer automation. According to David Autor, an economics professor at MIT, this is leading to a “hollowing out” of the US economy. While he doesn’t think technology like this is driving unemployment per se, he believes the job mix will inevitably change, and not necessarily for the better.
It’s the post-Watson era. Get used to it.