Feb. 7 — Much of the data of the World Wide Web hides like an iceberg below the surface. The so-called ‘deep web’ has been estimated to be 500 times bigger than the ‘surface web’ seen through search engines like Google. For scientists and others, the deep web holds important computer code and its licensing agreements. Nestled further inside the deep web, one finds the ‘dark web,’ a place where images and video are used by traders in illicit drugs, weapons, and human trafficking. A new data-intensive supercomputer called Wrangler is helping researchers obtain meaningful answers from the hidden data of the public web.

The Wrangler supercomputer got its start in response to the question, can a computer be built to handle massive amounts of I/O (input and output)? The National Science Foundation (NSF) in 2013 got behind this effort and awarded the Texas Advanced Computing Center (TACC), Indiana University, and the University of Chicago $11.2 million to build a first-of-its-kind data-intensive supercomputer. Wrangler’s 600 terabytes of lightning-fast flash storage enabled the speedy reads and writes of files needed to fly past big data bottlenecks that can slow down even the fastest computers. It was built to work in tandem with number crunchers such as TACC’s Stampede, which in 2013 was the sixth fastest computer in the world.

While Wrangler was being built, a separate project came together headed by the Defense Advanced Research Projects Agency (DARPA) of the U.S. Department of Defense. Back in 1969, DARPA had built the ARPANET, which eventually grew to become the Internet, as a way to exchange files and share information. In 2014, DARPA wanted something new – a search engine for the deep web. They were motivated to uncover the deep web’s hidden and illegal activity, according to Chris Mattmann, chief architect in the Instrument and Science Data Systems Section of the NASA Jet Propulsion Laboratory (JPL) at the California Institute of Technology.

“Behind forms and logins, there are bad things. Behind the dynamic portions of the web like AJAX and Javascript, people are doing nefarious things,” said Mattmann. They’re not indexed because the web crawlers of Google and others ignore most images, video, and audio files. “People are going on a forum site and they’re posting a picture of a woman that they’re trafficking. And they’re asking for payment for that. People are going to a different site and they’re posting illicit drugs, or weapons, guns, or things like that to sell,” he said.

Mattmann added that an even more inaccessible portion of the deep web called the ‘dark web’ can only be reached through a special browser client and protocol called TOR, The Onion Router. “On the dark web,” said Mattmann, “they’re doing even more nefarious things.” They traffic in guns and human organs, he explained. “They’re basically doing these activities and then they’re tying them back to terrorism.”

In response, DARPA started a program called Memex. Its name blends ‘memory’ with ‘index’ and has roots to an influential 1945 Atlantic magazine article penned by U.S. engineer and Raytheon founder Vannevar Bush. His futuristic essay imagined making all of a person’s communications – books, records, and even all spoken and written words – in fingertip reach. The DARPA Memex program sought to make the deep web accessible. “The goal of Memex was to provide search engines the information retrieval capacity to deal with those situations and to help defense and law enforcement go after the bad guys there,” Mattmann said.

Karanjeet Singh is a University of Southern California graduate student who works with Chris Mattmann on Memex and other projects. “The objective is to get more and more domain-specific (specialized) information from the Internet and try to make facts from that information,” said Singh said. He added that agencies such as law enforcement continue to tailor their questions to the limitations of search engines. In some ways the cart leads the horse in deep web search. “Although we have a lot of search-based queries through different search engines like Google,” Singh said, “it’s still a challenge to query the system in way that answers your questions directly.”

Once the Memex user extracts the information they need, they can apply tools such as named entity recognizer, sentiment analysis, and topic summarization. This can help law enforcement agencies like the U.S. Federal Bureau of Investigations find links between different activities, such as illegal weapon sales and human trafficking, Singh explained.

“Let’s say that we have one system directly in front of us, and there is some crime going on,” Singh said. “The FBI comes in and they have some set of questions or some specific information, such as a person with such hair color, this much age. Probably the best thing would be to mention a user ID on the Internet that the person is using. So with all three pieces of information, if you feed it into the Memex system, Memex would search in the database it has collected and would yield the web pages that match that information. It would yield the statistics, like where this person has been or where it has been sited in geolocation and also in the form of graphs and others.”

“What JPL is trying to do is trying to automate all of these processes into a system where you can just feed in the questions and and we get the answers,” Singh said. For that he worked with an open source web crawler called Apache Nutch. It retrieves and collects web page and domain information of the deep web. The MapReduce framework powers those crawls with a divide-and-conquer approach to big data that breaks it up into small pieces that run simultaneously. The problem is that even the fastest computers like Stampede weren’t designed to handle the input and output of millions of files needed for the Memex project.

The Wrangler data-intensive supercomputer avoids data overload by virtue of its 600 terabytes of speedy flash storage. What’s more, Wrangler supports the Hadoop framework, which runs using MapReduce. “Wrangler, as a platform, can run very large Hadoop-based and Spark-based crawling jobs,” Mattmann said. “It’s a fantastic resource that we didn’t have before as a mechanism to do research; to go out and test our algorithms and our new search engines and our crawlers on these sites; and to evaluate the extractions and analytics and things like that afterwards. Wrangler has been an amazing resource to help us do that, to run these large-scale crawls, to do these type of evaluations, to help develop techniques that are helping save people, stop crime, and stop terrorism around the world.”

Click here to view the entire article.

---

Source: Jorge Salazar, TACC