DARPA Takes on the Dark Web with MEMEX

Feb 11 2015 15:21 @patlalrique

dark web deep web darpa memex

A science data project from the Defense Advanced Research Projects Agency (DARPA) named [inlinetweet prefix=”@patlalrique” tweeter=”” suffix=””]MEMEX will contribute in part to accomplish the gigantic task of classify the whole internet[/inlinetweet]. This work is a major attempt to shine light on the Dark Web indexing and cataloging the part of internet that isn’t.

dark web deep web darpa memex tor

Most of the internet that is indexed and reachable are pages that are used, reached and read in an advertisement context. Google ranks pages based on popularity and try to understand them to deliver targeted advertisement. It is how most of the internet works. But the Dark Web escapes this reality as websites down there don’t want to be found, publicized or are simply not meant to be found, having a purpose aimed at a specific user group or organisation.

The move is clearly an attempt to find a way to monitor Tor and other proxy services. Since Tor is far from being the whole Dark Web, MEMEX is aimed at identifying what part of the hidden service sites is related to Tor traffic. It is unknown how much of website relates on Tor and .onion service but Dr. Chris White, program manager at DARPA estimates them at between 30,000 to 40,000, at any given time.

Another goal MEMEX is meant to achieve is very clear: DARPA wants to create a service (they say automated mechanism) that will allow the discovery of content on hidden websites and the access to that content. DARPA has high interests in understanding data from that chunk of the Internet. It will allow the intelligence agency to have a better understanding of the whole Internet as well. The Dark Web is connected in many ways to the surface. It is not entirely illegal after all.

With MEMEX will enable the agency to monitor many things like content movement, connections between content, location, device, similarities in content, websites and services. Since content is encrypted or hidden in the Dark Web, DARPA is trying to build tools that enable them to break the protections embedded and see the Dark Web intelligently, who feed it, with what type of content and all the hidden relationships between these variables.

sharing is caring please share k thanks!