Today large amounts of data is collected from numerous sources, such as social media, sensor feeds (e.g. cameras), and scientific data. There are over 1 billion websites on the world wide web today and the Annual global IP traffic will reach 3.3 ZB per year by 2021, or 278 exabytes (EB) per month. In 2016, the annual run rate for global IP traffic was 1.2 ZB per year, or 96 EB per month. The notion of Big Data emerges from the observation that 90 percent of the data available today has been created in just the past two years. From devices at the edge to large data centers crunching everything from corporate clouds to future energy technology simulations, the world is awash in data – being stored, indexed and accessed, says Intel. The goal of DARPA’s Hierarchical Identify Verify Exploit (HIVE) programme is to explore new and more efficient methods of processing large amounts of complex data.
DoD, has to make sense and make decisions based on large amount of data it collects like communications, intelligence, surveillance, and reconnaissance from drones, automated cybersecurity systems. Real-time predictive large-scale data analytics can provide decisive advantage to commanders across a range of military operations in the homeland and abroad, information supremacy, enhancing autonomy technologies and vastly improved situational awareness to aid warfighters and intelligence analysts, according to ARL.
In the big data era, information is often linked to form large-scale graphs. Graph analytics has emerged as a way to understand the relationships between these heterogeneous types of data, allowing analysts to draw conclusions from the patterns in the data and to answer previously unthinkable questions. By understanding the complex relationships between different data feeds, a more complete picture of the problem can be understood and some amount of causality may be inferred.
There is also an increasing need to make decisions in real time, which requires understanding how the inherent relationships in the graph evolve over time. This emerging data analytics technology is also used for applications like cyber defense and critical infrastructure protection that require analyzing huge data sets in real time.
Currently much of graph analytics is performed in large data centers on large cached or static data sets and the amount of processing required is a function of not only the size of the graph, but the type of data being processed. DARPA’s Hierarchical Identify Verify Exploit (HIVE) program that seeks to develop a generic and scalable graph processor that specializes in processing sparse graph primitives, and achieves 1000-times improvement in processing efficiency over standard processors.
In combination with emerging machine learning and other artificial intelligence techniques that can categorize raw data elements, and by updating the elements in the graph as new data becomes available, a powerful graph analytics processor could discern otherwise hidden causal relationships and stories among the data elements in the graph representations.
Most graph processing problems require large server-class type computers with a large size, weight, and power (SWaP) requirements. But the scale required limits what can be done in a tactical environment. HIVE is expected to overcome that challenge and enable processing of information at the tactical edge, Boyle said. “The hard problem is getting the processor down into a form factor and a SWaP footprint that is compatible with a tactical environment and then using it in an environment where you are really working towards this future of cognitive autonomy and intelligent systems,” he added.
“You are seeing companies shift from general-purpose computing devices to purpose built; that is what the HIVE chip is, it is a purpose-built chip just for graph processing,” he said. “This is a key enabler to the future that you hear the customers talking about. It is not just about being able to process graphs, this is one of the core technologies required for cognitive systems. That technology more broadly speaking will impact just about every aspect of war fighting in the future.”
The concept phase of the HIVE program extends though next year, with initial prototyping beginning in fiscal 2019. Chip fabrication could begin as early as fiscal 2020, the agency said.
IDST Monthly Access Membership Required
You must be a IDST Monthly Access member to access this content.