International Defense Security & Technology Your trusted Source for News, Research and Analysis
Related Articles
The rise of digital media has brought unparalleled opportunities for information sharing, but it has also created fertile ground for large-scale disinformation attacks. As the ability to generate falsified media—whether through altered images, deepfake videos, or fabricated text—has advanced rapidly, so too has the threat of automated disinformation at an unprecedented scale. Recognizing this growing challenge, the Defense Advanced Research Projects Agency (DARPA) launched the Semantic Forensics (SemaFor) program. The goal of SemaFor is to develop cutting-edge technologies that can automatically detect, attribute, and characterize falsified multi-modal media assets such as text, audio, images, and video.
In this technical article, we will delve into the specifics of the SemaFor program, its objectives, the technologies being developed, and the potential impact of this initiative in defending against disinformation attacks.
In recent years consumer imaging technology (digital cameras, mobile phones, etc.) has become ubiquitous, allowing people the world over to take and share images and video instantaneously. Mirroring this rise in digital imagery is the associated ability for even relatively unskilled users to manipulate and distort the message of the visual media. While many manipulations are benign, performed for fun or for artistic value, others are for adversarial purposes, such as propaganda or misinformation campaigns.
This manipulation of visual media is enabled by the wide-scale availability of sophisticated image and video editing applications that permit editing in ways that are very difficult to detect either visually or with current image analysis and visual media forensics tools. The problem isn’t limited to the fashion and cosmetics industries where photos are “touched-up” and “augmented” to make models look better and the results of skin-care products look (instantly) appealing — it’s spread to politics and now even business.
The most infamous form of this kind of content is the category called “deepfakes” — usually pornographic video that superimposes a celebrity or public figure’s likeness into a compromising scene. There are increasing instances of social media being abused and used to abuse the elections. Where things get especially scary is the prospect of malicious actors combining different forms of fake content into a seamless platform,” Andrew Grotto at the Center for International Security at Stanford University said. “Researchers can already produce convincing fake videos, generate persuasively realistic text, and deploy chatbots to interact with people. Imagine the potential persuasive impact on vulnerable people that integrating these technologies could have: an interactive deepfake of an influential person engaged in AI-directed propaganda on a bot-to-person basis.”
False news stories and so-called deepfakes are increasingly sophisticated and making it more difficult for data-driven software to spot. Though software that makes deepfakes possible is inexpensive and easy to use, existing video analysis tools aren’t yet up to the task of identifying what’s real and what’s been cooked up. The media and internet landscapes have seen manipulated videos, audio, images, and stories that spread disinformation, and the Defense Advanced Research Projects Agency (DARPA) is seeking solutions to help it detect and combat the manipulation.
However, existing automated media generation and manipulation algorithms are heavily reliant on purely data driven approaches and are prone to making semantic errors. For example, GAN-generated faces may have semantic inconsistencies such as mismatched earrings. These semantic failures provide an opportunity for defenders to gain an asymmetric advantage. A comprehensive suite of semantic inconsistency detectors would dramatically increase the burden on media falsifiers, requiring the creators of falsified media to get every semantic detail correct, while defenders only need to find one, or a very few, inconsistencies.
The Challenge of Multi-Modal Disinformation
Disinformation is not confined to one medium. False narratives can spread through a combination of manipulated text, images, audio, and video, often together in the form of multi-modal media campaigns. These campaigns are becoming increasingly sophisticated, leveraging advancements in machine learning and AI to create highly convincing but falsified content. Traditional methods of detecting falsified media are slow, labor-intensive, and often ineffective in the face of large-scale, automated disinformation attacks.
AI-generated media, such as deepfakes, have added an additional layer of complexity. Deepfakes are synthetic media in which a person in an image or video is replaced with someone else’s likeness in a highly realistic way. These tools can be used to spread misinformation at scale, making it nearly impossible for manual methods to keep up with the growing volume and sophistication of falsified content.
DARPA launched Semantic Forensics (SemaFor) program with aim to develop technologies to automatically detect, attribute, and characterize falsified multi-modal media assets (text, audio, image, video) to defend against large-scale, automated disinformation attacks.
Objectives of the SemaFor Program
In August 2019, DARPA announced the Semantic Forensics (SemaFor) program as part of a broad agency announcement aimed at addressing small but common errors generated by automated media manipulation systems. These errors, often found in synthetic media produced by generative adversarial networks (GANs), can include details like mismatched earrings on a fabricated image of a woman’s face. While these semantic errors are easier to spot than they are to avoid, DARPA’s goal is to harness cutting-edge technologies to detect and address these inconsistencies.
The SemaFor program, DARPA says, will explore ways to get around some of the weaknesses of current deepfake detection tools. Statistical detection techniques have been successful, but media generation and manipulation technology is advancing rapidly. Purely statistical detection methods are quickly becoming insufficient for detecting falsified media assets. Detection techniques that rely on statistical fingerprints can often be fooled with limited additional resources (algorithm development, data, or compute). Current media manipulation tools rely heavily on ingesting and processing large amounts of data, DARPA said, making them more prone to errors that can be spotted with the right algorithm.
DARPA’s SemaFor program aims to tackle the problem of automated disinformation by developing robust AI-driven technologies capable of detecting, attributing, and characterizing falsified content across various media types. Key objectives of the program include:
- Automated Detection of Falsified Media: Develop machine learning models that can automatically detect alterations or manipulations in text, audio, images, and video. This includes spotting subtle changes in media content that may not be easily detectable by the human eye or ear.
- Attribution of Media Assets: Enable the attribution of falsified media to its source, whether that be an individual, organization, or an AI model used to create the falsified content. This helps identify the actors behind disinformation campaigns and understand their motivations.
- Characterization of Media Content: Characterize the intent behind the manipulated content by analyzing the context and the nature of the falsifications. This involves understanding not just how a media asset was falsified but why, which can aid in identifying and combating large-scale disinformation efforts.
- Multi-Modal Analysis: Detect and analyze cross-media relationships in multi-modal disinformation. By considering text, audio, image, and video together, SemaFor aims to identify inconsistencies or correlations between various media types that point to coordinated falsification efforts.
- Scalability and Real-Time Detection: Create solutions that are scalable and capable of detecting falsified media in real-time to address the speed and volume of modern disinformation campaigns. Traditional manual methods are often too slow to respond to rapidly spreading false narratives; SemaFor seeks to automate the process to keep pace with evolving threats.
“From a defense standpoint, SemaFor is focused on exploiting a critical weakness in automated media generators,” said Dr. Matt Turek, the DARPA program manager leading SemaFor. “Currently, it is very difficult for an automated generation algorithm to get all of the semantics correct. Ensuring everything aligns from the text of a news story, to the accompanying image, to the elements within the image itself is a very tall order. Through this program we aim to explore the failure modes where current techniques for synthesizing media break down.”
“A key goal of the program is to establish an open, standards-based, multisource, plug-and-play architecture that allows for interoperability and integration,” DARPA said. “This goal includes the ability to easily add, remove, substitute, and modify software and hardware components in order to facilitate rapid innovation by future developers and users.”
Technical Areas of Focus
SemaFor seeks to develop innovative semantic technologies for analyzing media. Semantic detection algorithms will determine if media is generated or manipulated. Attribution algorithms will infer if media originates from a particular organization or individual. Characterization algorithms will reason about whether media was generated or manipulated for malicious purposes. These SemaFor technologies will help identify, deter, and understand adversary disinformation campaigns. “A comprehensive suite of semantic inconsistency detectors would dramatically increase the burden on media falsifiers, requiring the creators of falsified media to get every semantic detail correct, while defenders only need to find one, or a very few, inconsistencies,” DARPA said.
The SemaFor program is divided into four key technical areas: detection; attribution and characterization; explanation and integration; and evaluation and challenge curation. DARPA’s objective is for the resulting algorithms to outperform manual processes, while also providing transparent explanations of how conclusions are reached. Teams working on these technical areas will develop algorithms capable of identifying and attributing falsified multimedia content, which will then be integrated into a comprehensive system for further analysis.
1. Semantic Integrity Assessment
This area involves determining the semantic consistency of media content. For instance, the system would assess whether the information conveyed in a video (e.g., the words spoken by a person) is consistent with the accompanying text or image. By understanding the relationships between various media components, SemaFor can detect subtle mismatches that indicate falsification.
2. AI-Based Media Attribution
Attribution is key to understanding the source of falsified media. SemaFor aims to develop technologies that can track the origin of AI-generated content by identifying digital fingerprints left behind by specific generative models or editing tools. This will enable a detailed understanding of the tools and actors responsible for generating disinformation.
3. Explainable AI for Detection
One of the challenges of using AI to detect falsified media is the “black box” nature of many machine learning models, which makes it difficult to explain why the model flagged a particular piece of content as falsified. SemaFor is exploring methods for making AI decisions explainable, allowing users to understand how and why the model reached a particular conclusion. This increases trust in the system and helps security teams verify results.
4. Cross-Modal Analysis
Disinformation campaigns often rely on a combination of falsified media types. For example, an altered image might be paired with a fabricated news article or video. SemaFor’s cross-modal analysis capability focuses on detecting inconsistencies across these different media types by analyzing how well the text, audio, images, and video align with one another. Discrepancies between them could be indicative of manipulation.
5. Adaptive Learning
As disinformation tactics evolve, so too must detection systems. SemaFor aims to incorporate adaptive learning technologies that allow its detection models to improve over time, learning from new types of falsified media and disinformation techniques. This is critical for staying ahead of adversaries who continually refine their methods.
Expected Outcomes
SemaFor’s success will lead to several crucial outcomes in the fight against disinformation:
- Automated and Scalable Disinformation Detection: The tools developed through SemaFor will provide automated, scalable solutions for detecting falsified media, reducing the reliance on manual analysis and enabling rapid responses to disinformation campaigns.
- Robust Attribution Capabilities: Attribution tools will allow the identification of both human and AI sources behind falsified content, enabling law enforcement, government agencies, and other stakeholders to take appropriate action against those spreading disinformation.
- Enhanced Situational Awareness: The ability to detect, attribute, and characterize falsified media in real-time will enhance situational awareness, providing decision-makers with a clearer understanding of the threats they face and how to respond.
- Trust in Media: As falsified content becomes harder to detect manually, SemaFor’s tools will help restore public trust in digital media by providing reliable, automated verification of content authenticity.
TA1 performers will be responsible for developing algorithms that detect, characterize, and attribute manipulated media. These will be integrated into a larger system by TA2 performers, who will collaborate with TA1 teams to ensure scalability and cloud deployment. The integrated system will be reviewed by human analysts, emphasizing that this process is not entirely automated. Periodic proof-of-concept demonstrations will showcase the progress of the SemaFor system in each phase of the program.
TA3 performers will focus on evaluation and curation, conducting tests and analyzing results to refine the system. Notably, TA3 teams will also create and collect media content for testing purposes, highlighting DARPA’s ambition to develop tools that can not only detect but also simulate disinformation campaigns. TA3’s role will include establishing metrics, evaluation protocols, and a library of multi-modal media assets, positioning the U.S. Department of Defense (DoD) to anticipate and counter emerging disinformation threats.
Finally, in TA4, DARPA will focus on proactive threat modeling, curating state-of-the-art challenges to ensure SemaFor addresses evolving threats. Teams will collaborate to cover a wide range of threat models and scenarios, regularly updating the evaluation team with new challenges. This anticipatory approach will help keep the SemaFor program relevant and effective in combating future disinformation and media manipulation threats.
A key goal of the program is to establish a flexible, open architecture that facilitates innovation. By allowing interoperability and easy modification of software and hardware components, DARPA hopes to enable rapid advancements by future developers. The program’s success will be evaluated based on the performance of its algorithms in detection, attribution, characterization, and other areas, as well as its ability to anticipate future threats.
DARPA Awards
PAR Government Systems Corp., Rome, New York, was awarded an $11,920,160 cost-plus-fixed-fee contract for a research project under the Semantic Forensics (SemaFor) program. The SemaFor program will develop methods that exploit semantic inconsistencies in falsified media to perform tasks across media modalities and at scale. Work will be performed in Rome, New York, with an expected completion date of June 2024. Fiscal 2020 research, development, test and evaluation funding in the amount of $1,500,000 are being obligated at time of award. This contract was a competitive acquisition under a full and open broad agency announcement and 37 proposals were received.
In March 2021, DARPA announced the research teams selected to take on SemaFor’s research objectives. Teams from commercial companies and academic institutions will work to develop a suite of semantic analysis tools capable of automating the identification of falsified media. Arming human analysts with these technologies should make it difficult for manipulators to pass altered media as authentic or truthful.
Four teams of researchers will focus on developing three specific types of algorithms: semantic detection, attribution, and characterization algorithms. These will help analysts understand the “what,” “who,” “why,” and “how” behind the manipulations as they filter and prioritize media for review.
The teams will be led by Kitware, Inc., Purdue University, SRI International, and the University of California, Berkeley. Leveraging some of the research from another DARPA program – the Media Forensics (MediFor) program – the semantic detection algorithms will seek to determine whether a media asset has been generated or manipulated. Attribution algorithms will aim to automate the analysis of whether media comes from where it claims to originate, and characterization algorithms seek to uncover the intent behind the content’s falsification.
To help provide an understandable explanation to analysts responsible for reviewing potentially manipulated media assets, SemaFor also is developing technologies for automatically assembling and curating the evidence provided by the detection, attribution, and characterization algorithms. Lockheed Martin – Advanced Technology Laboratories will lead the research team selected to take on the development of these technologies and will develop a prototype SemaFor system.
“When used in combination, the target technologies will help automate the detection of inconsistencies across multimodal media assets. Imagine a news article with embedded images and an accompanying video that depicts a protest. Are you able to confirm elements of the scene location from cues within the image? Does the text appropriately characterize the mood of protestors, in alignment with the supporting visuals? On SemaFor, we are striving to make it easier for human analysts to answer these and similar questions, helping to more rapidly determine whether media has been maliciously falsified,” Turek said.
To ensure the capabilities are advancing in line with – or ahead of – the potential threats and applications of altered media, research teams are also working to characterize the threat landscape and devise challenge problems that are informed by what an adversary might do. The teams will be led by Accenture Federal Services (AFS), Google/Carahsoft, New York University (NYU), NVIDIA, and Systems & Technology Research.
Google/Carahsoft will provide perspective on disinformation threats to large-scale internet platforms, while NVIDIA will provide media generation algorithms and insights into the potential impact of upcoming hardware acceleration technologies. NYU provides a link to the NYC Media Lab and a broad media ecosystem that will provide insights into the evolving media landscape, and how it could be exploited by malicious manipulators. In addition, AFS provides evaluation, connectivity, and operational viability assessment of SemaFor in application to the Department of State’s Global Engagement Center, which has taken the lead on combating overseas disinformation.
Finally, ensuring the tools and algorithms in development have ample and relevant training data, researchers from PAR Government Systems have been selected to lead data curation and evaluation efforts on the program. The PAR team will be responsible for carrying out regular, large scale evaluations that will measure the performance of the capabilities developed on the program.
Applications in National Security
SemaFor’s technologies will have broad applications across national security and defense. By enabling rapid detection of falsified media in disinformation campaigns, SemaFor will help mitigate the impact of information warfare tactics used by hostile state and non-state actors. The military, intelligence agencies, and law enforcement can leverage these tools to protect critical operations from being undermined by fake news, deepfake videos, or other forms of manipulated media.
Moreover, the ability to attribute media to specific actors could be instrumental in cybersecurity efforts, enabling authorities to track down the origin of disinformation attacks and hold those responsible accountable. The integration of these tools into defense systems will help bolster the country’s resilience against cyber and information warfare.
Conclusion
DARPA’s Semantic Forensics (SemaFor) program is poised to become a critical asset in the battle against large-scale disinformation campaigns. By developing automated tools to detect, attribute, and characterize falsified media across multiple formats, SemaFor will significantly enhance national defense capabilities in the face of evolving threats. As AI-generated media becomes more prevalent and sophisticated, the technologies emerging from the SemaFor program will provide essential protection for the integrity of information in the digital age.
With its emphasis on explainability, adaptability, and cross-modal analysis, the SemaFor program represents a new frontier in the fight to safeguard public discourse and national security from the dangers of disinformation.
