Military systems are increasingly using software to support functionality, new capabilities, and beyond. Before a new piece of software can be deployed within a system however, its functional safety and compliance with certain standards must be verified and ultimately receive certification. As the rapid rate of software usage continues to grow, it is becoming exceedingly difficult to assure that all software considered for military use is coded correctly and then tested, verified, and documented appropriately.
“Software requires a certain level of certification – or approval that it will work as intended with minimal risks – before receiving approval for use within military systems and platforms,” said Dr. Ray Richards, a program manager in DARPA’s Information Innovation Office (I2O). “However, the effort required to certify software is an impediment to expeditiously developing and fielding new capabilities within the defense community.”
“Certification” is the process of determining that a system’s risk is acceptable. As the DOD and armed forces rely more on software and artificial intelligence platforms, it will be more critical to assure that the software they are deploying is coded correctly and that vulnerabilities are detected quickly. Current certification practices within the Department of Defense (DoD) are antiquated and unable to scale with the amount of software deployed.
Two factors prevent scaling: Today, the software certification process is largely manual and relies on human evaluators combing through piles of documentation, or assurance evidence, to determine whether the software meets certain certification criteria. The process is time consuming, costly, and can result in superficial or incomplete evaluations as reviewers bring their own sets of expertise, experiences, and biases to the process.
The amount of assurance evidence needed to determine software conformance to certification can be overwhelming to human subject matter experts, resulting in superficial, incomplete, and/or unacceptably long evaluations. Human evaluators also have unique expertise, experience, and biases that influence their approach to evaluations. Because certification requirements may be vague or poorly written, evaluators often must interpret what is intended. Combined, these factors result in inconsistencies over time and across evaluations.
Composed evaluations in a principled and trustworthy manner would allow subsystems or components to be evaluated independently. The results of those independent evaluations could then be leveraged as assurance evidence in the composing systems. This would amortize the effort of evaluating any component over all systems using that component. Current practice requires re-evaluating components and their assurance evidence in every system that employs them. The inability to use a divide-and-conquer approach to certification of large systems increases the cost and time required to perform these certifications. This creates additional time delays and review cycles.
A lack of a principled means of decomposing evaluations makes it difficult to create a balanced and trustworthy process that applies equally to all software. Further, each subsystem and component must be evaluated independently and re-evaluated before it can be used in a new system. “Just because a subsystem is certified for one system or platform does not mean it is unilaterally certified for all,” noted Richards. This creates additional time delays and review cycles.
“Software requires a certain level of certification – or approval that it will work as intended with minimal risks – before receiving approval for use within military systems and platforms,” said Dr. Ray Richards, a program manager in DARPA’s Information Innovation Office (I2O). “However, the effort required to certify software is an impediment to expeditiously developing and fielding new capabilities within the defense community.”
DARPA launched the Automated Rapid Certification Of Software (ARCOS) program in May 2019 with a goal to automate system risk assessment based on software assurance. It will automate the evaluation of software assurance evidence so that certifiers can rapidly determine if system risk is acceptable. DARPA is aiming to develop a process for continuous software certification and mission risk evaluation that can reduce impediments to developing and fielding new warfighting capabilities in a timely manner.
Automated Rapid Certification Of Software (ARCOS) program
ARCOS will create tools and a process that would allow for the automated assessment of software evidence and provide justification for a software’s level of assurance that is understandable. Taking advantage of recent advances in model-based design technology, “Big Code” analytics, mathematically rigorous analysis and verification, as well as assurance case languages, ARCOS seeks to develop a capability to automatically evaluate software assurance evidence to enable certifiers to rapidly determine that system risk is acceptable.
Two factors support the acceleration of software certification through the automation of evaluations. First, the DoD has articulated its intentions to have its contractors modernize their engineering processes in the DoD Digital Engineering Strategy. The goal of this strategy is to move away from document-based engineering processes and towards design models that are to be the authoritative source of truth for systems. Such a future does not lend itself to current certification practices, but it will facilitate the automated evaluation of assurance.
Second, advances in several technologies provide a basis for confidence that automated evaluation of assurance evidence to support certification is possible. Model-based design technology, including probabilistic model checking, may enable reasoning over a design in a way that quantifies uncertainty. So-called “Big Code” analytics have pioneered the application of semantic-based analytics to software and its associated artifacts. Mathematically rigorous analysis and verification provide the ability to develop software implementations that are demonstrably correct and sound. Assurance case languages provide us a means for expressing arguments on how software fulfills its certification goals, in a manner that is machine-readable.
To create this automated capability, ARCOS will explore techniques for automating the evidence generation process for new and legacy software; create a means of curating evidence while maintaining its provenance; and develop technologies for the automated construction of assurance cases, as well as technologies that can validate and assess the confidence of an assurance case argument. The evidence generation, curation, and assessment technologies will form the ARCOS tools and processes, working collectively to provide a scalable means of accelerating the pathway to certification.
Throughout the program’s expected three phases, evaluations and assessments will occur to gauge how the research is progressing. ARCOS researchers will tackle progressively more challenging sets of software systems and associated artifacts. The envisioned evaluation progression will move from a single software module to a set of interacting modules and finally to a realistic military software system.
DARPA Awards GrammaTech $7.6M for Safety and Certification Research in March 2020
GrammaTech, Inc., a leading developer of advanced commercial static analysis/software assurance products and advanced cybersecurity solutions, announces that it has been awarded a $7.6 million, four-year contract from Defense Advanced Research Projects Agency (DARPA).
Automated Rapid Certification of Software (ARCOS) is a DARPA program focused on generating evidence and assurance cases for a broad range of certification and/or accreditation standards. GrammaTech will develop technology that produces high-quality, traceable, and composable certification evidence for use in constructing assurance-case arguments for software that includes components available only in binary form. Although the specifics of individual accreditation and certification criteria differ across safety, security, airworthiness, and other certification standards, the criteria often share concerns about both the software attributes that should be evaluated and the types of evidence used to construct assurance cases.
GrammaTech is developing a set of tools for ARCOS that will enable the same testing strategies used for new software development to be employed for recertification of legacy software and provide better traceability, complete with rationale for why specific test results deliver sufficient requirements and structural code coverage. In addition, these tools will scale and automate test generation, execution, and test-suite maintenance to achieve measurably improved test coverage and completeness and decreasing time to deployment.
“Re-certification of software is an expensive activity,” says Alexey Loginov, Vice President of Research at GrammaTech, Inc. “It often requires significant amounts of tedious labor, that needs to be performed by experienced and hard-to-find personnel. The human factor in the process also results in inconsistent evaluation of security risk. The goal of our ARCOS contribution is to automate as much as possible of that work using GrammaTech’s advanced program-analysis capabilities.”
“GrammaTech’s research efforts move the needle in topics such as cyber security and, in this case, safety certification,” says Mike Dager, CEO at GrammaTech. “The results of these projects allow us to deliver greater value to our customers. This will directly benefit our customers that develop software against standards like DO-178C/DO-330, IEC 61508, ISO26262, CENELEC EN 50128, IEC 62443 and many others.”
GE Researchers Working to Speed Up Certification of Software for Critical Military and Industrial Systems
GE Research, the central technology development arm for the General Electric Company (GE), is leading a $10.5 million project through the Defense Advanced Research Agency’s (DARPA) Automated Rapid Certification of Software (ARCOS) program, to revolutionize the process of certifying software for critical military and industrial systems.
DARPA’s ARCOS program is focused on accelerating new developments that reduce the time and cost of certifying new software or software updates to the Department of Defense’s (DOD) large military platforms. Today, this process is largely manual and highly dependent on a variety of human subject matter experts that must evaluate millions of lines of code and other verification data when reviewing the certification of software for a typical system. The ARCOS program is aiming to achieve a more principled and automated certification process.
Kit Siu, a Principal Engineer on GE Research’s Controls and Optimization team, and Dr. Abha Moitra, a Principal Scientist in the Artificial Intelligence Group, are leading GE’s ARCOS project. Siu and Moitra are creating an automated assurance model that can quickly gather and curate the right data and evidence to analyze for certification. This is wholly different from a typical App update we might experience as consumers with our mobile devices or PCs. As part of GE’s project, Siu and Moitra will be developing and demonstrating an assurance model for a complex aerospace system, which involves a much bigger, more complex and safety critical application.
“A typical aerospace platform contains tens of millions of lines of software code,” Siu said. “If you want to make any software changes or updates to that system, you must build what is known as an assurance case where you gather and curate the right data and evidence much like a lawyer would do to prepare for a big court case.”
Moitra adds, “You must build a solid case that proves the changes being made will not compromise the system. To make this process faster, we will be using AI, semantic technology, natural language processing, automation and data analytics to gather and curate the right evidence to build a thorough assurance case in an expedited manner.”
“The best-case scenario outcome would be to complete software updates seamlessly within hours,” Siu explained. “You could show up at a test site in the morning, propose software changes based on observations from the previous day’s run, make the changes and ingest all the evidence of those updates into [the Rapid Assurance Curation Kit], and then have RACK generate a report that shows your changes were done safely, correctly, and securely and have approval for flight by lunchtime.”
“Without defining clear semantics, there can be a lot of misinterpretations, oversights, and rework. Researchers are using NLP to extract meaning out of documents traditionally written in plain text. Mapping that knowledge into a semantic model gives the ability to do analysis and queries. You can ask questions like, ‘Are there parts of the system not covered by satisfying test cases?’; ‘Are there parts of the systems where evidence is weak?’; ‘Have we met all the objectives in our development plan?’” Siu noted. “Being able to answer questions like these during the development process allows better systems to be built faster rather than focusing on ticking development workflow checkboxes.”
Distributed cryptography technology will also be deployed to enhance the security of digital assets in the associated critical systems.
“Guardtime Federal’s KSI digital integrity and provenance solutions provide users with confidence that their data has not been altered by an unauthorized party,” Siu said. “Using a data-agnostic and distributed cryptographic hash calendar, called the KSI Calendar, data is locked into this single trust anchor to provide mathematical proof of integrity, providing assurance that data have not been modified by an unauthorized party at any later point in time.”
GE Research is collaborating with GE Aviation Systems, a leading manufacturer of avionic systems; Galois, a leading developer of software to assess the trustworthiness of critical systems; and Guardtime Federal, which has developed a unique implementation of blockchain technology to secure the integrity of data used to manage critical systems.
Leading the team at Galois is Dr. David Archer, an expert in data-intensive systems and particularly in curating data and its provenance – the story of its origins, evolution, and resulting confidence in the data. “Unlike computer programs, which you can evaluate by formal methods or testing, data can’t speak about its trustworthiness. A key piece of our work in ARCOS will be to give users of assurance evidence data a quantifiable confidence based on where the data originated, who touched it, and when it was modified.”
To make the data used in critical systems more secure, Mr. Joseph Farrell from Guardtime Federal (GTF) is collaborating with Dr. Baoluo Meng from GE Research, to explore the application of Guardtime Federal’s KSI® Distributed Cryptography technology to provide secure provenance and data integrity for the artifacts being used for certification.
“As DARPA makes strides to increase the speed and automation of software certification processes, Guardtime Federal is excited to collaborate with GE Research to add digital integrity and provenance to the ARCOS evidence lifecycle,” said David Hamilton, president of Guardtime Federal. “Mathematically provable integrity is key to knowing that the data you are using comes from a trusted source, helping combat cyber threats from adversaries that are targeting our software and weapon system development pipelines.”