International Defense Security & Technology Your trusted Source for News, Research and Analysis
As the speed and frequency of releases increase, traditional application security teams cannot keep up with the pace of releases to ensure each release is secure. To address this, organizations need to build in security continuously across the SDLC so that DevOps teams can deliver secure applications with speed and quality. DevSecOps, a relatively new term in the application security (AppSec) space, is about introducing security earlier in the software development life cycle (SDLC) by expanding the close collaboration between development and operations teams in the DevOps movement to include security teams as well. The earlier you can introduce security into the workflow, the sooner you can identify and remedy security weaknesses and vulnerabilities.
DevSecOps is the next evolution of agile and builds on the agile principles by adding the following:
- Leverages Containers and Microservices concepts
- Leverages Cloud deployment for scalability and prototyping
- Continuous Integration/Continuous Delivery to rapidly prototype, test and deploy
- Leverage A/B testing and canary deployment for rapid feedback loops
- Embed security in the pipeline instead of an afterthought
Implementing DevSecOps
To implement DevSecOps, organizations should consider a variety of application security testing (AST) tools to integrate into their CI/CD process. Some commonly used AST tools follow:
Static application security testing (SAST)
SAST tools scan proprietary code, or custom code, for coding errors and design flaws that could lead to exploitable weaknesses. SAST tools are used primarily during the code, build, and development phases of the SDLC. Coverity is one such SAST tool.
Software composition analysis (SCA)
SCA tools such as Black Duck scan source code and binaries to identify known vulnerabilities in open source and third-party components. They also provide insight into security and license risks to accelerate prioritization and remediation efforts. In addition, they can be integrated seamlessly into a CI/CD process to continuously detect new open source vulnerabilities, from build integration to pre-production release.
Interactive application security testing (IAST)
IAST tools, working in the background during manual or automated functional tests, analyze web application runtime behavior. For example, the Seeker IAST tool uses instrumentation to observe application request/response interactions, behavior, and dataflow. It detects runtime vulnerabilities and automatically replays and tests the findings, providing detailed insights to developers down to the line of code where they occur. This enables developers to focus their time and effort on critical vulnerabilities.
Dynamic application security testing (DAST)
DAST is an automated black box testing technology that mimics how a hacker would interact with your web application or API. It tests applications over a network connection and by examining the client-side rendering of the application, much like a pen tester would. DAST tools do not require access to your source code or customization to scan your stack. They interact with your website and find vulnerabilities with a low rate of false positives. For example, Synopsys Web Scanner and Synopsys API Scanner DAST tools identify vulnerabilities on web applications and APIs, including web-connected devices such as mobile back-end servers, IoT devices, and any RESTful or GraphQL APIs.
DevSecOps in the U.S. Department of Defense
One of the biggest challenges facing the Department of Defense today is how to quickly get mission-critical capabilities into the hands of personnel. Before DevSecOps came to the U.S. Department of Defense, software delivery could take anywhere from three to ten years for big weapons systems. “It was mostly teams using waterfall, no minimum viable product, no incremental delivery, and no feedback loop from end users,” says Nicolas M. Chaillan, Chief Software Officer of the U.S. Air Force. Plus, “cybersecurity was mostly an afterthought.”
DevOps has enabled organizations to harness the automation and speed of deployment that cloud-native technologies such as containers and Kubernetes provide. However, if security is not tightly integrated into DevOps, organizations’ ability to take full advantage of the cloud-native model is severely diminished. Indeed, today’s rapid and frequent development cycles demand that security be an integral and shared part of the entire app life cycle: DevSecOps. For example, it is best practice never to patch a running container but, rather, to always rebuild and redeploy. To do this well and at speed, organizations must use automated unit and functional tests, and they need to integrate automated security gates.
DOD Enterprise DevSecOps Initiative holds incredible promise. Its goal is to bring “automated software tools, services and standards to DOD programs so that warfighters can create, deploy, and operate software applications in a secure, flexible, and interoperable manner.”
Chaillan and Peter Ranks, Deputy Chief Information Officer for Information Enterprise, DoD CIO, created the DoD Enterprise DevSecOps reference design, with a mandate to use CNCF-compliant Kubernetes clusters and other open source technologies. “The DoD Enterprise DevSecOps reference design defines the gates on the DevSecOps pipeline,” says Chaillan. “As long as teams are compliant with that reference design, they can get a DoD-wide continuous ATO (authority to operate).”
As hybrid cloud strategies go, the U.S. military certainly is taking a unique approach. Just like almost everything else, military organizations increasingly depend on software, and they are turning to an array of open source cloud tools like Kubernetes and Istio to get the job done. Those tools have to be deployed in some very interesting places, from weapons systems to even fighter planes. Yes, F-16s are running Kubernetes on the legacy hardware built into those jets. “One point for the team was to demonstrate that it could be done,” Chaillan said. He challenged the Air Force and its partners to get Kubernetes up and running on a jet in 45 days, and while that was as difficult as it sounds, the team met the goal and F-16s are now running three concurrent Kubernetes clusters, he said.
The military can’t really afford to fall too far behind the mainstream when it comes to the technology it needs for its mission. As rivals invest in their own software capabilities, they could get an edge on the battlefield. And the security implications of a glacial process required to update some of the military’s most critical applications are obvious. “It’s very important for us to reduce the attack surface and be able to mitigate threats,” Chaillan said.
Department of Defense Enterprise DevSecOps
Chaillan and his team decided to embrace open source software as the foundation of the new development platform, which they called the DoD Enterprise DevSecOps Initiative. This initiative specified a combination of Kubernetes, Istio, knative and an internally developed specification for “hardening” containers with a strict set of security requirements as the default software development platform across the military.
Software teams in different branches or regions have some discretion over how they use the tools at their disposal, but everything must be constructed on the layer provided by the Air Force’s Platform One team, and there are several things teams are not allowed to change, Chaillan said.
Chaillan expanded on that during a press conference hosted by the Cloud Native Computing Foundation following his talk, explaining that “we don’t want to get locked into any one thing.” The team chose to use Kubernetes specifically for this reason, with other projects like Istio providing security for the networking layer of the DoD stack: “We want to make sure Istio is running constantly across the entire stack,” Chaillan said.
There were, of course, lots of challenges along the way. Kubernetes was not designed for the disconnected environments that the military must use in many situations involving data that should not ever reach the internet, Chaillan said. He hinted that DoD will have a lot of suggestions for Kubernetes maintainers as they address this portion of the project, which could help pave the way for Kubernetes to be used in other sensitive operating environments.
“We’re very used to updating using the internet, and having connectivity to the internet, getting the updates directly from the internet. For us, we have to bring the entire stack with us” on jets or weapons systems that are disconnected from the internet by design, he said.
An Open Source Stack at DoD Scale
The scale at which the DoD operates is unlike almost all commercial operations; Chaillan had to train 100,000 people on the principles of DevSecOps, not to mention the new tools. “The culture shift has been interesting,” he said during the press conference, seemingly with some restraint. That scale is a reflection that the DoD will be using this new development platform for lots of mundane and unclassified applications: More than 2 million people work for the military, and most of them are not flying F-16s running Kubernetes.
“The jet is interesting, but it’s a tiny piece of the rest of the work we’re doing. We have a lot of business systems moving to cloud native environments, moving to microservices, being built right from the get-go,” Chaillan said. The entire DoD Enterprise DevSecOps Initiative stack is open source and available for anyone to check out, Chaillan said. Military agencies are “getting better” about releasing more of their work to the open source community, he said, also noting that “we’re not going to fork (open source) software.”
Automated Testing
Automated testing is a key part of DevSecOps. It is enabled by multiple tools that measure both test code coverage and test results. They are fully automated and do not require human action. It also enables new concepts like pair programming and peer code review.
Agile brings several new models for creating the right tests:
- Test-driven development (TDD) is a software development process that relies on very short development cycles: requirements are turned into very specific test cases first, then the software is built to pass the tests.
- Acceptance test–driven development (ATDD) is a development methodology based on communication between the business customers, the developers, and the testers. ATDD encompasses many of the same practices as specification by example, behavior-driven development (BDD), example-driven development (EDD), and support-driven development also called story test–driven development (SDD). All these processes aid developers and testers in understanding the customer’s needs prior to implementation and allow customers to be able to converse in their own domain language.
DevSecOps Archtecture
Checkmarx Delivers Containerized AppSec Solution to DoD’s Platform
Checkmarx, the global leader in software security solutions for DevOps, announced in Dec 2020 that it has been accepted into the U.S. Department of Defense’s (DoD) “Iron Bank” repository and is now available through the U.S. Air Force Platform One application portal. With this, Checkmarx furthers its commitment to supporting the public sector by making its automated application security testing (AST) solution available to all DoD agencies in the form of a hardened container, helping them to confidently build and release secure software while meeting the strict security and compliance requirements of the U.S. military.
A project of the U.S. Air Force designed to deliver the benefits of DevSecOps across the entire DoD, Platform One provides “Iron Bank,” a centralized artifacts repository, with a pre-approved collection of solutions that have undergone extensive auditing and approval steps to streamline Authority to Operate (ATO) processes. Checkmarx’s hardened container instance was developed in coordination with the DoD and has achieved a Certificate to Field (CtF) from the USAF Platform One team. This enables all DoD agencies and developers to easily acquire and integrate the Checkmarx solution into their DevOps environments and automatically insert security into the entire SDLC, while also avoiding lengthy ATO timelines.
Notably, this expands Checkmarx’s long-standing partnership with the DoD, already supporting the U.S. Navy’s Naval Information Warfare Center Pacific (NIWC PAC) division and the U.S. Air Force Business & Enterprise Systems Directorate (USAF BES), among other agencies, in their DevSecOps initiatives. “As software becomes more complex, bringing with it a vast attack surface, the DoD has made it a priority to arm development teams with best-in-class solutions to build and deploy applications in a more secure manner,” said Nicolas Chaillan, Chief Software Officer and Co-Lead for the DoD Enterprise DevSecOps Initiative, U.S. Air Force. “Checkmarx has been a valued partner to the USAF and DoD for years and this latest step in bringing a hardened version of their solution to Iron Bank and Platform One will be invaluable as we execute on our mission to shift to a DevSecOps model across our entire branch.”
Checkmarx offers automated solutions that simplify and speed up the process of security testing throughout software development. The company’s solutions integrate seamlessly with developer workflows and tools to quickly find and remediate vulnerabilities in both custom and open source code before software is released. Public sector agencies that leverage Checkmarx are able to integrate enterprise-grade security testing into their DevOps environments, while meeting compliance requirements for FISMA, NIST, and STIG, among others, and decreasing time to ATO.
“The Iron Bank and Platform One synergy is a truly logical way of bringing the benefits of faster time-to-market and lower development costs to a complex enterprise like the DoD,” said Peter Archibald, Federal Systems Manager, Checkmarx. “The genius and simplicity of this approach lies within the hardened containers. This is a significant evolution in how the DoD is innovating secure development.
Lockheed Martin conducted a flight test of U-2 Dragon Lady spy plane in Dec 2020 equipped with distributed processing software on board to establish a link with the ground station.
Lockheed Martin has conducted a flight test of U-2 Dragon Lady spy plane equipped with distributed processing software on board to establish a link with the ground station. Conducted by Lockheed Martin Skunk Works, the test used Kubernetes containerisation technology to demonstrate the capability. The move will help in advancing with the plan of creating a DevSecOps environment to deliver enhanced software capability to airborne assets in real-time.
Lockheed Martin Skunk Works vice-president and general manager Jeff Babione said: “The U-2 Kubernetes demonstration from mid-November not only advances the deployment pipeline for in-flight software upgrades but also operationally extends the computational resources for mission execution. “This additional capability makes it possible for the warfighter to quickly adapt to changing threat environments without costly or time-consuming system upgrades.”
The team utilised a Kubernetes Cloud configuration in the test. The technology was flown on the U-2 through an Enterprise Open System Architecture Mission Computer (EMC2), which represented the Open Mission Systems (OMS) mission computer currently being developed for the U-2 programme. During the flight, Kubernetes connected in-flight to a ground node extending the reconnaissance aircraft’s network-of-networks connectivity.
References and Resources also include:
https://thenewstack.io/how-the-u-s-air-force-deployed-kubernetes-and-istio-on-an-f-16-in-45-days/
