This is how you can secure your DevOps Cycle
As a modern cyber security professional for a corporate, you may get a lot of headache when working together with the people responsible for developing applications, the DevOps team (and vice versa). This article tries to explain why this is the case and how to structure good communication for a fruitful together in the company. Plus, it outlines two concrete strategies on how to continuously create more secure applications: security champions and tool integration.
Fig 1. Security needs to be involved from the start of an agile project.
Security Team's Requirements
What are the things that you as a security professional need when interacting with the DevOps team?
- When an application is developed in-house, the most important requirement as a security professional is to get involved in the process at all. Often, developers do not think and/or care about security. Therefore, security is only thought of shortly before the release. I have seen projects that were developed for years and then cancelled shortly before the production release, due to security considerations. In one case, a portal was designed to consolidate customer's contracts. It was stopped as it would have been a "single big honeypot" for the company. This cost the company millions of Euros for the development of the system. It could have easily been prevented if the security teams were involved early during the planning and development of the software.
- "Security is always a blocker." As long as this is the attitude that people responsible for a product have, you are in trouble. Only by instilling the attitude of "Security is an enabler", you will be involved earlier (or at all) and can work together on creating secure applications for your customers.
- You do not have a lot of time for comforting the DevOps department. You probably have enough to do even without taking the responsibility of the security for a new application. There are product owners who should take care of those kinds of things. The time that you invest in security should be spent as well as humanly possible. It does not make sense to explain basic security concepts to people creating and operating the application. You should be able to see the big picture of security in the company.
- If you invest time on checking the security of a single application, you want to concentrate on the important, difficult, and interesting part. Baseline security like a secure server configuration or up-to-date libraries should be the standard. The correct usage of frameworks to prevent vulnerabilities like SQL Injections and Cross-Site-Scripting should not be something you need to check. Instead, you can use OWASP ZAP or your favourite tool to check for flaws in the business logic of the application.
DevOps Team's Requirements
Try to take your DevOps Team's perspective now. What do they need when interacting with you?
- They probably hate to interact with you. Or at least they could think of thousands of different ways to use their time instead of talking to you. Their focus is to create meaningful applications to your customers. That is what they get paid for: Features
- They do not care about security. In most cases, there will not be User Stories that relate to security issues. Probably things like "Authentication" are implemented as it is a basic product requirement to have an internal area, but then that's it. The difference between authentication and authorisation is, where the problems usually start...
- If security is a topic, time is usually the greatest issue. Either the developer has many more important things to do and that prevents him from focusing on security. Or there is something which needs to be resolved by the developer ASAP. This is usually the time when security becomes relevant: Only if the business need is critical. Either because a vulnerability was detected and can lead to serious troubles if it stays unresolved or a customer is asking for a security assessment and suddenly the most unimportant topic of all time becomes a blocker.
Security and DevOps
Why is this an issue now? Security has been a problem for developers for ages...
In the waterfall development process, there was a planning phase before and a testing phase after the development. This is where security was involved. Setting up security requirements when creating the system specification and when it was checked before the operation started. Nowadays there are fast iterations and the planning is usually done in user stories. It is much harder to integrate security requirements when security needs to be involved every two weeks when new user stories are decided upon. In addition it is not feasible to do a manual penetration test every two weeks after a sprint. In contrast, manual tests were doable after a project period of multiple months or years.
Traditionally, security measures involved a lot of network and firewall rules. When networks were separated properly, a vulnerability in an application opened a much smaller attack surface than with interconnected networks (See also our blog on the interconnection of web applications then and now). E.g. network access to an air-gapped network for industrial controls is much harder to gain than jumping to the same control system over an infected website. With web applications for monitoring and steering everything, their vulnerabilities are much more important than ever for attackers to gain access to networks, steal data or sabotage companies.
DevSecOps – The Solution?
What does DevSecOps mean and is it the solution for all those problems?
DevSecOps is an attempt to bridge the gap between DevOps and Security. The same way that single people or teams take responsibility for development and operations, they are now also responsible for the security of their applications. This does not mean that you need to start developing applications now or that the DevOps engineers will do your work. It means that altogether, you take shared responsibility for the security of the application. You as a security specialist will support the development of the application (instead of restraining it).
To be your voice in the team and as a relief for you, there is the concept of security champions, which are "active members of a team that may help to make decisions about when to engage the Security Team". This helps you to engage non-security people in security aspects, establish a security culture, and scale through multiple (DevOps) teams. Often, product owners feel well in the role of a security champion, if security is an issue for them and their product. But also people with a development, operations or testing background are good fits for security champions. However, they need to have a standing in the team to be heard (and not overheard). Finally, these security champions should receive some training through courses or coaching to professionalise their work in the teams.
The security champions will be your "security voice of reason" during the regular sprint planning. When a user story is defined, they can add misuse-cases to it. For example, a user-story may define that a user of an application is able to download his invoices. The abuse-case may be to download different users' invoices. If this is forbidden within the acceptance criteria, there will be no doubt whether this security requirement has been implemented. It also serves as a guideline for testing the implementation of the feature.
Your job is it to provide the necessary guidelines and tools to the teams (and your security champions). This usually will be a mixture of the following:
Fig 2. Security Measures in the DevSecOps Cycle.
DevSecOps Teams and a secure DevOps Cycle
As shown in Figure 2, there are multiple ways in which security can be integrated in the DevOps cycle. Below is a DevSecOps tool overview in a table format:
|DevOps Phase||Possible Security Tools for this Stage|
Best Practices / Guidelines
Static Analysis (SAST)
|Test||Dynamic Analysis (DAST)
Interactive Analysis (IAST)
|Operate||Infrastructure as Code (IaC)
Logging & Alerting
Below, we dive deeper in some of the topics:
- Policies: What are the overall security policies for the company? The security champions and product owners can use threat models for their application in combination with your policies to create an overall security plan for the application. They will also be used for validation once features are released.
- Coding Guidelines: What are allowed technologies or libraries? How are frameworks used in a secure way? The DevOps teams then use this information to follow best practices of secure development. During peer reviews they check themselves whether the guidelines are followed.
- Vulnerability Scanner: Which scanners are used within the company to detect vulnerabilities early? Tools such as SonarQube for Static Analysis or Crashtest Security for Dynamic Analysis greatly help to identify any flaws or issues before the deployment – even in an agile environment. If the scanners are set-up and ready to use, it is easy for the teams to get started with them, immediately get feedback and have the benefit of continuous checks for unsecure software.
- Infrastructure-as-Code (IaC) Modules: How can a server be configured securely? How does a firewall need to be configured? Using tools like terraform, it is easy to create reproducible infrastructure. You can create modules e.g. for server setup in your preferred cloud. The module contains not only the server, but also network configuration, a firewall and access restrictions. This way the team can easily spawn their servers with a secure configuration. If you change the module due to new security requirements, all the existing infrastructure is automatically updated with the next release.
- Secret management: How can secrets be stored securely? It is always hard to store your secrets in a secure way. You do not want to end up revealing secrets when they are leaked in code repositories. If you provide a secret management service including guidelines and documentation on how to use it, you can control the secrets of the application, rotate them in case of an incident and monitor their usage. The teams have the control of which secrets they need and have an easy way to handle them.
- Threat Intelligence: Are there any new developments regarding the threat landscape? Important attacks on your infrastructure? This is one of the main tasks that you will keep in your hands. In case of incidents or a change in the attacks, you can take over and support the teams on topics where you are most needed.
- Logging & Alerting infrastructure: How do you keep track of the running application? Provide a logging infrastructure for all the teams. That way you can correlate events and detect incidents that would not have been visible otherwise. If you pre-configure alerts, the teams will be notified if there is an incident within their application. They will have the responsibility for it and only need to call you if the incident has security implications and they cannot handle it themselves.
- Vulnerability Disclosure Policy or Bug Bounty Program: What happens if an external researcher detects a vulnerability? You will be responsible to set-up and lead the vulnerability disclosure within the whole company. If you are just setting up such a program, you can pre-screen the issues and create comprehensible chunks of action. Then discuss the issues with the security champions and slowly transfer the responsibility for the issues to them. In the long run they will be capable to screen the reported issues for their team and only consult you in critical cases.
By using the tools mentioned above, you will end up with much more secure software. Every software is created by your guidelines that are manifested in the tool implementation. Additionally, you have a team of security champions that is spread throughout the company. If there are serious issues you will be involved. Security is thought about and integrated from the beginning of software projects till their operation and maintenance. Instead of being the DevOps people's enemy, you are their friend and they are happy to ask for your help whenever they need your support. One step closer on the way to continuous security.