Nirmata https://nirmata.com Kubernetes Security & Governance Tue, 19 Mar 2024 04:54:11 +0000 en-US hourly 1 https://nirmata.com/wp-content/uploads/2022/05/cropped-monochrone-512x512-1-32x32.png Nirmata https://nirmata.com 32 32 Unlocking Innovation: Nirmata at KubeCon EU in Paris https://nirmata.com/2024/03/11/nirmata-at-kubecon-paris-2024/ Mon, 11 Mar 2024 13:19:35 +0000 https://nirmata.com/?p=9495 We are very excited to announce that team Nirmata is a Silver sponsor of  KubeCon + CloudNativeCon Europe 2024 from March 19-22. Our team is all geared up for this event that promises innovation and collaboration for the future of cloud native technologies.  We are...

The post Unlocking Innovation: Nirmata at KubeCon EU in Paris first appeared on Nirmata.]]>
KCEU24 Nirmata LinkedIn 628x200

We are very excited to announce that team Nirmata is a Silver sponsor of  KubeCon + CloudNativeCon Europe 2024 from March 19-22. Our team is all geared up for this event that promises innovation and collaboration for the future of cloud native technologies.  We are eagerly looking forward to the opportunities to engage in insightful conversations with the cloud-native community and showcase what we are building at Nirmata to address challenges related to cloud-native security and governance. Join us for discussions on emerging trends in cloud-native technologies and how it can power the new world of generative AI.

Meet the Nirmata team in Paris

Here are the folks from Nirmata, who will be at KubeCon EU:

Anusha Hegde: Technical Product Manager at Nirmata. Anusha is enthusiastic about building distributed systems and everything cloud native!

Anushka Mittal: Anushka Mittal, Software Developer at Nirmata. She was an LFX Summer Mentee for CNCF-Kubernetes and contributed a policy report adapter to Falco, the cloud-native runtime security project. She then worked with Nirmata on their open-source project, Kyverno. 

Charles-Edouard Brétéché: Senior Software Engineer at Nirmata. Charles works on Kyverno to build the next level Kubernetes native policy engine. 

Damien Toledo: Co-founder and VP of Engineering at Nirmata. Damien builds global teams and delivers world class products.

Faeka Ansari: Policy writer intern at Nirmata. Faeka is a final year undergraduate pursuing CS, previously worked as an Linux foundation mentee at CNCF Istio, Kubernetes release team member for v1.29 and LWKD (https://lwkd.info/ a weekly newsletter summarizing code activity in the Kubernetes project) repo maintainer.

Jim Bugwadia: Co-founder and CEO of Nirmata. Jim does what it takes, and contributes to the cloud native community as a co-chair of the CNCF Kubernetes Policy Working Group and as a maintainer of Kyverno.

Mariam Fahmy: Software Engineer at Nirmata. Mariam has good knowledge and hands-on experience in docker, Kubernetes, AWS, terraform, ansible, and many DevOps tools.

Randi Belz: Strategic Partner Development Manager at Nirmata. Randi works across the partner and community ecosystem to drive mutual growth and success. Randi is also driving partner marketing and events. 

Ritesh Patel: Co-founder and VP Products at Nirmata. Ritesh loves working with customers and partners to build and deliver category leading products.

Shuting Zhao: Staff Engineer at Nirmata. Shuting works on Kyverno and several other cloud-native projects that help automate the deployment and operation of workloads and Kubernetes clusters.

Attend our sessions and activities

Bookmark these sessions from our team members and our community. For location, check the CNCF schedule. 

Tuesday, March 19 | 2.10pm-2.17pm (Lightning Talk)

Kyverno Top 10: Automate Kubernetes Security with Policy as Code

An exciting lightning talk on #Kyverno, the easiest solution for automating Kubernetes security using native policy-as-code. In this talk, learn about Kyverno’s top 10 features that set it apart as a cloud native policy management tool.

Wednesday, March 20 | 2.30pm-3.05pm

Enabling the Software Supply Chain Ecosystem with Notary Project   

Join this session to learn how the Notary Project is designed to be extensible, enabling other CNCF projects and ecosystem tools to implement solutions for authenticity and integrity across the supply chain. It also enables enterprises to leverage their existing investments while becoming more secure. 

Thursday, March 21 | 11am-11.35am

Faster, Safer, Serverless – Empowering Apache Spark Standalone Cluster on Kubernetes

Explore how to deliver a truly K8s-native Serverless Spark Service on Kubernetes, emphasizing speed, simplicity, with a new K8s operator for standalone cluster creation and job submission. Instead of relying solely on Spark, it also harnesses the elastic and police management power of Kubernetes with K8S metrics server, HPA and Kyverno, simplifying the workflow for Apache Spark itself, infra engineers, and users.

Thursday, March 21 | 2.30pm-3:05pm

Kubernetes Policy Time Machine: Where to Next? 

Join members of the Kubernetes Policy Working Group (WG) for a deep dive into each available flavor of policy, from built-in API resources and admission controls, to the new ValidatingAdmissionPolicies which are customizable using the Common Expressions Language (CEL), and powerful policy as code OSS projects that leverage dynamic admission controls. 

Thursday, March 21 | 4.30pm – 5:05pm 

Securing 900 Kubernetes Clusters Without PSP – Mercedes-Benz’ Journey to ValidatingAdmissionPolicies 

Delve into the journey of Mercedes-Benz of replacing PSPs and explore the reasons why PSS were not viable for their K8s fleet, their learnings on implementing Pod Security using popular tools like Kyverno, and why they landed on VAPs. 

Thursday, March 21 | 5.25pm –  6pm 

Running PCI-DSS Certified Kubernetes Workloads in the Public Cloud 

This talk will start by covering high-level PCI-DSS requirements and discuss how these were addressed using several CNCF and open source projects: * GitOps with ArgoCD * Network policies and visibility with Cilium * In-cluster image scanning and visualization with Trivy and Grafana * Open Policy Agent Gatekeeper or Kyverno for Policy as Code * Cilium Tetragon for runtime security visibility.

Friday March 22 |  11am – 12.30pm 

Contribfest: Kyverno – Let’s Build Together!

This hands-on session equips participants to contribute to Kyverno, the “Swiss army knife” of Kubernetes that elegantly solves critical challenges across security, automation, and compliance. Learn about Kyverno’s architecture, the role of each component, how to set up a development environment, and how to contribute to the project. This hands-on session will be led by Kyverno maintainers and organizers so that both developers as well as non-developers can contribute across the software base, sample policies, and documentation. Join us to shape the future of Kyverno together!

Friday March 22 |  2pm – 2.35pm

How I Met Your Software – an Image’s Sitcom of Consuming and Securing Software in Cloud Native!

The presenters will dissect the techniques of choosing the consumer software by comparing various distributions like Alpine, Wolfi and how a few of them are solving for maintaining the pace with upstream projects. 

Friday March 22  |  4pm – 4.35pm

From Chaos to Control: Cloud Native Governance with Kyverno!

Kyverno started life as a Kubernetes policy engine and has quickly evolved into a set of widely adopted tools for cloud native policy and governance. In the session, Raul, a Kyverno end user and platform lead at DE-CIX, will introduce the project for new users and discuss his organization’s journey with policy as code. Next, Kyverno maintainers Mariam Fahmy and Charles-Edouard will provide key updates and discuss advanced features including new sub-projects like Kyverno JSON and Kyverno Chainsaw, and share the project roadmap. To learn how to automate security concerns, or offer guardrails as part of Kubernetes platform, please join this session!  

Booths

If you are attending KubeCon Paris, feel free to drop by and say hello at the Nirmata booth F15 and Kyverno kiosk PP8-A in the CNCF project area. We are always happy to connect with you all as a part of the Kubernetes community, and we look forward to meeting you at the event. 

What’s Next

Please get in touch with us either on LinkedIn or Twitter.

And, if you need to understand where you or your organization is in the Kubernetes policy and governance journey, feel free to reach out to us for a discussion. Our team will be able to suggest the next steps forward. To get your hands dirty on our product, you can sign up for a free trial here.

If you want to see a customized demo or have any questions, feel free to reach out to us!

You can also Schedule a Meeting!

You can register for the event here.

For more information about Nirmata and our cloud-native solutions, visit us at Nirmata.com.

See you at KubeCon! 

The post Unlocking Innovation: Nirmata at KubeCon EU in Paris first appeared on Nirmata.]]>
Preventive Security vs Detection and Response https://nirmata.com/2024/03/04/preventive-security-vs-detection-and-response/ Mon, 04 Mar 2024 21:26:36 +0000 https://nirmata.com/?p=9470 Preventive security measures and detection and response strategies, particularly in the context of preventing misconfigurations versus runtime security, represent two fundamental approaches in the cybersecurity domain. Each plays a crucial role in an organization’s overall security posture. Below, we compare and contrast these approaches focusing...

The post Preventive Security vs Detection and Response first appeared on Nirmata.]]>
White Yellow Blue Modern Cyber Network Security Instagram Post

Preventive security measures and detection and response strategies, particularly in the context of preventing misconfigurations versus runtime security, represent two fundamental approaches in the cybersecurity domain. Each plays a crucial role in an organization’s overall security posture. Below, we compare and contrast these approaches focusing on their application, benefits, limitations, and key differences.

 

Preventive Security: Preventing Misconfigurations

This approach focuses on avoiding security incidents by ensuring correct configurations in code, servers, networks, Kubernetes, cloud environments, and other infrastructure components from the outset. It involves secure coding practices, Infrastructure as Code (IaC) for consistent deployments, and the use of static analysis tools and configuration management systems.

Benefits

Proactive Risk Reduction: By preventing misconfigurations, organizations can proactively reduce their attack surface and minimize vulnerabilities that could be exploited.

Cost Efficiency: It is generally more cost-effective to prevent security issues than to address them after they have been exploited.

Compliance Assurance: Helps in maintaining compliance with security policies and regulations by ensuring that configurations meet required standards from the start.

Limitations

Complexity and Overhead: Requires upfront investment in tools, training, and processes, which can be complex to implement and manage.

Dynamic Environments: In fast-changing cloud-native environments, maintaining zero misconfigurations can be challenging.

False Sense of Security: Sole reliance on preventive measures might lead to complacency, overlooking the need for detection and response capabilities.

 

Detection and Response: Runtime Security

This strategy focuses on identifying and responding to security threats in real-time or near real-time. It involves monitoring networks, systems, and applications for unusual or unauthorized behavior, and implementing automated responses to mitigate detected threats. Tools and practices include Intrusion Detection Systems (IDS), Security Information and Event Management (SIEM) systems, and automated incident response solutions.

Benefits

Adaptability to Emerging Threats: Offers the ability to detect and respond to new and evolving threats that were not prevented by initial security measures.

Insight and Intelligence: Provides valuable insights into the nature of attacks and attacker behavior, which can inform and improve preventive measures.

Continuous Protection: Ensures ongoing vigilance and protection throughout the lifecycle of systems and applications, adapting to changes and updates.

Limitations

Reactive Nature: While it can minimize the impact, this approach often deals with threats after they have occurred, which can still result in damage or data loss.

Resource Intensity: Effective detection and response require significant resources, including advanced tools and skilled personnel to monitor, analyze, and react to incidents.

Alert Fatigue: High volumes of alerts, not all of which are critical, can overwhelm security teams, potentially causing delays in responding to actual threats.

 

Key Differences

Timing and Focus: Preventive security aims to eliminate risks before they manifest, focusing on configuration and deployment stages. In contrast, detection and response deal with threats during runtime, focusing on identifying and mitigating attacks in progress or after they have occurred.

Approach: Prevention is about setting and maintaining secure standards and configurations to avoid vulnerabilities. Detection and response revolve around monitoring for deviations from normal operations and acting on them.

Scope of Protection: Preventive measures are often static, designed around known best practices and vulnerabilities. Detection and response are dynamic, capable of adapting to new and unforeseen attack vectors.

 

Admission Control with Policy as code for Preventive Security

Admission controllers with policy as code offer a powerful method to overcome some of the inherent limitations of preventive security through security automation. By integrating scanning into the deployment pipeline, organizations can enforce security policies automatically before applications are deployed or updated in environments like Kubernetes. In Kubernetes and cloud environments, admission controllers act as gatekeepers, evaluating requests to create or update resources against predefined policies coded into the system. This approach not only streamlines the enforcement of security standards but also ensures that any changes or deployments are automatically checked for compliance with security policies, thereby reducing human error, the risk of misconfigurations, the need for costly remediation efforts and downtime associated with security incidents. 

The ability to codify and automatically apply security policies across the board means that security scales with your infrastructure without a proportional increase in security staffing costs. Overall, the upfront investment in setting up admission controllers and implementing policy-as-code translates into long-term savings by preempting security issues, enhancing compliance, and streamlining security operations.

Policy as code allows for the codification of complex organizational security policies into actionable, automated checks that are consistently applied, making security scalable and more manageable. This method bridges the gap between the static nature of traditional preventive measures and the dynamic requirements of modern IT environments, providing a more adaptive and resilient security posture that can automatically adjust to new security policies and emerging threats without significant manual intervention.

 

Conclusion

Both preventive security through preventing misconfigurations and detection and response with runtime security are essential components of a robust cybersecurity strategy. The optimal approach involves a balanced investment in both, leveraging the proactive benefits of prevention to minimize misconfigurations and the reactive capabilities of detection and response to address threats that evade initial preventive measures. This integrated strategy ensures a comprehensive defense posture, capable of both preventing incidents and effectively dealing with those that occur.

 

The post Preventive Security vs Detection and Response first appeared on Nirmata.]]>
Proactive vs Reactive Security: A Paradigm Shift in Application Security https://nirmata.com/2024/02/09/proactive-vs-reactive-security-a-paradigm-shift-in-application-security/ Fri, 09 Feb 2024 22:03:24 +0000 https://nirmata.com/?p=9459 The shift from reactive to proactive measures marks a significant paradigm change for application security. This transformation is pivotal in the way organizations approach the security of their applications in an increasingly fast-paced and interconnected world. Let’s dive deeper into these concepts to understand their...

The post Proactive vs Reactive Security: A Paradigm Shift in Application Security first appeared on Nirmata.]]>
Untitled design (1)

The shift from reactive to proactive measures marks a significant paradigm change for application security. This transformation is pivotal in the way organizations approach the security of their applications in an increasingly fast-paced and interconnected world. Let’s dive deeper into these concepts to understand their impact on modern application security. 

Understanding Reactive Security

Reactive security refers to strategies and measures that organizations implement in response to security incidents that have already occurred. This approach typically involves:

  1. Incident Response: Reacting to security breaches or vulnerabilities after they have been exploited.
  2. Patch Management: Updating systems post-identification of vulnerabilities.
  3. Forensic Analysis: Investigating breaches to understand their impact and prevent future occurrences.
  4. Damage Control: Mitigating the impact of security incidents on business operations and customer trust.

While reactive security is essential for damage control and learning from past incidents, it often leads to higher costs and potential damage to reputation, as the security breach has already occurred.

The Rise of Proactive Security

Proactive security, on the other hand, is about anticipating, preventing, and mitigating potential security threats before they manifest into actual breaches. This approach includes:

  1. Continuous Risk Assessment: Continuously evaluating the IT environment to identify and mitigate potential vulnerabilities.
  2. Security by Design: Integrating security considerations into the application development process, rather than treating them as an afterthought.
  3. Automated Security Testing: Incorporating automated tools in the CI/CD pipeline to continuously check for misconfigurations and vulnerabilities.
  4. Threat Intelligence: Staying informed about new types of cyber threats and adapting strategies accordingly.
  5. Admission Control: Blocking and preventing insecure configurations instead of detecting them.
  6. User Education and Awareness: Training employees about security best practices to prevent security lapses due to human error.

 

Policy-as-Code: A Catalyst for Proactive Security

Policy-as-code (PaC) is a crucial tool in the transition towards proactive security. PaC involves defining and managing security policies in a code format, which can be version-controlled, automatically applied, and audited systematically. This approach allows organizations to:

  • Enforce Security Standards Automatically: Security policies can be automatically applied to every part of the software development lifecycle, ensuring that vulnerabilities are addressed early.
  • Prevent vs Detect: Policies can be proactively applied to prevent security issues instead of just detecting and reporting issues after the fact.
  • Scale Security Practices: PaC allows security practices to scale alongside the infrastructure and applications they protect, without requiring manual intervention.
  • Maintain Compliance: Automated compliance checks ensure continuous adherence to internal and external regulatory standards.
  • Reduce Human Errors: By automating policy enforcement, the likelihood of security breaches due to manual errors is significantly lowered.
  • Enhance Audit and Compliance Reporting: With PaC, creating audit trails and compliance reports becomes more straightforward, providing clear documentation for internal audits and regulatory compliance.
  • Improve Collaboration and Communication: PaC facilitates clearer communication and collaboration between security teams and developers, as security policies are defined in code, a language familiar to developers.
  • Be Agile and Rapidly Adapt: As security threats evolve, PaC enables organizations to rapidly adapt their security policies, ensuring they are always ahead of potential threats.
  • Ensure Consistent Application of Security Policies: Regardless of the scale or complexity of the environment, PaC ensures that security policies are applied uniformly, eliminating inconsistencies that can lead to vulnerabilities.

 

The Impact of the Shift

This paradigm shift from reactive to proactive security offers numerous benefits:

  • Reduced Risk of Breaches: By identifying and mitigating vulnerabilities early, the likelihood of breaches is significantly decreased.
  • Cost Efficiency: Addressing security issues in the early stages of development is generally less expensive than responding to a breach.
  • Enhanced Trust and Reputation: A proactive stance on security builds customer trust and enhances the organization’s reputation.
  • Regulatory Compliance: Proactive strategies often align better with regulatory requirements, reducing the risk of non-compliance penalties.

 

Conclusion

The shift from reactive to proactive security is not just a change in tactics but a fundamental change in mindset. It requires organizations to integrate security into every aspect of their operations and constantly evolve their strategies to stay ahead of threats. Policy-as-code is a key enabler in this transformation, offering a structured, scalable, and effective approach to embedding security into the fabric of modern applications. As the digital landscape grows in complexity, the importance of this proactive approach cannot be overstated. It is an essential element for any organization looking to safeguard its digital assets.

The post Proactive vs Reactive Security: A Paradigm Shift in Application Security first appeared on Nirmata.]]>
Securing OpenTofu with Nirmata Powered by Kyverno https://nirmata.com/2024/02/05/securing-opentofu-with-nirmata-powered-by-kyverno/ Mon, 05 Feb 2024 20:16:08 +0000 https://nirmata.com/?p=9451 As Infrastructure as Code (IaC) continues to gain popularity among DevOps practitioners for its efficiency and scalability, the recent Terraform license ambiguity has prompted the emergence of alternative solutions. OpenTofu, marking its first stable release, enters the scene as a robust IaC tool, providing users...

The post Securing OpenTofu with Nirmata Powered by Kyverno first appeared on Nirmata.]]>
2024 1 30 nirmata opentofu linkedin v2

As Infrastructure as Code (IaC) continues to gain popularity among DevOps practitioners for its efficiency and scalability, the recent Terraform license ambiguity has prompted the emergence of alternative solutions. OpenTofu, marking its first stable release, enters the scene as a robust IaC tool, providing users with a reliable option for managing and provisioning cloud infrastructure. Whether you are just beginning your journey with OpenTofu or contemplating a migration from other tools, it’s essential to enhance your infrastructure’s security and compliance posture. A key recommendation is to seamlessly integrate Policy-as-Code (PaC) into your OpenTofu workflows if they still need to be put in place.

Why scan your OpenTofu workflows?

As with any IaC, scanning OpenTofu files is crucial for ensuring the security, reliability, and compliance of cloud infrastructure deployments. By regularly scanning these files, organizations can proactively identify and rectify security vulnerabilities, misconfigurations, and compliance violations before they manifest in deployed environments.

How does Kyverno help?

Kyverno is the most popular cloud-native policy engine. If you’ve been keeping up with cloud-native security trends, Kyverno’s name has likely crossed your path, and how it simplifies policy enforcement for Kubernetes clusters with its low-code YAML syntax. Now, with the latest announcement of Kyverno for any JSON, you can leverage the benefits of the simplicity of Kyverno to write policies for scanning any OpenTofu file. 

This blog post demonstrates the power of Kyverno to scan OpenTofu files using NCTL (a command line tool developed by Nirmata). The CLI can be utilized in CI pipelines for the proactive detection of misconfigurations, enabling timely mitigation of issues and preempting potential security risks before they escalate.

NCTL and OpenTofu

Prerequisites

Before we get started, you need to install the following tools –

OpenTofu file to create an ECS task

The following is a Terraform file snippet for creating an ECS task. The image used here is `nginx:latest`.

resource "aws_ecs_task_definition" "service" {
  family = "service"
  container_definitions = jsonencode([
    {
      name      = "foo-service"
      image     = "nginx:latest"
      cpu       = 10
      memory    = 512
      essential = true
      portMappings = [
        {
          containerPort = 80
          hostPort      = 80
        }
      ]
    }
  ])
}

Kyverno Policy

The following Kyverno policy validates the `image` field specified in the `container_definitions` section and validates that a tag has to be present, and it should not be a mutable tag, i.e., `latest`

apiVersion: json.kyverno.io/v1alpha1
kind: ValidatingPolicy
metadata:
  name: check-ecs-image-tag
  annotations:
    policies.kyverno.io/title: Check ECS Image Tag 
    policies.kyverno.io/category: OpenTofu ECS Best Practices
    policies.kyverno.io/severity: High
    policies.kyverno.io/description: >-
      The ':latest' tag is mutable and can lead to unexpected errors if the
      image changes. A best practice is to use an immutable tag that maps to
      a specific version of an application. This policy validates that the image
      specifies a tag and that it is not called `latest`.
spec:
  rules:
    - name: check-ecs-image-tag-latest
      match:
        all:
        - type: (planned_values.root_module.resources[?type=='aws_ecs_task_definition'])
      assert:
        all:
        - message: An image tag or digest should be specified
          check:
             ~.(planned_values.root_module.resources[?type=='aws_ecs_task_definition'].values):
                  ~.(json_parse(container_definitions)):
                    image:
                      (contains(@, ':') || contains(@, '@sha256:')): true
        - message: Image tag `latest` should not be used
          check:
            ~.(planned_values.root_module.resources[?type=='aws_ecs_task_definition'].values):
                  ~.(json_parse(container_definitions)):
                    image:
                      (contains(@, ':latest')): false

Scan OpenTofu using NCTL

First, we will convert the Terraform file to a JSON format. Perform the following steps –

tofu init
tofu plan -out tofuplan.binary
tofu show -json tofuplan.binary | jq > tofuplan.json

Use NCTL to scan the `tofuplan.json` with the policy we defined above.

nctl scan terraform -r tofuplan.json -p check-ecs-image-tag.yaml

If the `image` does not contain any tag or a digest or contains the `latest` tag, then it will be reported as a failure. If any other tag is present, then the policy evaluates to pass.

Conclusion

With this simple yet powerful example, we saw how Kyverno is leveraged to write low-code policies to secure your OpenTofu files. NCTL can be used to build workflows that can integrate with your existing CI pipelines and detect misconfigurations as early as possible. Nirmata provides out-of-the-box policy sets that you can utilize to enforce security best practices in your organization. Watch out this space for more updates on policy and governance of cloud infrastructure!

To know more about Nirmata and our product offerings, visit our website or contact us! We would be happy to assist you in your Policy and Governance journey.

References

The post Securing OpenTofu with Nirmata Powered by Kyverno first appeared on Nirmata.]]>
Locked Doors, Untrusted Keys: Securing Containers in the Wake of Leaky Vessel Vulnerabilities https://nirmata.com/2024/02/02/locked-doors-untrusted-keys-securing-containers-in-the-wake-of-leaky-vessel-vulnerabilities/ Fri, 02 Feb 2024 18:48:51 +0000 https://nirmata.com/?p=9431 The recent buzz surrounding the actively exploited runc vulnerabilities “Leaky Vessels”  (CVE-2024-21626) serves as a timely reminder of two essential container security principles: image trust and comprehensive patching. Let’s delve into why these aspects are crucial for your containerized environments. Trustworthy Images are Foundational: It...

The post Locked Doors, Untrusted Keys: Securing Containers in the Wake of Leaky Vessel Vulnerabilities first appeared on Nirmata.]]>
2024 2 2 nirmata runc blog v3

The recent buzz surrounding the actively exploited runc vulnerabilities “Leaky Vessels”  (CVE-2024-21626) serves as a timely reminder of two essential container security principles: image trust and comprehensive patching. Let’s delve into why these aspects are crucial for your containerized environments.

  1. Trustworthy Images are Foundational: It might seem self-evident, but avoiding untrusted images is paramount. My experience conducting Kubernetes security reviews highlights this as the most frequent pitfall. In the context of the runc vulnerability, pulling an image from an unreliable source could grant attackers a direct entry point into your Kubernetes cluster. Remember, the attack surface expands with every untrusted image you incorporate.
  2. Patching Permeates the Entire Stack: While patching container orchestration platforms like Kubernetes is common practice, don’t neglect low-level components like runc. Often overlooked due to their seemingly insignificant nature, these pieces play a critical role. In this case, runc establishes and executes the container sandbox, making its security integral. Leaving it unpatched creates a gaping hole in your defenses, nullifying the effectiveness of security measures applied elsewhere.

By adhering to these fundamental principles, you can significantly strengthen your container security posture. Scrutinize image sources meticulously and prioritize patching across your entire container ecosystem, from high-level platforms to foundational components like runc. Remember, a chain is only as strong as its weakest link. Don’t let a seemingly minor vulnerability become your downfall. 

Suppose you are already a Nirmata customer running Nirmata Policy Manager or are running Kyverno in production. In that case, you can use the following policy to give you a better view of how impactful this new vulnerability is going to be. 

The policy will check your container runtimes for the vulnerable versions and produce an actionable report:


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-container-run-time
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: container-run-time-version
    match:
      any:
      - resources:
          kinds:
          - Node
    context:
    - name: cr_version
      variable:
        jmesPath: split(request.object.status.nodeInfo.containerRuntimeVersion, '://')[1]
    - name: cr_runtime
      variable:
        jmesPath: split(request.object.status.nodeInfo.containerRuntimeVersion, '://')[0]
    validate:
      message: "Your container runtime is vulnerable to CVE-2024-21626 & CVE-2024-23651: {{cr_runtime}}:{{cr_version}}"
      deny:
        conditions:
          any:
          - key: |-
              {{ cr_runtime == 'containerd' && semver_compare(cr_version, '<= 1.6.27') || false }}
            operator: Equals
            value: true
          - key: |-
              {{ cr_runtime == 'runc' && semver_compare(cr_version, '<= 1.1.11') || false }}
            operator: Equals
            value: true

Furthermore, you can detect containers trying to exploit the vulnerability by mounting /proc/self/fd/* note that it is recommended you run the policy in Audit prior to Enforce mode as potential false positives are expected.


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-proc-self-fd-workingdir
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: no-proc-self-fd-images
      match:
        any:
        - resources:
            kinds:
              - Pod
      validate:
        message: "Mounting /proc/self/fd is not allowed CVE-2024-21626."
        pattern:
          spec:
            containers:
              - =(workingDir): "!/proc/self/fd*"
            initContainers:
              - =(workingDir): "!/proc/self/fd*"
            ephemeralContainers:
              - =(workingDir): "!/proc/self/fd*"

You can use the following in order to block persons from blatantly trying to exploit Leaky Vessels. It will not be able to detect a container that has hidden its attempts to abuse the vulnerability deeper in its code, the scope of detection of such an attack is beyond Kyverno.


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-proc-self
spec:
  validationFailureAction: Enforce
  rules:
  - name: container-exec-block-proc-self
    match:
      any:
      - resources:
          kinds:
          - Pod/exec
          - Pod
    validate:
      message: "Pod {{ request.object.metadata.namespace }}/{{ request.object.metadata.name }} cannot use /proc/self/cwd, CVE-2024-21626"
      deny:
        conditions:
          any:
          - key: "{{ request.object.spec.containers[].args[] | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true
          - key: "{{ request.object.spec.initContainers[].args[] | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true
          - key: "{{ request.object.spec.ephemeralContainers[].args[] | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true
          - key: "{{ request.object.command | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-proc-self-mounting
spec:
  validationFailureAction: Enforce
  rules:
  - name: no-proc-self-images
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Pod {{ request.object.metadata.namespace }}/{{ request.object.metadata.name }} with images having /proc/self/fd/ in their layers are not allowed CVE-2024-21626."
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageData
          imageRegistry:
            reference: "{{ element.image }}"
        deny:
          conditions:
            any:
              - key: "{{ imageData.configData.history[].created_by | contains(@, '/proc/self/fd/') }}"
                operator: Equals
                value: true
      - list: "request.object.spec.initContainers"
        context: 
        - name: imageData
          imageRegistry:
            reference: "{{ element.image }}"
        deny:
          conditions:
            any:
              - key: "{{ imageData.configData.history[].created_by | contains(@, '/proc/self/fd/') }}"
                operator: Equals
                value: true
      - list: "request.object.spec.ephemeralContainers"
        context: 
        - name: imageData
          imageRegistry:
            reference: "{{ element.image }}"
        deny:
          conditions:
            any:
              - key: "{{ imageData.configData.history[].created_by | contains(@, '/proc/self/fd/') }}"
                operator: Equals
                value: true

Beyond Trust, Verify: Securing the Container Supply Chain with Signing

Container image signing plays a crucial role in ensuring the authenticity and integrity of the code within your containerized applications. It works like a digital fingerprint, allowing you to verify that the image you’re deploying hasn’t been tampered with or modified in any way. This explicitly blocks potential vulnerabilities such as Leaky Vessel by ensuring you never execute unknown code.

Benefits

  • Increased security: Signing helps prevent attackers from injecting malicious code into your images, potentially compromising your entire system.
  • Enhanced trust: Signing verifies the image’s origin and publisher, ensuring you’re deploying what you intended.
  • Improved compliance: Some industries and regulations mandate signed images for security and audit purposes.
  • Reduced risk: Identifying and mitigating vulnerabilities becomes easier when you know precisely what code you’re running.

Mechanisms

  • Signing Key Pair: You obtain a signing key pair with a private key (kept secret) and a public key (distributed).
  • Signing the Image: The private key is used to sign the container image cryptographically, creating a signature.
  • Verifying the Signature: On deployment, the public key and signature are used to confirm that the image hasn’t been altered.

Popular Tools

  • Cosign: An open-source tool based on the Sigstore standard, supporting keyless signing and various signing authorities.
  • Notary: An integrated solution from Docker for signing and verifying images within their ecosystem.
  • AWS Signer: A managed service from AWS for signing images stored in their Elastic Container Registry (ECR).
  • Venafi CodeSign Protect:  A managed X.509 and HSM service with enterprise-grade signing and verification.

 

Tie It All Together

If you do nothing else, please consider:

  1. Scan your nodes for vulnerable runc versions: Use the provided Kyverno policy to identify nodes running vulnerable versions of containerd or runc (CVE-2024-21626 & CVE-2024-23651). Patch affected nodes immediately.
  2. Audit your deployments for suspicious container behavior: Implement the Kyverno policy to detect containers attempting to mount /proc/self/fd/*, the technique used in the Leaky Vessels exploit. Run it in audit mode first to identify potential false positives.
  3. Block execution of untrusted images: Enforce the Kyverno policy that checks for images containing /proc/self/fd/ in their layers, potentially indicating tampering.

 

For long-term security, use Nirmata Policy Manager and Kyverno together to offer robust image security integrations, providing you with the ability to:

  1. Implement image signing and verification: This ensures the authenticity and integrity of your container images, preventing vulnerabilities like Leaky Vessels. Consider tools like Cosign, Notary, AWS Signer, or Venafi CodeSign Protect.
  2. Prioritize image trust: Only pull images from trusted sources and registries. Avoid untrusted images to minimize the attack surface.
  3. Maintain comprehensive patching: Patch not just Kubernetes but also low-level components like runc. A chain is only as strong as its weakest link.
  4. Use Nirmata Policy Manager: Leverage these tools to enforce security policies and automate vulnerability detection and mitigation.

 

If you would like to manage policies and governance easily across multiple clusters, check out what Nirmata is building at https://nirmata.com/.

The post Locked Doors, Untrusted Keys: Securing Containers in the Wake of Leaky Vessel Vulnerabilities first appeared on Nirmata.]]>
The Need for Speed: Optimizing Kyverno’s Performance https://nirmata.com/2024/01/30/the-need-for-speed-optimizing-kyvernos-performance-under-pressure/ Tue, 30 Jan 2024 22:09:45 +0000 https://nirmata.com/?p=9415   This blog post was co-authored by Khaled Emara. About Kyverno Kyverno is a policy engine designed for Kubernetes and cloud native workloads. Policies can be managed as Kubernetes resources, and no new language is required to write policies. Policy reports and exceptions are also...

The post The Need for Speed: Optimizing Kyverno’s Performance first appeared on Nirmata.]]>
motorcycle 1690452 1920

 

This blog post was co-authored by Khaled Emara.

About Kyverno

Kyverno is a policy engine designed for Kubernetes and cloud native workloads. Policies can be managed as Kubernetes resources, and no new language is required to write policies. Policy reports and exceptions are also Kubernetes resources. This approach allows using familiar tools such as kubectl, git, and kustomize to manage policies and results.  Kyverno policies can validate, mutate, generate, and clean up Kubernetes resources, as well as verify image signatures and artifacts to help secure the software supply chain. The Kyverno CLI can be used to test policies and validate resources as part of a CI/CD pipeline.

Since Kyverno operates as an admission controller, it needs to be very secure, highly scalable, and reliable, as high latency or other performance issues can impact cluster operations. In this post, we will detail some of the key performance improvements we have made to Kyverno, with the help of our customers and community, and how we test for load and performance for each new release of Kyverno.

Prior Performance

The Nirmata team was approached by a large multinational e-commerce company that was replacing Pod Security Policies with Kyverno policies. They operate some of the world’s largest clusters and have extremely stringent load and performance requirements.  While Kyverno worked well in normal conditions, they were seeing poor performance under high loads. Increasing memory and CPU resources helped a bit, but performance would quickly degrade when further load was applied.

Around the same time, a platform team at a leading automotive company was also evaluating Kyverno and one of the engineers contributed a load testing method they used with K6. This load testing tool allows writing API tests in Javascript. 

The teams simulated real-world scenarios to evaluate the system’s behavior under different levels of stress. This methodology provided valuable insights into Kyverno’s scalability, uncovering performance thresholds and potential pain points that required addressing. The load testing phase became instrumental in ensuring Kyverno’s robustness in handling varying workloads.

K6 has multiple executors, the most popular among which is the shared-iterations executor. This executor creates a number of concurrent connections called virtual users. The entirety of the number of iterations is then distributed among these virtual users.

We created a test wherein we installed Kyverno policies to enforce the Kubernetes pod security standards using 17 policies. Then, we developed an incompatible Pod test and measured how long Kyverno takes to reject the admission request.

With Kyverno 1.10, after applying the test above using 25vus and 250 iterations, we initially got the following results:

http_req_duration..........: avg=127.95ms min=65.63ms med=124.75ms max=245.33ms p(90)=177.64ms p(95)=190.64ms

For 25vus, 500 iterations:

http_req_duration..........: avg=116.47ms min=60.68ms med=111.5ms  max=284.21ms p(90)=158.41ms p(95)=175.03ms

For 25vus, 1000 iterations:

http_req_duration..........: avg=110.36ms min=55.69ms med=105.11ms max=292.46ms p(90)=146.54ms p(95)=161.88ms

For 50vus, 1000 iterations:

http_req_duration..........: avg=222.59ms min=54.26ms med=211.25ms max=497ms    p(90)=326.33ms p(95)=351.39ms

For 100vus, 1000 iterations:

http_req_duration..........: avg=430.43ms min=98.94ms med=382.43ms max=2.07s   p(90)=652.73ms p(95)=782.91ms

Profiling using pprof in golang

To gain deeper insights into Kyverno’s performance bottlenecks, the team employed the powerful pprof tool in Golang, by profiling the application allowed for a comprehensive analysis of CPU and memory usage, aiding in the identification of critical areas that required attention. 

This data-driven approach played a crucial role in steering the performance optimization efforts, enabling the team to make informed decisions on where to focus their efforts for maximum impact.

After enabling pprof in Kyverno and generating a flamegraph, we got the following:

SCR 20240129 sizs

 

From the graph, it was clear that checking preconditions and copying JSON takes a large amount of time. 

After investigating, it appears that it uses the internal “Context” to store information about the request rule and process variable information. The Context was stored in a JSON structure, and for each query, a marshal and unmarshal operation was required. 

It was also found that each request created a new JMESPath Interpreter, which was allocated a large function map for custom JMESPath filters, which is wasteful as these do not change over the life of the application.

Finally, we also watched the goroutine profile, and every time we did the test, it would increase to 4000 goroutines and decay at a languid pace of 20 goroutines per second. This caused a spike in memory. After further investigations, it appeared to be from the EventBroadcaster, which created a goroutine for each Event.

Changes Made

A number of changes were made to improve Kyverno’s overall performance. These changes are tracked in issue #8260, and the main ones are discussed below:

In-memory Golang Map for Context

One significant enhancement involved replacing the existing JSON structure used for storing context with an in-memory Golang map. The previous approach involved repetitive marshaling and unmarshaling, contributing to unnecessary overhead. By adopting an in-memory map, Kyverno reduced these operations, leading to a more efficient and performant context management system.

jsoniter Implementation

Recognizing the need for optimized JSON handling, the team transitioned from the standard Golang library to jsoniter. This lightweight and high-performance JSON library proved to be a strategic choice, significantly improving the efficiency of JSON-related operations within Kyverno. The adoption of jsoniter contributed to a more streamlined execution of tasks involving JSON structures.

Optimizing event generation

In our quest to enhance event processing efficiency on our Kubernetes cluster, we encountered a bottleneck with the default EventBroadcaster from client-go. The creation of a goroutine for every event led to memory spikes and frequent controller shutdowns, disrupting the stability of our applications.

To address this challenge, we opted for a customized solution, implementing a tailored Watcher function using client-go. This approach provided us with greater control over event creation and processing, allowing us to circumvent the issues associated with excessive goroutine generation.

The results were transformative – the optimized event processing solution alleviated memory spikes, eliminating controller shutdowns and enhancing the overall stability and reliability of our Kubernetes environment. This journey highlights the significance of tailoring solutions to specific Kubernetes requirements, showcasing the impact of a well-designed Watcher function on the dynamic orchestration landscape.

Optimizing policy matching

Kyverno manages an in-memory cache of policies and matches them to each incoming admission request. One area of optimization was to restructure the match logic so that more straightforward comparisons are performed first and more complex match or exclude checks are deferred. This reduces the time taken per request, and although it’s a small amount for each request, this processing time can add up under high loads.

Optimizing webhook configurations

Kyverno automatically manages webhook configurations to instruct the Kubernetes API on which API requests should be forwarded. The webhook configuration is done per-policy and in a fine grained manner to reduce the number of requests Kyverno needs to handle. An optimization made was to enable configuration and propagation of admission operations, e.g., CREATE, UPDATE, DELETE, or CONNECT from the policy rules to webhook configurations. This way, if a policy rule should only be applied for CREATE and UPDATE operations, Kyverno will not be invoked for a DELETE or CONNECT. 

Current Performance

The culmination of these changes brought about a substantial improvement in Kyverno’s overall performance. Metrics and benchmarks revealed notable reductions in response times, decreased resource consumption, and enhanced system stability. The optimizations made to context storage, JSON handling, and EventBroadcaster significantly contributed to a more responsive and reliable Kyverno, offering an improved experience for end-users and administrators.

http_req_duration..........: avg=15.52ms  min=6.87ms  med=15ms    max=36.81ms  p(90)=22.84ms  p(95)=25ms

As you can see, that’s almost an 8X improvement in average and p95 latency. Furthermore, since we resolved the OOMKill issue, we have been able to obtain stable performance results from higher iterations.

http_req_duration..........: avg=15.41ms  min=6.37ms  med=14.98ms max=31.02ms  p(90)=21.44ms  p(95)=23.4ms

25vus and 1000 iterations:

http_req_duration..........: avg=16.77ms  min=6.81ms  med=16.02ms max=47.92ms  p(90)=24.04ms  p(95)=26.61ms 

50vus and 1000 iterations:

http_req_duration..........: avg=31.39ms  min=7.16ms  med=28.83ms max=96.98ms  p(90)=49.58ms  p(95)=57.7ms

100vus and 1000 iterations:

http_req_duration..........: avg=64.34ms  min=13.06ms med=51.61ms max=236.42ms p(90)=120.99ms p(95)=160.19ms

Plotting the Average Request Duration we get:

 

SCR 20240129 skvw

As the results indicate, even under high load Kyverno’s average processing time stays well under 100 milliseconds.

Automating Tests

Recognizing the importance of maintaining performance gains, the Kyverno team implemented robust automated testing processes. These automated tests ensured that any future code changes, updates, or additions would undergo thorough performance validation. By automating the testing pipeline, the team established a proactive approach to identifying and addressing potential regressions, guaranteeing the longevity of the performance improvements achieved.

Our GitHub Actions workflow incorporates a custom K6 script that deploys pods intentionally violating Kubernetes Pod Security Policies (PSPs). These bad pods encounter Kyverno’s policies, triggering policy evaluation and enforcement. By varying the number of concurrent pods, we observe how Kyverno scales, ensuring its effectiveness even when handling large deployments.

But our testing continues beyond there. We didn’t only create tests that do simple validation – we created tests that delve into Kyverno’s diverse capabilities. We run additional scenarios to test mutating policies: K6 deploys pods with specific attributes, triggering Kyverno policies that modify resource configurations. This ensures seamless mutation and its impact on performance. Evaluate policy generation: We simulate scenarios where Kyverno automatically generates resource-specific policies based on pre-defined templates. K6 tests the efficiency and accuracy of this dynamic policy generation.

The benefits of this K6-powered approach are multifold:

  • Early detection of scalability issues: We catch performance bottlenecks before they impact production environments, ensuring a smooth and reliable Kyverno experience.
  • Confidence in Kyverno’s robustness: Rigorous testing instills confidence in Kyverno’s ability to handle large-scale deployments with grace and efficiency.
  • Continuous improvement: K6 data feeds into our CI/CD pipeline, enabling us to refine and optimize Kyverno’s performance continuously.

By integrating K6 with GitHub Actions, we’ve empowered Kyverno to scale gracefully and handle complex policy enforcement scenarios with confidence. This proactive approach ensures our Kubernetes environments remain secure and compliant, even as our infrastructure grows and evolves.

Conclusion

In conclusion, Kyverno’s journey toward performance optimization showcases the project’s dedication to delivering a high-performing and reliable solution for Kubernetes policy management. We are grateful to our customers and community for their support and assistance.

In addition to its unified governance solutions, Nirmata provides an enterprise distribution of Kyverno (called N4K) with long term support for Kyverno and Kubernetes releases, SLAs for CVEs and critical fixes, and 24×7 enterprise support required by enterprises since Kyverno constitutes an critical component for production environments. 

These changes were initially implemented into N4K v1.10 and have now all been contributed back to the upstream OSS distribution for the upcoming v1.12 release! 

The challenges faced, insights gained through profiling, and the strategic changes implemented collectively contribute to Kyverno’s evolution. As the project continues to scale, these experiences serve as valuable lessons, shaping the ongoing commitment to providing a secure and efficient policy engine within the CNCF ecosystem.

If you have not tried out Kyverno, get started at https://kyverno.io.

If you would like to manage policies and governance across clusters with ease, check out what Nirmata is building at https://nirmata.com/.

The post The Need for Speed: Optimizing Kyverno’s Performance first appeared on Nirmata.]]>
Preventing “Sys:All” vulnerability from ruining your day https://nirmata.com/2024/01/26/preventing-sysall-vulnerability-from-ruining-your-day/ Fri, 26 Jan 2024 02:30:49 +0000 https://nirmata.com/?p=9407 Google Kubernetes Engine (GKE) is a popular tool for managing containerized applications. However, a critical vulnerability has been discovered in GKE that could allow attackers to take control of a Kubernetes cluster. The vulnerability is called “Sys:All” and is caused by a misunderstanding of the...

The post Preventing “Sys:All” vulnerability from ruining your day first appeared on Nirmata.]]>
nirmata sys all blog v4

Google Kubernetes Engine (GKE) is a popular tool for managing containerized applications. However, a critical vulnerability has been discovered in GKE that could allow attackers to take control of a Kubernetes cluster. The vulnerability is called “Sys:All” and is caused by a misunderstanding of the “system:authenticated” group.

What is the attack vector?

The “system:authenticated” group is a default Kubernetes group that includes any authenticated Google account. This means that any Google account, not just verified users within an organization, can be used to gain access to a Kubernetes cluster. Attackers can exploit this vulnerability by using a Google account to authenticate themselves and then gain access to perform malicious activities.

How do we remediate it?

Google has taken steps to fix the most severe exploit in newer GKE versions, but many clusters remain vulnerable. To protect your clusters, you can use the Kyverno policy that restricts the use of the groups necessary to exploit “Sys:All.” This policy blocks the binding of ‘cluster-admin’ to the specific vulnerable groups/users: system:authenticated, system:unauthenticated, and system:anonymous. 

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-clusteradmin-rolebindings
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-restricted-cluster-admin-bindings
    match:
      resources:
        kinds:
        - ClusterRoleBinding
        - RoleBinding
    validate:
      message: "Binding ClusterRole 'cluster-admin' is restricted, system:authenticated, system:unauthenticated and system:anonymous are not allowed."
      deny:
        conditions:
          all:
          - key: "{{ request.object.subjects[].name }}"
            operator: AnyIn
            value:
            - system:authenticated
            - system:unauthenticated
            - system:anonymous 
         - key: "{{ request.object.roleRef.name }}"
           operator: Equals
           value: "cluster-admin"

With the policy in place, we can now ensure that cluster-admin can no longer assume the vulnerable groups and users.


# cat cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: good-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:test2

# kubectl apply -f cluster-role.yaml
Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding"
Name: "good-clusterrolebinding", Namespace: ""
for: "cluster-role.yaml": error when patching "cluster-role.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource ClusterRoleBinding//good-clusterrolebinding was blocked due to the following policies

restrict-clusteradmin-rolebindings:
  validate-restricted-cluster-admin-bindings: Binding ClusterRole 'cluster-admin'
    is restricted, system:authenticated, system:unauthenticated and system:anonymous
    are not allowed.

Conclusion

It is crucial to implement least-privilege access control practices for Kubernetes to prevent such vulnerabilities. While there are currently no known widespread attacks using this vulnerability, experts warn that it’s just a matter of time. Therefore, it is highly recommended that you take action now and protect your Kubernetes clusters from potential attacks.

If you are interested in modernizing and automating security with policy as code, check out Nirmata’s Policy Manager and reach out to set up a deep dive!

The post Preventing “Sys:All” vulnerability from ruining your day first appeared on Nirmata.]]>
Enhancing Application Security with Policy-as-Code https://nirmata.com/2024/01/18/enhancing-application-security-with-policy-as-code/ Thu, 18 Jan 2024 22:23:24 +0000 https://nirmata.com/?p=9381   In the rapidly evolving landscape of software development, security is a huge concern. The integration of policy-as-code (PaC) into the development process is revolutionizing how organizations safeguard their applications. This blog post delves into the significance of PaC in enhancing the security posture of...

The post Enhancing Application Security with Policy-as-Code first appeared on Nirmata.]]>
Neon Blue Glitch Youtube Thumbnail

 

In the rapidly evolving landscape of software development, security is a huge concern. The integration of policy-as-code (PaC) into the development process is revolutionizing how organizations safeguard their applications. This blog post delves into the significance of PaC in enhancing the security posture of modern applications, contrasting proactive security with reactive security, and highlighting the role of admission controllers in Kubernetes-based developer platforms. Additionally, we explore the benefits of policy-as-code in scanning infrastructure-as-code (IaC) manifests within CI/CD pipelines and discuss the advantages of central governance and visibility.

Proactive vs Reactive Security: A Paradigm Shift

Traditionally, security in software has been reactive: responding to threats and vulnerabilities after they have been exploited. This approach is no longer viable in a world where threats evolve faster than ever and your applications and infrastructure are dynamic. Proactive security, on the other hand, involves anticipating and mitigating risks before they become actual threats. Policy-as-code is a key enabler of proactive security. By implementing security policies, teams can automatically enforce standards and best practices, significantly reducing the likelihood of misconfigurations and vulnerabilities.

Policy-as-Code in Kubernetes: Empowering Admission Controllers

Kubernetes, the de facto standard for orchestrating containerized applications, presents unique security challenges. Admission controllers in Kubernetes are pivotal in ensuring that the configurations applied to the Kubernetes cluster adhere to the required security standards. By integrating policy-as-code with these controllers, organizations can automatically enforce consistent security policies at admission across all deployments. This integration empowers teams to prevent non-compliant resources from being deployed, thereby reducing the attack surface. Kyverno is an example of a policy engine that can be installed as an admission controller in Kubernetes.

Scanning Infrastructure-as-Code in CI/CD Pipelines

Infrastructure-as-code has transformed how infrastructure is provisioned and managed, allowing for consistent and repeatable deployments. However, this shift also necessitates a change in how infrastructure is secured. PaC plays a vital role here, enabling teams to scan IaC manifests for compliance and security issues as part of the CI/CD pipeline. This approach ensures that any potential security risks are identified and addressed early in the development cycle, long before the code is deployed. Kyverno, the Kubernetes-native policy engine can also be used to enforce policies in CI/CD pipelines.

Benefits of Policy-as-Code

  1. Automated Compliance: policy-as-code allows for the automation of compliance checks, ensuring that all deployed resources are in line with organizational policies and regulatory requirements.
  2. Early Detection of Vulnerabilities: By integrating policy-as-code into the development pipeline, misconfigurations and vulnerabilities can be detected and remediated early, reducing the risk of exploits in production.
  3. Consistency Across Environments: policy-as-code ensures that security policies are consistently applied across all environments, from development to production.
  4. Reduced Operational Overhead: Automating policy enforcement reduces the need for manual reviews and intervention, thereby lowering operational overhead.

Central Governance and Visibility

A central aspect of policy-as-code is the governance and visibility it provides across all infrastructure. Organizations can centrally manage and monitor security policies, ensuring that they are consistently applied across all environments. Developers and application owners can request exceptions to certain policies temporarily if needed through centralized, intuitive approval workflows. This centralized approach not only simplifies management but also provides clear visibility into the security posture of the entire infrastructure, making it easier to rapidly identify and address potential risks.

Conclusion

The adoption of policy-as-code is a game-changer in enhancing the security posture of modern applications. By shifting from a reactive to a proactive security approach, integrating policy-as-code with Kubernetes admission controllers such as Kyverno, scanning IaC manifests in CI/CD pipelines, and centralizing governance, organizations can significantly mitigate risks and ensure a robust security framework. As the digital landscape continues to evolve, the role of policy-as-code in maintaining the security and integrity of applications will become increasingly crucial.

Get our complimentary eBook – Securing Kubernetes Using Policy-as-Code – and learn more from Nirmata on the value and efficiencies of policy-as-code for development needs! Sign up here for your copy.

The post Enhancing Application Security with Policy-as-Code first appeared on Nirmata.]]>
Using nctl to enforce security in CI/CD pipelines https://nirmata.com/2024/01/16/nirmata-empowering-secure-platform-engineering/ Tue, 16 Jan 2024 21:53:15 +0000 https://nirmata.com/?p=9372 In today’s age of advanced technology platforms, organizations must proactively prioritize security concerns across all teams and platforms. Nirmata offers an efficient solution that allows platform engineering teams to do just that. Nirmata helps organizations stay proactive and ahead of potential security threats. One of...

The post Using nctl to enforce security in CI/CD pipelines first appeared on Nirmata.]]>
Untitled design 4

In today’s age of advanced technology platforms, organizations must proactively prioritize security concerns across all teams and platforms. Nirmata offers an efficient solution that allows platform engineering teams to do just that. Nirmata helps organizations stay proactive and ahead of potential security threats.

One of Nirmata’s distinctive features is its policy-as-code automation. This feature enables platform teams to govern security, operations, and best practices seamlessly across Kubernetes clusters, CI/CD pipelines, and cloud services. By automating policies, platform teams can eliminate the risk of human error and ensure that best practices are followed consistently across all teams.

Overall, Nirmata is invaluable for organizations seeking to enhance their platform engineering capabilities and stay ahead of potential security threats. By empowering platform teams to take a proactive approach and automate policies, Nirmata helps organizations operate more efficiently, reduce risk, and deliver better results.

blogimage1

CLI tools are essential for platform engineers and DevOps professionals, offering automation, efficiency, and versatility in managing infrastructure, deployment pipelines, and collaborative workflows. Their ability to perform tasks programmatically aligns seamlessly with the goals of DevOps practices.

 

Introducing NCTL: Nirmata’s Command Line Tool

NCTL (Nirmata Command Line Tool) is a comprehensive solution for managing the entire cluster governance lifecycle through a user-friendly Command Line Interface. NCTL empowers users with commands for onboarding and deboarding clusters, scanning clusters with best practices policies, and auto-remediating resources for policy violations! The tool offers a range of commands for various purposes:

blogimage2

NCTL makes handling Kubernetes cluster governance a breeze—it’s an all-in-one solution that boosts efficiency and saves time. Plus, it plays well with CI/CD pipelines and can be used programmatically, making cluster management smooth and seamless.

 

How to Get NCTL: Download Now!

NCTL is free to use and explore! It can be easily obtained from this link. Empower your platform engineering and DevOps journey with Nirmata’s powerful CLI tool, simplifying cluster governance and enhancing security best practices.

Stay tuned for more blog posts in this series. We will explore nctl and see more hands-on examples as we go ahead.

The post Using nctl to enforce security in CI/CD pipelines first appeared on Nirmata.]]>
GitOps and Mutating Policies: The Tale of Two Loops https://nirmata.com/2024/01/03/gitops-and-mutating-policies-the-tale-of-two-loops/ Wed, 03 Jan 2024 18:27:57 +0000 https://nirmata.com/?p=9360 Do policies that mutate or generate resources violate GitOps principles? In this blog post, I will show you how policy-based resource management can be complementary to GitOps, what benefits it provides, and how to use Kyverno to mutate and generate rules with popular GitOps tools...

The post GitOps and Mutating Policies: The Tale of Two Loops first appeared on Nirmata.]]>
Do policies that mutate or generate resources violate GitOps principles?

In this blog post, I will show you how policy-based resource management can be complementary to GitOps, what benefits it provides, and how to use Kyverno to mutate and generate rules with popular GitOps tools like Flux and ArgoCD.

Kubernetes Control Loops

The Controller pattern is a foundational concept in Kubernetes. Controllers are driven by declarative configurations that define the system’s desired state and run a control loop reconciling the current state with the desired state.

Kubernetes has several built-in controllers, like pod controllers. Kubernetes is extensible and supports custom controllers that can run inside of Kubernetes.

The GitOps Control Loop 

The GitOps pattern is another form of the controller pattern, where the desired state is versioned in Git, and the GitOps controller reconciles resources in clusters with resources declared in Git. 

The size of the control loops matters, and the tighter and more self-contained the loop, the better.  Tighter control loops are less error prone and more efficient, with fewer moving pieces. Tighter control loops can also offer greater security, as the attack surface is reduced when there are fewer dependencies on external systems.

The Policy Control Loop

Kubernetes offers many forms of policies, including API resources. Policy management solutions, like Kyverno and OPA/Gatekeeper, use dynamic admission controls to intercept API requests.

A Kubernetes policy controller is also a form of a control loop where the desired state is defined as a declarative policy. A policy controller applies policies to resource configurations in near real-time. Policies can be used to validate and block or flag insecure and misconfigured resources. Policies can also mutate, generate, and delete (cleanup) resources based on customizable criteria. 

Maintaining State in Git

“I want all cluster states to be stored in git” is often stated as a reason for not using mutation or generation policies. 

The goal of GitOps is to make the cluster state fully reproducible from Git and to store just enough to recover and track the state entirely.

For example, it does not make sense to store each Pod declaration in git as Kubernetes pod controllers like Deployments are used to manage the lifecycle of pods in a cluster. Hence, only pod controller declarations are stored in Git.

Similarly, policies can also be stored in Git and applied to clusters. A controller then watches the policy states, dynamically enforces required policies, and manages configurations.

The Advantages of Using Policies

Managing configurations using policies can bring numerous advantages. For instance, policies can simplify the configuration management process, enforce security measures, and promote best practices. In addition to these benefits, there are other advantages to using policies for configuration management.:

1. Just in Time Resource Management: policies can apply missing defaults and generate complete resources for security, isolation, multi-tenancy, or other concerns. Since this happens directly inside the cluster, it works with kubectl or any other Kubernetes client tool.

2. Tighter Security Controls: policy controllers, like Kyverno, register as dynamic admission controllers and monitor each API request. Since attackers will directly exploit security weaknesses within a cluster and bypass the CI/CD pipelines, admission controllers act as an essential line of defense. 

3. Separation of Concerns: policies allow cleanly separating concerns across development, operations, and security roles. For example, some policies may be mandated by the security team, whereas the operations team may manage other policies. A policy controller consolidates and applies policies across all. 

Avoiding Sync Errors with Server Side Apply (SSA)

Kubernetes Server Side Apply (SSA) is a feature that allows multiple controllers to collaborate on changes to a resource. 

Both FluxCD and ArgoCD can leverage SSA to coordinate changes with policy engines and other mutating admission controllers.

Kyverno and Flux

Flux enables SSA by default, so there is nothing special to configure.

This git repository, from Stefan Prodan, contains a detailed demo of how to use Kyverno policies to mutate a pod security context:

https://github.com/stefanprodan/gitops-kyverno 

Kyverno and ArgoCD

ArgoCD has added support for server-side diff in version 2.10. Here are the details:

https://argo-cd.readthedocs.io/en/latest/user-guide/diff-strategies/#server-side-diff

To enable SSA based diffs, an annotation must be specified on the application or globally via the “argocd-cmd-params-cm” config map:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true 
...

Demo

Here is a complete demonstration of how to use ArgoCD and Kyverno together for mutating resource configurations using policies:

Create a kind Cluster

Install kind and create a cluster:

kind create cluster --name=kyverno-argocd

Install Kyverno

kubectl create -f https://github.com/kyverno/kyverno/raw/main/config/install-latest-testing.yaml

Install Kyverno Policies

This command will install three Kyverno policies described below:

kubectl apply -f https://raw.githubusercontent.com/nirmata/demo-argocd-kyverno/main/config/kyverno-policies.yaml

This policy replaces all image tags with digests, which are immutable and more secure. Note that the policy does not perform any other form of image verification, like checking signatures or attestations:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: image-digest
 annotations:
   policies.kyverno.io/title: Convert tags to digests
   policies.kyverno.io/category: Supply Chain Security
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/minversion: 1.11.0
   policies.kyverno.io/description: >-
     Image tags are mutable and can be spoofed. This policy resolves
     image tags to digests which are immutable.
spec:
 validationFailureAction: Enforce
 rules:
   - name: replace-tag
     match:
       any:
       - resources:
           kinds:
             - Pod
     exclude:
       all:
       - resources:
           namespaces:
             - argocd     
     verifyImages:
     - imageReferences:
       - "*"
       required: false
       verifyDigest: true
       mutateDigest: true

This policy allows the Kubernetes cluster auto-scaler to evict pods that use emptyDir by adding the annotation `cluster-autoscaler.kubernetes.io/safe-to-evict: true` if a pod contains an emptyDir volume.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: add-safe-to-evict
 annotations:
   policies.kyverno.io/title: Add Safe To Evict
   policies.kyverno.io/category: Other
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/minversion: 1.6.0
   policies.kyverno.io/description: >-
     The Kubernetes cluster autoscaler does not evict pods that
     use hostPath or emptyDir volumes. To allow eviction of these
     pods the annotation cluster-autoscaler.kubernetes.io/safe-to-evict
     must be set to `true`.    
spec:
 rules:
 - name: annotate-empty-dir
   match:
     any:
     - resources:
         kinds:
         - Pod
   exclude:
     all:
     - resources:
         namespaces:
         - argocd
   mutate:
     patchStrategicMerge:
       metadata:
         annotations:
           +(cluster-autoscaler.kubernetes.io/safe-to-evict): "true"
       spec:         
         volumes:
         - <(emptyDir): {}

This policy turns off the auto-mount of service account tokens:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: service-account
 annotations:
   policies.kyverno.io/title: Disable Auto-Mount of Service Accounts
   policies.kyverno.io/category: Other
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/minversion: 1.6.0
   policies.kyverno.io/description: >-
     Kubernetes automatically mounts ServiceAccount credentials in each Pod.
     The ServiceAccount may be assigned roles allowing Pods to access API resources.
     Blocking this ability is an extension of the least privilege best practice and should
     be followed if pods do not need to speak to the API server to function.
     This policy ensures that mounting of these ServiceAccount tokens is blocked.
spec:
 rules:
   - name: disable-automount-sa
     match:
       any:
         resources:
           kinds:
           - Pod
     exclude:
       all:
       - resources:
           namespaces:
           - argocd
     mutate:
       patchStrategicMerge:
         spec:
           +(automountServiceAccountToken): false

Install ArgoCD

Note that a version greater than 2.10 is required:

kubectl create ns argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.0-rc1/manifests/install.yaml

Create the ArgoCD Application

Install the ArgoCD application:

kubectl apply -f https://raw.githubusercontent.com/nirmata/demo-argocd-kyverno/main/config/argocd-app/application.yaml

The Application manifest looks like this – note the required annotation:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  name: demo-app
  namespace: argocd
spec:
  destination:
    namespace: demo
    server: https://kubernetes.default.svc
  project: default
  source:
    path: config/kubernetes/
    repoURL: https://github.com/nirmata/demo-argocd
    targetRevision: HEAD
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

Once the application is deployed, we can check whether the policies have been applied. 

Check the running application for image tags:

kubectl -n demo get pod -o yaml | grep "image:"

This should show that images are references using digests:

    - image: ghcr.io/nirmata/demo-argocd:v1@sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
      image: sha256:f9d5de0795395db6c50cb1ac82ebed1bd8eb3eefcebb1aa724e01239594e937b

Check the running application for service accounts:

This command will check for any volumes and if automountServiceAccountToken is disabled.

kubectl -n demo get pod -o yaml | grep -i "serviceAccountToken"

The output should only contain instructions to turn off the auto-mounting of service account tokens:

    automountServiceAccountToken: false

Check the running application for cluster autoscaler annotations:

This command will check the pod annotations:

kubectl -n demo get pod -o yaml | grep -i "annotations" -A1

The output should show the “cluster-autoscaler.kubernetes.io/safe-to-evict: true” annotation:

    annotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

You can also view the pod and deployment details using ArgoCD:

 

Since Kyverno policy reports are namespaced Kubernetes resources, you can also check the compliance of each resource directly in the ArgoCD UI!

Conclusion

This blog post highlights the importance of mutating policies in securing and operating clusters and provides a tutorial on using Kyverno policies with FluxCD and ArgoCD. The post offers practical examples, such as adding pod security context defaults, necessary annotations, and converting image tags to digests during admission controls, all of which make sense to centralize using policies. If you’re interested in exploring other mutation policies, you can find them at https://kyverno.io/policies/?policytypes=mutate

Moreover, Kyverno also supports policies that generate resources, which can simplify self-service use cases for multi-tenancy and application isolation. To view examples of such policies, check out https://kyverno.io/policies/?policytypes=generate

Managing configurations using policies can bring numerous advantages. For instance, policies can simplify the configuration management process, enforce security measures, and promote best practices, while maintaining separation of concerns across developers, operations, and security teams.

For those looking to maximize the value of running Kyverno in production, Nirmata’s policy and governance platform offers a robust solution and a free trial! Simply head to https://try.nirmata.io/ to get started. 

 

The post GitOps and Mutating Policies: The Tale of Two Loops first appeared on Nirmata.]]>