How can I validate network policies within a Kubernetes cluster?
This document (000020041) is provided subject to the disclaimer at the end of this document.
Environment
- A Kubernetes cluster v1.15.2+
- kubectl access to the cluster
- Network Plugin (CNI) which supports Network Policies
Situation
Traffic flowing inside a kubernetes cluster is non-isolated by default so that each pod can communicate with every other pod. In some environments there is a need to ensure the proper isolation or restriction of each application in differing namespaces. Network Policies handle this microservice network segmentation and meet this need by defining the control to other entities' IP addresses (on OSI Layer 3) or network ports (on OSI Layer 4).
Access for Pod communication with other entities are identified with Network Policies and defined by other pods, the relevant namespaces, and IP CIDR Blocks. Pods and Namespaces are specified using a selector, like app=example .
Once the Network Policies are defined, how can a Kubernetes administrator test and validate them?
Resolution
Illuminatio is an open-source project written in Python3 by Inovex. It can run standalone, and is also available as a docker container. It creates test-cases for both the network policies and their inverse rules, generates an illuminatio-runner daemonset, tests all the cases against the defined network policies, and reports back on success or failure for each rule and inverted-rule. Illuminatio can use the current kubectl config for cluster access while working in the shell session, or designate the config file with the optional --kubeconfig flag.
Basic Usage
Assuming the Kubernetes admin has some network policies to test, the tool is very easy to use. It has three verbs to choose from, "clean", "generate" and, "run". The generate verb will only generate the tests, while clean removes them and run performs the test. Most users will want to use illuminatio clean run to start fresh, run the generated tests and report on their success. The results are also written to a configmap.
The following are some common examples of Network Policies, and how Illuminatio can assist with validation. Examples are taken from this network policies recipes github repo, and applied to a kubernetes cluster, in the default namespace. Validation is performed with Illuminatio instead of a temporary pod.
Deny Traffic to an application
Save this file as web-deny-all.yaml and then apply the network policy with kubectl -f web-deny-all.yaml. Notice it is deploying to the default namespace. Prepare the pod for this example with a selector of app=web.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-deny-all
spec:
podSelector:
matchLabels:
app: web
ingress: []
kubectl run --generator=run-pod/v1 web --image=nginx --labels app=web --expose --port 80
Show the current network policies, then run cases for all of them. Illuminatio will deploy a deamonset and run all the test cases, any passing tests show "success" in the last column of the report. Note: success indicates the test was successful, even if testing a connection denial.
$ kubectl get netpol
NAME POD-SELECTOR AGE
web-deny-all app=web 13m
$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Generated 1 cases in 0.0616 seconds
FROM TO PORT
default:app=web default:app=web -*
Ensure that Pods of DaemonSet illuminatio-runner are ready
Finished running 1 tests in 7.1175 seconds
FROM TO PORT RESULT
default:app=web default:app=web -* success
Limit Traffic to an application
Allow app=bookstore pods to communicate with only other app=bookstore pods.
kubectl run --generator=run-pod/v1 apiserver --image=nginx --labels app=bookstore,role=api --expose --port 80
Save the following as api-allow.yaml and issue kubectl apply -f api-allow.yaml.
Network policies are accumulative.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: api-allow
spec:
podSelector:
matchLabels:
app: bookstore
role: api
ingress:
- from:
- podSelector:
matchLabels:
app: bookstore
$ kubectl apply -f api-allow.yaml
networkpolicy.networking.k8s.io/api-allow created
$ kubectl get netpol
NAME POD-SELECTOR AGE
api-allow app=bookstore,role=api 5s
web-deny-all app=web 18m
$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Generated 5 cases in 0.0594 seconds
FROM TO PORT
illuminatio-inverted-default:app=bookstore default:app=bookstore,role=api -*
illuminatio-inverted-default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -*
default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -*
default:app=web default:app=web -*
default:app=bookstore default:app=bookstore,role=api *
Ensure that Pods of DaemonSet illuminatio-runner are ready
Finished running 5 tests in 13.2368 seconds
FROM TO PORT RESULT
illuminatio-inverted-default:app=bookstore default:app=bookstore,role=api -* success
illuminatio-inverted-default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -* success
default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -* success
default:app=web default:app=web -* success
default:app=bookstore default:app=bookstore,role=api * success
Allow whitelisted traffic for app=web
This policy will whitelist the app=web pods from the first example, with a new web-allow-all.yaml file. This Network Policy also voids the first example, by allowing all traffic. Because the traffic connections are allowed, Illuminatio recognizes this and avoids generating the negative (inverted) test cases.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-allow-all
namespace: default
spec:
podSelector:
matchLabels:
app: web
ingress:
- {}
$ kubectl apply -f web-allow-all.yaml
networkpolicy.networking.k8s.io/web-allow-all created
$ kubectl get netpol
NAME POD-SELECTOR AGE
api-allow app=bookstore,role=api 20m
web-allow-all app=web 32s
web-deny-all app=web 39m
$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Not generating negative tests for host ClusterHost(namespace=default, podLabels={'app': 'web'})as all connecti
ons to it are allowed
Generated 5 cases in 0.0551 seconds
FROM TO PORT
illuminatio-inverted-default:app=bookstore default:app=bookstore,role=api -*
illuminatio-inverted-default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -*
default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -*
default:app=bookstore default:app=bookstore,role=api *
*:* default:app=web *
Ensure that Pods of DaemonSet illuminatio-runner are ready
Finished running 5 tests in 13.4065 seconds
FROM TO PORT RESULT
illuminatio-inverted-default:app=bookstore default:app=bookstore,role=api -* success
illuminatio-inverted-default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -* success
default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -* success
default:app=bookstore default:app=bookstore,role=api * success
*:* default:app=web * success
Limit access to a Namespace
This policy will deny all traffic from other namespaces, limiting to just the current namespace. In other words, the secondary namespace allows connections internally, denying any from the default namespace in previous examples. Note how Illuimnatio tests all network policies, cluster-wide in all namespaces.
kubectl create namespace secondary
kubectl run --generator=run-pod/v1 web --namespace secondary --image=nginx \
--labels=app=web --expose --port 80
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
namespace: secondary
name: deny-from-other-namespaces
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}
$ kubectl apply -f deny-from-other-namespaces.yaml
networkpolicy "deny-from-other-namespaces" created"
$ kubectl get netpol -n secondary
NAME POD-SELECTOR AGE
deny-from-other-namespaces <none> 7s
$ kubectl get netpol -n default
NAME POD-SELECTOR AGE
api-allow app=bookstore,role=api 44m
web-allow-all app=web 24m
web-deny-all app=web 63m
$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Not generating negative tests for host ClusterHost(namespace=default, podLabels={'app': 'web'})as all connecti
ons to it are allowed
Generated 7 cases in 0.0621 seconds
FROM TO PORT
illuminatio-inverted-default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -*
default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -*
illuminatio-inverted-default:app=bookstore default:app=bookstore,role=api -*
illuminatio-inverted-secondary:* secondary:* -*
default:app=bookstore default:app=bookstore,role=api *
*:* default:app=web *
secondary:* secondary:* *
Ensure that Pods of DaemonSet illuminatio-runner are ready
Finished running 7 tests in 13.2361 seconds
FROM TO PORT RESULT
illuminatio-inverted-default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -* success
default:illuminatio-inverted-app=bookstore default:app=bookstore,role=api -* success
illuminatio-inverted-default:app=bookstore default:app=bookstore,role=api -* success
illuminatio-inverted-secondary:* secondary:* -* success
default:app=bookstore default:app=bookstore,role=api * success
*:* default:app=web * success
secondary:* secondary:* * success
Allow All Traffic from a certain Namespace
In this example, there are two namespaces, dev with purpose=testing and prod with purpose=production. The default namespace should allow connections from production but not dev. This is convenient for establishing policies along namespace boundaries. All previous network policies have been removed for this scenario.
kubectl run --generator=run-pod/v1 web --image=nginx \
--labels=app=web --expose --port 80
kubectl create namespace dev
kubectl label namespace/dev purpose=testing
kubectl create namespace prod
kubectl label namespace/prod purpose=production
The contents of the web-allow-prod.yaml file.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-allow-prod
spec:
podSelector:
matchLabels:
app: web
ingress:
- from:
- namespaceSelector:
matchLabels:
purpose: production
$ kubectl apply -f web-allow-prod.yaml
networkpolicy "web-allow-prod" created
$ kubectl get netpol -A
NAMESPACE NAME POD-SELECTOR AGE
default web-allow-prod app=web 44s
$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Generated 2 cases in 0.0645 seconds
FROM TO PORT
illuminatio-inverted-purpose=production:* default:app=web -*
purpose=production:* default:app=web *
Ensure that Pods of DaemonSet illuminatio-runner are ready
Finished running 2 tests in 7.1767 seconds
FROM TO PORT RESULT
illuminatio-inverted-purpose=production:* default:app=web -* success
purpose=production:* default:app=web * success
To view the results of the test programmatically, check the configmap for the illuminatio namespace, before performing another "clean" operation.
$ kubectl get cm -n illuminatio
NAME DATA AGE
illuminatio-cases-cfgmap 1 45s
illuminatio-runner-s87rw-results 2 40s
illuminatio-runner-z52gb-results 2 41s
$ kubectl get cm -n illuminatio illuminatio-runner-s87rw-results -o yaml
apiVersion: v1
data:
results: |
illuminatio-inverted-purposeproduction:illuminatio-dummy-nqtc7:
10.43.168.221:
'-80':
nmap-state: filtered
string: 'Test 10.43.168.221:-80 succeeded
Couldn''t reach 10.43.168.221 on port 80. Expected target to not be reachable'
success: true
prod:illuminatio-dummy-tc5v9:
10.43.168.221:
'80':
nmap-state: open
string: 'Test 10.43.168.221:80 succeeded
Could reach 10.43.168.221 on port 80. Expected target to be reachable'
success: true
runtimes: |
overall: error
tests:
illuminatio-inverted-purposeproduction:illuminatio-dummy-nqtc7:
10.43.168.221: 2.1185858249664307
prod:illuminatio-dummy-tc5v9:
10.43.168.221: 0.2853882312774658
kind: ConfigMap
metadata:
creationTimestamp: "2020-11-23T21:47:53Z"
labels:
illuminatio-cleanup: always
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:results: {}
f:runtimes: {}
f:metadata:
f:labels:
.: {}
f:illuminatio-cleanup: {}
manager: Swagger-Codegen
operation: Update
time: "2020-11-23T21:47:53Z"
name: illuminatio-runner-s87rw-results
namespace: illuminatio
resourceVersion: "906614"
selfLink: /api/v1/namespaces/illuminatio/configmaps/illuminatio-runner-s87rw-results
uid: 2c2f7434-d1ee-49c0-b77d-c11b7848f4da
Additional Information
Further Reading and Other Useful Links
- Securing Kubernetes Cluster Networking
- Example Network Policy Recipes, ahmetb/kubernetes-network-policy-recipes
- Inovex Illuminatio, Kubernetes Network Policy Validator
- Inovex/Illuminatio GitHub Project Page
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.