K8s Ingress with NSX Advanced Load Balancer
Abbreviations used in this article:
- NSX Advanced Load Balancer = NSX-ALB
- K8s = Kubernetes (8 letters between the K and s in Kubernetes)
- SSL = Secure Sockets Layer
- AKO = Avi Kubernetes Operator (AVI now a VMware product called NSX Advanced Load Balancer)
In one of my previous posts I wrote about how to install and configure AKO (Avi Kubernetes Operator) to use as Service type LoadBalancer.
This post will try to cover the basics of how to use NSX Advanced LoadBalancer by using AKO to handle our Ingress requests (ingress-controller).
For more information on Ingress in Kubernetes
An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.
Ingress Load Balancer Kubernetes Definition
Within Kubernetes or K8s, a collection of routing rules that control how Kubernetes cluster services are accessed by external users is called ingress. Managing ingress in Kubernetes can take one of several approaches.
An application can be exposed to external users via a Kubernetes ingress resource; a Kubernetes NodePort service which exposes the application on a port across each node; or using an ingress load balancer for Kubernetes that points to a service in your cluster.
An external load balancer routes external traffic to a Kubernetes service in your cluster and is associated with a specific IP address. Its precise implementation is controlled by which service types the cloud provider supports. Kubernetes deployments on bare metal may require custom load balancer implementations.
However, properly supported ingress load balancing for Kubernetes is the simplest, more secure way to route traffic. link
Getting AKO ready
While this post assumes AKO is already in place and working in your k8s clusters I will get straight to the parts that involve Ingress. If not head over here to read the official docs how to install Avi Kubernetes Operator (AKO). To verify that you AKO is ready to handle Ingress request, type in this and notice the output:
1$ kubectl get ingressclasses.networking.k8s.io 2NAME CONTROLLER PARAMETERS AGE 3avi-lb ako.vmware.com/avi-lb <none> 50d
Default secret for TLS Ingress
As AKO expects all ingresses with TLS termination to have a key and certificate specified, there is a couple of ways this can be done. We can go with a "pr service", meaning a dedicated set of keys and certs pr service or a default/common set of keys and certificates that AKO can use if nothing else is specified. To apply the common approach, one common key and certificate for one or more applications we need to add a secret for the AKO. Prepare your router-default.yaml definition file like this (the official docs wants you to put in your cert as is, that does not work so you need to base64 encode both keys and certs and paste in below):
1apiVersion: v1 2kind: Secret 3metadata: 4 name: router-certs-default 5 namespace: avi-system 6type: kubernetes.io/tls 7data: 8 tls.key: "base64 encoded" 9 tls.crt: "base64 encoded" 10 alt.key: "base64 encoded" 11 alt.crt: "base64 encoded" 12
To base64 encode your keys and certs this can be done like this:
If you have the keys and certs in a file, from whatever linux terminal type in:
1cat cert.pem | base64 -w 0 2cat key.pem | base64 -w 0
Then paste into the above yaml accordingly (tls.key:key, tls.crt:crt)
If you have both ECDSA and RSA certs use the alt.key and alt.crt to apply both.
As soon as everything is pasted, apply the yaml file
kubectl apply -f router-defaults.yaml
Apply your ingress service
To create an ingress service you need to define this in yaml. An example below:
1apiVersion: networking.k8s.io/v1 2kind: Ingress 3metadata: 4 name: "NameOfIngress-Service" 5 namespace: "Namespaceofwhereyour-service-resides" 6 labels: 7 app: "ifyouwant" 8 annotations: 9 ako.vmware.com/enable-tls: "true" #"Indicates to Avi that you want to use TLS" 10spec: 11 ingressClassName: avi-lb #"The default class name for AVI" 12 rules: 13 - host: "FQDN" 14 http: 15 paths: 16 - pathType: Prefix 17 path: / 18 backend: 19 service: 20 name: "which-service-to-point-to" 21 port: 22 number: 80
Hostrules and HTTPrules
As I mentioned earlier, you can also define specific rules pr server such as certificates. Here we can use Hostrules and HTTPrules to further adjust granular settings pr service. One Hostrule example below:
1apiVersion: ako.vmware.com/v1alpha1 2kind: HostRule 3metadata: 4 name: name-of-your-rule 5 namespace: namespace-of-your-service 6spec: 7 virtualhost: 8 fqdn: must-match-hostname-above # mandatory 9 fqdnType: Exact 10 enableVirtualHost: true 11 tls: # optional 12 sslKeyCertificate: 13 name: "name-of-certificate" # This must be already defined in your AVI controller 14 type: ref 15 alternateCertificate: 16 name: "name-of-alternate-cert" # This must be already defined in your AVI controller 17 type: ref 18 sslProfile: System-Standard-PFS 19 termination: edge
To get all features available head over to the official docs site here
Hostrule example from the official docs:
1apiVersion: ako.vmware.com/v1alpha1 2kind: HostRule 3metadata: 4 name: my-host-rule 5 namespace: red 6spec: 7 virtualhost: 8 fqdn: foo.region1.com # mandatory 9 fqdnType: Exact 10 enableVirtualHost: true 11 tls: # optional 12 sslKeyCertificate: 13 name: avi-ssl-key-cert 14 type: ref 15 alternateCertificate: 16 name: avi-ssl-key-cert2 17 type: ref 18 sslProfile: avi-ssl-profile 19 termination: edge 20 gslb: 21 fqdn: foo.com 22 includeAliases: false 23 httpPolicy: 24 policySets: 25 - avi-secure-policy-ref 26 overwrite: false 27 datascripts: 28 - avi-datascript-redirect-app1 29 wafPolicy: avi-waf-policy 30 applicationProfile: avi-app-ref 31 analyticsProfile: avi-analytics-ref 32 errorPageProfile: avi-errorpage-ref 33 analyticsPolicy: # optional 34 fullClientLogs: 35 enabled: true 36 throttle: HIGH 37 logAllHeaders: true 38 tcpSettings: 39 listeners: 40 - port: 8081 41 - port: 6443 42 enableSSL: true 43 loadBalancerIP: 10.10.10.1 44 aliases: # optional 45 - bar.com 46 - baz.com
Httprule example from the official docs:
1apiVersion: ako.vmware.com/v1alpha1 2kind: HTTPRule 3metadata: 4 name: my-http-rule 5 namespace: purple-l7 6spec: 7 fqdn: foo.avi.internal 8 paths: 9 - target: /foo 10 healthMonitors: 11 - my-health-monitor-1 12 - my-health-monitor-2 13 loadBalancerPolicy: 14 algorithm: LB_ALGORITHM_CONSISTENT_HASH 15 hash: LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS 16 tls: ## This is a re-encrypt to pool 17 type: reencrypt # Mandatory [re-encrypt] 18 sslProfile: avi-ssl-profile 19 destinationCA: |- 20 -----BEGIN CERTIFICATE----- 21 [...] 22 -----END CERTIFICATE-----