How to Use Istio With Kong Ingress Controller

, , , ,

Transforming software from monolith to micro services is becoming a trend in the past few years. Covid-19 has undoubtedly pushed this transformation further. Previously I had talked a lot about using Kong as the API gateway to secure, transform and monitor client requests to upstream services. What about traffics between internal services? The answer is using Service Mesh.

According to this article on solo.io, Istio has the most market share at the moment. In today’s post, I would show you how to use Kong Ingress Controller with Istio including demos of enabling mTLS, adding AuthorizationPolicy, change traffic distribution using VirtualService and DestinationRule.

Let’s get started.

Prerequisites:

  • Istioctl: If you haven’t installed it, you can follow the official doc.
  • Helm: I need to use helm to install Kong. Helm version is 3.7.1.
  • Your cluster need to be able to provision LoadBalancer. I am using metalLB locally.

Install Istio

As suggested by official documentation, I am going to install istio with minimal configuration profile first. Then I will install Kong Ingress Controller or Istio Ingress Gateway to allow requests entering the mesh.

1
2
3
istioctl install --set profile=minimal \
  --set meshConfig.accessLogFile=/dev/stdout \
  --skip-confirmation

Install BookInfo

We will use istio’s bookinfo in our demo.

1
2
3
4
5
6
# Create namespace bookinfo
kubectl create namespace bookinfo
# Add lable to namespace for auto sidecar injection.
kubectl label namespace bookinfo istio-injection=enabled
# Install bookinfo app
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo

Next I will show you some basic usage of Istio Ingress Gateway so we can compare it with Kong Ingress Controller. If you are only interested in testing Kong Ingress controller, you can go to the next section.

(Optional) Istio Ingress Gateway

Installation

  1. Add helm chart

    1
    2
    helm repo add istio https://istio-release.storage.googleapis.com/charts
    helm repo update

  2. Install Istio Ingress Gateway

    1
    2
    kubectl create namespace istio-ingress
    helm install istio-ingress istio/gateway -n istio-ingress --wait

  3. Once the pod is ready, let’s save load balancer IP to ISTIO_INGRESS_GATEWAY. 

    1
    export ISTIO_INGRESS_GATEWAY=$(kubectl -n istio-ingress get svc istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Let’s curl $ISTIO_INGRESS_GATEWAY

1
curl -s http://$ISTIO_INGRESS_GATEWAY -iv

We would get Connection refused error because istio ingress gateway requires Gateway resource to receive incoming HTTP/TCP connections.

1
2
3
4
5
*   Trying 172.18.18.150:80...
* connect to 172.18.18.150 port 80 failed: Connection refused
* Failed to connect to 172.18.18.150 port 80 after 0 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 172.18.18.150 port 80 after 0 ms: Connection refused

Create Gateway

Let’s create a Gateway resource.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

Then we can try sending the request again and we should be able to connect

1
curl http://$ISTIO_INGRESS_GATEWAY -iv

Create VirtualService

Next step we will use VirtualService to route requests to service productpage at port 9080.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: bookinfo 
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
EOF

Now we can access the product page.

1
curl http://$ISTIO_INGRESS_GATEWAY/productpage

Enable JWT auth

We can also enable JWT authentication on ingress gateway by applying below RequestAuthentication and AuthorizationPolicy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl apply -f - << EOF
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-authenticator
  namespace: istio-ingress
spec:
  selector:
    matchLabels:
      istio: ingress
  jwtRules:
  - issuer: istio-testing@aufomm
    jwksUri: https://raw.githubusercontent.com/istio/istio/release-1.11/security/tools/jwt/samples/jwks.json
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: istio-ingress
spec:
  selector:
    matchLabels:
      istio: ingress
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]
EOF

If we access curl http://$ISTIO_INGRESS_GATEWAY/productpage -i again, we will get 403 Forbidden.

1
2
3
4
5
6
7
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Fri, 26 Nov 2021 02:16:18 GMT
server: istio-envoy

RBAC: access denied

To access productpage again we need to pass a JWT token with authorization bearer header in our request.

1
curl http://$ISTIO_INGRESS_GATEWAY/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ"

Let’s move on and try Kong Ingress Controller.

Install Kong

I will install Kong in dbless mode in namespace kong and auto inject istio sidecar proxy.

1
2
3
4
5
6
7
8
9
10
11
12
# Add kong helm chart repo
helm repo add kong https://charts.konghq.com
# Update repo
helm repo update
# Create Namespace Kong
kubectl create namespace kong
# Label namespace to allow istio sidecar injection
kubectl label namespace kong istio-injection=enabled
# Install Kong with helm
helm upgrade -i my-kong kong/kong -n kong \
  --set image.tag=2.6 \
  --set ingressController.installCRDs=false --wait

Once the Pod is running, we can save kong proxy ip to KONG_PROXY_IP.

1
export KONG_PROXY_IP=$(kubectl -n kong get svc my-kong-kong-proxy -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Use Kong Ingress Controller

As I mentioned in my previous post, Kong ingress controller use CRDs and annotations to extend native kubernetes objects.

Create Ingress

Let’s start by creating an Ingress object. This ingress object has the same routes as the VirtualSerivce above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kong-istio-ingress
  namespace: bookinfo
spec:
  ingressClassName: kong
  rules:
  - http:
      paths:
      - path: /(productpage|login|logout)
        pathType: Exact
        backend:
          service:
            name: productpage
            port:
              number: 9080
  - http:
      paths:
      - path: /(|static|api/v1/products).*
        pathType: Prefix
        backend:
          service:
            name: productpage
            port:
              number: 9080
EOF

Now we can access /productpage via Kong.

1
curl $KONG_PROXY_IP/productpage

Enable JWT plugin

We will then use JWT plugin to restrict access. I am applying the plugin globally.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
kubectl apply -f - << EOF
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: jwt-user
  namespace: bookinfo
  annotations:
    kubernetes.io/ingress.class: "kong"
username: jwt-user
credentials:
  - jwt-key-rs256
---
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: jwt-auth
  namespace: bookinfo
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true" 
plugin: jwt
---
apiVersion: v1
kind: Secret
metadata:
  name: jwt-key-rs256
  namespace: bookinfo
type: Opaque
stringData:
  kongCredType: jwt
  key: 'istio-testing@aufomm'
  algorithm: RS256
  rsa_public_key: |
    -----BEGIN PUBLIC KEY-----
    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxAE7eB6qugXyCAG3yhh7
    pkDkT65pHymX+P7KfIupjf59vsdo91bSP9C8H07pSAGQO1MV/xFj9VswgsCg4R6o
    tmg5PV2He95lZdHtOcU5DXIg/pbhLdKXbi66GlVeK6ABZOUW3WYtnNHD+91gVuoe
    JT/DwtGGcp4ignkgXfkiEm4sw+4sfb4qdt5oLbyVpmW6x9cfa7vs2WTfURiCrBoU
    qgBo/+4WTiULmmHSGZHOjzwa8WtrtOQGsAFjIbno85jp6MnGGGZPYZbDAa/b3y5u
    +YpW7ypZrvD8BgtKVjgtQgZhLAGezMt0ua3DRrWnKqTZ0BJ/EyxOGuHJrLsn00fn
    MQIDAQAB
    -----END PUBLIC KEY-----
EOF

Now when we access /productpage route again,

1
curl $KONG_PROXY_IP/productpage -i

we should get 401 Unauthorized error.

1
2
3
4
5
6
7
8
9
10
HTTP/1.1 401 Unauthorized
date: Fri, 26 Nov 2021 03:35:48 GMT
content-type: application/json; charset=utf-8
content-length: 26
x-kong-response-latency: 0
server: istio-envoy
x-envoy-upstream-service-time: 0
x-envoy-decorator-operation: my-kong-kong-proxy.kong.svc.cluster.local:80/*

{"message":"Unauthorized"}

If we pass in an authorization bearer header, we are allowed to access the service again.

1
curl -s http://$KONG_PROXY_IP/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ"

Next we will explore some istio features and see how it works with Kong ingress controller.

Use Istio features

Monitor Mesh

Monitoring might be one of the most important features that you use service mesh for. I will use official sample yaml file to install kiali dashboard and prometheus and jaeger will be used to get some data from mesh

1
2
3
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/jaeger.yaml

Once pods is ready, we can access dashboard with istioctl dashboard kiali.

Let’s go to Graph and then let’s tick Security in the Display dropdown. Please make sure we are checking Namespace bookinfo. After that, let’s send some requests.

1
2
3
4
5
while true
do
curl -s http://$KONG_PROXY_IP/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ" > /dev/null
sleep 0.5
done

We should see below traffic flow.

As we can see not all traffics are encrypted.

Enable mTLS

Zero Trust architecture requires ALL traffics to be encrypted and verified. We can enable mTLS easily with istio to achieve this goal. What we need to do is to apply below PeerAuthentication in root namespace istio-system. This will enable strict mTLS in the mesh.

1
2
3
4
5
6
7
8
9
10
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF

If we visit curl $KONG_PROXY_IP/productpage now we will get curl: (56) Recv failure: Connection reset by peer error because the sidecar in Kong pod is expecting a client certificate to be passed in with the request. As Kong ingress controller is our entry point, we want to allow requests coming in. What we need to do is to disable PeerAuthentication in kong namespace so the traffic between Kong and Bookinfo remains encrypted and the client requests enter the mesh via Kong.

1
2
3
4
5
6
7
8
9
10
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: kong
spec:
  mtls:
    mode: DISABLE
EOF

Once that’s added, let’s try curl $KONG_PROXY_IP/productpage again. This time you would get upstream connect error or disconnect/reset before headers. reset reason: connection termination error. To fix this issue we need to annotate productpage service and our ingress.

1
2
kubectl -n bookinfo annotate service productpage ingress.kubernetes.io/service-upstream=true
kubectl -n bookinfo annotate ingress kong-istio-ingress konghq.com/preserve-host=false

The first annotation ingress.kubernetes.io/service-upstream=true allows Kong to send request to this service ip address instead of the pod IP.

The second annotation konghq.com/preserve-host=false helps us to NOT preserve request host header. The reason is that we want productpage to get the request from sidecar.

After applying annotations we can send our request again and we should see below in graph. As we can see ALL traffic are secured now.

Split Traffics

Kong can be used to load balance client requests to different backends. However when the backend services are not directly connected to Kong, there is not much Kong can do. For example, in our demo Kong has no control over which review pod productpage send requests to. In order to route requests as we want them to, we can use VirtualService and DistinationRule to fine tune traffic inside mesh.

Let me give you an example. Let’s continue sending our request to /productpage.

1
2
3
4
5
while true
do
curl -s http://$KONG_PROXY_IP/productpage -H "Authorization:Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJpc3Rpby10ZXN0aW5nQGF1Zm9tbSIsInN1YiI6ImlzdGlvLXRlc3RpbmdAYXVmb21tIiwidGVzdCI6ImlzdGlvIn0.bkWG-fESbgVUcWBSfa-i4q7ZNY_gk3LO18CDcjz07w74JaoDvSIn347LQrm5pF8r6Hb1lmhlC6p7y4iCDNeWPZ6azqpql4MPEAdM2sZvZphriJFj8XCuKsW4Qn2yOohsPm4QsRDUKxME8wMswPqfQo__0DSUgo7bv56HsBTrb7mAKoLw4qiv_MRjU9X_FwP75JgNh64rGUfKnSU1jWLgwlQH7_jcxHdSK57QvvFcjxgfSBAJ62vOmQld9goVQn8Q7M-cQmrgs3SDeuV0XhNjY3Ifp-jjx4PBphu5UPIZNyHWZ3nLM4GN518-DUbsYGYs1KbrLQZmlMk6oDSAyIdszQ" > /dev/null
sleep 0.1
done

Then go to Kiali Dashboard -> Graph and then select Traffic Distribution under Display dropdown. Because the default load balancing algorithm is ROUND_ROBIN, we can see requests are almost equally distributed amoung 3 subsets.

Let’s apply below VirtualService and DestinationRule. These two objects split the traffic to different review subsets based on its assigned weight. In the example I want 10% of traffic goes to v1, 60% goes to v2 and 30% goes to v3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: slipt-reviews
  namespace: bookinfo
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 10
    - destination:
        host: reviews
        subset: v2
      weight: 60
    - destination:
        host: reviews
        subset: v3
      weight: 30
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-subsets
  namespace: bookinfo
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
EOF

After a while we should see the traffic distrubution changes.

That’s all I want to show you today. This is just the tip of the iceberg, istio have a ton of more features that you can explore.

Thank you for reading, see you next time.