Kotlin

Service to Service call patterns – GKE with Anthos Service Mesh on a single cluster

 This is second in a series of posts exploring service to service call patterns in some of the application runtimes on Google Cloud. The first in the series explored service to service call patterns in GKE

This post will expand on it by adding in a Service Mesh, specifically Anthos Service Mesh, and explore how the service to service patterns change in the presence of a mesh. The service to service call with be across services in a single cluster. The next post will explore services deployed to multiple GKE clusters.

Set-Up

The steps to set-up a GKE cluster and install Anthos service mesh on top of it is described in this document – https://cloud.google.com/service-mesh/docs/unified-install/install, in brief these are the commands that I had to run in my GCP Project to get a cluster running:

export PROJECT=$(gcloud config get-value project)
export ZONE="us-west1-a"

# Create a GKE Standard cluster named "solo"
gcloud container clusters create solo \
    --project=${PROJECT} \
    --zone=${ZONE} \
    --enable-ip-alias \
    --machine-type=e2-standard-4 \
    --num-nodes=2 \
    --workload-pool=${PROJECT}.svc.id.goog

# Download the asmcli utility
curl https://storage.googleapis.com/csm-artifacts/asm/asmcli_1.11 > asmcli

# Install service mesh on the newly created "solo" cluster
./asmcli install \
  --project_id ${PROJECT} \
  --cluster_name solo \
  --cluster_location us-west1-a \
  --fleet_id ${PROJECT} \
  --enable_all  \
  --ca mesh_ca


# Determine the ASM revision
ASM_REVISION=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')

# Install an ingress gateway that can work with the service mesh
kubectl create namespace gw-namespace

kubectl label namespace gw-namespace \
  istio.io/rev=${ASM_REVISION} --overwrite

git clone https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git

kubectl apply -n gw-namespace \
  -f anthos-service-mesh-packages/samples/gateways/istio-ingressgateway

# Create a namespace to host applications with mesh proxy injected in
kubectl create namespace istio-apps
kubectl label namespace istio-apps istio-injection- istio.io/rev=${ASM_REVISION} --overwrite

If the installation of cluster and the mesh has run through cleanly, a good way to verify the installation is to see if the cluster gets registered as a Anthos managed cluster in the Google Cloud Console.

The services that I will be installing is fairly simple and looks like this:

Using a UI, the caller can make the producer behave in certain ways:

  • Introduce response time delays
  • Respond with certain status codes

This will help check how the mesh environment will behave in the face of these behaviors.

The codebase for the “caller” and “producer” are in
this repository – https://github.com/bijukunjummen/sample-service-to-service, there are kubernetes manifests available in the repository to bring up these services.

Behavior 1 – Mutual TLS

The first behavior that I want to see is for the the caller and the producer to verify each others identities by presenting and validating their certificates.

This can be done by adding in a istio DestinationRule for the producer, along these lines:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sample-producer-dl
  namespace: istio-apps
spec:
  host: sample-producer.istio-apps.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL      
---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sample-caller-dl
  namespace: istio-apps
spec:
  host: sample-caller.istio-apps.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

This also adds in the DestinationRule for the caller, this is because the caller gets the call from the browser via an
Ingress Gateway and even this call needs to be authenticated using mtls

Alright now that the set-up in place, the following is what gets captured as the request flows from the Browser to the Ingress Gateway to the Caller to the Producer.

The sign that the mTLS works is seeing the “x-forwarded-client-cert” header, this is in both the Callers headers coming in from Ingress-gateway, and in the “Producers” headers coming in from the Caller.

Behavior 2 – Timeout

The second behavior that I want to explore is the timeouts. A request timeout can be set for the call from the Caller to Producer by creating a Virtual Service for the Producer with the value set, along these lines:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: sample-producer-route
  namespace: istio-apps
spec:
  hosts:
    - "sample-producer.istio-apps.svc.cluster.local"
  http:
    - timeout: 5s
      route:
        - destination:
            host: sample-producer
            port:
              number: 8080

With this configuration in place a request from the caller with a delay of 6 seconds, causes the Mesh to timeout and present an error that looks like this:

The mesh responds with a http status code of 504 with a message of “Upstream timed out”. 

Behavior 3 – Circuit Breaker

Circuit breaker is implemented using a Destination Rule resource.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sample-producer-dl
  namespace: istio-apps
spec:
  host: sample-producer.istio-apps.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    connectionPool:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 15s
      baseEjectionTime: 15s

Here I have configuration which breaks the circuit if 3 continuous 5XX responses are received from the Producer in a 15 second interval, and then does not make a request for another 15 seconds

With this configuration in place a request with broken circuit looks like this:

The mesh responds with a http status code of 503 and a message of “no healthy upstream”

Conclusion

The neat thing is that in all scenarios so far, the way the Caller calls the Producer remains exactly the same, it is the mesh which injects in the appropriate security controls through mTLS and the resilience of calling service through timeouts and circuit breaker. 

Published on Java Code Geeks with permission by Biju Kunjummen, partner at our JCG program. See the original article here: Service to Service call patterns – GKE with Anthos Service Mesh on a single cluster

Opinions expressed by Java Code Geeks contributors are their own.

Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button