Software Development

Service to Service call patterns – Multi-cluster using Anthos Service Mesh

This is third blog post as part of a series exploring service to service call patterns in different application runtimes in Google Cloud.

The first post explored Service to Service call pattern in a GKE runtime using a Kubernetes Service abstraction

The second post explored Service to Service call pattern in a GKE runtime with Anthos Service mesh

This post will explore the call pattern across multiple GKE runtimes with Anthos Service mesh providing a simple way for secure calls to be made across clusters.

Target Architecture

A target architecture that I am aiming for is the following:

Here two different applications are hosted on two separate Kubernetes clusters in different availability zones and the Service(called “Caller”) in one cluster invokes the Service(called “Producer”) in another cluster.

Creating the Cluster and configuring Anthos Service Mesh

I have hosted a gist here with the steps to bring up:

  • 2 GKE clusters in us-west1-a and us-central1-a zones
  • Install Anthos service mesh on each of the clusters
  • Register the clusters to be centrally managed

Services Installation

Assuming that the 2 GKE clusters are now available, the first cluster holds the Caller and an Ingress Gateway to enable the UI of the caller to be accessible to the user. This is through a deployment descriptor which looks something like this for the caller:

apiVersion: apps/v1
kind: Deployment
  name: sample-caller-v1
    app: sample-caller
    version: v1
  replicas: 1
      app: sample-caller
      version: v1
        app: sample-caller
        version: v1
      serviceAccountName: sample-caller-sa
        - name: sample-caller
          imagePullPolicy: IfNotPresent
            - containerPort: 8080
            runAsUser: 1000
              memory: "256Mi"
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 3
              path: /actuator/health/readiness
              port: 8080

I have reproduced the entire yaml just for demonstration, there is nothing that should stand out in the file.

Along the same lines the Producer application is deployed to the second cluster.

Caller to Producer call – Hacky approach

Now that the 2 services are in place, how do they call each other. Let’s start with “Caller” attempting to reach “Producer” using the host name of “sample-producer”(sample-producer happens to the name of the Kubernetes service that producer application is installed with, in the second cluster). This fails with a message which looks like this:

This is reasonable as the service “sample-producer” does not exist in cluster 1 and exists only in cluster 2.

The next thought that I have is check the behavior if I were to artificially add in a Kubernetes service called “sample-producer”

apiVersion: v1
kind: Service
  name: sample-producer
    app: sample-producer
    service: sample-producer
    - port: 8080
      name: http
    app: sample-producer

Surprisingly adding such a service in “cluster 1” does work and sample-producer does resolve cleanly, even though the pods running “Producer” in cluster 2! Additionally adding in a virtual service and destination rule for the call from “Caller” to “Producer”, the timeout and circuit breaker works cleanly also:

kind: DestinationRule
  name: sample-producer-dl
  namespace: istio-apps
  host: sample-producer.istio-apps.svc.cluster.local
      mode: ISTIO_MUTUAL
      consecutive5xxErrors: 3
      interval: 15s
      baseEjectionTime: 15s
kind: VirtualService
  name: sample-producer-route
  namespace: istio-apps
    - "sample-producer.istio-apps.svc.cluster.local"
    - timeout: 5s
        - destination:
            host: sample-producer
              number: 8080

Why this works is still a mystery to me. Let me now get to the right way of making this work.

Caller to Producer call – Right Approach

The right approach to getting service to service call working across a cluster is to use a feature of Anthos called Multi-cluster service and is described in detail in this blog post and this how to post.

The short of it is that if a “ServiceExport” resource is defined in cluster 2 and if the same namespace exists in Cluster 1 then the Service is resolved using a host name of the form “service-name.namespace.svc.clusterset.local” and in my case this maps to “sample-producer.istio-apps.svc.clusterset.local”!. The ServiceExport resource looks something like this:

kind: ServiceExport
  namespace: istio-apps
  name: sample-producer

This is the only change that I have to make to the caller, instead of calling Producer using “sample-producer”, now it uses the host name of “sample-producer.istio-apps.svc.clusterset.local” and everything resolves cleanly and the call continues to work across the cluster.

View from the caller:

View from the Producer:


I hope this clarifies to some extent how service to service call can be enable across multiple clusters, even across regions. Anthos service mesh handles all the underlying mechanics of getting this to work and from a callers perspective all that changes is the hostname to invoke the service with. 

There are a few small catches, for eg, to get the Mutual TLS to work across clusters specific resources had to be created which I will leave as an exercise or please do reach out if it feels difficult to handle.

Published on Java Code Geeks with permission by Biju Kunjummen, partner at our JCG program. See the original article here: Service to Service call patterns – Multi-cluster using Anthos Service Mesh

Opinions expressed by Java Code Geeks contributors are their own.

Want to know how to develop your skillset to become a Java Rockstar?

Join our newsletter to start rocking!

To get you started we give you our best selling eBooks for FREE!


1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design


and many more ....


Receive Java & Developer job alerts in your Area

I have read and agree to the terms & conditions


Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Inline Feedbacks
View all comments
Back to top button