Home » Java » Enterprise Java » APIGEE API gateway log management with ELK (Elastic Search, Logstash and Kibana)

About Siva Janapati

Siva Janapati
Siva Prasad Janapati is a software engineer working in the e-Commerce domain. He has hands on experience on Java, JEE, Spring, Apache Solr, Node.js, JBoss and other open source/enterprise technologies. He loves to explore new technologies and trends.

APIGEE API gateway log management with ELK (Elastic Search, Logstash and Kibana)

 In this article we will see how we can  manage logs generated by APIGEE API gateway with Elastic Search, Logstash and Kibana. The below diagram highlights how log data flows through ELK stack for data visualization and monitoring.

APIGEE API gateway

As part of API gateway we will use MessageLogging policy to capture API proxy logs at proxy flow, target flow and post client flow based on the requirement. As of today, MessageLogging policy  supports syslog and file based logging(available for only on-premise installations). Below is MessageLogging policy configuration to send logs to syslog server.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="TestEnv-Message-Logging">
    <DisplayName>TestEnv Message Logging</DisplayName>
    <Syslog>
        <Message>{messageId}|{apiproxy.name}|{response.status.code}|{target.received.start.timestamp}|{target.received.end.timestamp}|{environment.name}|{client.received.start.timestamp}|{client.sent.end.timestamp}</Message>
        <Host>127.0.0.1</Host>
        <Port>514</Port>
    </Syslog>
</MessageLogging>

Now we will see logstash configuration to ingest data from syslog to logstash. Here I am not giving steps to setup ELK as that is out of scope.

input {
  file {
    path => "/Users/sjana/Documents/POC/ELK/APIGEE_LOG.log"
    start_position => "beginning"
   sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      separator => "|"
      columns => ["MessageId","API_PROXY_NAME","RESPONSE_STATUS_CODE","TARGET_RECEIVED_START_TIME","TARGET_RECEIVED_END_TIME","ENVIRONMENT_NAME","CLIENT_RECEIVED_START_TIME","CLEINT_SENT_END_TIME"]
      remove_field => "message"
  }
}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "apiproxies-log"
  }
  stdout {}
}

Once syslog data is ingested successfully to Logstash we can configure visualization based on available fields in index. Below is sample visualization created for API proxy response status codes.

APIGEE API gateway

In coming articles we will discuss another topic. Till then, Spread love for APIs!!!

Published on Java Code Geeks with permission by Siva Janapati, partner at our JCG program. See the original article here: APIGEE API gateway log management with ELK(Elastic Search, Logstash and Kibana)

Opinions expressed by Java Code Geeks contributors are their own.

(0 rating, 0 votes)
You need to be a registered member to rate this.
Start the discussion Views Tweet it!
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
6. Spring Interview Questions
7. Android UI Design
and many more ....
I agree to the Terms and Privacy Policy

Leave a Reply

avatar

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
Notify of