Tag Archives: Elasticsearch

This is the second part of the article series: Processing real-time data with Storm, Kafka, and ElasticSearch. 1. Introduction In the first part we described the problem and how we are going to solve it. To refresh your memory, the plan is to create a Data Reduction System of historic flight data (which you can freely download from here). We ...

This is an article of processing real-time data with Storm, Kafka and ElasticSearch. 1. Introduction How would you process a stream of real or near-real time data? In the era of Big Data, there are a number of technologies available that can help you in this task. In this series of articles we shall see a real example scenario and ...

Elasticsearch is great, but optimizing it for high load is always tricky. This won’t be yet another “Tips and tricks for optimizing Elasticsearch” article – there are many great ones out there. I’m going to focus on one narrow use-case – multitenant systems, i.e. those that support multiple customers/users (tenants). You can build a multitenant search engine in three different ...

So it happened that we had a production issue on the SaaS version of LogSentinel – our Elasticsearch stopped indexing new data. There was no data loss, as elasticsearch is just a secondary storage, but it caused some issues for our customers (they could not see the real-time data on their dashboards). Below is a post-mortem analysis – what happened, ...

The Elasticsearch engine Elasticsearch is one of the most widely search engines being used in a number of production deployments today. It is based on the Lucene search library and one of the key features it provides is a JSON-based query DSL on top of Lucene that provides an easier to use mechanism for interacting with the search engine. However ...

Choosing your indexing strategy is hard. The Elasticsearch documentation does have some general recommendations, and there are some tips from other companies, but it also depends on the particular usecase. In the typical scenario you have a database as the source of truth, and you have an index that makes things searchable. And you can have the following strategies: Index ...

 In this article we will see how we can  manage logs generated by APIGEE API gateway with Elastic Search, Logstash and Kibana. The below diagram highlights how log data flows through ELK stack for data visualization and monitoring. As part of API gateway we will use MessageLogging policy to capture API proxy logs at proxy flow, target flow and post ...

1. Introduction With the use of microservices, it has become easy to create stable distributed applications and get rid of many legacy problems. But the use of microservices has introduced a few challenges also and Distributed log management is one of them. As microservices are isolated hence they don’t share database and log files, so it becomes challenging to search, ...

Indonesian, or Bahasa Indonesia, is a very approachable language for westerners. It uses latin characters, there’s a clear structure, no tenses, no gender or plural forms and it contains many foreign words (as a German I especially enjoy the dutch influenced terms like knalpot for exhaust pipe). If you’re growing up outside of Asia Indonesia might be a quite distant ...