Tag Archives: Elasticsearch

So it happened that we had a production issue on the SaaS version of LogSentinel – our Elasticsearch stopped indexing new data. There was no data loss, as elasticsearch is just a secondary storage, but it caused some issues for our customers (they could not see the real-time data on their dashboards). Below is a post-mortem analysis – what happened, ...

The Elasticsearch engine Elasticsearch is one of the most widely search engines being used in a number of production deployments today. It is based on the Lucene search library and one of the key features it provides is a JSON-based query DSL on top of Lucene that provides an easier to use mechanism for interacting with the search engine. However ...

Choosing your indexing strategy is hard. The Elasticsearch documentation does have some general recommendations, and there are some tips from other companies, but it also depends on the particular usecase. In the typical scenario you have a database as the source of truth, and you have an index that makes things searchable. And you can have the following strategies: Index ...

 In this article we will see how we can  manage logs generated by APIGEE API gateway with Elastic Search, Logstash and Kibana. The below diagram highlights how log data flows through ELK stack for data visualization and monitoring. As part of API gateway we will use MessageLogging policy to capture API proxy logs at proxy flow, target flow and post ...

1. Introduction With the use of microservices, it has become easy to create stable distributed applications and get rid of many legacy problems. But the use of microservices has introduced a few challenges also and Distributed log management is one of them. As microservices are isolated hence they don’t share database and log files, so it becomes challenging to search, ...

Indonesian, or Bahasa Indonesia, is a very approachable language for westerners. It uses latin characters, there’s a clear structure, no tenses, no gender or plural forms and it contains many foreign words (as a German I especially enjoy the dutch influenced terms like knalpot for exhaust pipe). If you’re growing up outside of Asia Indonesia might be a quite distant ...

1. Introduction In this example, we shall demonstrate how to make use of Elasticsearch, a distributed free-text search and analysis database engine based on Apache Lucene with a simple maven-based Java client. We will be using the latest version of Elasticsearch, which is ES v6.1.2 while writing this post. For this example we use the following technologies: Maven 3 Java 8 Elasticsearch 6.1.2   ...

Using Elasticsearch 5, we had a field like drivers license number where values may include special characters and inconsistent upper/lower case behavior as the values were entered by the users with limited validation.  For example, these are hypothetical values: CA-123-456-789 WI.12345.6789 tx123456789 az-123-xyz-456 … In our application, the end user need to search by that field. We had a business ...

In the previous article we created a simple adapter from ElasticSearch’s API to Reactor’s Mono, that looks like this: import reactor.core.publisher.Mono; private Mono indexDoc(Doc doc) { //... } Now we would like to run this method at controlled concurrency level, millions of times. Basically, we want to see how our indexing code behaves under load, benchmark it. Fake data with jFairy First, we ...