Idan Fridman

About Idan Fridman

Idan is Software engineer with experience in Server side technologies. Idan is responsible for various infrastructure models in the software industry(Telecommunications, Finance).

My Exprience using ElasticSearch NodeJS and ExpressJS

General requirement

Part of a system I am currently working on is manipulating raw data. The data being sent from multiple client devices (Android/Iphone devices, etc..) ~ 100,000 requests per day.
The raw data needs to be collected and ready for later use. It gotta be searchable and aggregated for calculations.

I had little experience with Solr but this time I wanted to try ElasticSearch(ES). I read about ES out of the box clustering and shards capabilities so I gave it a shot.

 

ES

The installation is simple and quick. ES ships with comfortable UI dashboard(Marvel) and a command line tool to execute queries.

After playing with the framework I created my first Index and Mapping types (MyIndex, MyType).

I didn`t know at that point how exactly my data was going to be structured, so I went into “all-fields-querying’. Meaning that for a single input searching the ES engine will go through all documents and search in each document’s field. The matching documents will return.

We may achieve this by using the field _all. That field includes flatten text of one or more other fields within the document indexed. The field is populated by default (if enabled).

The query is pretty easy:

GET /_search
{
    "match": {
        "_all": "Android Galaxy S5 device"
    }
}

ES dashboard also ships with a tool called Sense. With this tool you can Index and execute queries straight into ES without using any client (It’s good for testing and developing purposes).

I suggest you going throw mapping, analyzing, boosting, aliasing,etc…

NodeJS

So after playing with ES I decided to write the server side layer to be as my ES facade.

My intention was to write simple client with a single search line that will send requests to the ES via the server layer and render the results to the screen.

ES is working with Json’s as its input and output. I thought about using javascript which working comfortably with json’s.

Nodejs is server side layer also using JS syntax. That’s why JavaScript makes Node a perfect language for interfacing with ElasticSearch. Dig it?

I grabbed the ES client implementation( elasticsearchclient) for NodeJS and after coding I completed the integration between node and ES. Here is a sample code from the Node side for querying the ES:

var index = "myindex";
var type = "mytype";
ElasticSearchClient = require('elasticsearchclient');
var Q = require('q');


var serverOptions = {
    host: 'localhost',
    port: 9200
};

var elasticSearchClient = new ElasticSearchClient(serverOptions);

function performSearch(termToSearch) {

    var deferred = Q.defer();
    console.log("Request handler 'search' was called.");
    var qryObj = {
        "query" : {
            "term" : { "name":termToSearch }
        }
    };
    elasticSearchClient.search(index, type, qryObj).
        on('data', function (data) {
            // console.log(data)
            deferred.resolve(JSON.parse(data));
        })
        .on('error', function (err) {
            console.log(err);
            return deferred.resolve(err);
        })
        .exec();
    return  deferred.promise;
}

exports.performSearch = performSearch;

NodeJS is single threaded that implemented with event looper. Therefore the “heavy” logic needs to be asynchronously executed. I am using Q in my code.
Q is a library for making and composing asynchronous Promises in JavaScript.

This way I can “free” the event looper and create the asynchronous requests to my ES engine and get clean callbacks through the code layers using Q.

After a while I found the need of having extensive API. I used ExpressJS for this purpose.

ExpressJS

I wanted MVC framework to provide a rest-api-like layer that will handle all API requests in my server side(NodjeJS).

I went to ExpressJS which appears to be the most popular web framework for Node.js among web developers (We as experienced open-source developers know that community matters!).

I am obsessed with project structure. I am coming from the Java world and sometimes javascript projects seems to me like a Jungle. I was looking for a popular and conventional way to structure my NodeJS project together with ExpressJS.

So after playing with the framework I found out that with Yeoman and generator-express you quickly generate project structure of Nodejs including all necessary expressJS libraries to get you going.

My project tree looked like this:

nodejstree

Example of ExpressJS controller:

var express = require('express');
var router = express.Router();
var esService = require('../services/esService');
var Q = require('q');



router.get("/search", function (req, res) {
    var termToSearch = req.query.termToSearch;
    console.log("termToSearch=" + termToSearch);
    Q(esService.performSearch(termToSearch)
    ).then(function (data) {
            res.send("Session: %j", data);
        });
})

The controller invoking the perfomSearch method using esService and returning the results back to the JS client asynchronously using promises.

This is the JS client code which rendering the results into HTML element:

* I used prettyPrint library to print the json result nicely on screen:

function handleClick() {
    var inputParam = document.querySelector("#myinput").value;
    $.get('http://localhost:3000/search?termToSearch=' + inputParam, function (responseText) {
        console.log(responseText);
        $("#resultlist").append(prettyPrint(responseText));
    });

    return false;
}

 
es

Idan.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

Leave a Reply


3 × = fifteen



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close