Tag Archives: Big Data

This is an article of processing real-time data with Storm, Kafka and ElasticSearch. 1. Introduction How would you process a stream of real or near-real time data? In the era of Big Data, there are a number of technologies available that can help you in this task. In this series of articles we shall see a real example scenario and ...

The big data challenge The concept of big data is understood differently in the variety of domains where companies face the need to deal with increasing volumes of data. In most of these scenarios the system under consideration needs to be designed in such a way so that it is capable of processing that data without sacrificing throughput as data ...

Back in ancient history (2004) Google’s Jeff Dean & Sanjay Ghemawat presented their innovative idea for dealing with huge data sets – a novel idea called MapReduce Jeff and Sanjay presented that a typical cluster was made of 100s to few 1000s of machines with 2 CPUs and 2-4 GB RAM each. They presented that in the whole of Aug ...

With the release of latest influxdb-java driver version came along the InfluxbMapper. To get started we need to spin up an influxdb instance, and docker is the easiest way to do so. We just follow the steps as described here. Now we have a database with some data and we are ready to execute our queries. We have the measure ...

IT and big data projects can be complex and are often ambiguous by nature. In the case of big data projects, hypotheses about the information contained within the data are either proved or rejected, with new hypotheses then arising. Similarly, assumptions about project requirements are often built into IT projects and stacked on top of one another in advance. Yet, ...

Why would a data scientist use Kafka Jupyter Python KSQL TensorFlow all together in a single notebook? There is an impedance mismatch between model development using Python and its Machine Learning tool stack and a scalable, reliable data platform. The former is what you need for quick and easy prototyping to build analytic models. The latter is what you need ...

As continuation of big data query system blog, i want to share more techniques for building Analytics engine. Take a problem where you have to build system that will be used for analyzing customer data at scale.           What options are available to solve this problem ? – Load the data in your favorite database and have ...

In-memory approaches can achieve blazing speed by putting the working set of the data into the system memory. When all data is kept in memory, the need to deal with issues arising from the use of traditional spinning disks disappears. This means, for instance, there is no need to maintain additional cache copies of data and manage synchronization between them. ...

Editor’s Note: At Strata+Hadoop World 2016 in New York, MapR Director of Enterprise Strategy & Architecture Jim Scott gave a presentation on “Handling the Extremes: Scaling and Streaming in Finance.” As Jim explains, agility is king in the world of finance, and a message-driven architecture is a mechanism for building and managing discrete business functionality to enable agility. In order to ...