Home » Author Archives: Joe Stein

Author Archives: Joe Stein

Apache Hadoop HDFS Data Node Apache Mesos Framework

apache-hadoop-logo

Intro This project allows running HDFS on Mesos. You should be familiar with HDFS and Mesos basics: http://mesos.apache.org/documentation/latest/ https://hadoop.apache.org/docs/r2.7.2/hdfs_design.html         Project requires: Mesos 0.23.0+ JDK 1.7.x Hadoop 1.2.x or 2.7.x Mesos in Vagrant Project includes vagrant environment, that allows to run Mesos cluster locally. If you are going to use external Mesos cluster, you can skip this section. 1. ...

Read More »

Getting Started with Heron on Apache Mesos and Apache Kafka

software-development-2-logo

Heron has been Open Sourced, woo! Heron is Twitter’s  distributed stream computation system for running Apache Storm compatible topologies in production. A Heron topology is a directed acyclic graph used to process streams of data. Heron topologies consist of three basic components: spouts and bolts, which are connected via streams of tuples. Below is a visual illustration of a simple topology: Spouts are ...

Read More »

Open Source Cloud Formation with Minotaur for Mesos, Kafka and Hadoop

apache-hadoop-logo

Today I am happy to announce “Minotaur” which is our Open Source AWS based infrastructure for managing big data open source projects including (but not limited too): Apache Kafka, Apache Mesos and Cloudera’s Distribution of Hadoop. Minotaur is based on AWS Cloud Formation. The following labs are currently supported:           Apache Mesos Apache Kafka Apache Zookeeper Cloudera Hadoop ...

Read More »

Resource scheduling and task launching with Apache Mesos and Apache Aurora at Twitter

java-interview-questions-answers

Episode # 23 of the podcast was a talk with Bill Farner Bill explained how Twitter, using Apache Mesos and Apache Aurora, gets more for their money for the hardware and saves engineering time (both development and operations) by utilizing fine grained resources scheduling across their infrastructure. Bill talked a bit how the power of what he saw and experienced at Google with Borg ...

Read More »

Reporting Metrics to Apache Kafka and Monitoring with Consumers

software-development-2-logo

Apache Kafka has been used for some time now by organizations to consume not only all of the data within its infrastructure from an application perspective but also the server statistics of the running applications and infrastructure.  Apache Kafka is great for this. Coda Hale’s metrics’s has become a leading way to instrument your JVM applications capturing what the application is doing ...

Read More »

XML to Avro Conversion

java-interview-questions-answers

We all know what XML is right?  Just in case not, no problem here is what it is all about.                   <root> <node>5</node> </root> Now, what the computer really needs is the number five and some context around it. In XML you (human and computer) can see how it represents context to ...

Read More »

Getting started with Apache Mesos and Apache Aurora using Vagrant

vagrant-logo

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. Think of it as the “kernel” for your data center. Paco Nathan talked about this on one of the All Things Hadoop Podcasts. Features:             Fault-tolerant replicated master using ZooKeeper Scalability to 10,000s of nodes Isolation between tasks ...

Read More »

Technology Decisions Are About Trade Offs and Solving Problems

software-development-logo

At some point in the last decade we hit the inflection point where distributed systems, and all their complexities, became the common reality. Maybe it was the need to change how we scale since CPU clocks are not getting any faster… Maybe it was the Google MapReduce and/or Amazon Dynamo papers… Or maybe it was just the RedSox winning the ...

Read More »

Big Data Open Source Security

apache-hadoop-logo

In security there has never (IMHO) been enough open source solutions and Bruce Schneier has written about this several times in the past, and there’s no need to rewrite the arguments again. Now with “NoSQL” and “Big Data” Open Source trends in the market place Security finally has an intersection… a union if I may where new solutions to solve ...

Read More »

Want to take your Java skills to the next level?

Grab our programming books for FREE!

Here are some of the eBooks you will get:

  • Advanced Java Guide
  • Java Design Patterns
  • JMeter Tutorial
  • Java 8 Features Tutorial
  • JUnit Tutorial
  • JSF Programming Cookbook
  • Java Concurrency Essentials