View on GitHub

The Hadoop Ecosystem Builder

The Buildoop Project

Buildoop Project is an open source collaboration project that provides templates and tools to help you create custom Linux-based systems based on Hadoop ecosystem.

Buildoop Project is suitable for Big Data developers based on Hadoop

It’s suitable both for:

You can use any Linux distribution as foundation (Centos 6.x, Ubuntu 12.04.x, …) and create add-ons with Buildoop to convert this distribution in a Hadoop Linux distribution.

Buildoop Software Development life-cycle

The Buildoop Project fits in the integration phase in a classical software development cycle. You can puts together all bits of Hadoop ecosystem and perform a set of tests as a group

Smiley face

Buildoop Project components

The Buildoop Project is splitted in the following parts:

  1. Buildoop engine: a make-like build command line tool.

  2. Metadata: A set of metadata information with “buildoop recipes”. “buildoop recipes” specify how a particular package is built. It includes all the package dependencies, source code locations, configuration, compilation, build, install and remove instructions.

  3. SIT: System Integration Test Framework. The individual software modules are combined and tested as a group by means of virtualized infrastructure.

  4. Configuration artifacts: Set of configuration management files for operating system deploy and software configuration.

Smiley face

Lambda Architecture Use Case with Buildoop

The following is a real production implementation of Lambda Architecture for Linux Audit Framework analysis using Buildoop as package building system.
The whole cluster is monitored with Linux Audit enabled. The audit.log for each node is converted to Avro and is injected using Flume into kafka, the log is processed in the batch layer for long term storage analysis. The same log entry is processed in the speed layer for incremental frame analysis, or for other on-line analysis, for example the current logged users in the cluster.
The whole tool set is built using Buildoop.

Smiley face

The same picture in terms of Buildoop recipe packages:

Smiley face

More in Buildoop Project Presentation

The Buildoop Project from Javi Roman