Apache Mesos

Apache Mesos is an open source cluster manager that handles workloads in a distributed environment through dynamic resource sharing and isolation. Mesos is suited for the deployment and management of applications in large-scale clustered environments. Mesos brings together the existing resources of the machines/nodes in a cluster into a single pool from which a variety of workloads may utillize. Also known as node abstraction, this removes the need to allocate specific machines for different workloads. Companies such as Twitter, Airbnb and Xogito utilize Apache Mesos.

Originally developed at the University of California at Berkley, Mesos sits between the operating system and the application layer and basically acts as a data center kernel. Mesos isolates the processes running in a cluster, such as memory, CPU, file system, rack locality and I/O, to keep them from interfering with each other. Such isolation allows Mesos to create a single, large pool of resources to offer workloads.

Apache Mesos uses master-agent architecture in conjunction with frameworks to manage and isolate resource requests. The agent daemons run on each cluster node. These application frameworks, such as Apache Spark, Chronos, Apache Aurora and Mesosphere Marathon, each specialize in different jobs and workloads. Spark offers Mesos-powered and standalone cluster computing. Marathon and Aurora are geared towards long-running services and specialize scaling services. Chronos is a distributed and fault-tolerant scheduler (a replacement for cron) which automatically starts and stops services and manages failures. Without frameworks in place, specific workloads could consume all the pooled resources. Mesos also utilizes Apache Zookeeper, part of Hadoop, to synchronize distributed processes to ensure all clients receive consistent data and assure fault tolerance.

Each framework consists of at least two crucial components: a scheduler and executor. Schedulers register with the Mesos master to get resources, and executors launch the command or program that runs tasks on the slaves. The master offers resources to each framework, but it is the framework’s scheduler that chooses which of those available resources to use. After a framework accepts the resources offered by the master, it sends a description of the tasks back to the master. The master then sends these tasks to the slave, and the executor on the slave launches the tasks.

Mesos can also manage and run Docker containers with Chronos and Marathon frameworks. Some of Mesos' features include multi-resource scheduling.