CDAP Blog

Efficient Use of Hadoop Cluster with YARN Capacity Scheduler

As organizations see an increase in Hadoop adoption, there is a spike in both the number of jobs that are run on a Hadoop cluster, as well as the number of tenants utilizing the cluster. Effectively utilizing a Hadoop cluster becomes important from an administration perspective. Consolidating data and allowing multiple tenants to share a … Read more


Deploying CDAP packages from source via Coopr

chrisg

Developing features for CDAP follows a similar workflow as working on many projects. Developers have their local checkout of the source, make modifications in a feature branch, build and test locally on their development machines, push their branch, and submit a pull request for code review. During this process, developers build CDAP clusters (for testing) … Read more


Hadoop Vendor OS Support Matrix

chrisg

Developing our open source data application platform, CDAP, which runs on top of Apache™ Hadoop® can be a challenging task. It requires testing of many different configurations, on multiple vendors of Hadoop, and on lots of different distributions of Linux. Setting up and testing all of these configurations can be extremely difficult without a simple reference of supported Linux distributions … Read more


Multitenancy for Hadoop: Namespaces

bhooshan

As a data processing platform, Hadoop‘s popularity today is often attributed to its cost-effectiveness, derived equally from the usage of commodity hardware and from the ability to co-locate work on shared compute and storage resources. Sharing resources allows organizations to maximize the throughput and utilization of a small number of large clusters instead of managing a large … Read more


Data-driven job scheduling in Hadoop

Julien Guery

Triggering the processing of data in Hadoop—as soon as enough new data is available—helps optimize many incremental data processing use-cases, but is not trivial to implement. The ability to schedule a job (such as MapReduce or Spark) to run as soon as there’s a certain amount of unprocessed data available—for instance, in a set of … Read more


CDAP v2.8.0 is out in the wild

I am very happy to announce that the latest release of our flagship product – the Cask Data Application Platform (CDAP) – v2.8.0 is now available for everyone to download. This release has a bunch of cool features that our customers, partners and the community want: Namespaces (provides application and data isolation that enables multi-tenancy) … Read more



The Shift to Realtime Processing: An Easier Way to Build Hadoop Apps

I was inspired by the recent Google I/O talk on Cloud Dataflow, a data processing service used internally at Google, which evolved from a model based on MapReduce and successor stream processing technologies such as MillWheel and FlumeJava. Based on the premise of focusing on your application logic rather than the underlying infrastructure, I set … Read more