Drill

Drill is a database system designed for Big Data exploration. It is an open-source, distributed SQL query system based on Google's Dremel query system, and it features a columnar execution engine. Drill is the only distributed SQL engine in the world that does not require schemas. It was designed from the ground up, and it supports many NoSQL databases and file systems, with the ability for a single query to join data from multiple types of datastores, like MongoDB or HBase for instance.

History

In 2010, Google published a paper titled "Dremel: Interactive Analysis of Web-Scale Datasets" that described a scalable database system designed for "interactive analysis of nested data". The Dremel system is available today under Google's BigQuery system. Development of Apache Drill began in 2012, with the goal of replicating the capabilities of Dremel. Initial goals of the system included support for multiple storage systems, file formats, query languages, and data sources, as well as the ability to scale over 10,000 servers and process petabytes of data in seconds.

Query Interface

SQL

Concurrency Control

Optimistic Concurrency Control (OCC)

Drill supports Optimistic Concurrency Control. It plans queries in fragments, assuming that all of the fragments can be completed in parallel without interfering with each other. Larger fragments are broken into smaller fragments, which are run in clusters until the whole fragment is complete.

Checkpoints

Not Supported

Drill adopts optimistic query execution, which assumes that failures occur rarely during queries. Therefore, it does not take checkpoints. With its pipelined query execution model, single queries are simply reran when they fail.

Foreign Keys

Supported

Drill supports the usage of foreign keys within the schemas of the datastores that it gathers data from.

Data Model

Column Family

Drill features a JSON self-describing data model that supports language independence and loosely defined, weak data typing. This data model uses on-the-fly schema discovery, also known as late binding, to begin the execution of queries without having to know the structure of the data. Through this data model, Drill can handle data with evolving schemas or even no schemas at all.

Drill's internal data representation is columnar and hierarchical, which allows for efficient SQL processing without the need to flatten data into rows. The data model supports queries on complex/nested data as well as evolving data structures.

System Architecture

Shared-Disk

Drill Logo
Website

https://drill.apache.org

Source Code

https://github.com/apache/drill

Tech Docs

https://drill.apache.org/docs/

Developer

Apache Foundation

Country of Origin

US

Start Year

2012

Project Type

Open Source

Written in

Java

Supported languages

SQL

Inspired By

BigQuery

Compatible With

HBase, MongoDB

Operating Systems

Linux, OS X, Windows

Licenses

Apache v2

Wikipedia

https://en.wikipedia.org/wiki/Apache_Drill