Pig Represents Big Data as data flows. Pig is a high-level platform or tool which is used to process the large datasets. It provides a high-level of abstraction for processing over the MapReduce. It provides a high-level scripting language, known as Pig Latin which is used to develop the data analysis codes. First, to process the data which is stored in the HDFS, the programmers will write the scripts using the Pig Latin Language. Internally Pig Engine(a component of Apache Pig) converted all these scripts into a specific map and reduce task. But these are not visible to the programmers in order to provide a high-level of abstraction. Pig Latin and Pig Engine are the two main components of the Apache Pig tool. The result of Pig always stored in the HDFS.
Note: Pig Engine has two type of execution environment i.e. a local execution environment in a single JVM (used when dataset is small in size)and distributed execution environment in a Hadoop Cluster.
Need of Pig: One limitation of MapReduce is that the development cycle is very long. Writing the reducer and mapper, compiling packaging the code, submitting the job and retrieving the output is a time-consuming task. Apache Pig reduces the time of development using the multi-query approach. Also, Pig is beneficial for programmers who are not from Java background. 200 lines of Java code can be written in only 10 lines using the Pig Latin language. Programmers who have SQL knowledge needed less effort to learn Pig Latin.
- It uses query approach which results in reducing the length of the code.
- Pig Latin is SQL like language.
- It provides many builtIn operators.
- It provides nested data types (tuples, bags, map).
Evolution of Pig: Earlier in 2006, Apache Pig was developed by Yahoo’s researchers. At that time, the main idea to develop Pig was to execute the MapReduce jobs on extremely large datasets. In the year 2007, it moved to Apache Software Foundation(ASF) which makes it an open source project. The first version(0.1) of Pig came in the year 2008. The latest version of Apache Pig is 0.18 which came in the year 2017.
Features of Apache Pig:
- For performing several operations Apache Pig provides rich sets of operators like the filtering, joining, sorting, aggregation etc.
- Easy to learn, read and write. Especially for SQL-programmer, Apache Pig is a boon.
- Apache Pig is extensible so that you can make your own process and user-defined functions(UDFs) written in python, java or other programming languages .
- Join operation is easy in Apache Pig.
- Fewer lines of code.
- Apache Pig allows splits in the pipeline.
- By integrating with other components of the Apache Hadoop ecosystem, such as Apache Hive, Apache Spark, and Apache ZooKeeper, Apache Pig enables users to take advantage of these components’ capabilities while transforming data.
- The data structure is multivalued, nested, and richer.
- Pig can handle the analysis of both structured and unstructured data.
| Apache Pig | MapReduce |
|---|---|
| It is a scripting language. | It is a compiled programming language. |
| Abstraction is at higher level. | Abstraction is at lower level. |
| It have less line of code as compared to MapReduce. | Lines of code is more. |
| Less effort is needed for Apache Pig. | More development efforts are required for MapReduce. |
| Code efficiency is less as compared to MapReduce. | As compared to Pig efficiency of code is higher. |
| Pig provides built in functions for ordering, sorting and union. | Hard to perform data operations. |
| It allows nested data types like map, tuple and bag | It does not allow nested data types |
Applications of Apache Pig:
- For exploring large datasets Pig Scripting is used.
- Provides the supports across large data-sets for Ad-hoc queries.
- In the prototyping of large data-sets processing algorithms.
- Required to process the time sensitive data loads.
- For collecting large amounts of datasets in form of search logs and web crawls.
- Used where the analytical insights are needed using the sampling.
Types of Data Models in Apache Pig: It consist of the 4 types of data models as follows:
- Atom: It is a atomic data value which is used to store as a string. The main use of this model is that it can be used as a number and as well as a string.
- Tuple: It is an ordered set of the fields.
- Bag: It is a collection of the tuples.
- Map: It is a set of key/value pairs.