Category: Big Data & Analytics

AVRO – Environment Setup ”; Previous Next Apache software foundation provides Avro with various releases. You can download the required release from Apache mirrors. Let us see, how to set up the environment to work with Avro − Downloading Avro To download Apache Avro, proceed with the following − Open the web page Apache.org. You will see the homepage of Apache Avro as shown below − Click on project → releases. You will get a list of releases. Select the latest release which leads you to a download link. mirror.nexcess is one of the links where you can find the list of all libraries of different languages that Avro supports as shown below − You can select and download the library for any of the languages provided. In this tutorial, we use Java. Hence download the jar files avro-1.7.7.jar and avro-tools-1.7.7.jar. Avro with Eclipse To use Avro in Eclipse environment, you need to follow the steps given below − Step 1. Open eclipse. Step 2. Create a project. Step 3. Right-click on the project name. You will get a shortcut menu. Step 4. Click on Build Path. It leads you to another shortcut menu. Step 5. Click on Configure Build Path… You can see Properties window of your project as shown below − Step 6. Under libraries tab, click on ADD EXternal JARs… button. Step 7. Select the jar file avro-1.77.jar you have downloaded. Step 8. Click on OK. Avro with Maven You can also get the Avro library into your project using Maven. Given below is the pom.xml file for Avro. <project xmlns=”http://maven.apache.org/POM/4.0.0″ xmlns:xsi=” http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd”> <modelVersion>4.0.0</modelVersion> <groupId>Test</groupId> <artifactId>Test</artifactId> <version>0.0.1-SNAPSHOT</version> <build> <sourceDirectory>src</sourceDirectory> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.7.7</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro-tools</artifactId> <version>1.7.7</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <version>2.0-beta9</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>2.0-beta9</version> </dependency> </dependencies> </project> Setting Classpath To work with Avro in Linux environment, download the following jar files − avro-1.77.jar avro-tools-1.77.jar log4j-api-2.0-beta9.jar og4j-core-2.0.beta9.jar. Copy these files into a folder and set the classpath to the folder, in the ./bashrc file as shown below. #class path for Avro export CLASSPATH=$CLASSPATH://home/Hadoop/Avro_Work/jars/* Print Page Previous Next Advertisements ”;

AVRO – Reference API ”; Previous Next In the previous chapter, we described the input type of Avro, i.e., Avro schemas. In this chapter, we will explain the classes and methods used in the serialization and deserialization of Avro schemas. SpecificDatumWriter Class This class belongs to the package org.apache.avro.specific. It implements the DatumWriter interface which converts Java objects into an in-memory serialized format. Constructor S.No. Description 1 SpecificDatumWriter(Schema schema) Method S.No. Description 1 SpecificData getSpecificData() Returns the SpecificData implementation used by this writer. SpecificDatumReader Class This class belongs to the package org.apache.avro.specific. It implements the DatumReader interface which reads the data of a schema and determines in-memory data representation. SpecificDatumReader is the class which supports generated java classes. Constructor S.No. Description 1 SpecificDatumReader(Schema schema) Construct where the writer”s and reader”s schemas are the same. Methods S.No. Description 1 SpecificData getSpecificData() Returns the contained SpecificData. 2 void setSchema(Schema actual) This method is used to set the writer”s schema. DataFileWriter Instantiates DataFileWrite for emp class. This class writes a sequence serialized records of data conforming to a schema, along with the schema in a file. Constructor S.No. Description 1 DataFileWriter(DatumWriter<D> dout) Methods S.No Description 1 void append(D datum) Appends a datum to a file. 2 DataFileWriter<D> appendTo(File file) This method is used to open a writer appending to an existing file. Data FileReader This class provides random access to files written with DataFileWriter. It inherits the class DataFileStream. Constructor S.No. Description 1 DataFileReader(File file, DatumReader<D> reader)) Methods S.No. Description 1 next() Reads the next datum in the file. 2 Boolean hasNext() Returns true if more entries remain in this file. Class Schema.parser This class is a parser for JSON-format schemas. It contains methods to parse the schema. It belongs to org.apache.avro package. Constructor S.No. Description 1 Schema.Parser() Methods S.No. Description 1 parse (File file) Parses the schema provided in the given file. 2 parse (InputStream in) Parses the schema provided in the given InputStream. 3 parse (String s) Parses the schema provided in the given String. Interface GenricRecord This interface provides methods to access the fields by name as well as index. Methods S.No. Description 1 Object get(String key) Returns the value of a field given. 2 void put(String key, Object v) Sets the value of a field given its name. Class GenericData.Record Constructor S.No. Description 1 GenericData.Record(Schema schema) Methods S.No. Description 1 Object get(String key) Returns the value of a field of the given name. 2 Schema getSchema() Returns the schema of this instance. 3 void put(int i, Object v) Sets the value of a field given its position in the schema. 4 void put(String key, Object value) Sets the value of a field given its name. Print Page Previous Next Advertisements ”;

AVRO – Quick Guide ”; Previous Next AVRO – Overview To transfer data over a network or for its persistent storage, you need to serialize the data. Prior to the serialization APIs provided by Java and Hadoop, we have a special utility, called Avro, a schema-based serialization technique. This tutorial teaches you how to serialize and deserialize the data using Avro. Avro provides libraries for various programming languages. In this tutorial, we demonstrate the examples using Java library. What is Avro? Apache Avro is a language-neutral data serialization system. It was developed by Doug Cutting, the father of Hadoop. Since Hadoop writable classes lack language portability, Avro becomes quite helpful, as it deals with data formats that can be processed by multiple languages. Avro is a preferred tool to serialize data in Hadoop. Avro has a schema-based system. A language-independent schema is associated with its read and write operations. Avro serializes the data which has a built-in schema. Avro serializes the data into a compact binary format, which can be deserialized by any application. Avro uses JSON format to declare the data structures. Presently, it supports languages such as Java, C, C++, C#, Python, and Ruby. Avro Schemas Avro depends heavily on its schema. It allows every data to be written with no prior knowledge of the schema. It serializes fast and the resulting serialized data is lesser in size. Schema is stored along with the Avro data in a file for any further processing. In RPC, the client and the server exchange schemas during the connection. This exchange helps in the communication between same named fields, missing fields, extra fields, etc. Avro schemas are defined with JSON that simplifies its implementation in languages with JSON libraries. Like Avro, there are other serialization mechanisms in Hadoop such as Sequence Files, Protocol Buffers, and Thrift. Comparison with Thrift and Protocol Buffers Thrift and Protocol Buffers are the most competent libraries with Avro. Avro differs from these frameworks in the following ways − Avro supports both dynamic and static types as per the requirement. Protocol Buffers and Thrift use Interface Definition Languages (IDLs) to specify schemas and their types. These IDLs are used to generate code for serialization and deserialization. Avro is built in the Hadoop ecosystem. Thrift and Protocol Buffers are not built in Hadoop ecosystem. Unlike Thrift and Protocol Buffer, Avro”s schema definition is in JSON and not in any proprietary IDL. Property Avro Thrift & Protocol Buffer Dynamic schema Yes No Built into Hadoop Yes No Schema in JSON Yes No No need to compile Yes No No need to declare IDs Yes No Bleeding edge Yes No Features of Avro Listed below are some of the prominent features of Avro − Avro is a language-neutral data serialization system. It can be processed by many languages (currently C, C++, C#, Java, Python, and Ruby). Avro creates binary structured format that is both compressible and splittable. Hence it can be efficiently used as the input to Hadoop MapReduce jobs. Avro provides rich data structures. For example, you can create a record that contains an array, an enumerated type, and a sub record. These datatypes can be created in any language, can be processed in Hadoop, and the results can be fed to a third language. Avro schemas defined in JSON, facilitate implementation in the languages that already have JSON libraries. Avro creates a self-describing file named Avro Data File, in which it stores data along with its schema in the metadata section. Avro is also used in Remote Procedure Calls (RPCs). During RPC, client and server exchange schemas in the connection handshake. General Working of Avro To use Avro, you need to follow the given workflow − Step 1 − Create schemas. Here you need to design Avro schema according to your data. Step 2 − Read the schemas into your program. It is done in two ways − By Generating a Class Corresponding to Schema − Compile the schema using Avro. This generates a class file corresponding to the schema By Using Parsers Library − You can directly read the schema using parsers library. Step 3 − Serialize the data using the serialization API provided for Avro, which is found in the package org.apache.avro.specific. Step 4 − Deserialize the data using deserialization API provided for Avro, which is found in the package org.apache.avro.specific. AVRO – Serialization Data is serialized for two objectives − For persistent storage To transport the data over network What is Serialization? Serialization is the process of translating data structures or objects state into binary or textual form to transport the data over network or to store on some persisten storage. Once the data is transported over network or retrieved from the persistent storage, it needs to be deserialized again. Serialization is termed as marshalling and deserialization is termed as unmarshalling. Serialization in Java Java provides a mechanism, called object serialization where an object can be represented as a sequence of bytes that includes the object”s data as well as information about the object”s type and the types of data stored in the object. After a serialized object is written into a file, it can be read from the file and deserialized. That is, the type information and bytes that represent the object and its data can be used to recreate the object in memory. ObjectInputStream and ObjectOutputStream classes are used to serialize and deserialize an object respectively in Java. Serialization in Hadoop Generally in distributed systems like Hadoop, the concept of serialization is used for Interprocess Communication and Persistent Storage. Interprocess Communication To establish the interprocess communication between the nodes connected in a network, RPC technique was used. RPC used internal serialization to convert the message into binary format before sending it to the remote node via network. At the other end the remote system deserializes the binary stream into the original message. The RPC serialization format is required to be as follows − Compact − To make the best use