Category: Big Data & Analytics

Aug 10

Tableau – Data Sources

Tableau – Data Sources ”; Previous Next Tableau can connect to all the popular data sources which are widely used. Tableau’s native connectors can connect to the following types of data sources. File Systems such as CSV, Excel, etc. Relational Systems such as Oracle, Sql Server, DB2, etc. Cloud Systems such as Windows Azure, Google BigQuery, etc. Other Sources using ODBC The following picture shows most of the data sources available through Tableau’s native data connectors. Connect Live The Connect Live feature is used for real-time data analysis. In this case, Tableau connects to real-time data source and keeps reading the data. Thus, the result of the analysis is up to the second, and the latest changes are reflected in the result. However, on the downside, it burdens the source system as it has to keep sending the data to Tableau. In-Memory Tableau can also process data in-memory by caching them in memory and not being connected to the source anymore while analyzing the data. Of course, there will be a limit to the amount of data cached depending on the availability of memory. Combine Data Sources Tableau can connect to different data sources at the same time. For example, in a single workbook you can connect to a flat file and a relational source by defining multiple connections. This is used in data blending, which is a very unique feature in Tableau. Print Page Previous Next Advertisements ”;

Aug 10

Variance

Statistics – Variance ”; Previous Next A variance is defined as the average of Squared differences from mean value. Combination is defined and given by the following function: Formula ${ delta = frac{ sum (M – n_i)^2 }{n}}$ Where − ${M}$ = Mean of items. ${n}$ = the number of items considered. ${n_i}$ = items. Example Problem Statement: Find the variance between following data : {600, 470, 170, 430, 300} Solution: Step 1: Determine the Mean of the given items. ${ M = frac{600 + 470 + 170 + 430 + 300}{5} \[7pt] = frac{1970}{5} \[7pt] = 394}$ Step 2: Determine Variance ${ delta = frac{ sum (M – n_i)^2 }{n} \[7pt] = frac{(600 – 394)^2 + (470 – 394)^2 + (170 – 394)^2 + (430 – 394)^2 + (300 – 394)^2}{5} \[7pt] = frac{(206)^2 + (76)^2 + (-224)^2 + (36)^2 + (-94)^2}{5} \[7pt] = frac{ 42,436 + 5,776 + 50,176 + 1,296 + 8,836}{5} \[7pt] = frac{ 108,520}{5} \[7pt] = frac{(14)(13)(3)(11)}{(2)(1)} \[7pt] = 21,704}$ As a result, Variance is ${21,704}$. Print Page Previous Next Advertisements ”;

Aug 10

Tableau – Save & Delete Worksheet

Tableau – Save & Delete Worksheet ”; Previous Next An existing worksheet can be both saved and deleted. This helps in organizing the contents in the Tableau desktop environment. While you can save a worksheet by clicking the save button under the main menu, you can delete a worksheet using the following steps. Deleting the Worksheet To delete a worksheet, right-click on name of the worksheet and choose the option ‘Delete Sheet’. The following screenshot shows the worksheet has been deleted. Print Page Previous Next Advertisements ”;

Aug 10

Transformations

Statistics – Transformations ”; Previous Next Data transformation refers to application of a function to each item in a data set. Here $ x_i $ is replaced by its transformed value $ y_i $ where $ y_i = f(x_i) $. Data transformations are carried out generally to make appearance of graphs more interpretable. There are four major functions used for transformations. $ log x $ – logarithm transformations. For example sound units are in decibels and is generally represented using log transformations. $ frac{1}{x} $ – Reciprocal Transformations. For example Time to complete race/ task is represents using speed. More the speed lesser the time taken. $ sqrt{x} $ – Square root Transformations. For example areas of circular ground are compared using their radius. $ {x^2} $ – Power Transformations. For example to compare negative numbers. logarithm and Square root Transformations are used in case of positive numbers where as Reciprocal and Power Transformations can be used in case of both negative as well as positive numbers. Following diagrams illustrates the use of logarithm transformation to compare population graphically. Before transformation After transformation Print Page Previous Next Advertisements ”;

Aug 10

Tableau – Show Me

Tableau – Show Me ”; Previous Next As an advanced data visualization tool, Tableau makes the data analysis very easy by providing many analysis techniques without writing any custom code. One such feature is Show Me. It can be used to apply a required view to the existing data in the worksheet. Those views can be a pie chart, scatter plot, or a line chart. Whenever a worksheet with data is created, it is available in the top right corner as shown in the following figure. Some of the view options will be greyed out depending on the nature of selection in the data pane. Show Me with Two Fields The relation between two fields can be visually analyzed easily by using various graphs and charts available in Show Me. In this case, we choose two fields and apply a line chart. Following are the steps − Step 1 − Select the two fields (order date and profit) to be analyzed by holding the control key. Step 2 − Click the Show Me bar and choose line chart. Step 3 − Click the Mark Label button on the scrollbar. The following diagram shows the line chart created using the above steps. Show Me with Multiple Fields We can apply a similar technique as above to analyze more than 2 fields. The only difference in this case will be the availability of fewer views in active form. Tableau automatically greys out the views that are not appropriate for the analysis of the fields chosen. In this case, choose the field’s product name, customer name, sales and profit by holding down the control key. As you can observe, most of the views in Show Me are greyed out. From the active views, choose Scatter View. The following diagram shows the Scatter View chart created. Print Page Previous Next Advertisements ”;

Aug 10

Type I & II Error

Statistics – Type I & II Errors ”; Previous Next Type I and Type II errors signifies the erroneous outcomes of statistical hypothesis tests. Type I error represents the incorrect rejection of a valid null hypothesis whereas Type II error represents the incorrect retention of an invalid null hypothesis. Null Hypothesis Null Hypothesis refers to a statement which nullifies the contrary with evidence. Consider the following examples: Example 1 Hypothesis – Water added to a toothpaste protects teeth against cavities. Null Hypothesis – Water added to a toothpaste has no effect against cavities. Example 2 Hypothesis – Floride added to a toothpaste protects teeth against cavities. Null Hypothesis – Floride added to a toothpaste has no effect against cavities. Here Null hypothesis is to be tested against experimental data to nullify the effect of floride and water on teeth”s cavities. Type I Error Consider the Example 1. Here Null hypothesis is true i.e. Water added to a toothpaste has no effect against cavities. But if using experimental data, we detect an effect of water added on cavities then we are rejecting a true null hypothesis. This is a Type I error. It is also called a False Positive condition (a situation which indicates that a given condition is present but it actually is not present). The Type I error rate or significance level of Type I is represented by the probability of rejecting the null hypothesis given that it is true. Type I error is denoted by $ alpha $ and is also called alpha level. Generally It is acceptable to have Type I error significance level as 0.05 or 5% which means that 5% probability of incorrectly rejecting the null hypothesis is acceptable. Type II Error Consider the Example 2. Here Null hypothesis is false i.e. Floride added to a toothpaste has effect against cavities. But if using experimental data, we do not detect an effect of floride added on cavities then we are accepting a false null hypothesis. This is a Type II error. It is also called a False Positive condition (a situation which indicates that a given condition is not present but it actually is present). Type II error is denoted by $ beta $ and is also called beta level. Goal of a statistical test is to determine that a null hypothesis can be rejected or not. A statistical test can reject or not be able to reject a null hypothesis. Following table illustrates the relationship between truth or falseness of the null hypothesis and outcomes of the test in terms of Type I or Type II error. Judgment Null hypothesis ($ H_0 $) is Error Type Inference Reject Valid Type I Error (False Positive) Incorrect Reject Invalid True Positive Correct Unable to Reject Valid True Negative Correct Unable to Reject Invalid Type II error(False Negative) Incorrect Print Page Previous Next Advertisements ”;

Aug 10

Tableau – Paged Workbook

Tableau – Paged Workbook ”; Previous Next A paged workbook is used to save the view of the data in different pages for different values of the dimension or measure. A common example is to see how each type of products have performed against each other in a specific sales region. As each of the values of product type is stored as a separate page, we can view them one at a time or see it as a range of values. Creating Paged Workbook The paged workbook contains worksheets which have fields put in the page shelf. Consider an example of studying the profit of various sub-category of products in different regions. Following are the steps. Step 1 − Create a bar chart with two dimensions and one measure. In this case, drag the Measure Profit to the columns shelf and the dimensions sub-category, and Region to the rows shelf as shown in the following screenshot. Step 2 − Drag the Sub-Category field again to the page shelf. You will see that a page control is automatically added, just below the Pages shelf. This page control provides the following features to navigate through the pages in a view − Jump to a specific page Manually advance through the pages Automatically advance through pages In this case, we will see how to jump to a specific page and how to get the automatic display of pages. To go to a specific page, click on the drop-down on the page control and select Accessories. The chart seen in the following screenshot appears. Step 3 − For automatic display of pages, keep the show history checkbox ticked and click the play button. You can then see an automatic play of different pages of sub categories. While the current Sub-Category value is shown with a dark color, the previous values are shaded with light color. The following screenshot illustrates this. Print Page Previous Next Advertisements ”;

Aug 10

Tableau – Fields Operations

Tableau – Fields Operations ”; Previous Next Tableau has many features to manipulate the fields present in Tableau data pane. You can rename the fields or combine two fields to create one field. Such operations help in better organization of the dimensions and measures, as well as accommodate two or more fields with the same name for better data analysis. Following are the important examples of such Field Operations. Adding Fields to Worksheet You can add any field to the worksheet by right-clicking and choosing the option Add to Sheet. You can also drag and drop the fields into different shelves present in the worksheet, like Columns shelf, Rows shelf, Filters shelf, and many other shelves under the Marks card. The following diagram shows the right-click option. Combining Two Fields You can combine two dimension fields to create one field. This combined field has a name which is a combination of the individual fields. The values in the dimension get combined to a single value by joining the two strings into one string separated by a comma. However, this default name can be changed by using the rename field operation. The following diagram shows the step to combine two fields. Searching Fields You can search for names of fields by using the search box option. Writing first three or more letters of the field name brings out the result showing only the fields whose name contains these letters. Reordering Fields You can change the position of fields by simply dragging them up and down. In the following example, we drag the field customer name to the place between state and city. This is usually done to bring similar fields together which are frequently used for analysis. Print Page Previous Next Advertisements ”;

Aug 10

Zookeeper – Home

Zookeeper Tutorial PDF Version Quick Guide Resources Job Search Discussion ZooKeeper is a distributed co-ordination service to manage large set of hosts. Co-ordinating and managing a service in a distributed environment is a complicated process. ZooKeeper solves this issue with its simple architecture and API. ZooKeeper allows developers to focus on core application logic without worrying about the distributed nature of the application. The ZooKeeper framework was originally built at “Yahoo!” for accessing their applications in an easy and robust manner. Later, Apache ZooKeeper became a standard for organized service used by Hadoop, HBase, and other distributed frameworks. For example, Apache HBase uses ZooKeeper to track the status of distributed data. This tutorial explains the basics of ZooKeeper, how to install and deploy a ZooKeeper cluster in a distributed environment, and finally concludes with a few examples using Java programming and sample applications. Audience This tutorial has been prepared for professionals aspiring to make a career in Big Data Analytics using ZooKeeper framework. It will give you enough understanding on how to use ZooKeeper to create distributed clusters. Prerequisites Before proceeding with this tutorial, you must have a good understanding of Java because the ZooKeeper server runs on JVM, distributed process, and Linux environment. Print Page Previous Next Advertisements ”;

Aug 10

Sqoop – Export

Sqoop – Export ”; Previous Next This chapter describes how to export data back from the HDFS to the RDBMS database. The target table must exist in the target database. The files which are given as input to the Sqoop contain records, which are called rows in table. Those are read and parsed into a set of records and delimited with user-specified delimiter. The default operation is to insert all the record from the input files to the database table using the INSERT statement. In update mode, Sqoop generates the UPDATE statement that replaces the existing record into the database. Syntax The following is the syntax for the export command. $ sqoop export (generic-args) (export-args) $ sqoop-export (generic-args) (export-args) Example Let us take an example of the employee data in file, in HDFS. The employee data is available in emp_data file in ‘emp/’ directory in HDFS. The emp_data is as follows. 1201, gopal, manager, 50000, TP 1202, manisha, preader, 50000, TP 1203, kalil, php dev, 30000, AC 1204, prasanth, php dev, 30000, AC 1205, kranthi, admin, 20000, TP 1206, satish p, grp des, 20000, GR It is mandatory that the table to be exported is created manually and is present in the database from where it has to be exported. The following query is used to create the table ‘employee’ in mysql command line. $ mysql mysql> USE db; mysql> CREATE TABLE employee ( id INT NOT NULL PRIMARY KEY, name VARCHAR(20), deg VARCHAR(20), salary INT, dept VARCHAR(10)); The following command is used to export the table data (which is in emp_data file on HDFS) to the employee table in db database of Mysql database server. $ sqoop export –connect jdbc:mysql://localhost/db –username root –table employee –export-dir /emp/emp_data The following command is used to verify the table in mysql command line. mysql>select * from employee; If the given data is stored successfully, then you can find the following table of given employee data. +——+————–+————-+——————-+——–+ | Id | Name | Designation | Salary | Dept | +——+————–+————-+——————-+——–+ | 1201 | gopal | manager | 50000 | TP | | 1202 | manisha | preader | 50000 | TP | | 1203 | kalil | php dev | 30000 | AC | | 1204 | prasanth | php dev | 30000 | AC | | 1205 | kranthi | admin | 20000 | TP | | 1206 | satish p | grp des | 20000 | GR | +——+————–+————-+——————-+——–+ Print Page Previous Next Advertisements ”;