Category: python Data Science

Python – Processing JSON Data ”; Previous Next JSON file stores data as text in human-readable format. JSON stands for JavaScript Object Notation. Pandas can read JSON files using the read_json function. Input Data Create a JSON file by copying the below data into a text editor like notepad. Save the file with .json extension and choosing the file type as all files(*.*). { “ID”:[“1″,”2″,”3″,”4″,”5″,”6″,”7″,”8” ], “Name”:[“Rick”,”Dan”,”Michelle”,”Ryan”,”Gary”,”Nina”,”Simon”,”Guru” ] “Salary”:[“623.3″,”515.2″,”611″,”729″,”843.25″,”578″,”632.8″,”722.5” ], “StartDate”:[ “1/1/2012″,”9/23/2013″,”11/15/2014″,”5/11/2014″,”3/27/2015″,”5/21/2013”, “7/30/2013″,”6/17/2014”], “Dept”:[ “IT”,”Operations”,”IT”,”HR”,”Finance”,”IT”,”Operations”,”Finance”] } Read the JSON File The read_json function of the pandas library can be used to read the JSON file into a pandas DataFrame. import pandas as pd data = pd.read_json(”path/input.json”) print (data) When we execute the above code, it produces the following result. Dept ID Name Salary StartDate 0 IT 1 Rick 623.30 1/1/2012 1 Operations 2 Dan 515.20 9/23/2013 2 IT 3 Tusar 611.00 11/15/2014 3 HR 4 Ryan 729.00 5/11/2014 4 Finance 5 Gary 843.25 3/27/2015 5 IT 6 Rasmi 578.00 5/21/2013 6 Operations 7 Pranab 632.80 7/30/2013 7 Finance 8 Guru 722.50 6/17/2014 Reading Specific Columns and Rows Similar to what we have already seen in the previous chapter to read the CSV file, the read_json function of the pandas library can also be used to read some specific columns and specific rows after the JSON file is read to a DataFrame. We use the multi-axes indexing method called .loc() for this purpose. We choose to display the Salary and Name column for some of the rows. import pandas as pd data = pd.read_json(”path/input.xlsx”) # Use the multi-axes indexing funtion print (data.loc[[1,3,5],[”salary”,”name”]]) When we execute the above code, it produces the following result. salary name 1 515.2 Dan 3 729.0 Ryan 5 578.0 Rasmi Reading JSON file as Records We can also apply the to_json function along with parameters to read the JSON file content into individual records. import pandas as pd data = pd.read_json(”path/input.xlsx”) print(data.to_json(orient=”records”, lines=True)) When we execute the above code, it produces the following result. {“Dept”:”IT”,”ID”:1,”Name”:”Rick”,”Salary”:623.3,”StartDate”:”1/1/2012″} {“Dept”:”Operations”,”ID”:2,”Name”:”Dan”,”Salary”:515.2,”StartDate”:”9/23/2013″} {“Dept”:”IT”,”ID”:3,”Name”:”Tusar”,”Salary”:611.0,”StartDate”:”11/15/2014″} {“Dept”:”HR”,”ID”:4,”Name”:”Ryan”,”Salary”:729.0,”StartDate”:”5/11/2014″} {“Dept”:”Finance”,”ID”:5,”Name”:”Gary”,”Salary”:843.25,”StartDate”:”3/27/2015″} {“Dept”:”IT”,”ID”:6,”Name”:”Rasmi”,”Salary”:578.0,”StartDate”:”5/21/2013″} {“Dept”:”Operations”,”ID”:7,”Name”:”Pranab”,”Salary”:632.8,”StartDate”:”7/30/2013″} {“Dept”:”Finance”,”ID”:8,”Name”:”Guru”,”Salary”:722.5,”StartDate”:”6/17/2014″} Print Page Previous Next Advertisements ”;

Python – NoSQL Databases ”; Previous Next As more and more data become available as unstructured or semi-structured, the need of managing them through NoSql database increases. Python can also interact with NoSQL databases in a similar way as is interacts with Relational databases. In this chapter we will use python to interact with MongoDB as a NoSQL database. In case you are new to MongoDB, you can learn it in our tutorial here. In order to connect to MongoDB, python uses a library known as pymongo. You can add this library to your python environment, using the below command from the Anaconda environment. conda install pymongo This library enables python to connect to MOngoDB using a db client. Once connected we select the db name to be used for various operations. Inserting Data To insert data into MongoDB we use the insert() method which is available in the database environment. First we connect to the db using python code shown below and then we provide the document details in form of a series of key-value pairs. # Import the python libraries from pymongo import MongoClient from pprint import pprint # Choose the appropriate client client = MongoClient() # Connect to the test db db=client.test # Use the employee collection employee = db.employee employee_details = { ”Name”: ”Raj Kumar”, ”Address”: ”Sears Streer, NZ”, ”Age”: ”42” } # Use the insert method result = employee.insert_one(employee_details) # Query for the inserted document. Queryresult = employee.find_one({”Age”: ”42”}) pprint(Queryresult) When we execute the above code, it produces the following result. {u”Address”: u”Sears Streer, NZ”, u”Age”: u”42”, u”Name”: u”Raj Kumar”, u”_id”: ObjectId(”5adc5a9f84e7cd3940399f93”)} Updating Data Updating an existing MongoDB data is similar to inserting. We use the update() method which is native to mongoDB. In the below code we are replacing the existing record with new key-value pairs. Please note how we are using the condition criteria to decide which record to update. # Import the python libraries from pymongo import MongoClient from pprint import pprint # Choose the appropriate client client = MongoClient() # Connect to db db=client.test employee = db.employee # Use the condition to choose the record # and use the update method db.employee.update_one( {“Age”:”42”}, { “$set”: { “Name”:”Srinidhi”, “Age”:”35”, “Address”:”New Omsk, WC” } } ) Queryresult = employee.find_one({”Age”:”35”}) pprint(Queryresult) When we execute the above code, it produces the following result. {u”Address”: u”New Omsk, WC”, u”Age”: u”35”, u”Name”: u”Srinidhi”, u”_id”: ObjectId(”5adc5a9f84e7cd3940399f93”)} Deleting Data Deleting a record is also straight forward where we use the delete method. Here also we mention the condition which is used to choose the record to be deleted. # Import the python libraries from pymongo import MongoClient from pprint import pprint # Choose the appropriate client client = MongoClient() # Connect to db db=client.test employee = db.employee # Use the condition to choose the record # and use the delete method db.employee.delete_one({“Age”:”35”}) Queryresult = employee.find_one({”Age”:”35”}) pprint(Queryresult) When we execute the above code, it produces the following result. None So we see the particular record does not exist in the db any more. Print Page Previous Next Advertisements ”;

Python – Measuring Variance ”; Previous Next In statistics, variance is a measure of how far a value in a data set lies from the mean value. In other words, it indicates how dispersed the values are. It is measured by using standard deviation. The other method commonly used is skewness. Both of these are calculated by using functions available in pandas library. Measuring Standard Deviation Standard deviation is square root of variance. variance is the average of squared difference of values in a data set from the mean value. In python we calculate this value by using the function std() from pandas library. import pandas as pd #Create a Dictionary of series d = {”Name”:pd.Series([”Tom”,”James”,”Ricky”,”Vin”,”Steve”,”Smith”,”Jack”, ”Lee”,”Chanchal”,”Gasper”,”Naviya”,”Andres”]), ”Age”:pd.Series([25,26,25,23,30,25,23,34,40,30,25,46]), ”Rating”:pd.Series([4.23,3.24,3.98,2.56,3.20,4.6,3.8,3.78,2.98,4.80,4.10,3.65])} #Create a DataFrame df = pd.DataFrame(d) # Calculate the standard deviation print df.std() Its output is as follows − Age 7.265527 Rating 0.661628 dtype: float64 Measuring Skewness It used to determine whether the data is symmetric or skewed. If the index is between -1 and 1, then the distribution is symmetric. If the index is no more than -1 then it is skewed to the left and if it is at least 1, then it is skewed to the right import pandas as pd #Create a Dictionary of series d = {”Name”:pd.Series([”Tom”,”James”,”Ricky”,”Vin”,”Steve”,”Smith”,”Jack”, ”Lee”,”Chanchal”,”Gasper”,”Naviya”,”Andres”]), ”Age”:pd.Series([25,26,25,23,30,25,23,34,40,30,25,46]), ”Rating”:pd.Series([4.23,3.24,3.98,2.56,3.20,4.6,3.8,3.78,2.98,4.80,4.10,3.65])} #Create a DataFrame df = pd.DataFrame(d) print df.skew() Its output is as follows − Age 1.443490 Rating -0.153629 dtype: float64 So the distribution of age rating is symmetric while the distribution of age is skewed to the right. Print Page Previous Next Advertisements ”;

Python – Processing CSV Data ”; Previous Next Reading data from CSV(comma separated values) is a fundamental necessity in Data Science. Often, we get data from various sources which can get exported to CSV format so that they can be used by other systems. The Panadas library provides features using which we can read the CSV file in full as well as in parts for only a selected group of columns and rows. Input as CSV File The csv file is a text file in which the values in the columns are separated by a comma. Let”s consider the following data present in the file named input.csv. You can create this file using windows notepad by copying and pasting this data. Save the file as input.csv using the save As All files(*.*) option in notepad. id,name,salary,start_date,dept 1,Rick,623.3,2012-01-01,IT 2,Dan,515.2,2013-09-23,Operations 3,Tusar,611,2014-11-15,IT 4,Ryan,729,2014-05-11,HR 5,Gary,843.25,2015-03-27,Finance 6,Rasmi,578,2013-05-21,IT 7,Pranab,632.8,2013-07-30,Operations 8,Guru,722.5,2014-06-17,Finance Reading a CSV File The read_csv function of the pandas library is used read the content of a CSV file into the python environment as a pandas DataFrame. The function can read the files from the OS by using proper path to the file. import pandas as pd data = pd.read_csv(”path/input.csv”) print (data) When we execute the above code, it produces the following result. Please note how an additional column starting with zero as a index has been created by the function. id name salary start_date dept 0 1 Rick 623.30 2012-01-01 IT 1 2 Dan 515.20 2013-09-23 Operations 2 3 Tusar 611.00 2014-11-15 IT 3 4 Ryan 729.00 2014-05-11 HR 4 5 Gary 843.25 2015-03-27 Finance 5 6 Rasmi 578.00 2013-05-21 IT 6 7 Pranab 632.80 2013-07-30 Operations 7 8 Guru 722.50 2014-06-17 Finance Reading Specific Rows The read_csv function of the pandas library can also be used to read some specific rows for a given column. We slice the result from the read_csv function using the code shown below for first 5 rows for the column named salary. import pandas as pd data = pd.read_csv(”path/input.csv”) # Slice the result for first 5 rows print (data[0:5][”salary”]) When we execute the above code, it produces the following result. 0 623.30 1 515.20 2 611.00 3 729.00 4 843.25 Name: salary, dtype: float64 Reading Specific Columns The read_csv function of the pandas library can also be used to read some specific columns. We use the multi-axes indexing method called .loc() for this purpose. We choose to display the salary and name column for all the rows. import pandas as pd data = pd.read_csv(”path/input.csv”) # Use the multi-axes indexing funtion print (data.loc[:,[”salary”,”name”]]) When we execute the above code, it produces the following result. salary name 0 623.30 Rick 1 515.20 Dan 2 611.00 Tusar 3 729.00 Ryan 4 843.25 Gary 5 578.00 Rasmi 6 632.80 Pranab 7 722.50 Guru Reading Specific Columns and Rows The read_csv function of the pandas library can also be used to read some specific columns and specific rows. We use the multi-axes indexing method called .loc() for this purpose. We choose to display the salary and name column for some of the rows. import pandas as pd data = pd.read_csv(”path/input.csv”) # Use the multi-axes indexing funtion print (data.loc[[1,3,5],[”salary”,”name”]]) When we execute the above code, it produces the following result. salary name 1 515.2 Dan 3 729.0 Ryan 5 578.0 Rasmi Reading Specific Columns for a Range of Rows The read_csv function of the pandas library can also be used to read some specific columns and a range of rows. We use the multi-axes indexing method called .loc() for this purpose. We choose to display the salary and name column for some of the rows. import pandas as pd data = pd.read_csv(”path/input.csv”) # Use the multi-axes indexing funtion print (data.loc[2:6,[”salary”,”name”]]) When we execute the above code, it produces the following result. salary name 2 611.00 Tusar 3 729.00 Ryan 4 843.25 Gary 5 578.00 Rasmi 6 632.80 Pranab Print Page Previous Next Advertisements ”;

Python Data Science – Pandas ”; Previous Next What is Pandas? Pandas is an open-source Python Library used for high-performance data manipulation and data analysis using its powerful data structures. Python with pandas is in use in a variety of academic and commercial domains, including Finance, Economics, Statistics, Advertising, Web Analytics, and more. Using Pandas, we can accomplish five typical steps in the processing and analysis of data, regardless of the origin of data — load, organize, manipulate, model, and analyse the data. Below are the some of the important features of Pandas which is used specifically for Data processing and Data analysis work. Key Features of Pandas Fast and efficient DataFrame object with default and customized indexing. Tools for loading data into in-memory data objects from different file formats. Data alignment and integrated handling of missing data. Reshaping and pivoting of date sets. Label-based slicing, indexing and subsetting of large data sets. Columns from a data structure can be deleted or inserted. Group by data for aggregation and transformations. High performance merging and joining of data. Time Series functionality. Pandas deals with the following three data structures − Series DataFrame These data structures are built on top of Numpy array, making them fast and efficient. Dimension & Description The best way to think of these data structures is that the higher dimensional data structure is a container of its lower dimensional data structure. For example, DataFrame is a container of Series, Panel is a container of DataFrame. Data Structure Dimensions Description Series 1 1D labeled homogeneous array, size-immutable. Data Frames 2 General 2D labeled, size-mutable tabular structure with potentially heterogeneously typed columns. DataFrame is widely used and it is the most important data structures. Series Series is a one-dimensional array like structure with homogeneous data. For example, the following series is a collection of integers 10, 23, 56, … 10 23 56 17 52 61 73 90 26 72 Key Points of Series Homogeneous data Size Immutable Values of Data Mutable DataFrame DataFrame is a two-dimensional array with heterogeneous data. For example, Name Age Gender Rating Steve 32 Male 3.45 Lia 28 Female 4.6 Vin 45 Male 3.9 Katie 38 Female 2.78 The table represents the data of a sales team of an organization with their overall performance rating. The data is represented in rows and columns. Each column represents an attribute and each row represents a person. Data Type of Columns The data types of the four columns are as follows − Column Type Name String Age Integer Gender String Rating Float Key Points of Data Frame Heterogeneous data Size Mutable Data Mutable We will see lots of examples on using pandas library of python in Data science work in the next chapters. Print Page Previous Next Advertisements ”;

Python Data Science – SciPy ”; Previous Next What is SciPy? The SciPy library of Python is built to work with NumPy arrays and provides many user-friendly and efficient numerical practices such as routines for numerical integration and optimization. Together, they run on all popular operating systems, are quick to install and are free of charge. NumPy and SciPy are easy to use, but powerful enough to depend on by some of the world”s leading scientists and engineers. SciPy Sub-packages SciPy is organized into sub-packages covering different scientific computing domains. These are summarized in the following table − scipy.constants Physical and mathematical constants scipy.fftpack Fourier transform scipy.integrate Integration routines scipy.interpolate Interpolation scipy.io Data input and output scipy.linalg Linear algebra routines scipy.optimize Optimization scipy.signal Signal processing scipy.sparse Sparse matrices scipy.spatial Spatial data structures and algorithms scipy.special Any special mathematical functions scipy.stats Statistics Data Structure The basic data structure used by SciPy is a multidimensional array provided by the NumPy module. NumPy provides some functions for Linear Algebra, Fourier Transforms and Random Number Generation, but not with the generality of the equivalent functions in SciPy. We will see lots of examples on using SciPy library of python in Data science work in the next chapters. Print Page Previous Next Advertisements ”;

Python – Processing XLS Data ”; Previous Next Microsoft Excel is a very widely used spread sheet program. Its user friendliness and appealing features makes it a very frequently used tool in Data Science. The Panadas library provides features using which we can read the Excel file in full as well as in parts for only a selected group of Data. We can also read an Excel file with multiple sheets in it. We use the read_excel function to read the data from it. Input as Excel File We Create an excel file with multiple sheets in the windows OS. The Data in the different sheets is as shown below. You can create this file using the Excel Program in windows OS. Save the file as input.xlsx. # Data in Sheet1 id,name,salary,start_date,dept 1,Rick,623.3,2012-01-01,IT 2,Dan,515.2,2013-09-23,Operations 3,Tusar,611,2014-11-15,IT 4,Ryan,729,2014-05-11,HR 5,Gary,843.25,2015-03-27,Finance 6,Rasmi,578,2013-05-21,IT 7,Pranab,632.8,2013-07-30,Operations 8,Guru,722.5,2014-06-17,Finance # Data in Sheet2 id name zipcode 1 Rick 301224 2 Dan 341255 3 Tusar 297704 4 Ryan 216650 5 Gary 438700 6 Rasmi 665100 7 Pranab 341211 8 Guru 347480 Reading an Excel File The read_excel function of the pandas library is used read the content of an Excel file into the python environment as a pandas DataFrame. The function can read the files from the OS by using proper path to the file. By default, the function will read Sheet1. import pandas as pd data = pd.read_excel(”path/input.xlsx”) print (data) When we execute the above code, it produces the following result. Please note how an additional column starting with zero as a index has been created by the function. id name salary start_date dept 0 1 Rick 623.30 2012-01-01 IT 1 2 Dan 515.20 2013-09-23 Operations 2 3 Tusar 611.00 2014-11-15 IT 3 4 Ryan 729.00 2014-05-11 HR 4 5 Gary 843.25 2015-03-27 Finance 5 6 Rasmi 578.00 2013-05-21 IT 6 7 Pranab 632.80 2013-07-30 Operations 7 8 Guru 722.50 2014-06-17 Finance Reading Specific Columns and Rows Similar to what we have already seen in the previous chapter to read the CSV file, the read_excel function of the pandas library can also be used to read some specific columns and specific rows. We use the multi-axes indexing method called .loc() for this purpose. We choose to display the salary and name column for some of the rows. import pandas as pd data = pd.read_excel(”path/input.xlsx”) # Use the multi-axes indexing funtion print (data.loc[[1,3,5],[”salary”,”name”]]) When we execute the above code, it produces the following result. salary name 1 515.2 Dan 3 729.0 Ryan 5 578.0 Rasmi Reading Multiple Excel Sheets Multiple sheets with different Data formats can also be read by using read_excel function with help of a wrapper class named ExcelFile. It will read the multiple sheets into memory only once. In the below example we read sheet1 and sheet2 into two data frames and print them out individually. import pandas as pd with pd.ExcelFile(”C:/Users/Rasmi/Documents/pydatasci/input.xlsx”) as xls: df1 = pd.read_excel(xls, ”Sheet1”) df2 = pd.read_excel(xls, ”Sheet2”) print(“****Result Sheet 1****”) print (df1[0:5][”salary”]) print(“”) print(“***Result Sheet 2****”) print (df2[0:5][”zipcode”]) When we execute the above code, it produces the following result. ****Result Sheet 1**** 0 623.30 1 515.20 2 611.00 3 729.00 4 843.25 Name: salary, dtype: float64 ***Result Sheet 2**** 0 301224 1 341255 2 297704 3 216650 4 438700 Name: zipcode, dtype: int64 Print Page Previous Next Advertisements ”;

Python Data Science – NumPy ”; Previous Next What is NumPy? NumPy is a Python package which stands for ”Numerical Python”. It is a library consisting of multidimensional array objects and a collection of routines for processing of array. Operations using NumPy Using NumPy, a developer can perform the following operations − Mathematical and logical operations on arrays. Fourier transforms and routines for shape manipulation. Operations related to linear algebra. NumPy has in-built functions for linear algebra and random number generation. NumPy – A Replacement for MatLab NumPy is often used along with packages like SciPy (Scientific Python) and Mat−plotlib (plotting library). This combination is widely used as a replacement for MatLab, a popular platform for technical computing. However, Python alternative to MatLab is now seen as a more modern and complete programming language. It is open source, which is an added advantage of NumPy. ndarray Object The most important object defined in NumPy is an N-dimensional array type called ndarray. It describes the collection of items of the same type. Items in the collection can be accessed using a zero-based index. Every item in an ndarray takes the same size of block in the memory. Each element in ndarray is an object of data-type object (called dtype). Any item extracted from ndarray object (by slicing) is represented by a Python object of one of array scalar types. We will see lots of examples on using NumPy library of python in Data science work in the next chapters. Print Page Previous Next Advertisements ”;

Data Science Python – Getting Started ”; Previous Next What is Data Science ? Data science is the process of deriving knowledge and insights from a huge and diverse set of data through organizing, processing and analysing the data. It involves many different disciplines like mathematical and statistical modelling, extracting data from it source and applying data visualization techniques. Often it also involves handling big data technologies to gather both structured and unstructured data. Below we will see some example scenarios where Data science is used. Recommendation systems As online shopping becomes more prevalent, the e-commerce platforms are able to capture users shopping preferences as well as the performance of various products in the market. This leads to creation of recommendation systems which create models predicting the shoppers needs and show the products the shopper is most likely to buy. Financial Risk management The financial risk involving loans and credits are better analysed by using the customers past spend habits, past defaults, other financial commitments and many socio-economic indicators. These data is gathered from various sources in different formats. Organising them together and getting insight into customers profile needs the help of Data science. The outcome is minimizing loss for the financial organization by avoiding bad debt. Improvement in Health Care services The health care industry deals with a variety of data which can be classified into technical data, financial data, patient information, drug information and legal rules. All this data need to be analysed in a coordinated manner to produce insights that will save cost both for the health care provider and care receiver while remaining legally compliant. Computer Vision The advancement in recognizing an image by a computer involves processing large sets of image data from multiple objects of same category. For example, Face recognition. These data sets are modelled, and algorithms are created to apply the model to newer images to get a satisfactory result. Processing of these huge data sets and creation of models need various tools used in Data science. Efficient Management of Energy As the demand for energy consumption soars, the energy producing companies need to manage the various phases of the energy production and distribution more efficiently. This involves optimizing the production methods, the storage and distribution mechanisms as well as studying the customers consumption patterns. Linking the data from all these sources and deriving insight seems a daunting task. This is made easier by using the tools of data science. Python in Data Science The programming requirements of data science demands a very versatile yet flexible language which is simple to write the code but can handle highly complex mathematical processing. Python is most suited for such requirements as it has already established itself both as a language for general computing as well as scientific computing. More over it is being continuously upgraded in form of new addition to its plethora of libraries aimed at different programming requirements. Below we will discuss such features of python which makes it the preferred language for data science. A simple and easy to learn language which achieves result in fewer lines of code than other similar languages like R. Its simplicity also makes it robust to handle complex scenarios with minimal code and much less confusion on the general flow of the program. It is cross platform, so the same code works in multiple environments without needing any change. That makes it perfect to be used in a multi-environment setup easily. It executes faster than other similar languages used for data analysis like R and MATLAB. Its excellent memory management capability, especially garbage collection makes it versatile in gracefully managing very large volume of data transformation, slicing, dicing and visualization. Most importantly Python has got a very large collection of libraries which serve as special purpose analysis tools. For example – the NumPy package deals with scientific computing and its array needs much less memory than the conventional python list for managing numeric data. And the number of such packages is continuously growing. Python has packages which can directly use the code from other languages like Java or C. This helps in optimizing the code performance by using existing code of other languages, whenever it gives a better result. In the subsequent chapters we will see how we can leverage these features of python to accomplish all the tasks needed in the different areas of Data Science. Print Page Previous Next Advertisements ”;

Python Data Science – Matplotlib ”; Previous Next What is Matplotlib? Matplotlib is a python library used to create 2D graphs and plots by using python scripts. It has a module named pyplot which makes things easy for plotting by providing feature to control line styles, font properties, formatting axes etc. It supports a very wide variety of graphs and plots namely – histogram, bar charts, power spectra, error charts etc. It is used along with NumPy to provide an environment that is an effective open source alternative for MatLab. It can also be used with graphics toolkits like PyQt and wxPython. Conventionally, the package is imported into the Python script by adding the following statement − from matplotlib import pyplot as plt Matplotlib Example The following script produces the sine wave plot using matplotlib. Example import numpy as np import matplotlib.pyplot as plt # Compute the x and y coordinates for points on a sine curve x = np.arange(0, 3 * np.pi, 0.1) y = np.sin(x) plt.title(“sine wave form”) # Plot the points using matplotlib plt.plot(x, y) plt.show() Its output is as follows − We will see lots of examples on using Matplotlib library of python in Data science work in the next chapters. Print Page Previous Next Advertisements ”;