Mahout – Recommendation ”; Previous Next This chapter covers the popular machine learning technique called recommendation, its mechanisms, and how to write an application implementing Mahout recommendation. Recommendation Ever wondered how Amazon comes up with a list of recommended items to draw your attention to a particular product that you might be interested in! Suppose you want to purchase the book “Mahout in Action” from Amazon: Along with the selected product, Amazon also displays a list of related recommended items, as shown below. Such recommendation lists are produced with the help of recommender engines. Mahout provides recommender engines of several types such as: user-based recommenders, item-based recommenders, and several other algorithms. Mahout Recommender Engine Mahout has a non-distributed, non-Hadoop-based recommender engine. You should pass a text document having user preferences for items. And the output of this engine would be the estimated preferences of a particular user for other items. Example Consider a website that sells consumer goods such as mobiles, gadgets, and their accessories. If we want to implement the features of Mahout in such a site, then we can build a recommender engine. This engine analyzes past purchase data of the users and recommends new products based on that. The components provided by Mahout to build a recommender engine are as follows: DataModel UserSimilarity ItemSimilarity UserNeighborhood Recommender From the data store, the data model is prepared and is passed as an input to the recommender engine. The Recommender engine generates the recommendations for a particular user. Given below is the architecture of recommender engine. Architecture of Recommender Engine Building a Recommender using Mahout Here are the steps to develop a simple recommender: Step1: Create DataModel Object The constructor of PearsonCorrelationSimilarity class requires a data model object, which holds a file that contains the Users, Items, and Preferences details of a product. Here is the sample data model file: 1,00,1.0 1,01,2.0 1,02,5.0 1,03,5.0 1,04,5.0 2,00,1.0 2,01,2.0 2,05,5.0 2,06,4.5 2,02,5.0 3,01,2.5 3,02,5.0 3,03,4.0 3,04,3.0 4,00,5.0 4,01,5.0 4,02,5.0 4,03,0.0 The DataModel object requires the file object, which contains the path of the input file. Create the DataModel object as shown below. DataModel datamodel = new FileDataModel(new File(“input file”)); Step2: Create UserSimilarity Object Create UserSimilarity object using PearsonCorrelationSimilarity class as shown below: UserSimilarity similarity = new PearsonCorrelationSimilarity(datamodel); Step3: Create UserNeighborhood object This object computes a “neighborhood” of users like a given user. There are two types of neighborhoods: NearestNUserNeighborhood – This class computes a neighborhood consisting of the nearest n users to a given user. “Nearest” is defined by the given UserSimilarity. ThresholdUserNeighborhood – This class computes a neighborhood consisting of all the users whose similarity to the given user meets or exceeds a certain threshold. Similarity is defined by the given UserSimilarity. Here we are using ThresholdUserNeighborhood and set the limit of preference to 3.0. UserNeighborhood neighborhood = new ThresholdUserNeighborhood(3.0, similarity, model); Step4: Create Recommender Object Create UserbasedRecomender object. Pass all the above created objects to its constructor as shown below. UserBasedRecommender recommender = new GenericUserBasedRecommender(model, neighborhood, similarity); Step5: Recommend Items to a User Recommend products to a user using the recommend() method of Recommender interface. This method requires two parameters. The first represents the user id of the user to whom we need to send the recommendations, and the second represents the number of recommendations to be sent. Here is the usage of recommender() method: List<RecommendedItem> recommendations = recommender.recommend(2, 3); for (RecommendedItem recommendation : recommendations) { System.out.println(recommendation); } Example Program Given below is an example program to set recommendation. Prepare the recommendations for the user with user id 2. import java.io.File; import java.util.List; import org.apache.mahout.cf.taste.impl.model.file.FileDataModel; import org.apache.mahout.cf.taste.impl.neighborhood.ThresholdUserNeighborhood; import org.apache.mahout.cf.taste.impl.recommender.GenericUserBasedRecommender; import org.apache.mahout.cf.taste.impl.similarity.PearsonCorrelationSimilarity; import org.apache.mahout.cf.taste.model.DataModel; import org.apache.mahout.cf.taste.neighborhood.UserNeighborhood; import org.apache.mahout.cf.taste.recommender.RecommendedItem; import org.apache.mahout.cf.taste.recommender.UserBasedRecommender; import org.apache.mahout.cf.taste.similarity.UserSimilarity; public class Recommender { public static void main(String args[]){ try{ //Creating data model DataModel datamodel = new FileDataModel(new File(“data”)); //data //Creating UserSimilarity object. UserSimilarity usersimilarity = new PearsonCorrelationSimilarity(datamodel); //Creating UserNeighbourHHood object. UserNeighborhood userneighborhood = new ThresholdUserNeighborhood(3.0, usersimilarity, datamodel); //Create UserRecomender UserBasedRecommender recommender = new GenericUserBasedRecommender(datamodel, userneighborhood, usersimilarity); List<RecommendedItem> recommendations = recommender.recommend(2, 3); for (RecommendedItem recommendation : recommendations) { System.out.println(recommendation); } }catch(Exception e){} } } Compile the program using the following commands: javac Recommender.java java Recommender It should produce the following output: RecommendedItem [item:3, value:4.5] RecommendedItem [item:4, value:4.0] Print Page Previous Next Advertisements ”;
Category: mahout
Mahout – Discussion
Mahout – Discussion ”; Previous Next Apache Mahout is an open source project that is primarily used in producing scalable machine learning algorithms. This brief tutorial provides a quick introduction to Apache Mahout and explains how it can be applied to make recommendations and organize documents in more useable clusters. Please enable JavaScript to view the comments powered by Disqus. Print Page Previous Next Advertisements ”;
Mahout – Classification
Mahout – Classification ”; Previous Next What is Classification? Classification is a machine learning technique that uses known data to determine how the new data should be classified into a set of existing categories. For example, iTunes application uses classification to prepare playlists. Mail service providers such as Yahoo! and Gmail use this technique to decide whether a new mail should be classified as a spam. The categorization algorithm trains itself by analyzing user habits of marking certain mails as spams. Based on that, the classifier decides whether a future mail should be deposited in your inbox or in the spams folder. How Classification Works While classifying a given set of data, the classifier system performs the following actions: Initially a new data model is prepared using any of the learning algorithms. Then the prepared data model is tested. Thereafter, this data model is used to evaluate the new data and to determine its class. Applications of Classification Credit card fraud detection – The Classification mechanism is used to predict credit card frauds. Using historical information of previous frauds, the classifier can predict which future transactions may turn into frauds. Spam e-mails – Depending on the characteristics of previous spam mails, the classifier determines whether a newly encountered e-mail should be sent to the spam folder. Naive Bayes Classifier Mahout uses the Naive Bayes classifier algorithm. It uses two implementations: Distributed Naive Bayes classification Complementary Naive Bayes classification Naive Bayes is a simple technique for constructing classifiers. It is not a single algorithm for training such classifiers, but a family of algorithms. A Bayes classifier constructs models to classify problem instances. These classifications are made using the available data. An advantage of naive Bayes is that it only requires a small amount of training data to estimate the parameters necessary for classification. For some types of probability models, naive Bayes classifiers can be trained very efficiently in a supervised learning setting. Despite its oversimplified assumptions, naive Bayes classifiers have worked quite well in many complex real-world situations. Procedure of Classification The following steps are to be followed to implement Classification: Generate example data Create sequence files from data Convert sequence files to vectors Train the vectors Test the vectors Step1: Generate Example Data Generate or download the data to be classified. For example, you can get the 20 newsgroups example data from the following link: http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz Create a directory for storing input data. Download the example as shown below. $ mkdir classification_example $ cd classification_example $tar xzvf 20news-bydate.tar.gz wget http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz Step 2: Create Sequence Files Create sequence file from the example using seqdirectory utility. The syntax to generate sequence is given below: mahout seqdirectory -i <input file path> -o <output directory> Step 3: Convert Sequence Files to Vectors Create vector files from sequence files using seq2parse utility. The options of seq2parse utility are given below: $MAHOUT_HOME/bin/mahout seq2sparse –analyzerName (-a) analyzerName The class name of the analyzer –chunkSize (-chunk) chunkSize The chunkSize in MegaBytes. –output (-o) output The directory pathname for o/p –input (-i) input Path to job input directory. Step 4: Train the Vectors Train the generated vectors using the trainnb utility. The options to use trainnb utility are given below: mahout trainnb -i ${PATH_TO_TFIDF_VECTORS} -el -o ${PATH_TO_MODEL}/model -li ${PATH_TO_MODEL}/labelindex -ow -c Step 5: Test the Vectors Test the vectors using testnb utility. The options to use testnb utility are given below: mahout testnb -i ${PATH_TO_TFIDF_TEST_VECTORS} -m ${PATH_TO_MODEL}/model -l ${PATH_TO_MODEL}/labelindex -ow -o ${PATH_TO_OUTPUT} -c -seq Print Page Previous Next Advertisements ”;
Mahout – Environment
Mahout – Environment ”; Previous Next This chapter teaches you how to setup mahout. Java and Hadoop are the prerequisites of mahout. Below given are the steps to download and install Java, Hadoop, and Mahout. Pre-Installation Setup Before installing Hadoop into Linux environment, we need to set up Linux using ssh (Secure Shell). Follow the steps mentioned below for setting up the Linux environment. Creating a User It is recommended to create a separate user for Hadoop to isolate the Hadoop file system from the Unix file system. Follow the steps given below to create a user: Open root using the command “su”. Create a user from the root account using the command “useradd username”. Now you can open an existing user account using the command “su username”. Open the Linux terminal and type the following commands to create a user. $ su password: # useradd hadoop # passwd hadoop New passwd: Retype new passwd SSH Setup and Key Generation SSH setup is required to perform different operations on a cluster such as starting, stopping, and distributed daemon shell operations. To authenticate different users of Hadoop, it is required to provide public/private key pair for a Hadoop user and share it with different users. The following commands are used to generate a key value pair using SSH, copy the public keys form id_rsa.pub to authorized_keys, and provide owner, read and write permissions to authorized_keys file respectively. $ ssh-keygen -t rsa $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys $ chmod 0600 ~/.ssh/authorized_keys Verifying ssh ssh localhost Installing Java Java is the main prerequisite for Hadoop and HBase. First of all, you should verify the existence of Java in your system using “java -version”. The syntax of Java version command is given below. $ java -version It should produce the following output. java version “1.7.0_71” Java(TM) SE Runtime Environment (build 1.7.0_71-b13) Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode) If you don’t have Java installed in your system, then follow the steps given below for installing Java. Step 1 Download java (JDK <latest version> – X64.tar.gz) by visiting the following link: Oracle Then jdk-7u71-linux-x64.tar.gz is downloaded onto your system. Step 2 Generally, you find the downloaded Java file in the Downloads folder. Verify it and extract the jdk-7u71-linux-x64.gz file using the following commands. $ cd Downloads/ $ ls jdk-7u71-linux-x64.gz $ tar zxf jdk-7u71-linux-x64.gz $ ls jdk1.7.0_71 jdk-7u71-linux-x64.gz Step 3 To make Java available to all the users, you need to move it to the location “/usr/local/”. Open root, and type the following commands. $ su password: # mv jdk1.7.0_71 /usr/local/ # exit Step 4 For setting up PATH and JAVA_HOME variables, add the following commands to ~/.bashrc file. export JAVA_HOME=/usr/local/jdk1.7.0_71 export PATH= $PATH:$JAVA_HOME/bin Now, verify the java -version command from terminal as explained above. Downloading Hadoop After installing Java, you need to install Hadoop initially. Verify the existence of Hadoop using “Hadoop version” command as shown below. hadoop version It should produce the following output: Hadoop 2.6.0 Compiled by jenkins on 2014-11-13T21:10Z Compiled with protoc 2.5.0 From source with checksum 18e43357c8f927c0695f1e9522859d6a This command was run using /home/hadoop/hadoop/share/hadoop/common/hadoopcommon-2.6.0.jar If your system is unable to locate Hadoop, then download Hadoop and have it installed on your system. Follow the commands given below to do so. Download and extract hadoop-2.6.0 from apache software foundation using the following commands. $ su password: # cd /usr/local # wget http://mirrors.advancedhosters.com/apache/hadoop/common/hadoop- 2.6.0/hadoop-2.6.0-src.tar.gz # tar xzf hadoop-2.6.0-src.tar.gz # mv hadoop-2.6.0/* hadoop/ # exit Installing Hadoop Install Hadoop in any of the required modes. Here, we are demonstrating HBase functionalities in pseudo-distributed mode, therefore install Hadoop in pseudo-distributed mode. Follow the steps given below to install Hadoop 2.4.1 on your system. Step 1: Setting up Hadoop You can set Hadoop environment variables by appending the following commands to ~/.bashrc file. export HADOOP_HOME=/usr/local/hadoop export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HADOOP_INSTALL=$HADOOP_HOME Now, apply all changes into the currently running system. $ source ~/.bashrc Step 2: Hadoop Configuration You can find all the Hadoop configuration files at the location “$HADOOP_HOME/etc/hadoop”. It is required to make changes in those configuration files according to your Hadoop infrastructure. $ cd $HADOOP_HOME/etc/hadoop In order to develop Hadoop programs in Java, you need to reset the Java environment variables in hadoop-env.sh file by replacing JAVA_HOME value with the location of Java in your system. export JAVA_HOME=/usr/local/jdk1.7.0_71 Given below are the list of files which you have to edit to configure Hadoop. core-site.xml The core-site.xml file contains information such as the port number used for Hadoop instance, memory allocated for file system, memory limit for storing data, and the size of Read/Write buffers. Open core-site.xml and add the following property in between the <configuration>, </configuration> tags: <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration> hdfs-site.xm The hdfs-site.xml file contains information such as the value of replication data, namenode path, and datanode paths of your local file systems. It means the place where you want to store the Hadoop infrastructure. Let us assume the following data: dfs.replication (data replication value) = 1 (In the below given path /hadoop/ is the user name. hadoopinfra/hdfs/namenode is the directory created by hdfs file system.) namenode path = //home/hadoop/hadoopinfra/hdfs/namenode (hadoopinfra/hdfs/datanode is the directory created by hdfs file system.) datanode path = //home/hadoop/hadoopinfra/hdfs/datanode Open this file and add the following properties in between the <configuration>, </configuration> tags in this file. <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.name.dir</name> <value>file:///home/hadoop/hadoopinfra/hdfs/namenode</value> </property> <property> <name>dfs.data.dir</name> <value>file:///home/hadoop/hadoopinfra/hdfs/datanode</value> </property> </configuration> Note: In the above file, all the property values are user defined. You can make changes according to your Hadoop infrastructure. mapred-site.xml This file is used to configure yarn into Hadoop. Open mapred-site.xml file and add the following property in between the <configuration>, </configuration> tags in this file. <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> mapred-site.xml This file is used to specify which MapReduce framework we are using. By default, Hadoop contains a template of mapred-site.xml. First of all, it is required to copy the file from mapred-site.xml.template to mapred-site.xml file using the following command. $