Category: tensorflow

TensorFlow – Hidden Layers of Perceptron In this chapter, we will be focus on the network we will have to learn from known set of points called x and f(x). A single hidden layer will build this simple network. The code for the explanation of hidden layers of perceptron is as shown below − #Importing the necessary modules import tensorflow as tf import numpy as np import math, random import matplotlib.pyplot as plt np.random.seed(1000) function_to_learn = lambda x: np.cos(x) + 0.1*np.random.randn(*x.shape) layer_1_neurons = 10 NUM_points = 1000 #Training the parameters batch_size = 100 NUM_EPOCHS = 1500 all_x = np.float32(np.random.uniform(-2*math.pi, 2*math.pi, (1, NUM_points))).T np.random.shuffle(all_x) train_size = int(900) #Training the first 700 points in the given set x_training = all_x[:train_size] y_training = function_to_learn(x_training) #Training the last 300 points in the given set x_validation = all_x[train_size:] y_validation = function_to_learn(x_validation) plt.figure(1) plt.scatter(x_training, y_training, c = ”blue”, label = ”train”) plt.scatter(x_validation, y_validation, c = ”pink”, label = ”validation”) plt.legend() plt.show() X = tf.placeholder(tf.float32, [None, 1], name = “X”) Y = tf.placeholder(tf.float32, [None, 1], name = “Y”) #first layer #Number of neurons = 10 w_h = tf.Variable( tf.random_uniform([1, layer_1_neurons], minval = -1, maxval = 1, dtype = tf.float32)) b_h = tf.Variable(tf.zeros([1, layer_1_neurons], dtype = tf.float32)) h = tf.nn.sigmoid(tf.matmul(X, w_h) + b_h) #output layer #Number of neurons = 10 w_o = tf.Variable( tf.random_uniform([layer_1_neurons, 1], minval = -1, maxval = 1, dtype = tf.float32)) b_o = tf.Variable(tf.zeros([1, 1], dtype = tf.float32)) #build the model model = tf.matmul(h, w_o) + b_o #minimize the cost function (model – Y) train_op = tf.train.AdamOptimizer().minimize(tf.nn.l2_loss(model – Y)) #Start the Learning phase sess = tf.Session() sess.run(tf.initialize_all_variables()) errors = [] for i in range(NUM_EPOCHS): for start, end in zip(range(0, len(x_training), batch_size), range(batch_size, len(x_training), batch_size)): sess.run(train_op, feed_dict = {X: x_training[start:end], Y: y_training[start:end]}) cost = sess.run(tf.nn.l2_loss(model – y_validation), feed_dict = {X:x_validation}) errors.append(cost) if i%100 == 0: print(“epoch %d, cost = %g” % (i, cost)) plt.plot(errors,label=”MLP Function Approximation”) plt.xlabel(”epochs”) plt.ylabel(”cost”) plt.legend() plt.show() Output Following is the representation of function layer approximation − Here two data are represented in shape of W. The two data are: train and validation which are represented in distinct colors as visible in legend section.

TensorFlow – Multi-Layer Perceptron Learning Multi-Layer perceptron defines the most complicated architecture of artificial neural networks. It is substantially formed from multiple layers of perceptron. The diagrammatic representation of multi-layer perceptron learning is as shown below − MLP networks are usually used for supervised learning format. A typical learning algorithm for MLP networks is also called back propagation’s algorithm. Now, we will focus on the implementation with MLP for an image classification problem. # Import MINST data from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets(“/tmp/data/”, one_hot = True) import tensorflow as tf import matplotlib.pyplot as plt # Parameters learning_rate = 0.001 training_epochs = 20 batch_size = 100 display_step = 1 # Network Parameters n_hidden_1 = 256 # 1st layer num features n_hidden_2 = 256 # 2nd layer num features n_input = 784 # MNIST data input (img shape: 28*28) n_classes = 10 # MNIST total classes (0-9 digits) # tf Graph input x = tf.placeholder(“float”, [None, n_input]) y = tf.placeholder(“float”, [None, n_classes]) # weights layer 1 h = tf.Variable(tf.random_normal([n_input, n_hidden_1])) # bias layer 1 bias_layer_1 = tf.Variable(tf.random_normal([n_hidden_1])) # layer 1 layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, h), bias_layer_1)) # weights layer 2 w = tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])) # bias layer 2 bias_layer_2 = tf.Variable(tf.random_normal([n_hidden_2])) # layer 2 layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, w), bias_layer_2)) # weights output layer output = tf.Variable(tf.random_normal([n_hidden_2, n_classes])) # biar output layer bias_output = tf.Variable(tf.random_normal([n_classes])) # output layer output_layer = tf.matmul(layer_2, output) + bias_output # cost function cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits( logits = output_layer, labels = y)) #cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(output_layer, y)) # optimizer optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost) # optimizer = tf.train.GradientDescentOptimizer( learning_rate = learning_rate).minimize(cost) # Plot settings avg_set = [] epoch_set = [] # Initializing the variables init = tf.global_variables_initializer() # Launch the graph with tf.Session() as sess: sess.run(init) # Training cycle for epoch in range(training_epochs): avg_cost = 0. total_batch = int(mnist.train.num_examples / batch_size) # Loop over all batches for i in range(total_batch): batch_xs, batch_ys = mnist.train.next_batch(batch_size) # Fit training using batch data sess.run(optimizer, feed_dict = { x: batch_xs, y: batch_ys}) # Compute average loss avg_cost += sess.run(cost, feed_dict = {x: batch_xs, y: batch_ys}) / total_batch # Display logs per epoch step if epoch % display_step == 0: print Epoch:”, ”%04d” % (epoch + 1), “cost=”, “{:.9f}”.format(avg_cost) avg_set.append(avg_cost) epoch_set.append(epoch + 1) print “Training phase finished” plt.plot(epoch_set, avg_set, ”o”, label = ”MLP Training phase”) plt.ylabel(”cost”) plt.xlabel(”epoch”) plt.legend() plt.show() # Test model correct_prediction = tf.equal(tf.argmax(output_layer, 1), tf.argmax(y, 1)) # Calculate accuracy accuracy = tf.reduce_mean(tf.cast(correct_prediction, “float”)) print “Model Accuracy:”, accuracy.eval({x: mnist.test.images, y: mnist.test.labels}) The above line of code generates the following output −

TensorFlow – Introduction TensorFlow is a software library or framework, designed by the Google team to implement machine learning and deep learning concepts in the easiest manner. It combines the computational algebra of optimization techniques for easy calculation of many mathematical expressions. The official website of TensorFlow is mentioned below − Let us now consider the following important features of TensorFlow − It includes a feature of that defines, optimizes and calculates mathematical expressions easily with the help of multi-dimensional arrays called tensors. It includes a programming support of deep neural networks and machine learning techniques. It includes a high scalable feature of computation with various data sets. TensorFlow uses GPU computing, automating management. It also includes a unique feature of optimization of same memory and the data used. Why is TensorFlow So Popular? TensorFlow is well-documented and includes plenty of machine learning libraries. It offers a few important functionalities and methods for the same. TensorFlow is also called a “Google” product. It includes a variety of machine learning and deep learning algorithms. TensorFlow can train and run deep neural networks for handwritten digit classification, image recognition, word embedding and creation of various sequence models.

TensorFlow – TensorBoard Visualization TensorFlow includes a visualization tool, which is called the TensorBoard. It is used for analyzing Data Flow Graph and also used to understand machine-learning models. The important feature of TensorBoard includes a view of different types of statistics about the parameters and details of any graph in vertical alignment. Deep neural network includes up to 36,000 nodes. TensorBoard helps in collapsing these nodes in high-level blocks and highlighting the identical structures. This allows better analysis of graph focusing on the primary sections of the computation graph. The TensorBoard visualization is said to be very interactive where a user can pan, zoom and expand the nodes to display the details. The following schematic diagram representation shows the complete working of TensorBoard visualization − The algorithms collapse nodes into high-level blocks and highlight the specific groups with identical structures, which separate high-degree nodes. The TensorBoard thus created is useful and is treated equally important for tuning a machine learning model. This visualization tool is designed for the configuration log file with summary information and details that need to be displayed. Let us focus on the demo example of TensorBoard visualization with the help of the following code − import tensorflow as tf # Constants creation for TensorBoard visualization a = tf.constant(10,name = “a”) b = tf.constant(90,name = “b”) y = tf.Variable(a+b*2,name = ”y”) model = tf.initialize_all_variables() #Creation of model with tf.Session() as session: merged = tf.merge_all_summaries() writer = tf.train.SummaryWriter(“/tmp/tensorflowlogs”,session.graph) session.run(model) print(session.run(y)) The following table shows the various symbols of TensorBoard visualization used for the node representation −

Discuss TensorFlow TensorFlow is an open source machine learning framework for all developers. It is used for implementing machine learning and deep learning applications. To develop and research on fascinating ideas on artificial intelligence, Google team created TensorFlow. TensorFlow is designed in Python programming language, hence it is considered an easy to understand framework.

TensorFlow – CNN And RNN Difference In this chapter, we will focus on the difference between CNN and RNN − CNN RNN It is suitable for spatial data such as images. RNN is suitable for temporal data, also called sequential data. CNN is considered to be more powerful than RNN. RNN includes less feature compatibility when compared to CNN. This network takes fixed size inputs and generates fixed size outputs. RNN can handle arbitrary input/output lengths. CNN is a type of feed-forward artificial neural network with variations of multilayer perceptrons designed to use minimal amounts of preprocessing. RNN unlike feed forward neural networks – can use their internal memory to process arbitrary sequences of inputs. CNNs use connectivity pattern between the neurons. This is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Recurrent neural networks use time-series information – what a user spoke last will impact what he/she will speak next. CNNs are ideal for images and video processing. RNNs are ideal for text and speech analysis. Following illustration shows the schematic representation of CNN and RNN −

TensorFlow – XOR Implementation In this chapter, we will learn about the XOR implementation using TensorFlow. Before starting with XOR implementation in TensorFlow, let us see the XOR table values. This will help us understand encryption and decryption process. A B A XOR B 0 0 0 0 1 1 1 0 1 1 1 0 XOR Cipher encryption method is basically used to encrypt data which is hard to crack with brute force method, i.e., by generating random encryption keys which match the appropriate key. The concept of implementation with XOR Cipher is to define a XOR encryption key and then perform XOR operation of the characters in the specified string with this key, which a user tries to encrypt. Now we will focus on XOR implementation using TensorFlow, which is mentioned below − #Declaring necessary modules import tensorflow as tf import numpy as np “”” A simple numpy implementation of a XOR gate to understand the backpropagation algorithm “”” x = tf.placeholder(tf.float64,shape = [4,2],name = “x”) #declaring a place holder for input x y = tf.placeholder(tf.float64,shape = [4,1],name = “y”) #declaring a place holder for desired output y m = np.shape(x)[0]#number of training examples n = np.shape(x)[1]#number of features hidden_s = 2 #number of nodes in the hidden layer l_r = 1#learning rate initialization theta1 = tf.cast(tf.Variable(tf.random_normal([3,hidden_s]),name = “theta1”),tf.float64) theta2 = tf.cast(tf.Variable(tf.random_normal([hidden_s+1,1]),name = “theta2”),tf.float64) #conducting forward propagation a1 = tf.concat([np.c_[np.ones(x.shape[0])],x],1) #the weights of the first layer are multiplied by the input of the first layer z1 = tf.matmul(a1,theta1) #the input of the second layer is the output of the first layer, passed through the activation function and column of biases is added a2 = tf.concat([np.c_[np.ones(x.shape[0])],tf.sigmoid(z1)],1) #the input of the second layer is multiplied by the weights z3 = tf.matmul(a2,theta2) #the output is passed through the activation function to obtain the final probability h3 = tf.sigmoid(z3) cost_func = -tf.reduce_sum(y*tf.log(h3)+(1-y)*tf.log(1-h3),axis = 1) #built in tensorflow optimizer that conducts gradient descent using specified learning rate to obtain theta values optimiser = tf.train.GradientDescentOptimizer(learning_rate = l_r).minimize(cost_func) #setting required X and Y values to perform XOR operation X = [[0,0],[0,1],[1,0],[1,1]] Y = [[0],[1],[1],[0]] #initializing all variables, creating a session and running a tensorflow session init = tf.global_variables_initializer() sess = tf.Session() sess.run(init) #running gradient descent for each iteration and printing the hypothesis obtained using the updated theta values for i in range(100000): sess.run(optimiser, feed_dict = {x:X,y:Y})#setting place holder values using feed_dict if i%100==0: print(“Epoch:”,i) print(“Hyp:”,sess.run(h3,feed_dict = {x:X,y:Y})) The above line of code generates an output as shown in the screenshot below −

TensorFlow – Optimizers Optimizers are the extended class, which include added information to train a specific model. The optimizer class is initialized with given parameters but it is important to remember that no Tensor is needed. The optimizers are used for improving speed and performance for training a specific model. The basic optimizer of TensorFlow is − tf.train.Optimizer This class is defined in the specified path of tensorflow/python/training/optimizer.py. Following are some optimizers in Tensorflow − Stochastic Gradient descent Stochastic Gradient descent with gradient clipping Momentum Nesterov momentum Adagrad Adadelta RMSProp Adam Adamax SMORMS3 We will focus on the Stochastic Gradient descent. The illustration for creating optimizer for the same is mentioned below − def sgd(cost, params, lr = np.float32(0.01)): g_params = tf.gradients(cost, params) updates = [] for param, g_param in zip(params, g_params): updates.append(param.assign(param – lr*g_param)) return updates The basic parameters are defined within the specific function. In our subsequent chapter, we will focus on Gradient Descent Optimization with implementation of optimizers.

TensorFlow – Gradient Descent Optimization Gradient descent optimization is considered to be an important concept in data science. Consider the steps shown below to understand the implementation of gradient descent optimization − Step 1 Include necessary modules and declaration of x and y variables through which we are going to define the gradient descent optimization. import tensorflow as tf x = tf.Variable(2, name = ”x”, dtype = tf.float32) log_x = tf.log(x) log_x_squared = tf.square(log_x) optimizer = tf.train.GradientDescentOptimizer(0.5) train = optimizer.minimize(log_x_squared) Step 2 Initialize the necessary variables and call the optimizers for defining and calling it with respective function. init = tf.initialize_all_variables() def optimize(): with tf.Session() as session: session.run(init) print(“starting at”, “x:”, session.run(x), “log(x)^2:”, session.run(log_x_squared)) for step in range(10): session.run(train) print(“step”, step, “x:”, session.run(x), “log(x)^2:”, session.run(log_x_squared)) optimize() The above line of code generates an output as shown in the screenshot below − We can see that the necessary epochs and iterations are calculated as shown in the output.

TensorFlow – Linear Regression In this chapter, we will focus on the basic example of linear regression implementation using TensorFlow. Logistic regression or linear regression is a supervised machine learning approach for the classification of order discrete categories. Our goal in this chapter is to build a model by which a user can predict the relationship between predictor variables and one or more independent variables. The relationship between these two variables is cons −idered linear. If y is the dependent variable and x is considered as the independent variable, then the linear regression relationship of two variables will look like the following equation − Y = Ax+b We will design an algorithm for linear regression. This will allow us to understand the following two important concepts − Cost Function Gradient descent algorithms The schematic representation of linear regression is mentioned below − The graphical view of the equation of linear regression is mentioned below − Steps to design an algorithm for linear regression We will now learn about the steps that help in designing an algorithm for linear regression. Step 1 It is important to import the necessary modules for plotting the linear regression module. We start importing the Python library NumPy and Matplotlib. import numpy as np import matplotlib.pyplot as plt Step 2 Define the number of coefficients necessary for logistic regression. number_of_points = 500 x_point = [] y_point = [] a = 0.22 b = 0.78 Step 3 Iterate the variables for generating 300 random points around the regression equation − Y = 0.22x+0.78 for i in range(number_of_points): x = np.random.normal(0.0,0.5) y = a*x + b +np.random.normal(0.0,0.1) x_point.append([x]) y_point.append([y]) Step 4 View the generated points using Matplotlib. fplt.plot(x_point,y_point, ”o”, label = ”Input Data”) plt.legend() plt.show() The complete code for logistic regression is as follows − import numpy as np import matplotlib.pyplot as plt number_of_points = 500 x_point = [] y_point = [] a = 0.22 b = 0.78 for i in range(number_of_points): x = np.random.normal(0.0,0.5) y = a*x + b +np.random.normal(0.0,0.1) x_point.append([x]) y_point.append([y]) plt.plot(x_point,y_point, ”o”, label = ”Input Data”) plt.legend() plt.show() The number of points which is taken as input is considered as input data.