In this chapter, we will be discussing about MXNet’s multi-dimensional array format called ndarray.
Handling data with NDArray
First, we are going see how we can handle data with NDArray. Following are the prerequisites for the same −
Prerequisites
To understand how we can handle data with this multi-dimensional array format, we need to fulfil the following prerequisites:
-
MXNet installed in a Python environment
-
Python 2.7.x or Python 3.x
Implementation Example
Let us understand the basic functionality with the help of an example given below −
First, we need to import MXNet and ndarray from MXNet as follows −
import mxnet as mx from mxnet import nd
Once we import the necessary libraries, we will go with the following basic functionalities:
A simple 1-D array with a python list
Example
x = nd.array([1,2,3,4,5,6,7,8,9,10]) print(x)
Output
The output is as mentioned below −
[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.] <NDArray 10 @cpu(0)>
A 2-D array with a python list
Example
y = nd.array([[1,2,3,4,5,6,7,8,9,10], [1,2,3,4,5,6,7,8,9,10], [1,2,3,4,5,6,7,8,9,10]]) print(y)
Output
The output is as stated below −
[[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.] [ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.] [ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]] <NDArray 3x10 @cpu(0)>
Creating an NDArray without any initialisation
Here, we will create a matrix with 3 rows and 4 columns by using .empty function. We will also use .full function, which will take an additional operator for what value you want to fill in the array.
Example
x = nd.empty((3, 4)) print(x) x = nd.full((3,4), 8) print(x)
Output
The output is given below −
[[0.000e+00 0.000e+00 0.000e+00 0.000e+00] [0.000e+00 0.000e+00 2.887e-42 0.000e+00] [0.000e+00 0.000e+00 0.000e+00 0.000e+00]] <NDArray 3x4 @cpu(0)> [[8. 8. 8. 8.] [8. 8. 8. 8.] [8. 8. 8. 8.]] <NDArray 3x4 @cpu(0)>
Matrix of all zeros with the .zeros function
Example
x = nd.zeros((3, 8)) print(x)
Output
The output is as follows −
[[0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0.]] <NDArray 3x8 @cpu(0)>
Matrix of all ones with the .ones function
Example
x = nd.ones((3, 8)) print(x)
Output
The output is mentioned below −
[[1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.]] <NDArray 3x8 @cpu(0)>
Creating array whose values are sampled randomly
Example
y = nd.random_normal(0, 1, shape=(3, 4)) print(y)
Output
The output is given below −
[[ 1.2673576 -2.0345826 -0.32537818 -1.4583491 ] [-0.11176403 1.3606371 -0.7889914 -0.17639421] [-0.2532185 -0.42614475 -0.12548696 1.4022992 ]] <NDArray 3x4 @cpu(0)>
Finding dimension of each NDArray
Example
y.shape
Output
The output is as follows −
(3, 4)
Finding the size of each NDArray
Example
y.size
Output
12
Finding the datatype of each NDArray
Example
y.dtype
Output
numpy.float32
NDArray Operations
In this section, we will introduce you to MXNet’s array operations. NDArray support large number of standard mathematical as well as In-place operations.
Standard Mathematical Operations
Following are standard mathematical operations supported by NDArray −
Element-wise addition
First, we need to import MXNet and ndarray from MXNet as follows:
import mxnet as mx
from mxnet import nd
x = nd.ones((3, 5))
y = nd.random_normal(0, 1, shape=(3, 5))
print(''x='', x)
print(''y='', y)
x = x + y
print(''x = x + y, x='', x)
Output
The output is given herewith −
x= [[1. 1. 1. 1. 1.] [1. 1. 1. 1. 1.] [1. 1. 1. 1. 1.]] <NDArray 3x5 @cpu(0)> y= [[-1.0554522 -1.3118273 -0.14674698 0.641493 -0.73820823] [ 2.031364 0.5932667 0.10228804 1.179526 -0.5444829 ] [-0.34249446 1.1086396 1.2756858 -1.8332436 -0.5289873 ]] <NDArray 3x5 @cpu(0)> x = x + y, x= [[-0.05545223 -0.3118273 0.853253 1.6414931 0.26179177] [ 3.031364 1.5932667 1.102288 2.1795259 0.4555171 ] [ 0.6575055 2.1086397 2.2756858 -0.8332436 0.4710127 ]] <NDArray 3x5 @cpu(0)>
Element-wise multiplication
Example
x = nd.array([1, 2, 3, 4]) y = nd.array([2, 2, 2, 1]) x * y
Output
You will see the following output−
[2. 4. 6. 4.] <NDArray 4 @cpu(0)>
Exponentiation
Example
nd.exp(x)
Output
When you run the code, you will see the following output:
[ 2.7182817 7.389056 20.085537 54.59815 ] <NDArray 4 @cpu(0)>
Matrix transpose to compute matrix-matrix product
Example
nd.dot(x, y.T)
Output
Given below is the output of the code −
[16.] <NDArray 1 @cpu(0)>
In-place Operations
Every time, in the above example, we ran an operation, we allocated a new memory to host its result.
For example, if we write A = A+B, we will dereference the matrix that A used to point to and instead point it at the newly allocated memory. Let us understand it with the example given below, using Python’s id() function −
print(''y='', y)
print(''id(y):'', id(y))
y = y + x
print(''after y=y+x, y='', y)
print(''id(y):'', id(y))
Output
Upon execution, you will receive the following output −
y= [2. 2. 2. 1.] <NDArray 4 @cpu(0)> id(y): 2438905634376 after y=y+x, y= [3. 4. 5. 5.] <NDArray 4 @cpu(0)> id(y): 2438905685664
In fact, we can also assign the result to a previously allocated array as follows −
print(''x='', x)
z = nd.zeros_like(x)
print(''z is zeros_like x, z='', z)
print(''id(z):'', id(z))
print(''y='', y)
z[:] = x + y
print(''z[:] = x + y, z='', z)
print(''id(z) is the same as before:'', id(z))
Output
The output is shown below −
x= [1. 2. 3. 4.] <NDArray 4 @cpu(0)> z is zeros_like x, z= [0. 0. 0. 0.] <NDArray 4 @cpu(0)> id(z): 2438905790760 y= [3. 4. 5. 5.] <NDArray 4 @cpu(0)> z[:] = x + y, z= [4. 6. 8. 9.] <NDArray 4 @cpu(0)> id(z) is the same as before: 2438905790760
From the above output, we can see that x+y will still allocate a temporary buffer to store the result before copying it to z. So now, we can perform operations in-place to make better use of memory and to avoid temporary buffer. To do this, we will specify the out keyword argument every operator support as follows −
print(''x='', x, ''is in id(x):'', id(x))
print(''y='', y, ''is in id(y):'', id(y))
print(''z='', z, ''is in id(z):'', id(z))
nd.elemwise_add(x, y, out=z)
print(''after nd.elemwise_add(x, y, out=z), x='', x, ''is in id(x):'', id(x))
print(''after nd.elemwise_add(x, y, out=z), y='', y, ''is in id(y):'', id(y))
print(''after nd.elemwise_add(x, y, out=z), z='', z, ''is in id(z):'', id(z))
Output
On executing the above program, you will get the following result −
x= [1. 2. 3. 4.] <NDArray 4 @cpu(0)> is in id(x): 2438905791152 y= [3. 4. 5. 5.] <NDArray 4 @cpu(0)> is in id(y): 2438905685664 z= [4. 6. 8. 9.] <NDArray 4 @cpu(0)> is in id(z): 2438905790760 after nd.elemwise_add(x, y, out=z), x= [1. 2. 3. 4.] <NDArray 4 @cpu(0)> is in id(x): 2438905791152 after nd.elemwise_add(x, y, out=z), y= [3. 4. 5. 5.] <NDArray 4 @cpu(0)> is in id(y): 2438905685664 after nd.elemwise_add(x, y, out=z), z= [4. 6. 8. 9.] <NDArray 4 @cpu(0)> is in id(z): 2438905790760
NDArray Contexts
In Apache MXNet, each array has a context and one context could be the CPU, whereas other contexts might be several GPUs. The things can get even worst, when we deploy the work across multiple servers. That’s why, we need to assign arrays to contexts intelligently. It will minimise the time spent transferring data between devices.
For example, try initialising an array as follows −
from mxnet import nd z = nd.ones(shape=(3,3), ctx=mx.cpu(0)) print(z)
Output
When you execute the above code, you should see the following output −
[[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] <NDArray 3x3 @cpu(0)>
We can copy the given NDArray from one context to another context by using the copyto() method as follows −
x_gpu = x.copyto(gpu(0)) print(x_gpu)
NumPy array vs. NDArray
We all the familiar with NumPy arrays but Apache MXNet offers its own array implementation named NDArray. Actually, it was initially designed to be similar to NumPy but there is a key difference −
The key difference is in the way calculations are executed in NumPy and NDArray. Every NDArray manipulation in MXNet is done in asynchronous and non-blocking way, which means that, when we write code like c = a * b, the function is pushed to the Execution Engine, which will start the calculation.
Here, a and b both are NDArrays. The benefit of using it is that, the function immediately returns back, and the user thread can continue execution despite the fact that the previous calculation may not have been completed yet.
Working of Execution Engine
If we talk about the working of execution engine, it builds the computation graph. The computation graph may reorder or combine some calculations, but it always honors dependency order.
For example, if there are other manipulation with ‘X’ done later in the programming code, the Execution Engine will start doing them once the result of ‘X’ is available. Execution engine will handle some important works for the users, such as writing of callbacks to start execution of subsequent code.
In Apache MXNet, with the help of NDArray, to get the result of computation we only need to access the resulting variable. The flow of the code will be blocked until the computation results are assigned to the resulting variable. In this way, it increases code performance while still supporting imperative programming mode.
Converting NDArray to NumPy Array
Let us learn how can we convert NDArray to NumPy Array in MXNet.
Combining higher-level operator with the help of few lower-level operators
Sometimes, we can assemble a higher-level operator by using the existing operators. One of the best examples of this is, the np.full_like() operator, which is not there in NDArray API. It can easily be replaced with a combination of existing operators as follows:
from mxnet import nd import numpy as np np_x = np.full_like(a=np.arange(7, dtype=int), fill_value=15) nd_x = nd.ones(shape=(7,)) * 15 np.array_equal(np_x, nd_x.asnumpy())
Output
We will get the output similar as follows −
True
Finding similar operator with different name and/or signature
Among all the operators, some of them have slightly different name, but they are similar in the terms of functionality. An example of this is nd.ravel_index() with np.ravel() functions. In the same way, some operators may have similar names, but they have different signatures. An example of this is np.split() and nd.split() are similar.
Let’s understand it with the following programming example:
def pad_array123(data, max_length): data_expanded = data.reshape(1, 1, 1, data.shape[0]) data_padded = nd.pad(data_expanded, mode=''constant'', pad_width=[0, 0, 0, 0, 0, 0, 0, max_length - data.shape[0]], constant_value=0) data_reshaped_back = data_padded.reshape(max_length) return data_reshaped_back pad_array123(nd.array([1, 2, 3]), max_length=10)
Output
The output is stated below −
[1. 2. 3. 0. 0. 0. 0. 0. 0. 0.] <NDArray 10 @cpu(0)>
Minimising impact of blocking calls
In some of the cases, we have to use either .asnumpy() or .asscalar() methods, but this will force MXNet to block the execution, until the result can be retrieved. We can minimise the impact of a blocking call by calling .asnumpy() or .asscalar() methods in the moment, when we think the calculation of this value is already done.
Implementation Example
Example
from __future__ import print_function
import mxnet as mx
from mxnet import gluon, nd, autograd
from mxnet.ndarray import NDArray
from mxnet.gluon import HybridBlock
import numpy as np
class LossBuffer(object):
"""
Simple buffer for storing loss value
"""
def __init__(self):
self._loss = None
def new_loss(self, loss):
ret = self._loss
self._loss = loss
return ret
@property
def loss(self):
return self._loss
net = gluon.nn.Dense(10)
ce = gluon.loss.SoftmaxCELoss()
net.initialize()
data = nd.random.uniform(shape=(1024, 100))
label = nd.array(np.random.randint(0, 10, (1024,)), dtype=''int32'')
train_dataset = gluon.data.ArrayDataset(data, label)
train_data = gluon.data.DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=2)
trainer = gluon.Trainer(net.collect_params(), optimizer=''sgd'')
loss_buffer = LossBuffer()
for data, label in train_data:
with autograd.record():
out = net(data)
# This call saves new loss and returns previous loss
prev_loss = loss_buffer.new_loss(ce(out, label))
loss_buffer.loss.backward()
trainer.step(data.shape[0])
if prev_loss is not None:
print("Loss: {}".format(np.mean(prev_loss.asnumpy())))
Output
The output is cited below:
Loss: 2.3373236656188965 Loss: 2.3656985759735107 Loss: 2.3613128662109375 Loss: 2.3197104930877686 Loss: 2.3054862022399902 Loss: 2.329197406768799 Loss: 2.318927526473999