# Tensor Operations

## Prerequisites

### Tensor Operations

#### Introduction

• A tensor operation is an operation that either:
• Uses a tensor as one or more of it's inputs
• Or produces a tensor as it's output

#### Component-wise Operations

• A component-wise (or element-wise) operation acts on each value individually.
• Component-wise operations usually produce tensors of the same shape as the input tensor.
• Tensor addition operates on two tensors of the same shape, and produces a tensor also of the same shape.
• Each value in the output tensor is the sum of the values in the input tensors at the same location.
• Code (slow):
``````import numpy as np

assert a.shape == b.shape

c = np.empty(a.shape)
for index in np.ndindex(a.shape):
c[index] = a[index] + b[index]

return c

a = np.array([[1, 2],
[3, 4]])

b = np.array([[5, 6],
[7, 8]])

print(c)
# prints [[ 6  8]
#         [10 12]]
``````
• Code (fast):
``````import numpy as np

a = np.array([[1, 2],
[3, 4]])

b = np.array([[5, 6],
[7, 8]])

c = a + b
print(c)
# prints [[ 6  8]
#         [10 12]]
``````
##### Relu
• relu stands for rectified linear unit.
• If an element is positive, it is used as-is.
• If an element is negative, 0 is used.
• Code (slow):
``````import numpy as np

def relu(a):
r = np.empty(a.shape)
for index in np.ndindex(a.shape):
r[index] = max(a[index], 0)
return r

a = np.array([[1, -2],
[3, -4]])

b = relu(a)
print(b)
# prints [[1 0]
#         [3 0]]
``````
• Code (fast):
``````import numpy as np

a = np.array([[1, -2],
[3, -4]])

b = np.maximum(a, 0)
print(b)
# prints [[1 0]
#         [3 0]]
``````

• Component-wise operations such as addition rely on the two input tensors having the same shape.
• However sometimes it is necessary to perform operations on vectors with different shapes.
• Broadcasting is an operation which increases the rank of a tensor.
• Add a new axis, which by default has a length of 1.
• Extend the length of the axis by cloning the values.
• Code:
``````import numpy as np

a = np.array([[1, 2],
[3, 4]])
print(a.shape)  # prints (2, 2)

a = np.expand_dims(a, axis=0)
print(a.shape)  # prints (1, 2, 2)

a = np.concatenate([a] * 3, axis=0)
print(a.shape)  # prints (3, 2, 2)

print(a)
# prints:
#   [[[1 2]
#     [3 4]]
#    [[1 2]
#     [3 4]]
#    [[1 2]
#     [3 4]]]``````
• Numpy operations automatically perform broadcasting if there inputs are different shapes:
``````import numpy as np

a = np.array([[1, 1],
[3, 2]])
b = np.array([2.5, 1.5])

c = np.maximum(a, b)
print(c)
# prints:
#   [[2.5 2.5]
#    [3.  4. ]]``````
• For broadcasting to be successful, the shape of the larger ranked tensor should end with the shape of the snaller ranked tensor:
``````import numpy as np

smallest_rank = min(a.ndim, b.ndim)
return a.shape[-smallest_rank:] == b.shape[-smallest_rank:]

a = np.array([[1, 2],
[3, 4]])

canBroadcast(np.array([5, 6, 7]), a)  # false``````

#### Tensor Product

• Tensor product of two 1D tensors:
• When both tensors are vectors, their product produces a scalar.
• To find the product, multiply the corresponding elements and sum the results.
• Code (slow):
``````import numpy as np

def product(a, b):
assert a.shape == b.shape
assert a.ndim == 1

total = 0
for i in range(len(a)):
total += a[i] * b[i]

a = np.array([3, 6, 2])
b = np.array([1, 3, 8])

c = product(a, b)
print(c) # prints '37'``````
• Code (fast)
``````import numpy as np

a = np.array([3, 6, 2])
b = np.array([1, 3, 8])

c = np.dot(a, b)
print(c) # prints '37'``````
• #### Tensor product of two 2D tensors:

• When both tensors are matrices, their product produces a matrix.
• The size of the new matrix has the number of columns from the first matrix, and the number of rows from the second matrix
• Each value is the result of applying the tensor product of a row from the first matrix with a column from the second matrix.
• Code (slow):
``````import numpy as np

def product(a, b):
assert a.ndim == 2
assert b.ndim == 2
assert a.shape[1] == b.shape[0]

c = np.empty((a.shape[0], b.shape[1]))
for index in np.ndindex(c.shape):
row = a[index[0], :]
column = b[:, index[1]]
c[index] = np.dot(row, column)

return c

a = np.array([[1, 4], [3, 7], [8, 4]])
b = np.array([[3, 6, 2], [5, 1, 10]])

c = product(a, b)
print(c)
# prints
#   [[23. 10. 42.]
#    [44. 25. 76.]
#    [44. 52. 56.]]``````
• Code (fast):
``````import numpy as np

a = np.array([[1, 4], [3, 7], [8, 4]])
b = np.array([[3, 6, 2], [5, 1, 10]])

c = np.dot(a, b)
print(c)
# prints
#   [[23. 10. 42.]
#    [44. 25. 76.]
#    [44. 52. 56.]]``````
• The tensor product also works for tensors with higher ranks.
• If the ranks of two tensors are different then the normal broadcasting rules apply.

#### Reshaping

• Reshaping is a tensor produces a new tensor with the same number of values, but a different shape.
``````import numpy as np

x = np.array([[1, 2],
[3, 4],
[5, 6]])

x = x.reshape((2, 3))
print(x)
# prints:
#   [[1 2 3]
#    [4 5 6]]

x = x.reshape((1, 6))
print(x)
# prints:
#   [[1 2 3 4 5 6]]``````
• Reshaping a tensor can also involve changing the rank:
``````import numpy as np

x = np.array([[1, 2],
[3, 4],
[5, 6]])

x = x.reshape(6)
print(x)
# prints:
#   [1 2 3 4 5 6]``````

#### Transposing

• Transposing a matrix involves converting all its rows into columns:
``````import numpy as np

x = np.array([[1, 2],
[3, 4],
[5, 6]])

x = x.transpose()
print(x)
# prints:
#   [[1 3 5]
#    [2 4 6]]``````