Getting a Handle on Vector for Machine Learning

Getting a Handle on Vector for Machine Learning

On Thursday, March 28 2019

The word vector comes from the Latin for “carrier”, reason might be the nature of vector to move from one location to another. In physics, a Vector is an object that moves around a physical space. In Data Science and Machine Learning we generalize this idea to think vector as a list of attributes of an object. So Vectors are used to represent objects, observations and predictions.

Say we want to represent a house having 2 bedrooms, 1 bathroom, 150-meter square and cost around $ 20,000. Based on the information we know about this house, we can represent the house as a vector:

$$house = \begin{pmatrix} 2 \\ 1 \\ 150 \\ 20000 \end{pmatrix}$$

Based on this vector, our machine learning predicts that there is 45% chance that this house will give me profit if I buy, 5% chance that it will be a bad deal(loss) and 50% chance that the price of house will not change at all. We can represent this prediction in a vector as:

$$class\_probabilities = \begin{pmatrix} 0.45 \\ 0.05 \\ 0.5 \end{pmatrix}$$

Vectors in Python

There are multiple ways we can define a vector in Python, the simplest on being a list:

house = [2, 1, 150, 20000]

Python Lists are very good, but in Machine Learning we have deal with lot of scientific calculations,so it is much convenient to represent a vector in Python as a NumPy array as Numpy array does the thing what we expect when we add, multiply vectors.

import numpy as np
house = np.array([2, 1, 150, 20000])

Vector Arithmetic

In this section, we will illustrate simple vector-vector arithmetic, where all operations are performed element-wise.

Vector Addition

Two vector of equal length when added produce a new vector of the same length. Each element of the new vector at index i is calculated as the sum of the other element at that particular index i.

import numpy as np

house1 = np.array([2,1,150, 20000])
house2 = np.array([2, 1, 200, 29000])

house3 = house1 + house2

print(house3)
[    4,     2,   350, 49000]

Vector Subtraction

Two vectors of equal length can be subtracted to produce a new vector of the same length. Each element of the new vector at index i is calculated as the difference of the other elements at that particular index i.

Vector Multiplication

Two vector of equal length can be multiplied together to create a new vector of same length. Like with addition and subtraction, vector multiplication is done element wise.

$$a=b * c$$
import numpy as np

a = np.array([1, 2, 3])

b = np.array([2, 3, 4])

c = a * b

print(c)

[ 2  6 12]

Vector Division

Two vector of equal length can be divided element wise to calculate a new vector of same length.

Vector Norms

The term Norm is typically used to represent the length or magnitude of a vector. The norm of vector v is denoted by ||v||.

Norm Properties

1. For any vector \(\mathbf{v}\), \(\mathbf{||v||}\) is a non-negative real number.$$\\$$ 2. For any vector \(\mathbf{v}\), \(\mathbf{||v||}\) is zero if and only if \(\mathbf{v}\) is a zero vector.$$\\$$ 3. Triangle inequality, For any vectors \(\mathbf{x}\) and \(\mathbf{y}\), \[||x + y|| \le ||x|| + ||y||\]$$\\$$ 4. For any vector \(\mathbf{v}\) and any scalar \(\alpha \), \[||\alpha v|| = |\alpha |||v||\]$$\\$$

Different Vector Norms

1. Manhattan Norm or Taxicab Norm

The \(L^1\) Norm or Manhattan Norm or Taxicab Norm of vector \(\mathbf{v}\) \( \in \) \(R^n\) is defined as \[||v||_1 = \sum\limits_{i = 1}^n {{|v_i|}} \]. $$\\$$ The \(L^1\) norm is calculated as the sum of the absolute vector values, where the absolute value of a scalar uses the notation \(\mathbf{|v|}\). \[||v||_1 = |v_1| + |v_2| + |v_3| + .. + |v_n|\]

2. Euclidean Norm

The \(L^2\) Norm or Euclidean of vector \(\mathbf{v}\) \( \in \) \(R^n\) is defined as \[||v||_2 = \sqrt{\sum\limits_{i = 1}^n {v_i^2}} \]. $$\\$$ The \(L^2\) norm simply calculates the Euclidean distance from the origin to the point identified by \(\mathbf{v}\). $$\\$$ The \(L^2\) norm is calculated as the square root of the sum of the squared vector values \[||v||_2 = \sqrt {v_1^2 + v_2^2 + v_3^2 + .. + v_n^2} \]

3. Vector Max Norm

The \(L^\infty\) norm or vector max norm of vector \(\mathbf{v}\) \( \in \) \(R^n\) is defined as \[||v||_\infty = \mathop {\max |v_i|}\limits_i \] The \(L^\infty\) norm is calculated by returning the the maximum value of vector as \[||v||_\infty = \max (v_1,v_2,..,v_n)\]
# Import numpy
import numpy as np

# Import Linear Algebra package from NumPy
import numpy.linalg as LA

# Define a simple vector

v = np.array([1,2,3])

# Calculate L1 norm

L1_norm = LA.norm(v,1)

print(L1_norm) # 6.0

# Calculate L2 norm

L2_norm = LA.norm(v, 2)
print(L2_norm) # 3.7416573867739413

# Calculate max norm
# For max norm import infinity

from math import inf
max_norm = LA.norm(v, inf)

print(max_norm) # 3.0

Brownlee (2018) MarcPeterDeisenroth (2019) IanGoodfellow (2019) Ageron (2019)


Bibliography

Ageron. Math - Linear Algebra. url: https://github.com/ageron/handson-ml/blob/master/math_linear_algebra.ipynb, 2019. [Online; accessed 19-April-2019].

Jason Brownlee. Basics of Linear Algebra for Machine Learning. Machine Learning Mastery, 2018.

Aaron Courville Ian Goodfellow, Yoshua Bengio. Deep Learning. Www.deeplearningbook.org. www.deeplearningbook.org, 2019.

Cheng Soon Ong Marc Peter Deisenroth, A. Aldo Faisal. Mathematics for Machine Learning. Cambridge University Press. Cambridge University Press, 2019.

Newsletter

A rover wearing a fuzzy suit doesn’t alarm the real penguins. Then What Alarms The Penguins. Subscribe Me To Find Out.