Getting a Handle on Vector for Machine Learning
The word vector comes from the Latin for “carrier”, reason might be the nature of vector to move from one location to another. In physics, a Vector is an object that moves around a physical space. In Data Science and Machine Learning we generalize this idea to think vector as a list of attributes of an object. So Vectors are used to represent objects, observations and predictions. Brownlee (2018)
Say we want to represent a house having 2 bedrooms, 1 bathroom, 150-meter square and cost around $ 20,000. Based on the information we know about this house, we can represent the house as a vector:
$$house = \begin{pmatrix} 2 \\ 1 \\ 150 \\ 20000 \end{pmatrix}$$
Based on this vector, our machine learning predicts that there is 45% chance that this house will give me profit if I buy, 5% chance that it will be a bad deal(loss) and 50% chance that the price of house will not change at all. We can represent this prediction in a vector as:
$$class\_probabilities = \begin{pmatrix} 0.45 \\ 0.05 \\ 0.5 \end{pmatrix}$$
Vectors in Python
There are multiple ways we can define a vector in Python, the simplest one being a list:
house = [2, 1, 150, 20000]
Python Lists are very good, but in Machine Learning we have deal with lot of scientific calculations,so it is much convenient to represent a vector in Python as a NumPy array as Numpy array does the thing what we expect when we add, multiply vectors.
import numpy as np
house = np.array([2, 1, 150, 20000])
Vector Arithmetic
In this section, we will illustrate simple vector-vector arithmetic, where all operations are performed element-wise.
Vector Addition
Two vector of equal length when added produce a new vector of the same length. Each element of the new vector at index i is calculated as the sum of the other element at that particular index i.
import numpy as np
house1 = np.array([2,1,150, 20000])
house2 = np.array([2, 1, 200, 29000])
house3 = house1 + house2
print(house3)
[ 4, 2, 350, 49000]
Vector Subtraction
Two vectors of equal length can be subtracted to produce a new vector of the same length. Each element of the new vector at index i is calculated as the difference of the other elements at that particular index i.
Vector Multiplication
Two vector of equal length can be multiplied together to create a new vector of same length. Like with addition and subtraction, vector multiplication is done element wise.
$$a=b * c$$import numpy as np
a = np.array([1, 2, 3])
b = np.array([2, 3, 4])
c = a * b
print(c)
[ 2 6 12]
Vector Division
Two vector of equal length can be divided element wise to calculate a new vector of same length.
Vector Norms
The term Norm is typically used to represent the length or magnitude of a vector. The norm of vector v is denoted by ||v||.
Norm Properties
1. For any vector \(\mathbf{v}\), \(\mathbf{||v||}\) is a non-negative real number.$$\\$$ 2. For any vector \(\mathbf{v}\), \(\mathbf{||v||}\) is zero if and only if \(\mathbf{v}\) is a zero vector.$$\\$$ 3. Triangle inequality, For any vectors \(\mathbf{x}\) and \(\mathbf{y}\), \[||x + y|| \le ||x|| + ||y||\]$$\\$$ 4. For any vector \(\mathbf{v}\) and any scalar \(\alpha \), \[||\alpha v|| = |\alpha |||v||\]$$\\$$Different Vector Norms
1. Manhattan Norm or Taxicab Norm
The \(L^1\) Norm or Manhattan Norm or Taxicab Norm of vector \(\mathbf{v}\) \( \in \) \(R^n\) is defined as \[||v||_1 = \sum\limits_{i = 1}^n {{|v_i|}} \]. $$\\$$ The \(L^1\) norm is calculated as the sum of the absolute vector values, where the absolute value of a scalar uses the notation \(\mathbf{|v|}\). \[||v||_1 = |v_1| + |v_2| + |v_3| + .. + |v_n|\]2. Euclidean Norm
The \(L^2\) Norm or Euclidean of vector \(\mathbf{v}\) \( \in \) \(R^n\) is defined as \[||v||_2 = \sqrt{\sum\limits_{i = 1}^n {v_i^2}} \]. $$\\$$ The \(L^2\) norm simply calculates the Euclidean distance from the origin to the point identified by \(\mathbf{v}\). $$\\$$ The \(L^2\) norm is calculated as the square root of the sum of the squared vector values \[||v||_2 = \sqrt {v_1^2 + v_2^2 + v_3^2 + .. + v_n^2} \]3. Vector Max Norm
The \(L^\infty\) norm or vector max norm of vector \(\mathbf{v}\) \( \in \) \(R^n\) is defined as \[||v||_\infty = \mathop {\max |v_i|}\limits_i \] The \(L^\infty\) norm is calculated by returning the the maximum value of vector as \[||v||_\infty = \max (v_1,v_2,..,v_n)\]# Import numpy
import numpy as np
# Import Linear Algebra package from NumPy
import numpy.linalg as LA
# Define a simple vector
v = np.array([1,2,3])
# Calculate L1 norm
L1_norm = LA.norm(v,1)
print(L1_norm) # 6.0
# Calculate L2 norm
L2_norm = LA.norm(v, 2)
print(L2_norm) # 3.7416573867739413
# Calculate max norm
# For max norm import infinity
from math import inf
max_norm = LA.norm(v, inf)
print(max_norm) # 3.0
MarcPeterDeisenroth (2019) IanGoodfellow (2019) Ageron (2019)
Bibliography
Ageron. Math - Linear Algebra. url: https://github.com/ageron/handson-ml/blob/master/math_linear_algebra.ipynb, 2019. [Online; accessed 19-April-2019]. ↩
Jason Brownlee. Basics of Linear Algebra for Machine Learning. Machine Learning Mastery, 2018. ↩
Aaron Courville Ian Goodfellow, Yoshua Bengio. Deep Learning. Www.deeplearningbook.org. www.deeplearningbook.org, 2019. ↩
Cheng Soon Ong Marc Peter Deisenroth, A. Aldo Faisal. Mathematics for Machine Learning. Cambridge University Press. Cambridge University Press, 2019. ↩