Chapter 1 Mathematical Foundation 

1. 1.1 The relationship between scalars, vectors, and tensors 1

2. 1.2 What is the difference between a tensor and matrix? 1

3. 1.3 Matrix and vector multiplication results 1

4. 1.4 Vector and matrix norm induction 1

5. 1.5 How to judge a matrix to be positive? 2

6. 1.6 Derivative Bias Calculation 3

7. 1.7 What is the difference between derivatives and partial derivatives? 3

8. 1.8 Eigenvalue decomposition and feature vector 3

9. 1.9 What is the relationship between singular values and eigenvalues? 4

10. 1.10 Why should machine learning use probabilities? 4

11. 1.11 What is the difference between a variable and a random variable? 4

12. 1.12 Common probability distribution? 5

13. 1.13 Example Understanding Conditional Probability 9

14. 1.14 What is the difference between joint probability and edge probability? 10

15. 1.15 Chain Law of Conditional Probability 10

16. 1.16 Independence and conditional independence 11

17. 1.17 Summary of Expectations, Variances, Covariances, Correlation Coefficients 11

1.1 The relationship between scalars, vectors, and tensors:

• Scalars: Scalars are single, real, or complex numbers, denoted by a single letter (e.g., a). They have magnitude but no direction. Scalars are often used to represent quantities like temperature, mass, or distance.

• Vectors: Vectors are quantities that have both magnitude and direction. They are represented as ordered lists of numbers enclosed in square brackets (e.g., [x, y, z]). Common examples include position vectors, velocity vectors, and force vectors.

• Tensors: Tensors are multi-dimensional arrays that can represent data in higher-dimensional spaces. Scalars and vectors can be thought of as tensors of rank 0 and rank 1, respectively. Tensors of rank 2 and higher can represent more complex data structures. For example, a matrix is a rank-2 tensor.

1.2 What is the difference between a tensor and a matrix:

• A matrix is a specific type of tensor. It is a two-dimensional array of numbers, often used for linear transformations, solving systems of equations, and representing data in tabular form.

• A tensor is a more general mathematical object that can have multiple dimensions (rank greater than 2). While a matrix is a rank-2 tensor, tensors can have higher ranks and are used in various areas of mathematics and physics, including machine learning.

1.3 Matrix and vector multiplication results:

• Matrix x vector = a new vector. This operation is commonly used in linear transformations. If you have a matrix A and a vector v, the result Av is another vector.

• Matrix x matrix = a new matrix. The multiplication of matrices is associative but not commutative, meaning that the order of multiplication matters.

1.4 Vector and matrix norm induction:

• • The norm of a vector or matrix is a measure of its size or magnitude.

  • For a vector x, the L2 norm (Euclidean norm) is given by:

1.5 How to judge a matrix to be positive:

• • A matrix is considered positive definite if, for any non-zero vector x, the quadratic form xTAx is always positive (greater than zero). 

  • A matrix is considered positive semidefinite if xTAx is non-negative (greater than or equal to zero) for all non-zero vectors x.

  • The positivity of a matrix can also be defined by examining its eigenvalues. If all eigenvalues are positive (for a positive definite matrix) or non-negative (for a positive semidefinite matrix), then the matrix satisfies the criteria.

1.6 Derivative Bias Calculation:

• • In machine learning, you might be calculating gradients during the training of neural networks, and biases are included in these calculations.

  • To calculate derivatives involving biases, you typically treat the bias terms as constants, and then compute the derivative with respect to the variable(s) of interest using standard calculus rules, such as the chain rule if the bias terms are inside a non-linear activation function.