Deep Learning 101: Lesson 6: Tensors
This article is part of the “Deep Learning 101” series. Explore the full series for more insights and in-depth learning here.
Tensors are a fundamental concept in AI and machine learning, and can be thought of as multidimensional containers for data. Most of us are already familiar with scalars, vectors, and matrices. Tensors can be seen as a generalization that includes all of these concepts, along with higher dimensional data structures. They serve as a versatile term to represent different data types and containers, allowing us to work with more complex and multidimensional data sets.
The image shows two sections, “1-D” and “2-D,” representing one- and two-dimensional linear regression models. In the “1-D” section, there is a table listing pairs of ‘x’ and ‘y’ values that suggest data points for a simple linear regression. The formula y=ax+b describes the relationship between ‘x’ and ‘y’, where ‘a’ is the slope (0.413) and ‘b’ is the y-intercept (0.161). The “2-D” section is prepared for two-dimensional data, with an empty table set to contain ‘x1’, ‘x2’, and ‘y’ values. The regression model extends to two predictors, represented by the equation:
y = a1 x1 + a2 x2 + b
where a1 and a2 denote the coefficients for ‘x1’ and ‘x2’, and ‘b’ is the y-intercept.
The below matrix equation further translates this model into matrix notation:
y = XA + b
Here, y is the outcome vector, X is the predictor matrix, A is the coefficient matrix, and b is the intercept vector. Their dimensions are given as (3, 1) for y, (3, 2) for X, (2, 1) for A, and (1) for b, showing how matrix multiplication is used to compute y from X and A, with the addition of b.
Tensor examples
Tensors are a type of data structure used in linear algebra and are particularly important in various applications, including physics, computer science, and data analysis. They are generalizations of more familiar mathematical entities such as scalars, vectors, and matrices to an arbitrary number of dimensions. In the following, we will explore the concept of tensors by looking at their different forms, starting from a simple matrix to more complex structures.
2-Dimensional Tensor
A matrix is the simplest form of a tensor, essentially a 2-dimensional array. Consider a matrix with the shape [2, 4], which means it has 2 rows and 4 columns. This matrix is a rectangular grid filled with numbers, such as integers from 0 to 7. In linear algebra, this matrix is called a 2nd-rank tensor because it has two dimensions and requires two indices to identify each element.
In the diagram above, we have used the red arrow and red square brackets to indicate the column dimension, which is 4 in the 2-D tensor example above. Similarly, we have used the green arrows and green square brackets to indicate the row dimension, which is 2 in the above example.
3-Dimensional Tensor
Extending the 2-dimensional tensor, a 3-dimensional tensor can be visualized as a stack of matrices. With a shape denoted by [3, 2, 4], it consists of 3 layers, where each layer is a 2x4 matrix. This structure is called a 3rd-rank tensor because it requires three indices to locate a particular element within the tensor. In this tensor, the numbers are filled sequentially, creating a block of numbers from 0 to 23, which extends our grid into the third dimension.
Note that we have used a blue arrow and a blue square bracket to denote the third dimension in a 3-dimensional tensor, which has the value 3 in the example above.
4-Dimensional Tensor
Proceeding to even higher dimensions, a 4-dimensional tensor has the form [2, 3, 2, 4]. This structure can be thought of as a list of 3-dimensional tensors, with two such lists forming the complete tensor. It contains numbers from 0 to 47, arranged in an abstract four-dimensional space. This tensor is a 4th rank tensor, since four indices are needed to specify the position of each element.
Creating Tensors from Raw Data
Tensors are the fundamental building blocks of many modern computational frameworks, especially in the fields of machine learning and data analysis. By definition, a tensor is a generalized n-dimensional array, where ’n’ denotes the rank or order of the tensor. The rank of a tensor defines the number of dimensions, called axes, that the tensor has. The power of tensors comes from their ability to model complex relationships and efficiently represent high-dimensional data structures.
To illustrate the concept of tensors and how they can be constructed from a sequence of raw data, consider the following examples, where a sequence of integers from 0 to 15 serves as our raw data:
Data: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
Rank 2 Tensor:
If we arrange this sequence into a rank 2 tensor (similar to a 2D array or matrix), we can define its shape as [2, 8]. This means that the tensor has two dimensions: the first dimension has a length of 2, and the second dimension has a length of 8. Such a structure can be used to represent matrices, where the first axis can represent rows and the second axis can represent columns. The tensor thus contains two arrays, each containing eight elements of the data sequence. Here is how we can represent it in tensor form:
Tensor
rank: 2
shape: [2,8]
values:
[[0, 1, 2 , 3 , 4 , 5 , 6 , 7 ],
[8, 9, 10, 11, 12, 13, 14, 15]]Rank 3 Tensor:
Progressing to a rank 3 tensor introduces an additional dimension, which can be thought of as depth. A tensor of the form [2, 4, 2] can be thought of as two layers of 2D arrays, each layer containing four rows and two columns. This format is particularly useful for applications such as computer vision, where an image can be represented as a tensor with dimensions corresponding to height, width, and color channels. Here is how we can represent it in tensor form:
Tensor
rank: 3
shape: [2,4,2]
values:
[[[0 , 1 ],
[2 , 3 ],
[4 , 5 ],
[6 , 7 ]],
[[8 , 9 ],
[10, 11],
[12, 13],
[14, 15]]]Creating tensors from raw data involves arranging sequences of numbers into multidimensional arrays. Tensors can be of different ranks, with each rank representing a certain number of dimensions. For example, a rank 2 tensor can be visualized as a matrix, while a rank 3 tensor can be thought of as a stack of matrices. The flexibility and power of tensors make them essential for modeling complex relationships and representing high-dimensional data in machine learning and data analysis.
Summary
Tensors are versatile, multidimensional data structures that are critical to AI and machine learning, enabling efficient representation and manipulation of complex data sets. They generalize familiar concepts such as scalars, vectors, and matrices to higher dimensions, enabling sophisticated data modeling and computational tasks. By organizing raw data into tensors of various ranks, we can leverage their capabilities to perform advanced analytics and build powerful machine learning models, ultimately improving our understanding and ability to process high-dimensional data.
4 Ways to Learn
1. Read the article: Tensors
2. Play with the visual tool: Tensors
3. Watch the video: Tensors
4. Practice with the code: Tensors
Previous Article: Stochastic Gradient Descent
Next Article: Perceptron
