Pytorch Functions – tensor(), fill_diagnol(), append(), index_copy()
This article aims to share some PyTorch functions that will help you a lot in your deep learning and data science journey. Each function will be explained using two write examples and one example where you can’t use those functions. So let’s get started.
PyTorch is an open-source machine learning library, it contains a tensor library that enables to create a scalar, a vector, a matrix or in short we can create an n-dimensional matrix. It is used in computer vision and natural language processing, primarily developed by Facebook’s Research Lab. It is open-source software and released under the modified BSD (Barkley Software Distribution) license.
Our Four functions:
Function 1 – torch.tensor()
This function enables us to create PyTorch tensors. Tensor could be anything i.e. it could be a scalar, it could be a 1-dimensional matrix, it could be an n-dimensional matrix.
Why we need tensors?
Whenever we want to compute any matrix computations in our deep learning model the first and most important task is converting our data frames into numpy array and then to tensors or if we are working on some image classification problem we have to convert those images into PyTorch tensors.
Syntax and parameters:
data: data that we want to convert into tensor. dtype: data type of that data whether it is torch32, torch64 etc. device: type of your device whether it is cpu or gpu(cuda). requires_grad: if it is True, then gradients of that tensor will be saved in torch.grad property.
Code: To create vectors using this function.
In the above example, t1 is the tensor that contains a simple 1-dimensional array. In, t2 we inside the torch. Tensor we have used a single float element but due to that single, our whole t2 tensor has converted into float type. In, t3 we have forcefully set dtype = torch.int32 to change the data types of a tensor.
Code: To create a tensor using a matrix.
In the above example, we have created a t4 tensor that is 3*3 tensor. Similarly, if we want to create an n-dimensional tensor we can create that, like tensor having 4 dimensions or many more.
Example 3 ( the most common mistake):
In this example, we will talk the commonly attempted mistake during creation of a tensor. In example 2, we have learned that we can create n-dimensional array.
Here we have created a tensor of having two dimensions but each dimension is not containing equal no. of elements i.e. each dimension is not having the same length. Because of this. The compiler will throw an error, saying…
ValueError Traceback (most recent call last) <ipython-input-21–28787d136593> in <module> 1
# Example 3?-?breaking (to illustrate when it breaks)?-?? 2 torch.tensor([[1, 2], [3, 4, 5]])
ValueError: expected sequence of length 2 at dim 1 (got 3)
So what we learned is, we can create an n-dimensional array but all the dimensions should having same length otherwise we will get an error.
Function 2 – fill_diagonal_()
Fill_diagonal() fill the main diagonal with fill_value but that matrix should be a 2D matrix, for a ndim > 2 matrix number of rows should be equal to number of columns and if it is not then we have to set wrap = True so that diagonal value can repeat itself.
Syntax and parameters:
fill_diagonal_(fill_value, wrap=False) -> tensor
fill_value: tensor matrix whose diagonals we want to fill. wrap: it takes boolean, it enables us to work with a non-square matrix.
In this example, firstly we will create a tensor of size(3, 3) using torch.zeros(3, 3).
a is a tensor of size(3, 3) which have all of it’s element zero i.e. a is a zero matrix
now what we want is to replace all of it’s diagonal values with .4, so this we can do with the help of fill_diagonal_().
If, we look at tensor a it will look like
tensor([[4., 0., 0.], [0., 4., 0.], [0., 0., 4.]])
Example 2 :
As we learned from example 1, that we can replace diagonal elements using this function but what if we have a tensor in which no. of rows is not equal to no. of columns, then in this scenario we have to set the “wrap” parameter of fill_diagonal_().
You can look at this code,
In the above code, firstly we created a zero tensor of size(9, 4), then we filled it’s diagonal with 5 but we didn’t set wrap = True. Now let’s look at tensor A,
tensor([[5., 0., 0., 0.], [0., 5., 0., 0.], [0., 0., 5., 0.], [0., 0., 0., 5.], [0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.]])
Here you can see that once diagonal reaches to the end of the length of a particular dimension then the rest of the diagonal was not changed
So to solve this problem we have to set wrap as True. Look at this code,
Now take a look at tensor B
tensor([[6., 0., 0., 0.], [0., 6., 0., 0.], [0., 0., 6., 0.], [0., 0., 0., 6.], [0., 0., 0., 0.], [6., 0., 0., 0.], [0., 6., 0., 0.], [0., 0., 6., 0.], [0., 0., 0., 6.]])
You will observe that even diagonal reached to the end of a particular dimension, it again starts changing the diagonals form next dimension.
Example 3(common mistake):
let’s assume that you have a zero tensor of size(4, 5) and you want to change its diagonal with .3 but if you want to change the data type of that tensor to int32, then probably you will think to set a dype = torch.int32 in fill_diagonal_() as,
B = b.fill_diagonal_(6, True)
But, here you have to remember a little thing that fill_diagonal_() only takes two arguments as parameter, one is data that you want to put in diagonal and another one is wrap for working with non-square tensor,
So, the above code will throw an error as,
Traceback (most recent call last) <ipython-input-26–133c4c6a6759> in <module>
1 # Example 3?-?breaking (but if try to set it’s data type it throws an error)
2 a = torch.zeros(4, 5)
– ? 3 a.fill_diagonal_(.3, dtype = torch.float32) TypeError: fill_diagonal_() got an unexpected keyword argument ‘dtype’
Function 3 – expand(*size)
- Returns a new view of the self tensor with singleton dimensions expanded to a larger size.
- Passing -1 as the size for a dimension means not changing the size of that dimension.
- Tensor can be also expanded to a larger number of dimensions, and the new ones will be appended at the front. For the new dimensions, the size cannot be set to -1.
- Expanding a tensor does not allocate new memory, but only creates a new view on the existing tensor where a dimension of size one is expanded to a larger size by setting the stride to 0. Any dimension of size 1 can be expanded to an arbitrary value without allocating new memory.
Syntax and parameter:
Example 1 :
if we have a singleton-dimension matrix then we can expand it’s singleton-dimension, using expand()
lool at tensor a,
tensor([[5, 5, 5, 5, 5, 5, 5, 5],
[3, 3, 3, 3, 3, 3, 3, 3],
[8, 8, 8, 8, 8, 8, 8, 8],
[4, 4, 4, 4, 4, 4, 4, 4]])
Initially, a was a matrix that contains a singleton dimension, now using expand() we expanded it’s singleton dimension into 8
In the above example, we put the second size as -1 which means we don’t want to expand it’s dimension.
Example 3(common mistake) :
This throws an error because according to the definition of expand() we have to match one size to the non-singleton dimension but as we have two singleton dimensions so we can’t fulfill the condition. Error….
RuntimeError: The expanded size of the tensor (6) must match the existing size (2) at non-singleton dimension 1. Target sizes: [2, 6]. Tensor sizes: [2, 2]
Function 4 – index_copy_(dim, index, tensor) ? Tensor
It enables us to copy a tensor into a self tensor in a given index that is defined in index tensor it contains three arguments
- index tensor to hold indices order
- tensor that we want to copy in our self tensor
Now, if you look at the output tensor…
tensor([[0., 1., 9.],
[3., 9., 1.],
[0., 0., 0.],
[7., 6., 9.],
[0., 0., 0.],
[0., 0., 0.]])
In the above example
we copied t tensor into self_tensor with an order that is defined in i tensor
we can perform the same thing of example 1 with two non-zero tensors
Example 3(common mistake )
Traceback (most recent call last) <ipython-input-6-fb983dc7560f> in <module>
3 b = torch.tensor([[3, 9]]) 4 i = torch.tensor() —-> 5 a.index_copy(1, i, b) RuntimeError: index_copy_(): Source/destination tensor must have same slice shapes. Destination slice shape: 2 at dimension 1 and source slice shape: 1 at dimension 0.
The above example throws an error because we have tried to give dim = 1
but, dimth element of the index should contain the same no of element as much element our b tensor have.
Conclusion: so whenever we need to copy a tensor into a self tensor with a specified order we can use this function
Reference Links –