Related Articles

# Continuous Kernel Convolution

• Last Updated : 14 Aug, 2021

Continuous Kernel convolution was proposed by the researcher of Verije University Amsterdam in collaboration with the University of Amsterdam in a paper titled ‘CKConv: Continuous Kernel Convolution For Sequential Data‘. The motivation behind that is to propose a model that uses the properties of convolution neural networks and Recurrent Neural networks in order to process a long sequence of image data.

#### Convolution Operation

Let x ∶R → RNc and ψ ∶ R → RNc be a vector-valued signal and kernel on R, such that x = {xc}Nc and ψ = {ψc} NC c=1. The convolution operation can be defined as: However, practically the input signal x is gather from sampling. Thus, the input signal and convolution can be defined as

• Input signal: • Convolution: and the equation that is centered around t is given by: Now for continuous kernel convolution, we will use a convolution kernel ψ as continuous function parameterized over a small NN called MLPψ. It takes (t−τ ) as input and outputs the value of the convolution kernel at that position ψ(t−τ ). The continuous kernel can be formulated by: If the sampling factor is different from training sampling factor, then we can perform convolution operation in following ways: #### Recurrent Unit

For the input sequence . The recurrent unit is given by:  where U, W, V are the input-to-hidden, hidden-to-hidden and hidden-to-output connections of the unit. h(τ ), y˜(τ ) depict the hidden representation and the output at time-step τ, and σ represents a point-wise non-linearity.

Now, we unroll the above equation for t steps: where h(−1) is the initial state of the hidden representation. h(t) can also be represented in following way: The above equation provides us with following conclusion:

• Vanishing gradient and exploding gradient problem in RNN is caused by the term x(t-τ) τ steps back in the past being multiplied with an effective convolution weight ψ(τ )=WτU.
• Linear recurrent unit can be defined as the convolution of input and exponential convolution functions.

#### MLP Continuous Kernel

Let {∆ti=(t − τi)}N i=0 be a sequence of relative positions. The convolution kernel MLPψ is parameterized by a conventional L-layer neural network: MLP where, is used to add non-linearity such as ReLU.

### Implementation

• In this implementation, we will be training the CKconv model on the sMNIST dataset, for this implementation, we will be using colaboratory that is provided to us by Google.

## Python3

 # code credits: https://github.com/dwromero/ckconv# first, we need to clone the ckconv repository from Github! git clone https://github.com/dwromero/ckconv # Now, we need to change the pwd to ckconv directorycd ckconv # Before actually train the model,# we need to first install the required modules and libraries! pip install -r requirements.txt # if the above command fails, please make sure that following modules installed using# command belowpip install ml-collections torchaudio mkl-random sktime wandb # Now to train the model on sMNIst dataset, run the following commands! python run_experiment.py --config.batch_size=64 --config.clip=0 \--config.dataset=MNIST --config.device=cuda --config.dropout=0.1 \--config.dropout_in=0.1 --config.epochs=200 --config.kernelnet_activation_function=Sine \--config.kernelnet_no_hidden=32 --config.kernelnet_norm_type=LayerNorm \--config.kernelnet_omega_0=31.09195739463897 --config.lr=0.001 --config.model=CKCNN \--config.no_blocks=2 --config.no_hidden=30 --config.optimizer=Adam \--config.permuted=False --config.sched_decay_factor=5  --config.sched_patience=20 \--config.scheduler=plateau
• Below are the results of the above training of CKCNN on sMNIST data: CkConv training Result

#### Conclusion

• Ckconv is able to very complex and non-linear function easily.
• Contrary to RNNs, CKConvs do not rely on any form of recurrence for considering large memory horizons and have global long-term dependencies.
• CKCNNs do not make use of Back-Propagation Through Time(BPTT). Consequently, CKCNNs can be trained in parallel.
• CKCNNs can also be deployed at resolutions other than the resolution on which it is trained.

#### References:

Attention reader! Don’t stop learning now. Get hold of all the important Machine Learning Concepts with the Machine Learning Foundation Course at a student-friendly price and become industry ready.

My Personal Notes arrow_drop_up