hand gesture recognition from rgb-d data using 2d and 3d convolutional neural networks: a comparative study

Despite considerable enhances in recognizing hand gestures from still images, there are still many challenges in the classification of hand gestures in videos. the latter comes with more challenges including higher computational complexity and arduous task of representing temporal features. hand movement dynamics, represented by temporal features, have to be extracted by analyzing the total frames of a video. so far, both the 2d and 3d convolutional neural networks (cnns) have been used to manipulate the temporal dynamics of the video frames. 3d cnns can extract the changes in the consecutive frames and tend to be more suitable for the video classification task; however, they usually need more time. on the other hand, using techniques like tiling, it is possible to aggregate all the frames in a single matrix and preserve the temporal and spatial features. this way, using 2d cnns, which are inherently simpler than 3d cnns, can be used to classify the video instances. in this paper, we compare the application of 2d and 3d cnns for representing temporal features and classifying hand gesture sequences. additionally, providing a two-stage two-stream architecture, we efficiently combined color and depth modalities and 2d and 3d cnn predictions. the effects of different types of augmentation techniques are also investigated. the results obtained confirm that an appropriate usage of 2d cnns outperforms a 3d cnn implementation in this task.