Made for my fourth semester with Python. The goal of this project was to create a real time gesture recognition app which is able to recognize american sign language gestures. The model that this app uses is able to recognize gestures for digits '0' to '9'. The notebook explains how this model was made.
- real time: gesture recognition in real time.
- multiple hands: multiple hands can be recognized at once.
- ambidextrous: recognize gestures on both left and right hand.
- debug: visualize hand skeleton, confidence & more.
- CD into the project folder.
cd gesture-recognition
- Create and start a new python environment.
python -m venv venv
venv/Scripts/activate
- Install all requirements using the requirements.txt file.
pip install -r requirements.txt
- Run the gesture.py file.
python gesture.py
- Close the application by pressing
q, and deactivate the enrionment.
deactivate
The approach document explains the score of the project and my approach to different topics like choosing a hand pose estimation model, getting a dataset and training a model. It also contains some reccomentations for all these subjects.
The Sign Language Digits Dataset was used to train and test the model.
Mavi, A., (2020), “A New Dataset and Proposed Convolutional Neural Network Architecture for Classification of American Sign Language Digits”, arXiv:2011.08927 [cs.CV]
This repository does NOT contain this data. So, if you want to process the images yourself you will have to download the dataset and put all the subfolders into a folder called images.
The structure should look like this:
.
├─ 🗋 Gesture recognition.ipynb
├─ 🗁 images/
│ ├─🗀 0/
│ ├─🗀 1/
│ ├─🗀 2/
│ ├─🗀 4/
│ ├─🗀 5/
│ ├─...
If you wish to use your own dataset click Here to see how.
If you want to work on pre-processing the dataset yourself you can use the notebook. It's not necessary to process all the images yourself. the dataframes directory contains both the raw points data and the pre-processed data.
The gesture-points-raw data is not cleaned and contains a number of missing values. This dataframe can be used if you want to try your own technique of pre-processing.
The gesture-points-processed contains the pre-processed dataframe. This is the pre-processed version of the raw dataframe. All missing values have been fixed, the data has been normalized and a flipped version of the dataframe has been appeded. This dataset can be used if you want to try out your own technique of modeling.
The notebook should also work with other datasets of gesture images. But you have to make sure that the images are put into a folder called images. The structure should look the same as if you were using the original dataset. The openpose hand model is NOT included inside this repository. This model can be downloaded here, and needs to be put in the openpose directory.
There are some limitations for the dataset that need to be taken into account:
- The hand in the image must be larger than 60x60 pixels.
- Lower exposure images work better.
In the Dataset restrictions you can see how i found these limitations. In the approach document you can also find some more reccomendations for the dataset.
This software is licensed under MIT