Sign language recognition model

Sign language recognition is the task of recognizing the sign performed by deaf people. Sign gestures can beisolated (one sign per video) or continuous (several signs per video). In this assignment, you will develop a deeplearning model for sign language recognition at the sentence level. This task is similar to the video captioningtask where the input will be a sign video and the output will be the performed sign(s) by the signer in the textformat. An example of the input and output of the system is shown below.

In this assignment you need to do the following:

– Read and prepare the dataset

o The dataset is provided as images (

https://drive.google.com/file/d/13lsr4ncy27YDaVbg6…

)

o There are 10 different sentences performed by three signers

o There are 80 frames extracted from each video’s sample

– Use word embeddings to represent the ground truth text when you feed it to the model

– For each frame, use MobileNetV2 to extract features from the last fully connectedlayer before the classification layer of the MobileNetV2 pretrained model. You will use thesefeatures as inputs for the following questions.

– Develop an encoder-decoder model to recognize the sign video (video captioning)

o Select the best architecture that gives good results

– Develop an encoder-decoder model with attention to recognize the sign video(video captioning) – Report the results using word error rate (WER) metric- Bonus [30 points]

o The first three best results (lowest WER)

o Develop a transformer model for the same problem.