ASL Translator

Dataset

Original Data on Kaggle

In this project, we create our own dataset, the Dataset-A, by arranging the following data from Kaggle.com:

Dataset-A

We want to challenge if our model is general and robust or not, so we build this hybrid dataset. Links for download:

Structure of Dataset-A

In both train data and test data, they contain alphabet A to Z, 26 classes folder of right hand image instances data.

Training Data

All data are the subset of dataset1
We get rid of some images that cannot pass our image-pipeline in dataset1
The image count for each alphabet is approximately to the amount 2220.

Here's the chart of our image count distribution in Dataset-A training set:

Testing Data

For each alphabet, we select 555 image instances (2220 * 0.2 = 555) from dataset2, dataset3 and dataset4

Here's the chart of our image count distribution in Dataset-A testing set:

Methodology

a. Data Preprocessing

The demo images are in the folder pipeline-demo, the image file name prefix indicates the pipeline fucntion type

image-pipeline, with two different type

I. General Pipeline:

This kind of pipeline can be used in any kinds of preprocessing stage.

// work flow
1. roi normalization (by mediapipe)
2. background normalization (by rembg)
3. skin normalization
4. channel normalization
5. resolution normaliztion

II. Training Pipeline:

This kind of pipeline can only be used in training preprocessing stage.

// work flow
1. background normalization (by rembg)
2. roi normalization (by mediapipe)
3. skin normalization
4. channel normalization
5. resolution normaliztion

Data Augmentation

Implement by keras.ImageDataGenerator with zoom_range=0.1,, width_shift_range=0.1, height_shift_range=0.1, shear_range=0.1
Implement by keras.model.Engine, we create our own Spatial Transformer Layer stn().

b. Model Building

You can download our models at here: saved_models_v1

Normal Model - Pure CNN Structure without Spatial Transform Layers:

The implement code is in asl_model/models.py-get_model_1()

model = models.Sequential()

model.add(layers.Conv2D(32, (3, 3), strides=(1, 1), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.BatchNormalization())

model.add(layers.MaxPool2D((2, 2)))

model.add(layers.Conv2D(64, (3, 3), strides=(1, 1), activation='relu'))
model.add(layers.Dropout(0.2))
model.add(layers.BatchNormalization())

model.add(layers.MaxPool2D((2, 2)))

model.add(layers.Conv2D(128, (3, 3), strides=(1, 1), activation='relu'))
model.add(layers.BatchNormalization())
model.add(layers.MaxPool2D((2, 2)))

# finish feature extraction
model.add(layers.Flatten())

model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dropout(0.25))

model.add(layers.Dense(26, activation='softmax'))

STL Model - Spatial Transform Layer with CNN Structure

The implement code is in asl_model/stl/struct_a/model.py-get_stn_a_model_8()

input_layers = layers.Input((size, size, 1))
x = stn(input_layers)

x = layers.Conv2D(32, (3, 3), strides=(1, 1), activation='relu',
                  kernel_initializer=initializers.glorot_uniform(seed=STN_SEED))(x)
x = layers.BatchNormalization()(x)

x = layers.Conv2D(64, (3, 3), strides=(1, 1), activation='relu',
                  kernel_initializer=initializers.glorot_uniform(seed=STN_SEED))(x)
x = layers.BatchNormalization()(x)

x = layers.Conv2D(128, (3, 3), strides=(1, 1), activation='relu',
                  kernel_initializer=initializers.glorot_uniform(seed=STN_SEED))(x)
x = layers.BatchNormalization()(x)

x = layers.Flatten()(x)

x = layers.Dense(512, activation='relu')(x)
x = layers.Dropout(0.5)(x)

x = layers.Dense(256, activation='relu')(x)
x = layers.Dropout(0.25)(x)

output_layers = layers.Dense(26, activation="softmax")(x)

model = tf.keras.Model(input_layers, output_layers)

c. Model Training

Basic

57717 train images, 20% will become the validation data
14430 test image, 555 test images for each alphabet

First, data-structure-selection

Select the best structure for normal-model and stl-model. With following hyper-parameters:

lr = 0.001
epoch = 10
batch_size = 128

optimizer=tf.keras.optimizers.Adam(learning_rate = lr),
loss='categorical_crossentropy',
metrics=['accuracy']

Second, use callback function to train the best model of each type. With following settings:

BATCH = 128
EPOCH = 100 # max epoch

# call back functions
es_callback = tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)
reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau()

loss="categorical_crossentropy" 
optimizer="adam" 
metrics=["accuracy"]

d. Model Evaluation

Normal Model

Validation Data - Epoch Accuracy

Train : 0.9917 (Orange)
Valid : 0.9864 (Blue)

Validation Data - Epoch Loss

Train : 0.02555 (Orange)
Valid : 0.05084 (Blue)

Testing Data - Total Accuracy : 89.4%

Testing Data - F1-Score Report:

STL Model

Validation Data - Epoch Accuracy

Train : 0.9871 (Orange)
Valid : 0.9883 (Blue)

Validation Data - Epoch Loss

Train : 0.05062 (Orange)
Valid : 0.04649 (Blue)

Testing Data - Total Accuracy : 90.6%

Testing Data - F1-Score Report:

There are more evaluation charts in the folder charts

Getting Starting

Environment

conda create --name aslt python=3.8 -y

(Options) - If you want to use jupyter run these commands

// actactivate venv
pip install ipykernel
python -m ipykernel install --user --name aslt --display-name "ASLT"

install Pytorch for package rembg

get pyTorch install instructions on pytorch.org For example:

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

Name		Name	Last commit message	Last commit date
Latest commit History 98 Commits
Archive		Archive
Reference-Src-Code		Reference-Src-Code
WALL		WALL
app-logs		app-logs
asl_model		asl_model
asl_translater		asl_translater
charts		charts
demo-data		demo-data
image_pipeline		image_pipeline
img_for_dev		img_for_dev
normal-logs		normal-logs
notebooks_files		notebooks_files
past_record_files		past_record_files
pipeline-demo		pipeline-demo
scripts-for-gen-Dataset-A		scripts-for-gen-Dataset-A
stl-logs		stl-logs
.gitignore		.gitignore
Demo_predicitons.ipynb		Demo_predicitons.ipynb
EDA-DATASET-A.ipynb		EDA-DATASET-A.ipynb
EDA-Original-Kaggle-Data.ipynb		EDA-Original-Kaggle-Data.ipynb
Final-Evaluate-Normal-Model.ipynb		Final-Evaluate-Normal-Model.ipynb
Final-Evaluate-STL-Model.ipynb		Final-Evaluate-STL-Model.ipynb
README.md		README.md
Tune-model-comparison.ipynb		Tune-model-comparison.ipynb
data_tool.py		data_tool.py
dataset_candidate.md		dataset_candidate.md
download_dataset_by_gdown.py		download_dataset_by_gdown.py
gdown_dataset_a.py		gdown_dataset_a.py
pipeline_demo.py		pipeline_demo.py
predict_demo_images.py		predict_demo_images.py
preprocessing.md		preprocessing.md
requirements.txt		requirements.txt
run.py		run.py
test_asl_model.py		test_asl_model.py
test_bg_remover.py		test_bg_remover.py
test_hand_detector.py		test_hand_detector.py
test_img_pipeline.py		test_img_pipeline.py
test_pipeline.py		test_pipeline.py
test_web_cam_work.py		test_web_cam_work.py
webcam_dev_tool.py		webcam_dev_tool.py

juancvargas/ASL_Translator

Folders and files

Latest commit

History

Repository files navigation

ASL Translator

Dataset

Original Data on Kaggle

Dataset-A

Structure of Dataset-A

Training Data

Testing Data

Methodology

a. Data Preprocessing

image-pipeline, with two different type

I. General Pipeline:

II. Training Pipeline:

Data Augmentation

b. Model Building

Normal Model - Pure CNN Structure without Spatial Transform Layers:

STL Model - Spatial Transform Layer with CNN Structure

c. Model Training

Basic

First, data-structure-selection

Second, use callback function to train the best model of each type. With following settings:

d. Model Evaluation

Normal Model

Validation Data - Epoch Accuracy

Validation Data - Epoch Loss

Testing Data - Total Accuracy : 89.4%

Testing Data - F1-Score Report:

STL Model

Validation Data - Epoch Accuracy

Validation Data - Epoch Loss

Testing Data - Total Accuracy : 90.6%

Testing Data - F1-Score Report:

Getting Starting

Environment

install Pytorch for package rembg

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 3

Languages

Packages