Update lip reading example

[seujung]

Description

Add lip reading model using gluon

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[roywei]

@mxnet-label-bot add[Example, Gluon, pr-awaiting-review]

[aaronmarkham]

I haven't been able to test this end-to-end yet. I've tried the data download process a couple of times, but have had to restart due to connectivity and space issues. I'll try again later, but I thought I'd at least give some initial feedback.
I'm looking forward to seeing this work. It seems like a very cool example. Thanks for sharing it.

[aaronmarkham]

Might make it easier if you used paths relative to where this is in the examples folder and where the data gets downloaded.

[soeque1]

Changed

[aaronmarkham]

Suggested change

	Spli the number(sam_num) into numbers by n_tile
	Split the number(sam_num) into numbers by n_tile

[soeque1]

Fixed

[aaronmarkham]

Please add these to your prerequisites list:

• scikit-image
• scikit-video
• dlib
• tqdm

Also, I tried to run it without a GPU and couldn't get it to work with:

python3 main.py --use_gpu False 
# or 0

Raises this error:

mxnet.base.MXNetError: [06:52:06] src/ndarray/ndarray.cc:1233: GPU is not enabled

[aaronmarkham]

I built the project on a GPU instance this time and was able to run main.py. However, I immediately get a dump of a lot of these errors:

  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 166, in _recursive_fork_recordio
    if depth >= max_depth:
RecursionError: maximum recursion depth exceeded in comparison

Looks like it ran that 200 times and failed each time.

[vandanavk]

is this PR good to go @thomelane @larroy ?

[thomelane]

@aaronmarkham any chance you could zip up all of the preprocessed files, to avoid aws s3 sync downloading loads of small files? would definitely save time and bandwidth.

And then we need to get those instructions into the README, so people don't try to download and preprocess the data themselves.

[aaronmarkham]

@thomelane Ok, I'm creating the tar files now. The reason I didn't do that is that I kept getting disconnected and wanted to be able to resume a sync. If you're pulling a 15gb file and have to start over, well, that's no fun.
At least people will have the option for either route once I've uploaded the files.
Rather than force another CI pass, if the code is alright here, let's merge it and then I can update the README in a follow-up PR.

[aaronmarkham]

I put the tar files in a separate bucket so you can pick how you want to download.

To get the tar files:

 aws s3 sync s3:https://mxnet-public/lipnet/data-archives .

Or to download them by link:
https://mxnet-public.s3.amazonaws.com/lipnet/data-archives/align.tgz
https://mxnet-public.s3.amazonaws.com/lipnet/data-archives/datasets.tgz

To get the folders (unzipped):

 aws s3 sync s3:https://mxnet-public/lipnet/data .

[thomelane]

@aaronmarkham thanks for uploading! sure, you can add the instructions to the readme in a different commit, wouldn't that need a CI run anyway, or has that been optimised now to ignore markdown changes?

@seujung the model seems to be training okay (i.e. loss going down), but still noticeable differences between target and prediction. How good are the predictions on a correctly trained model? Also noticed that things like learning rate aren't explicitly defined, are the defaults correct for this model?

[soeque1]

@aaronmarkham I checked the file you uploaded. It was nice as intended.
@thomelane @seujung and I are working together for this example.
(1) After we did some experiments, we decided to use default learning rate.
(2) We checked the prediction, decoded using beam search.

It takes too long time to train this model. The main reason is the decode part (beam search) on validation data (def infer_batch). Actually, we do not need decode all the validation examples during training, so we skip this or check only one mini batch example. To speed up, we only check the decoded result using infer.py not main.py (train).
But it helps to understand how good the result is.

(3) Although the loss is still decreasing, I attach the pre-trained model.
Model
Sample

You can get the result.

python infer.py model_path='checkpoint/epoches_81_loss_15.7157'

Or
You can resume training.

python main.py model_path='checkpoint/epoches_81_loss_15.7157'

[thomelane]

Great, thanks for clarifying @soeque1!

[thomelane]

LGTM

[szha]

Suggested change

	# LipNet: End-to-End Sentence-level Lipreading
	<!---
	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at
	http:https://www.apache.org/licenses/LICENSE-2.0
	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License. See accompanying LICENSE file.
	-->
	# LipNet: End-to-End Sentence-level Lipreading

[aaronmarkham]

License isn't required on readme files. @szha if you feel strongly about adding it, I'm going to modify the readme in another PR later today and I can add it then.