Skip to content

Latest commit

 

History

History
166 lines (114 loc) · 5.5 KB

README.md

File metadata and controls

166 lines (114 loc) · 5.5 KB
Raw

Timeception for Complex Action Recognition

Keras Keras Keras

This code repository is the implementation for the paper Timeception for Complex Action Recognition. We provide the implementation for 3 different libraries: keras, tensorflow and pytorch.

Timeception for Complex Action Recognition

Citation

Please consider citing this work using this BibTeX entry

@inproceedings{hussein2018timeception,
  title     = {Timeception for Complex Action Recognition},
  author    = {Hussein, Noureldien and Gavves, Efstratios and Smeulders, Arnold WM},
  booktitle = {CVPR},
  year      = {2019}
}

How to Use?

Keras

Using keras, we can define timeception as a sub-model. Then we use it along with another model definition. For example, here we define 4 timeception layers followed by a dense layer for classification.

from keras import Model
from keras.layers import Input, Dense
from nets.layers_keras import MaxLayer
from nets.timeception import Timeception

# define the timeception layers
timeception = Timeception(1024, n_layers=4)

# define network for classification
input = Input(shape=(128, 7, 7, 1024))
tensor = timeception(input)
tensor = MaxLayer(axis=(1, 2, 3))(tensor)
output = Dense(100, activation='softmax')(tensor)
model = Model(inputs=input, outputs=output)
model.summary()

This results in the model defined as:

Layer (type)  Output Shape              Param #   
================================================
(InputLayer)  (None, 128, 7, 7, 1024)   0         
(Timeception) (None, 8, 7, 7, 2480)     1494304   
(MaxLayer)    (None, 2480)              0         
(Dense)       (None, 100)               248100    
================================================
Total params: 1,742,404
Tensorflow

Using tensorflow, we can define timeception as a list of nodes in the computational graph. Then we use it along with another model definition. For example, here a functions defines 4 timeception layers. It takes the input tensor, feedforward it to the timeception layers and return the output tensor output.

import tensorflow as tf
from nets import timeception

# define input tensor
input = tf.placeholder(tf.float32, shape=(None, 128, 7, 7, 1024))

# feedforward the input to the timeception layers
tensor = timeception.timeception_layers(input, n_layers=4)

# the output is (?, 8, 7, 7, 2480)
print (tensor.get_shape())
PyTorch

Using pytorch, we can define timeception as a module. Then we use it along with another model definition. For example, here we define 4 timeception layers followed by a dense layer for classification..

import numpy as np
import torch as T
from nets import timeception_pytorch

# define input tensor
input = T.tensor(np.zeros((32, 1024, 128, 7, 7)), dtype=T.float32)

# define 4 layers of timeception
module = timeception_pytorch.Timeception(input.size(), n_layers=4)

# feedforward the input to the timeception layers 
tensor = module(input)

# the output is (32, 2480, 8, 7, 7)
print (tensor.size())

Installation

We use python 2.7.15, provided by Anaconda 4.6.2, and we depend on the following python packages.

  • Keras 2.2.4
  • Tensorflow 1.10.1
  • PyTorch 1.0.1

Training

Testing

Fine-tuning

Pretrained Models

Charades

We will add all pretrained models for Charades by the end of April. For testing, start with the script ./scripts/test_charades_timeception.sh. In order to change which baseline is uses for testing, set the -- config-file using on of the following options.

2D-ResNet-152

Timeception on top of 2D-ResNet-152 as backnone.

Config File Backbone TC Layers Frames mAP (%) Model
charades_r2d_tc3_f32.yaml R2D 3 32 30.37 Link
charades_r2d_tc3_f64.yaml R2D 3 64 31.25 Link
charades_r2d_tc4_f128.yaml R2D 4 128 31.82 Link
I3D

Timeception on top of ResNet-152 as backnone.

Config File Backbone TC Layers Frames mAP (%) Model
charades_i3d_tc3_f256.yaml I3D 3 256 33.89 Link
charades_i3d_tc3_f512.yaml I3D 3 512 35.46 Link
charades_i3d_tc4_f1024.yaml I3D 4 1024 37.20 Link
3D-ResNet-100

Timeception on top of 3D-ResNet-100 as backnone.

Config File Backbone TC Layers Frames mAP (%) Model
charades_r3d_tc4_f1024.yaml R3D 4 1024 41.1 Link

Kinetics 400

We will add all pretrained models for Kinetics 400 by the end of June.

License

The code and the models in this repo are released under the GNU 3.0 LICENSE.