DeepMM is a multimodal deep learning package based on keras and tensorflow that combines representations of high-cardinality categorical features together with text-based features in a single deep learning architecture for regression and binary classification use cases.

This model employs the idea of categorical entitiy embeddings (see [https://arxiv.org/abs/1604.06737]) for mapping highly sparse one-hot encoded categorical features into a latent lower-dimensional feature space. A bi-interaction pooling layer (as proposed by He et al. 2017 [https://arxiv.org/abs/1708.05027]) is incorporated to account for second-order feature interactions. An LSTM-based sub-network is used to process the sequential text features.

The architecture is oriented on other deep learning approaches for processing sparse features, such as:

• He et al. (2017) Neural factorization machines for sparse predictive Analysis [https://arxiv.org/abs/1708.05027]
• Cheng et al. (2016) Wide and deep learning for recommender systems [https://arxiv.org/abs/1606.07792]
• Guo et al. (2017) DeepFM: A Factorization Machine-based neural network for CTR prediction [https://arxiv.org/abs/1703.04247]
• Wang et al. (2017) Deep & Cross Network for ad click predictions [https://arxiv.org/abs/1708.05123]

General outline of the multimodal model architecture (with concatenation of categorical embedding vectors) with four generic categorical features (C1-C4):

The package exposes the architecture as a keras.models.Model object based on Keras' functional API. It supports the integration of pre-trained word embedding vectors.

from deepmm.models import DeepMultimodalModel

# Preprocess data and combine modalities into a single matrix
# ...

model = DeepMultimodalModel(task='regression', num_unique_categories=num_unique_categories, cat_embedding_dim=16,
                            txt_vocab_size=vocabulary_size, txt_embedding_dim=EMBEDDING_DIM, txt_max_len=MAX_LEN,
                           txt_weights=embedding_matrix,
                           cat_hidden_neurons=[100,50,10], cat_dropout=[0.1, 0.2, 0.2], cat_bi_interaction=True,
                           txt_lstm_neurons=32, txt_dropout=0.2, final_hidden_neurons=[64, 32], final_dropout=[0.3, 0.3])