Skip to content

Commit

Permalink
initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
mkocabas committed Oct 12, 2021
0 parents commit 308fd4a
Show file tree
Hide file tree
Showing 48 changed files with 7,619 additions and 0 deletions.
113 changes: 113 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
License

Software Copyright License for non-commercial scientific research purposes
Please read carefully the following terms and conditions and any accompanying documentation before you download
and/or use the SPEC model, data and software, (the "Model & Software"), including 3D meshes, software, and scripts.
By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use
of this github repository), you acknowledge that you have read these terms and conditions, understand them, and
agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use
the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights
under this License

Ownership / Licensees
The Model & Software and the associated materials has been developed at the

Max Planck Institute for Intelligent Systems (hereinafter "MPI").

Any copyright or patent right is owned by and proprietary material of the

Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter
collectively “Max-Planck”)

hereinafter the “Licensor”.

This software includes the SMPL Body Model. By downloading this software, you are agreeing to be bound by the terms of the SMPL Model License

https://smpl.is.tue.mpg.de/modellicense

which is necessary to create SMPL body models.

SMPL bodies that are generated with SPEC can be distributed freely under the SMPL Body License

https://smpl.is.tue.mpg.de/bodylicense

License Grant
Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:

To install the Model & Software on computers owned, leased or otherwise controlled by you and/or your organization;
To use the Model & Software for the sole purpose of performing non-commercial scientific research, non-commercial
education, or non-commercial artistic projects;
Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation,
incorporation in a commercial product, use in a commercial service, or production of other artifacts for
commercial purposes. The Model & Software may not be reproduced, modified and/or made available in any form to
any third party without Max-Planck’s prior written permission.

The Model & Software may not be used for pornographic purposes or to generate pornographic material whether
commercial or not. This license also prohibits the use of the Model & Software to train methods/algorithms/neural
networks/etc. for commercial use of any kind. By downloading the Model & Software,
you agree not to reverse engineer it.

No Distribution
The Model & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered
for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive
purposes only.

Disclaimer of Representations and Warranties
You expressly acknowledge and agree that the Model & Software results from basic research, is provided “AS IS”,
may contain errors, and that any use of the Model & Software is at your sole risk. LICENSOR MAKES NO REPRESENTATIONS
OR WARRANTIES OF ANY KIND CONCERNING THE MODEL & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY
LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor
makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of
the Model & Software, (ii) that the use of the Model & Software will not infringe any patents, copyrights or other
intellectual property rights of a third party, and (iii) that the use of the Model & Software will not cause any
damage of any kind to you or a third party.

Limitation of Liability
Because this Model & Software License Agreement qualifies as a donation, according to Section 521 of the German
Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only.
If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee
for the resulting damage.

Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have
arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be
liable in accordance with the German Product Liability Act in the event of product liability. The foregoing
applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall be excluded.
Patent claims generated through the usage of the Model & Software cannot be directed towards the copyright holders.
The Model & Software is provided in the state of development the licensor defines. If modified or extended by
Licensee, the Licensor makes no claims about the fitness of the Model & Software and is not responsible
for any problems such modifications cause.

No Maintenance Services
You understand and agree that Licensor is under no obligation to provide either maintenance services,
update services, notices of latent defects, or corrections of defects with regard to the Model & Software.
Licensor nevertheless reserves the right to update, modify, or discontinue the Model & Software at any time.

Defects of the Model & Software must be notified in writing to the Licensor with a comprehensible description
of the error symptoms. The notification of the defect should enable the reproduction of the error.
The Licensee is encouraged to communicate any use, results, modification or publication.

Publications using the Model & Software
You acknowledge that the Model & Software is a valuable scientific resource and agree to appropriately reference
the following paper in any publication making use of the Model & Software.

Subjects' Consent
All subjects gave informed written consent to share their data for research purposes. You agree to delete data or
change their use, in case a subject changes or withdraws their consent.

Citation:

@inproceedings{SPEC:ICCV:2021,
title = {{SPEC}: Seeing People in the Wild with an Estimated Camera},
author = {Kocabas, Muhammed and Huang, Chun-Hao P. and Tesch, Joachim and M\"uller, Lea and Hilliges, Otmar and Black, Michael J.},
booktitle = {Proc. International Conference on Computer Vision (ICCV)},
pages = {11035--11045},
month = oct,
year = {2021},
doi = {},
month_numeric = {10}
}

Commercial licensing opportunities
For commercial uses of the Model & Software, please send email to [email protected]

This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.
195 changes: 195 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
# SPEC: Seeing People in the Wild with an Estimated Camera [ICCV 2021]

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)]()
[![report](https://img.shields.io/badge/Project-Page-blue)](https://spec.is.tue.mpg.de/)
[![report](https://img.shields.io/badge/ArXiv-Paper-red)](https://arxiv.org/abs/2110.00620)

> [**SPEC: Seeing People in the Wild with an Estimated Camera**](https://arxiv.org/abs/2110.00620),
> [Muhammed Kocabas](https://ps.is.tuebingen.mpg.de/person/mkocabas),
> [Chun-Hao Paul Huang](https://ps.is.tuebingen.mpg.de/person/chuang2),
> [Joachim Tesch](https://ps.is.tuebingen.mpg.de/person/jtesch),
> [Lea Müller](https://ps.is.tuebingen.mpg.de/person/lmueller2),
> [Otmar Hilliges](https://ait.ethz.ch/people/hilliges/),
[Michael J. Black](https://ps.is.tuebingen.mpg.de/person/black),
> *International Conference on Computer Vision (ICCV), 2021*
<p float="center">
<img src="docs/assets/spec_gif.gif" width="100%" />
</p>

## Features

SPEC is a camera-aware human body pose and shape estimation method.
It both predicts the camera parameters and SMPL body model for a given image.
[CamCalib](camcalib) predicts the camera parameters.
[SPEC](spec) uses these parameters to predict SMPL body model parameters.

This implementation:
- has the demo code for SPEC and CamCalib implemented in PyTorch.
- achieves SOTA results in SPEC-SYN and SPEC-MTP datasets.
- shows how to perform evaluation on SPEC-SYN and SPEC-MTP datasets.

## Updates

- 13/10/2021: Demo and evaluation code is released.

## Getting Started

SPEC has been implemented and tested on Ubuntu 18.04 with python >= 3.7.
If you don't have a suitable device, try running our Colab demo.

Clone the repo:

```shell
git clone https://github.com/mkocabas/SPEC.git
```

Install the requirements using virtualenv or conda:

```shell
# pip
source scripts/install_pip.sh

# conda
source scripts/install_conda.sh
```

## Running the Demo

### SPEC

First, you need to download the required data
(i.e our trained model and SMPL model parameters). It is approximately 1GB.
To do this you can just run:

```shell
source scripts/prepare_data.sh
```

Then, running the demo is as simple as:

```shell
python scripts/spec_demo.py \
--image_folder data/sample_images \
--output_folder logs/spec/sample_images
```
Sample demo output:

<p float="left">
<img src="docs/assets/COCO_val2014_000000386912_000000.jpg" width="70%" />
</p>

Here the green line is the horizon obtained using estimated camera parameters.
On the right, the ground plane is visualized to show how accurate the global translation is.

### CamCalib

If you are only interested in estimating the camera parameters of an image,
run the CamCalib demo:

```shell
python scripts/camcalib_demo.py \
--img_folder <input image folder> \
--out_folder <output folder> \
--show # visualize the raw network predictions
```

This script outputs a pickle file which contains the predicted camera parameters for each input image along with an output
image which visualizes the camera parameters as a horizon line. Pickle file contains:
```
'vfov' : vertical field of view in radians
'f_pix': focal length in pixels
'pitch': pitch in radians
'roll' : roll in radians
```

## Google Colab

## Training

Training instructions will follow soon.

## Datasets

Pano360, SPEC-MTP, and SPEC-SYN are new datasets introduced in our paper.
You can download them from the [Downloads](https://spec.is.tue.mpg.de/download.php)
section of our [project page](https://spec.is.tue.mpg.de).

For Pano360 dataset, we have released the Flickr image ids which can be used to download images
using FlickrAPI.
We have provided a [download script](scripts/download_flickr.py) in this repo.
Some of the images will be missing due to users deleting their photos.
In this case, you can also use `scrape_and_download` function provided
in the script to find and download more photos.

After downloading the SPEC-SYN, SPEC-MTP, Pano360, and [3DPW](https://virtualhumans.mpi-inf.mpg.de/3DPW/)
datasets, the `data` folder should look like:

```shell
data/
├── body_models
│   └── smpl
├── camcalib
│   └── checkpoints
├── dataset_extras
├── dataset_folders
│   ├── 3dpw
│   ├── pano360
│   ├── spec-mtp
│   └── spec-syn
├── sample_images
└── spec
└── checkpoints
```

## Evaluation
You can evaluate SPEC on SPEC-SYN, SPEC-MTP, and 3DPW datasets by running:
```shell
python scripts/spec_eval.py \
--cfg data/spec/checkpoints/spec_config.yaml \
--opts DATASET.VAL_DS spec-syn_spec-mtp_3dpw-test-cam
```

Running this script should give results reported in this table:

| | W-MPJPE | PA-MPJPE | W-PVE|
|--- | --- | --- | ---|
|SPEC-MTP | 124.3 | 71.8 | 147.1|
|SPEC-SYN | 74.9 | 54.5 | 90.5|
|3DPW | 106.7 | 53.3 | 124.7|

## Citation

```bibtex
@inproceedings{SPEC:ICCV:2021,
title = {{SPEC}: Seeing People in the Wild with an Estimated Camera},
author = {Kocabas, Muhammed and Huang, Chun-Hao P. and Tesch, Joachim and M\"uller, Lea and Hilliges, Otmar and Black, Michael J.},
booktitle = {Proc. International Conference on Computer Vision (ICCV)},
pages = {11035--11045},
month = oct,
year = {2021},
doi = {},
month_numeric = {10}
}
```
## License

This code is available for **non-commercial scientific research purposes** as defined in the [LICENSE file](LICENSE). By downloading and using this code you agree to the terms in the [LICENSE](LICENSE). Third-party datasets and software are subject to their respective licenses.

## References

We indicate if a function or script is borrowed externally inside each file.
Here are some great resources we benefit:

- Pano360 dataset preprocessing is borrowed from [Single View Metrology in the Wild](https://github.com/Jerrypiglet/ScaleNet).
- Most of the utility functions depends on our another paper [PARE](github.com/mkocabas/PARE).

Consider citing these works if you use them in your project.



## Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].
Empty file added camcalib/__init__.py
Empty file.
Loading

0 comments on commit 308fd4a

Please sign in to comment.