This GitHub repository contains code and data for our paper A Bayesian-Symbolic Approach to Reasoning and Learning in Intuitive Physics at NeurIPS 2021.

If you use the code in your research, please cite us using the following BibTeX entry:

@inproceedings{xu2021bsp,
  title={A {B}ayesian-Symbolic Approach to Reasoning and Learning for Intuitive Physics},
  author={Kai Xu and Srivastava, Akash and Gutfreund, Dan and Sosa, Felix and Ullman Tomer and Tenenbaumm, Joshua B. and Sutton, Charles},
  booktitle={The 35th Conference on Neural Information Processing Systems (NeurIPS)},
  year={2021},
}

/

• data/: generated data and processed data
  • phys101: selected scenes from Physics 101 (Wu et al., 2016); originally from https://phys101.csail.mit.edu/
  • synth: 3 types of synthetic scenes
  • ullman: visual stimulus used in Ullman et al. (2019); originally from https://github.com/tomeru/LPDS/
• notebooks/: interactive notebooks to extract or analyse experimental results
  • demo-world.ipynb: example notebook to demonstrate how to run simulation with BayesianSymbolic.jl
  • monly.ipynb: notebook to extract results for M-step only results
  • em.ipynb: notebook to extract results for complete EM results
  • phys101.ipynb: notebook to extract results for PHYS101 results
  • ullman.ipynb: notebook to extract results for ULLMAN results
• paper/: interactive notebooks to produce artifects for the paper
  • raw/: extracted results
  • figures.ipynb: notebook to make figures
  • tables.ipynb: notebook to make tables
• scripts/: executable scripts to generate or pre-process data or to run experiments
  • evalexpr_efficiency.jl: script to compare different implementations for expression evaluations
    • This does not affect the results but only performance.
  • generate-synth.jl: script to generate SYNTH
  • helper.jl: common helper functions used by scripts
  • neural.jl: concrete architecture and hyper-parameters for neural baselines
  • preprocess-ullman-master.jl: master script to process a scene from ULLMAN, calling scripts/preprocess-ullman.py
  • preprocess-ullman.py: script to process scenes from ULLMAN
  • runexp-ullman.jl: script to run an experiment on ULLMAN; for other datasets, use runexp.jl
  • runexp.jl: script to run an experiment on a given dataset and specific hyper-parameters
    • All datasets but ULLMAN are supported; for ULLMAN, use runexp-ullman.jl
  • ullman_hacks.jl: ULLMAN specific grammar
    • See the comments on hacks in the beginning of the file
• src/: source codes for simulation and the BSP algorithm
  • BayesianSymbolic.jl/: world construction and simulations
  • data/: data processing functions
    • phys101.jl: PHYS101 specific functions
    • preprocessing.jl: generic data preparation functions
    • ullman.jl: ULLMAN specific functions
    • ullman.txt: ground truth information for ULLMAN
  • scenarios/: scenarios implemented in Turing.jl
    • bounce.jl: the BOUNCE scenario from SYNTH
    • fall.jl: the FALL scenario from PHYS101
    • magnet.jl: the MAGNET scenario
      • This is not used in the paper
    • mat.jl: the MAT scenario from SYNTH
    • nbody.jl: the NBDOY scenario from SYNTH
    • spring.jl the SPRING scenario from PHYS101
    • ullman.jl: the scenario from ULLMAN
  • analyse.jl: quantitative and visual analysis
  • app_inf.jl: functions for approximate inference
  • dataset.jl: functions for loading datasets (SYNTH, PHYS101 and ULLMAN)
  • exp_max.jl: types and interfaces for the EM algorithm
  • neural.jl: generic implementations of neural baselines
  • sym_reg.jl: functions for symbolic regression
  • utility.jl utility functions for simulation and loss computation
• =suupl/=: supplementary materials
  • bounce_inspection: visualisation for results discussed in appendix C.3.1
  • generalization: visualisation for results discussed in appendix C.3.2
• Manifest.toml: the exact package version of this environment
• master.jl: master scripts to run a batch of experiments, calling scripts in scripts/
• Project.toml: the dependency of this environment
• README.md: this file you are reading

BSP is implemented with Julia and some of the dependencies or scripts also rely on Python.

Please follow https://julialang.org/downloads/ to download and install Julia. Make sure julia is available in your executable path. Then from the root of this repo, you can do julia -e "import Pkg; Pkg.instantiate()" to instantiate the environment.

There are a few more steps to have Julia properly linked with Python, which is explained next.

You will need to have Python installed and a virtual environment set up. The virtual environment should have the following packages

• matplotlib
• pandas
• wandb

To properly link this virtual environment with Julia, please follow https://github.com/JuliaPy/PyCall.jl.

You will also have all necessary Python dependencies to run scripts/preprocess-ullman.py. Please see the libraries imported in the script.

Once these steps are done, you are ready to run the scripts.

Run the following experiments

• julia master.jl efficiency synth/nbody
• julia master.jl efficiency synth/bounce
• julia master.jl efficiency synth/mat

Collect the results using notebooks/monly.ipynb and make the figure using paper/figures.ipynb

Run the following experiments

• julia master.jl ablation synth/nbody
• julia master.jl ablation synth/bounce
• julia master.jl ablation synth/mat

Collect the results using notebooks/monly.ipynb and make the figure using paper/figures.ipynb

Run the following experiments

• julia master.jl em synth/nbody

Collect the results using notebooks/em.ipynb and make the figure using paper/tables.ipynb

Run the following experiments

• julia master.jl phys101 fall
• julia master.jl phys101 spring

Collect the results using notebooks/phys101.ipynb and make the figure using paper/tables.ipynb

Run the following experiments

• julia master.jl ullman

Collect the results using notebooks/ullman.ipynb and make the table using paper/tables.ipynb & figure using paper/figures.ipynb

Set the environment variable JULIA_NUM_THREADS=10 before running any scripts will enable multiple-threading (e.g. 10 threads in this example) whenever it’s programmed to do so. For example, master.jl executes a batch of experiments and it is programmed to run them in a multi-threading manner which can benefit from setting this environment variable.