Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics
Sean Welleck, Peter West, Jize Cao, Yejin Choi
AAAI 2022
This repo contains code for automated (SAGGA) and rule-based (robustness, compositionality) failure discovery and verification.
Please cite our work if you found the resources in this repository useful:
@inproceedings{welleck2022symbolic,
title={Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics},
author={Sean Welleck and Peter West and Jize Cao and Yejin Choi},
booktitle={AAAI},
year={2022},
url={https://arxiv.org/pdf/2109.13986.pdf}
}
We use utilities from the implementation of [Lample & Charton].
- Clone their [repo]. The repo filepath is then provided as a
--symbolic-math-repo-pathcommand line argument in the scripts below.
Then setup our code by running:
python setup.py developWe use pretrained models and data from [Lample & Charton].
From their [repo]:
- Download and unpack the
FWD + BWD + IBPmodel [link].
The model path is provided as a--model-pathcommand line argument in the scripts below.
For experiments involving validation primitives:
- Download the Integration
FWD,BWD,IBPdatasets.
python admath/genetic.py --basis polynomial_robustness --mutation-params robustness
python admath/genetic.py --basis trig_robustness --mutation-params robustness python admath/genetic.py --fitness-type target_problems --basis target --mutation-params distancepython admath/genetic.py --basis polynomial --fitness-type length_penalty
python admath/genetic.py --basis trig --ops trig --fitness-type trig_length_penaltypython admath/genetic.py --fitness-type target_length --target-length 10 --tau 0.05 --seed-size 50 --generation-size 300 --archive-size-terminate 5000
python admath/genetic.py --fitness-type target_length --target-length 20 --tau 0.05 --seed-size 50 --generation-size 300 --archive-size-terminate 5000
python admath/genetic.py --fitness-type target_length --target-length 40 --tau 0.05 --seed-size 50 --generation-size 300 --archive-size-terminate 5000Generate
python admath/robustness.py --output-dir ./output
==> Wrote to ./output/simple_robustness_all.jsonCheck and verify
python admath/verify.py \
--generations ./output/simple_robustness_all_generations.json \
--method sympy \
--early-stop \
--output-dir ./output
==> Wrote to ./output/simple_robustness_all_verified_sympy_sagemath.jsonCompositionality uses verified functions from the Robustness experiment via the --generations-file parameter.
python admath/compositionality.py \
--generations-file ./output/simple_robustness_all_verified_sympy.json
==> Wrote to ./output/simple_comp_all.jsonThen do the "check and verify" step above.
See notebooks/.