To start, we need to download some Haskell packages to fuzz.
This is most easily done with the ./prepare.sh
script.
See the section "Obtaining Packages, Continued" at the bottom for how to do this manually
You can automatically generate and run test suites for modules we now how to
fuzz by running the script ./test-all-working.sh
To run the synth tool (for testing), use the command
stack run -- --searchterm [DEFS_FILE] TYPE_NAME
where DEFS_FILE is an optional file that includes package-specific definitions.
Note that the prelude in searchterm/prelude.txt
is loaded by default.
The rulecheck
project is responsible for generating a test suite for the
rewrite rules in a given package. Before using it, you should tell it the
appropriate values for $PACKAGEDIR
and $RULESFILE
. This can be done by
modifying appropriate values in the Config.hs
file.
To generate a suite for a single package $PACKAGENAME
, run the command:
stack run -- $PACKAGENAME
If you want to generate a suite for every package in $RULESFILE
, this can be
done by running without any arguments:
stack run
After generating a test suite for $PACKAGENAME
, the appropriate tests will be
created in package-tests/$PACKAGENAME-test
.
You can then easily run the fuzz test by running stack test
from inside the
generated directory.
The above two steps can be done at once with the command ./test.sh $PACKAGENAME
Just edit ./test-all-working.sh
, it should be straightforward
This can be done by adding a file in
test-template/extra-arbitrary-instances/$PACKAGENAME.hs
.
You may want to do this to expose additional modules, functions, etc.
This can be done by putting modified versions of the file in the vendored
directory. The script ./apply-vendored-patches.sh
will apply the patches.
You can use scripts in the auto
directory to download packages containing
rewrite rules. Here's how to do it in three steps (from the auto directory):
./run.sh main find --listing $LISTINGFILE
./run.sh main download --listing $LISTINGFILE --scratch $PACKAGEDIR
./run.sh main extract --listing $LISTINGFILE --scratch $PACKAGEDIR --output $RULESFILE
where $LISTINGFILE
is a temporary file to store the package information from
hackage. $PACKAGEDIR
is where the packages will be downloaded, and
$RULESFILE
contains information on packages containing rewrite rules.
IMPORTANT Current we expect $PACKAGEDIR
to be named haskell-packages
and
placed adjacent to this directory (i.e. ../haskell-packages
). AND you need
to update the path of haskellPackagesDir
in Config.hs
.
The first thing you need to do is to download the modified version of haddock
that can generate the definition files. This can be done by downloading the zip
file:
https://github.com/zgrannan/haddock/archive/refs/heads/rulecheck-9.0.2.zip
Then run cabal install
in that directory.
To generate the defs for a package, first switch to the directory for that
package. Then, FROM THAT DIRECTORY run the generate-defs-file.sh
script, i.e.
$RULECHECK_DIR/generate-defs-file.sh
. The result should be a file defs.txt
in the current directory.
Various issues can arise during both test suite generation and running the generated test suites. Below are some common issues, and how they can be addressed.
The test generation logic runs separately for each file in a project that contains rewrite rules. However, some files actually shouldn't be analyzed for the following reasons:
- The file is actually a test file (for programs that analyze Haskell code, for example)
- The rewrite rule only applies to other version of GHC besides 9.0.2
- The rule isn't actually a rule at all (for example, it could been commented out). This is possible because rule identification uses a regular expression and could have false positives.
These files can be skipped by adding an entry in filesToSkip
in the
Config.hs
. Note that the filesToSkip
entry matching is performed via an
infix match, so facilitating skipping multiple files at once.
If a whole package seems to have a problem, you can also skip it via the
packagesToSkip
in Config.hs
.
It's possible that required imports are not in scope. The importsForPackage
function in Config.hs
defines what imports should be added for each test file
for a package. You can modify that function to add necessary imports and then
re-generate the test suite.
Basically hlint should be happy and the test suites should pass:
make lint test
You can format (if you have stylish-haskell
installed) with:
make format