This repository contains the source code, data, and documentation for the research paper titled "Rematch: Robust and Efficient Knowledge Graph Matching for Improved Structural and Semantic Similarity".
An example of rematch’s similarity calculation for a pair of AMRs. After AMRs are parsed from sentences,
rematch has a two-step process to calculate similarity. First, sets of motifs are generated. Second, the two sets are
used to calculate the Jaccard similarity (intersecting motifs shown in color).
- [Author 1 Name] ([Affiliation])
- [Author 2 Name] ([Affiliation])
- ...
Knowledge graphs play a pivotal role in various applications, such as question-answering and fact-checking. Abstract Meaning Representation (AMR) represents text as knowledge graphs. Evaluating the quality of these graphs involves matching them structurally to each other and semantically to the source text. Existing AMR metrics are inefficient and struggle to capture semantic similarity. We also lack a systematic evaluation benchmark for assessing structural similarity between AMR graphs. To overcome these limitations, we introduce a novel AMR similarity metric, rematch, alongside a new evaluation for structural similarity called RARE. Among state-of-the-art metrics, rematch ranks second in structural similarity; and first in semantic similarity by 1--5 percentage points on the STS-B and SICK-R benchmarks. Rematch is also five times faster than the next most efficient metric.
[List relevant keywords here.]
-
Clone the repository:
git clone https://github.com/Zoher15/Rematch-RARE.git
-
Create and activate conda Environment:
conda env create -f rematch-rare.yml
conda activate rematch-rare
- License and download AMR Annotation 3.0
- Preprocess data by:
bash methods/preprocess_data/preprocess_amr3.sh <dir>
<dir>
is the directory where youramr_annotation_3.0_LDC2020T02.tgz
file is located
Steps to reproduce these results:
- Generate Randomized AMRs with Rewired Edges (RARE):
python experiments/structural_consistency/randomize_amr_rewire.py
- Evaluate any metric on RARE test:
bash experiments/structural_consistency/structural_consistency.sh <metric>
<metric>
should be one ofrematch
,smatch
,s2match
,sembleu
,wlk
orwwlk
. Depending on the metric, this could take a while to run.
Steps to reproduce these results:
-
Parse AMRs from STS-B and SICK-R:
a. Parse
AMR3-structbart-L-smpl
andAMR3-joint-ontowiki-seed42
by:bash experiments/semantic_consistency/parse_amrs.sh
b. (optional) Parse
Spring
by cloning the repo. Also download and unzip the AMR3 pretrained checkpoint. Ensure that the resulting unzipped file (AMR3.parsing.pt
) is in the cloned repo directoryspring/
. Then run (requires cuda):bash experiments/semantic_consistency/parse_spring.sh <spring_dir>
<spring_dir>
is where your Spring repo clone is located.c. (optional) Parse
Amrbart
by cloning my fork of the AMRBART repo. To resolve errors not addressed by the original repo, alsogit reset --hard 4110f1e
. Then run (requires cuda):bash experiments/semantic_consistency/parse_amrbart.sh <amrbart_dir>
<amrbart_dir>
is where your Amrbart repo clone is located. -
Evaluate a metric on the test set:
bash experiments/semantic_consistency/semantic_consistency.sh <metric> <parser>
<metric>
should be one ofrematch
,smatch
,s2match
,sembleu
,wlk
orwwlk
.<parser>
should be one ofAMR3-structbart-L-smpl
,AMR3-joint-ontowiki-seed42
,spring_unwiki
oramrbart_unwiki
. Ensure the chosen<parser>
has been executed in the previous step.
Please follow the instructions in the Bamboo repo. Do note that by default, Bamboo uses Pearsonr, but for our analysis we chose Spearmanr. That change can be made easily in the evaluation script by using find and replace. The word pearsonr
needs to be replaced with spearmanr
.
AMR Metric | Time(s) | RAM(GB) |
---|---|---|
smatch | 927 | 0.2 |
s2match | 7718 | 2 |
sembleu | 275 | 0.2 |
WLK | 315 | 30 |
rematch | 51 | 0.2 |
Steps to reproduce this experiment:
- Generate the time testbed by:
python experiments/efficiency/generate_matchups.py
- Evaluate a specific
<metric>
, one ofrematch
,smatch
,s2match
,sembleu
orwlk
:bash experiments/efficiency/efficiency.sh <metric>
- If all metrics have been executed, the plots from the paper can be reproduced by:
python experiments/efficiency/plot_complexity.py