'ORGANIZATION' entity is not correctly identified

[Anusha-12]

I tired the evaluation script to validate the synthetic_data present in the data directory. The evaluation score is NaN for 'ORGANIZATION', it is not able to correctly predict the output

`

[omri374]

Thanks, we'll look into this. Apologies for the delayed response.

[Anusha-12]

Hi, is there any update with this?

[omri374]

Apologies for the delay. Which evaluation script are you running? and with which model/nlp engine?
I tried to run it as is (on the first 300 samples in the synthetic dataset, and this is what I got:

If you're getting other results, please share a reproducible example and make sure you're using the latest code in the repo.

[Anusha-12]

yes, I am using the Presidio Analyzer with 'Jean-Baptiste/roberta-large-ner-english' and I got different results. I will upload the graph

[omri374]

I tried to reproduce with this model. This is the configuration I used:

from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import TransformersNlpEngine, NerModelConfiguration


# Here we define a transformers based NLP engine, 
# but you can use this cell to customize your Presidio Analyzer instance

# Define which model to use
model_config = [{"lang_code": "en", "model_name": {
    "spacy": "en_core_web_sm",  # use a small spaCy model for lemmas, tokens etc.
    "transformers": "Jean-Baptiste/roberta-large-ner-english"
    }
}]


# Map transformers model labels to Presidio's
model_to_presidio_entity_mapping = dict(
    PER="PERSON",
    LOC= "LOCATION",
    ORG="ORGANIZATION"
)

ner_model_configuration = NerModelConfiguration(labels_to_ignore = ["O"], 
                                                model_to_presidio_entity_mapping=model_to_presidio_entity_mapping)

nlp_engine = TransformersNlpEngine(models=model_config,
                                   ner_model_configuration=ner_model_configuration)

# Set up the engine, loads the NLP module (spaCy model by default) 
# and other PII recognizers
analyzer_engine = AnalyzerEngine(nlp_engine=nlp_engine)

Results did show ORGANIZATION metrics, but please try to reproduce too and let me know if you were able to get this working.