Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PromptNode doesn't support isolated node evaluation #3275

Closed
Timoeller opened this issue Sep 24, 2022 · 2 comments
Closed

PromptNode doesn't support isolated node evaluation #3275

Timoeller opened this issue Sep 24, 2022 · 2 comments
Labels
P2 Medium priority, add to the next sprint if no P1 available topic:pipeline type:bug Something isn't working

Comments

@Timoeller
Copy link
Contributor

Describe the bug
When running pipeline.eval_batch with a GenerativeQAPipeline the isolated node eval options seems to be ignored.

Additional context

  1. When running the same pipeline without batching it adds the isolated node eval. Only in eval_batch it dissmisses the parameter.
  2. Running eval_batch with an ExtractiveQAPipeline does not ignore the isolated eval param.

To Reproduce
please take eval labels from the eval tutorial

    retriever = BM25Retriever(
        document_store=document_store
    )
    generator = Seq2SeqGenerator(model_name_or_path="vblagoje/bart_lfqa")
    pipeline = GenerativeQAPipeline(generator=generator, retriever=retriever)
    eval_result = pipeline.eval_batch(labels=eval_labels[:5],
                                      params={"Retriever": {"top_k": 10}, "Generator": {"top_k": 2}},
                                      sas_model_name_or_path="cross-encoder/stsb-roberta-large",
                                      add_isolated_node_eval=True)

@Timoeller Timoeller added the type:bug Something isn't working label Sep 24, 2022
@masci masci added the P2 Medium priority, add to the next sprint if no P1 available label Nov 24, 2022
@masci masci added P3 Low priority, leave it in the backlog and removed P2 Medium priority, add to the next sprint if no P1 available labels Jan 25, 2023
@julian-risch julian-risch added P2 Medium priority, add to the next sprint if no P1 available and removed P2 Medium priority, add to the next sprint if no P1 available P3 Low priority, leave it in the backlog labels Mar 13, 2023
@bogdankostic bogdankostic removed their assignment Jul 6, 2023
@bogdankostic bogdankostic changed the title Failed to add isolated node eval when running pipeline.eval_batch with a GenerativeQAPipeline PromptNode doesn't support isolated node evaluation Jul 6, 2023
@bogdankostic
Copy link
Contributor

#5291 adds support for isolated node evaluation in BaseGenerator.

Given that we removed Seq2SeqGenerator and RAGenerator and recommend to use PromptNode instead, I investigated the support for isolated node evaluation in PromptNode. However, I found that we currently do not support it at all. Let's keep this issue open to think about if and how we want to support isolated node evaluation in PromptNode.

@masci
Copy link
Contributor

masci commented Dec 15, 2023

Closing as won't fix

@masci masci closed this as completed Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 Medium priority, add to the next sprint if no P1 available topic:pipeline type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants