FEAT: Crescendo part 1 (scorers) #203
Conversation
| @@ -2,7 +2,7 @@ | |||
| "cells": [ | |||
| { | |||
There was a problem hiding this comment.
can remove this file for now
| @@ -113,3 +115,10 @@ def _validate(self, scorer_type, score_value): | |||
| raise ValueError(f"Float scale scorers must have a score value between 0 and 1. Got {score_value}") | |||
| except ValueError: | |||
| raise ValueError(f"Float scale scorers require a numeric score value. Got {score_value}") | |||
| elif scorer_type == "severity": | |||
| try: | |||
There was a problem hiding this comment.
I think we can use float_scale for this; there is a method to normalize any value to one between 0.0 and 1.0
IMO we should delete this
There was a problem hiding this comment.
I dont think we should. The severity documented are specific int values. While we can represent it as a float (e.g. 2-> 0.02) I think it would be confusing for a caller.
| # These LLMs are heavily aligned to resist engaging in illegal or unethical topics as a means to avoid contributing to responsible AI harms. | ||
| # However, a recent line of attacks, known as "jailbreaks", seek to overcome this alignment. Intuitively, jailbreak attacks aim to narrow | ||
| # the gap between what the model can do and what it is willing to do. In this paper, we introduce a novel jailbreak attack called Crescendo. | ||
| # Unlike existing jailbreak methods, Crescendo is a multi-turn jailbreak that interacts with the model in a seemingly benign manner. |
There was a problem hiding this comment.
This looks like a great start!
But can we remove all non-scoring pieces to this PR? E.g. this, anything and anything with the orchestrator
E.g. remove this, orchestrator, etc. (you can keep the code for future PRs, just branch off of your current branch and remove them from this from the current PR)
| description: | | ||
| A variant of the crescendo attack technique | ||
| harm_category: NA | ||
| author: Ahmed Salem |
There was a problem hiding this comment.
I'd remove these for now
| @@ -264,6 +264,13 @@ def __str__(self): | |||
| return self.strategy.apply_custom_metaprompt_parameters(**self.kwargs) | |||
|
|
|||
|
|
|||
| class ToolCall(BaseModel): | |||
There was a problem hiding this comment.
it's not clear to me what this is
|
|
||
| self._harm_category = harm_category | ||
|
|
||
| if azure_content_safety_key is not None and azure_content_safety_endpoint is not None: |
There was a problem hiding this comment.
Can we add env keys for this and get the default values? See most of our targets on examples of how to do this. It should also be added to our env_example :)
There was a problem hiding this comment.
e.g. self._api_key = default_values.get_required_value(
env_var_name=self.API_KEY_ENVIRONMENT_VARIABLE, passed_value=api_key
)
|
|
||
| request = AnalyzeTextOptions(text=request_response.converted_value, output_type="EightSeverityLevels") | ||
| # Analyze text and geting the results for the category specified | ||
|
|
There was a problem hiding this comment.
we should use AnalyzeTextAsync and await the method
| """ | ||
| self.validate(request_response) | ||
|
|
||
| request = AnalyzeTextOptions(text=request_response.converted_value, output_type="EightSeverityLevels") |
There was a problem hiding this comment.
It would be nice to do images here too, and just AnaylzeImage depending ont the reqeust_response data type
There was a problem hiding this comment.
(not a blocker, but nice to have)
| # Analyze text and geting the results for the category specified | ||
|
|
||
| response = self._azureCFClient.analyze_text(request) | ||
| result = next((item for item in response.categories_analysis if item.category == self._harm_category), None) |
There was a problem hiding this comment.
Set score_type to "float_scale"
And for score_value call this (I think they range from 1 to 8?)
score_value = self.scale_value_float(float(parsed_response["score_value"]), 1, 8)
| score = Score( | ||
| score_type="severity", | ||
| score_value=result.severity, | ||
| score_value_description="severity", |
There was a problem hiding this comment.
description can be "none" or to be consistent it would be with whatever azure defines as the specific descriptions for each of the levels
| from pyrit.models import PromptRequestPiece | ||
|
|
||
|
|
||
| class AzureContentFilter(Scorer): |
There was a problem hiding this comment.
I'd rename to AzureContentFilterScorer (and rename this class)
There was a problem hiding this comment.
For this PR, you may want to just have "azure_content_filter_scorer". That is a concrete piece of work that is close to done and would immediately provide value. Then You could title this PR "Azure Content Filter Scorer"
The second PR I would include the other three scorers: eval_prompt, meta_judge_prompt, and refuse_judge_propmt
I really like these three prompts eval_prompt, meta_judge_prompt, and refuse_judge_propmt. Broadly, I think they should be broken up into two separate scorers
- SelfAskVerifyScore, which contains meta_judge prompt and is a TrueFalse scorer
- SelfAskConversationObjectiveScorer, I would also probably make this a TrueFalse scorer, and put the value from 0 to 1 in metadata. I like that result_percentage, we may add something like that to all TrueFalse scorers, but for now I think metadata will allow you to do operations on it.
IMO this would allow us to reuse, and each of these will be useful in other scoring situations.
There are some nits too, like the prompts should go in datasets.
I also really like the various crescendo scorers. I would make that the second pr. Broadly, I think they should be broken up into three separate scorers
|
Closing in favor of #275 |
Description
Implement the first part of the crescendo attack strategy. So far adding the relevant scorers and prompt templates.
Tests and Documentation
Added unit tests and scoring notebook sections to test out the scorers.