Merge pull request #53 from anthropics/skill-dev-garvan

Classification Guide

skills/README.md

@@ -0,0 +1,12 @@
+# Claude Skills
+
+Welcome to the Skills section of the Anthropic Cookbook! This directory contains a collection of guides that showcase specific skills and capabilities where Claude excels. Each guide provides an in-depth exploration of a particular skill, discussing potential use cases, prompt engineering techniques to optimize results, and approaches for evaluating Claude's performance.
+
+## Guides
+
+- **[Classification with Claude](./classification/guide.ipynb)**: Discover how Claude can revolutionize classification tasks, especially in scenarios with complex business rules and limited training data. This guide walks you through data preparation, prompt engineering with retrieval-augmented generation (RAG), testing, and evaluation.
+
+
+## Getting Started
+
+To get started with the Skills guides, simply navigate to the desired guide's directory and follow the instructions provided in the `guide.ipynb` file. Each guide is self-contained and includes all the necessary code, data, and evaluation scripts to reproduce the examples and experiments.

skills/classification/data/train.tsv

@@ -0,0 +1,69 @@
+text label
+I just got my auto policy renewal bill and the cost seems to be more than what I usually pay. Could you explain the reason for the increase? Billing Inquiries
+The premium for my car insurance renewal is higher than I expected. Can you clarify why there was a price increase compared to my previous policy? Billing Inquiries
+I need some help understanding why my auto insurance bill for the upcoming term is more expensive than it has been in the past. Could you provide an explanation? Billing Inquiries
+I would like to modify the coverage limits and deductibles on my homeowners insurance policy. What choices do I have available? Policy Administration
+Can you tell me what my options are for changing the deductibles and coverage limits associated with my home insurance policy? Policy Administration
+I'm interested in adjusting the deductibles and coverage amounts for my homeowners policy. Could you outline the different possibilities I have? Policy Administration
+Another vehicle hit my parked car, but I didn't get any information from the other driver. What's the process for submitting a claim in this situation? Claims Assistance
+My parked vehicle was damaged by another car, but I don't have the details of the other driver. How should I go about filing an insurance claim? Claims Assistance
+Someone hit my car while it was parked, but I couldn't get their information. Can you explain what I need to do to file a claim without the other driver's details? Claims Assistance
+I don't fully grasp the distinction between comprehensive and collision coverage on my auto policy. Could you explain what each type of coverage entails? Coverage Explanations
+What's the difference between collision and comprehensive car insurance? I'm not sure I understand what's included in each one. Coverage Explanations
+Can you help me understand how collision coverage differs from comprehensive coverage in terms of what kinds of incidents and damages are covered under my auto policy? Coverage Explanations
+I'm thinking about changing my insurance provider to your company. Would you be able to give me a quote that is equivalent to or better than my existing coverage? Quotes and Proposals
+I'm considering moving my insurance policies to your firm. Is it possible for you to provide a quote that offers the same or better coverage compared to what I currently have? Quotes and Proposals
+I'm interested in switching my insurance to your company, but I want to ensure I'm getting a good deal. Can you provide me with a competitive quote that meets or exceeds my current coverage level? Quotes and Proposals
+I attempted to sign in to my online account, but it's saying my password isn't correct, even though I never changed it. What could be causing this problem? Account Management
+When I tried logging into my web-based account, it told me my password was invalid, but I'm certain I didn't change it. Can you help me figure out why this is happening? Account Management
+I'm having trouble accessing my online insurance account. It says my password is wrong, but I'm sure I'm using the right one. What might be going on here? Account Management
+There's an unfamiliar charge from your company on my credit card statement that doesn't match my usual premium payment. Could you please investigate this for me? Billing Disputes
+I've noticed a charge on my credit card from your business that I don't recognize, and it's a different amount than my typical insurance premium. Can you look into this and let me know what it's for? Billing Disputes
+I spotted an unidentified charge from your insurance company on my credit card bill, and it's not the same as my regular premium payment. Would you mind checking on this and explaining the charge to me? Billing Disputes
+Your company denied my claim for water damage in my kitchen, but I think it should be covered based on my policy. What's the process for appealing this decision? Claims Disputes
+I filed a claim for water damage to my kitchen, but it was rejected even though I believe my policy covers it. Can you explain how I can dispute this denial and have my claim reconsidered? Claims Disputes
+My water damage claim for my kitchen was turned down, but I feel it should be covered under the terms of my insurance policy. What steps can I take to challenge this decision and get my claim approved? Claims Disputes
+I'm trying to decide between a high-deductible health plan and a traditional PPO for my insurance coverage. Can you explain the advantages and disadvantages of each option to help me choose? Policy Comparisons
+I'm considering two different health insurance plans - a high-deductible policy and a standard PPO. Could you outline the benefits and drawbacks of both to assist me in making a decision? Policy Comparisons
+I received some correspondence about modifications to my insurance policy, but I don't fully understand what they entail. Would you be able to provide more information about these changes? General Inquiries
+Your company sent me a letter describing some adjustments to my policy, but the details are a bit unclear to me. Can you give me a better explanation of what these changes mean for my coverage? General Inquiries
+I got a letter in the mail outlining some updates to my insurance policy, but I'm unsure how they affect me. Could you clarify the implications of these changes? General Inquiries
+I recently added another vehicle to my auto policy, but the multi-car discount I was told I would receive isn't showing up on my bill. Can you please review this and make sure the discount is applied? Billing Inquiries
+I added a second car to my insurance policy recently, but my billing statement doesn't include the multi-vehicle discount I was promised. Could you double-check this and ensure I'm getting the appropriate discount? Billing Inquiries
+After adding a new car to my auto insurance, I noticed that the multi-car discount I was expecting isn't reflected on my latest bill. Would you mind looking into this and verifying that the discount has been properly applied to my account? Billing Inquiries
+My home suffered hail damage, and I need to submit a claim. However, I'm unsure whether I should file it under my homeowners policy or my separate windstorm coverage. What would you advise? Claims Assistance
+Hail caused some damage to my house, and I want to file an insurance claim. Should I submit it through my homeowners policy or my windstorm policy? I'm not certain which one is appropriate for this situation. Claims Assistance
+I have to file a claim for hail damage to my home's roof, but I don't know if it should be filed under my primary homeowners insurance or my additional windstorm policy. Can you provide guidance on which policy to use? Claims Assistance
+My auto insurance includes coverage for rental cars, but I'm not clear on the specifics. What are the daily and per-incident limits for rental car coverage under my policy? Coverage Explanations
+I know my car insurance policy has some kind of rental car coverage, but I don't fully understand the details. Can you tell me how much is covered per day and per event when I need to rent a vehicle? Coverage Explanations
+My auto policy mentions rental car coverage, but I'm unsure about the particulars. Could you explain what the daily and per-incident allowances are for rental car expenses under my insurance plan? Coverage Explanations
+I'm in the market for life insurance and am considering both term and whole life policies. Can you clarify the differences between the two and offer some guidance on which one might be best for me? Quotes and Proposals
+As I shop around for life insurance, I'm weighing the options of term life versus whole life coverage. Could you explain how these two types of policies differ and provide some advice on choosing between them based on my needs? Quotes and Proposals
+I want to set up paperless billing and have my premiums automatically deducted from my account. What's the process for enabling these features through the online portal? Account Management
+Can you walk me through the steps to go paperless and set up automatic payments for my insurance premiums using my online account? Account Management
+I'd like to switch to paperless statements and set up auto-pay for my insurance bills. How can I make these changes through my web-based account? Account Management
+I suspect I'm being charged twice for the motorcycle endorsement on my auto policy. The cost of my premium increased, but I don't see an itemized charge for the endorsement. Can you help me understand what's going on? Billing Disputes
+It looks like I might be getting double-billed for my motorcycle endorsement. My insurance rates went up, but the endorsement fee isn't listed as a separate item. Could you clarify this for me and make sure I'm not being overcharged? Billing Disputes
+I think there may be a duplicate charge for my motorcycle endorsement, as my premium has increased without the endorsement being clearly listed on my bill. Can you investigate this and explain why the price has gone up? Billing Disputes
+Following an accident, the repair estimate for my car is greater than the amount approved by your claims adjuster. What's the best way to contest this difference and get the full cost of repairs covered? Claims Disputes
+I took my vehicle to the shop after an accident, and their estimate for the work is higher than what your adjuster authorized. How can I dispute this discrepancy and ensure that the entire repair bill is paid for by my insurance? Claims Disputes
+My insurance agent mentioned a "vanishing deductible" option, but I don't entirely grasp what that means. Can you provide some additional details about how this feature works and what benefits it offers? General Inquiries
+I heard something about a "disappearing deductible" from my insurance representative, but I'm not completely clear on the concept. Could you give me a more thorough explanation of this option and how it might impact my policy? General Inquiries
+I'm a bit confused about the "disappearing deductible" my agent brought up. Would you mind elaborating on what this entails and how it could affect my insurance coverage? General Inquiries
+As I prepare for retirement, I'm considering making some adjustments to my life insurance policy. What changes would you recommend based on my upcoming shift in lifestyle and financial needs? Policy Administration
+With retirement on the horizon, I'm thinking about updating my life insurance coverage to better suit my changing circumstances. Can you suggest some modifications I should make to my policy as I enter this new phase of life? Policy Administration
+Given my impending retirement, I believe it's time to reassess my life insurance needs and make any necessary alterations to my coverage. What advice can you offer about adapting my policy to best fit my post-retirement situation? Policy Administration
+My home's basement was flooded due to a malfunctioning sump pump, and I'm uncertain whether this type of water damage is covered by my homeowners insurance policy. Can you clarify which kinds of water damage are included in my coverage? Coverage Explanations
+After a sump pump failure caused my basement to flood, I'm wondering if my homeowners policy will cover the resulting water damage. Could you explain what types of water-related incidents are typically included under this type of insurance? Coverage Explanations
+I received a notice that my insurance policy is being canceled for non-payment, but I'm almost positive I paid my premium before the due date. Could you please look into this matter and reinstate my coverage if there's been an error? Billing Disputes
+According to the cancellation notice I received, my insurance is being terminated for failure to pay my premium on time. However, I believe I made the payment by the deadline. Can you investigate this situation and restore my policy if a mistake has occurred? Billing Disputes
+I got a letter saying my insurance is being canceled because of an overdue payment, but I think I submitted the payment prior to the cutoff date. Would you be able to research this issue and reactivate my policy if it turns out there was a misunderstanding or processing error? Billing Disputes
+I brought my car to the repair shop following an accident, but their estimate for the work is significantly higher than the amount quoted by your insurance appraiser. How should I proceed in resolving this price discrepancy? Claims Disputes
+After getting into an accident, I took my vehicle to the mechanic for repairs. However, the shop's estimate is much steeper than what your appraiser initially provided. What's the best course of action for addressing this difference in cost? Claims Disputes
+The auto body shop I went to after my accident gave me a repair estimate that's way above what your appraiser said it would be. Can you advise me on how to handle this situation and come to a resolution on the price of the repairs? Claims Disputes
+I'm comparing two health insurance plans that have comparable monthly premiums but different out-of-pocket maximum amounts. Based on my health history, which option would you suggest I choose? Policy Comparisons
+I've narrowed down my health insurance choices to two policies with similar premiums but contrasting out-of-pocket maximums. Considering my medical background, which plan do you think would be the wiser selection? Policy Comparisons
+Given my past medical experiences, I'm trying to decide between two health insurance policies that have nearly equivalent premiums but vary in their out-of-pocket maximum limits. Which one would you recommend for my specific circumstances? Policy Comparisons
+Can you provide more information about how your accident forgiveness program operates and what criteria must be met to be eligible for this benefit? General Inquiries
+I'm interested in learning more about your accident forgiveness offering. Could you explain the details of how it works and what qualifications policyholders need to satisfy to take advantage of this feature? General Inquiries
+I'd like to better understand your company's accident forgiveness option. Would you mind describing how it functions and outlining the requirements customers must fulfill to be eligible for this perk? General Inquiries

skills/classification/evaluation/README.md

@@ -0,0 +1,57 @@
+# Evaluations with Promptfoo
+
+
+
+### Pre-requisities 
+To use Promptfoo you will need to have node.js & npm installed on your system. For more information follow [this guide](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) 
+
+You can install promptfoo using npm or run it directly using npx. In this guide we will use npx. 
+
+*Note: For this example you will not need to run `npx promptfoo@latest init` there is already an initialized `promptfooconfig.yaml` file in this directory* 
+
+See the official docs [here](https://www.promptfoo.dev/docs/getting-started) 
+
+
+
+### Getting Started
+The evaluation is orchestrated by the `promptfooconfig.yaml` file. In this file we define the following sections:
+
+- Prompts
+ - Promptfoo enables you to import prompts in many different formats. You can read more about this [here](https://www.promptfoo.dev/docs/configuration/parameters).
+ - In this example we will load 3 prompts - the same used in `guide.ipynb` from the `prompts.py` file:
+ - The functions are identical to those used in `guide.ipynb` except that instead of calling the Anthropic API they just return the prompt. Promptfoo then handles the orchestration of calling the API and storing the results.
+ - You can read more about prompt functions [here](https://www.promptfoo.dev/docs/configuration/parameters#prompt-functions). Using python allows us to reuse the VectorDB class which is necessary for RAG, this is defined in `vectordb.py`.
+- Providers
+ - With Promptfoo you can connect to many different LLMs from different platforms, see [here for more](https://www.promptfoo.dev/docs/providers). In `guide.ipynb` we used Haiku with default temperature 0.0. We will use Promptfoo to experiment with an array of different temperature settings to identify the optimal choice for our use case.
+- Tests
+ - We will use the same data that was used in `guide.ipynb` which can be found in this [Google Sheet](https://docs.google.com/spreadsheets/d/1UwbrWCWsTFGVshyOfY2ywtf5BEt7pUcJEGYZDkfkufU/edit#gid=0).
+ - Promptfoo has a wide array of built in tests which can be found [here](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic).
+ - In this example we will define a test in our `dataset.csv` as the conditions of our evaluation change with each row and a test in the `promptfooconfig.yaml` for conditions that are consistent across all test cases. Read more about this [here](https://www.promptfoo.dev/docs/configuration/parameters/#import-from-csv)
+- Transform
+ - In the `defaultTest` section we define a transform function. This is a python function which extracts the specific output we want to test from the LLM response. 
+- Output
+ - We define the path for the output file. Promptfoo can output results in many formats, [see here](https://www.promptfoo.dev/docs/configuration/parameters/#output-file). Alternatively you can use Promptfoo's web UI, [see here](https://www.promptfoo.dev/docs/usage/web-ui).
+
+
+### Run the eval
+
+To get started with Promptfoo open your terminal and navigate to this directory (`./evaluation`).
+
+Before running your evaluation you must define the following enviroment variables:
+
+`export ANTHROPIC_API_KEY=YOUR_API_KEY` 
+`export VOYAGE_API_KEY=YOUR_API_KEY`
+
+From the `evaluation` directory, run the following command. 
+
+`npx promptfoo@latest eval`
+
+If you would like to increase the concurrency of the requests (default = 4), run the following command. 
+
+`npx promptfoo@latest eval -j 25` 
+
+When the evaluation is complete the terminal will print the results for each row in the dataset.
+
+You can now go back to `guide.ipynb` to analyze the results!
+
+