Add traceloop / OpenLLMetry integration docs

[nirga]

No description provided.

[TuanaCelik]

Hi @nirga - Thanks for your contribution to the haystack integrations. Could I ask how this integrates with Haystack? To add this to the Haystack integrations page I think it's best to have a bit more information on how people can use it with Haystack pipelines, what steps they have to take to make monitoring start etc. Any additional info would be helpful!

[nirga]

Thanks @TuanaCelik for your response. That's the magic of it! We're monkey-patching Haystack internally so devs don't need to do anything and they get visibility in their existing observability platform.

[TuanaCelik]

@nirga it's still not fully clear to me. Here's a colab that you can see me trying to use this: https://colab.research.google.com/drive/1AIfIQBUZsSzeETRipBJtXOx2Wnld2QXJ?usp=sharing

Here's some things I've been able to figure our, if it makes sense to you, adding them to the integration page would be super useful 🙏

1. It seems that even though I can pip install and call Traceloop.init I would need to create an account to be able to access the UI. If this is correct, then would be great to add this info to the integration page.
2. I think we need to add a from traceloop.sdk import Traceloop for that line to work
3. Although I've added the init call in the colab, I'm not seeing anything show up in the UI after running some of these code cells. ANd looking at this pate: https://traceloop.com/docs/python-sdk it looks like users would have to add a decorator around their Haystack functions? Is that the case? If not how does it work?

Thanks in advance for all the info :)

[TuanaCelik]

More experiments :)
I've added this in that colab:

from traceloop.sdk.decorators import workflow

@workflow(name="video pipeline run")
def run(query: str):
  result = video_rag_pipeline.run(query)
  print(result['answers'][0].answer)
  return result['documents']

by looking at the sourcecode of openllmetry, and then I run the new 'run()' function. But still can't see any dashboard.
Any idea what I might be doing wrong?

[nirga]

@TuanaCelik excited to update that I ended up building a custom OpenTelemetry instrumentation for Haystack. It's all happening behind the scenes, and I was able to auto-instrument OpenAI with just one line of code (which is really exciting!). So for example this is what I ran:

import os
from haystack.nodes import PromptNode, PromptTemplate, AnswerParser
from haystack.pipelines import Pipeline
from traceloop.sdk import Traceloop

Traceloop.init(app_name="haystack_app")

prompt = PromptTemplate(
    prompt="Tell me a joke about {query}\n",
    output_parser=AnswerParser(),
)

prompt_node = PromptNode(
    model_name_or_path="gpt-4",
    api_key=os.getenv("OPENAI_API_KEY"),
    default_prompt_template=prompt,
)

pipeline = Pipeline()
pipeline.add_node(component=prompt_node, name="PromptNode", inputs=["Query"])

query = "OpenTelemetry"
result = pipeline.run(query)
print(result["answers"][0].answer)

And here are the results on our dashboard (have similar results on Honeycomb and Datadog):

And here are the results from your colab (you can track the trace from getting the data from weaviate all the way to OpenAI which is nice)

Let me know your thoughts and whether it makes sense to merge this.

[TuanaCelik]

@nirga this looks great, definitely makes sense to merge this. I think we could add these screenshots and code examples and info that you provide here to the .md file here too. I'm happy to do that later today and ask for you to have a look though. Excited for the community to see this :)

PS with some more info: Haystack is currently going through a major update to 2.0, which will still take some months. So we would likely ping you to make sure the integration still works when that happens or help you with figuring out how to update it. When the time comes...

[nirga]

Sounds good! 😃 thanks!

[TuanaCelik]

Already have a question. I think I'm doing something wrong, tried running my colab again and I go to the URL which the following line produces:

from traceloop.sdk import Traceloop

Traceloop.init(app_name="YouTube Haystack Demo")

But when I get to the last run function and run it, the traceloop dashboard is still empty. I don't have an account or anything though with traceloop. Could this be the issue? - I also see that the code snippet above creates a key for me. Should I use that somewhere?

[nirga]

That was a bug specific for OpenTelemetry + Colab (occurred because the spans weren't flushed after running a cell). Fixed now with v0.0.42 of the SDK.

[TuanaCelik]

Not on colab, but I got it to work on my local notebook! Making some suggestions and merging today 🎉