Questions about CI/CD and Automated Conversation Testing

[sfacci]

Documentation is unclear/incomplete

Have fully automated conversations with a CX agent (powerful for regression testing!)

The documentation suggests that this library has features which allow creation of automated conversations with your agent, but I could not find any description of how this might be done anywhere in the repo.

If this is referring to specific methods included in the library, an example should be written in a pynb file, or the documentation should be clarified that this requires using methods or approach from another library. What test approach is being suggested? A python script living somewhere in GCP that uses SCRAPI to create an agent, and then what?

Create, Update, Delete, Get, and List for all CX resources types (i.e. Intents, Entity Types, Pages, Flows, etc.)

Google's Cloud Skill Boost training claims that SCRAPI can help with CI/CD and release automation but this documentation says very little about that. https://www.cloudskillsboost.google/course_templates/1002/video/466428

It says we can integrate with Bitbucket, but there is no guide on how to do that, or what can be stored there. It seems Dialogflow CX itself only integrates with GitHub. Are we to understand that SCRAPI code can be stored in Bitbucket and then accessed from Cloud Run or another GCP service somehow? Only from the gcloud CLI? If we use SCRAPI, does this mean our agent must be rebuilt from python code each time it is deployed? Or can it work with an agent built in the UI and synced to GitHub?

[kmaphoenix]

@sfacci

Have fully automated conversations with a CX agent (powerful for regression testing!)

All of the Sample code is in the examples folder at the top level of the repo.
The notebook you're likely looking for is this one:
https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/examples/nlu_analysis_series/nlu_evaluation_testing.ipynb

If you'd prefer to build a script yourself, the best starting points are from the DialogflowConversation class or the Sessions class.

The former (DialogflowConverstaions) is a wrapper around Sessions to make it easier for handling multi-turn conversations programmatically.
If you want the base implementation you can use Sessions directly.

What test approach is being suggested? A python script living somewhere in GCP that uses SCRAPI to create an agent, and then what?

This is 100% dependent on your setup, org structure, development capabilities, etc.
Some users simply use the jupyter notebook in a Google Colab and that is good enough for them.
Others will convert to a .py file and put into a serverless setup like Cloud Function / Run and trigger it periodically using Scheduler or something similar.
The library is meant to be flexible so you can use it however you see fit.

Google's Cloud Skill Boost training claims that SCRAPI can help with CI/CD and release automation but this documentation says very little about that

For CI/CD, the easiest approach is to utilize the Export / Restore features.
Each Agent can be exported as a JSON Package which works great with any git setups because you and easily view file diffs as they are made.

The exact push/pull workflow depends on your development setup, but a general one is something like this:

• Users make changes in the CX UI
• Periodically the Agent is Exported in JSON Package format and pushed to git repo (you choose the cadence)
• At the git repo level, you handle merge conflicts, etc.
• From the git repo, you pull the entire Agent in JSON format then use the Restore method to load the Agent to whatever destination you want

Again, there can be many variations on this depending on your setup.
Perhaps no one uses the UI at your company and instead would prefer to make operations directly on the JSON.
Or maybe you want each user to have their own working copy of the Agent (like a feature branch) and then you merge those together at the repo level.

To be clear, SCRAPI itself doesn't manage any of the Git functionality.
It just helps to facilitate some of the functions like Export / Restore in a more programmatic way.

You could also incorporate CRUD functions into this workflow as well, which many orgs do.
For example, when moving between GCP Environments / Projects, you might need to update Webhook endpoints, specific content in the bot, etc.
You can use the CRUD methods in SCRAPI to accomplish these actions quickly and easily.

I'll leave this issue open for a few days in case you have any questions.
Hope this helps.