-
-
Notifications
You must be signed in to change notification settings - Fork 240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slice management: Create slices in UI and apply them over inspections #606
Conversation
backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java
Outdated
Show resolved
Hide resolved
4e735af
to
88271e8
Compare
e7bc854
to
90ca6bd
Compare
backend/src/main/java/ai/giskard/web/rest/controllers/SliceController.java
Outdated
Show resolved
Hide resolved
|
||
const items = computed(() => { | ||
// TODO: Find a solution for this typing issue | ||
let result: any[] = slices.value.map(slice => ({ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe have a type AutocompleteItem
defined as following
{
text: string,
name: string,
index: number | null
}
For each item in the slices
array have either the index in the array or ID of the slice to retrieve it latter on and for the __NEW_SLICE__
item have -1
I think it would be cleaner.
There's a buggy behaviour when switching from less selective slice to more selective one when we're outside of a new range: Screen.Recording.2023-01-10.at.23.29.14.mov |
Slice gets unselected after being edited: Screen.Recording.2023-01-10.at.23.31.49.mov |
backend/src/main/resources/config/liquibase/changelog/20221215151953_changelog.xml
Outdated
Show resolved
Hide resolved
backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java
Outdated
Show resolved
Hide resolved
- Current dataset row is reset on slice application - Editing will keep the slice selected if it was and refresh
Thanks for the functional feedback, it was an oversight on my part. I've solved the first 2 by:
As for the sorting, I'm open to any solution (name or creation date). Sorting by name would still cause a move when editing the name, which we may want to avoid ? I'll take a look at the code feedback tomorrow :) |
General commentI feel that it's a bit cumbersome to iterate while creating a slicing function. My workflow was:
on each step, I had to close the modal and reopen it again. It's a bit disappointing, especially if there's a silly syntax error. It could be great if we could quickly validate the slice function with a chosen dataset without leaving the editing mode.That was the idea behind the "Run" button on this mockup Secondly, I might have missed some discussions last week, but I think slice creation being part of an inspection makes slices too hidden for users. It also unnecessarily requires a model to exist (without which you won't be able to create an inspection). In principle, a slice function should only require a dataset, so I think the previous location of the slice functions list (next to datasets) made more sense. |
Hey Andrey, thanks for the feedback again. I'll reply here to some of the points you mentionned:
6 I agree with this too, using it yesterday made me realise it's not a good workflow currently. As for your general feedback, we discussed it with Jean Marie and since Slices are (for now) only relevant for inspections we decided to put them there. With a future version having a test catalogue, we imagined we could have a "function catalogue" that works in a similar way, with slicing functions, perturbation functions and other types as the product evolves. There is an in-between then and now that we need to figure out if the dropdown solution isn't good enough for now, that's for sure |
backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java
Outdated
Show resolved
Hide resolved
backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java
Outdated
Show resolved
Hide resolved
backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java
Outdated
Show resolved
Hide resolved
backend/src/main/java/ai/giskard/service/ml/MLWorkerService.java
Outdated
Show resolved
Hide resolved
result = mlWorkerService.filterDataset(client, dataset, slice.getCode()); | ||
Files.createDirectories(cachedSliceFile.getParent()); | ||
Files.createFile(cachedSliceFile); | ||
try (DataOutputStream outputStream = new DataOutputStream(Files.newOutputStream(cachedSliceFile))) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's not expensive to write cache as text file, not binary. It'll make our life easier when debugging, plus we already store prediction results as text, so this cache file won't be larger than that
backend/src/main/java/ai/giskard/web/rest/controllers/InspectionController.java
Outdated
Show resolved
Hide resolved
backend/src/main/java/ai/giskard/web/rest/controllers/InspectionController.java
Outdated
Show resolved
Hide resolved
I agree with both solutions you proposed for 4 and 6. As for the definitive place for the slices list, lets keep it inside inspection for now, but keep the "General comment" in mind for the moment when we start using slices in tests too. I'd still enhance the slice editor to be able to try to run it on a dataset directly without leaving the editor |
Small todo for myself, increase slice name length limit, too restrictive atm |
# Conflicts: # frontend/src/api.ts
@Googleton , could you have a last look at Sonar results? I don't think everything's coming from your changes though... |
# Conflicts: # backend/src/main/java/ai/giskard/service/FileLocationService.java # frontend/src/generated-sources/index.ts # frontend/src/generated-sources/j2ts-generated-metadata.json
# Conflicts: # frontend/src/views/main/profile/UserProfile.vue
…oc/397-slice-management
Kudos, SonarCloud Quality Gate passed! |
Description
Slices are now in Giskard!
The UI is currently only located inside of the Inspector. This is what it looks like:
![Screenshot 2023-01-10 at 10 37 11](https://user-images.githubusercontent.com/1703741/211515295-bfa15b14-e6bf-4bf7-bab4-ea47a5d62244.png)
![Screenshot 2023-01-10 at 10 37 29](https://user-images.githubusercontent.com/1703741/211515384-b2bdcaaf-a4ae-46d9-9058-ad9ffa2934ef.png)
![Screenshot 2023-01-10 at 10 37 45](https://user-images.githubusercontent.com/1703741/211515508-8713e053-2d2d-4623-8a49-23fcd2397c74.png)
![Screenshot 2023-01-10 at 10 38 29](https://user-images.githubusercontent.com/1703741/211515685-d06ec29e-b995-4925-98f1-872ea4f4ddcb.png)