Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slice management: Create slices in UI and apply them over inspections #606

Merged
merged 23 commits into from
Feb 2, 2023

Conversation

Googleton
Copy link
Contributor

@Googleton Googleton commented Dec 14, 2022

Description

Slices are now in Giskard!

The UI is currently only located inside of the Inspector. This is what it looks like:
Screenshot 2023-01-10 at 10 37 11
Screenshot 2023-01-10 at 10 37 29
Screenshot 2023-01-10 at 10 37 45
Screenshot 2023-01-10 at 10 38 29

@Googleton Googleton linked an issue Dec 14, 2022 that may be closed by this pull request
@andreybavt andreybavt self-requested a review December 14, 2022 17:13
@andreybavt andreybavt added the wip label Dec 29, 2022
@Googleton Googleton changed the title Poc/397 slice management Slice management: Create slices in UI and apply them over inspections Jan 10, 2023
@Googleton Googleton marked this pull request as ready for review January 10, 2023 09:36
frontend/src/components/SliceDropdown.vue Outdated Show resolved Hide resolved

const items = computed(() => {
// TODO: Find a solution for this typing issue
let result: any[] = slices.value.map(slice => ({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe have a type AutocompleteItem defined as following

{
  text: string,
  name: string,
  index: number | null
}

For each item in the slices array have either the index in the array or ID of the slice to retrieve it latter on and for the __NEW_SLICE__ item have -1

I think it would be cleaner.

@andreybavt
Copy link
Contributor

There's a buggy behaviour when switching from less selective slice to more selective one when we're outside of a new range:

Screen.Recording.2023-01-10.at.23.29.14.mov

@andreybavt
Copy link
Contributor

Slice gets unselected after being edited:

Screen.Recording.2023-01-10.at.23.31.49.mov

@andreybavt
Copy link
Contributor

Slices change order in the dropdown. I had test first and all after, then modified test and it became second
image

Should we sort them by name?

- Current dataset row is reset on slice application
- Editing will keep the slice selected if it was and refresh
@Googleton
Copy link
Contributor Author

Thanks for the functional feedback, it was an oversight on my part.

I've solved the first 2 by:

  • Resetting the dataset row number to 0 (well, 1) when applying a slice
  • Selecting the new edited slice instance if it was selected previously

As for the sorting, I'm open to any solution (name or creation date). Sorting by name would still cause a move when editing the name, which we may want to avoid ?

I'll take a look at the code feedback tomorrow :)

@andreybavt
Copy link
Contributor

Another general feedback: let's have a friendly message when you apply a filter that's too strict so that there are no matching rows instead of showing empty forms.

image

P.S.
no need to do it at midnight ;)

@andreybavt
Copy link
Contributor

  1. Let's limit the max height of a name element in the slice creation modal so that the update/cancel buttons don't disappear

image

  1. Could you remove the autocomplete (control+space feature)from the code editor that results into a black horizontal line? Have a look at the monaco editor on test creation screen. Also I don't think we need an overview widget neither since the slicing functions will usually be small.

image

  1. Minor improvement: could you add autofocus on "name" field when the modal opens? It could also be more intelligent and focus on name when you create a slice and code when you edit it

  2. I'd make a distinction between an "Edit slice" modal title and the slice name itself, maybe print the slice name on a new line?
    image

  3. Long slice names break saving operation. There could be a character limit on the UI and validation on the backend to prevent it.
    image

  4. When playing with the feature I found it a bit long to navigate to slice editing, since I had to adapt my slice function frequently and see the results. I think we could remove 1 click by placing edit/delete icons next to the slice name in the list. Maybe in order to make the UI lighter we could only show these buttons for the slice that you've currently hovering over
    image

General comment

I feel that it's a bit cumbersome to iterate while creating a slicing function. My workflow was:

  • write a simple function
  • check that it works (no syntax error, the resulting rows are the ones that I expected)
  • modify the condition to make it more sophisticated

on each step, I had to close the modal and reopen it again. It's a bit disappointing, especially if there's a silly syntax error.

It could be great if we could quickly validate the slice function with a chosen dataset without leaving the editing mode.That was the idea behind the "Run" button on this mockup

Secondly, I might have missed some discussions last week, but I think slice creation being part of an inspection makes slices too hidden for users. It also unnecessarily requires a model to exist (without which you won't be able to create an inspection). In principle, a slice function should only require a dataset, so I think the previous location of the slice functions list (next to datasets) made more sense.

@Googleton
Copy link
Contributor Author

Googleton commented Jan 11, 2023

Hey Andrey, thanks for the feedback again.

I'll reply here to some of the points you mentionned:

  1. Modal titles will be modified to "Editing slice" and "Creating slice", and the name will not be displayed. Since the name field is directly underneath, it just doubles the same information so it can be left out.

6 I agree with this too, using it yesterday made me realise it's not a good workflow currently.
I believe we could have a "Run" or "Validate" button that checks the code against a dataset, but I think having a small pen icon next to the "X" in the slice dropdown would allow for quicker access to editing it.
This way, a user will work on their code, validate it. When they want to test to see how many rows return, they can save. But then they have to open the dropdown, open the menu, open the modal.. Having a shortcut to the modal for the current slice would be nicer IMO.

As for your general feedback, we discussed it with Jean Marie and since Slices are (for now) only relevant for inspections we decided to put them there. With a future version having a test catalogue, we imagined we could have a "function catalogue" that works in a similar way, with slicing functions, perturbation functions and other types as the product evolves.

There is an in-between then and now that we need to figure out if the dropdown solution isn't good enough for now, that's for sure

common/proto/ml-worker.proto Outdated Show resolved Hide resolved
backend/src/main/java/ai/giskard/service/SliceService.java Outdated Show resolved Hide resolved
backend/src/main/java/ai/giskard/service/SliceService.java Outdated Show resolved Hide resolved
result = mlWorkerService.filterDataset(client, dataset, slice.getCode());
Files.createDirectories(cachedSliceFile.getParent());
Files.createFile(cachedSliceFile);
try (DataOutputStream outputStream = new DataOutputStream(Files.newOutputStream(cachedSliceFile))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's not expensive to write cache as text file, not binary. It'll make our life easier when debugging, plus we already store prediction results as text, so this cache file won't be larger than that

@andreybavt
Copy link
Contributor

ack, we discussed it with Jean Marie and since Slices are (for now) only relevant for inspections we decided to put them there. With a future version having a test catalogue, we imagined we could have a "function catalogue" that works in a similar

I agree with both solutions you proposed for 4 and 6. As for the definitive place for the slices list, lets keep it inside inspection for now, but keep the "General comment" in mind for the moment when we start using slices in tests too.

I'd still enhance the slice editor to be able to try to run it on a dataset directly without leaving the editor

@Googleton
Copy link
Contributor Author

Small todo for myself, increase slice name length limit, too restrictive atm

# Conflicts:
#	frontend/src/api.ts
@andreybavt
Copy link
Contributor

@Googleton , could you have a last look at Sonar results? I don't think everything's coming from your changes though...

@andreybavt andreybavt removed the wip label Feb 2, 2023
Googleton and others added 3 commits February 2, 2023 14:06
# Conflicts:
#	frontend/src/views/main/profile/UserProfile.vue
@sonarcloud
Copy link

sonarcloud bot commented Feb 2, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

3.5% 3.5% Coverage
0.0% 0.0% Duplication

@andreybavt andreybavt merged commit d94fe2f into main Feb 2, 2023
@Hartorn Hartorn deleted the poc/397-slice-management branch September 13, 2023 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Slice management
3 participants