What are the requirements in how people report the SCI?

[jawache]

We've mentioned in a few places in the spec things that need to be reported as part of the calculation of SCI. I suspect we probably need to have a clear section regarding how you report it and also, right from the start we should specify an exact format so we can parse things with software. I would even go as far as to suggest using YAML or JSON, really prescribe this.

NOTE: We need to allow companies to not reveal data that they would consider proprietary or anything that they think gives them a competitive advantage.

So the SCI will report these bits of data.

Product Name *
Contact Name / Email *
Organisation
GUID - A unique identifier for this particular product/resource/service. *
SCI - the actual number. *
R - The baseline *
Date - The date this calculation was made *
Software boundary - A free text field to describe the software boundary this calculation refers to. *
Calculation methodology - Any details about your calculation methodology? If you did some MUSTs instead of SHOULDs perhaps this is the place to discuss it?
More info - A link to a doc with more info, maybe you published a longer piece on your company website or in a GH repo about exactly how you calculated it.

(*) Mandatory.

I'm wondering if GSF should maintain the database of all the SCI's people have reported, for now we can just ask people to submit a PR to a repository.

Thoughts?

[atg-abhishek]

Hellz yea @jawache! I really like this actually!

I would go so far as to say that the formatting (YAML) would be great because it allows for in-line commenting and is still machine-readable.

And maintaining the DB here would be great because that will also help:

1. people find how others in the community are exercising the SCI
2. show how pervasive it is
3. Ask questions in our discussions sections
4. suggest improvements and things that are missing that we should account for

And probably other things that I haven't thought of just yet, but I would say that this is something that should be discussed in our WG call too and see what other ideas we can get for this.

[jawache]

Yeah YAML is a good compromise of human and machine-readable. For the V1 release, I would say that just a GH repo is enough, one YAML file per app?

Maybe we'd also need to impose some naming conventions on the scope/filename to give some structure to the folders? I.e. like java package names?

com.microsoft.azure.ml ?

[atg-abhishek]

I like both of those @jawache ! We can maybe set that out as templates in the repo so that people can use that as a starting point (just like how we have templates for issues on GH?) cc @seanmcilroy29

[SaraEmilyBergman]

Using this as a thread to comment on the format (sorry for slight fork @Sealjay ).
I think yaml is fine, same with json honestly. I'd rather suggest (and by suggest I mean have it as a MUST) a few (like 1,2 or 3) formats that are easily accessible to people over than just saying "machine readable". It is easier to open up to more formats later. E.g. You MUST report the SCI in a yaml or json format.

I don't think publishing in a pdf and website is bad per se, as long as you also publish the required doc. What do you all think about that?

[Sealjay]

I've written using the unique GUID for the folder rather than package name for now - as some reports may not be associated with packages, or they can conflict between languages. Up for discussing further

[Sealjay]

As mentioned, I think we should define the hierarchical structure of the machine readable data format - but then allow consumers of the spec to select their day format of choice. Be that json, yaml, or at worst (IMHO) xml, etc.
So we shouldn't discourage PDF or website publishing, but we should state that the authoritative source of truth be machine readable, in my opinion.

[Sealjay]

So @SaraEmilyBergman and @jawache added good points around having the GSF maintain a central repo of data outputs.

There is previous for a central service with optional opt out- like the OSSF maintaining https://github.com/ossf/allstar allstar as a managed service for the common good, but also letting people roll their own.

[Sealjay]

As mentioned by @Henry-WattTime - feedback for this by and before Tuesday please - I'll then raise the pull request.

[jawache]

Before we talk about format I think we need to decide what exactly are we going to ask people to report, what does an SCI disclosure contain?

I've marked as mandatory these:

Product Name *
Contact Name / Email *
Organisation *
GUID - A unique identifier for this particular product/resource/service. *
SCI - the actual number. *
R - The baseline *
Date - The date this calculation was made *
Software boundary - A free text field to describe the software boundary this calculation refers to. *

Thoughts @SaraEmilyBergman @Sealjay?

[SaraEmilyBergman]

Nice! Couple of thoughts (these might be too detailed but I'm just going for it):

1. Why should contact be mandatory? I'm just thinking it can be hard for open source projects and the likes? Or can it be like a "legal person/entity" and not a "natural person"?
2. I think we should be more crisp on the SCI, e.g. that it is the CI and possible also the C. And if not C the reason for not providing C (this is listed as a must in the spec right now)
3. For R, include an optional comments field on why R was not in the set of predefined R's
4. As an optional field I'd like to include "software version", for apps I'm thinking this can be very relevant and easier to relate to over date.

I looked through the list and compared it to the spec and it looks like all the MUST and SHOULD's are covered :)

[Sealjay]

Here we go - I implemented your thoughts @SaraEmilyBergman @jawache in #123

[buchananwp]

Propose we use similar structure to the ML reproducibility checklist:
https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf

[Sealjay]

I started to do this - you can see I commented this out with <!-- - but I tried a table format instead. What do you think? PR in #123

[jawache]

I think we need to define three things here.

1. What do we want people to report.
2. The format of the report.
3. The method of reporting (repo that people publish to).

Although given the time we have left, I'm starting to think this is going to be a real challenge to get all this agreed on by Tuesday.

[SaraEmilyBergman]

Is it worth scheduling an extra meeting for discussing these? I'm also a bit concerned over if we have the time to agree.

[Sealjay]

I'm available tomorrow afternoon CEST/UK time, and Thursday early if that works? I've put this into #123 for review

[Sealjay]

@SaraEmilyBergman , I've proposed a time for review, but we can move it if people are busy :)

[Sealjay]

Hey folks - we've hit the Tuesday deadline, so I'm going to review the slack comments and crack on. I ended up flying Monday, so was a bit out of internet signal. I'm going to catch up on the slack, and put a pull request in for review today. That should give us two days to tweak and review with some focus @SaraEmilyBergman @jawache

[Sealjay]

This can be found in #123