extract files from json?

[Fil]

to make it easier to create multiple files from a single data loader, we could add json as an "extractible" format.

For example, a call to _file/foo/bar.csv would return the csv from foo.json:

{
  "bar.csv": "a,b\n1,2"
}

A more "clever" approach would be to extract foo/bar.json or foo/bar.csv from foo.json like so:

{
  "bar": [{"a": 1, "b": 2}]
}

this would require the extractor to know how to re-encode the object. Maybe worth it for a few select formats (csv, json)?

[mbostock]

I’m not a fan of this idea, but I appreciate you sharing it nonetheless. I would prefer to stick to existing, well-known multi-file formats and not try to invent a new one. Another format to consider is multipart.

Maybe instead of a new format, though, there could be some way to indicate that a data loader writes to the file system (in a specific directory) rather than writing to stdout? There are other cases where it’s easier to write to the file system than to write to stdout, even if you’re just generating one file.

[Fil]

In defense of JSON, it is an existing and well-known file format for key/value data. It would be very nice to have the possibility to output several "variables" from a basic JS data loader by writing console.log(JSON.stringify({a, b, c, d})), and the possibility to fetch FileAttachment("loader/a.json").

(However I reckon that the value itself must be JSON too, which limits the scope of that format.)

For multipart see #1246. (It's possible that this would be useful, but overall it doesn't seem that it will reduce the amount of work we ask of the user, which is what I am trying to do with this JSON idea.)

A way to use the filesystem would certainly be very helpful for many use cases (though, again, not the one I have in mind here); I've split it into #1247.

Closing this issue because I don't plan to work on it if there's no consensus. But we should continue the discussion about how to make data loaders simpler (for faster iteration/exploration), since every time I create a new thing I don't like that I have to look up in our examples how to create a zip file.