observable convert

[mbostock]

A minimal implementation of converting an Observable notebook to Markdown. For example:

observable convert https://observablehq.com/@observablehq/plot-scatterplot/2

This outputs a plot-scatterplot,2.md containing:

<div style="color: grey; font: 13px/25.5px var(--sans-serif); text-transform: uppercase;"><h1 style="display: none;">Plot: Scatterplot</h1><a href="/plot">Observable Plot</a> › <a href="/@observablehq/plot-gallery">Gallery</a></div>

# Scatterplot

Given a dataset with two quantitative dimensions, the [dot](https://observablehq.com/plot/marks/dot) mark creates a scatterplot.

```js echo
Plot.dot(penguins, {x: "culmen_length_mm", y: "culmen_depth_mm"}).plot()
```

You can also use shorthand:

observable convert @observablehq/plot-scatterplot/2

This is mostly designed under the worse is better philosophy. I didn’t try to implement anything fancy, like parsing Observable JavaScript and rewriting it to vanilla JavaScript. The expectation is that it’ll probably break, but the code will be right there so it should be “easy” to rewrite the code manually. And in many cases, I don’t think we’ll be able to rewrite the code automatically. Maybe we could have some opt-in (or opt-out) smart rewrites in the future, but this feels like a useful thing to include to get things going!

I’d also like to allow it to use your credentials to convert private notebooks if you observable login.

[mythmon]

I’d also like to allow it to use your credentials to convert private notebooks if you observable login.

FYI we currently don't support requesting private notebooks via API keys as individuals, which would be a blocker for this. It is something that I've discussed with the gardening team as wanting to fix soon though. I think we understand the path of how to get there though, especially post workspace migration.

[mbostock]

FYI we currently don't support requesting private notebooks via API keys as individuals, which would be a blocker for this.

Yep, I didn’t expect it to work yet — just noting it for future work!

[Fil]

As an aside, it looks like some of the lines that are output by the yarn scripts are sent to stdout (😭), so my new page starts and ends with cruft:

> yarn observable convert https://observablehq.com/@observablehq/plot-scatterplot/2 > docs/penguins.md

[Fil]

I will probably have to use this script in large quantities (say, to port over certain galleries…), and I wonder what I should do when I see a repeating pattern. Is it better to try and make this script smarter, or to write an independent script to use as a post-processor (and that matches what I care about without trying to be generic)?

[mootari]

Is it better to try and make this script smarter

Fwiw, I'd be more than happy to help with that.

[mbostock]

I can’t do anything about yarn but we could offer an --output option.

As I said in the description, I think we could make this smarter in the future, but that anything smart should be opt-in or opt-out since it is unlikely to be correct in all cases. And so I think the best place to start is to simply preserve the user’s code with no edits. And of course you could pipe the output to a helper script for post-processing too.

I’d also like this to be available for bulk conversion, too:

• Converting all notebooks in a collection
• Converting all notebooks in a workspace
• Converting a notebook and its transitive imports
• Downloading file attachments too

Perhaps this suggests that --output should be a required argument (or it could default to the slug or id) because we’ll always want the ability to output multiple files.

[mbostock]

Okay, this is ready — it just needs tests!

[Fil]

I was surprised that the replace confirm would stop the whole operation; instead, I thought (given the wording of the question) that answering No would skip that file and continue to the next file.

If we don't change that, the message in that case ("Cancelled convert") is slightly wrong: the download is in a partial state (what's been downloaded already is already here). It would be more accurate to say "Stopped convert". If we wanted to cancel, we should use a temp download directory and leave a clean slate if there is any error.

Also, it seems strange to download a file then verify if we already have a copy.

Happy to work on these changes if you think they make sense. But I don't want to block because this is still good enough to include today.

[mbostock]

Added a caveat…

[trebor]

approving to unblock. a few thoughts:

• does it make sense to drop the md file directly into doc?
• the comma in the name feels rough

that said perhaps both of these are a good way to indicate that it's not always just gonna work and needs some inspection.

[mbostock]

What does this mean? “drop the md file directly into doc?”

[trebor]

What does this mean? “drop the md file directly into doc?”

ah yes sorry, assuming that convert might be typically be run from the project root directory, the file could be placed into the docs directory.

[mbostock]

Ah, yeah, maybe. Not sure yet how people will use it. 🙏

[Fil]

the comma in the name feels rough

this should only happen when the original notebook's slug contains a /:

> yarn observable convert @d3/bar-chart/2
◇  Downloaded bar-chart,2.md in 296ms
◇  Downloaded alphabet.csv in 596ms

because we can't use a slash in a file name (the choice of a comma in that case is a little arbitrary, but it lowers the risk of conflicting with another notebook (e.g. bar-chart-2 if we used a dash).