Base documentation

[timholy]

Just a reminder issue: we need fixes or guidance on the proper way to proceed for the documentation changes. Those of us who have tried haven't gotten very far yet.

If there's anything needed to "build the docs" (e.g., doc/genstdlib.jl), instructions should be put somewhere appropriate.

[ViralBShah]

Should this have the 0.4 milestone label?

[timholy]

Yes, since it's the only thing I know of that's left as a blocker.

[jakebolewski]

@one-more-minute how do we add documentation now? I asked @jiahao and he does not know either.

[MichaelHatherly]

how do we add documentation now?

#12435 still needs to be completed for markdown docs to be converted and extracted properly, but at the moment adding an entry in helpdb.jl something like:

doc"""
```rst
::
           foobar(x)

Docs for ``foobar``.
```
"""
foobar

and a corresponding line somewhere in the stdlib files:

.. function:: foobar(x)

followed by make and ./julia ./doc/genstdlib.jl should splice the docstring into place.

To write this inline rather than in helpdb.jl it should work just fine moving what's in helpdb.jl to above functions, macros, etc. but currently docs/bootstrap.jl isn't defined early enough to allow anything prior to arrays.jl to be documented inline. #12511 removes most of that restriction by moving docs bootstrapping to earlier and could use a couple more eyes to check if I'm doing anything bad there.

@one-more-minute please do correct me if I've got anything mixed up.

[tkelman]

Thanks for all your help here @MichaelHatherly.

What would converting the docs to markdown buy us right now, if we don't have replacements for the website, doctests, linkchecks, cross-references, and any other things we currently use Sphinx for? If some of these things can be built in parallel on branches or in separate repos (using Lexicon anywhere that makes sense?) then it should, but this doesn't seem mature enough to use for 0.4. So #12435 doesn't seem release-critical to me, but correct me if there's some reason I'm missing that it should be.

We need to make sure that all recent doc additions are consistently ending up in both places (check https://github.com/JuliaLang/julia/commits/master/base/docs/helpdb.jl vs https://github.com/JuliaLang/julia/commits/master/doc for consistency), and that the few conflicts and RST losses from #11943 get cleaned back up and restored.

[MichaelHatherly]

What would converting the docs to markdown buy us right now, if we don't have replacements for the website, doctests, linkchecks, cross-references, and any other things we currently use Sphinx for?

Other than having nicer looking docs in the REPL, not a lot at the moment until we've got cross-references etc. supported.

If some of these things can be built in parallel on branches or in separate repos

Maybe the manual / stdlib docs should live in a separate repo (JuliaManual.jl) and doc/genstdlib.jl could just be run there once daily / on every merge so that we don't have to run that script manually. People who make changes / additions would then just need to add the docstring and the marker for the final location in the manual. (Just a random thought though, not sure how suitable that would be.)

doesn't seem release-critical to me

Changes and tweaks to the docs during an RC phase would be alright to do, wouldn't they? They wouldn't really be changing/adding new features to the language.

[tkelman]

Changes and tweaks to the docs during an RC phase would be alright to do, wouldn't they? They wouldn't really be changing/adding new features to the language.

Yes. But would like to avoid having to carefully review massive thousands-of-lines diffs if we don't need to do so right this moment.

[MichaelHatherly]

But would like to avoid having to carefully review massive thousands-of-lines diffs

Yeah, definitely. I think we could probably make those kind of doc changes piece by piece, even manually if needed, which should hopefully avoid that.

[MikeInnes]

@tkelman "Moving the docs to markdown" is really two seperate issues. One is simply changing the format of the docstrings to markdown, which #12435 implements. This is actually a very simple and non-disruptive change, albeit touching a lot of docstrings, since we can continue outputting RST and keep all the sphinx stuff around. We also don't need to implement a bunch of supporting features like cross-references, since we can just convert the basic ones and attack the rest later on.

The second is moving the manual itself to markdown. That's a much bigger deal, since all the docstrings must be in the right format (which means implementing crossrefs etc.) and on top of that we need to sort out the new website etc. I'm completely in agreement that that's not a reasonable goal for a nearby release.

[timholy]

Questions:

• Do we have a git management issue? IIUC, in doc/stdlib, foo.rst should just contain "stubs" for the documentation, and foo.rst should be a committed file. When you run genstdlib.jl, it will re-write foo.rst, populating it with the entries from base/docs/helpdb.jl. The .rst files are used to generate the HTML documentation.
  The problem I see with this workflow is that, post-genstdlib.jl, git will recognize this as an altered file, one that sure looks like it should be committed. To me it seems we should have foo_stubs.rst be the committed file, and make foo.rst the resulting processed file from which the HTML documentation is generated.
• Can you flesh out how more complex examples work? What if I have separate documentation for bar(x::Real) and bar(x::AbstractString)? For print and Profile.print? I'm guessing the latter is straightforward, but I note that the former may not be working: check out the current profile.rst and compare it to the version in the release-0.3 branch: https://raw.githubusercontent.com/JuliaLang/julia/release-0.3/doc/stdlib/profile.rst.
  EDIT: the output of ?Profile.print is fine, but the on-the-web version isn't: http:https://docs.julialang.org/en/latest/stdlib/profile/.

[timholy]

It would also be good to have the generated .rst files be emblazoned with

.. This file is auto-generated. DO NOT EDIT.

[jakebolewski]

Ok thanks, this seems a bit convoluted at the moment. I think there needs to be a toplevel issue enumerating the steps / path forward for 0.5. It seems like there is a lot of work still left to do to make the entire transition of the manual to Markdown. What are the closest competitors to Sphinx for writing long form documentation?

[timholy]

I'd change my suggestion, and have foo.rst be the git-committed version, and foo_gen.rst be the version produced by genstdlib.jl. That way we can add *_gen.rst to .gitignore.

[ufechner7]

Could rippledoc be an alternative to Sphinx?
http:https://www.unexpected-vortices.com/sw/rippledoc/index.html

[tkelman]

@one-more-minute I agree the format conversion is theoretically straightforward, but it's a large change that needs to be vetted to make sure it's translating everything correctly. It's a nice-to-have if it makes the display nicer but I don't think we need to block the release on it. The more substantial "Replace Sphinx entirely" goal has a lot more missing pieces that will be difficult to implement. Good goal that can hopefully be built incrementally during 0.5 and onward.

There is a bit of a git issue here, but I think the simplest plan for right this second is to commit your changes both to helpdb, and post-genstdlib.jl RST docs. The RST docs are "partially autogenerated" and splitting them up into separate files seems like more trouble than it's worth.

[StefanKarpinski]

Let's move discussion of transitioning the manual to Markdown over to #12489 erm #12573.

[IainNZ]

@StefanKarpinski , this is #12489...?

[StefanKarpinski]

Sorry, wrong link.

[MikeInnes]

@timholy Apologies for not being quicker to respond, but hopefully #12703 answers your questions. I know you had at least one doc-related PR open, let me know if I can help out with that in any way.

[timholy]

No worries, now it's up to me to catch up with what y'all have been doing! I'll try to figure it out and ping you if I need help. Thanks for your herculean efforts here.

I'm assuming this can be deservedly closed.