x/pkgsite: generate static docs

[moraes]

It is fairly trivial to compile a given Go version and run the doc server locally to
search for an older API.

But it would be nice if golang.org provided access to docs per release. One simple
solution to this is to add an option to godoc to generate static docs on disk -- just a
tree of static files (search would be disabled of course). Then docs for old versions
could be made available somewhere in golang.org.

One additional benefit is that projects that want to ship docs could generate the static
files and provide the html files for download, or create pdfs, chms etc based on them.

PS: I tried searching for a similar issue; sorry if this is a duplicate.

[bradfitz]

Comment 1:

This is being worked on, kinda.
Now that godoc can run on App Engine, we'll be doing something like running different
subdomains of golang.org for old releases.
Andrew knows the details.

Owner changed to @adg.

Status changed to Accepted.

[adg]

Comment 2:

To generate static docs, just run this command:
godoc -http=localhost:8080 &
wget -r -np http:https://localhost:8080/pkg/
We're working on providing online docs for older versions of Go.

[moraes]

Comment 3:

I haven't thought about using wget. Nice. For those interested in this trick, here are
some notes.
First, edit robots.txt in the go root directory, and remove "Disallow: /". Otherwise
only the index will be downloaded because wget respects robots.txt.
Then start godoc pointing to the project path:
godoc -path="/path/to/project" -http=:8080
And finally this a more complete command to get a working static docs, including static
files and with proper links:
wget -r -np -N -E -p -k http:https://localhost:8080/pkg/
-r  : download recursive
-np : don't ascend to the parent directory
-N  : don't retrieve files unless newer than local
-E  : add extension .html to html files (if they don't have)
-p  : download all necessary files for each page (css, js, images)
-k  : convert links to relative

[rsc]

Comment 4:

Labels changed: added priority-later.

[adg]

Comment 5:

Owner changed to ---.

Status changed to HelpWanted.

[adg]

Comment 7:

Labels changed: added godoc.

[gopherbot]

Comment 8 by pedromorgan:

Yes please

[josharian]

Comment 9:

You can ask wget to ignore robots.txt by adding the flag `-e robots=off`.

[rsc]

Comment 10:

Labels changed: added go1.3maybe.

[rsc]

Comment 11:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 12:

Labels changed: added repo-tools.

[kegsay]

I would also like godoc to be able to generate static docs. The proposed workaround (hosting all the files and wgeting it) works but it crawls the entire contents of your GOPATH, which is less than ideal because that then bloats the static docs incredibly.

I still want things like the builtins to link correctly to docs, so some incantation of go list to pull out the dependencies of the project would be ideal. Unfortunately, I can't just write a bash script to do this because using godoc -html path/to/package just returns the raw HTML without any CSS, and no JS to handle section expansion.

EDIT: Managed to do this in the end with a fairly noddy script.

[gopherbot]

Change https://golang.org/cl/72890 mentions this issue: godoc/internal/render: add render package for text formatting

[dsnet]

As part of my work on #18342, I wrote simple static generator for godoc, that I realized is really well suited for resolving this issue. Is there any interest in including it's generated output in each release?

You can see statically generated docs at: https://static-hotlinks.digitalstatic.net/

[ghost]

As part of my work on #18342, I wrote simple static generator for godoc, that I realized is really well suited for resolving this issue. Is there any interest in including it's generated output in each release?

@dsnet Is this static generator available somewhere? I'd like to give it a try for hosting docs for a few packages.

[JohnStarich]

I've created a tool gopages based on the suggestions outlined here. Perhaps that may be useful for those generating internal enterprise docs for static sites.

A built-in solution would be excellent, nonetheless.

[edaniels]

Wow @JohnStarich, beat me by 6 hours. I've created a similar tool that only outputs docs for direct dependencies here :) https://github.com/viamrobotics/govanity

[dsnet]

For the record, that old Gerrit CL of mine has been partially mirrored over to https://github.com/dsnet/godoc. My original intention for that repo is to serve as a testing bed for various godoc proposals (e.g., #7873, #25444, #44447, #45533). If others want to use it to statically generate godoc pages, then they are free to do so.

[HRogge]

This is really annoying, especially because the anchors of links to source files are too the wrong line number... which is normally not recognized because godoc uses javascript to jump to the line too.

[3052]

you can generate with Doc2Go:

https://github.com/abhinav/doc2go

it also supports generating different versions of a module

[adonovan]

Is this feature (browsing old versions) essentially redundant with the "Added in go1.x" caption that pkg.go.dev displays next to each symbol? See:

• x/pkgsite: show the stdlib/module version when a symbol was added #42933
• x/pkgsite: show the version when a symbol was added to a package #37102

pkg.go.dev should also display such a caption for the package as a whole, but does not do so yet. See:

• x/pkgsite: "added in version" does not account for new packages #51013

[adonovan]

the title is "generate static docs", so I dont see what your comment has to do with anything. currently the go tool cannot generate static docs, or even serve them for that matter, to my knowledge

I realize that's the title, but if you read the first note of the issue it is merely proposed as "one simple solution"; the actual problem described is

it would be nice if golang.org provided access to docs per release.

[adonovan]

could you explain why you seem to be steering the conversation away from static docs and towards a server setup, across multiple issues? I get it, it seems you prefer that approach, but thats not the only approach.

x/tools/cmd/godoc is deprecated and unmaintained, and doesn't work with modules, and for those reasons I would like to delete it and close all open issues associated with it. However, there are a number of remaining obstacles to doing so, such as features or modes of operation that godoc supports but its proposed replacements do not. In addition, there are requests such as this one for new features to be added to godoc, which is of course not going to happen.

My goal is to make sure that the needs or problems motivating these issues are properly addressed by the newer tools. As usual, addressing the need or problem does not always mean implementing the feature exactly as proposed. In this case, the needs appear to be ways for readers to (a) understand which version a symbol appeared in the Go std API, which could be addressed by "Added in go1.x" captions as in pkg.go.dev; and (b) view specific versions of an arbitrary package's docs, which is already addressed by pkg.go.dev (for public packages) or a local pkgsite instance (for private ones).

Is there some other need addressed by static docs here that I have missed?

[adonovan]

people want to be able to generate HTML files for a given module or package.

But why? That's my question, and none of the lines you quote contains an answer.

"Generate static docs" is a solution, not a problem. So what is the problem? I identified two problems, both of which (I claim) are solved by other means. What others remain?

is your plan to just get the serve option working, then close the static requests as wont fix? if so it would be nice if you are at least up front about it, so people can pursue other options.

As I said in my previous note, cmd/godoc will not be getting new features, so this issue as narrowly defined by its first comment is already "won't fix". Far from "avoiding the issue", I'm trying to understand what problems remain so that we can solve them by other means. No-one is stopping anyone else from pursuing other options.

I would appreciate it if you take your rhetoric down a notch in future.

[edaniels]

Is there any plan then to have the go doc incantation become pkg site? For modern languages it seems imperative to have a documentation tool/command. It would be disappointing to see this go.

I had no clue this was a deprecated tool (I guess the source code says so?)

also @alandonovan, if no more work will be done on this, why not just close the issue and point people to pkgsite and its corresponding github issues since you mentioned it's missing some features. The back and forth seems silly for something that the Go team has committed to abandoning

[maxatome]

people want to be able to generate HTML files for a given module or package.

But why? That's my question, and none of the lines you quote contains an answer.

Just serving godoc in an already running web server accepting only static files? No need to run a host or a specific program just to dynamically serve files that almost never change, for example.

The first message mentioned the creation of PDF file. Why not? Why the user should launch a web server + a web crawler to do this conversion?

I am sure we can find plenty of other examples of static files uses...

++

[adonovan]

[@maxatome] Just serving godoc in an already running web server accepting only static files? No need to run a host or a specific program just to dynamically serve files that almost never change, for example. The first message mentioned the creation of PDF file. Why not? Why the user should launch a web server + a web crawler to do this conversion?

These all seem rather hypothetical: for example, PDFs from godoc or pkgsite are not very pleasant to read, and you would still need to run a browser to render the HTML. But at least you are addressing my question.

OK, I will leave this issue open, and we can interpret "godoc" as "pkgsite".

[@edaniels ] Is there any plan then to have the go doc incantation become pkg site?

Yes, there is a plan to make go doc -http fmt.Println start an instance of pkgsite (or rather, a version of it shorn of heavyweight dependencies), and open a browser to the documentation for the chosen package or symbol. Oddly there isn't an issue for the feature itself, but the preparatory work is #61399.

[Update: I filed this feature request:]

• proposal: cmd/go: doc -http should start a pkgsite instance and open a browser #68106

[edaniels]

Sounds great!

[3052]

people want to be able to generate HTML files for a given module or package.

But why? That's my question, and none of the lines you quote contains an answer.

judging by your comment here and others, your position seems to be "I don't think static generation is useful" is that really your position? do you really not understand why people would want to do this? personally I think the responsive design of https://pkg.go.dev is awful, and want to get away from it. I much prefer the older style, which https://godocs.io seems to be similar to. so my goal is to have a tool that can do

SOME-TOOL SOME-MODULE

and I get a folder of HTML files, that I can serve on GitHub or where I want. this is an extremely common scenario/request, so why you are having a hard time grasping it, or why it could be useful is beyond me. with the current approach, user would need to literally rent a VPS just to host their own docs, since its forcing a server approach instead of static generation.

[torbenschinke]

But why? That's my question, and none of the lines you quote contains an answer.

Our use case is to hand out a (static) documentation for our customers which use one or more of our private go modules in their products.

[seankhliao]

OK, I will leave this issue open, and we can interpret "godoc" as "pkgsite".

Let's just retitle this.

[adonovan]

judging by your comment here and others, your position seems to be "I don't think static generation is useful" is that really your position? do you really not understand why people would want to do this?

No, I don't have a position on the issue yet. I can imagine lots of ways it might be useful, but that's just my imagination (or @maxatome's, three comments back). I wanted to hear about actual concrete problems before deciding on a solution.

personally I think the responsive design of https://pkg.go.dev is awful, and want to get away from it. I much prefer the older style, which https://godocs.io seems to be similar to.

Well, that's really a separate question of what HTML is generated for a page. If pkgsite were to support static generation, perhaps it should use a print-friendly stylesheet.

[micahhausler]

I can imagine lots of ways it might be useful, but that's just my imagination. I wanted to hear about actual concrete problems before deciding on a solution.

At Amazon, I can safely say we have thousands of developers writing and reading Go. Our source code is stored in an internal Git hosting, and our home-grown build system has a build tool that leverages an internal Go module proxy. One of the language-agnostic outputs of our build system for a particular package (along with sources or binaries) is a documentation folder. Our build viewer can serve HTML files for a built release, which is a common pattern from other languages like Java (javadoc), Python (pydoc), Ruby (rdoc or 3rd party yard), Rust (rustdoc), C/C++ (3rd party doxygen). For Java, Python, Ruby, and Rust those are all first-party tools that our build tooling automatically calls to generates docs.

Today, in order to view Go documentation for a package, I have to check out source code on a machine (assuming I have source viewing/checkout permissions, which are separate from release consumption/viewing. For Go thats roughly equivalent, but not for all languages/builds), build it, spawn a process, possibly set up a port-forward (say, if I'm building on a remote Linux box), and then I can finally view the documentation. I can't tell you how many times I've had to repeat this process when I just want to browse another team's Go package.

The ability to invoke a maintained tool with roughly the arguments (to @3052's point)

TOOL -output-directory ./docs PACKAGE... 

would be in line with what other programming languages already support and would dramatically simplify the developer experience at Amazon when viewing docs for internal Go packages.

Because there is not one single filesystem that contains all versions of all packages' source code, and packages have differing permissions, running a fleet of pkgsite processes for all internal code, or spawning a per-user pkgsite process is not going to work for us.

If pkgsite were to support static generation, perhaps it should use a print-friendly stylesheet.

Our use case does not need a print-friendly version, as the vast majority of use cases would be online.

While I've described the Amazon internal use case here, but I think the general contours of this use case probably applicable elsewhere. I can also speak to previous places I've worked where I fronted statically generated Python and JavaScript HTML docs with an authenticated webserver, and Go was the odd one out.

[jimmyfrasche]

(A print friendly stylesheet should be a separate issue. That would be equally as useful for someone wanting to print from/save a pdf of a page on go.dev and nothing about being dynamic or static factors into adding print styles.)

I agree that static documentation is important.

I worry about creating issues where things have to be done twice for the static vs. dynamic presentation or skew when someone adds a feature but no one checks to make sure how it works in the static version or scope creep because someone has a use case where they need to have all packages in a module go on one page or have each type get its own page and so on.

Maybe it would suffice to have a -json flag that output all the raw data. (And when I say raw I mean even the doc comments should be the go/doc/comment AST instead of html fragments).

Then there just needs to be a frontend process that takes that and applies the data to templates, but, importantly, there could be many different such frontends depending on taste and need.

Essentially, instead of

$ pkgsite -static -output dir pattern

you'd have

$ pkgsite -json pattern | static-pkgsite dir

but you could have many such static-pkgsite commands.

(pkgsite could also serve this same json dynamically for headless operation, but that would also be another issue).