Support documentation on functors

[edolstra]

For example, given a functor like

id = {
  __args = [ "x" ];
  __doc = ''The identity function. It takes a single argument *x* and returns it.'';
  __functor = _: x: x;
};

the repl can now print documentation about it:

nix-repl> :doc lib.id
Synopsis: lib.id x

    The identity function. It takes a single argument x and returns it.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-21/16032/1

[andir]

Alternative idea:

nix-repl> f = builtins.addDoc "some doc string" (a: b: a + b)
nix-repl> x = builtins.addDoc "another doc string" { a = "1"; }
nix-repl> :d f
    some doc string
nix-repl> :d x
    another doc string

Without having spent too much time thinking about this the implementation could add a references to a String to the Value struct. This means we can document all and every single expression in the language, something that was always missing to generate docs from a set of functions (without resorting to some kind of hacks).

I don't really like the current ergonomics around the __functor implementation. builtins.typeOf functor always returns set. It is easy to get the functor syntax wrong and end up in an inifite loop (see #5515).

[sternenseemann]

@andir's proposal would also allow for arbitrary values to be documented which could be quite nice.

[edolstra]

@andir That was my original thought, but the problem is that Values don't have enough space to add annotations like docstrings. Functors on the other hand already are attrsets.

[andir]

@andir That was my original thought, but the problem is that Values don't have enough space to add annotations like docstrings. Functors on the other hand already are attrsets.

You mean we can't add a pointer to the Value struct that defaults to nullptr? If so, why is that?

[edolstra]

Right, it would make every Value 8 bytes larger.

[andir]

I've actually gone ahead and made that change (with another pointer value in each Value instance). It is very rough around the edges but in principle it works:
8e25232

./benchmark/bin/benchmark ./with-docs/bin/nix-instantiate ~/dev/nixos/nixpkgs/nixos/tests/simple.nix
maximum RSS:        median = 561232.0000  mean = 561240.4444  stddev =     67.8953  min = 561180.0000  max = 561412.0000
soft page faults:   median = 138058.0000  mean = 138058.3333  stddev =      2.4495  min = 138055.0000  max = 138063.0000
system CPU time:    median =      0.2016  mean =      0.2019  stddev =      0.0077  min =      0.1907  max =      0.2136
user CPU time:      median =      1.6646  mean =      1.6743  stddev =      0.0275  min =      1.6316  max =      1.7097
elapsed time:       median =      2.0381  mean =      2.0230  stddev =      0.0382  min =      1.9672  max =      2.0772

./benchmark/bin/benchmark ./without-docs/bin/nix-instantiate ~/dev/nixos/nixpkgs/nixos/tests/simple.nix
maximum RSS:        median = 559624.0000  mean = 559636.8889  stddev =     63.0485  min = 559572.0000  max = 559744.0000
soft page faults:   median = 137672.0000  mean = 137674.1111  stddev =      5.3955  min = 137670.0000  max = 137687.0000
system CPU time:    median =      0.2137  mean =      0.2162  stddev =      0.0142  min =      0.1917  max =      0.2353
user CPU time:      median =      1.6506  mean =      1.6574  stddev =      0.0234  min =      1.6267  max =      1.6927
elapsed time:       median =      2.0295  mean =      2.0180  stddev =      0.0306  min =      1.9757  max =      2.0683

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-generate-documentation-from-arbitrary-nix-code/22292/7

[mightyiam]

@edolstra can we meet to discuss options regarding this, please?

[roberth]

@edolstra My old PR #1652 achieves this without attrset overhead and with less syntax. Docs should be as easy as possible to write.

:doc wasn't a thing back then, but I'll be happy to update the code to work with the new command instead, if you agree with the source position based approach.

[mightyiam]

Hey, @roberth, we are a team of three, working in mob programming format on core documentation issues in the Nix ecosystem. We'd love to coordinate efforts with you.

[mightyiam]

We are @a-kenji, @101v and I.

[roberth]

Hey, @roberth, we are a team of three, working in mob programming format on core documentation issues in the Nix ecosystem. We'd love to coordinate efforts with you.

Hi @mightyiam and the mob,
You might have read in the other PR that Eelco and I disagree about the documentation format; comments vs __functor.
I don't plan to work on this controversial issue, so feel free to take over.

@edolstra How about having both? Documentation seems important enough to warrant the relative complexity of having both the added power of __functor and the ease of use of doc comments.

[mightyiam]

Thank you, @roberth . We intend to treat #228 as the tracking issue for this feature in general.

We are working on a draft design that is solely from a user's perspective, user being a documentation author and consumer.
Will post an update there as soon as we have something to show.