Merge pull request JuliaLang#18588 from MichaelHatherly/mh/markdown-docs

Manual Conversion

CONTRIBUTING.md

@@ -100,59 +100,80 @@ Coveralls shows functionality that still needs "proof of concept" tests. These a
 
 *By contributing documentation to Julia, you are agreeing to release it under the [MIT License](https://github.com/JuliaLang/julia/tree/master/LICENSE.md).*
 
-Julia's documentation is stored in the `doc` directory, and like everything else can be modified using `git`.
+Julia's documentation source files are stored in the `doc/` directory and all docstrings are found in `base/`. Like everything else these can be modified using `git`. Documentation is built with [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl), which uses Markdown syntax. The HTML documentation can be built locally by running
 
-Julia's documentation is built with [Sphinx](http:https://sphinx-doc.org/contents.html), which supports (and Julia's docs rely heavily on) [ReST directives](http:https://docutils.sourceforge.net/docs/ref/rst/directives.html). To build the documentation locally, run
+```
+make docs
+```
 
- make -C doc html
+from Julia's root directory. This will rebuild the Julia system image, then install or update the package dependencies required to build the documentation, and finally build the HTML documentation and place the resulting files in `doc/_build/html/`.
 
-or
+> **Note**
+>
+> When making changes to any of Julia's documentation it is recommended that you run `make docs` to check the your changes are valid and do not produce any errors before opening a pull request.
 
- make -C doc latex
+Below are outlined the three most common types of documentation changes and the steps required to perform them. Please note that the following instructions do not cover the full range of features provided by Documenter.jl. Refer to [Documenter's documentation](https://juliadocs.github.io/Documenter.jl/stable/) if you encounter anything that is not covered by the sections below.
 
-from Julia's root directory. Sometimes errors only show up in one of them, so if you're preparing a pull request it is nice if you've checked both formats before you submit.
+#### Modifying files in `doc/src/`
 
-Existing docstrings now live primarily in `base/docs/helpdb.jl`.
-It is a goal over time to move the docstrings inline to their respective method definitions.
-If you want to edit the body of a method docstring, run the `doc/genstdlib.jl` script to regenerate the restructured text files **after** you have already rebuilt Julia.
-(From the top-level source directory, you can do this with `make julia-genstdlib`.)
-If you want to edit an existing docstring signature, you **first** have to change the signature in the `doc/stdlib` `..function` or `..data` definition (not the auto-generated content) and *then*
-edit the helpdb.jl or inline method docstrings. The existing signatures in the `doc/stdlib/*.rst` files are pattern matched to base docstrings and the new content overwrites the content in `doc/stdlib/`.
-The signature definitions **must** be in sync or else the pattern match will fail and documentation will be lost in the result.
-To add entirely new methods to the `stdlib` documentation, first add the signature in the appropriate `doc/stdlib/*.rst` file before writing the docstring, rebuilding Julia, and re-running `doc/genstdlib.jl`.
-Pattern matching requires that multiline method signatures' inter-line character alignment in `doc/stdlib/*.rst` match that in the corresponding docstring. In the following example,
+Most of the source text for the Julia Manual is located in `doc/src/`. To update or add new text to any one of the existing files the following steps should be followed:
 
-```julia
-"""
- foo(bar, baz,
- qux, quux)
+1. update the text in whichever `.md` files are applicable;
+2. run `make docs` from the root directory;
+3. check the output in `doc/_build/html/` to make sure the changes are correct;
+4. commit your changes and open a pull request.
 
-Foo `bar`, `baz`, `qux`, and `quux`.
-"""
-```
+> **Note**
+>
+> The contents of `doc/_build/` does **not** need to be committed when you make changes.
 
-will only match entries in `doc/stdlib/*.rst` beginning with
+To add a **new file** to `doc/src/` rather than updating a file replace step `1` above with
 
-```
-.. function:: foo(bar, baz,
- qux, quux)
-```
+1. add the file to the appropriate subdirectory in `doc/src/` and also add the file path to the `PAGES` vector in `doc/make.jl`.
 
-Note that the second line of the signature is indented by four spaces relative to `foo(bar, baz,`
-in the first line of the signature. This leading indent matches the indent used in the
-docstring exactly. If it did not match, such as in the following example,
+#### Modifying an existing docstring in `base/`
 
-```
-.. function:: foo(bar, baz,
- qux, quux)
-```
+All docstrings are written inline above the methods or types they are associated with and can be found by clicking on the `source` link that appears below each docstring in the HTML file. The steps needed to make a change to an existing docstring are listed below:
+
+1. find the docstring in `base/`;
+2. update the text in the docstring;
+3. run `make docs` from the root directory;
+4. check the output in `doc/_build/html/` to make sure the changes are correct;
+5. commit your changes and open a pull request.
+
+> **Note**
+>
+> Currently there are a large number of docstrings found in `base/docs/helpdb/Base.jl`. When any of these docstrings are modified please move them out of this file and place them above the most appropriate definition in one of the `base/` source files.
 
-where three spaces instead of four are used then running `genstdlib.jl` will print a warning
-and not update the docstring.
+#### Adding a new docstring to `base/`
 
-It is encouraged to write all new docstrings in Markdown markup. If you need to write a more complicated docstring that contains cross-references or citations it can be written in a restructured text codeblock.
-Many of the existing docstrings are currently restructured text codeblocks and these will be transitioned to Markdown over time. RST codeblocks are delineated with the triple-quote (\`\`\`rst \`\`\`) Makdown codeblock syntax.
-The content of the codeblock is spliced directly into the final restructured text document unmodified.
+The steps required to add a new docstring are listed below:
+
+1. find a suitable definition in `base/` that the docstring will be most applicable to;
+2. add a docstring above the definition;
+3. find a suitable `@docs` code block in one of the `doc/src/stdlib/` files where you would like the docstring to appear;
+4. add the name of the definition to the `@docs` code block. For example, with a docstring added to a function `bar`
+
+ ```julia
+ "..."
+ function bar(args...)
+ # ...
+ end
+ ```
+
+ you would add the name `bar` to a `@docs` block in `doc/src/stdlib/`
+
+  ```@docs
+  foo
+  bar # <-- Added this one.
+  baz
+  ```
+
+5. run `make docs` from the root directory;
+6. check the output in `doc/_build/html` to make sure the changes are correct;
+7. commit your changes and open a pull request.
+
+#### Doctests
 
 Examples written within docstrings can be used as testcases known as "doctests" by annotating code blocks with `jldoctest`.
 
@@ -161,7 +182,9 @@ Examples written within docstrings can be used as testcases known as "doctests"
  "DOCSTRING TEST"
  ```
 
-A doctest needs to match an interactive REPL including the `julia>` prompt. To run doctests you first run `make julia-genstdlib` then `make -C doc doctest`.
+A doctest needs to match an interactive REPL including the `julia>` prompt. To run doctests you need to run `make -C doc check` from the root directory.
+
+#### News-worthy changes
 
 For new functionality and other substantial changes, add a brief summary to `NEWS.md`. The news item should cross reference the pull request (PR) parenthetically, in the form `([#pr])`; after adding this, run `./julia doc/NEWS-update.jl` from the `julia` directory to update the cross-reference links. To add the PR reference number, first create the PR, then push an additional commit updating `NEWS.md` with the PR reference number.
 

Makefile

@@ -105,10 +105,7 @@ julia-debug julia-release : julia-% : julia-ui-% julia-sysimg-% julia-symlink ju
 
 debug release : % : julia-%
 
-julia-genstdlib: julia-sysimg-$(JULIA_BUILD_MODE)
- @$(call PRINT_JULIA, $(JULIA_EXECUTABLE) $(call cygpath_w, $(JULIAHOME)/doc/genstdlib.jl))
-
-docs: julia-genstdlib
+docs: julia-sysimg-$(JULIA_BUILD_MODE)
  @$(MAKE) $(QUIET_MAKE) -C $(BUILDROOT)/doc
 
 check-whitespace:
@@ -120,21 +117,11 @@ endif
 
 release-candidate: release testall
  @$(JULIA_EXECUTABLE) $(JULIAHOME)/contrib/add_license_to_files.jl #add license headers
- @$(JULIA_EXECUTABLE) $(JULIAHOME)/doc/genstdlib.jl
  @#Check documentation
  @$(JULIA_EXECUTABLE) $(JULIAHOME)/doc/NEWS-update.jl #Add missing cross-references to NEWS.md
- @$(MAKE) -C $(BUILDROOT)/doc unicode #Rebuild Unicode table if necessary
- @$(JULIA_EXECUTABLE) $(JULIAHOME)/doc/DocCheck.jl > $(BUILDROOT)/doc/UNDOCUMENTED.rst 2>&1 #Check for undocumented items
- @if [ -z "$(cat $(BUILDROOT)/doc/UNDOCUMENTED.rst)" ]; then \
- rm $(BUILDROOT)/doc/UNDOCUMENTED.rst; \
- else \
- echo "Undocumented functions found in doc/UNDOCUMENTED.rst; document them, then retry"; \
- exit 1; \
- fi
- @$(MAKE) -C $(BUILDROOT)/doc html SPHINXOPTS="-n" #Rebuild Julia HTML docs pedantically
- @$(MAKE) -C $(BUILDROOT)/doc latex SPHINXOPTS="-n" #Rebuild Julia PDF docs pedantically
- @$(MAKE) -C $(BUILDROOT)/doc doctest #Run Julia doctests
- @$(MAKE) -C $(BUILDROOT)/doc linkcheck #Check all links
+ @$(MAKE) -C $(BUILDROOT)/doc html
+ @$(MAKE) -C $(BUILDROOT)/doc pdf
+ @$(MAKE) -C $(BUILDROOT)/doc check
 
  @# Check to see if the above make invocations changed anything important
  @if [ -n "$$(git status --porcelain)" ]; then \
@@ -329,8 +316,9 @@ define stringreplace
  $(build_depsbindir)/stringreplace $$(strings -t x - $1 | grep '$2' | awk '{print $$1;}') '$3' 255 "$(call cygpath_w,$1)"
 endef
 
-install: $(build_depsbindir)/stringreplace $(BUILDROOT)/doc/_build/html
+install: $(build_depsbindir)/stringreplace
  @$(MAKE) $(QUIET_MAKE) all
+ @$(MAKE) $(QUIET_MAKE) docs
  @for subdir in $(bindir) $(libexecdir) $(datarootdir)/julia/site/$(VERSDIR) $(docdir) $(man1dir) $(includedir)/julia $(libdir) $(private_libdir) $(sysconfdir); do \
  mkdir -p $(DESTDIR)$$subdir; \
  done
@@ -379,7 +367,6 @@ endif
  # Copy documentation
  cp -R -L $(build_docdir)/* $(DESTDIR)$(docdir)/
  cp -R -L $(BUILDROOT)/doc/_build/html $(DESTDIR)$(docdir)/
- -rm $(DESTDIR)$(docdir)/html/.buildinfo
  # Remove perf suite
  -rm -rf $(DESTDIR)$(datarootdir)/julia/test/perf/
  # Remove various files which should not be installed

base/Enums.jl

@@ -29,7 +29,7 @@ end
 """
  @enum EnumName EnumValue1[=x] EnumValue2[=y]
 
-Create an [`Enum`](:obj:`Enum`) type with name `EnumName` and enum member values of
+Create an `Enum` type with name `EnumName` and enum member values of
 `EnumValue1` and `EnumValue2` with optional assigned values of `x` and `y`, respectively.
 `EnumName` can be used just like other types and enum member values as regular values, such as
 

base/abstractarray.jl

@@ -267,9 +267,9 @@ linearindexing(::LinearIndexing, ::LinearIndexing) = LinearSlow()
 Return `true` if the specified indices `I` are in bounds for the given
 array `A`. Subtypes of `AbstractArray` should specialize this method
 if they need to provide custom bounds checking behaviors; however, in
-many cases one can rely on `A`'s indices and [`checkindex`](:func:`checkindex`).
+many cases one can rely on `A`'s indices and [`checkindex`](@ref).
 
-See also [`checkindex`](:func:`checkindex`).
+See also [`checkindex`](@ref).
 """
 function checkbounds(::Type{Bool}, A::AbstractArray, I...)
  @_inline_meta
@@ -304,7 +304,7 @@ usually in a 1-for-1 fashion,
  checkbounds_indices(Bool, (IA1, IA...), (I1, I...)) = checkindex(Bool, IA1, I1) &
  checkbounds_indices(Bool, IA, I)
 
-Note that [`checkindex`](:func:`checkindex`) is being used to perform the actual
+Note that [`checkindex`](@ref) is being used to perform the actual
 bounds-check for a single dimension of the array.
 
 There are two important exceptions to the 1-1 rule: linear indexing and
@@ -714,8 +714,8 @@ A[iter] = 0
 ```
 
 If you supply more than one `AbstractArray` argument, `eachindex` will create an
-iterable object that is fast for all arguments (a [`UnitRange`](:obj:`UnitRange`)
-if all inputs have fast linear indexing, a [`CartesianRange`](:obj:`CartesianRange`)
+iterable object that is fast for all arguments (a `UnitRange`
+if all inputs have fast linear indexing, a `CartesianRange`
 otherwise).
 If the arrays have different sizes and/or dimensionalities, `eachindex` returns an
 iterable that spans the largest range along each dimension.
@@ -1512,7 +1512,7 @@ sub2ind(::Tuple{}, I::Integer...) = (@_inline_meta; _sub2ind((), 1, 1, I...))
 """
  sub2ind(dims, i, j, k...) -> index
 
-The inverse of [`ind2sub`](:func:`ind2sub`), returns the linear index corresponding to the provided subscripts.
+The inverse of [`ind2sub`](@ref), returns the linear index corresponding to the provided subscripts.
 
 ```jldoctest
 julia> sub2ind((5,6,7),1,2,3)
@@ -1789,7 +1789,7 @@ promote_eltype_op(op, A, B, C, D...) = (@_pure_meta; promote_eltype_op(op, eltyp
 """
  map!(function, collection)
 
-In-place version of [`map`](:func:`map`).
+In-place version of [`map`](@ref).
 """
 map!{F}(f::F, A::AbstractArray) = map!(f, A, A)
 function map!{F}(f::F, dest::AbstractArray, A::AbstractArray)
@@ -1848,7 +1848,7 @@ end
 """
  map!(function, destination, collection...)
 
-Like [`map`](:func:`map`), but stores the result in `destination` rather than a new
+Like [`map`](@ref), but stores the result in `destination` rather than a new
 collection. `destination` must be at least as large as the first collection.
 """
 map!{F}(f::F, dest::AbstractArray, As::AbstractArray...) = map_n!(f, dest, As)

base/abstractarraymath.jl

@@ -210,7 +210,7 @@ julia> circshift(b, (-1,0))
  1 5 9 13
 ```
 
-See also [`circshift!`](:func:`circshift!`).
+See also [`circshift!`](@ref).
 """
 function circshift(a::AbstractArray, shiftamt)
  circshift!(similar(a), a, map(Integer, (shiftamt...,)))

base/array.jl

@@ -261,7 +261,7 @@ julia> eye(A)
  0 0 1
 ```
 
-Note the difference from [`ones`](:func:`ones`).
+Note the difference from [`ones`](@ref).
 """
 eye{T}(x::AbstractMatrix{T}) = eye(T, size(x, 1), size(x, 2))
 

base/arraymath.jl

@@ -7,7 +7,7 @@
 
 Transform an array to its complex conjugate in-place.
 
-See also [`conj`](:func:`conj`).
+See also [`conj`](@ref).
 """
 function conj!{T<:Number}(A::AbstractArray{T})
  for i in eachindex(A)

base/associative.jl

@@ -115,7 +115,7 @@ end
  merge!(d::Associative, others::Associative...)
 
 Update collection with pairs from the other collections.
-See also [`merge`](:func:`merge`).
+See also [`merge`](@ref).
 """
 function merge!(d::Associative, others::Associative...)
  for other in others
@@ -138,7 +138,7 @@ end
 """
  keytype(type)
 
-Get the key type of an associative collection type. Behaves similarly to [`eltype`](:func:`eltype`).
+Get the key type of an associative collection type. Behaves similarly to [`eltype`](@ref).
 """
 keytype{K,V}(::Type{Associative{K,V}}) = K
 keytype(a::Associative) = keytype(typeof(a))
@@ -147,7 +147,7 @@ keytype{A<:Associative}(::Type{A}) = keytype(supertype(A))
 """
  valtype(type)
 
-Get the value type of an associative collection type. Behaves similarly to [`eltype`](:func:`eltype`).
+Get the value type of an associative collection type. Behaves similarly to [`eltype`](@ref).
 """
 valtype{K,V}(::Type{Associative{K,V}}) = V
 valtype{A<:Associative}(::Type{A}) = valtype(supertype(A))

base/asyncmap.jl

@@ -232,14 +232,14 @@ asyncmap(f, c...) = collect(AsyncGenerator(f, c...))
 """
  asyncmap!(f, c)
 
-In-place version of [`asyncmap()`](:func:`asyncmap`).
+In-place version of [`asyncmap()`](@ref).
 """
 asyncmap!(f, c) = (for x in AsyncCollector(f, c, c) end; c)
 
 
 """
  asyncmap!(f, results, c...)
 
-Like [`asyncmap()`](:func:`asyncmap`), but stores output in `results` rather returning a collection.
+Like [`asyncmap()`](@ref), but stores output in `results` rather returning a collection.
 """
 asyncmap!(f, r, c1, c...) = (for x in AsyncCollector(f, r, c1, c...) end; r)

base/base64.jl

@@ -19,7 +19,7 @@ export Base64EncodePipe, Base64DecodePipe, base64encode, base64decode
 
 Returns a new write-only I/O stream, which converts any bytes written to it into
 base64-encoded ASCII bytes written to `ostream`.
-Calling [`close`](:func:`close`) on the `Base64EncodePipe` stream
+Calling [`close`](@ref) on the `Base64EncodePipe` stream
 is necessary to complete the encoding (but does not close `ostream`).
 """
 type Base64EncodePipe <: IO
@@ -166,10 +166,10 @@ end
  base64encode(writefunc, args...)
  base64encode(args...)
 
-Given a [`write`](:func:`write`)-like function `writefunc`, which takes an I/O stream as its first argument,
+Given a [`write`](@ref)-like function `writefunc`, which takes an I/O stream as its first argument,
 `base64encode(writefunc, args...)` calls `writefunc` to write `args...` to a base64-encoded
 string, and returns the string. `base64encode(args...)` is equivalent to `base64encode(write, args...)`:
-it converts its arguments into bytes using the standard [`write`](:func:`write`) functions and returns the
+it converts its arguments into bytes using the standard [`write`](@ref) functions and returns the
 base64-encoded string.
 """
 function base64encode(f::Function, args...)

base/bitarray.jl

@@ -33,7 +33,7 @@ end
  BitArray{N}(dims::NTuple{N,Int})
 
 Construct an uninitialized `BitArray` with the given dimensions.
-Behaves identically to the [`Array`](:func:`Array`) constructor.
+Behaves identically to the [`Array`](@ref) constructor.
 """
 BitArray(dims::Integer...) = BitArray(map(Int,dims))
 BitArray{N}(dims::NTuple{N,Int}) = BitArray{N}(dims...)
@@ -1187,7 +1187,7 @@ end
 """
  flipbits!(B::BitArray{N}) -> BitArray{N}
 
-Performs a bitwise not operation on `B`. See [`~`](:ref:`~ operator <~>`).
+Performs a bitwise not operation on `B`. See [`~`](@ref).
 
 ```jldoctest
 julia> A = trues(2,2)
@@ -1632,7 +1632,7 @@ rol!(B::BitVector, i::Integer) = rol!(B, B, i)
 
 Performs a left rotation operation, returning a new `BitVector`.
 `i` controls how far to rotate the bits.
-See also [`rol!`](:func:`rol!`).
+See also [`rol!`](@ref).
 
 ```jldoctest
 julia> A = BitArray([true, true, false, false, true])
@@ -1701,7 +1701,7 @@ ror!(B::BitVector, i::Integer) = ror!(B, B, i)
 
 Performs a right rotation operation on `B`, returning a new `BitVector`.
 `i` controls how far to rotate the bits.
-See also [`ror!`](:func:`ror!`).
+See also [`ror!`](@ref).
 
 ```jldoctest
 julia> A = BitArray([true, true, false, false, true])