I really enjoyed reading this, and I don't know Clojure at all. This is a problem I've encountered several times while building design systems/component libraries. It's a very universal documentation problem, regardless of what platform it's targeting.

I have done the same with React - but it gets messy. Something nice like in the post would be great for that.

Too often I see very poor Storybook sites in the wild (a web based design system visual explorer, somewhat like the screenshot in the post) where no thought was given to this exact problem.

Instead of something meaningful like in the post, I see:

    () => <TextInputStorybook />

As the only avaliable code. The point of the code blocks is so people can copy paste. When I see that, I know I'm in for a rough ride because the code quality inevitably isn't any good either.

For reference, this is easily the best design system I know of: https://seek-oss.github.io/braid-design-system/components/Te... I use this as a reference when I work on my own.

In elixir you can run docs as (init) tests too. The doctest macro is generating the tests from the docs within the module. Maybe this might be interesting together with this article.

The fifth time this has been posted in just over a week... it must be a REALLY good article! :)

My bad... the sixth time (the fifth time was under a different title).

> What if our macro read the source file?

> Like, actually went to the file system, opened a file, and read its content? We already have the file name conveniently stored in file, and luckily Clojure keeps sources around.

> So this is what I ended up with:

> (defn slurp-source [file key]

Looks like a function to me (which is good).

it can't be a function because the source is only guaranteed to be available at compile time, rather than at runtime.

It's reading code from a file, which you can make arrangements for to be there at any time you want.

Files are not directly relevant to macros.

The code that a macro invocation has access to by design is the datum which comprises that invocation itself. That may have come from an expression in a file, and may be annotated in some way with the location info. Other than that, macros do not necessarily have access to their textual source code. No such thing necessarily exists because macros can be invoked by code that was generated (possibly by other macros or in other ways) and so never existed as characters in a file.

I don't have a clear picture of the architecture of the system that the article refers to, but the author presents, as his end result, a defn form, which defines a function and not a macro.

If a macro is needed to get that function to be called at compile time, that would be a minor hooking detail, and not an instance of a macro being used to perpetrate a complex code transformation. The function just needs a file name and a string.

(It looks as if the author's system may be interactive; the code files are there all the time and can be changed, and he wants the view of the files to refresh?)

It could be a helper-function which gets called by a macro expansion at compile-time. One would define the feature as a function which is available at compile-time and the macro calls it during expansion.

A macro is a perfectly normal function. It's simply called at compile time, with its contents as its argument.

no you're misunderstanding me.

I'm saying that the macro's purpose is to read the source file and produce something at compile time. It's not a "normal" (read: runtime) function, as this feature cannot be implemented using a function that only executes at runtime.

Sure, but the macro function is a normal function - it has inputs, outputs, and (preferably) should be pure. There's nothing special about the macro function itself, really.

You can see it explicitly in Common Lisp, where you can easily access and replace a macro-function with a custom one that you DEFUN yourself, like this.

    CL-USER> (defun my-quote-fn (form env)
               (declare (ignore env))
               (destructuring-bind (operator &rest body) form
                 (assert (eq operator 'my-quote))
                 (list* 'quote body)))
    MY-QUOTE-FN

It's a perfectly normal function, I can call it right away:

    CL-USER> (my-quote-fn '(my-quote (+ 2 2)) nil)
    '(+ 2 2)

I can also tell the compiler that this perfectly normal function is a macro function...

    CL-USER> (setf (macro-function 'my-quote) #'my-quote-fn)
    #<FUNCTION MY-QUOTE-FN>

...and the compiler will start using it to expand macros:

    CL-USER> (macroexpand-1 '(my-quote (+ 2 2)))
    '(+ 2 2)
    T

What is special about macros, and what you describe as the "feature that cannot be implemented", is the language's ability to call that function at compilation time and feed its output back into the compiler. This is the link that is missing in most programming languages.

> call that function at compilation time

that's why it's called a macro and not a function. You're using the word function in the mathematical sense, and not in the lisp sense.

The irrefutable fact is that the author presents the core logic of the file-reading solution as a defn, not a defmacro. It may be driven by a macro, but evidently that part is not noteworthy enough to be presented.

I mean, sorta, but Lisp macro writers learn early on that macros are also functions, it's just they operate on a list of symbols, and are called in a different phase.

So saying "it can't be a function" doesn't ring true, either.

The real trick here is dodging ASTs, which, after trying to use in so many parse-the-code projects, really aren’t needed all the time but are put pretty highly on the pedestal

An AST can carry the original source representation on each node as metadata — best of both worlds.

Thankfully this hiccup style seems less common in recent code. ie. where the index in the vector has a hidden meaning.. Now vectors are typically used for array of the same datatype. I think this makes more sense as basically if you see a vector, you can shove it in to map/filter/etc.

People now prefer using maps - even if it's a bit more verbose. Not sure why records fell out of fashion. You get the terseness of hiccup and the named keys of maps

Just wait for using full blown CL macros then.

Eh? Pretty sure they're more powerful. And readable, unlike the Scheme ones that look as elegant and well integrated as a caravan being pulled by a Porsche =).

Used in the way the author of the article presents, or generally the way you could use them in Clojure, Racket and Scheme macros are pretty similar. However at least in Racket you have a lot more control over certain parts of the way compilation affects the macro (look up phases for example). I assume Scheme has similar constructs, as does CL.

Note this is ignoring anything about the reader, for which Racket has substantially more powerful functionality around than Clojure.

TXR Lisp:

  $ cat slurp-source.tl
  (defun slurp-source (path key)
    (let* ((lines (file-get-lines path))
           (match (member-if (op contains key) lines))
           (tail (rest match))
           (indent (find-min-key tail : (op match-regex @1 #/\s*/)))
           (dedent (mapcar (op drop indent) tail)))
      `@{dedent "\n"}\n`))
  $ txr -i slurp-source.tl
  1> (put-string (slurp-source "slurp-source.tl" "let*"))
       (match (member-if (op contains key) lines))
       (tail (rest match))
       (indent (find-min-key tail : (op match-regex @1 #/\s*/)))
       (dedent (mapcar (op drop indent) tail)))
  `@{dedent "\n"}\n`))
  t

Wow!