A Common Lisp library for constructing interactive, composable HTML forms, entirely on the server-side; no JavaScript is necessary.
- Interactive: The way a “conformlet” is rendered may change depending on the user’s actions. For example, a resizeable list with “add” and “remove” buttons is possible. Each element of the list can contain more form fields!
- Composable: Parts of a form can be re-used. For example, there might be a datepicker form element that you will want to include in multiple places throughout a form. “Higher-order” parts of a form are also possible, for example, a resizeable list of arbitrary sub-conformlets.
The most well-known solution to server-side form composability, was presented in a 2008 paper, The Essence of Form Abstraction. Formlets, the name of the solution presented in that paper, have since been implemented in many programming languages, including Racket and Common Lisp. Formlets achieve composability, but not interactivity. It’s impossible to define a “list” formlet that contains a varying number of child formlets based on which buttons the user clicks.
Conform still uses a formlet-like design, so we credit the paper’s authors by naming each composable form component a “conformlet”.
Conform is optimized for making forms that represent a persistent server-side object, rather than forms you prepare and then submit once. For instance, Conform would be a great choice for:
- An “edit profile” page.
- A page editor for a CMS (Content Management System).
- Editing a hotel booking.
To use Conform for transient (non-persistent) data, you have to either give up the interactivity
features (which rely on knowing what the data “used to be” before each form submission) or come up
with a creative way to persist the data (eg, cryptographically signing it and storing it in a hidden
<input>).
Conform is not tightly coupled to any web server. That being said, all built-in conformlets, and all the examples here, are tied to a certain HTML generator: html->string. You may find it helpful to read the html->string documentation before reading the examples. To use any other HTML generator you would have to reimplement all built-in conformlets, but wouldn’t have to mess with the Conform framework itself.
Conform will be available on Quicklisp soon, hopefully. Until then, just put the project
directory into your local Quicklisp directory (eg, ~/quicklisp/local-projects/), then run
(ql:quickload :conform)
Load the conform/examples system, then run (conform/examples:start 4321) to start a
Hunchentoot server on https://localhost:4321 with a few Conform examples.
At the time of writing, I run an example server at https://markasoftware.com:4321.
Conform includes two macros, place and with-places, that simplify passing a mutable object as
an argument to a function. (place (car \*my-list\*)) will expand into something to the effect
of (cons (lambda () (car \*my-list\*)) (lambda (new-val) (rplaca \*my-list\* new-val))), ie, a
cons cell containing a reader and writer for the given place, respectively. (with-places) can
then be used to take a result of (place) and bind it to what seems a normal, setf-able variable:
(let ((foo (place (car *my-list*))))
(with-places (foo)
(incf foo)))
;; equivalent to (incf (car *my-list*))This is used in Conform to simplify passing part of a data structure to a subconformlet. For
example, in an edit profile conformlet, you might want to pass the user’s first name to the
string-field conformlet, allowing the string-field to both read and update the first name.
This could be achieved by passing (place (slot-value *profile* 'first-name)) to string-field,
for example.
Before we can talk about the juicy interactive conformlets, we have to discuss how to make a simple conformlet. Here’s one:
(defun full-name (val)
"val should be a Place pointing to a cons (\"first-name\" . \"last-name\")"
(with-places val
(conformlet ()
`(div (class "full-name-field")
,(string-field (place (car val))
"First Name")
,(string-field (place (cdr val))
"Last Name")))))First, note that our conformlet is just a function – there’s no defconformlet or similar.
Second, val is passed using (place), as described earlier, so we can both read and write it.
The string-field conformlets modify the place passed in as a value every time the form is
submitted.
Here’s another conformlet: A number with buttons to increase or decrease it by a certain amount.
(defun onerous-number (val delta)
(conformlet (:places val)
`(div (class "onerous-number")
"The number currently is: " val
,(button (lambda () (incf val delta))
"Increment")
,(button (lambda () (decf val delta))
"Decrement"))))In the full-name example, we explicitly wrote with-places. In onerous-number, though, we
provide the :places argument to conformlet, which is shorthand for the same thing.
The first argument to each button is an “Onclick Event Handler”, which is called when the button
is pressed. Remember that Conform does not use JavaScript, so all buttons are of type="submit"
and therefore trigger a form submission and page reload.
Here’s a “higher-order” conformlet that duplicates a field:
(defun duplicator (val subconformlet)
"Call subconformlet on both cells of val, which should be a place pointing to a cons."
(conformlet (:places val)
`(div (class "duplicator")
,(funcall subconformlet (place (car val)))
,(funcall subconformlet (place (cdr val))))))The subconformlet should be a function that takes, as a single argument, a (place) output. For
example, to use onerous-number with a delta of 4 as the subconformlet:
(duplicator (place *my-cons-cell*) (rcurry #'onerous-number 4))curry and rcurry are very common when handling higher-order conformlets.
Note that duplicator could actually be implemented without passing in val as a Place, because
it never setf’s directly on val.
Conform has separate “render” and “event handling” phases. The “render” phase only generates HTML,
with no side effects. The “event handling” phase is when the form responds to the POST data. For
example, in the onerous-number example, the lambdas on the button conformlets are called
during the event handling phase.
We can introduce a (custom-event) that will be processed during the event handling phase, after
all other event handlers. This is useful for post-processing data from sub-conformlets.
The following legal-agreements conformlet demonstrates using a custom event to implement a
custom event handler, onagree, that is called only when all the boxes are checked.
(defun legal-agreements (onagree)
(conformlet ()
(let ((agreed-tos)
(agreed-privacy-policy)
(agreed-forfeit-assets))
;; this will be run after all other event handlers have been run.
(custom-event
(when (and agreed-tos agreed-privacy-policy agreed-forfeit-assets)
(funcall onagree)))
`(div (class "legal-agreements")
,(checkbox-field (place agreed-tos)
"Do you agree to the terms of service?")
,(checkbox-field (place agreed-privacy-policy)
"Do you agree to the privacy policy?")
,(checkbox-field (place agreed-forfeit-assets)
"Do you agree to forfeit all of your assets to EvilCorp LLC?")))))Pretty cool! This also demonstrates a common pattern: Passing local variables to sub-conformlets,
then using a custom-event to process those local variables and conditionally pass something on
to the parent conformlet (in this case, calling onagree). Here’s another instance of the
pattern:
(defun uppercase-string (val)
(conformlet (:places val)
(let ((temp val))
(custom-event
(setf val (string-upcase temp)))
(string-field (place temp) "Enter string:"))))I mentioned that custom events are processed after all other events under the current conformlet. It’s possible to get finer control over the order in which the events of sub-conformlets are processed.
TODO: order example
Here comes the fun part!
(defun simple-list (val subconformlet default)
(conformlet (:places val)
`(div (class "simple-list")
,(loop for i from 0 below (length val)
collect (let ((k i))
(funcall subconformlet (place (nth k val)))))
,(button (lambda ()
(appendf val (list default)))
"Add New"))))Ain’t that easy? Note the important binding of k to i, which is necessary to ensure that the
current value is captured during each iteration, else all the subconformlets would refer to the
same i, which would, at the end of iteration, equal (length val). Here’s a more advanced
list:
(defun swapcar (cons1 cons2)
(declare (cons cons1 cons2))
(let ((temp (car cons1)))
(rplaca cons1 (car cons2))
(rplaca cons2 temp)))
(defun advanced-list (val subconformlet make-default)
(declare (function subconformlet make-default))
(conformlet (:places val)
`(div (class "form-list")
,(loop for i from 0 below (length val)
collect (let ((k i)) ; capture the value
`(div ()
,(funcall subconformlet (place (nth k val)))
(div ()
,(button (lambda () (metatilities:delete-item-at val k)) "Delete")
,(when (> k 0)
(button (lambda ()
(swapcar (nthcdr (1- k) val)
(nthcdr k val)))
"Move up"))
,(when (< k (1- (length val)))
(button (lambda ()
(swapcar (nthcdr k val)
(nthcdr (1+ k) val)))
"Move down"))))))
(div (class "pure-controls")
,(button (lambda ()
(appendf val (list (funcall make-default))))
"Add new")
,(button (lambda ()
(setf val (shuffle val)))
"Shuffle")
,(button (lambda ()
(setf val (nthcdr (ceiling (length val) 2) (shuffle val))))
"Thanos")))))Once you’ve defined all the conformlets you need, render the form:
(html->string
`(form (method "POST" action "")
,(render-form "form_prefix"
#'hunchentoot:post-parameter
(eq :post (hunchentoot:request-method*))
(some-conformlet))
(button (type "submit") "Submit form")))The first argument to render-form is a prefix that will be added to the name attributes of all
fields. The next argument is a function that returns the value of a post parameter, given the
string name of the post parameter. The third argument is a boolean indicating whether the current
request contains a form submission; this will be nil on initial page load. The rest of the
arguments are evaluated in an implicit conformlet statement.
It’s common to, after a POST request, redirect the user back to the same page to avoid breaking the browser’s back button. Implementation is trivial with Conform and Hunchentoot:
(render-form "prefix" #'hunchentoot:post-parameter (eq :post (hunchentoot:request-method*))
(custom-event
(hunchentoot:redirect (hunchentoot:request-uri*)))
(rest-of-form-here))In case you’re unfamiliar, hunchentoot:redirect causes a non-local exit.
Every form framework has validation utilities. Even the original formlet paper discusses a simple extension to the basic formlet system for it! Unfortunately, the most user-friendly form validation doesn’t play together nicely with interactive forms, so we have to make compromises.
There are a few different ways to display the results of form validation. One is to display all
validation failures at the top of the page. Another is to display validation failures right
alongside the field that failed to validate. The latter approach is problematic in Conform; it
breaks the important property that conformlets are pure functions of their arguments. Keeping
track of arbitrary local state for each conformlet is not trivial; each conformlet would need to
keep track of “which is which” among their children conformlets. For example, if you re-order
items of a list, you would need to somehow communicate that the conformlets were rearranged, not
just that the val was rearranged. React (a JavaScript UI framework) encounters similar issues,
and even their overengineered solution often requires manual intervention.
Thus, the limit of Conform’s validation is sad stuff like this:
(defun verified-string (val verifier error-text label)
(conformlet (:places val)
(let ((unverified-val val))
(custom-event
(if (funcall verifier unverified-val)
(setf val unverified-val)
(push error-text *form-errors*)))
(string-field (place unverified-val) label))))You need some top level code to display the errors:
;; render-form dynamically binds *form-errors* to nil for us.
(render-form "my-prefix" #'hunchentoot:post-parameter (eq :post (hunchentoot:request-method*))
`(form (method "POST" action "")
,(loop for error in *form-errors*
collect `(div (class "form-validation-error") ,error))
,(some-other-conformlet)))I think having only top-level errors is acceptable for a couple reasons:
- If the form is especially large, people won’t have to scroll through it to find where they made a mistake – it’s all at the top.
- JavaScript can be used to perform preliminary client-side validation at the point of the error. While the big point of a server-side-only form framework is to avoid requiring JavaScript, there’s nothing wrong with progressively enhancing the webpage with JS.
- HTML5 supports a lot of form validation, even with JavaScript disabled, through attributes such as maxlength and field types such as url, to help the user find errors before submitting.
Conform works in three stages:
- First Render stage: The form is rendered using whatever data was present before the user submitted. This render should yield the exact same form that was originally served to the user. The purpose of the first render stage is to determine which event handlers should be registered. The HTML output of this stage is discarded.
- Event Handling stage: Now that we know which events exist, they are executed as appropriate.
- Second Render stage: Now that the data has been updated by events, the form is re-rendered to reflect those changes.
The First Render stage relies on the persistent data not having changed since the user loaded the form. This assumption could be broken if the user has multiple tabs open, modifies the data in one tab, then submits the form in the other tab. For example, if there’s a list present, then in one tab the user deletes an item from the list, it’s impossible to meaningfully process the other form’s submission because it contains data about a field that no longer exists.
Here’s a strategy for detecting and preventing such unsafe scenarios:
(defun change-protect (val mtime-place subconformlet)
(conformlet (:places (val mtime-place))
(let ((new-mtime (get-universal-time)))
;; instead of using (place), manually specify getter and setter using cons. While it's
;; possible to use (place), it's convoluted and requires additional sub-conformlets to get the
;; ordering right
`(,(string-input (cons (constantly new-mtime)
(lambda (last-mtime)
;; non-local jump if mtime is not what we expected.
(assert (= last-mtime mtime-place))
(setf mtime-place new-mtime)))
'(type "hidden"))
;; ensure that mtime is processed first
(conformlet (:order 1)
,(funcall subconformlet (place val)))))))It uses a special hidden <input> which stores the last time the form was rendered. This must be
persisted somewhere, and the persistence location passed as mtime-place. If the form is
submitted, and the stored render time is not the same as the persisted one, it means that the
val may have been modified since the form was rendered, and the form should not be safely
processed. It’s necessary to perform a non-local jump; there is no other way to “abort” the
pending event handlers.
Read the function documentation strings for the built-in fields to learn about them.
The body of a conformlet form mustn’t cause side effects. It’s alright to
have side effects in event handlers, though.
Say your form has a list of strings, plus a delete button next to each string that deletes it. What if the user modifies a string and deletes said string, all in the same submission? Now we see that “ordering” is important: As long as Conform processes the edit before the deletion, all is well: The edit is peformed and then thrown away, in the same order that the user performed the actions. Imagine what would happen if the deletion was processed first instead: The edit might be applied to the wrong element of the list, or to an out-of-bounds element, causing an error. Conform gives the developer control over the order in which events are executed to avoid these issues.
Sub-conformlet event handlers are ordered according to the :order parameter passed to their
conformlet statements. custom-event statements are executed after all sub-conformlet event
handlers. All built-in conformlets except for buttons have an order of 0. Buttons have an order
of 1, because buttons submit the form and thus a button press is the last thing that happens
before form submission. (This button rule means that Conform has the correct behavior by default
in the deletion scenario described in the preceding paragraph). If multiple event handlers have
the same order (very common, because all non-button conformlets have an order of zero), their
event handlers are executed in the order the conformlets were instantiated during the render
phase.
Order is “local” to the current conformlet statement. Eg, if you have the following structure:
- A: Top-level
- B:
:order 2- C:
:order 599
- C:
- D:
:order 5
- B:
Conformlet D’s event handlers will be executed after conformlet C’s.