There are several names associated with the ability to execute a source block from within an org document. The tech that backs most of this ability is called babel. It has as its primary goal the ideas of literate programming and reproducible research. What has made it a popular feature is the combination of babel with many of the other emacs features.
As some examples:
- Using babel with tools like gnuplot, plantuml, ditaa and graphviz allows presenters to build diagrams dynamically from their notes.
- The integration with tools like TRAMP in emacs allows developers to dynamically build reports from running processes on remote boxes.
- Emacs ledger mode and the org mode spreadsheets have empowered some org mode users to build complex budget reports from their financial data.
- Clocking and some of the project management tools in org mode have allowed contractors to dynamically build timesheets from work notes.
- Or… I suppose you could use it for what it was intended for, as a means of facilitating a research paper.
Babel really made orgmode the original jupyt*r notebook.
Babel treats a source block in an org document as an active function call that can be called in various ways.
Here we have a basic python source block:
##+BEGIN_SRC python :var x=5 print(str(x)) ##+END_SRC
There are a couple of things of note here:
- The source block states what language it belongs to, in this case python.
- We have specified a variable x that is set to the value of 5.
- We are able to access that variable from within the source block.
Executing this source block will produce a #+RESULTS comment:
#+RESULTS: : 5
The following are the currently supported executable source blocks:
- plantuml
- graphviz
- ditaa
- gnuplot
- python
- javascript / js
- powershell
- cmd / bat
- sh / bash
Variables can come from a few different sources other than the source block fence itself. Variales can be in property blocks or property comments as seen below:
:PROPERTIES:
:header-args: :var g=global
:header-args:python: :var x=5
:var: v=10
:END:
#+PROPERTY: header-args: :var y=11
#+PROPERTY: header-args:python: :var z=12
Language specific variables can be defined as well using the language suffix
Variables are only one type of parameter than can be specified as an argument. Options such as results, noweb-href, eval, cache etc can all be specified in the manner shown above. In addition there is a global option that allows you to specify options for all blocks in any org file unless overriden.
The defaults can be found in the core settings file:
"orgBabelDefaultHeaderArgs": ":results replace :exports code :cache no :noweb no",
While variables are nice, babel becomes even more useful when connected to data tables and lists in your documents. Tables and lists can both have a name. This name allows them to be referenced as the source of a variable.
#+NAME: my-source-block | name | value | | Hello | 25 | | World | 42 | ##+BEGIN_SRC python :var DATA=my-source-block print(DATA) ##+END_SRC #+RESULTS: : [['name', 'value'], ['Hello', 25], ['World', 42]]
Up till now we have been ignoring one of the important parts of babel which is the execution type. Babel supports 2 execution modes:
- Output - This is the default and what we have seen so far. The stdout of a source block is captured and used as the return value.
- Value - This is more the traditional function call style.
##+BEGIN_SRC python :results value return 5 ##+END_SRC #+RESULTS: : 5
The results keyword really has 4 dimensions
- Execution Type
- output vs value - what do we capture?
- Output Handling
- table vs list vs raw - does the output get interpreted as something else?
- Formatting
- verbatim vs html vs org vs drawer vs code - do we wrap the output in something?
- Output Mode
- append vs prepend vs silent vs replace - how do we insert the output into the buffer?
Each results keyword can have a combination of those 4.
##+BEGIN_SRC python :results output table drawer
print("Hello World")
##+END_SRC
#+RESULTS:
:results:
| Hello | World |
:end:
Here we have formatted the output as a table and put it in a drawer. You can also output the results to a file with the :file <filename> parameter.
Here is some powershell doing the same thing as our python
#+NAME: in-table
| a | b | c | d | e |
| 1 | 2 | 3 | 4 | 5 |
| 6 | 7 | 8 | 9 | 0 |
##+BEGIN_SRC powershell :var DATA=in-table :results table
$DATA | %{"$_"}
##+END_SRC
#+RESULTS:
| a | b | c | d | e |
| 1 | 2 | 3 | 4 | 5 |
| 6 | 7 | 8 | 9 | 0 |
Org Mode treats source blocks like functions in some strange programming language tapestry. The call statements lets you call a source block with a different set of parameters. Executing one of these blocks will produce a new set of results below the call, but with the alternate set of parameters.
#+CALL: name-to-call(x=5)
Sometimes you would like to specify a quick little source snippet inline in your text. This can be done with the inline source block. Inline blocks should get parsed out by an exporter, support parameters and can be used for dynamic elements such as page numbers and other dynamics inline content.
src_python[:var x=5]{print("hello" + str(x))} {{{results(=hello5=)}}}
Variable inputs can not only be tables, lists or plain old data types but they can be other named source blocks. Here is a silly example. In this little example the second source block will have “hi” as its output. It has chained the results into x and used that variable in the print statement.
##+NAME: my-function
##+BEGIN_SRC python
print("hi")
##+END_SRC
##+BEGIN_SRC python :var x=my-function
print(x)
##+END_SRC
This can be really powerful for situations where you have one language well suited for a certain kind of data manipulation. For example, you use powershell to pull some data from azure. You then pipe that data into a different language to help manipulate the data in some fashion. For example, python. Finally you push the output of the python script into a gnuplot source block and graph it to visualize what is going on.
Sometimes it is handy to avoid recalculating something if the inputs have not changed. Org has a mechanism called cache. Be careful as this can have odd side effects if your code has side effects.
:cache yes
When you turn on caching the results block will get a SHA1 hash. This is a hash of the source code and parameters being passed to generate that output. If the hash matches the block will NOT get re-evaluated when you try to execute it later.
Things get even more interesting once we start to look at the sbe (Source Block Execute) table function. With this we come full circle. Not only can tables act as inputs to source blocks but source blocks can act as inputs to tables.
Remember that example in the chaining section above? We chained hi from the output of one source block into a variable and used it as input. Here we are calling that same block but providing a different input. Here the input is just a string. The result then gets inserted into my table. I can even pass cell references from the table as parameters to those functions.
| pulled from python |
| hello world |
#+TBLFM:@2$1=sbe('my-function',x="hello world")
Here is an example of this in action. The evalutation of the table takes a bit as it is waiting on images to be generated by graphviz ditaa and plantuml, the resulting file links are passed in the table. You can bring up this little unit test by calling the “Org Show Source Tests” method. You will need plantuml, ditaa and graphviz installed to run the tests.
The sbe calls chain to a method that is using a table or a list as source. The image examples generate an image and use a test source block to validate the file exists on disk.
Literate programming and reproducible research require several things:
- You can easily publish your document.
- You can easily have someone else run your source code with your data.
- All of your code and data can live in one seamless package.
These things are supported by the 3 core pillars of babel:
- Execute
- Run a source block inline in my code.
- Tangle
- Extract my source code with embeded data into pure source.
- Export
- Export my document to another format.
- Execute we have already been covering at length in this document.
- Export is supported by our html exporter, the reveal js exporter for presentations and pandoc for conversion to a ton of other formats.
- Finally tangling is supported by our detangler. Org Tangle File will attempt to create pure source files from the source found in your org document.
Babel has support for some security features. The eval parameter can be used to stop a block from being executed:
Either of the following will stop a block from ever being executed
:eval never :eval no
In addition you can use the query option to cause the system to prompt with a popup on whether or not you would like to execute the source block:
:eval query
NOTE: query or never can be used as a global option in your settings file to avoid accidental execution.
One final tool makes the literate programming toolset complete. This is support for the NoWeb macro language. Org has limited support for noweb macros. Much like macros in C or C++ NoWeb lets you paste either your source code between source blocks OR the results of the execution of the source block.
Here is a simple example where we paste the source in the block above into our source block.
##+begin_src python :noweb-href print-it
print("Hello World")
##+end_src
##+begin_src python :noweb yes
<<print-it>>
##+end_src
Just like you can paste a blocks contents into another block you can paste the results:
##+BEGIN_SRC python :var x=10 :noweb-ref evalsource :results raw
print("print('hi "+str(x)+"')")
##+END_SRC
##+BEGIN_SRC python :noweb yes
<<evalsource(x=20)>>
##+END_SRC
##+RESULTS:
: hi 20
This is kind of a crazy example. Here we have a noweb macro calling another source block that when run generates some code that is pasted into the first block and executed to generate “hi 20” as a result.
Remember the power of this system is that you can mix and match. Choose the right tool for the job.
You can add your own source block handlers. In fact, you can replace the built in ones. If you do not like that the built in python handler uses the version of python built in to sublime you can change the:
"builtinSourceBlockHandlers": [LIST]
Remove python from that list and then add your own python.py file in:
Packages/User/orgsrc/python.py
That will let you write your own handler. Any language that orgmode knows how to syntax highlight can have a source handler written for it. I am working to eventually support as many languages as I can. That said, there will always be cases where you need a language I cannot yet support.
Right now the easiest way to do this is to take a look at the powershell.py or cmd.py handlers in the orgsrc folder inside this package. You will see examples of how a handler can be written. You will need to write a couple of methods and I encourage you to share your handlers with me so I can include them in the main package or reference them from this documentation.