When this last round of discussion of visual programming happened I had a minor epiphany.
For a little background, I’ve maintained that visual programming goes absolutely nowhere until we have visual diffs that work (work as in workflow). I’ve been saying that since before the UML Trough of Disillusionment kicked into high gear. Without diffs, without analysis, we are going nowhere fast. Almost every other link of the lifecycle is still intact with visual code but focusing on an editor without diffs breaks the chain. There’s no producing and maintaining commercial grade software without it. It’s either toy programs, or absolutely heroic effort, and what else could you have done with that much energy?
The epiphany was this: motion detection has been a feature of digital video since at least the MPEG days. Someone with a background in video compression needs to tackle the problem of doffing visual code. Figuring out how the code should look (mechanical sympathy) to facilitate this.
> Without diffs, without analysis, we are going nowhere fast.
The victory of plaintext coding over visual programming is another example of worse is better. Semantic, language-aware diffs exist, but I see them much less frequently than dumb plaintext diffs. Intelligent code search exists, but in many cases grep is just as good. IDEs come with very advanced refactoring tools, but I still prefer to refactor using vim macros.
When we write code in plaintext, we're expressing our code in a lower level of abstraction with really great tooling. Doesn't matter if it's java or yaml or something custom—plaintext tools work on all of them. Visual programming languages can't do this. There's no language-agnostic vim or git for visual programming. Even if we wanted to invent sort of unified language-agnostic visual programming abstraction, the ecosystem isn't there, and there's no guarantee it'd get adopted.
So I think plaintext will remain king for the foreseeable future. Visual tools have to use human-readable, human-editable plaintext as their source of truth, if they want to succeed.
I work with a few proprietary languages and many tools don't have semantic information but it's still tolerable as plain text. I suspect I'd be stuck using awful tools if these were visual languages.
I think of it like comparing cuniform and hieroglyphs to phonetic alphabets.
The pictures are easier to learn and use but they are not as expressive and precise as the more abstract text. Text is a marvelous innovation. The fact that most people on the planet can communicate complex ideas and emotions as well as program machines with it (in many different dialects) is mind blowing.
Using pictures to program feels like a regression.
The challenge with visual programming from my perspective is how to get the abstraction level just right. It's like Terraform/HCL modules: too granular, you wind up passing in a billion parameters, and the module adds little value. Too high-level, too many assumptions, also adds little value.
I have an idea that I think would be suitable for visual programming. Every time I think about it, I come to the conclusion that first you need a DSL, or at least a JSON Schema or something.
If you have a DSL/schema, then you also have a way to diff.
Then, you implement your visual programming environment, and visual diffs turn into effectively communicating those differences visually. Faded color to indicate a deleted node and connector, bold parameter and value to indicate a change, etc.
I think this idea has promise but I wonder how it would work when many of the visualizations I create to help explain logic wouldn't fit neatly into a DSL/schema. For example, I might use red in some places to refer to data from a particular client, or underlines to indicate constants, or arrows to indicate things that are related from a business perspective but not data or logic flow. I guess this could be standardized, but that almost defeats the purpose of having the computer do the heavy lifting.
In other words, not only does understanding these diagrams assume a ton of knowledge in the problem domain, but the crude tools at hand for creating them (lines, words, symbols, shapes, colors) can be repurposed in so many ways by different people and different projects that I don't see how generic motion detection will be able to detect high level changes. Sure, detecting changed nodes and descriptions might be easy, and possibly useful, but I think there will never be a substitute for human comments like "removed XYZ because the boss said so" or "renamed foo to bar because of IRS regulation so-and-so."
>The challenge with visual programming from my perspective is how to get the abstraction level just right.
It is a big problem if you are trying to develop a genric visual programming language. But much less of a problem if you restrict the problem domain to something like data processing, signal processing or image processing.
Node-Red "Projects" feature has version control features that include a diffing tool (running on git). I've tried UML code generators in the past and was a skeptic of NR at first, but for headless applications subject to recurring change or needing high-level access to complex stacks (like video compression) its really impressive. Don't get me wrong, its still glorified NodeJs and not for every job but its worth a peek...
I get what you are trying to convey but that usage of the term is not helpful in the general case as it confuses what mechanical sympathy actually means.
Mechanical sympathy is simply this: abstractions that acknowledge the underlying embodying stratum of a dynamic construct. In code, that translates to software that attempts at harmonizing the software abstraction with the operational characteristics of the underlying computing device.
The thing about visual programming, at least a lot of them, is they store the data as an ASCII version under the hood, but render and edit it visually. So for those, normal diff still works, and the renderer can highlight things accordingly.
GitHub has had diff for images for years, and there's a web automation testing (think: Selenium) vendor called Aplitools that takes snapshots of web pages and diffs those too.
I don't know of any non-proprietary tools, but at least the tech is out there.
Another place where visual diffs would be indispensable is file-system organization. Each node in a file tree contains another graphable data structure: permissions! Especially on network shares where user groups are more common.
As long as there's some structured record of changes made to permissions and structure, those can be visualized.
I've been experimenting with a project I'm calling guish, a bidirectional command line editor for writing bash pipelines with GUI modules. The best way to explain it is that text written in the CLI prompt updates the GUI and changes to the GUI updates the CLI prompt in turn. It operates by working on the bash AST and has step-by-step outputs so you can see the result of each step in the pipeline's output.
Looks wonderful. I think being able to type examples from a textbook or a tutorial into this and see each stage visually would make people learn a lot faster.
I don't believe in visual programming ever replacing code for general programming. However, many examples exist when visual programs work for narrow domain-specific applications. This works because such applications allow exposing only high-level domain-specific abstractions. This reduces complexity enough to be a good fit for visual representation.
Nah, I wouldn't be too sure. Each generation introduces their own new abstractions on top of whatever existed before. Any code that was written before their era is outdated legacy unreadable stuff, and it can only be fixed by writing new shiny modern blazing fast rocketemoji stuff. It doesn't matter that existing code already works, it must be changed anyway, and we all know that change==progress.
Maybe it won't happen during the next decades. Most likely it won't happen until all currently-alive maintainers of Linux "retire", and most programmers alive are from generations that grew up being pressured to use either Rust or React.
So I can easily imagine a future where there's two main camps: low level code in Rust (where "low level" now means "anything up to, and including, the web browser"); and high level code in some graphical thingy that compiles down to WASM.
And the path I see where we'll reach that point, is by the Rust community continuing to do their thing, and by SaaS companies continuing to do their thing (VM->Docker->"Serverless"[1]->"Codeless"[2]->"Textless"[3]).
With all that, I think it's possible that visual programming might become the dominant way of doing general programming. Not the only way (after all COBOL is still a thing today, so it would be like that), but I can see how what's "normal" is shifted up one level of abstraction higher, so that "code" in that future is seen the same way as "assembly" today; or "assembly" in that future is seen the same way as "punch cards" today.
Those old timers think their Rust language is good enough with their memory safety and stuff, and prefer to ignore decades of progress on programming. Separating the code in text files? Yeah no wonder they keep having incidents like the DeseCRATE or Cargottem supply chain attacks (years 2038 and 2060 respectively), if they install dependencies willy-nilly without even looking at the code they're bringing in (nobody wants to look at dependencies with that primitive tooling). If they used modern tooling instead, they would be able to easily tell at a glance which "nodes"[4] are trying to do suspicious stuff just by their location or relationships with other nodes; or even restrict network or filesystem access to a region of the canvas. They say "well, just use capabilities", but then the code looks like an unreadable mess and way too error prone, when modern tooling lets you just draw a square on a canvas and "any node[4] inside the square can read filesystem, everything outside it can't", without needing to modify any of the nodes[4] themselves. It eliminates whole types of vulnerabilities and whole types of human errors.
(/s, mostly)
[1]: Still needs servers, but you don't worry about it because servers are too low level and you only want to focus on the important stuff (just code).
[2]: Still needs code, but you don't worry about it because code is too low level and you only want to focus on the important stuff (just English).
[3]: Still needs text, but you don't worry about it because text is too low level and you only want to focus on the important stuff (just visual concepts).
Non-programmer here. I’ve long desired to program Arduino microcontrollers but haven’t had the ability and/or the patience to learn programming. I’m a very visual person and get confused if I have to keep more than a few lines of code in my head. Even if I understand how I want a signal to be processed into an action, when the flow chart in my head needs to be translated into text I get overwhelmed.
A solution arrived for me in the form of a number of visual arduino IDEs. Flprog, Outseal, and Open PLC. They all allow programming in ladder logic, and Flprog and Open PLC allows for function block programming. Ladder logic is useful for many tasks involving discrete I/O (Ex. A button latches a contact which turns on a relay which turns off a light and turns on a motor.) Function block programming has enabled me to perform more complicated functions like creating an HMI using a menu system displayed on an LCD connected via I2C. These are tasks that I wouldn’t have a chance of accomplishing without the complex arduino code being contained in graphical functions blocks which I can wire together and configure with a gui.
Prior to compilation, Flprog displays the raw arduino code in the arduino IDE, which gives me a chance to exam the actual code produced by my function block program. I find it to be very educational.
As a non-programmer, I have no idea if this embedded programming use case can be generalized to more complex programming tasks, but for my limited use case it’s been transformational.
> I’m a very visual person and get confused if I have to keep more than a few lines of code in my head. Even if I understand how I want a signal to be processed into an action, when the flow chart in my head needs to be translated into text I get overwhelmed.
It looks like you've got things sorted. But another option would be to write your flowchart down on paper ... get your idea out of your head in a visual medium, and then turn that into code without needing to hold the whole thing in your head at once.
Former machine technician. I used to repair machines in factories. I’ve been lurking on hacker news for over a decade because I find the work of programmers fascinating, but my brain just doesn’t cope well with code. I do however, understand machines and control logic, so for me visual programming is a sweet spot.
Interesting. Looks like the Flprog website is largely in Russian and Outseal seems to be Indonesian. Is there a good introduction to ladder logic in English?
OpenPLC is a good English introduction. You're right, the biggest downside to these programs is that support and documentation are not in English. The Outseal IDE gives you the option of setting it English, although error messages still get output in Indonesian. Flprog also allows you to switch to English within the IDE.
If you're willing to pay a little extra for hardware you could go with Allen Bradley and their Connected Components workbench software.
not a well thought out take, but i think visual programming should stop trying to encode application logic and focus on:
- building good UIs rapidly (e.g. improving & simplifying layout options, look & feel, etc.)
- make debugging trivial & powerful
- make deployment trivial
Visual Basic was on the right track in almost every way except as a language. Hypercard maybe had the language thing figured out: use english as a base?
Hooking up loops and conditionals visually is a waste of type, people can type it out with a little instruction with a good IDE & code completion that caters to non-coders. Focus on the tooling around the code & the things visual development has proven good for (laying out UIs, debugging) and then the stuff that people don't want to think about (deploy)
> Hypercard maybe had the language thing figured out: use english as a base?
I wrote some stuff in applescript, which is similar / descended from HyperTalk. I would say it's the worst language I've used; it's so hard to use because of pretending to be natural. It doesn't help when Apple forgot how to write documentation somewhere around 2008, so there's no authoritative docs, and it's unsearchable because it looks like english words.
Literally anything else would be preferable, but applescript is the best way to automate MacOS. Thankfully, I only had to use it once.
applescript was a nightmare with zero autocompletion and a ton of magical strings and bad syntactic choices, some inherited from hypertalk, some not
i don't think an english-derived scripting language would be a bad thing (well, I wouldn't, since I created https://hyperscript.org, now would i?) if the tooling was good, I don't see a huge difference between someone dragging and dropping a for loop and an if statement and just typing
for thing in whatever
if the thing's field is false
do something
The problem with Hypercard is that it used an extremely restricted subset of English constructions. Once you got used to the limitations, it was fine, but could be very frustrating for users learning what sorts of phrasing Hypercard expected.
Allowing too large a subset of English ends up allowing more ambiguous statements and more user surprise. Also, more complex grammar increases the chances of mistakes in implementation.
My current experience with LLM hallucination makes it clear that at present, we can't just throw an LLM at the parsing and semantic analysis side of programming language implementation.
Non-coders will cease to exist about a half hour into building their first visual program. Even non-coders are going to want a place to put some code. They will get sick of all the clicking, clicking and dragging and wonder if there is a better way.
Yeah, I feel like this blog post misunderstood the original post it was responding to. The argument wasn't simply "add visualization where it makes sense". It was "diagrams are more effective for dynamic analysis, as opposed to representing static structure, which code already represents effectively." Debugging being the #1 candidate, as you pointed out.
It has a real place among novice programmers. We even have some experts who use it as a fun alternative to writing text. I don't see visual systems as an effective way to replace everything us experts are doing but they've gotten a ton of mileage in the jr. and learning domain. Scratch (https://scratch.mit.edu/) being another obvious example.
See the email in my profile if you are interested in roles and I'll see if we can find something that fits.
Isn't this how some UI frameworks work(ed)? You had graphical design tool that then generated some blob (or xml) that you then included in your application.
More extreme example would be the various business process etc things, like KIE/Drools/jBPM, or BPEL/BPMN, that also allow various ways of mixing various visual and traditional practices.
I think the common thing with all of these is that in the end no developer actually liked them.
Now thinking of this, one of the problems is that there is no suitable generic graphics format that could be used as a base for these sort of visual programming ideas. So there isn't any text-editor analog for visual programming currently, instead all visual programming environments I've seen use some specialized custom tools and formats.
In that thread, I mentioned the work on systems analysis and visualization done by the people behind Moose. (a framework for Pharo Smalltalk).
I didn’t follow that project for a while. It seems that their latest work is called Glamorous Toolkit, and the presentation is worth watching and relevant to this thread:
Visual Programming is fine for very specific cases: Graph-based connection of inputs and outputs. It was used for DirectShow GraphEdit and ComfyUI, and works find there. Just please provide a way to also type things in manually as well.
Visual programming has a long history. See Nassi-Shneiderman-Diagram's, which are even a German (Now eu) DIN standard. And you have Drakon from the Russian space program.
As a matter of fact, we _do_ have something that is more 'visual' or 'symbolic' than written speech, and it has a critical role in certain abstract forms of writing. It's mathematical notation (or, sometimes, symbolic logic) and it's already great.
Thus, any sufficiently advanced visual programming environment is indistinguishable from rendered LaTeX.
Presumably as an example of where it doesn't need to go? The graphical representations are incomprehensible at first glance. I'm sure you can learn to understand them, but at that point why not just learn how to read the textual representation? Especially when it doesn't even uphold its claim of being equivalent. The textual representation shows some kind of error handling, but I can find nothing in the graphical representation that indicates that there is error states.
It's interesting to see how different programming languages approach building software.
Languages like Lisp are all about planning out the whole program structure (the "Abstract Syntax Tree") upfront, kind of like a blueprint. But modern tools and visual programming often take a more piece-by-piece approach. You build smaller parts first and then connect them together, gradually building up the whole thing.
This bottom-up method feels more natural to how we solve problems in real life. Visual tools make it even easier because you can just drag and drop components instead of worrying about complex code structures.
I wonder if this kind of visual, incremental approach could make programming more accessible to everyone. It might help bridge the gap between our everyday thinking and the formal world of code.
> Languages like Lisp are all about planning out the whole program structure (the "Abstract Syntax Tree") upfront, kind of like a blueprint.
Lisp seems like a poor example because the emphasis there is building incrementally in an interactive REPL session. I think this style of programming would be very amenable to visual programming.
I was referring to the fact that in Lisp everything is an expression, so technically Lisp doesn't stop you from writing something like this, whereas in Python you'd have to define distinct variables and nest them afterwards:
((if (< 1 0) + *) 42 100)
Now imagine if this were a bunch of deeply nested expressions...
I guess I just don't see the problem that you are seeing for expression-based languages + visual programming. Are you saying that it would be cluttered if there were too many things without names?
I'm saying that expression-based languages encourage a style of programming which would resemble deep hierarchies in a visual language. They don't enforce abstractions via statements and whatnot.
> Sebastian argues that we shouldn’t try to replace all code with visual programming but instead only add graphics where it makes sense
I just interviewed Daniel Kallin, the maker of the language/tool Nomnoml (https://www.nomnoml.com/), for making beautiful diagrams, which is one such place where it makes sense [0].
He had an expression I liked, wanting to make a tool/lang that felt like "drawing with ascii".
I think this is the way for visual programming. You must do nice DSLs side by side with the visual representation. The text must be source of truth. It _has_ to be that way! That's how computers work!
---
Don't let the hard to pronounce name throw you off, Nomnoml is a super fast and handy tool for making beautiful visualizations, hand crafted over 10 years and with no ads or login and fully open source.
Totally agree. It also hard to do diagrams in a lot of document management systems so you end up with this crazy mix of Word docs and Visio diagrams. In the 90s they tried to fix this with OpenDoc and OLE but somehow that got out of fashion.
Yeah but it turns so niche you write visual 3D interfaces in Unity for your specific application. Terminal editor improvements are visual programming.work iterm2 I can see command history and dig into problems.
There’s a huge advantage to see programming for non programmers. Do you know how hard it is to explain hypervisors, VMMs and OverlayFS to non programmers? I gave them a simple example they need to make a simple text update to a Photoshop file. It costs a renewed font license for the latest but essentially an application library they can open the legal version they bought on windows see it look sort of okay looking on osx make changes to the file and be done with it.
One thing ChatGPT taught me was people don’t understand code. SQL was supposed to be ChatGPT of the day. Executives had no concept the “power” of essentially an NLP on top of structured data.
And for developers a high level AutoCad like view of what’s talking to what, etc. is valuable. I don’t need to visualize a merge sort but maybe a realistic abstract demo non programmers and programmers can work against.
I got tired of corrupt paste programmers mocking up design in Figma. I want to show you acquired Product X on SAP and it’s a different version. Using token data I want to be like as of right now you see realistic data and users, zoom in, show it merge in a datalake and produce an auditable report.
And that it runs strategies so algorithms (probably AI) can show aggregated date and where things might be failing. Zoom out see two icons from two company’s on different software and a generic data lake, zoom in and see and update data. See a message in Skack/email/etc. indicating whatever. Show elementary that consultant leads products like Twilio run off AWS and telemetry shows they don’t use but one feature and we can swap it out. All ok a running system without awkwardly switching windows. It helps executives get the big picture,’itnactualu works and you drill down enough you see type forwarding and an IDL for the programmers.
When was the last time you used merge sort not in std library or visualization would have helped you solve it? And yes I’m not talking just about corporate software.
Just the other day I was wanting to show an overlayfs/squashfs. I mean I’ll go down to C or god help me ASM so it’s not technical ability. It’s usually not even big vs little indian that trips people up but the big picture.
I wish so bad things like Salesforce would release their code, it is very rare someone would “steal” it and companies I know would benefit greatly and still pay full cost if I could rip out things and see their documentation is wrong and not have to go to COBOL or disassemble to me that’s visual programming. “INT32 actually a mutable type oh yeah they also made up their own types. That’s in red. In fact here’s a point map of all types that don’t match specs. First question why (in 1994 a driver didn’t work right so we did some weird type forwarding) ok… but in huge systems I’m not sure visual programming would help. Maybe weird things like btree in rust is better until you make an roc call outside of rust for some reason?
For a little background, I’ve maintained that visual programming goes absolutely nowhere until we have visual diffs that work (work as in workflow). I’ve been saying that since before the UML Trough of Disillusionment kicked into high gear. Without diffs, without analysis, we are going nowhere fast. Almost every other link of the lifecycle is still intact with visual code but focusing on an editor without diffs breaks the chain. There’s no producing and maintaining commercial grade software without it. It’s either toy programs, or absolutely heroic effort, and what else could you have done with that much energy?
The epiphany was this: motion detection has been a feature of digital video since at least the MPEG days. Someone with a background in video compression needs to tackle the problem of doffing visual code. Figuring out how the code should look (mechanical sympathy) to facilitate this.
reply