The TSDB is now called 'jd'[0] FWIIW. Funny, I've been using J on and off since then and I'm still not sure if it's genius or insane. It, along with the other APLs, is definitely its own thing, just as much as Lisp is (my previous favorite weird language was Yann LeCun and Leon Bottou's "Lush,"[1] which I continue to think is an amazing design). Once you get a little bit proficient at it, you finally realize what an actually high level programming language could be.
I think it's an idea whose time has come in some ways: hardware is more array-like than ever. J has evolved a lot from 806 to the present 902 beta; lots of performance increases from moving away from the single core paradigm (where it excelled anyway). Lots more to unlock there as well, and some interesting J-based startups brewing[2]. I also think it's one of the best user communities I know of.
Oh yeah, Art's new language Shakti[3] should probably be mentioned. Lots of people excited about this.
I started with Lush a long time ago before being obliged to switch to MATLAB and then C.
Recently I have been trying to learn Futhark which is a ML like language generating parallel code (E.g. OpenCL or CUDA) from pure functional code with nested array structures. I find this exciting because when using the GPU with a thin layer like OpenCL directly, you have to write kernels from scratch without much abstraction possible. Higher level APIs like PyTorch or Jax give you easy ways to flexibly structure the computation but you just have to hope for kernel fusion. In Futhark this is in principle explicit and well handled.
ISPC is another example of a C like language where SIMD style parallelism is unambiguous while staying relatively high level.
Yeah, I've been looking for some higher level way of taking advantage of GPUs. Pascal Jasmin wrote a doodad for J talking to Arrayfire, but I never got it to work properly.
Lush: it just felt comfy for a data science workflow. No DB adapters, or even real csv readers, but you could do whatever you needed in the interpreted language and go pretty fast with the compiled bits. Still feels like the future.
That's extremely cool. Julia has something very similar, in the form of CuArrays [0].
I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware.
OpenCL was pretty good in this a few years ago. I had some kernels which ran across AMD and Nvidia GPUs as well as many core Xeon systems without changing anything. To squeeze the highest performance out, you still had to tune for the device, but it was a great start.
Now, all vendors seem to have dropped support for OpenCL, though it will live on nicely in PortableCL, which implements it atop LLVM, allowing it to target anything LLVM does, which finally goes further to achieve the open-ness the original standard aspired to, than the vendors themselves at the time.
> I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware.
Then look no further than GPUCompiler.jl. Tim factored out all the platform agnostic parts of CUDA.jl (successor of CuArrays.jl and CudaNative.jl) into GPUCompiler.jl and that is now being used by the AMDGPU.jl package. It works today!
Thanks! I had hit some of those pages before, but their ontology isn't immediately obvious, so thanks for the clarification. In particular I've definitely looked at AMDGPU.jl recently, but as the page says: "not all features (and especially performance) are up to par with CUDA.jl" - hence my impression of AMD being a second-class citizen. Still, it's great that we have anything at all, and I'm extremely grateful for the hard work of those in the Julia community. It's a remarkable achievement.
> In particular I've definitely looked at AMDGPU.jl recently, but as the page says: "not all features (and especially performance) are up to par with CUDA.jl" - hence my impression of AMD being a second-class citizen.
That just takes developer time to fix. Currently it’s basically just a one-man show.
Questions are answered in no time, the folks are very friendly and treat everyone on eye level. A great role model for any programming language / software communities!
I'm still waiting for Shakti to have more info on their site. As on now, it seems like it is more tailored for Art's traditional clients of banks and not the common engineer.
I've learned a bunch of different languages[1], but J (and its friends APL and K) has been both the one that I don't feel I have the patience to learn AND the one that I feel like I might fall in love with if I actually learned it.
The incomprehensible programs just seem to hint that there is some sort of maximal information density where every keystroke provides maximal productivity.
There's also something artistically appealing about starting at a program and making a deliberate one character change that makes all the difference ... although on the other hand, that's kind of scary too.
[1] - I learned (in this approximate order): C, Java, Lua, C#, Ruby, Python, Lisp, Forth, Prolog, Clojure, OCaml, Haskell, Erlang, F#, R, Racket, and Rust. I also took a look at languages like: ATS, Cyclone, C++, Idris, Agda, D, and Perl 6. Of all those languages it really feels like APL/J/K are the strange ones.
I struggled a lot too. I think the main issue is the problem domain in which these languages excel at.
I like compilers, systems programming, distributed systems... but math? statistics? I'm admittedly terrible at that and find it really uninteresting. It doesn't help that most literature on J tries to teach its language concepts showing you how to solve math problems. I have no need for a fancy calculator.
For what it's worth, I like K and Klong and found them very approachable. Basically because you can get away by writing Scheme with m-expressions.
My general belief is that these languages would be far more useful as embedded languages; occasionally there is a need for the core of some complicated system to take advantage of their power, but the same design that makes them good at math makes them horrible for writing APIs or database access.
They work better than databases for accessing databases - nsl.com has examples like tables.k , a reasonably efficient in memory ordered-and-mostly-relational database in all of 14 lines of K.
And a variety of spreadsheets, from 2 to 100 lines.
They were sort-of-ok for API access, but Shakti’s FFI is actually incredibly straightforward - comparable to writing a C extern definition only much terser (‘cause it’s Arthur...)
APL isn't always written (⊂∘⍋⌷⊢), APL has a traditional notation which is like other procedural languages:
:If databaseResult.Rows.Count > 5
:AndIf warnOnLargeData
:For Row :In databaseResult.Rows
logToFile 'Careful, that's a lot of data, e.g. ', Row[1]
:EndFor
:EndIf
and Dyalog APL is a .Net language, which has extended this style with :Namespace and :Class and :Access and :Signature ways to build objects and methods to make your API and export it as a .DLL for use by other .Net languages, or plugin to ASP.Net and make a web service[1], or accessing most things C# can[2], e.g. if you want to use a System.Collections.ArrayList it's right there:
and that means things like System.Data.DataTable and SQL Connections for your database access. .Net isn't as popular as JavaScript and Rust, but it is a pretty large, stable, capable and established ecosystem to be able to hook into.
The core of APL symbols more or less /are/ an embedded language in the wider APL, just that wider APL is not showy for codegolf and is not unusual and pretty symbols, and you can't use it /without/ the symbols.
[1] See the examples shipped in the install folder like "C:\Program Files\Dyalog\Dyalog APL-64 18.0 Unicode\Samples\asp.net\webservices\"
[2] I expect not /everything/ C# can; C# is the first-class .Net language, but if you have things written in C# you can use them.
What makes them bad for APIs or databases? Both k and q are very practical general purpose languages (J and APL less so) that would be an appropriate choice for almost any kind of app, imo.
q especially looks mostly like a normal procedural language, and can be used like one too.
The first time I watched the video my response was that he was literally doing magic or that it was some sort of joke. Over time the video has become more approachable and I have a pretty good idea of what is going on. However, if I was asked to approach game of life in APL myself then I think I would still be pretty lost.
It's a kind of sleight of hand. It purports to be showing off the power of APL; what it's really showing off is the cleverness of someone who realized that the rules of Life could be expressed as a series of array operations.
APL has a kind of mystique, but all it really is is a bunch of NumPy functions with a quirky DSL (of course, APL came first and heavily inspired NumPy). But we've quietly been doing the APL experiment in the background, by pressuring programmers to avoid pure Python in favor of NumPy functions for speed, and the results are in - you can indeed push the array paradigm very far. But not quite far enough to be general and efficient.
The jury is still out on whether the quirky DSL is worth anything. My suspicion is that having names for array operators is very valuable indeed, but dedicated symbols for them much less so, and cryptic ASCII digraphs are barking completely up the wrong tree. But if you really want the weird syntax - well, it's not exactly a difficult language to parse, and Unicode is a thing now. If the notation is worthwhile, there's no real reason why any language couldn't have APL built-in as a DSL, just like regex.
Rank polymorphism is genuinely important and useful and a lot of languages are missing it.
Cryptic symbols, I agree, less so. If your language doesn't arbitrarily restrict identifiers then you can just define the APL operators as functions and write code that looks like APL - I tried doing this in Scala once, and it worked ok. But it's hard enough to convince people to take the time to learn what "map" or "filter" or "<* " does, never mind anything further in that direction.
Someone who was _very competent_ with J gave a demo of it at one of the Recurse Center (ne Hackruiter) _Brain Dump_ meetups back in the day and broke the crowd's collective brain. It might as well have been Brainfuck to all of us, but the proof was in the ... STDOUT. I believe they were demonstrating the standard factorial, map/reduce, etc. sorts of solutions you'd expect, but I remember being shocked by the small amount of code required to achieve the desired outcome. To the point the article appears to make, this may or may not be a good thing in terms of team productivity and maintainability, but it was damn impressive.
There's a very elegant Haskell quicksort implementation that misses out on performance because it's not in-place. It's frequently used as a demonstration. The performant version looks as bad as C.
I'd be more interested in a J/K demonstration that plays to the language's strengths. How about live-commentating e-sports?
* Player gets first blood. Commentator executes search query in J across DB. *
"We've seen $PLAYER_NAME contribute to 65% of first blood situations across their last 10 matches."
Idiomatic K solutions (and from what I hear, J as well) tend to be faster than idiomatic C solutions to the same problems.
It originally seemed like magic to me, but really - there is no magic. It is simpler in K and J to process more things, in more stages, but with completely predictable and very pipelineable operations. The J/K program often does 2 or 4 times as many operations, but has virtually everything prefetched to L1 before it’s needed so it runs as fast or even faster.
CPUs are constantly converging towards the K model (GPUs were always there).
K is often shorter and simpler than J for these kinds of things, although J’a much richer vocabulary pays off if the problem domain lends itself to J’s immense built in vocabulary.
I have a hard time understanding where apl/j/k maintenance costs could be higher than java / c#
As A. Hsu say, a java method name is long enough to encode a complete algorithm. the drag of having to maintain MLoC of boilerplate with all the opportunities for bad naming, bad style, NPE and such is so huge.
Funny, that phrase can also describe k itself... It's a mindset, or cult, depending on who you chat with about it. Very powerful, and somewhat obtuse, which often go together.
Fwir, Kdb+ is written in k which is sorta APL, and then there's q for writing kdb+ programs which is easierish. But still an adventure.
kdb+ is written in C, but q is (largely) written in k. One of the first things kdb+ does on startup is load the file `q.k`, which is a k file which defines the q language. The interpreter then switches from interpreting k to interpreting q.
Interestingly enough you can fairly easily program your own language (written in q/k) on top of that, and set the interpreter to evaluate that language by default. Kx offers a Python integration layer called EmbedPy which (among other things) exposes a Python evaluator as the "p" language in kdb+.
But do they implement the same programming paradigm, or is there any fundamental differences
For example, C# and Java implement the same paradigm , they are different, many argue which provide a better set of feature, but I would not call the differences fundamental
C#, F#, implement completely different paradigms
F# and Clojure, same paradigm, but have fundamental differences, F# being static and come from the ML family , Clojure dynamic and come from the Lisp family
So APL to J, is more like what to what, C# to Java, or F# to Clojure ??
APL and J are similar but for a pretty fundamental difference: APL has nested array theory while J is flat array theory. In practice, this does not make them that different to the user, except APL's model is arguably more elegant since it's "turtles all the way down" (J array elements are not array themselves).
K on the other hand, is really different! K doesn't have true multidimensional
arrays, but lists of vectors. So it's ideal for its use case: finance (1d/2d numeric tables).
It's more like C# to Java. Setting aside the different character sets used, a lot of code written in one can be translated almost directly to the other. Whereas F# and Clojure, while using many similar idioms, have fundamentally different approaches to types which significantly alters your program structure and design approach.
Both J and APL work on the same type of things (principally, arrays and nested arrays) with functions being applied over the arrays or across multiple arrays. And both languages encourage a point-free or tacit style of programming (though J, from my limited experience with both, seems to push this a bit further sometimes). For a functional paradigm example it's probably more like the difference between SML and Ocaml, or between two similar lisps like Common Lisp and Emacs Lisp. There are clear differences in focus in the two language designs, but much more in common (again, setting aside the choice of characters, you can find an equivalent for most APL symbols in J and vice versa without needing to create too many new definitions).
That is what I was meaning, to me C and FORTRAN are mostly "the same" except at a surface level, and FORTRAN has multidimensional arrays and regions (a slight abstraction that can help the compiler, in particular, for automatic parallelization of loops)
J seems like a language I should learn using flash cards. Make a flash card for each character, for each operation, and just learn all the operators, and then I will know J.
Any truth to this?
I suppose it might be like telling someone, "you know conditionals and loops, now you know how to program!"
Definitely no; I tried something similar with the J dictionary. The right way to learn is to go through "J for C programmers" (it comes with the language), or just grab NuVoc[0] and start writing code for something non trivial. Labs can also help for areas you're not familiar with. Its the composition in J which carries meaning; and stuff like hooks and forks (more or less, writing J AST's directly) make it non-obvious in early days.
Memory failing me; it used to be part of the IDE, with a copy of it and the dictionary stored on local storage, but I guess J has changed enough recently it needed to go up on the wiki where it can be changed. You can always find it from nuvoc.
Usr Pri JfC LJ Phr Dic Voc !:
All these are interesting; some primers, some the old version of the dictionary (I think it's depreciated for now). I end up !: a lot for the foreigns.
It would be a lot like that. There's a mindset shift to thinking of arrays as geometric objects to manipulate in space instead of as boundaries on valid indices. One of my favorite examples is computing a moving average over some vector of input data. Instead of thinking about looping over each sub-range to find its average, and having that nested inside a loop over possible sub-ranges, you can build a 2D array whose rows are time-shifted versions of the original 1D array. Sum that column-wise and divide by the number of rows to get your answer.
Understanding how you can build that iteration space as reified data gives you a lot more expressive power than you get from remembering that `|.` means "rotate." It's easy to look up which symbol corresponds to a chosen operation (and you'll memorize the ones you use a lot anyhow).
In my experience -- flash cards might help a little, but writing code is the only way to internalize it. Read the books/tutorials and follow along until your feet are a little wet.
Keep in mind that it's not just about the verbs (operators) in isolation, it's also about sentence structure. While you can get quite far in the language without learning how to write a fork or a hook (two related ways of combining verbs), it will be near impossible to read someone else's code until you practice these forms yourself.
Having said that, if you're anything like me you will have the NuVoc page [1] open basically all the time while writing or reading J. It's hard to keep them all straight, esp. the verbs that you don't use very often.
I've been experimenting with flashcards that prompt me to write or solve some problem, versus the more common memorization usage. Not necessarily solving the same problem (write FizzBuzz in SomeLang) but maybe more complex (select a public web API with a JSON source, get the data, process it, and make a simple report). A way to reinforce understanding of different libraries or tools. I haven't figured out the scheduling on that in Anki yet, though. The same ratings can still be applied, even if it takes an hour to do it it's "how easy was that hour?" Did I have to pull open a lot of language and library references? Was I able to, without much effort, create the basic project template and start filling it out? Or did I have to look up the details on defpackage again?
Then combine that with more explicit memorization/recall cards like I use for Spanish. "In Common Lisp what non-destructive function filters a sequence based on a predicate?" "remove-if".
I think this should work well as a way to develop an understanding when paired with books/tutorials on the language and its libraries.
Good luck, I hope it works! Consider adding some flash cards with grammatical forms to mix in with your "character" cards. e.g. a monadic hook "(f g) y" is equivalent to "y f g y". And some short idiomatic phrases, e.g. "(#~ f) y" means -- approximately! -- "copy the values y_i of y wherever (f y_i) is true." You can find examples of these little phrases on the J wiki.
Thanks. It's still an experiment, and about to be restarted (was going to be learning modern C++ and some APIs for work, but direction shifted and I'm now on a Java project, which I haven't used in a long time). I'd actually like to try this for J, but work and Spanish (to speak with the in-laws) are my personal priorities right now. Perhaps in October I'll start studying J to use for this next Advent of Code. I have been getting the itch to relearn it. I learned a large chunk of it circa 2010, but haven't touched it since, then APL in 2015 or 2016. I've never used either professionally or for a major project so a lot of things have been forgotten.
I'm basing my experiment on what I learned using Anki for language learning. Much like you said, you need larger statements and to develop an understanding of them. For Spanish I have some loose writing/speaking prompts and some reading prompts (not copying the text, but a prompt to go to today's newspapers and read some articles and explain them to my wife or to read some paragraphs from a book). How well I can complete these without needing a dictionary or to ask for assistance determines how I mark the cards. It seems to work decently, they come up almost like pop quizzes in school classes since the spaced repetition system means I don't see the same prompts each day. I think it should work to have project prompts for programming languages in the same manner.
I stumbled across J years ago when I was messing with Supercollider[0]. I wish I had a use for J, it's very cool.
Slight tangent, Supercollier is one of the best langs I've ever played with. Smalltalk + J. Totally blew my mind when I first wrapped my brain around it!
Some of these languages are an excellent fit for specific problems. I still think an array based language is best for sql queries rather than set based. Any time you hear something like "select the first....", you don't want a set, you want an ordered list.
Shameless plug: I'm currently working through Project Euler problems using J, and the language is obscenely well-suited to this kind of problems. The terseness has grown on me, as for the other thing - well, I guess code in any language can look like line noise, it's all up to the programmer.
If someone is interested in array languages, I would start with learning q, then k, J, and finally APL. q and k are much more conventional and less complicated than J/APL. They also tend to be more practical, and less mathematical. They also don't have multi-dimensional arrays, so the code tends to be both less general and less confusing. Once you've gotten familiar with them, J and APL won't seem as weird.
Moreover, it's entirely possible to write regular procedural code in q and k, so it's easier to transition to the array language mindset.
My intuition for array-based languages is that you can use the "obscure" code, which heavily uses the standard vocabulary, for the low-level details of the business logic, but that you would still maintain a set of high-level verbs (DSL?) to describe in more clear terms what the hell the code is doing.
However, the few real world projects that I've seen in these languages won't use any kind of DSLs and the argument usually is "for array-based language programmers, this is actually very clear and concise". I need to improve my J to verify that :)
I think it's an idea whose time has come in some ways: hardware is more array-like than ever. J has evolved a lot from 806 to the present 902 beta; lots of performance increases from moving away from the single core paradigm (where it excelled anyway). Lots more to unlock there as well, and some interesting J-based startups brewing[2]. I also think it's one of the best user communities I know of.
Oh yeah, Art's new language Shakti[3] should probably be mentioned. Lots of people excited about this.
[0]https://code.jsoftware.com/wiki/Jd/Overview
[1]http://lush.sourceforge.net/
[2] https://www.monument.ai/ for example
[3] https://shakti.com/