Surely their research is not about shipping broken software though, then your goals would be in conflict.
I suppose your point is that the researcher's goal in not exactly to ship working software either, but wouldn't that put the researcher's goals at worst neutral then?
Likewise your goal is not to publish research, but it is also not to actively work against it either. From their standpoint you're also at worst neutral.
It's also worth pointing out that the motivating factor doesn't necessarily have to be the same for each party to have a common goal. I'd argue that this is how it actually is most of the time.
I do work for a customer so that my boss doesn't fire me and I get my paycheck, whereas my boss does work for the customer to bring money to their company and to not go bankrupt. The customer helps with the work we're doing for them because they want something out of the project that is their money's worth. Our motivations are different, but the goal is the same - to build a thing that works.
My point is that it just seems very likely that some common ground can be found between you and the researchers, regardless of your individual motivations and since there's really no inherent conflict either.
> regardless of your individual motivations and since there's really no inherent conflict either.
Within organizations, there's an inherent conflict between any orthogonal goals.
In theory there's no conflict, but in practice, there is constant competition for time and resources. This creates conflict between any groups whose goals are not aligned, including groups whose goals are completely unrelated. This is also why organizational politics is the way it is.
you stated the condition/assumption - that the goals are orthogonal
I believe these are not - researchers publishing how to create good software and developers creating software may be seen as goals as aligned as fixing an incident and publishing a post mortem, or as writing an RFC and implementing it. Or publishing a post on how you remodeled your infrastructure.
They diverge at some point, yes, but that's not orthogonal at all
> researchers publishing how to create good software
I disagree that this is what researchers are doing (at least from my point of view). This is actually an area where I agree with the article, there's a gigantic gulf between researchers and practitioners. The things that academia puts out are not, generally, what I would consider to be good software.
I think this fundamentally comes down to a difference in the definition of "good" between the two camps. So far as I can tell (not being an academic), the academic definition of "good" seems to revolve around software having certain provable characteristics. My definition of good software involves the software exhibiting useful characteristics. And those are, generally speaking, orthogonal. If not sometimes inhibiting each other.
But, of course I would think this, I'm a practitioner. The academics probably have similar complaints about me.
I can see this is going to be one of those HN discussions that goes back and forth forever, maybe somebody should do some research on how orthogonal the goals really are!
If the researcher makes a neutral contribution and someone else you could hire with the same resources would make a strongly positive contribution you're always going to favor the strongly positive contribution.
In short, the incentives/requirements in academia have changed so that producing working software is no longer feasible. He says he got his PhD with exactly 0 publications, became a professor with 5.
Nowadays, you have to have 5 publications to get your PhD and god knows how many to become professor.
That means 1 paper successfully published per year if you want to get your PhD in 5 years. Doing software is not possible, it is too risky and just takes too long.
Seems like a pov, I like working with researchers and shipping interesting working software.
Why is it a problem they are at odds? Its not a researchers problem to create scalable production/consumer software or hell even git repos or libraries for you. Its yours! Thats your interest.
I’ve run into both. I sit between a research organization and a production organization and have to produce artifacts on both sides, this suits me quite well because I’m very much an applied research person. If it’s not feasible or tangible then it tends not to interest me much.
Doing applied research and working with people who want to do that tends to work reasonably well with production, because the point is to research techniques and algorithms and so forth that help improve production. Theoretical or more future-focused research tends to need to focus on building minimal scaffolding to be able to test ideas, when that kind of scaffolding gets integrated into production software, that’s where the problems arise, because by necessity it’s a kind of MVP. I am right now working on rearchitecting a component that was built this way and labeled as production without going through hardening first (though it did go through a great deal of testing it didn’t get the code review it needed).
In the end I suppose my point is that research and production aren’t inherently at odds, or even orthogonal. Applied research can be valuable to improving production software by studying current stable code and processes and researching improvements, but trying to do blue-sky research by incorporating code into production can be orthogonal or in some cases truly at odds with stability. It matters a great deal who is doing it, and how it’s managed.
Preaching to a colleague, I also work on applied research, very similar to you. I also try to implement papers on my own most weekends, a little coffee and POC haha, or "this will never work".
As an interesting counterpoint, a16z (the VC firm) recently released a new cryptographic system - a zkVM (zero knowledge virtual machine) that was a collaboration between academic researchers and engineers on their staff.[1],[2]
> students design a small study or experiment, collect data, analyze them, and figure out what (if anything) they’ve proven
Based on ample experience, both first and second hand, that's going to fail. It is really hard to instill proper methodology, domain modelling, and statistics in students, and since the premise already is that it "would not disrupt other curricula" (PM speak for: little time and attention), students are going to walk away with the wrong ideas and think that they now understand experimental science.
I did both a group-project software engineering course and a research course in my undergraduate Computer Science program. It's definitely possible to do.
Bridging the gap between research and practical programs is so great there’s an entire position, the RSDE, around it. Which you only encounter in places you have researchers and want to make their work a reality.
Unfortunately, most researchers are publishing work about improving process, creating new and interesting things, and developing new ways of thinking about the science of computation, instead of what actually makes money, throwing out half-baked apps faster. There’s really no need for a formal understanding of what you’re actually doing and why if you can throw out a few trial-and-error projects and one gains enough traction.
These are pretty interesting talks but they very much feel like just random talks from any conference? Interesting but I have no way to apply them if I wasn't writing a tool for developers.
edit: On another level I am put off by their tendency to describe things such as "teaching undergraduates" or "debugging problems" as algorithms. I understand that you can't really analyze unclear, hazy things without putting them down in clear terms, but something in me doesn't really accept that you _can_ do this without losing so much detail and information as to make your conclusion useless.
They do point out that SWE often think of studies this way:
> Most software developers in industry have never heard of any findings more recent than The Mythical Man-Month: Essays on Software Engineering (which few of them ever actually read) and routinely dismiss studies as “not statistically significant,” even when those studies are carefully done and directly relevant to their work.
Ah yes, the front page featuring the recorded talks kind of hides the real substance. Here's the full collection of papers, by category https://neverworkintheory.org/category/
Awesome, thanks. I jumped into the first one that grabbed my attention: CDD - cognitive driven development (link below, 15min video).
I love the premise but find the recommended results silly. TL;DW: cognitive research shows humans have working memory limited to 7 +/- 2 things. This means that classes should be limited in what they do. So far, I agree. And I think this can be extended, in practice.
However, they back up this idea with a single case study and use a decorator to help count complexity to keep the classes small and state that methods not extending beyond 24 lines of code are ways to do this. This will just lead to cargo culture like "don't comment code."
The decorator is not needed because code should (authoring or in review) be obvious if handling more than 5 to 9 things. The number of lines might be correlated to the number of things a function does, but that doesn't mean cognitive overhead. Like, a 100 line function that maps state abbreviations to state names is fine.
Based on the presented data and mental performative drop off, it is not 7 +/- 2, it is really five + 0 to 2. I hereby dub this the Rule of Five.
A practical example better than what they present in the talk: early returns. Every nested block adds to cognitive load, early returns allow you to discard that load as you continue reading through the function. The cyclomatic complexity reaching over 5 levels means many people will start to fail to keep all the blocks in working human memory. Therefore, early returns are a good implementation of Cognitive Driven Development.
The case study they used seems meh. With a couple decades of experience with high growth start ups becoming "real" companies, I just don't buy their results on decorators and function line length. But they are on to something with limiting code to keep minimal cognitive overload down.
I’ve heard this before, and I can see how some cases (returning from random points in deeply nested loops or something) could cause issues, but in my experience early returns at the start of a function for error handling make things far simpler and more explicit compared with other options.
It really bugs me when it is blindingly obvious a professor doesn’t actually know how cognition works. The actual study of human performance is outside their field, therefore untested or flawed assumptions form the basis of their research.
Good point: as you become an expert in the given system, you can encapsulate or abstract away multiple things into one.
A silly example: when you are learning to dress yourself as a kid, zippers are hard, shoe laces are hard, putting left shoe on left food often gets mixed up. All these little details. When you are older, these details take no mental capacity and you can "get dressed" as a single concept.
There’s also a weird dynamic where this applies to production but not consumption. I can’t remember which, but there’s a social media site that is donation based. Every year they do a fundraising drive. Every year the users flip out they would try to raise money with Everything Going On Right Now.
The founder posted on their blog about how every year has things going on, and they still need to raise money. And the users completely backlash against all forms of monetization.
What I didn’t see anyone mention is the users are still using the site. Apparently using the site is really important. Just not making it, or earning a living while making it.
I was working with a non-profit that taught people around the world how to teach tech skills in their local communities. There was a shift to focusing on recent political events which was a major reason for my leaving. My take was if you personally want to do that, go for it. However, reacting to recent political events was not part of the group's mission. I wanted to spend my limited time on the group's primary mission of education.
First time I hear of this, but one issue is that they've reviewed a pretty overwhelming number of papers, which dilutes their impact.
In general, practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems - programming languages or libraries - is more impactful, but much higher-effort.
> practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems
Isn't that kind of the whole problem? Practitioners ignoring research results because all they want to look at is what other practitioners do? Or are you really suggesting that researchers also become practitioners who develop working systems based on the ideas they discover?
Perhaps I'm misreading, but your statement seems to encapsulate the entire problem the article is highlighting.
Researchers should become practictioners to some extent, if they want their ideas to spread. Example : Donald Knuth and TeX. Racket programming language. etc...
Of course in this publish-or-perish era, this is probably no longer possible.
> the real challenge in software engineering research is not what to do about ChatGPT or whatever else Silicon Valley is gushing about at the moment.
I know this is really just a turn of phrase, but at the rate language models are becoming programming models, it is hard to think of any problem with the lack of hard engineering, science, math, ..., in what we call "Computer Engineering", "Computer Science", "Computer Math", ..., that won't be heavily impacted by model/co-model driven development.
For instance, a major programming language of the near future might develop adaptively as multiple coder models communicate between each other. Reviewing, testing and improving upon each other's designs and co-optimizing the way they communicate code as they do so. Resulting in representations and solutions to existing development problems completely opaque to the humans currently working on those problems.
The rising effect of massive data + compute outdoing slow careful research in yet another area.
Today's models are already being trained across the board on practical/actual and theoretical/research artifacts. No persistent pernicious borders there.
> But why would anyone want that? I agree that this would result, but my takeaway is "and therefore we shouldn't do that".
My guess is that a lot of CEO's already consider the implementation details of their companies' software systems as opaque. So that is essentially a pre-existing market for performative but opaque systems. CEO's. CFO's. Shareholders, etc.
Even in engineering organizations. Look at how Boeing and NASA both demoted critical engineering concerns when external concerns became overwhelmingly more important to their leaders.
Now factor in the models actually doing a good job, checking each other, and generally getting better year to year.
> > any problem with the lack of hard engineering, science, math, …
> To be fair, that does exclude quite a lot of problems.
I had some trouble parsing the comment, but I eventually settled on the reading that it means what it literally says, a problem caused by the lack of engineering in computer engineering, of science in computer science, and of math in "computer math" (a phrase I have never heard), rather than a problem other than hard engineering, science, and math, which I agree with you excludes an awful lot of problems.
Culturally, I find we’re simply not motivated by the same things. Researchers want to publish research. I want to ship working software.
The more researchers I’ve met, the more convinced I become that those two goals are at odds with one another.