Hacker News new | past | comments | ask | show | jobs | submit login
Why is everything so hard in a large organization? (graphthinking.blogspot.com)
492 points by physicsgraph 68 days ago | hide | past | favorite | 292 comments



It seems like a lot of people here have been in an ineffective large organisation and an effective startup. I've worked at an effective large organisation and an ineffective startup, so perhaps I can shed some light on what these kind of processes are for:

* You don't get to be a large organisation without accumulating generations of previous products. Well, unless you're google and can regularly fire your customers without going bust. but part from them, you start to need processes just to track what's going on. (The ineffective startup accumulated previous product attempts too, and is still paying for a lot of pointless infra because no-one still there knows which ones can be switched off).

* At a large scale, it starts to be pretty difficult to maneuver if each dept of 30-50 people has built or contracted all their own infra & services. Some divergence is useful for agility, but a lot of it is just waste and actually slows you down when you want two depts to work together on something - or even prevents useful collaboration. An effective organisation will make the tradeoff consciously.

* When you complete dozens of projects per year, it starts to be possible to invest in researching and rolling out best practices in areas where a startup just has to go with the simple/obvious answer. When my startup employer moved, it was pretty onerous, and there was a bunch of stuff we never found again. The large org had so many offices that it had a full time dept just for moving offices. (They weren't dumb, each move consolidated offices -but they kept buying other companies). When they moved us, it was like magic - we went home on a friday and went to work on a monday in the new office, and everything was set up. Well, we had to set up the lab ourselves, but


> It seems like a lot of people here have been in an ineffective large organisation and an effective startup.

I was at an effective startup that turned into an ineffective large organization before my own eyes. Your three (excellent) points ring very true.

However, the company actually tried to implement each of your three points and recognized their importance. The problem was in the execution.

Old products: These were neither retired nor given appropriate resources, so they instead decayed over time. Empty promises about longevity were made and then broken. Attempts were made to outsource maintenance to countries with the cheapest engineers, which backfired when each update become more buggy than the last. The end result was a lot of customers and early adopters getting burned and a destruction of our reputation in the industry.

Shared infrastructure: The C-levels saw shared infrastructure as an opportunity to reduce costs and speed up development, but they took it too far. At the worst, they wanted to have one gigantic team of back-end engineers, one gigantic team of front-end engineers, and so on, spread across every project across the entire company. Individual engineers were assigned to many projects and each had to deal with multiple managers fighting for their time. End result was that nothing got done and politics reigned supreme as the way to get attention on your project.

Best practices: The company hired “subject matter experts” to enforce best practices. They were expected to be involved in everything across the company. Imagine being hired as the “security best practices owner” and then told that you’re accountable for the security of everything at the whole company. Instead of promoting best practices, they went into full defensive mode and interfered with the shipment of every product and feature that they didn’t have time to work on. We would have products ready to ship and have the best practices experts panicking to halt it because they wouldn’t have time to look at it for months after completion. Getting them involved at the start was a pipe dream.

The overarching issue was that they tried to do big company practices in the move fast and break things startup style. For every well-intentioned move toward being a proper company they managed to screw it up by trying to inject startup-style speed and improvisation.


> I was at an effective startup that turned into an ineffective large organization before my own eyes

Somewhat related, I've been in effective teams that turned into organizations, a couple of times. One of the main issues I've seen is that they're especially prone to survivorship bias, on both accounts of process and people.

You get the usual "this is how we managed to get there, it must have been right, let's do more of it", and people get protective or suspicious if you propose anything else.

There's also a tendency of over-attributing success to the people who happened to be here while the team/product became successful. They tend to be promoted to responsibilities, their voice is over-amplified and their opinion becomes dogma.

The result is that rather than adapting to the turning point both organizationally and architecturally, you end up with dungeon keepers protecting their babies under the cover of "this is our culture". The growth is not painful and rather than re-thinking things from the ground up, we correct by patches and ad-hoc fixes - approvals, reviews, build queues, more planning, backlog sign-offs and such.


> There's also a tendency of over-attributing success to the people who happened to be here while the team/product became successful. They tend to be promoted to responsibilities, their voice is over-amplified and their opinion becomes dogma.

How do you fix this? Assume all success is pure luck and ban dogmatic things like style guides and code reviews and force entire rewrites of sowftware and infrastructure every few years so that best practices change and don't become dogmatic?

One of the allures to getting in early somewhere is that you get to build the foundation and become the thought leader and define to an extent which dogmas shall be adhered to because you have the context to make those decisions. I'm sure it can get overbearing when abused, but I also think it's quite presumptuous to join a mid-stage startup thinking you're going to come in and uproot everything and make things happen your way. If that's what you want go make the sacrifice and join a small place and earn it. And grow the business with it. And then incorporate it into what you can bring to the table as an experienced industry professional so you can be hired as a director somewhere else when you're ready to move and do it at larger scale. IDK...


    The overarching issue was that they tried to do big
    company practices in the move fast and break things startup
    style. For every well-intentioned move toward being a
    proper company they managed to screw it up by trying to
    inject startup-style speed and improvisation.
What would have been a better way of handling the transition?


> What would have been a better way of handling the transition?

The problems all came from the top-down direction. Replacing the CEO with an experienced big company leader could have solved so many issues.

The CEO was good at founding startups and running small teams, but that’s all he really wanted to do. He was trying to run a big company like a small startup and actively fought big-company organization styles.

Companies like Microsoft, Facebook, and Google don’t try to pretend they’re still startups. They embrace the fact that they’re a big company and they make it work. They’re not perfect, but it’s better than a slowly failing big company that can’t accept that change is necessary.


>it’s better than a slowly failing big company that can’t accept that change is necessary.

Or worse, a big company that is in a growth/high profit area that takes in enough money to not fail despite dozens of missteps.

If you do the wrong thing long enough and don't go out of business, then the wrong thing becomes "our way".


One of my big realizations as I've gotten more work experience is just how much momentum matters in business. Brand awareness along with existing customers and relationships is huge.

A big established company can get away with a shocking amount of incompetence for a really long time, while a very well run startup can easily fail.


Scale that up and you can see the same effect with entire countries.


Mostly because annexation of conquered territory is illegal. Without it at least Eastern Congo, possibly all of it, would be part of Rwanda. There are a lot of failed states in the world. They'd fail at defending themselves from a competent military too.


Scale that up and you can see the same effect with entire generations.


So are we using google as an example of what to do or what not to do?


Yes


Agreed. It's sometimes hard to distinguish the things Google does that contributes to its success from the things Google can get away with (waste, etc.) because of its success.


Isn't that a conglomerate in a nutshell? "Our random incompetency averages with our random competency, such that we're able to avoid bankruptcy and remain a going concern longer than smaller, less diversified companies."


> If you do the wrong thing long enough and don't go out of business, then the wrong thing becomes "our way".

You just described most of retail banking and insurance (and I assume, utilities).

It took the 2008-prompted investment-retail bank mergers and ACA admin costs caps to nudge these industries into the 21st century.

The financial side of health care providers is still a nightmare (even though the actual medical side is amazing at continuous improvement).


So much this. There is a very big difference between running a startup and a large / publicly traded company. There is no shame in a founder hiring a good ceo to manage the company after the startup phase. I worked for a company where they essentially grew through grit and insane hours and became established and went public. Then everything started going to hell. CEO was just not prepared to run a proper company and as a result constant production crashes, employee and customer churn. Company survives by constantly diluting and selling more shares now.


But that's also extremely risky, it's really not easy finding good CEOs and not easy to tell if one is going to be good or not (see Apple replacing Jobs in the 80s).


This. The things that get a start-up to IPO are not what make a successful company after IPO. Different leaders are needed, Google succeeded by bringing such a CEO on bord. It is hard to see companies on a good trajectory, from start up to mature public entity, fail due to these purely self inflicted things.


This is very true. A CEO of a startup is used to operating under Dunbar's number. For him to not actually not know everyone is hard to admit.


> Individual engineers were assigned to many projects and each had to deal with multiple managers fighting for their time.

The misunderstanding I saw across several organizations of different sizes was the mistaken rationale that surely, I should be able to take up "just" 25% of your time. When there are 5 other groups, there is nothing at all sure about that statement. Either you are lucky to get 10% or you're riding that employee into burnout and don't understand how they just up and quit one day.


There’s a lot of anti-patterns. Big orgs tend to reward careerism and punish risk. The biggest reason for that is that as you grow, the consequences of failure exponentially increase.

Startup pivots are failures made good. Big org failures means people get fired or arrested.


> Old products: These were neither retired nor given appropriate resources, so they instead decayed over time.

This is the biggest one where I work. Call it technical debt, legacy systems, or aborted migrations - we have multiple generations of products doing the same thing.

Management comes in bright eyed with a desire to migrate to the new system. Not too bad at first, we get some quick wins. Then the 80/20 rule hits, people realise how hard the rest will be and we decide to put it on hold temporarily (i.e. forever).

This is the worst of both worlds, as we now only have the most complicated and ugly applications using the old system. There is nothing easy to train new people on. Over time fewer and fewer people truly understand the old system, work gets slower, and more error prone.

But hey, there's another new system coming along - maybe we can move to that, along with all the fresh easy work we setup in the last few years. Cue xkcd comic about a solution making more problems.

The other side of this is policy and procedure cruft: having so many different systems makes mistakes more likely. Every mistake means an investigation and an item added to a multi-page policy document. You'd never get everything done if you fully adhered to it, but it makes people feel safe.


Sounds like they switched to Matrix management. There must be some studies of when this switch is most effect, and to what degree.


I’ve concluded that anyone using the term “best practices” in earnest doesn’t understand them.


This is GREAT summary!

I work in a large organization, in an industry that is naturally slow. The work (in the industry) is slow because it's critical to deliver stable and reliable solution because large part of population and economy rely on this industry SW/HW. I have heard a lot of complains on how the process is slow, but in many cases, building on top of existing work takes a long time. It takes long time to understand legacy work, to go through older studies and projects, and to find spots to add contributions. Not all of us work on green field projects, but many of us maintain code/work that doesn't look "cool".


Another reason that I observed is that large organizations are extremely afraid of mistakes regardless of expected loss. Such companies appoint gatekeepers, and those gatekeepers are by nature risk averse. The gatekeepers will simply reject any idea by default, unless there is a well-proven precedent. As a result, anything moves slowly.


> Another reason that I observed is that large organizations are extremely afraid of mistakes regardless of expected loss.

Not what I have seen. Big companies with big profits will throw millions away on dumb ideas on the chance they make money, when they do not.


Also true. In the meantime, wanting to use Guava 33 instead of Guava 27 will get rejected immediately. Using Spark instead of AWS Batch is an absolute no no. I guess what qualifies as risk is also highly asymmetrical.


> Well, unless you're google and can regularly fire your customers without going bust

This is only true for external products. When I was at Google it always seemed to me that the majority of engineering effort is focused on the inside, and that the same effect exists there.


My experience as well, although my large org sometimes seems to move to slowly it's loads better than the startups I've been at.

One thing I'm realizing is really useful, and which we're doing more and more, is putting "program managers" in charge of projects, totally separate from "project managers", which really should be called "people managers". With that split we can have a senior engineer who is interested in management work across multiple teams without worrying about the day to day personnel issues. This only works if everyone supports this approach but here it's working well.


This sounds like terminology issues. Classically a project manager manages a project (a think with a start and an end, reasonably well defined) and a programme manager manages a set of related projects. People managers depending on intention sound like either engineering team leads, engineering heads of, or line managers.


One crucial difference is around authority. Traditional project managers can generally exercise authority over the people doing the work to get what they want.

With a program manager/people manager split, it might be the case that the person leading the project has to truly lead, because the actual authority sits with the people manager. May or may not be a better structure.


I don't find that to much of an issue - in fact its alot easier. having a conversation with someone about whether this is the best project in the company for them to be working on rather than whether they should be there at all.

maybe the first conversation leads to the second, but its not me :)


> Traditional project managers can generally exercise authority over the people doing the work to get what they want

I don't think this is that different; people have more static line managers, and are assigned to projects, which project managers then govern.


This sounds a lot like the matrix team structure. Can program managers assign work to engineers? And if so, how do you deal with project and program managers giving conflicting tasks?


Yes, the program managers can assign work and they tend to work closely with the involved people managers to balance workload.


I hate matrix management so much.

It's gimmicky and a pain in the arse. I know management does do a job, but it's hard to justify needing two instead of one.

My view I that matrix style management is a solution to a communication problem you haven't figured out.

Instead of fixing the communication throughput problem, your adding more bandwidth and scaling out the number of nodes in hopes it magically fixes the problem.

And it sort of does fix the communication problems, that's why people like it. The trade off is that as an IC its arese to have to jerk two cocks at the same time.


"At a large scale, it starts to be pretty difficult to maneuver if each dept of 30-50 people has built or contracted all their own infra & services."

This is one of the biggest risks I've seen, for both large organizations and small ones.

Large organizations are built of small organizations, and a bunch of effective small organizations will each be moving and shaking, and building or buying things that become part of their infrastructure. Then one day you wake up and discover that your overall organization is paying for 27 different solutions to the same problem.

Then, any attempt to collapse that down to something reasonable results in zero productivity for quite a while and whole lot of pissed off people.


The question is, is it really worth it to collapse it down?


That depends on so many things.

One of the things I've seen cripple large organizations is waiting for the common infrastructure / services becoming mature enough to be used in production.

Teams are kept in a holding pattern, sometimes for years, while waiting for the teams to deliver their common infrastructure / services solutions, and they're actively discouraged from building their own solutions, because the enterprise-wide solution is "just around the corner".


Internal FUD?


It depends, but generally yes. Collapsing to 1 may or may not be a good thing, but you probably want less than 3-5 different DBMS, CMS, Cloud providers, printing services, source control systems, office supply/equipment, etc.


And suddenly I'm reminded of the web app I put together that needed three DB drivers...


At some level, absolutely. Your organizations aren't usually totally independent units - they need to interoperate with other organizations.

To the extent that the spaghetti of infrastructure prevents effective interop, that's a problem.

And as the need for interop grows between two orgs the cost of the various hacks, shims, and other cheap adaptations, escalates, until unification of infrastructure starts looking awwwwfully tempting.

Of course, unification comes with its own set of problems.


> Some divergence is useful for agility, but a lot of it is just waste and actually slows you down when you want two depts to work together on something - or even prevents useful collaboration. An effective organisation will make the tradeoff consciously.

Often an effective strategy for this to purposefully diverge and converge. Allow a new product to strike it out on its own, and if it survives follow back up and homogenize the infra so that maintenance is cheaper.

Takes a lot of labor, but that’s usually something large organizations have in spades.


This is a great summary! Large orgs are designed to move slow, it's a necessity of calculated moves. For example, single policy, product or process changes can affect thousands of people and many millions of dollars. These things take time as the ripple effects are far and wide.

This is not to say that innovation isn't possible. In my experience, the way to forge ahead in large orgs is to achieve progress by example, advocacy and influence.


From what I hear, even Google is mired by years of product, organisational and tooling debt.


That would make sense as to why they kill of products, in attempt to reign this in. In reality, killing products is still pretty uncommon even for Google.

Just today, I logged into Google Calendar on my iPhone and the mobile UI looked like it hadn't changed since 2008 and it was missing a ton of features.


The mobile web UI for calendar has indeed been abandonware for about 5 years and even before that it was starting to look pretty crusty. I think largely because nobody cares and everyone on mobile uses the app. The mobile web interface of Gmail hasn't changed in years either but it is still maintained and also wonderfully good.


Yes, and it actively manages it. Occasionally, one of the potential outcomes of this management becomes visible to the outside world, when it sunsets a product. Other times, lack of investment in this management also becomes visible, when a product is clearly languishing.

As in any other company, headcount is always limited, and some products are funded by nothing but the willpower of particular passionate engineers/managers, which results in a very low lottery bus factor.


Mire is a good word to describe what is going through my mind when I see their recent search results.


What would you say are the drivers that enable effectiveness in an org, regardless of its size?


Just off the top of my head:

An effective responsibility culture and structure. Those who are responsible for something must have the means to effect it. There must be a means of agreeing who is responsible for something and an effective way to find who is responsible when someone is blocked. Things should not languish because no-one is responsible, nor be trampled in because everyone wants a piece. (This is culture as well as structure, because everyone needs to internalise what a good responsibility structure looks like so that they negotiate it into place in a distributed way, rather than having it imposed top down.)

Prioritise making intelligent decisions over raw speed. In my effective employer, if your project was late, you were required to take a step back and really work out when it was realistically deliverable and make sure you genuinely understood what was now needed - better to say ONCE 'it needs another 3 months' than 'just a another 2 weeks' 8 times. (The ineffective company had a 5 day horizon and would burn everyone out constantly trying to deliver next week) Related to this is that profligate agility can work against intelligence. The effective org could afford what would seem like a huge amount of process, but it actually took less of our time than the agile process in the ineffective org because we didn't need to change our minds as often.

Making effective use of staff's skills. The means everything from: Leaders need to recognise what they aren't expert in, not having a destructive people culture, giving people larger pieces of work that they can master rather than splitting everything into a million fragments, allowing for development rather than people getting type-cast.


  >  profligate agility can work against intelligence
thank you for putting into words something i've been inuiting for a while now... i couldn't put a finger on what was going at some workplaces i've been in, but this really resonates with me

it seems in our chase for "agility" above all else, we are sacrificing our intelligence along the way...


two keys: trust and (efficient) communication. these may or may not be sufficient for a given org, but they’re certainly necessary ingredients. they lead to highly effective teams via autonomy and intrinsic motivation (teams that achieve desirable outcomes efficiently without oversight).

these two factors are extraordinarily hard to cultivate consistently, especially during high growth. this is incidentally why management is so hard, and so often “bad”, because managers are often encouraged to focus on metrics, like financials, rather than cultivating trust and fostering effective communication.

kanban (in japanese manufacturing) is a quintessential example of a management system designed to foster trust (no-fault problem-solving orientation) and communication (simple, immediate, and well-known signaling).

edit: should also have mentioned kaizen as a companion to kanban in this context.


I will be curious to see how founder-led companies operate as they get large. Mainly companies like AirBnB, Cloudflare & Coinbase.

I don't think it's impossible for a company to get large and still be somewhat effective.


Can you give an example (non-code related, if you have one) of the first bullet?


Documentation. When you're working on a new build, everything is frequently just in your head or those of your teammates, and that isn't a problem initially. And if you have one main product, you don't need to spend a lot of time on documenting which resources were for what. When you have a bunch of old ones, which you may not actually often spend much time on, it's much more important to be able to figure out why that old thing is running, exactly what was shipped to customers when (in the case of physical product). Otherwise you risk having to do archaeology every time you turn round. Or just not having the information when some customers says that X has stopped working on a device that you shipped 5 years ago but guaranteed for 10.


> google and can regularly fire your customers

They don't fire the customers (advertisers) they fire the users.


It's hard because it's meant to be hard.

Once you have a large org that's making money, you don't really know exactly why it makes money. Sure, there's annual filings and there's still the elevator pitch of what the org does. But the actual how of how it works is difficult to describe in causal terms: if team X or Y was not there, the business would do better/worse? How does this team interact with that? It's really a bunch of informal relationships between different people, and both the people and the relationships change over time.

So what you want to do when you know the whole sorta-works and you want to keep it that way is you pour glue all over it. You try to formalize processes, you give people titles and you make hierarchies. That way you attempt to stop the firm from inadvertently slipping into dysfunction.

The side effect is of course that everyone who works there can see ways to improve things, but they can't see a way to get those improvements implemented.


It's like the complaints about Wikipedia you read on Hacker News.

Someone smart who knows something wants to add it and gets tangled up in "useless" beaurocracy. Why can't they just let me do it!?

Because if they did just let people do stuff, then lots of other people you consider idiots would have already done things you consider stupid and or bad.

Basically everyone wants rules for the idiots so they don't ruin things and at the same time to not be included in the list of idiots.

That would sound harsh though, so mostly people just moan about rules, rather than them not being exempted, but I don't think anyone genuinely wants the rules to be dropped for everyone.


The main criticism of wikipedia here afaik is of "deletionism", and I don't think that framing fits for that - most people are completely happy if other people also get to make articles about their pet interests.


Wikipedia doesn't have an idiot problem. It has a smart people with an agenda problem.


It also has an idiot problem. The idiots are there, and you will occasionally run into them. I've found pages in the course of natural browsing which were suffering from both obvious and very subtle vandalism.


The solution for that is review by a smart person who is empowered to consider the context and use her professional judgement, not a static set of criteria.


Unfortunately, a 'smart person' in the employ of one industry might not agree with a 'smart person' in the employ of another industry.

For example, consider two smart Wikipedia editors working on the climate change and global warming pages... one from the fossil fuel sector, one from the renewable energy sector.


The sensibilities of your gatekeepers are your organization’s culture. If you don’t like them, you have a deeper problem which rules aren’t going to fix.


Precisely, rules are implemented by people, and people are often idiots.


1) Such smart person might not even exist

2) If she exists, and is as smart as she should be, she will realize quite quickly that her power can be used for personal gain and she will become corrupted. Power corrupts, etc. and you can't really entertain any kind of organizational solution without considering game theory of the situation.


I want no idiots, not rules.


Idiots is a bad term for this. Novices need rules before they are competent. After a certain level of expertise you start breaking the rules, develop your own specialized style, because you know when to break them and how to make up new ones. And we're all novices in some contexts, very rarely anything above that.

The problem is not rules/"idiots"/novices. The problem is when people who make/enforce the rules are not fit to do so, cling to the rules and apply them with sweeping generality. The people who create those problems are typically incompetent but eager or even zealous at the same time.

They are resistant to learning, addicted to power and/or afraid of change and certainly afraid of appearing incompetent. So they never get past that level where they start to recognize when rules need to change or need to get broken.


It's not just novices. Humans are just error prone and make mistakes. Processes to help prevent those mistakes from making material impact on the business are critical to making it sustainable. Honestly though, these processes, if done well can improve velocity of the organization. For example having a comprehensive set of precommit tests may enable your team to continuously ship from head every day. An alternative process to achieve the same effect may only allow shipping once every 3 months. This sort of thing applies to non-technical processes as well.


It’s not just errors. There is the effect of bounded rationality: The organization is too complex to consider everything, so intelligent people make decisions which have unintended bad effects.


I'm fucking brilliant, I'm literally the opposite of an idiot. I swear.

Ask me to navigate a bus station in a provincial Chinese city and I'm a complete fucking moron though. Ask me how I know.

Point being, there's no such thing as idiocy without context.


We are all somewhere on the idiot scale.


Not only that, but we have multiple on differing levels of the idiot scale depending on the topic at hand. The biggest idiots I've encountered are brilliant people outside their knowledge domain.


The other way around I hope ;)


*The biggest idiots I've encountered are brilliant people within their knowledge domain.

Exhibit A: me writing ;p


Don’t worry :) I read it as “The biggest idiots I've encountered are brilliant people (who are [idiots when they are]) outside their knowledge domain.”


  >  we have multiple on differing levels of the idiot scale depending on the topic at hand
or even depending on the time of day, or current stress/work levels

alot of things can affect the idiot scale...


That's very narrow way of thinking ... I believe myself to be idiot in multiple different ways which can have multiple dimensions with multiple different scales.


And the real problem is that it is only a really small set of things one person can have even mediocre competence during one lifetime, whereas the space of things where you can be completely idiot is infinite. So, as a very good approximation, we all are complete idiots.


The trouble is that even if everyone has great ideas, implementing all of them at once is idiotic.

You need some brakes on the organization to assure that the rate of change is manageable and that marginal ideas are implemented more slowly if at all.


If it's better to slow down the good and the bad ideas, red tape, organizational brakes, and a "default nope" culture are great.

It might also be better to have a much faster pace and more freewheeling "default don't care; if you can make it work, it'll stay" which also then requires a relatively ruthless culling mechanism (even if slightly unfair) for the things that don't work.


What complaints about Wikipedia are you referring to? I post a lot on HN and I also edit Wikipedia. I don't recall any complaints of that kind.


I block Wikipedia from all search results, mainly because so many links to sources are broken or paywalled. This is kind of ridiculous for an outfit that claims all information must be properly sourced and no 'original research' is allowed.

It would seem fairly easy for Wikipedia to examine its own pages for broken links and flag any such link (and its related content) as unreliable. Wonder what that would look like...


I often talk about this when I do consulting work: the difference between succeeding accidentally and succeeding deliberately.

(If they are not succeeding, they can't afford to hire a consultant, of course)

It's my firm opinion that there is a U shaped curve of chaos - new companies do not yet know what they should be doing to make money, and large companies are inherently too complex for any one person (or subset of people) to actually understand what is happening.

This is why I prefer to work with and for medium sized companies - companies that can have goals and metrics that lead to success, but are not yet so large that they are inherently unknowable.


I find medium sized companies more frustrating. IMO, they are trying to become large companies and do things that are status quo expectations of large orgs. Yet, without realizing the ways in which those things have constrained you from moving as quickly as they are or did when smaller. It’s like adding a bunch of rigidity to the process and then asking why the flexibility has been reduced.

Feedback loops of communication is what I view as the biggest inefficiency. If you CC 10 people on an email, you’ll still be chasing 3 of them for a response next week.


>This is why I prefer to work with and for medium sized companies

Out of curiosity, how do you define a medium sized company? How many people, how much revenue, etc? The way people define this varies widely, so just trying to understand how you define it.


More people than it is feasible to get into one meeting at a time :) I usually am thinking of 20+ to <200 people - probably my sweet spot is around 50-75, but it depends a lot on the people at the company. Basically the size where they start wanting to have things figured out, instead of just being amazed they work at all.

In startup terms? Private companies, usually post Series A (usually B or C) but not yet private equity unicorn rounds. In market cap terms it's 100m+ to <2b or ARR ~2m to ~25m for a sass-style company. still a "small cap" in the grand scheme of things.


Thanks! I think with all the well known tech behemoths (50K+ employees), it's easy to lose perspective and assume that a company like Databricks (2K+ employees) or Stripe (4K+ employees) is a medium sized company in comparison. I appreciate the clarification.


Yeah I mean obviously they are compared to foogbookazonpple, but I think they're past the point where you can individually go in and make substantive change, really.


I honestly wonder if it IS possible to know roughly how a company works and be able to make tactical improvements throughout. This is dismissed as micromanaging, and most people lack the curiosity necessary to accomplish it, but could a CEO/CTO actually have a good enough understanding to make changes throughout an organization and fix the dumb things that every employee knows but doesn’t have the authority to change? I suppose most executives and management spend a ton of time in meetings and not making such tactical decisions, but I do wonder if it is possible.

I’m reminded of SpaceX, who is run by Gwynne Shotwell and Elon Musk. Gwynne keeps the whole business machine running and manages existing programs so well that Elon has the bandwidth to dig down to a fractal level and address a lot of the weird issues & bottlenecks that everyone knows about while leading new programs extremely fast. Or at least, that’s the story (doubtless the CEO meddling has negative effects, too, but overall seems to work at least as well as traditional organizations that big). Also, maybe that can only happen in a business like SpaceX which is filled with a bunch of fantastic workers who are just really driven to make things work at all levels.


This is similar to the premise behind my "software archeology" tool idea. While working as a lowly developer at a medium-large company, I kept finding case after case where badly coded software was doing dumb/wasteful things, but I lacked the authority to fix problems from a systems perspective.

I fantasized about what if there were some kind of internal company web app, a portal or dashboard, where people at the highest levels, if they cared (I am not even if sure they did), could drill down to a "fractal level" (as you called it) and see what the software is actually coded to do, presented in a way that makes sense to a non-programmer/higher-level understanding.

The elevator pitch for this kind of IT product would be: "Software runs your business, and you don't know how it works."

Unfortunately I never could figure out how to actually turn this into a real thing. None of the common approximations of software (modeling) do a very good job of capturing subtle aspects or of being more understandable than the code itself. The one conclusion I did come to is that it would have to be something where low level developers interpret and model the code to digest it for whatever format the portal uses, and not some kind of automated distillation or analysis of the code.


This is a genuinely great idea.

I think it might in fact be impossible but if you ever figure it out you'll win all the marbles.


Agreed this is a great idea. But imo you don't need such high resolution.

Your target customers might end up being other programmers or managers though.


When you get down to it, many apparently "dumb" things are about power and status and political arm wrestling and compromise among departments.


Why does it have to be the CEO? My first association was that Winston Wolfe character from Pulp Fiction. Wouldn´t it be neat if companies had that kind of fixer who you could call for the real hard problems? Like problems that arise from the structure of the organization and cannot be fixed on a micro level. Asking half joking, half serious.


That's who I aspire to be, but my batting average isn't that fabulous, maybe .200, and it's really difficult to sell to people, because you have to catch it right on the cusp of being a recognizable problem.

So people are inherently skeptical, and by the time it becomes obvious that they need some outside heavy hitter, it's too late for that strategy to be effective.

I'd really like to systematize it, to be known as the solution for that kind of problem, but that's a different skillset than actually solving the problem, of course.


i have a bit of a reputation for doing this, and a few personal relationships with business leaders who have tapped me for these kind of projects. in the end its kind of dysfunctional every time because various people in the org own the relevant responsibilties already and you are basically micromanaging them against their will as a prelude to termination or reorganization.

its not like the solutions are usually that hard to figure out, it's always a problem with people and their incentives ultimately.

the fixer role is basically to give confidence to the CEO that disempowering certain people is safe and there's a path out.


I too thought of SpaceX while reading your first paragraph. I've heard Elon Musk described as a "nanomanager".


I bet he plays the “bring me a rock” game with all of his reports.


Really? Is that an argument for “bring me a rock” being a successful strategy?

SpaceX Falcon 9 and Falcon Heavy vs Vulcan and New Glenn and Ariane 6.

SpaceX Starlink vs OneWeb.

SpaceX Dragon Crew vs Boeing Starliner.

SpaceX Starship HLS vs Blue Origin HLS.


What game is this? I'm not familiar with it



This is the moderation fallacy, assuming that the middle is better than the extremes and not just as likely to be the worst of both worlds.

Even if the optimum is somewhere in the middle, the middle region is a spectrum and the part you picked might be worse than the ends. (Picture a sine-wave shaped graph of quality vs size.)


I've heard this effect put into more positive terms: If the ship is on a good course, making it hard to steer also makes it hard to sink.


There are no courses that don't eventually intersect with land, or other ships...


Other ships, who can say. But if you are at about 60 degrees south of the equator and keep going due west (or east, of course) you can keep going forever without ever hitting land. You'll pass just below the southernmost point of Chile and just above the nothernmost parts of Antartica.


> if you are at about 60 degrees south of the equator and keep going due west (or east, of course) you can keep going forever without ever hitting land.

No, this requires you to constantly steer left (or right, of course). So you are not traveling on a straight line.


Same loxodrome though.


I wasn't sure if there literally wasn't a great circle route that didn't include land in it. I am moderately surprised to find out there isn't[0]. But I would be more surprised if there wasn't some line of latitude you could follow around antartica that never hits land.

[0]https://www.technologyreview.com/2018/04/30/143150/computer-...


There’s a quote attributed to adam smith “there’s a lot of ruin in a nation”.

A large enterprise such as Google, GE, or IBM can be poorly managed for decades before being economically forced to change.

I observed a team making 25 MM per engineer in a large company that made one change per year. There are products at google that require 400+ pages of documentation/proof to change.


If the analogy is continued, it might also be harder to steer away from an iceberg.


Which is probably an intended part of the analogy.

On the other hand in these orgs you have people crying "iceberg!" all the time.


But they are so big and strong that there would be no danger from an iceber... oh


So many examples of this. Blockbuster vs NetFlix seems like one.


Blockbuster v Netflix is odd though, in that Blockbuster actually did pivot to the DVD by mail subscriptions really well.

Well at least operationally it was a good product, and imo better than Netflix's, but I have no idea if helped or hurt them financially.

But Blockbuster's established sources for content/disks should have been an advantage against the stories of Netflix having to go buy retail copies of movies in cases of uncooperative distributors.

It was just Phase 2 and moving to streaming where they fell so far behind. (Plus Redbox's uprising didn't help, which is a separate failure to pivot.)


That wasn't necessarily inability to steer. Blockbuster had an opportunity to own Netflix at one point and said no. Just like Docker was given a chance to steward Kubernetes and decided to compete against it instead.


Until something big happens (Kodak?)

But failing can be good and bad.


I keep this link sitting around and enjoy dusting it off every couple of months or so

https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...


>Once you have a large org that's making money, you don't really know exactly why it makes money. Sure, there's annual filings and there's still the elevator pitch of what the org does. But the actual how of how it works is difficult to describe in causal terms: if team X or Y was not there, the business would do better/worse? How does this team interact with that? It's really a bunch of informal relationships between different people, and both the people and the relationships change over time.

Sociologists have quantified these sorts of relationships almost a hundred years ago with graphical modelling.


> That way you attempt to stop the firm from inadvertently slipping into dysfunction.

And thereby deliberately push it into dysfunction. At least when seen by an impartial observer.


The concerns of a business and the concerns of its engineering organization don't always align. On the one hand, you're maximizing for net income. On the other, you'd like to minimize the friction to make changes. The business organization needs to see value in allowing easy change and be convinced that the engineering org isn't going to quickly and easily kill their golden goose with that ability.


Killing their golden goose is kind of the point though. Obviously the people whose job is literally dealing with the bureaucracy are not going to like anyone making any moves towards making things more efficient.


Then you get people nagging about how big companies are bad at security with leaks.

It is easy to nag when they never had to secure anything or operate at scale of 100's of employees and 1000's of users.

It is just hard even if you have good practices in place, because checking that everything is in place and training new people to follow rules is a lot of work.


Big mistake in the article to claim that a single engineer could build Amazon, Facebook, or Google. Amazon is the most egregious.

> . A software developer could write a simple e-commerce website in a few days; Amazon employs 1,335,000 people [source].

Those 1.4 million people are not coding the e-commerce app but making e-commerce happen as package handlers, drivers, finance teams, etc. on top of that Amazon has many non e-commerce related projects: building hardware (Kindles, Alexa, etc) , AWS and support, and on and on.


That's what I thought, too. Amazon is a bad example, maybe for Twitter, you could make the case. But not for the reasons the article mentions: Sure, there's some overhead due to communication and misaligned incentives, but most of the work goes into the hidden complexity that even the simplest tasks contain when rolled out to millions of people. And I'm not only talking about technical complexity - I'd be interested how many people Twitter employs just for checking reported tweets.


> ...that even the simplest tasks contain when rolled out to millions of people.

Wasn't this precisely the focus of the article?


> Wasn't this precisely the focus of the article?

"A software developer could write a proof-of-concept search engine in a few days; Alphabet employs 130,000 people "

This suggests that those 130,000 people are all software developers, and that they all work on the search engine. I think clarifying that ambiguity is a pretty appropriate response.

The reason Alphabet employees 130K people isn't because "running a search engine at scale" requires it, it's because they're in 30 other lines of business.


Maybe I misread it, but I understood that the article said that the simplest tasks get complex when dozens of people are involved in solving it. I'm not saying that's wrong, of course working with a large team creates overhead. But giving the headcount of amazon and google as evidence for this is a bit misleading.


I wouldn't really listen to the case on either FB or Twitter unless the person arguing the point is familiar with using the products as an advertiser. That is the core product, after all.


>Big mistake in the article to claim that a single engineer could build Amazon, Facebook, or Google.

You misread it. The author is not equating them; he's contrasting them. Simplicity-of-proof-of-concept vs complexity of real-world-AmaFaceGoog.


The author puts as a reason some internal objectives of the corporation not visible to an observer behind the simple idea of their website. But contrast focused on them is wrong, it is comparison between apples and asteroids. The complexity of a corporation is created from external factors: E.g. Google or Facebook are not a search engine or social network (both even in current scale would require significantly less people to implement and maintain). They are advertisement businesses compliant with local regulations, serving different audiences and optimizing variety of financial KPIs set by investors. Just naming and understanding all those factors is beyond intellectual abilities of a single person and this is why big organizations are really difficult: it is not due to complexity of relationships between people or misalignment (they are just derivatives), but due to complexity of the problems such organizations are dealing with.


I agree with you on amazon, but I think the author does have a point.

Take FB as an example. FB have 60k employees, 12X the 2012 IPO number. It also costs 12X to run facebook today. They don't do that much more today than in 2012. They mostly just do things a more complicated way.

IMO, the reason for that is actually revenue. Beyond a point, there's no market discipline reason for mitigating costs. FB is profitable enough with its >30% margins. Investors don't want them to cut the salary bill. They want FB to hire people, invest, do more stuff & grow.

If FB revenues had plateaued around $10bn, I'm pretty sure FB would still exist. FB would have 90% fewer employees and users would not experience much difference.

Once you have all these extra people, and basically the same set of tasks at hand... sluggish ways of doing things and inefficiency are basically guaranteed. If 500 devs are now working on an area that 50 people previously ran, it's going to look like ineffectiveness at the ground level.

A lot of the software industry is like that. Half the industry, like most internal dev orgs are not very capable or motivated, and things look sluggish at ground level. The other half is flooded with cash and great devs, but teams are bigger than actually necessary.


Not a great example. FB’s product surface is easily 10x 2012. If you think FB == news feed, go dig into the app. Marketplace didn’t exist in 2012, for example. Also, IG acquisition was in 2012 post-IPO, WhatsApp and Oculus were after that.


Possibly. I've had this conversation a few times and I find it mostly unresolveable.

There's no useful measure of FB's efficiency, what 10X more FB means... so we can't externally measure the claim. Whether or not someone agrees with me or you probably relates to what we think about work & efficiency broadly.


I think there are two parts, one which is resolvable and one which isn't.

FB doesn't "do that much more today than in 2012" - completely resolvable. I hear statements like this all the time. FB/Google/Amazon don't do more, they just do it bigger, or they just hire engs to keep them from the competition, etc. FB had 3000 employees total at IPO. I can easily name 10 brand new things launched/acquired since IPO that each have a massive user base and at least 1000 engineers on them now. I'm sure people could do the same for Google, Amazon, Apple.

The unresolvable part - are the 60K employees at FB today individually doing as much as the 3K employees were at IPO? Personally, I doubt it. The luxury of being big means you can try more ambitious stuff that may never launch, overbuild on infra and process, etc. The flipside is that niche features you've never heard of can drive millions in revenue bc of the scale of the userbase and business customers.


I disagree... "10 brand new things launched/acquired" is not a measurable quantity. You could tally up the employees that worked at these companies pre acquisition... but to total is a lot bigger.

There's also the fact that in software, you have something once you build it. Software is cumulative, so you'd expect it to grow as long as someone is working on it.

There are various points of evidence, but they're all arguable.


I think you just want to argue. Using IG as an example, if you think a completely separate app with a billion monthly active users isn’t doing anything more than FB did at IPO, you’re never going to be convinced. If you think selling the most popular VR headset and building the infra for a metaverse isn’t more than FB was doing at IPO, you’re never going to be convinced.


I mean there's also like the super not obvious stuff that comes with scale. Facebook wanted to hire me to join their team that does supply chain optimization to ensure that computers get to their datacenters fast enough. That's like a reasonable thing for them to have but it's totally removed from the product.


Good point, but is "supply chain optimization to ensure that computers get to their datacenters fast" a task that is not required at 2012 FB-scale, or is it an optimisation that FB just didn't have the scale for in 2012.

The former implies inefficiencies of scale. The latter implies efficiencies. I think both exist.


And yet Marketplace has amazingly poor UX and only has traction because it fills a niche of classifieds with RL identity.

To the larger points here, this illustrates another major failing of large organizations that achieve de facto monopoly status due to network effect: arrogance towards the users that made them so large in the first place.


> Take FB as an example. FB have 60k employees, 12X the 2012 IPO number. It also costs 12X to run facebook today. They don't do that much more today than in 2012.

They do.

In 2012, their advertising revenue was $5B. In 2020, it was $86B. They are "doing" 17x more now.

I don't mean that from your perspective of revenue leading to wasteful spending, but from an advertising product perspective. If you're making $86B in advertising revenue, then every junior data scientist whose entire job might be to optimise ad click-throughs for cat food in Mongolia by 3% is a net win for the company. If they are getting better at cat food ads, they are doing more.


More revenue, yes.

That was my point. Revenue rises, and everything else grows with it regardless of need.

Whether or not the marginal junior data scientist is a net win for the company.... I suspect not at a company like FB. Efficiency is hard. Factories are efficient because margins are slim and capital costs are high. FB is the opposite. There's no external disciplinarian forcing managers to find ways of minimizing junior data scientist headcount.

FB profit margin this year is 37%. When margins are that high, there's not much incentive to cut fat. If profit margin is 3%, a 1% efficiency gain is a major win. At 37%... different game.


My question: what is new about what FB is doing currently? Are they attempting to maintain their current influence and social network? Stem the tide of anti social media talk? Encourage controversial misinformation? What do they need the employees for? Is it to improve their image for investors?


>> what is new about what FB is doing currently?

You'd find the same sort of answers at FB that you'd find at citibank. They have regulatory challenges, content moderation challenges. Corporate structure challenges left over from acquisitions or whatnot.

I'm sure there are a ton of items on the todo/doing/done lists. There are "new and exciting" projects, oculus VR, whatever is happening with the marketplace, etc.

Same thing at citibank. From the perspective of a manager there, I imagine it looks like they're doing a ton of stuff all the time.


> Big mistake in the article to claim that a single engineer could build Amazon, Facebook, or Google.

The article does not claim that. You misread it. Right there, in the quote you included:

> A software developer could write a simple e-commerce website in a few days; Amazon employs 1,335,000 people

It says "simple" right in the middle. Then it immediately says afterward that Amazon employs 1.3M people, which clearly indicates that the author does not consider Amazon to be "simple" (and this holds regardless of whether the 1.3M people are factually related to e-commerce), and that he does not expect his readers to, either.


The author is trying to use Amazon’s workforce size to support his argument that

“given their organization's internal objectives, they are forced to build huge workforces to accomplish what seems like "simple" tasks.”

“internal objectives” is not why Amazon needs 1.4 million people to complete the “simple” task described. And if you redefine internal objectives to include everything else that Amazon does to make money then it’s really saying nothing at all.


This is probably due to engineer's bias. We like to look at something like Amazon and think "I could build that". It's just a shopping site?

A business person would know all of the things you are talking about that make that e-commerce site as successful as it is.

I have stumbled across many existing companies that do things that I have thought about developing but they employ 2 people, have no marketing and no-one knows about them.

That said, I also agree that there is a weird exponential growth of team size with a product that isn't always much bigger. Our SaaS company turns over about 3M/year with a dev team of 6. I was looking at another similar company the other day (granted slightly bigger/more complicated but only 2 main products) and they had about 200 devs in their about page!


I’m not going to say it’s the same, but I have a bit of a story here.

My first gig was at a small software retailer, straight out of high school and balancing study at uni. Picture a GameStop competitor but selling home/commercial software and licensing as well as games.

The owner’s approach was to employ a series of teenagers who saw this as an opportunity to get that elusive “foot in the door” first job experience despite being paid less than half minimum wage for the hours we were actually paid for (hard not to envy today’s junior Devs walking into jobs out of uni today, yet alone the eye watering salaries, but then again there’s something to be said with romanticised rear vision for working your way up the ladder during that golden age of the early modern web).

Two of us, while balancing sales and service to customers, and keeping the store running (receiving and unpacking stock, boxing up and posting customer orders, phone calls, stocktake, cleaning, directing the occasional casual staff), without any formal education, with tools that don’t match today (no especially useful libraries or tools, eg pre-jQuery, Ajax just catching on and feeling like magic, plain PHP, .NET 2.0 without an ORM etc.), built and maintained a website that in my view rivalled Amazon’s UX at the time (we developed some novel ways to group and present licensing and editions that surpassed Amazon in this space).

On top of this we developed and maintained a bespoke WinForms POS/inventory/accounting system that handled all sales and ordering etc., supporting tools to handle pricing/forecasting, automation scripts to keep customers updated etc., and a series of integrations with suppliers to pull near-live stock availability and pricing for perhaps 40k SKUs from around 80 suppliers (none of whom had APIs so we were screen scraping custom portals, parsing the odd PDF price list, or best case pulling CSV email attachments).

The net effect was with minimal operational effort we could effectively compete with far larger businesses and provide a better experience.

This included the ability, with near zero-touch, to surface a comprehensive catalogue of stock while maintaining minimal inventory, automatically take customer orders, place corresponding supplier orders based on the best price at that point in time (rationalising and aligning across price lists, with consideration of preferential contract terms or shipping timelines), publish near-live pricing and ETA changes, administer a customer loyalty program, and provide automatic updates to customers.

We just got stuff done because we didn’t know any better.

I know this is several scales lower in complexity than Amazon, eg fraud detection, volume of catalog and sales, reseller accounts etc. but again we were not 1000s of Stanford and MIT educated engineers.

Incidentally this has shaped my view of the undervalued power of having bespoke software supporting your core business operations, despite also driving home the risks and perils of reinventing the wheel.


My first software job was also right out of high school in June of 2006.

I was recruited by a small consulting firm that had a specific project in mind that aligned with a fake project I had on a resume. My resume was fake because there was a BCIS project in high school to create a resume and post it, but I forgot to take it down, but I didn't let them know that either.

I bluffed my way passed two interviews, was given a $30k salary and benefits. Within a month I was running the team, and within 6 months I had shipped my first product (an ocean shipping contract management system). This is after they had been trying to write this software for over 6 years with no success.

The company then decided that I was worth more than the rest of the team, took me into a meeting told me that I would be getting more responsibilities, and asked if I cared if they laid off the rest of the team. I agreed with them thinking that by laying them off I'd get a big raise... I did not, I just got longer hours and even some weekend work!

So after a month of that exploitation, I quit to focus on a couple of side projects that were almost making enough money for me since I still lived with my parents.

First jobs (especially in "industry")as an undereducated 18 year old kid are crazy exploitative. I didn't know the difference between salary and hourly, or the fact that salary means you work as long as we want you for the same pay as if you didn't work at all.


Sounds like a startup but then what happens? You start getting success so now you have a new salesperson but they need lots of help because they aren't as techie as you. You might even start needing 24 hour support and a dedicated returns line.

You then realise that you are the main part of the success so you now want your own office and a team to manage. Of course, they are not as sharp as you so you have lost your skill in return for trying to train and manage them - some don't even really want to be there.

Then the boss decides to get an investor to help scale the business - you need the support people on-board before you can start charging for support - and that person is now putting you under pressure because you are not updating the website quickly enough, there are bugs that people are complaining about etc.

It's a fairly classic story. You either stick, fold or go FANG!


> We just got stuff done because we didn’t know any better.

Despite knowing I'm more effective at avoiding major pitfalls, guiding the team in the right direction, building software that lasts.... I miss this feeling, so much.


Well, 13 people worked on Instagram when FB acquired it for a billion. So... productivity varies.


Ronald Coase won the nobel prize in economics for his "transaction costs" solution to the "why do firms even exist" question. A lot of economics is about asking a question the right way.

He started with "if markets are so great and stuff, why do companies run like soviets & chiefdoms in practice? Why doesn't the product team just contract a UI designer, a testing team and such. Hi solution was "transaction costs." IE, it's too hard to find a designer every time you need a button.

Economics and social science generally is prone to a "defend my model" problem. Once you have a model, there's a tendency to see everything in terms of that model. A Coasian sees everything in transaction costs terms. Coasians ran around

This author seems to see everything in incentive & game theory terms. Management theory often prefers group psychology explanations like Dunbar's number.

IMO, you should probably keep all in mind at once, and remember that none are entirely true. Even if you see the problem as "Prisoners Dilemma," the solution may not be "breaking the prisoner's dilemma."

Human cooperation in groups to achieve aims is basically the secret sauce of human civilization, culture & what makes us different from other animals. If it could be predictably engineered, a lot of problems would have been solved already.

Most, seemingly common sense solutions fail miserably. Trying to break prisoners' dilemmas by nailing all responsibility to someone (a) may be a lot more disruptive and aggressive than you imagined and (b) will often result in elaborate CYA responsibility avoidance. See enterprise consulting for a snapshot of a mature system described this way.

One way or another, this problem has occured to many people over the years. It's not an easy one. Perhaps, it's the hard question of social science.


> Most seemingly common sense solutions fail miserably. Trying to break prisoners' dilemmas by nailing all responsibility to someone (a) may be a lot more disruptive and aggressive than you imagined and (b) will often result in elaborate CYA responsibility avoidance.

Or, that is what the management chain does and this is the only way to scale it.

There is also a perspective on this which is along the lines of Turner's social evolution ideas.

To recap that, bureaucracy is recast as a gene for a successful organization. For Turner, this is in some sense a very literal survival of the fittest, because he is asking the question, “why do modern militaries and governments regress to The State as their model, rather than the other governmental styles through history?” And his answer is that The State, or bureaucracy, is legit a better gene, it propagates itself and it propagates the organism better.

“But, the thing we know about bureaucracies is that they are super inefficient, how can that be the best way to go?!” Well, maybe we need to revisit that. The modern military that is structured like a bureaucracy is vastly vastly larger than the militaries that are not. Is there a forcing function that would drive efficiency way down at these resource scales? And once he's asking that question the answer seems obvious, the same answer as in OP: corruption. In computing terms, a tree with fan-out 5 has 5/4 as many nodes as leaf nodes, with 10 you can drop to 10/9, so you have an 11% - 25% intrinsic overhead. (Substantially more in terms of cost of labor since they also want to be paid more than the front-line employee, which again suggests incentives.) We look at this minimum 20% waste and are shocked by it, but the claim is that the transition to bureaucracy starts to happen precisely when you are moving so many resources that the 2% of bad actors that slip through your vetting can sipon off 5x-10x what they actually need and produce a 10-20% corruption. This combination sets a Dunbar’s number, rather than it being some aspect of human evolutionary psychology.

The cost was built into the game rules at these scales, it stops being waste and starts being opportunity cost, in the economic sense... Put another way, maybe it is not the hard question of social science, but a constraint in which social science necessarily operates?


I'm not sure that I understand the point correctly, but if I do...

One of the point Dawkins' used his original meme concept to demonstrate is that a meme may not be beneficial to the person or culture carrying it. The meme just needs to persist and reproduce. He is the selfish gene guy, after all.

Also I agree that cultures evolve, but I think the analogy to natural evolution by natural selection can be overstretched. This dynamic exists and can even be dominant, but there are other evolutionary dynamic going on.

For myself, my "Big Theory of Bureaucracy" is simply that it accumulates over time. The larger the scale, the less competition and creative destruction happen so bureaucracy can keep growing undisturbed.

You could analogise corporate bureaucracy to just about any traditional culture: the US senate, the catholic church or some small island tribe.

The US senate does this ritual around debt ceilings. There are written and unwritten rules that make no sense from the outside. It's all based on a rule from 100 years ago, and stopped serving its initial purpose generations ago. Meanwhile, the whole process has gained incidental important roles in coalition consolidation & legislation bundling. Those now depend on debt ceilings. It can be resolved either by vote or a ceremony whereby a small metal object is place inside a large building. The whole thing requires a lot of paperwork.

50 years from now "debt ceiling" might refer exclusively to a quaint ritual where a president gives some guy with a special hat a special coin. At this point, it's impossible to tell bureaucracy from ritual. It's just stuff that needs doing in very specific ways for reasons that are not externally logical. They have history, dependencies, tradition, etc.

Maybe bureaucracy is just an inelegant version of that.


I think the evolution of organizations is super interesting stuff. But I would say that a lot of the paradigms that have been true for Millenia have changed rapidly in the last few decades, which is an extremely short period of time in the history of our social evolution. For example:

(1) 100 years ago we didn't have reliable instant communication. If you wanted to send a message across oceans it could take days or weeks. As a result having a "chain of command" was critically important to survival. For example a military commander prior to the 20th century could not make decisions regarding an army thousands of miles away, and thus needed to delegate decision making authority. In 2021 a military commander (Or leader) can instantly make a decision from thousands of miles away. Or perhaps a computer could make those decisions instead.

(2) Data-driven decision making the name of the game in most large corporations. The dinosaurs are still making decisions based on reports every week/month/quarter. The most cutting edge organizations are operating on a different level. For example, if a particular product or service isn't performing at a certain price, you don't need a bureaucracy to optimize it. Automated software could easily A/B test pricing schemes in different regions, and come up with the most optimized conclusions, all without any humans in the loop. Ride-share software and food delivery apps are already ahead of the curve here, and like it or not, this outsources a lot of people in the business of decision making.

Not to go too far down a rabbit hole, but I think that "bureaucracy ... as a gene for a successful organization" may be a gene that soon goes extinct. Computers, data, and software can do bureaucracy much better than humans can. And if the trend continues we may see the entire structure of organizations, leadership, and decision making make a complete shift from what we even expect an organization to be, possibly closer to resembling cybernetic organism rather than a government or traditional corporate leadership structure.


> Human cooperation in groups to achieve aims is basically the secret sauce of human civilization, culture & what makes us different from other animals.

I don’t know… Albert Einstein alone makes us pretty different from other animals.


Albert Einstein wasn't alone. He was engaged with a scientific community, through papers, publications and such. He used maths developed by others. The scientific method actually calls for multiple people, even if only for objective POVs.

Even though he wasn't on a team, that still is a model of communication. We call that model science.

The scientific revolution was largely a revolution in the way scientists were organised. They became connected to one another via publication, critical dialogue and such. The scientific method is, among other things a model of cooperation. How to replicate, publish, review and convince one another of things.


Sure, I buy that.

But if you think chimps would produce the general theory of relativity if we could just teach them to publish their thoughts, then… I don’t buy that.


It’s less the ability to publish their thoughts, and more the ability to understand the published thoughts of others. I do think that almost any creature with enough presence of mind to do that could get to the theory of general relativity in a couple million years. Being able to pass intricate knowledge to future generations is really what defines humans in comparison to chimps.

Although I suppose it’s hard to say exactly what the limited level of intelligence needed to understand a few paragraphs of writing is.


The Coasians sentence seems to have run away after "Coasians ran around" ! Or did you mean "ran aground" ?


..ran around explaining how everything could be understood as transaction costs at work, in much the same way neoclassicals thought their marginal price dynamics explained everything.

oops


The article actually jumps right to the three major points that I see, rephrased to: (1) misaligned priorities, (2) policy flux, and (3) attrition.

The most damning trend I have witnessed first hand is that directors have extreme agency and are most of the time not incentivized to help other directors, especially if they’re under a different VP or SVP. The high-level leadership are too detached from the details (as they should be — but this is the thing that is most different in large orgs — it becomes more and more difficult to know everything, eventually becomes impossible, and then even knowing “enough” becomes impossible) to make centralized decisions so you’re left with one organization at a standstill with another, which causes attrition in the lower ranks of people who just want to get stuff done. That attrition results directly in knowledge loss and overall reduced productivity, which hampers most teams from breaking out of their set of problems. And it all just gets worse with time.

Large companies have way more guardrails and standards than smaller companies and they also change at a greater rate. Since these policies so rarely help improve or ship features, it’s just an ever-increasing source of productivity loss from the perspective of someone who wants to get stuff done. When internal politics are complained about, this is typically the root cause.

Hiring and retaining talent becomes harder as all companies grow and previous top talents becomes disenfranchised over time. Average talent hits an inflection point and then begins a descent. This just adds fuel to the fire, but I believe things would still become extremely difficult without this trait as well.

All of this becomes only more difficult to deal with over time as the company increases in size and age.

Nepotism, cronyism, etc.. all factor in as well, but those issues also affect small organizations and their impact there is even higher.


> Since these policies so rarely help improve or ship features, it’s just an ever-increasing source of productivity loss from the perspective of someone who wants to get stuff done. When internal politics are complained about, this is typically the root cause.

"Why do I have to fill in this mountain of paperwork, in triplicate, just to get permission to do this basic activity that is an essential part of my job, and hence the permission being granted is a foregone conclusion anyway?"

"Eleven years ago a guy named Bob tried something similar and screwed up terribly."

"But not the same?"

"No, but similar. So anyway now we have a 'process' to make sure nothing like that will ever happen again."

"Is Bob liable to make the same mistake?"

"Oh no, he's learned his lesson. Also, he was fired a year later, so he's not around anyway."

"So, has anyone made the same mistake since?"

"No."

"Do you think I'll make the same mistake?"

"I don't believe so, no."

"So why do I have to fill out the paperwork to avoid making a mistake we all know I won't make, especially now that I've been made aware of it."

"Just in case."

"Just in case, what?"

"Well, how do we know that you won't make mistakes? We have to have proof, some sort of written evidence."

"But you can't prove ahead of time that people won't make mistakes! People don't make mistakes on purpose, so no amount of paperwork will ever stop mistakes happening."

"You're just being argumentative now. You're not special, everyone has to do the paperwork."

"Why everyone? You just said only Bob ever made the mistake!"

"HR told us that we can't make rules just for Bob, so everyone has to follow the same rules. That's fair."

"But you don't need the paperwork any more, you got rid of Bob!"

"Yes, but the process has been established, it would take a lot of time and effort -- and paperwork -- to change it now."

"It's less than making everyone jump through unnecessary hoops just to do their jobs!"

"Maybe, but I'm very busy. Back-to-back meetings to discuss important matters, like how several staff are behind on their paperwork and may need to be disciplined."

etc, etc, etc...

I've had conversations like that -- nearly verbatim -- at several organisations in contexts such as:

"Why does it take four weeks to add a firewall rule?"

and:

"Why is the minimum permitted change turnaround time two months?"

and:

"Why can't person responsible for system X have access to said system X?"


The stupid thing is that the reverse also happens. I happen to have a random conversation with a business member that is complaining about something ridiculously simple in our system, and that it’s been in the pipeline to fix for months.

My mind boggles, since it’s literally a 5 minute fix, and somewhere over our head some managers have been protecting their little fiefdoms at the expense of this poor guy that does something stupid by hand for an hour every day.

5 minutes later the fix is deployed, and we just saved 200 hours per year of pointless labor.


The terrifying part for me,is that by now I've been on both sides of that conversation multiple times - and the third,less seen side, where the paying client explicitly and specifically asks and expects you to implement such a Process as part of "lessons learned".

It's noble to say "ouch! well let's make sure we don't do THAT again!", but yeah ; over time the processes and procedures do expand ad infinitum :-/


> expects you to implement such a Process as part of "lessons learned".

I sometimes refer to this as "scar tissue" - an over-response to a trauma that causes ongoing issues more so than the initial injury.


One problem I’ve seen is someone implements a process, a really nice process, but has zero clue how it fits into the big picture. It’s clearly designed as if it’s the only process someone needs to complete.

What you end up with is a half dozen “really nice processes” all stacked up and suddenly what took 1 week now takes 4-6 weeks to complete.

Each process is justifiable on its own but put together make no sense at all.


Looking back at a megacorp experience, I could read most of your exact text but replacing the Bob's mistake with Bob's intentional fraud - and then the process does start making sense. People do make mistakes on purpose; sure, a small minority of people, but in a large organization that will be multiple cases every year.


Does that justify wasting hours upon hours of every employees time per year? Somehow businesses do not look at this kind of thing as ‘a cost of doing business’.


It really does not. Conservative sketch: assume 10k ppl, avg cost $50k/y, process/procedure/policy overhead 30%, is 150M/y. For that to make sense you'd need 15x $10M fraudulent or single point stupidity errors per year. And that is assuming that the bullshit actually catches most errors.

In my experience the 30% is conservative, from my measurements of large orgs. High overhead does not reduce errors, instead it increases both frequency and severity. It also kills innovation, thinking, adaptability, etc.


This pales in comparison to the tender process required of government. The logic goes something like this:

Before: In the past, companies would bribe bureaucrats in order to land sweetheart deals. There was not enough competition and hence the government was paying too much.

After: There is now a 10% chance of landing a contract after a month of paperwork. Only Accenture, Deloitte, and a handful of their direct competitors can afford to risk the time and effort required to win a contract. They don't actually do anything, they just subcontract everything while skimming their cut off the top. This isn't corruption however, it's just a nice legal business transaction.

I regularly see projects that might take 1 guy 6 months balloon out to 20+ people for multiple years across three orgs and costing tens of millions of dollars, all in the name of "avoiding corruption".

The best part is that because the projects end up costing so much, they have to be "taken more seriously", which means that the tender process must be "thorough", which means more papework and more managers and paper pushers on both sides of the fence.

There's an exponential cost to this a bit like the rocket equation...


The real answer is typically "These rules is why I have my job, I can't remove them since I want to stay". It is like when you automate 90% of someone's job they usually don't get happy. This is the same thing but the opposite, bureaucrats de-automate tasks to create new jobs.


One shd also consider that the process is often the only thing which really holds large megacorps together, there is nothing else. You could have a turnover of all staff, as long as you keep the process it is still the same megacorp. That's probably the reason management often says 'process over results' because it is the heart of the company.


Reminds me of that anectode where the father would ask the son to take the clay pots and go fetch water from the fountain. He would yell at the son to not break the pots and slap him. When people asked why he punished the son before even going, he said: "What's the point of punishment after he breaks the pots?"


Everyone sees that the check could have saved them that one time. Nobody sees the cost of the check, applied over and over and over...


See the work on Scaling Laws that https://en.wikipedia.org/wiki/Geoffrey_West has been working on at the https://en.wikipedia.org/wiki/Santa_Fe_Institute, there are loads of interviews on youtube where he summaries his work.


Thanks for the recommendation, I just grabbed his book :)


> The Prisoner's dilemma is the reason.

It's not that simple.

A bigger org is slower often because it does bigger things. No, you can't just buy a tractor even though it'll be cheaper and faster for your project than to do all the work to do internal billing to rent the tractor the company already owns.

Because if everyone did that then we'd have a thousand tractors and there's nowhere to put them, and we don't have time to sell them, nor do we know how to sell them.

And your button on your website is not a one-line change because you're contracting with the US government and the contract has ADA clauses, and it needs 34 language translations.

And no you can't make that change because last time someone improvised in this space we got dragged in front of congress and were made to promise to not do that again. So you need to follow procedures, and talk to product councel for exemptions.

And you can't change that API because 6000 developers rely on that, and they need to launch before cyber monday.

Sure, in that last case that's prisoners dilemma misalignment, but to be clear it's you who are misaligned with the goals of the business, so don't do it.

It's not reasonable to suggest a solution of "why don't everyone just stop and do what I want". If you want to tell 10k people to stop what they're doing to work on your thing, then it'd better be right. Which means you'd better know what those 10k people were already doing. And you don't.

And you can't just do it alone because it's 100 human-years worth of effort even if nobody's in your way, and you need to launch within 2 years or it's pointless to launch at all.

That's what hard about making large changes in large orgs.

Actually, correction: it's not even that simple.


A big problem is that orgs get so big that norms can't be gatekept by individuals.

In a small org, Bob down the hall enforces standards. In large orgs, they need to apply to the whole org -- so you have documents and committees. The latter are much slower and more difficult to change.


Large organizations are hard for startup people but easy for people who want a straightforward job. In a sense sitting in meetings and making decks is an easy way to make a decent amount of money.

If you are self-driven and seeking to improve things and do cool work these jobs are psychologically quite difficult.

I was always frustrated, first as an enterprise guy, then as a consultant.

Eventually I started a company and quit my job, and then it clicked.


My job is anything but straight-forward in a large org. I have witnessed entire departments or "orgs" sit idle for almost a year while management did a huge dance to figure out the budget. This involved a lot of reliance on various lower lever employees. Once that was done, finally, we then began the process of figuring out how to spend the money fast enough in the time we had left until the next budget-dance--fast approaching.

I was an engineer (high level but still) and spent most of my time helping management not look like complete idiots to their managers.

When it came to actual engineering I had to beg the help of so many shared services that I rarely knew how to proceed. It usually came down to finding someone who had been there their whole career to find out which one of their friends knew how to initiate some process...to get an EC2 instance in AWS.

High degree of chaos and stress and nothing of real value got done. To top it all off, when you find out who your customer is you realize they are guilty of war crimes.

I went there to build data pipelines for high volume telemetry. I sort of worked on that, but not really.

I met numerous people who lamented that they would look back on their career and be able to say they went to some meetings. It is not healthy, on many levels.

"Psychologically difficult" is a very diplomatic take.


Ha! Thanks, in my journey through enterprise (that was very similar to yours) I learned a lot about diplomacy :)


Another thought is that it might actually get easier as you get older.

I worked at a large org in my early twenties. It sucked. Now I work for a large tech company (FAANG) in my thirties. It is not nearly so bad.

The org might just be a better fit, but also I think people just listen to me more now that I'm older. I'm not just some kid who doesn't know anything. I can talk to people and get things done without fighting uphill so much.

Also, I'm more realistic. In my twenties I just wanted things to be simple and logical. Now I've accepted people aren't simple and that's just how things are, so I'm more tolerant of dealing with politics.


It me, straightforward-job-haver at a FAANG. I bounced around from startups to small companies, and eventually ended up at big tech due to the anxiety/uncertainty small orgs were giving me. A lot of (maybe privileged) people like that drive as it motivates them, but to me it was more concern about what I would do if the paychecks stopped. I'm much more happy working on boring shit at a big tech company. Not here to change the world, just to work with competent people and plan for my future.


I've said no several times when asked if I'd be interested in management - I like writing code and solving problems, management is only vaguely necessary in some bureaucratic unfortunate way. I would say that that approach to writing software defined my 20s. But in the past few years I've become fascinated by the problem of organization and it's made me appreciate good management so much more - getting more than two people to act in concert is _hard_. Getting 10 or 30 or 1000 of them is so much harder. If I think about it too much I start wondering how anything ever gets done at all, anywhere


This is part of why I want to move towards management. I'm not an incredibly gifted developer, I don't live and breathe code, but I do understand the process and the challenges that get tossed at devs, and would love to make their lives easier.


I’m a small company guy - worked in startups for 30 years - and every time I’ve had to work with senior managers who come in after cutting their teeth in large companies I end up heading for the door.

The goals and motivations of large companies are really different to small companies. Large companies are all about risk management - at heart they are about revenue defence - while effective small companies should be all about attack. Being disruptive in a small company is the reason they exist; disruption in a large company is confronting and to be resisted. Going from one to the other is likely to be super stressful.

Unfortunately, as I’ve experienced, you do get these same problems in small companies run by big company people who think they know better. In my experience they just add weight. They don’t understand YAGNI.

IMO the discipline and skills needed to run a startup, especially pre revenue, are very different from those needed for an established company. It’s not that one is less disciplined than the other. But the size of a small company gives them benefits and room to move that a large company can’t touch. Small companies are already risky so a bit of extra risk doesn’t count for much. It’s to be embraced.


Currently working at a startup where a lot of recent hires have come from large organizations. All of them cite "bureaucracy and red tape" as reasons for joining a small company, but at least half of them have set about establishing exactly the sort of bureaucracy and red tape they claim to be fleeing.

I think that large company experience is valuable, but you probably don't want to be the startup they're cutting their teeth on. And if you're going to absorb those types of people, best to do it gradually. Make sure they don't comprise a majority of your workforce.


Thank you so much for writing this, you could easily be talking about my current company. I'm actually losing the battle against bureaucracy, and feeling pretty lost. Despite my long experience, the more I push back, the more I seem to reinforce the idea that I don't know what I'm doing - because I refuse to waste my time on bullshit. I'm the founder, and you'd think I'd have a say, but I don't. Yet I've been doing this for long enough to know that, at least, I'm less wrong than they think.

At least I'm not the only one to experience this! But I have no idea how to tackle it.


I think the article has some good points about interpersonal comms and alignment problems and their solutions but doesn't actually mention some of the structural / political issues associated with really large organisations. For example, I have worked at or seen organisations that are actually loose groupings of 'cottage industries' (often called divisions or similar) that have separately evolved local solutions to problems (technical /commercial, etc) with a small centralised head-office. Moving between divisions is hard because they are running separate systems. Aligning their processes can improve efficiencies but can also lead to loss of local knowledge or inappropriate one-size-fits-all solutions.


The biggest example of your last point that I've seen first hand is Electronic Arts and Frostbite.

Mandating that all EA games be made with their own engine has caused so many delays, rushed games, and broken products by teams who have already proven they can deliver amazing games on their own terms.


I worked for such a fragmented company before. The team I joined was effectively still the startup that had been acquired by that company 7-8 years prior to my joining. Since the group continued to make profits that were growing most of the time, they were largely left autonomous by head office for 90% of their work even over such a large time period. There were people in some of the US offices which formerly had belonged to that startup that would describe themselves as working for that startup despite joining after the startup had already been absorbed into the parent company. This division wasn't the only such division in the company.

There were certainly negatives, an internal transfer was effectively a new hire, just one who already had their email/git/vpn access sorted and probably already spoken to a few of the cross org teams like IT or security, but overall I never felt as much organisational friction as other teams I've worked in large companies.

Anyway, the parent company was acquired by a much larger one, that much larger one acquired a similar size company with a much more top down/unified approach to direction, and given that unified approach they were much more effective at selling a vision to our parent companies execs, so a lot of them were put in charge despite the original parent company being profitable and the new acquisition not being so.

There was a big push to standardise/centralise/etc. and in that time a few products stagnated and we lost customers. The newly acquired company has a pretty poor track record in dead/broken acquisitions from this pattern, so it shouldn't have been news that prioritizing this has costs. There was a lot of grumbling at being made move to "standardised" solutions that were suboptimal in meaningful ways, but were preferred at high levels because they were homegrown.

Everything certainly took longer in the new company, dealing with centralised, barely informed non-stakeholders who still had red tape in place to get anything done, along with a strong impulse to just copy other teams rather than come up with a solution that cleanly and effectively solves the problem. I left during this process, and a lot of the people who were the core of the startup-turned-business-unit also left, the largest group now at a rapidly growing competitor.


> Everything certainly took longer in the new company, dealing with centralised, barely informed non-stakeholders who still had red tape in place to get anything done, along with a strong impulse to just copy other teams rather than come up with a solution that cleanly and effectively solves the problem.

Yes. The last place I worked was a UK organisation that had around 5000 staff, many of whom were software engineers. The org was divided into 5 or so different divisions, each of which had its own customers, sales teams, PMs, etc. Crucially, each division had its own resourcing function, so new projects could exploit local knowledge to find engineers who had domain knowledge, and could even consider career development opportunities when picking staff for new bids/projects ('Fred is interested in trying this, he's not a perfect fit but is competent and motivated to learn').

Anyway, a new CEO (who himself had an engineering background) joined the organisation and apparently in an early meeting asked 'how many engineers do we have across the company?'. Nobody actually knew. So after a while he started an initiative to launch a company wide resource management process. Although well intentioned, it was a disaster. We ended up with global resource managers who didn't actually know any of the people concerned. A lot of our work was specialised engineering (e.g. simulation development, signal processing, etc) and it proved impossible to develop a global skills framework with all of the right categories. After a very painful year, bid teams and projects reverted to local resourcing to find the staff who had relevant skills and domain knowledge.

I'm not against standardizing things across organisations (indeed, my main function was trying to drive technical commonality) but like optimization, you have to choose the right things to standardize, and the right things to leave untouched.


I’ve given the opposite side of this question a lot of thought: given how hard it is to run a large organization, how do they stay in business?

The only answer I’ve come up with is that economies of scale are much more important than most people think. And the economies of scale of a Walmart/GM/P&G/AT&T are so big that they override all the petty stupidity in those firms.


Although it depends on the business the megacorp is in, reputation is also important.

Would you entrust your live saving to FinanZly? Would you go on a family vacation in a car from DriveFlow.io?

I think megacorps are mainly successful due to being the safe choice.

People love loss aversion, and you never go wrong with Bank of America or Ford.


Very true. “Nobody got fired for buying IBM” is a subtle form of economy of scale. And perhaps allowed them to extend their relevance well beyond what their internal disfunctions should have allowed.


K-Mart was eventually killed despite the economies of scale. Then again they were badly run starting in the 80s, and it only took another 30 years


Very good example. They did everything wrong, got looted along the way, and still took more than two decades to finally die.


I tried to get people to record meetings and link them to features being discussed, so that working on a feature you can review everything that was said. IMHO it would reduce the amount of unnecessary conversation by 50% and save everyone a lot of time and trouble. The initiative failed, because not everyone was eager to give consent. Its their right, but I think if you dont want something to be recorded, it probably shouldnt be said at all in a work meeting.


Meeting recordings have terrible signal to noise ratio. Instead of putting effort into creating a concise document to describe the meeting output and actions to take, the burden is placed on others to shift through an hour video. I also see it used as a CYA technique: "oh but we recorded this decision, it's your fault for not watching hours of our meeting videos to know this".


It's a tricky balance between a useful records, and stifling peoples comfort to discuss and speak openly.

General solution is to send "meeting notes" at the end - sounds bureaucratic, but it's exactly what you describe - a chance to document and verify "hey all, this is why we agreed on, right?" As well as provide that summary to others who missed it. Ultimately way more useful than transcripts and thus always worth it's time - yes one person will spend 20mjn writing it up, but it'll save time of at least 20min times number of people missing or distracted , plus any time that would've been lost due to misalignment or poor memory etc.


Yep thats the normal way. But this can fail on many levels, for instance if the notetaker doesnt understand the problem as well as people talking, they kinda just agree to whatever was written because the thing is still fresh in their mind, then someone who wasnt even there comes to work on the feature a month later and the notes are no good, they need to chase a bunch of people.

Even with a noisy transcript you at least dont loose anything, and everyone can still make notes in an async collaborative way

> stifling peoples comfort to discuss and speak openly

I know some people feel this way, which is why I wouldnt force it on anyone. But I think its a bit silly and they should get over it, were not on the meeting for appearances.


It's not appearances or self consciousness. Have you actually asked people and tried to deeply understand why they don't like it?

It's difference between freely brainstorming with colleagues, and speaking on the formal record that anybody can access at any time under any context or lack thereof for any purpose and with full attribution. No more jokes, informal chit chat, putting out there ideas, or sticking your neck out etc. And linking it to features sounds like a great way to fingerpoint, scapegoat and blame :-(

It's all theoretical and unlikely and paranoid until it inevitably happens :-/

(fwiw, after 20+ years in the field, I am not consenting to recording; and if I do, you'll get the appropriately shortest most formal safest contribution imaginable. Only the things I would be comfortable preceding with "it is the formal position of the company / team to...". Feel free to judge, until or unless you experience or witness the downfall that can be of sufficient magnitude to eliminate all upside of recording :)


Thanks for your post, I understand these issues better now. But I think there still can be some middle ground solution. I struggle with memory, I cant remember what was said on the meeting a few minutes ago. I often have to chase people even if I was there for the note taking because they often lack some context. For the few meetings we did record, I was able to go through it while working to keep the context fresh in my mind constantly. But of course, I would not want to create a situation where that can be abused.

A recorded video can be transcribed and deleted, and the transcript can be anonymised and potentially summarised by an algorithm. I think this could at least lead to better (although much longer) notes, while protecting the participants. What would you think about such tool, if it existed?


Honestly - I still think best solution is to instill culture of meeting notes; yes, first few will suck - they'll either lack crucial detail, context & background; or they'll be overly verbose and hard to read. But like anything else, team/people get better at it - and fast, if team agrees to make it a priority; pick the right people to be designated to collect and send meeting notes, coach them, provide constructive feedback. You can also consider taking turns - once people sit in both shoes (note taker, as well as note consumer), enlightenment and understanding occurs. Bluntly - it feels you perhaps haven't been on receiving end of a good set of meeting notes; it's a skill for sure, but an eminently develop-able one, and will never ever not be useful - even if just for yourself! :)

The technical solution, in my view, would a) not be worthwhile the effort to setup [transcribing and creating summary algorithms - I get that'd be fun, but that's not the same as feasible ;] and b) would have too many points of failure (now we need to both trust it'll delete and anonymize etc). If there was a COTS tool... dunno. Maybe. transcribing is already built into most web meeting software these days, and we value our meeting notes way way way way more.

Thing to understand is - making and sending meeting notes isn't (necessarily:) a burdensome, bureaucratic task that has no technical value; once you've developed the skill, you'll find you're keeping great notes for yourself. I use OneNote, I have a tab for meetings, and every meeting I go to, I keep good notes (I'm a touch typist which helps hugely, to be fair). During and after meeting I cut/trim/organize them. If I need to send them out, it's a few minutes to format them nicely for others. If I don't need to send them, a) They're useful for me to refer to and b) They're useful even if I never refer to them, as I keep track of what's going on during meeting, organize my thoughts, and ask more and better questions because I see the gaps. As we go, I keep bullet points, I order them, make them hierarchical, reorder and so on; frequently I'll screenshare them and it actually makes the meeting better as it prevents looping around, reduces uncertainty, and shows gaps / what's left much more easily.

My 100 Croatian Lipa :->


That's why notes are then distributed for review, allowing people to say when the notes as taken don't accurately reflect what was discussed.


Might be easier with transcripts, but with recordings it takes a lot of effort to find the correct recording, and correct place to find the correct information that you have a question about. As well as needing to know that your question might be in one of the recordings. So it just also becomes easier to re-ask the question even if it was already discussed somewhere.


I agree, and tbh we had automatic transcripts as well (a feature of MS Teams), but its the same issue, you need to be able to record/capture in the first place


As an in-the-trenches developer, I've almost never been able to get a productivity initiative such as this into effect.

The obvious answer is, "it's my fault" and that's highly possible.

However, I never really see anybody else accomplishing such things either.


Hard disagree. Why would I consent to be recorded in the workplace? My job isn't to be recorded.


I think the author actually nails the primary reason in an aside:

"As though that were not bad enough, in some organizations there's a separate compounding problem: limited resources. For an organization, attention resources are {time, money, staffing}. In that case you get to layer on the negotiation for resources among competing teams within an organization."

While I think the other points are valid, this process of resource allocation alone probably explains most of the problem. As an organization grows, its resources grow proportionally, but the communication required to allocate those resources grows non-linearly. The overhead needed to get $50K from each of two people is more than twice that needed to get $100K from one person. Add in the other issues raised, and this core problem is exacerbated even further.


It is as simple as having too many middle managers. It doesn't matter how large or small the companies are.

Each one need to justify the existence of their department. And to get promoted, they usually need to hire more middle managers.

These departments will inevitably have confusing directives and even non-sensical overlapping products.

Eventually the built-up inertia becomes too large.


The triad of solutions to the prisoners dilemma gives some really fast heuristics. e.g. if time-boxed, then reduce consensus; if hero req'd, give up credit; if consensus req'd, give up time commitments, etc.

The best easy book I can recommend about large organizations is Smith & DeMesquita's "Dictator's Handbook" because it provides a practical applied model for determining incentives between groups. Reality is, the people you are dealing with are running their own mental version of the model, where the coalition that keeps them employed and prevailing is made up of essentials, influentials, and interchangeables. They intuit a salience matrix of stakeholders (decision power, vs. how much stakeholder cares), and while everything is dynamic, it is usually within these parameters. It's a tool that can provide fast insight into group/project/org behavior pretty well.


"Everything" really means "changes that I think would be better for me/the company/the world", because if you are happy with what the company is doing, you don't need to change anything.

In a truly large organization, it takes institutional knowledge to identify who will have to make a change, who maintains it long term, and who will be impacted.

Then you need to convince each group that the change you want is desirable. Sometimes this is as easy as convincing the person in charge of the group. A large one-time change is often easier than a small change to daily procedures.

As your knowledge of how the organization operates grows, so will your ability to find the necessary people and phrase things in terms of their interests. In a thousand-plus person corporation, I would expect this to take at least a year or two.


Which is why politics is always an important part of large companies. In order to get anything done you need to convince a lot of important people that it would benefit them to allow this project. Those people then consider how much they and others could leverage the project to expand their power and influence in the company, if it isn't net positive for them they probably wont agree to it.


The increase in people increases the relationships and amount of communication geometrically.

The increase in levels is like a game of telephone and reduces the fidelity of the message/strategy/goal from executive down to line level team members.


I think one of the only large companies who has worked this out is Apple. I would love to know how they pulled this off.


Given their recent moves you could argue they're still learning like the rest of us.


For anyone interested, I wrote a fairly detailed case study of some of the difficulties faced when trying to reinvent the technology stack at one of the world's largest car rental companies:

"Why are large companies so difficult to rescue (regarding bad internal technology)"

http://www.smashcompany.com/business/why-are-large-companies...


My personal experience with large organizations (over 100,000 people) is that everything is so hard because 10-20% of the people do most of the work and the others are not just minor contributors, but many times just roadblocks.

For example 1 hour ago I was discussing with my business counterpart about a project we are working on. We just got approval to hire 4 more people (that means we have to hire them, it is not optional) and half of them are diversity targets (meaning: skills are irelevant, we will get them anyway) and the other 2 will be hired and given to us without knowing what are the skills we need. Based on such organization dynamics, we plan to be sufficient without these 4 people and if any of them has any useful skills, it will be a nice surprise. At the same time, we need to give them some work to do, so we need to invent some roles that should be as harmless as possible.

This is just a project, in my department of ~100 people about half are just mouth breathing, but it makes it a lot more complicated for the ~ 20-30 that do work to just do their work: some of these people have invented roles to rubber stamp steps in the projects, so they are adding 10-20 steps and a few days of work just to justify their existence. With the department half the size, everything would be much simpler, faster and even more productive.


Group think is well known, but it's also a force that works against individual think. Similar versions of the same symptom emerges as group hearing, group speach, and group action.

Thinking alone is easy. Thinking with someone is hard. Now keep adding people. You stop thinking.

Same with hearing. Listening to one person is easy. How about two people. If they don't share their time, you're listening to two people talk at once. Same with actions.

With more people, thoughts become more ideological and less mobile. Same with speech, same with what they listen to, and same with their actions. And each individual is not only physically a fraction of the group, but so are their senses. And to act humane or to simply perform on an individual level is directly hindered by this group dynamic. Individuals then become lethargic, and begin to experience greater stress and pain. You could also think of this as the Group Flu. I know I've experienced it. It's the feeling you get when you know you need to talk to someone higher up just to be heard. It's the feeling you get when you're talking to a customer service rep and you know you're not talking to a person.

Now take a large organization and we have a group of groups.

We already know one group cannot do it all, and one group is too inefficient. But if we're just creating more groups, the problem persists.

So what you need is to undo the group dynamic. But then again, with a big successful organization, maybe change is what you don't want? Ideology is what you do want, and suppressing the individuals leads to your ongoing success.


There's also a balancing effect from being in a group. Thinking alone is easy, as is going insane in your own belief. Other people acts as averager, additional data points.


Balancing how?

Say each member has a point, and additional members have counterpoints to balance the group.

What exactly happened? As a member with a point countered, you've effectively been silenced. You have a group where extremes are neutralized. That is either a set goal by whomever controls the group, or works towards the greater ideology that becomes the group voice and group think that no one member can overturn.

You don't need to be in a group to have additional data points.

Thinking alone might be easy, but not thinking in a group is also easy. That's what most members end up doing.


Maybe you've never been stuck long enough in your own head, but i did spend years with big blind spots about life, and the ability to talk with others would have saved my many many years.

Also you sounds biased against groups. Sharing points of view don't equate being silenced. Having more data alone still means you're the sole interpreter, you perspective / understanding of the world might still be too focused on part of the space. Others can widen the angle, change your position. Of course politics can distort things too, but that's outside my comment. I was talking about conceptual group, small, free of agendas. Just a bunch of people saying what they see. Not every group is ruled by someone (even though most of the time there are such implicit social structures, I agree)


You don't need to be in a group to talk to other people. You also qualify "group" with functionless groups.

I am biased against group think. But also group speech, group emotion, and group everything else, as they are all inferior to their individual counterparts. Group emotion is a mob.

If you've been in a group where the group changes your independent behavior, or you relinquish your decisions to the group, you've just downgraded yourself.

And if you look at all the stupid in the world, most are done by groups.


> The Prisoner's dilemma is the reason. There might be a way of doing complex multi-participant tasks that is better than what's being done currently, but the incentives for each participant are not aligned. Even when that lack of alignment produces a suboptimal outcome, each participant in the process lacks skin in the game. Each participant in the process creating the suboptimal outcome is not accountable for the consequence of their collective action.

The article goes on to describe solutions without actually explaining how the Prisoner's Dilemma operates in large companies.

This makes the article less valuable than it could be because it interferes with the reader developing a mental model of the problem.

For a take on this that does present a very clear (and possibly actionable) mental model, see "The Gervais Principle":

https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...

The idea is based on the axiom that "organizations don’t suffer pathologies; they are intrinsically pathological constructs."


The biggest component of the answer is simple: risk aversion. The status quo has been working so far, and making changings introduces risk that things might no longer work. An organization will probably exhibit risk aversion comparable to the most risk averse decision maker in the a particular decision chain. This imparts the status quo a type of inertia.


I have definitely experienced effective small org -> ineffective large org.

Lots of make work, "lawyers don't get paid to simplify". Growing from small to large company you can end up with a lot of bureaucrats that tell you things like "this policy is what big cos do", but is really entrenching their position and keeping them employed. Creates adversarial relationship between devs and IT/ops/sec.

They don't have it easy, because one leak or bad PR due to data and they're shot into space.

Best to move to small effective teams, move fast, and show value. Often by currying favour of C/SVP you can get barriers removed fast and get things done.

Ecosystem is large, so find your niche.

Seen entire teams destroyed by well meaning PMs and other idealists implementing a "system" leading to dev/project slowdown and even abandon. Adding a couple check boxes to a JIRA process seems simple, but every choice is multiplied by the number of developers you have. You can see this in places that implement scrum poorly.

I'm reminded of this slide: https://speakerdeck.com/holman/how-github-uses-github-to-bui...

Overly complex security policies, JIRA check pointing, and constant meetings can grind you up. A lot of it is mistrust of employees, or employees also don't have enough context to do what they need to do.

It is also entirely possible that I am a particular type, attracted to small/medium fast movers, and then I get tired of battling security/ops/managers for access to systems/data I need to do work, or the new sets of flaming hoops to jump through to do completely mundane tasks.

You can always roll the dice! Bonne chance!


I think a large part of the problem is over-specialization.

Once you have groups with a leader that are in charge of [something] narrow, they start trying to make [something] more important, more complex, etc, than it should be. For various reasons...exposure for career progression, empire building, etc.

Then, when you need to get something from an idea into production, you're dealing with some large number of these groups. All of which have some form of demand management (jira tickets or similar), some form of requirements you have to meet, different ways of interaction, and so on.

And, some of those groups have to work with other groups to do "task a" that you asked for. So you get this complicated dependency graph of tasks that's not even visible to you.


The larger an organization gets, the more typical a chunk of society it is, so its effectiveness gets closer to the societal average, and further from whatever can make a small organization special. I think this explanation covers all other explanations I've seen.


Couple of big things missing/brushed aside, really.

1) Uncertainty. On larger things, success is often uncertain (e.g., on new products, etc.), so the final consequences of the decisions are not known in advance

2) Generally, running processes well across an organization is not something that comes naturally. It needs investment in skills. For example, most meetings should have an agenda and should end with next steps as well as agreed and assigned actions. Tracking and planning similarly need effort and skill.

It might very well be that on top of it the incentives are wrong for people to take risks in (1) or developed skills for (2) or even to cooperate in the first place, but most people are not naturally good at administration to start with.


How much space do I have?

I think the skin-in-the-game is very significant as the OP says. In a smaller organisation, you might have a person who does several things and one of those is HR. The existence and success of the company depends on hiring well so they do their best. Also, the more efficient they can be (sometimes the worse they can be) the quicker they can get back onto other things.

In a large corporate, you now have an HR team. There is usually much less feedback on performance and it is easier to blame the org or the market for poor hires.

Another issue is that most employees like most people are only average. A few are terrible and a few are excellent. We can be picky in a small company and find that great person with alignment to the vision etc. What about in your team of 10? You aren't going to find 10 great people so you get a whole range and now some are doing much less than 10th of the work of the team so the excellent people are now picking up the slack.

Maintaining multiple communication paths is hard and risky. I worked with a Bank where each requirement spec was signed off by about 15 people (compliance, legal, marketing etc.) how complicated is it to make 1 change? Holidays, other priorities, change of staff etc. can make the smallest thing take really long.

Ego: Some people simply cannot compromise. If you are the CEO, you might get away with it but what if someone else is throwing their toys out of the pram because e.g. the legal team need your product to look terrible for compliance reasons? Things you should just accept take weeks or months to argue out.

Specialism: The head of e.g. marketing should have the final say on all things marketing ideally. But in many large companies, they might have a disagreement with people who might or might not know more about something but vocalise it anyway. This is magnified if you don't really trust your department heads are good at their job.

Oh, so many more things.

Is there an example out there of a large organisation that does work well or are there just too many variables to make large organisations work period?


The first half of the article is ok but the part about large organisations is completely misguided.

I think the author is conflating prisoner's dilemma with the communication challenges in a large organisation, which arises naturally from their sheer size (as other's pointed out, these companies work in a bazillion products in parallel, not in a single one as the author claims).

As an actual employee of one such company, one of the ways they reduce the impact of this is with Principal/Staff level engineers. This is similar to the concept of hero mentioned there. These high rank ICs oversee projects that span teams or orgs, and align them to achieve global positive outcomes.


I would say, the bystander effect. The more people you have, the more you have the tendency to think "oh, someone else will pick it up".

I am guilty of that myself. In a small team where I know no one is coming to help me, I step up and become 10x productive.


Not sure about the bystander effect applied within an organization but hasn't this theory been discredited following recent research?

Cf. https://psycnet.apa.org/doiLanding?doi=10.1037%2Famp0000469


Does everyone try to help? And if only one person helps in a crisis, even most of the time, how does that translate to organizational psychology?


I don't know if this topic has a name (study of work social structures) but it fascinates me. I see so much counter intuitive events, misery, and low efficacy that I cannot stop thinking about what are the major forces at play


The field is called Organizational Behavior if you're looking for scholarly info


godly many welcomes :)


Systems theory tells us that in large organizations nobody actually does what they say they do. For example, one person building a boat is a boat-builder. But nobody at General Dynamics Electric Boat can properly be said to be building boats (or ships).

Thus, in a large organization, you can only improve the ability of individuals to do what they actually do, which may or may not improve the ability to the organization to accomplish its nominal purpose.


Communication multiples due to the number of people involved.


I’ve experienced this myself in companies that have gone from <100 people to thousands over a handful of years. In the small days it’s easy to stay current and coordinated with everything. As the company grows new teams are added, there’s more specialization and teams begin looking more inward as their size has grown.

You don’t realize it right away but your processes, methods, and networks are breaking down with the new scale and what was once effective stops being so. But it’s hard to detect right away - it’s the “boiled frog” problem.

The company defines themes or goals or priorities and small cross team groups are assembled but it only gives the illusion of being coordinated. You still have a bunch of teams vaguely working towards solving a greater problem but not actually working together.

There’s a general sense of “losing our way” and disillusion from the veterans who don’t recognize the company anymore and attrition sets in causing even more disillusion and attrition. The company is growing headcount but it seems like everyone is leaving.

I think this is the point (hopefully we’ll before) the company realizes the old ways that got them there don’t work so well as to get them to the next place.

What I’ve found works is senior management that can not only identify what the company priorities should be but also understand that every team should participate in it and allow the teams to determine how to execute. Defining programs and having coordinators work with the stakeholders helps scale communication and coordination. It’s another layer that wasn’t needed in earlier days of the company but becomes critical in my experience.

And the cycle begins again. If the company does achieve the next level of scale similar problems will set in as coordinating 3000 people is just as different than coordinating 15,000 as it was from 700 to 3000. More layers of indirection are required.


The fun thing about Dunbar's number is that he nails these transitions pretty cleanly. The breaks happen (roughly) at every 3x in growth. What works for 5 people works for 15, but breaks at 16 (plus/minus 2, depends on the team).

What works for 16 works up to 45/50 (3X15).

Next breakpoint at 150 (3x50), the 'classic' Dunbar number.

likewise at 500, 1500, 4K,...

and it all comes down to communication bandwidth. The same scaling laws which lead to the cortical columns in the visual cortex specializing into different kinds of signal detection (edges, motion, ...). You can only maintain close connections with so many people.


I find this amazingly insightful.

One thing I've been thinking lately is, what if everyone had as part of their "goals"/"performance review" structure -- how have you helped another team (in a non-close branch of reporting structure) do their job/accomplish their goals better?

It's probably naively optimistic to think there's a bureaucratic solution to bureaucracy though, which is basically what that suggestion is.


I wonder how "bullshit jobs" and the fact a lot of people in large organizations are not really into the mission factor into this.


I would ad busy work, stuff that isn't per she useless but not really needed right now. Easy to end up doing that instead of the really needed things, because they make you look productive while not being counter productive. Result is nothing advances. Usually people do that when the right-now things are hard and involve facing some hard truths. Even more so if that is implicitly encouraged by higher ups (management, founders...).


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: