In Germany the app was developed by research institutes. Then canceled. Then SAP picked it up and charged 10 million Euros for what is essentially a few 1000 lines of UI code. The German Telekom got the same amount, just „because“ and will additionally charge 2-3 million per month mostly for the hotline (with limited capacity).
So the total cost will be 50-100 millions. Despite the fact that most technical work was already done by Apple and Google for free.
I‘m not even talking about the fact that my dad got a new phone to run it - sold by T-Mobile (Telekom) of course.
Not sure where you get the 1000 lines from. That's ridiculous. `cloc` counts 16k lines of Kotlin for the Android app and 22k lines of Swift for the iOS app. That does not even include other aspects like the server side code or support documentation. Also it's not only the app that had to be build: It required all the notification infrastructure and coordination to deliver test result to the app. This is really one of the very few big government projects where I can't really complain about the costs. There was only one attempt at getting this done right and in time. They managed both.
You're right that the parent comment is exaggerating, but 20 million for a project of this size is still a bit crazy. The same app cost Austria "only" 2 or 3 million, IIRC.
Plus, I just don't understand all the NIH. Why isn't this kind of stuff being developed in one place and then forked by the respective countries instead of everyone creating their own tiny island?
That said, I don't have any major gripes with how the code turned out, just with the cost.
Yep, am positively surprised how the product turned out. Plus the launch also went well I think, including a comic manual and even an AMA on reddit; I didn't read of any glitches or anything bad either.
I really can’t understand this for a COVID-19 contact tracing app. You should find out how to give feedback on this to make them change availability to global.
It boggles my mind that I have to remind many app publishers that we live in a globalized world and that it’s a natter of a few minutes work to just select the countries to make the app available in. More so for free apps where the publisher doesn’t have to worry about taxes and payments.
Even for apps that are not localized to English or other languages, the publishers should prefer to get them out to as many people as possible in the whole wide world who may be visiting or on a long stay in a specific country.
In this case, the app developer can surely include help text that explains where the app is meant to be used or would be useful (it’s also not like people with a German App Store account living in some other country would be able to use this app effectively).
Many times it has happens due legal restrictions that developer has in place in his org . It is worse when the publisher is a government body or contracted by one
Sadly, no. It's been left to the states, and almost none of the states are planning on using apps[0].
This boggles my mind. Sure, it will take a reasonable effort to get widespread use of these apps. But the impact would be huge. Even low rates of app use have a real chance of lowering the reproduction rate significantly, and reasonably broad usage bring it less than 1.0.
The technology is designed in the right way, where anonymous data is being collected with no app required. People can download the app right after they get a positive test and report themselves positive retroactively. Same on the other side, you can download the app and know how much contact you've had with positive cases retroactively.
I'm no longer surprised the US federal government is doing nothing to push this. But I expected more out of the states.
Where I come from (.au) we paid Boston Consulting $8mil to copy/paste the source code from the Singapore governments open sourced implementation.
The government was then talking it up with "If enough people download the app, we'll let you go to football games again!!!"
Current media reports say none of our local health departments have found _any_ contacts using it that they hadn't already found via manual contact tracing.
(I think 100 apps+backends for $10mil is a bit optimistic, if those backends are required to be HA and support a significant percentage of a countries population. As a single datapoint, where I work has many iOS+Android+backend app projects for gov departments that serve tens to hundreds of thousands of users that cost them high 5 or low 6 figures plus 4 figures a month support/hosting, but if they wanted me to support the 60% of the population (~15mil users) that our CovidSafe app was targeting it'd most likely come to more like mid to high 6 figures, and certainly 5 figures a month to host/support it. So I'd have said maybe 20 apps realistically for that price. Still just as fraudulent a rorting of taxpayer money though...)
I honestly think this will be a net loss financially for SAP and Telekom. To get it done so quickly in parallel to the contact tracing API being just developed will have required a team of their best developers. The lost opportunity cost for this alone will be huge. Then it is not just development, but also QA, community management (git hub issues are in the hundreds), legal, project management, etc. I think the primary reason SAP and Telekom even offered to participated at all was political pressure.
The loss in public reputation if they had failed was not worth even participating.
Also, having worked in a corporate setting would have told you that you can't launch anything for less than 1m EUR/USD, can't launch two things for less than 5m and 3 for less than 20m.
I do work in a corporate setting and have done so, as a consultant, for years. I know the numbers very well, both from what I and others charge and what large companies actually budget.
If you would look at what the app actually has to do to „handle the pandemic“ you‘d find most apps would be more complex.
Same for what you call concurrent users. Those users have no identity at all and essentially need to anonymously download a bunch of static files which are the same for everyone from a bucket periodically. Put a CDN in front. Done.
Infrastructure for the one we have here in .au, which started out as a copy/paste from the .sg opentrace app's git repo, is minuscule.
The only backend touches that apps do is a few login/register APIs when you first launch it, and if you ever test positive and agree to send your data it sends a big json blob to an AWS Lambda nodejs function that pretty much just uploads the json to S3.
I don't recall how complex the login/register stuff is, but I don't remember thinking "That's gonna need heroics to register 10 or 20 million users quickly" when reviewing the code.
Simpler really. No need to get the Facebook SDK, DoubleClick SDK, and Cambridge Analytics plugins set up and configured with your payment details correct and tested...
(And the MSS/GRU/NSA/Mossad will all insert their code into your app bundles without you needing to do anything...)
Nope, it's not 1000 lines of UI code. It is a project with lots of pressure (timeline, politics, ...) and two years of maintenance. Including two apps (iOS and Android) multiple Backend systems and a concept, project management (coordination between government, testing laboratories, ...) looking at the team size (educated guess based on GitHub stats) the hourly rate is on the higher side, but it's still low for a government contract and in case it has effect (which is still not proven) peanuts compared to the financial cost of COVID.
I said a few 1000 and I actually browsed the code on Github when it came out.
There were just six weeks to develop it.
And the concepts where developed before by public research institutes for a tiny fraction of that price (not included).
(If you insist, here is a small calculation. Let‘s say we charge a whopping 500 Euros per hour per engineer. In other words an engineer that costs 1 million per year. And we have 6 weeks time which was all they had. And let’s take 20 people for that. And let‘s add 500.000 for some overhead. Then that‘s still just about 3 million. And I‘ve used insane numbers for a product that trivial.)
First mistake: you calculate 6 weeks of dev time ignoring that SAP got 11.5M including maintenance for 2 years. Certainly amount of work in two years time is lower than initial dev work, but not zero.
Secondly you are only taking in "development work", but such a contract entails other things
- certainly there is a penalty if they deliver too late
- there is legal risk to cover, if they deliver crap in some legal sense they are liable
- probably SAP had other contracts which they had to postpone
- count in all the lawyers writing the contract :-D
And yes, they certainly made a profit out of this and yes a proper biding procedure might be better, but given the timelines and the transparency I think it's ok.
Okay, and the additional 1.5 million should suffice to „maintain“ an app that will not need or get any new features.
Let‘s assume another million for lawyers „writing the contract“. It‘s still far off.
Additionally let‘s remember the 20 million is just for the development. It does not include running the thing. That‘s what the other 50+ million are for.
I don‘t agree. I run a business myself. I charge plenty. I know how much DAX size companies charge as well. The numbers don‘t add up, and this is especially true for something that could have been done pro bono by SAP in the current situation where pretty much everyone tries to do whatever they can.
I can‘t reply to marvion below so I put it here: The total initial cost is 20 million. As I said SAP gets half. These „operation“ costs by Telekom do not actually include any operations. The actual operation (including the hotlines) is estimated at 2-3 millions per month. But it actually depends, it‘s not a flat fee.
Operations for two years are actually not 20 but 40M. But this includes two phone hotlines, thus actual labor cost. (And by then the development costs are close to irrelevant already ...)
> but such a contract entails other things - certainly there is a penalty if they deliver too late - there is legal risk to cover,
... celebratory cocaine for the entire sales team, new yacht bonuses for the entire executive team, another few mil into the CTO's golden handshake fund, and a brown paper bag full of unmarked notes for the gov procurement guy. Poor bastards probably barely broke even on the project - certainly not enough to pay out any bonuses to the devs or project managers...
20M for the boost of Germany's worldwide status is a drop in the ocean. The money was well spent, Germany did not embarrass themselves by penny pinching. It needed to work, and it appears to. Compare this to the UK who is still flapping around.
It doesn‘t work great. I actually run it on my phone. It has silly bugs, despite the simplicity.
Additionally, for that price you could have easily have 3 independent teams develop it and take the best one and still be cheaper and better.
I hope someone will challenge this in court. I don‘t think they can get away without a call for bids which should have been done (across Europe, as the law requires).
There is this contact tracing framework in iOS and Google Play Services and that causes a few errors. Some users get a message like "not possible in your region" and some get some random error codes about rate limits and such errors, which bubble up from those frameworks and can only be fixed there.
Yeah. It‘s the framework which does pretty much all of the interesting technical work, if I may say so. It does the communication with other phones and determines if there was a contact and generates, stores and maintains the tokens and given the server data figures out matches and so on. It has a tiny, simple to use API on top.
It was jointly developed by Apple and Google. For free.
That's what I thought, it seems that the entire project is basically a fancy wrapper around the Exposure Notification apis provided by Apple and Google.
There's some kind of cognitive dissonance around your idea that these companies deserve so much compensation for designing a solution that relies so heavily (entirely) on these features, and yet be free from responsibility for serious bugs that arise from this decision.
I mean, if there was a public tender there's little to complain, if there wasn't then it seems right to, even if it's "a drop in the ocean".
And frankly nobody thinks "wow, Germany has a working COVID app" since almost everyone else has it too. Even Italy, amongst a ton of screw ups, has managed to produce a perfectly fine open source contact tracing app.
Yeah, I don’t get all these people criticizing the cost. Also, when a government is dealing with a flagship corporation of their country, cost is a secondary or tertiary concern anyways. It’s a subsidy in part, and it’s also just that $10m for a nation state government to get it right the first time is just so so so inexpensive. I think SAP let them off easy this time, if anything.
An illegal subsidy. Public contracts require a bidding process, usually across Europe.
It‘s unfair that smaller companies don‘t get a chance.
I reject the idea that Germany should prop up its biggest companies, excluding what really is the backbone of the German economy (which works very differently from the US one) and also excluding our partners across all of Europe.
I think the UK's decision to switch to Google and Apple's framework was actually the right decision, although it's all rather moot and a waste of money as I can't see any contact tracing app work if nobody downloads it (see France).
If one of these apps leads to avoidance of a handful of serious cases (by identifying clusters earlier and thus reducing spread) it has paid for itself. An ICU bed costs _at least_ 2000 euro a day (that's for a normal one, ongoing operational costs only; rush-job new ICU capacity is presumably more expensive); it doesn't take too many averted hospitalisations to pay off one of these apps.
Obviously, it would have been better if the German app has been cheaper. But rush jobs are expensive, and the cost involved is pretty minimal relative to the cost of managing the pandemic writ large. If it's even slightly effective, it was worth it.
I think a lot of people could have made a good working app for a lot less. But on the other side: if SAP and Telekom had failed no one would have blamed the government.
And if the both had failed their reputation would have suffered a lot. Well it's not the best as it is, so well.
They have a top team and delivered on time. Thats rare in software development.
And 20 million are nothing compared to the tax break which costs about 3 trillion.
For the government it's money well spend in two german flagship companies.
My guess most of it was insurance money in case something went wrong, so these companies would have taken the fall.
Don't bother, HN users LOVE expensive government IT projects because it artificially inflates their own self worth. Remember those 20 lines of python you wrote before lunch was really hard and worth millions. You are a ROCKSTAR!
Hey developers of German Corona Warn-App. Just wanted to say even I have installed it, not because I'm convinced it'll help too much (the verdict is still open I guess) but because it's a good project result given the constraints, and there's no doubt an effort was made to deliver the best possible privacy, so I didn't want to be the one boycotting it or something.
The total cost of the app (over a 2-year timeline) is about the same as 1-2 days of Covid testing in Germany. If it helps to reduce the spread of the pandemic it will be absolutely worth it.
I think what's worse is that I suspect the whole project won't be all that useful, and a drop in the bucket compared to society-wide (enforced) change in behavior.
From everything I've read, this kind of app 1) is only really useful if enough people actively use it, and 2) have serious issues with false positives because of inherent positional inaccuracy (or inability to be accurate without sucking battery).
In regards to 1, in my country at least, unless using the app would be mandatory, I know many more people who will refuse to use it or be too lazy to use it. And I'm pretty sure hell will freeze over before the government can make usage (and owning a smartphone) mandatory.
So, false positives may not matter in some conditions. Ireland is currently reporting anything from 5 to 30 cases per day, doing anything from 3000 to 10000 tests per day, and has capacity for at least 15000 tests per day. If the app doubles the number of tests being done and finds a few new cases, that's fine; it's still a lot better than NOT finding those cases.
These apps would, granted, be far less useful in places with major ongoing outbreaks and inadequate testing infrastructure; lockdowns should be used to bring numbers down to the level where this sort of thing is workable.
(1) is probably a greater problem. Ireland has about 30% uptake, which probably isn't enough.
Yeah, after I wrote my comment I figured that perhaps I overstated how useless the app would be. I imagine even with relatively low uptake it'll have /some/ effect. Even a little would help, I suppose.
Relative to other COVID-related economic damage this is peanuts. It's acceptable to throw a few tens of millions at a wall and hope that something sticks.
> The German government says its app cost 20 million euros ($22.7 million) to develop and will require 2.5 million to 3.5 million euros per month to operate. It’s available in German and English, with Turkish and other languages to follow.
I cannot speak for Germany, but I have a bit of experience in how public health projects are funded in the US. Usually, the set of incentives are misaligned for maintaining products.
Research institutes are typically funded through contract grants, where you as a principle investigator are supposed to be using the money for a research project. Obviously there are exceptions, but if you want to make a large name for yourself in academia, doing maintenance projects isn't the way to get yourself promoted; all the flashy rewards are in publishing novel cutting edge ideas in high ranking journal.
I am not an expert on how things are in Germany, but if they're anything similar to the US experience, I'm not surprised at all that they would fail due to the misaligned incentives.
Because it was a mess. A privacy mess. Also because the fact that supporting iOS well requires using the Apple framework was ignored. And also lot‘s of politics.
The company is part of the development. Afaik many apps are open source.
"Donating it" means offloading the support and profit from Linux Foundations status.
Nearform is involved in integrating the backend into the countries health system... so they offload the app and just do the work that gets paid well. ?
The app has been open sourced by the Irish Government, not Nearform who were the developers. The copyright is owned by the HSE (the Irish public health authority).
The app being open source does benefit Nearform, though, as they can sell their services customising the app for other governments.
Large-scale deployments in Ireland (1.4 million out of 4.9 million people) and Germany (12 million out of 80 million people), anyway. In Ireland it has lead to the discovery of a number of positive cases, which is positive; if they'd been discovered at all through conventional means that would have taken longer and the people concerned could have been spreading it for longer.
There has also been successful deployments in Taiwan and Singapore, but AIUI they don't use the Apple/Google API and are far more intrusive.
The Australian app (COVIDsafe) does not use the Apple/Google API becuase the Goverment wanted location tracking which is banned under their terms.
This means the COVIDsafe app chews up battery life and is vulnerable to power saving features which shut down its communication after a short amount of time on iOS in particular.
It would have been far better to use the API and obtain the location data from elsewhere (Google, Ad networks, Mobile Carriers) if really needed. If this were a less free country I'd suspect the location tracking was part of a government spying program but instead I just chalk it up to incompetence.
They didn't even build the app. They just skinned an open source one which was made before the APIs came out. I doubt the government has the talent available to update it to use the new apis.
The problem with contact tracing apps is that their effectiveness has a quadratic relation with the adoption rate. They would be mostly useful to trace "random" contacts (since you tend to know which of your friends and family you saw within the last 2 weeks), but to be able to trace a possible infection both the infected and the "recipient" have to have the app and active. In Germany they said their 12M installs is "good" but that's just 15% of the population, so if you'd take a random pair the chance to detect a link is 0.15*0.15 = 2.25% ... so it's obvious that this isn't going to move the needle much. So in order for this to be effective, you need to have very high adoption rates, and it can only help in addition to other measures (wearing masks, hygene, avoiding indoor gatherings, ...)
I completely agree with you that this cannot possibly replace other measures such as masks, hygiene, and avoiding large indoor gatherings. Nor should it.
I think there is an additional confusion though. This is really an exposure notification app. If you used only this, and not traditional contact tracing, you would have to have a lot of adoption. But I think the general recommendation is to use both (1) these exposure notification applications and (2) traditional contact tracing by people trained to do it. Then you don't need as large an adoption to get utility. Traditional contact tracing can be effective, especially for people you know you contacted. But it is slow, and does not handle well the people you don't know. By combining them, there is a greater odd of notifying those who may be potentially infected. As long as you're also doing traditional contact tracing and other measures, a lower adoption rate for these applications can still be valuable.
Full disclosure, I work for the Linux Foundation. But I still believe this anyway :-).
The other problem is which people are likely to install the app. It basically requires a modern Android or iOS smartphone with internet connectivity, which probably excludes the really at-risk groups that are of most concern: https://www.dw.com/en/loved-or-loathed-how-germanys-coronavi...
So, elderly people are less likely to use a smartphone (though I suspect not THAT much less likely). But the people they come in contact with probably do use one, mostly. Generally these should work on any smartphone under five years old.
Trying to create hardware token for this. Because myself and many friends are not able to use this. And if I think of old people not able to handle smartphone, this should be useful.
https://github.com/Lurkars/esp-ena
It has fallen off because most states have decided not to use an app. The exposure notification itself can only be leveraged by government/health officials.
I released a blog post (10min read) this past weekend breaking down digital contact tracing in the U.S. I also discussed why Google and Apple's solution isn't sufficient.
Most iPhones now have the API available, but for some reason it has fallen to the states to provide apps that work with those APIs, and (no surprise) 90% have elected not to. I expect the feature to be quietly dropped in a future release since it's clear nobody really wants it.
“In order to use the Exposure Notification APIs, You must be a government entity, such as a government health services organization, or a developer who has been endorsed and approved by a government entity to develop an Application on behalf of a government for COVID-19 response efforts. Entitlement Profile(s) are limited to one (1) Application per country unless the country has a regional approach, or as otherwise agreed by Apple.”
The Chinese Health Code app has been extremely widely deployed and has probably contributed significantly to containment. I’m not sure how much has been written about exactly how it works though.
This pandemic is global so anything that helps countries to combat the virus is a good thing. The Irish app seems pretty good but the great thing about open source is that you can branch it to make your own app or just use the code to learn how others resolved the technical issues associated with creating a decentralized contact tracing app.
Of course an app is not a panacea but as developers we can't create vaccines or provide medical assistance. Its not perfect, as it depends on getting enough people to use it. No mean feat when there are people who don't even believe Covid exists or think it is only dangerous to older people and don't want to take any precautions like wear masks or socially distance. Even then, there are lots of technical challenges trying to use a technology like Bluetooth for a purpose for which it was never intended.
There will be a vaccine at some point (plenty of anti-vaccine people too, which might be a problem for the future) and there will be better treatments in the medium term but right now, speed is important and if it helps States to save time and get a contact tracing app quickly then it has to be a good thing. That way we are all safer.
I wasn't sure whether to remove that as a gratuitous adjective. Even without that claim, it's an interesting development, especially as the app cost only €850k.
In contrast, our nearest neighbour scrapped their app last month after spending over £11MM (~€12MM) and are now rebuilding on the Apple/Google API: https://www.bbc.com/news/technology-53095336
I didn’t mean it that way, I’m just not sure how much I could talk about it. For one, because I was involved but only so much. And two, because it’s not really my thing to talk about per se.
As for problems, not really any problems. It was quite straight forward. I’m just not sure my company would appreciate me speaking on their behalf about details.
Edit: but I can answer like, general technical details and things like that.
I wouldn't put the focus on it being ONLY 850k, that seems a competent price, I would really like to know how in the world they spent 11M on the uk one...
I don’t know specifics about the UK version, but most of those costs likely weren’t on the app proper. They did run a field test, for example.
For comparison, the German version cost millions, too. That included multi-language support, setting up help lines, security evaluation, load testing millions of concurrent users and, IIRC, connecting to quite a few of different hospital computer systems.
Add in time pressure, and costs go up. Germany did get a quality system, judging by what the spokesman for the Chaos Computer Club said (paraphrasing: ‘we have the unfamiliar problem that there is so little to complain about’. They never endorse stuff, so that’s about as endorsing as their statements can get)
It also includes the DPIA, which in itself can be an expensive process. There was a lot of concern/press in Ireland pre-release about the privacy implications of the app. It's hot on the heels of a "mandatory but not compulsory" national ID card fiasco that was determined to be unlawful by the privacy regulator last year: https://www.thejournal.ie/psc-no-legal-basis-4766822-Aug2019...
We ran a field test too. But again; I probably can’t talk about that much. Sorry. If you have more in depth technical questions I might be able to answer them.
We had a team of great devs and designers who worked on this. What most would consider “par for the course” in relation to an MVP startup for instance.
With these things, you're really trying to get saturation (I think it's usually estimated that they need to get to about 60% adoption to be very useful), so in that context 1.5 million downloads in Ireland is more successful than 100m in India (though neither is good enough).
That’s a little misleading though because the percentage of people who install the country’s contact tracing app is driven much more by the quality of the people than the quality of the app. So this app might be garbage, I don’t think this statistic says much one way or the other.
I think it's more about public trust in the app than anything else. Nearform, and the government, were quite effective in establishing public trust; I think it helped that it was open source.
There's still a fair amount of people that are suspect that the published binary doesn't match the source and the app could be doing nefarious things. It will be interesting to see how this sentiment progresses. Reproducible builds would help in this regard.
Really, I work in the industry (it) and haven't known one person to raise this.
I would say the majority of people in Ireland know the various social network apps are a bajillion times more intrusive than the covid checker. Im fairly sure most people "trust" the government not to be "spying" on us
You could write the cleanest, most bug-free and user-friendly app, release it in the US, and likely never see this level of adoption (I believe is the point that is being made here).
So far it seems like a given country/jurisdiction standardizes on one app for everyone to use. (In theory, apps could be interoperable, but I don't know of this having happened.)
So, the citizens' choice isn't between one app and another app. It is only whether or not to use the app that has been chosen for them.
Therefore, their choice doesn't tell you much about the quality of the app. The primary driver of the decision is whether or not they're willing to take steps to fight the disease.
It's similar to being served dinner on an airplane with only one meal option. They bring you the food, and you either eat it or you don't. The choice doesn't tell you much about whether the food was good. It's more of a reflection of whether you were hungry. The only thing it tells you about food quality is that it (if you eat it) it was somewhere between barely acceptable and excellent.
> The primary driver of the decision is whether or not they're willing to take steps to fight the disease.
Here in Australia, our gov is extremely incompetent at delivering successful IT projects. They also have a terrible track record with securing data, scaling services (eg national-level IT outages), don't admit to mistakes, etc. :(
> It's more of a reflection of whether you were hungry.
No. To go with your example, if you know the people preparing the meal are incompetent, don't follow anything like reasonable hygiene standards, have previously caused significant outbreaks of disease themselves, and each time they're caught they promise to do better (but actually don't do so)... would you eat something they've prepared for you? Especially prepared at short notice where corner-cutting is expected? ;)
Trying to say that only idiots (etc) wouldn't install such an application is seriously wrong. ;)
It's an unjustified claim and a little demeaning, but not obviously wrong. But citizens choosing to believe in the effectiveness of digital contact tracing is a far larger variable in the success of these apps than the quality of the app itself.
So the total cost will be 50-100 millions. Despite the fact that most technical work was already done by Apple and Google for free.
I‘m not even talking about the fact that my dad got a new phone to run it - sold by T-Mobile (Telekom) of course.
And hardly anyone complained.