Hacker News new | past | comments | ask | show | jobs | submit login
AWS now allows customers to pay for their usage in advance (amazon.com)
153 points by msmithstubbs on July 21, 2021 | hide | past | favorite | 182 comments



I really wish you could just designate a group of resources as unimportant, set a billing limit, and let Amazon nuke everything / delete your files / whatever, if you go over the limit. Everytime I try to learn cloud infrastructure stuff I'm terrified of the literally infinite bill that might show up from a typo a month down the line.


> I'm terrified of the literally infinite bill that might show up from a typo a month down the line

Whilst this might sound funny, we were surprised to see it as a common use-cases with users putting https://github.com/infracost/infracost in their CI/CD pipelines to act as safety net. Currently it only works for Terraform users, but we plan to add other infra-as-code tools in the future. We're also discussing how we can do this for people who don't use infra-as-code in https://github.com/infracost/infracost/issues/840 but it's not clear what the workflow could look like for them. Perhaps having separate AWS accounts with a budget alert that emails you to run https://github.com/rebuy-de/aws-nuke is a work-around just now.

(I'm co-founder of Infracost)


> Perhaps having separate AWS accounts ...

You absolutely must, MUST, MUST be using separate AWS accounts for separate purposes. You can have as many as you’d like and roll up the billing into one actual paying account.

This is a win for accountability (roll up dev and easily see the split out for separate environments), but more importantly for security as it limits the blast radius for any one environment. Combined with per-account budget alerts it’s a win across the board.


It may be a 'must' for security but from a UX perspective it is a horrible experience.

Does it make sense for one team to have 10+ AWS accounts per service because 'security'? How about if each team out of 1000s in your company has 10 AWS accounts per service?

We run our service in 3 geographic regions and have a separate AWS account for each region and stage despite each account supporting resources in multiple regions. Considering that we have 4~ services that is roughly 40 AWS accounts for just one team with less than 10 people.

What I'm describing above is the 'best practice' way to manage AWS accounts at scale. It is insane and saying 'security' does not magically make this reasonable.


I was so happy when I finally got cross-account roles working so I could use a nice drop down and seamlessly switch between my accounts. So cool!

Then I learned because they’re saving it all browser-side I had to rebuild the whole menu whenever I first used a new browser or computer? Whaaaat? Of all people, AWS console users have to be highly likely to be using multiple devices/browsers. Having to recreate your own prefs at each new environment is nuts.


Not to mention that the there is a pretty small limit on how many can show up in the drop down (I don't remember how many) so it isn't very scalable if you follow the recommendations to create a lot of accounts.

Plus you have to look up the account id in order to set it up initially.



Second this recommendation. Our TAM suggested it and it works great. You can even install the Chrome plugin in the Edge browser.


This add on is highly recommended inside of Amazon and AWS, too.


The UX issue you're describing...can and should be solved with UX.

While security and UX are oftentimes in tension, in this case they don't have to be. It would not be that hard to be signed into multiple accounts and allow you to switch seamlessly between them (allow the tagging of each account, such that you can say, effectively, "show me dev us-east-1" vs "show me us-east-1" vs "show me dev", slicing and dicing between accounts that way). At that point, separating infra across accounts becomes semantically meaningful, and you can slice/dice in whatever way seems best (so you could have a full account for a single service, sure. Or an environment. Or a region. Or a combination of those, only service-Foo in us-east-1 for dev. Whatever level of granularity you want; trading off instead between the security of isolation with the convenience of colocation, which should be the actual UX cost; infra in the us-east-1 account has a harder time communicating with the infra in the us-west-1 account).


I already set this up. My customers are 5-10 man shops, and they have 5 different AWS Accounts: One for billing, one for Build Infrastructure, one each for Dev/Staging/Prod. Sometimes marketing is treated as a separate product team and their website has it's own staging/prod accounts (No real need for "dev" in that case).

Users login to the Build Infra account and then Assume Role into the others - There's a list of magic links that does the assume role. There's also a list that is added to ~/.aws/config that does the equivalent: They configure one IAM key, and the rest are assumed automatically by the CLI or client libraries (Requires relatively recent client libraries; Java only started supporting this within the last year or two)


Many people shit on GCP, but this is an area where they really shine.

You can set budgets by project and easily allocate costs and address accountability issues across teams or products.

The value depends on how you operate.


I happily use 40+ accounts per service, and don't think it's an undue burden. Accounts are free and represent a convenient natural boundary for data, access, and oopsie-daisy mitigation.


"Undue burden" and good UX is a wide chasm. In the last few months of my last role we went from the 1 account to multiple separating environments and the additional cognitive load and extra work migrating was not trivial (plus trying to keep costs down duplicating things for migration).

I can see how starting with a pattern of "account per X" would create intuitive boundaries. When you say "per service" what kind of service do you mean? Business related web service API? AWS product? Other? Interested in what boundary line made sense for you given the large number of accounts you say you're happy with using.


I would say one account per business-related Web service, per stage, is a sensible way to break it up. That gives you some visibility into what services are talking to what other services, logical cost breakdowns, and access controls pretty much where you'd want them.


With one giant caveat imho — I have a root account, an admin account, a common account (load balancer, database) and then customer-specific accounts. Was working great, using Terraform for consistency, sharing VPC where made sense, etc… until I had an issue and realized that my paid support plan only covered the root account. From what I understand you have to get a separate support plan, with a paid minimum ($100 per for business plan), for each account if you’re gonna need tech support, and you can’t pool until you’re in the $15K+ monthly spend: “AWS Support fees are calculated on a per-account basis for Business and Developer Support plans. For Enterprise Support, you are billed based on the aggregate monthly AWS charges for all your account IDs subscribed to Enterprise Support.”

Really soured me on the setup, tbh.


If you roll them all up into one Organization, then everything should be properly aggregated.


This is true. It does add additional complexity, especially if you have to do cross-account access, but the tooling for that is improving over time.


This seems silly to me. I (personally) think it is much more likely for your computer to be stolen/hacked/ransomed than a single account credential to be leaked. If so, "the blast radius" will be whatever you're logged into ... and if you're logged into everything, what's the point?


For AWS, blast radius includes things like “developer fucked up on using SSM and exceeded the rate limit for the entire account”. Or “developer failed to set API Gateway rate limit on their trivial app and brought down everything else in the account”


Because you should have 2fa set up and your access to AWS accounts should expire after 1 hour. Also, you likely have full disk encryption enabled, and the person stealing your laptop is unlikely to know who you work for and are more interested in selling it.

If someone acquires credentials, they are usually multi use and long term. And it can go unnoticed if an ec2 instance is span up running crypto mining on your dime, only for you to notice at the end of the day that your estimated bill has shot through the roof


I think most of the cost for medium-large sized business are elastic(number of pods, bandwidth cost depends on requests per second, storage cost for many things increases linearly with users etc).


Yep - it seems to depend on the architecture too (e.g. companies that lift-and-shift to the cloud use VMs heavily). We're discussing ideas on https://github.com/infracost/infracost/issues/730, e.g. could CloudWatch be used to fetch the usage so user has context of what those elastic services used last week/month.


Didn't imagined that this functionality would be present. Looks very useful and I would try it out for my terraform setup!


Unfortunately it's also not just resources that cost money.

A couple of fun billing surprises I've seen.

1. A bug in a system that uploaded quite a lot of data to Amazon S3 caused it to hit the S3 API to the tune of about $10K/day. Because AWS billing is usually 2-3 days lagged, it took us 3 days to notice. We fixed it right away once we found it. Goodby to that $30K.

2. An engineer did an Athena query that happened to walk many TB of data. And they unknowingly did it in us-west-2, but the data was in us-east-1. So that resulted in a cross region transfer to the tune of $10K for that single query.


Did you contact support in either event? Just curious if they were at all willing to adjust your bill.


I think GCP's official method for doing this is pretty similar to what you describe. You basically create a cloud function that disables billing if your bill goes over a configured limit. It's not perfect, because there's a tiny bit of lag between usage and billing calculation, but you'll only end up with a few dollars over the limit instead of thousands. Truly the nuclear option though.


I did this last year for my project, except instead of disabling billing which would nuke everything, I wrote a service that runs every day, looks up my remaining monthly budget and sets the daily quotas on the APIs I use so they can't use more than my budget. (Which wouldn't be necessary if they offered monthly quotas to match the monthly billing period, but they don't.)

Then last month I got an email saying "Hey, those quotas you were setting using the API documented to set quotas, those were actually not being enforced the whole time because of undocumented issues with our systems." So basically you can't rely on the documented behavior of these systems, there's no good way to test whether your code is correct or whether your limits will work without actually exceeding your budget for real, and the whole thing is a clusterfuck. When you get a surprise bill you just have to throw yourself at the mercy of whichever first line billing support rep is randomly assigned to your case.

Limiting your bill to something less than "potentially infinite" is just a basic fundamental feature that shouldn't require rolling your own bill-monitoring service relying on poorly documented and malfunctioning APIs with no provision for testing. There's no excuse strong enough to explain why the cloud providers can't do something reasonable here.


And this is something that should've been added years ago. How many people have decided not to use these services because trying things out to learn seemed too risky? They're not going to gain these skills either, so they argue for alternatives when they actually need these capabilities.


This official method is so broken that it's embarrassing that they recommend it. It looks like a solution, but it doesn't work.

The "tiny bit of lag" between usage and billing calculation explodes when there's a lot of usage - in my case, a broken job tried resubmitting itself continuously, and the lag increased to 8 hours and $5000 just when I needed the alert the most. My team's response time was 5 minutes... After the 8 hour GCP lag.

Very similar to this guy's story: https://blog.tomilkieway.com/72k-1/

I had to go back and forth with them on email for weeks, and ultimately threaten them with a draft blog post with a lot of graphs and screenshots of their recommendations for them to cancel the bill.


Oh, on the GCP story I was always reminded of this:

https://blog.tomilkieway.com/72k-1/


Wow, well they had some pretty fundamental design problems that the author points out. Infinite recursion due to back linking is a pretty easy way to max out your bill. I'm glad that Google forgave the bill at least.


> GCP's official method for doing this is … a cloud function that disables billing if your bill goes over a configured limit

I’d love it if GCP’s official method were to disable billing if your bill went over a limit.

Sadly, I suspect it would just disable systems instead.


How does "disabling billing" but not "disabling systems" work?

Is this like asking the phone company "When I reach my plan limits, stop charging me money but let me keep making calls?"


That was my point.

I think GP was incorrect in the mechanism there.

I quoted GP verbatim, they said disable billing rather than disable service or delivery or system.


Yeah it has firmly kept me away from AWS, Google cloud, and similar.

I use Vultr or Digitalocean if I need a server somewhere because at least it's just a pre-set cost.


Absolutely this! The uncertainty freaks me out. I don't have money to pay thousands of dollars as a result of some bug or attack.

I'm in love with DigitalOcean because you know the price you'd pay each month. If it's just 5 USD or 5000 USD, it's what you expected, nothing more.

I believe the rest of the clan (Linode, Vultr, etc.) give you the same certainty.


> I use Vultr or Digitalocean if I need a server somewhere because at least it's just a pre-set cost.

To be fair, AWS Lightsail should be an option too. Lightsail machines come with a fairly competitive amount of bandwidth.


Can you not get charged for bandwidth overages? What happens if you use 3x your transfer limit?


My VPS shuts down and I get an email at 95% usage I think, I can turn it back on if needed and pay for more bandwidth.


> I really wish you could just designate a group of resources as unimportant, set a billing limit, and let Amazon nuke everything / delete your files / whatever, if you go over the limit.

That's exactly how it should work. It would even be useful if I could designate my development / testing account as unimportant by default so everything can be nuked to limit spending.

I think it's the kind of thing that will only be solved by regulation. The government needs to institute the concept of capped overages for cloud providers where if I set my budget to $100 / month they aren't allowed to send me a 100x bill for $10k.

Here's the 9 year old request for the same thing on Azure.

https://feedback.azure.com/forums/170030-signup-and-billing/...

I politely asked @AzureSupport (on Twitter) if they could have someone provide an update, but they didn't deliver on their promise to follow up :-(


If not possible to cap price, starting with the capacity limiter on S3 and bandwidth limit at VPC level would do.

The possibility that someone flood the server even for static resources causing bandwidth spiked Bill is scary.


> The possibility that someone flood the server even for static resources causing bandwidth spiked Bill is scary.

Genuinely curious, is this just a side-effect of the cloud craze or did DDoS attacks become so powerful that old-school approaches of appropriately-sized bare-metal infrastructure with finite but unmetered bandwidth are no longer viable?

The way I see it, you can provision enough unmetered bandwidth to cover your typical load + a safety margin at a flat rate per month, and worst case scenario if the attack is big enough you merely get downtime (allowing you to re-evaluate the situation and decide whether to throw more bandwidth at the problem or purchase attack mitigation services) instead of an infinite bill?

My current ISP gives me 1Gbps unmetered. Worst case scenario the connection is saturated but at no point the ISP will come to me and ask for extra money.


You could still run many systems just fine on private infrastructure with at most a business-class Internet connection to your office or a colo bill for putting your servers somewhere more central. This didn't magically stop working just because someone got paid a lot of money to do PR for cloud services. By the time you take into account the financial costs and inherent risks of cloud hosting, maybe more things should still run that way than actually do.

The practical problem today is that cloud now has so much mindshare, justified or otherwise, that the ecosystem around private hosting is diminished. Finding good people with the required admin skills, good sources of equipment, even good software to run local versions of automation we take for granted in the cloud, can be harder than it used to be.

I won't be surprised if in a few years some huge tech firm we all thought had faded into obscurity enjoys a new lease of life by offering a set of locally hosted equivalents to popular cloud services that are also easy to administer and scale but come with a lot more predictability because they run on the customer's own infrastructure.


We still use bare-metal at Automattic. All our global-scale admin stuff is open source... it shouldn't be surprising that bash scripts aren't all that interesting. People want it written in Go, with Raft-consensus to think for us humans, running on blockchain.


Are they published somewhere on https://github.com/Automattic?


No, it all predates git/Github: https://code.svn.wordpress.org/servermattic/README

If you go up a few levels you can find more interesting things as well...

Someone wrote an good tutorial on using it here: https://codeseekah.com/2012/03/19/cross-server-deployment-wi...


One big problem with that is the dichotomy between "cloud" and "open source" - people will pay for SaaS but they absolutely balk at paying for licenses.


In this hypothetical scenario the real money might be in consultancy. "Sure, we can get your organisation set up with OpenNotAWSBecauseTrademarks. Our rates are $20K/consultant/week and we expect to bring a team of 5 for a fortnight." It just has to be a comparable cost and financial structure to how a large organisation trying to escape from cloud lock-in would have otherwise expected to engage their cloud architecture consultants or cloud security red team or other cloud specialists and then you're in the game.


Technology is a good business because a small labor input can scale to a very large impact. I'm sure there is a place for consultancy but I don't see it winning against "scale" in the long term.


Licenses are a major PITA when you want to be spinning machines up and down all the time. Some enterprise vendors have pay as you go solutions, but many don’t.

I get the impression that some enterprise vendors don’t offer pay as you go solutions because it would put their sales staff out of work, and because they wouldn’t be able to use a “how much can you afford?” pricing model.


That threat even has its own name now: a denial-of-wallet attack.

The limited protections available against this threat from the big cloud providers have to be seen as a warning sign. It's only a matter of time before any small business using these services for hosting can be subject to sudden shakedowns by criminals. "Nice business-critical infrastructure you have there, be a shame if anything were to happen to it." Some of the providers do offer a DoS mitigation service, but the cost for the higher levels can start to look like a shakedown itself.


Set an SNS alert to sent an email/SMS message to your phone if your monthly bill goes over whatever $X you decide. I've had this set on my personal account for years and it isn't too hard to configure, most of it is just point and click via the SNS and CloudWatch GUIs and is pretty foolproof.


From all of the horror stories I've heard, it is not foolproof. For one, don't you get notified after the usage and charge happens? So one mistake that causes a large spike and the notification is too late.


This is true, it isn't really possible to get a near instantaneous real-time feed of every single charge from all of the different AWS services you may be using, because they are all unrelated and do their logging / billing differently. IE EC2 will scrape and upload your iptables data-usage info to s3 and then that will get scraped and generated into a daily billing/money report etc, and there are thousands of things such as this between all of AWS services.

This likely will just alert you somewhat quickly after something has spiked and been running for a number of hours/day, most likely.


Single most obvious customer obsessed (their tenet BTW) feature they could add, but after over a decade of requests, it's seemingly clear they won't. It keeps me from playing with AWS for side projects as well. Their loss.


This is something that everyone seems to ask for (I know I'd love it), but they haven't implemented it. To me that suggests that they _can't_.

My guess is that billing lags enough that they can't stick to a price cap, which means that they either have to guarantee the price cap and swallow the difference, which could be exploited by malicious users to get free compute, or they have to say that there's a delay on it which makes the cap fairly useless.

Some of these services are billed by such small increments I can't even imagine how complex billing for them is in practice. I'd be surprised if bills are eventually consistent within 24 hours.

I wouldn't be surprised if we see an announcement like billing being guaranteed after 1 hour at some point in the not too distant future, but I'd be surprised if we see realtime caps.


I’m on the Bill Computation Team for AWS, and this is exactly it. The work behind the scenes is insane. BillComp is the oldest and most brittle part of AWS. There are modernization efforts underway, so stay tuned.

Trust me, they hear you.


> This is something that everyone seems to ask for (I know I'd love it), but they haven't implemented it. To me that suggests that they _can't_.

Or maybe it is a costly implementation that would not bring any profits.

The strange thing is that the lack of this feature seems too incur a cost as it causes more calls to customer support. So, maybe it's that implement this feature will reduce profit more that it will reduce cost.


When I fill my tank with gas, there's a preauthorization with my credit card before I'm allowed to pump a single drop. It seems like a similar arrangement could be made here w/r to hourly level billing. And it would be a huge improvement over the current situation which scares me away.


Oddly enough, Budgets seem to work, since I've gotten alerted to runaway services fast enough (I set it at 80% of my previously-free monthly AWS credits) to be able to log in and fix them, or shut them down.


I think the same, it's put me off using anything but the free tier for learning. Azure was slightly better but still not ideal.


I've read that some people use a pre-paid credit card with a $1 spending limit when setting up their playground accounts. Seems like a reasonable approach.


You will still owe the incurred charges and AWS can send it to collections.


AWS is oddly dysfunctional recently.

They nerfed the $100 of AWS credits for Alexa developers with zero notice this month, which caused me to incur overages this and last month.

I've gotten last month's bill waived, but still received a passive-aggressive email with bad English by a Territory Account Sales person from my region about how my account could be suspended, if I didn't reply to the email within the day. I'm not sure I would trust said person to handle my accounts, even if I was on a corporate budget.

I'm still in the process of moving most of my workload away from AWS.


I do this. I’d much rather have AWS needing to call me to negotiate / collect than having $15k go through my CC as a legit authorized charge.


Unless they call you, refuse to negotiate and still send it to collections as it is (at least in their mind) a legitimate charge.

All these stories of providers giving "good will" credit for these massive charges really concerns me when you look at how other parts of these companies ignore their customers or only reply with scripted responses.


That doesn't make much sense. You would still be on the hook for the eventual bill. This sounds like a showerthought hashtag lifehack.


It does change the dynamic / comfort though. Would you rather ask AWS to please revert $5k they put on your card, or talk with them about $5k they'd like to charge you but can't?


It doesn't change the dynamic though.

At their revenue, don't care about 5K charge, they can send to collections / sell to 3rd party collections agencies.

They do care about keeping you happy as a customer since your employers will be swayed by their employees.

So the former is much more likely to succeed, the latter will just make you look like a scammer.

At larger sums - they will do much more rigorous checks to avoid issues.


It doesn't change the dynamic for AWS. It doesn't change for many of us. But it does for example for a student who forgot to terminate a stack and suddenly can't afford rent/utilities/shopping until the charge is resolved. These are amounts which can really mess up people's lives for weeks.


The former. But if we're talking about $5M instead, I'd be completely terrified of both options.


Oh yes, please. And to all the other commenters that suggest workarounds: Yes, better than nothing, but not exactly a solution to get beginners on board. AWS is complicated enough even without all the billing headaches.


This is probably what I find most frustrating about the way things are done now. I'd really like to get some idea of how these cloud services work. Every time I'm stopped by the concern that I'd rack up a huge bill and have to rely on the goodwill of a large corporation to have the debt forgiven.

I understand the role and the necessity for "the cloud", but it's a re-invention of the role of the mainframe. I hate seeing one of the most notable aspects of the microcomputer era go away which the ability of a motivated individual to gain computer skills using an individual's resources.


> a re-invention of the role of the mainframe.

A publicly accessible mainframe, where anyone anywhere in the world can script the provision of machines and other resources with little more than terminal and a text editor.

That would have been utopian science fiction in the heyday of the mainframe.


This is not to be dismissed. AWS billing is too opaque[1] and a misconfigured or buggy plugin can very well trigger resources infinitely[2] to a big nasty surprise.

[1] https://news.ycombinator.com/item?id=21694835

[2] https://wordpress.org/support/topic/amazon-cloudfront-invali...


If this is an issue, use Lightsail or a tier 2 provider (like DigitalOcean)


That doesn't solve for the AWS only resources.


Yeah, no fixing that.

Billing can be 24 hours delayed.


Then you aren't learning AWS, which was the stated goal.


This terrifying scenario is kinda common. We've come across a bunch of tweets like: https://twitter.com/alexwlchan/status/1399095011178958851

This inspired us to add billing limits to our SaaS product so that users don't have be in scary situations with bill run offs: https://mediamachine.io/blog/protect-your-customers-with-bil...


This is the reason I have always stayed away from AWS and stuck to Digital Ocean/Linode. I'm sure I'm not the only one. But I am always surprised to see people complaining about this and still using AWS.


That fear of a huge bill is real and much more common than you think.


It's a rational fear as well. It happens more often than one would think.


I think confusion around billing has to be intentional at this point. I would guess they are making >$1b every year due to users not understanding the consequences of their actions fully.


Just use the free tier? You’re notified when you’re approaching the free limit.

AWS, anecdotally, has removed 5k++ mistakes I’ve made with little question.

(One example they forgave due to my carelessness: ECS and Fargate service with logging to CloudWatch but with verbose logging on. The bill was 8k that month for just CloudWatch usage)


I have only asked for one refund, which was clearly the result of a bug on Amazon's part, and they haggled the whole way. They were quick to a 50% refund and slow to a 100% refund.


I've never had a refund denied. One was for 20k on an account that only billed that much monthly. If it's an honest mistake they'll wipe it if you have any history with them.


I've had $30k, and later $120k refunded on an account that billed ~$20-25k monthly. Both covered 100% of the overage.

AWS is the one major tech company where I've never had any issue getting in touch with a real human who has been empowered to actually fix my issues.

The only thing that's been required from us was to show them we were taking reasonable steps to prevent it happening again.


It's great that they forgave you. I know a startup that incurred a $30k bill that they didn't forgive. The startup folded.

AWS's unknowable policy for the cost of errors represents a huge risk for individuals and small businesses. It puts a lot of people off.


I'm sorry, what?

> AWS's unknowable policy for the cost of errors represents a huge risk for individuals and small businesses.

Assume you're paying for your own mistakes, and be thankful when you don't have to? That's a pretty easy policy.


AWS billing practices are horrible, and they are increasingly more “Oracle” like in their approach.

I had a security issue related to a SaaS product which led to a $7k AWS line item when someone started sending a LIST request to S3 buckets billions of times. They would not consider refunding.

Now I’m having a bunch of problems terminating some AWS Orgs accounts and they are being deliberately difficult in getting it tidied up whilst I’m incurring significant costs.

The whole billing stuff is complex and opaque and there aren’t enough controls and limits on spend. I feel like I need to dedicate 1 x FTE at least on AWS cost control which is a high cost for a small business.

As a CTO, I’ve previously influenced $millions in spend on AWS, but would be very nervous putting my reputation on the line to spend big with them in future. I’m frankly losing trust in their commercial approach.


Anecdata, but my experience as CTO of a startup, a hedge fund, and a bank has been the opposite.

I’ve never had an unexpected cost they didn’t readily credit back, provided we were taking the recommended and reasonably easy steps to keep on top of costs and limits.


The problem is relying on this "good will" and "one time only" to credit back compared with having a way to set hard billing limits so you don't need to have this conversation as a part of your business as usual. Mistakes will always happen with something as complex as this and that's what billing and rate limits are supposed to protect your against.


I made a mistake with glacier (old transfer pricing model was horribly unobvious, I think that they changed it since then) which costed me few hundred bucks instead of expected few pennies. I asked for refund, because I read about people in the same situation being refunded but all I got from support is pricing page and FAQ link. I don't expect any goodwill from Amazon, I'm not the kind of person who would fight over refund, so I just paid and forgot about it, but had some bad taste in my mouth.


I used to work at AWS, and my experience helping customers with these types of issues was almost without exception a credit/refund would be applied for any honest mistake that had corrective or preventative steps already in flight.

I say almost without exception, because the one case that wasn’t true was a Glacier transfer case like you described (except an order of magnitude larger in cost). We made it right for the customer in other ways. But I’m still seething years later about how poor and experience it was and how uncharacteristically unmoving and not customer obsessed whatever the decision making chain were on that particular issue. Just wanted to let you know you’re not alone, and it’s not just customers that had a bad taste from that experience.


Whats your monthly spend? I used to work for an org with 50K monthly spend none cared at AWS about us. Now I work for a big org with very serious spend and it's night and day we can get access to eng. quickly we have regular meetings with PMs and get our requests for AWS features put onto roadmap etc.


I bet that's the case for the GP. They probably spend millions of dollars, so they get catered to and think it's the same treatment normal people get. I call it VIP vision. People don't even realize they're getting special treatment and assume their results are merit based rather than money based.


I mentioned 3 wildly different orgs for that reason.


We recently helped a small client of ours discover a cost increase where AWS RETROACTIVELY increased their costs for a service near the end of the month for previous days without letting them know.

We were a bit shocked to see this happen and it was a very subtle increase that was sort of hidden in Cost Explorer unless you spent hours digging into it and comparing your past invoices.

(I'm a co-founder of CloudForecast)


Extraordinary claims require extraordinary evidence.


I don't think it's an extraordinary claim but some evidence would be nice because the comment made me paranoid


I run at AWS for 5 years and use more than 25 services and keep an eye at a huge sum very carefully year never seen this happening so it's pretty extraordinary for me. I dont ever remember AWS increasing prices for any service at all.


It really is. AWS never increases prices, let alone retroactively. They make such massive margins where they do make money they absolutely do not need to use underhanded tactics to milk their customers who are more than happy to hand out money of their own volition.

They give out hundreds of thousands of dollars in credit just so you can use their crack.

The GP's comment's claim isn't just extraordinary, it's out there with "I saw aliens and they probed me". Possible? Physically, yes. Unlikely? Quite the understatement.


Which service and whay API?


What was the service that they retroactively increased the cost of?


At least you can do something about open S3 buckets with relative ease, I got slammed by people scanning zones on Route53.

The way Route53 pricing works, you could get a 1000$ bill any day - for DNS... Having just a single domain on there is enough.


> and they are increasingly more “Oracle” like in their approach.

Ironically the Oracle cloud seems more price-reasonable (for now).


The Oracle approach is:

- Good terms with proprietary lock-in.

- Milk that cow for all it's worth.

It's more nuanced than that -- I gave an oversimplified model -- but I've never seen anyone come out ahead doing business with Oracle long-term.


What's the point, if there isn't a discount for paying upfront?

Will some people/businesses prefer this because it's not 'credit'—does AWS scrobble to your Credit Report in any country?

I am failing to see the appeal here...


For companies operating on a cash basis with a standard Jan-Dec fiscal calendar (e.g. most small businesses), this would allow you to deduct future spending by prepurchasing AWS credits. It locks away whatever money you dedicate to it but that’d be peanuts compared to paying income tax on it in order to carry it forward as retained earnings.


I don't think that works the way you suggest, but I also admit the guidance is unclear.

Reg. Section 1.461-1(a)(1) provides the following:

If an expenditure results in the creation of an asset having a useful life which extends substantially beyond the close of the taxable year, such an expenditure may not be deductible, or may be deductible only in part, for the taxable year in which made.

https://www.law.cornell.edu/cfr/text/26/1.461-1

If you buy 10+ months of AWS credits in December and have a Jan-Dec fiscal year, I'd argue that you bought "an asset having a useful life which extends substantially beyond the close of your taxable year"


This isn’t purchase of a capitalizable asset, it’s renting as an operational expense ;)


Why not use a dedicated escrow service for that, which wouold work with all expenses, not just AWS?


If it smells like a checking account then it’s going to be treated as a checking account.


This is for when the departmental budget has a little cash left at the end of the fiscal year and they need to spend it.


I worked on this feature when I was at Amazon—and the demographic we were after were mostly government & some non-profit organizations. The biggest thing with these orgs were that they _needed_ to have a clear and structured per-month budget over a long period (~year) OR they had to use up their funds before their grants “expired”

Also on a technical note, this allowed for some nice internal data models/patterns that could be utilized in further use-cases


I work at AWS, but I wasn't involved in this feature, so this isn't anything more than speculation on my part. I've certainly talked to customers who would time their reserved instances and savings plan purchases based on the USD exchange rate for their local currency. This could make sense for those customers too, who often don't have USD denominated bank accounts.


Other comments have covered cases like departments having money left over in their quarterly budgets, or companies looking to spend in a particular quarter for earnings/tax deduction reasons, or reducing currency risk by hedging forex prices. But the biggest use by far that I've seen for this is government/public orgs that are prevented by outdated laws/auditing regulations/processes from using pay-as-you-go models. They are forced by their accounting department/government grant to treat infra expenses as capex and have zero budget to expense them as opex (this model assumes an on-prem physical plant for an IT department). Previously AWS had a way to get around part of that with reserved instances, this solution is more comprehensive.


The pricing on reserved instances is so appealing over on-demand instances, though, that people are using it for more than just opex vs. capex accounting. You legitimately save money by buying in advance.


> What's the point, if there isn't a discount for paying upfront?

In a past life, I did some work with government clients who preferred to be charged up-front in a lump sum, because it was much easier for them to get funding for that than a recurring subscription.


I like those kinds of services for personal projects. I don't always have enough money on my bank card and I'm lazy to go to ATM to fill my card with cash, so those monthly payments could be missed very easily. If that leads to service disruption, it's very annoying. It's much easier to load a balance for year and forget about it until next year.


Our bank recently started charging us negative interest on any balance over €150K in our checking account. So I wouldn't mind pre-paying a bit if the balance gets too high. Alas, it seems this is US only for now.


Even though I'm no where close to the world's best AWS developer, I happened onto a contract for a major stock exchange and ended up writing code that truly commands fleets of computers. What I learned: There are plenty of ways to optimize your AWS spend. Hire consultants, especially ones that do so on commission. There are so many tricks that work 99% of the time and will save you a ton. For example, read-before-update on DynamoDB. Puts are expensive, reads are cheap. Depending on your data you may be able to do the diff in code and only push the delta. There are many other optimizations. If you're a growing business it helps to get help with this stuff. I never would have guessed were it not for the pros that worked for my client.


I’m curious about infra of stock exchanges these days. Any resources you could point to? I’m especially curious how they handle sudden volatility and spike in trading. We usually talk about high frequency testing but very less about the infra that consumes all of those trades


I'm under a couple strict NDAs, unfortunately, and I didn't work on the trading engine, just software around piping giant amounts of data around to a ton of consumers. But the stuff I worked on heavily utilized AWS wherever possible and the emphasis was on prioritization. No matter what it did to the code, if high priority data came in the swarm of computers had to switch to it as soon as possible.


Does this mean I can set up a static website on S3, pre-pay fir the next hundred years of hosting costs and then pretty much forget about it? Because I would genuinely love to be able to do that.


No. S3, like most AWS services, has uncapped costs. If you experience higher than expected load, such as a DDoS attempt, you'll burn through the preallocated spend and you'll still get a bill afterward.

This doesn't appear to actually shut down the resources once the preallocated spend is exhausted. Its just a way to pay for bills preemptively instead of when you receive them. Its an accounting thing, not a new feature.


Just put your S3 bucket behind a CloudFront CDN and you're golden :)


I think you could, yes. It’s a different question as to how fast you’d hit the limit, but definitely possible to do a “this site can only have 100000 visits” type art project.


I've also been thinking about that! I wonder if https://archive.org/web/ is an alternative though, as in could I pay them so they could mirror it for a 100 years?


I would absolutely love to be able to donate a domain name to the Internet Archive plus a lump sum cash donation and have them keep it hosted in perpetuity.


Sign me up too, I've got a (very small) site that I would like to outlive me; my plan is to attempt to set it up with a large balance at NearlyFreeSpeach.net and also put the account identifier in an HTML comment so that motivated people could increase its balance in the future.

I would be very interested in other credible perpetual hosting plans.


You'd probably be better off signing up for an Oracle Always-Free tier as there's no billing information stored should anything run into costs. But as the name implies, it's always free, so your performance, bandwidth and space allocation is substantially lower than the paid options.


Yes, but no. You could pre-pay for the next 100 years, but there’s no guarantee you would get 100 years of service. Nothing stopping AWS increasing prices during that period, and you’d be subject to those increases just like everyone else.


Oh I have been thinking about something similar as well. Basically a I buy a single HTML page that will last at least 20-30 years.


You can do this today with the decentralized cloud: https://docs.storj.io/dcs/how-tos/host-a-static-website/host...


Of course it has its own cryptocurrency.


what are the odds that that service exists in 5 years? Or 10 years? I'm confident AWS will.


Not much of a feature.

If that could be used as a hard limit that would be more interesting


I wish Digital Ocean would allow this. My country's debit/credit cards don't work online reliably, my attached cards can start getting rejected randomly any time. I'm always nervous about getting my account suspended due to missed payments, DO is pretty forgiving thankfully.


Interesting, I had the opposite experience. The cardholder forgot what Digital Ocean was and placed a chargeback. Do immediately locked my account which had been in good standing for years. I couldn't log in the console or API to do anything. I wrote about it here if you're interested to learn more: https://news.ycombinator.com/item?id=25806086

Linode is very similar pricing/offering and has incredible customer service. I'm very happy with them.


Linode allows you to pre-fund your account.


They do with PayPal at least.

Their emails even use language like "you need to top up your account".


Why are the top comments companies promoting their solution? Don’t get me wrong, I think it’s find to do so I just don’t expect them at the top.


I don't see any top 1st level comments promoting anything. The only promotions I see are comments to the top comments which is hard to avoid!


Demand for a solution is probably quite high.


This has been increasingly prevalent on HN, and I'd (eventually) like to see something done about it. Sure, Hacker News is a project incubator at heart, so it will naturally have a higher ratio of CEOs:normal_users. That doesn't excuse how obnoxious it is seeing someone plug their SAAS-of-the-day on seemingly innocuous information (like how Fig hitched a ride on a Brew PSA).

It's frustrating me to the point where I might just leave this site. I'm sick and tired of this new-wave guerilla marketing.


Cheaper to park your money to AWS, rather than pay negative interests on your bank account.


I can't wait until I can trade my credit with other AWS users.


No mention of discounts, so this is probably a purely cashflow / tax management system.


Discount?


Is this for Tax benefits? Where you could put in all your annual net profits for AWS credit?


Not sure about those but it'll be incredibly useful for research grant funding monies. Most research grants are "use it or lose it" so if you have any essential infrastructure, capital with short shelf lives/frequent replacement needs, etc. you want/need after the end of the grant, you pay for it in advance.

A group I worked with bought about 5 years worth of a specific consumable they needed to continue working, 2-3 year service contract with a vendor to maintain aspects of things so some work could continue and be leveraged for future grants, and hosting/software licenses were often purchased for long time horizons in advance, where possible.

With use it or lose it money, you use it. Whether money should be provisioned that way and coming in under budget should be punished is another story...


Oh this is a nice way to lock in all the money from Research Grants. I remember reading on Twitter about some of the research requiring massive amount of compute resources. ( Like a whole region of AWS ). This AWS money pool usage makes sense in that context.


You also can use this to meet credit card minimum spending for credit card bonuses.


I thought of this too, but it looks like they only allow transfers from US bank accounts for prepayment.


No, GAAP solves for this.


This is really nice. Now just add that when the amount is met, everything stops. Or maybe dropped into glacier to accrue charges.

I’d like this to work like a prepaid phone.


Awesome can't wait to give one of the richest companies on the planet an interest free loan.


I use Glacier For cold storage of family videos and photos. I have pre-paid for the next 10 years of expected usage. I just wanted to be sure that we would never lose that data, so I think advanced billing is great.


I was thinking about switching from Digital Ocean+Cloud66 to AWS but all comments about invoices and saas helping forecast aws invoice they convinced me to stay with Digital Ocean


AWS’s on demand pricing is high but their reserved pricing doesn’t leave a lot of flexibility.

Why is it just 1 or 3 years? What if I only need it for 6 months or 2 years? Can’t I just get a discount proportional to a custom length of time?

Why can’t I choose the amount of money I want to pay under the “partial upfront” option?

Why can I only reserve some AWS services and not others? Why can’t I reserve a certain amount of S3 storage for example?


Nope.

Used AWS for 3 years at a decent sized agency. It seems we underestimated how much not to forget checking and scrutinize every line item in the bill because our lighsail instances had another DB attached to it that we had no idea about, but was charging a crazy fee (converting our local currency to dollars = 19x)

There was much finger-pointing.


But unless you plan to block your card and ignore AWS' email(might not be a healthy thing for business), how will prepaying bad?


What I want is to assign pre-paid limits or just plain limits for a given resource group


ELI5 does this mean I can prepay and not be at risk for more than I have prepaid?


No. It just means that you can give money to AWS without having a bill, you are still responsible for the charges incurred regardless of how much you paid in advance.


Sorry but I did not understand the ‘cool’ part. With Linode & Webfaction I was able to prepay via credit card too. What is the advantage? To get block me if the credit is too low for s specific service?


Believe it or not a big part of cloud migration is figuring out how to cost it and get the finance people on board with after-the-fact operational expenses (*aaS) replacing capital/labor expenses (servers, sysadmins, network engineers, etc). When I worked in defense contracting I sat through half a dozen meetings with cloud vendors and virtually all of them took the time to explain how the costing model was distinct from on-prem, how to estimate and budget, governance, etc. At the end of the day many orgs with deep pockets also have very entrenched financial processes. AWS is doing everything that it can to make a play for these dollars by creating on-ramps such as this one.


Fwiw - GCP already does this through "Enterprise Agreements"

This is largely desired by customers with complicated acquisitions and budget allocation periods (Government)


How is what AWS doing legal btw? How can they charge people without limits? Isn’t this abusive?


How is that abusive? You pay for what you use and that’s it.


If you make a mistake you can be down one million dollar when you wake up


Wasn't this already a negotiable option?


Gotta fund the next space trip somehow.

/s


The AWS unexpected bills service has competition.


I'm surprised it took this long for AWS to launch something as basic as this. As others in the thread have mentioned, the core problem of tracking your AWS costs and where they're coming from is still a very hard problem for most organizations. Especially startups.

I'm a co-founder of https://www.vantage.sh/ which helps organizations track their AWS costs and we'll look at incorporating Advance Pay balances into the platform.


>I'm surprised it took this long for AWS to launch something as basic as this.

I'm not surprised. I'm convinced AWS has strategically focused on making costs difficult to keep on top of so you just pick a service, assume it's magically cost optimized for you and use it even though that's not reality.

Side note, I love the Vantage EC2 instance comparison chart, I've used it a few times recently and it made my life so much easier. Thank you and your team(s) for providing this freely and publicly: https://instances.vantage.sh/


STORJ DCS (Decentralized Cloud Storage) has enabled users to pay in advance with crypto since day 1.


They haven’t even had a working service since day 1 (still don’t?…)? I consulted for a couple of blockchains startups a few years ago, and this was the biggest piece of perpetual vaporware I came across. Good for them if they’ve finally managed to have a working product, but I wouldn’t be relying on it to work for a week, let alone some actually long period of time.


That site is weird. I get a "not found", then two seconds later the page loads. If that's my first interaction with the domain, I'm definitely not giving them money.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: