Hacker News

Yeah, that the main reason I never use services like Google Cloud if I don't have to, it's impossible to have a hard cap, and anyone pretending to be an expert, is just off. Google says that they can't provide a hard cap because that would mean shutting down all your services..bla bla, but at least give users the option.

That's actually crazy. So I can build a project I love, that does good, but somehow get in a situation where I'm accidentally paying 30.000€ (or 50.000€) to a big tech company? How is that fair? I mean yes, as a software engineer, you ought to reflect on all possible weaknesses, but there was a time when overlooking something meant something completely different than being down 30/50k. That is actually life-altering.

This should be illegal. If a contractor your hired to swap out a tile on your bathroom floor billed you for remodelling your back garden, you would obviously have the legal right to refuse that.

> The Gemini API supports monthly spend caps at both the billing account tier and project levels. These controls are designed to protect your account from unexpected overages, and the ecosystem to ensure service availability

https://ai.google.dev/gemini-api/docs/billing#project-spend-...

As a manager I avoid Google Cloud for this kind of customer-service disasters; but as someone who has dealt with large-scale billing systems in the telecom world, probably similar to that of Google Cloud, I am not surprised that it takes 10 minutes to consolidate all the usage logs of a customer for billing.

For telephony, it sometimes takes days when roaming is involved.

You have to imagine TB/sec of data, if not more, coming from thousand of potential sources, and queuing for aggregation to the proper company account, all having to be auditable. This is not a small engineering feat and it can't be real-time.

With that said, telcos usually include in their business model around 2-3% of bad debt (i.e. revenue that won't get paid), which accounts for frauds like this one. Given that the customer seems in good faith and has taken measures upon being notified, Google should manage this bill shock a bit more elegantly.

Moreover, the fact that this happened immediately after this key opened the AI gates means that pirates permanently scan for the permissions of all the keys they could gathers. Google could and should detect that and act upon it.

It's like a fire alarm system that goes off 30 mins after the it senses a fire. Good stuff.

Yet my phone service provider is able to cut off my internet access from the Kb I go over the limit…

For everyone here, you should be able to use quotas in order to set actual limits, you can lower them yourself as a practical solution.

https://docs.cloud.google.com/docs/quotas/view-manage

Quotas are real time or near real time.

Real time spend limits are probably never going to happen. Actual $ amounts are calculated by a centralized billing system offline in batch.

It sounds easy but it’s bonkers complicated, because of things like discounts, free tiers, committed usage, currency conversions and having to support every payment and deal structure in GCP.

Individual eng teams rarely actually think in dollar amounts, they think in the abstraction which is quotas.

hard cap it's technically impossible

These companies can sell your personal information in a microsecond in an advertising auction, but somehow can't figure out how to give you timely alerts that stop their cash flow.

Big shock.

Happened to me to, luckily only was 40$, restricted the api the next day. They were using gemini 3 flash which I am not using.

This is clearly setup for VC backed companies where shareholders don't care about spend as long as they can brag about investing in this cool start up at dinner parties. Normal and true business should stay away.

Shirky’s principle at work is all

Yet another good reason to use a pre-paid service.

There are many to choose from now, like Openrouter.com, PPQ.ai, and routstr.com.

> So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.)

Yes, it's technically+business impossible. To implement a hard cap, a bill never to go over, they'd have to cut your service, but also delete all your data in databases, object storage, data lake, etc. This is simply not an option, so they take the different option of authorising support to wave surprise surcharges / billing DDoSes.

I'd buy the technically impossible angle.

Even if you manage to get your microservices to synch every penny spent to your payment account at realtime (impossible) you still have to waiver the excess, losing some money every time someone goes past their quota.

I get furious every time this comes up and somehow there are bootlickers ready to defend big tech on it.

My ~2 person small business was almost put out of business due to a runaway job. I had instrumented everything perfectly according to the GCP instructions - as soon as billing went over the cap the notification was hooked up to a kill switch, which it did instantly.

GCP sent the notification they offered as best practice 6 HOURS late. They did everything they could to not credit my account until they realized I had the receipts. They said an investigation revealed their pipeline was overwhelmed by the number of line items and that was the reason for the lag. ... The exact scenario it is supposed to function in. JFC.

Exactly my thoughts, can not really understand how delayed alerts are acceptable... Have you managed to settle the cost with Google, what was the outcome?

Which cloud provider actually prioritises features that cut off your money supply? Because AWS sure as shit doesn't either.

> Sure it would probably make Google less money short term, but surely that's more preferable to providing devs with such a poor experience that they'd never recommend your platform to anyone else again.

Welcome to late-stage capitalism, where there is no long-term thinking, only short-term profit stealing, and Fuck You I Got Mine.

> Google also has historically treated API keys as non-secrets, except with the introduction of the keys for LLM inference, then users are supposed to treat those secretly

This was reported a long time ago, and was supposed to be fixed by Google via making sure that these legacy public keys would not be usable for Gemini or AI. https://news.ycombinator.com/item?id=47156925 https://ai.google.dev/gemini-api/docs/troubleshooting#google... "We are defaulting to blocking API keys that are leaked and used with the Gemini API, helping prevent abuse of cost and your application data." Why are we hearing about this again?

theres not a single real gemini api key in the results

...JCip3SJw => Your API key was reported as leaked. Please use another API key.

...afnt0t-E => Your API key was reported as leaked. Please use another API key.

...-UYzYTYU => Your API key was reported as leaked. Please use another API key.

I think they all get immediately reported as leaked and invalidated.

Um. What? In what world are API keys not secrets?

My favorite Google LLM benchmark is asking Gemini models to create a script that fetches API usage (just request counts) for a project from GCP.

100% failure rate.

Call it for what it is, an antifeature, a trap for the user.

This is presumably by design: How can it be the vendor's fault if your custom billing protection implementation failed you at a critical time? Much harder to defend against a switch on their dashboard allowing billing overshoot.

having to glue pub/sub to a cloud function just to approximate a hard cap is the whole indictment. that's not a safety feature. that's you building your own brakes.

As the other user said - this would be an anti-feature and user hostile.

This is a sign that somehow there isn’t sufficient incentive to work on these features.

This is from my experience the same in AWS and Azure. I would love for a kill-switch if the usage goes above a critical threshold. 5 hours down time will not kill my app but a huge cloud bill might.

> Insanity

You mean cash machine

So someone took a picture of the key at the live training session or something? What's the suspected cause?

I wonder what happens if you just decide not to pay. Surely that would have some legal implications in the US, but what about elsewhere?

It's not typically a problem that usage is event driven. At least not for prepaid phone plans. Or debit cards. Or mailboxes. Or any myriad of prepaid or quota'd services. It's not rocket science, just a bad business practice on the part of Google.

>There's a delay between incurring costs and receiving budget notifications, so you might incur additional costs for usage that hasn't arrived at the time that all services are stopped.

This delay may be hours or days. I managed to spend $400 in 5 minutes.

Early Generative AI was popular with spammers before it became mainstream because it could be used to write infinite variations of spam messages. Making each message unique is more likely to bypass spam filters.

There are also a lot of AI use cases that require a lot of token spend to brute force a problem. Someone might want to search for security exploits in a codebase but they don’t want to spend the $50,000 in tokens from their own money. Finding someone’s key and using it as hard as possible until getting locked out could move these projects forward.

Totally speculating here, but maybe they provide some sort of LLM as a service, and they rotate stolen API keys in the background so they don't have to pay anything ?

Or they use the LLMs for criminal purposes (like automated social engineering) and so the API key can't be traced to their personal info (but they could also use a local model for this, so I don't know).

There are plenty of services offering AI inference at a discount. Some of these will be using your data for future distillation; others might be making use of bulk discounts and passing these through to a number of individual users (while taking on billing, support etc. risk) – and maybe some are just selling tokens falling off the back of a truck?

If they work for hostile state, the payoff is destruction of economy and social contract. Damage here, damage there. It all adds up.

Generate phishing mails, spam mails, disinformation pictures and videos.

Youtube has plenty of scam ads where well known people try to get you to sign up for the WhatApp group for financial tips

I want API keys with monthly and hourly quotas and RATE LIMITING.

like 50k requests per hour, above that 1/s/client up to 20 req/sec.

I don't want to shotgun my service for every user if one user is misbehaving. I want to set rate of bleeding

A cap is a more fundamental fix.

It’s easy to miss a setting especially if new features with opt-out are added

That's standard for Firebase apps. It's also recommended by Google (they describe the keys as "public by design").

It's "implied" throughout the whole post (or more like assumed that the reader understands this, because it's the basic premise of the problem). It's why they link to a post that explains the basic concept after a remark that "This describes our issue in more detail".

> tl;dr Google spent over a decade telling developers that Google API keys (like those used in Maps, Firebase, etc.) are not secrets. But that's no longer true: Gemini accepts the same keys to access your private data. We scanned millions of websites and found nearly 3,000 Google API keys, originally deployed for public services like Google Maps, that now also authenticate to Gemini even though they were never intended for it. With a valid key, an attacker can access uploaded files, cached data, and charge LLM-usage to your account. Even Google themselves had old public API keys, which they thought were non-sensitive, that we could use to access Google’s internal Gemini.

From Google themselves, in the Firebase docs:

> API keys for Firebase services are not secret. Firebase uses API keys only to identify your app's Firebase project to Firebase services, and not to control access to database or Cloud Storage data, which is done using Firebase Security Rules. For this reason, you do not need to treat API keys for Firebase services as secrets, and you can safely embed them in client code.

<https://firebase.google.com/support/guides/security-checklis...>

... or at least that's what it used to say, until they quietly updated the docs to say this:

> API keys for Firebase services are not secret. API keys for Firebase services only identify your Firebase project and app to those services. Authorization is handled through Google Cloud IAM permissions, Firebase Security Rules, and Firebase App Check.

> All Firebase-provisioned API keys are automatically restricted to Firebase-related APIs. If your app's setup follows the guidelines in this page, then API keys restricted to Firebase services do not need to be treated as secrets, and it's safe to include them in your code or configuration files.

Followed later by (in different section):

> Use your Firebase-provisioned API keys only for Firebase-related APIs. If your app uses any other APIs (for example, the Places API for Maps or the Gemini Developer API), use a separate API key and restrict it to the applicable API.

Yeah, the amount of people creating, running and maintaining websites yet don't understand how websites actually work in practice is very high and seems we haven't even come close to the ceiling yet.

They don't have to compute it in real time. They can cut service when they detect it reached the cost and the difference is free of charge.

Overcharge protection doesn't have to be free. It could be +5% on prices or a fee of 25% when you reach the threshold.

They would have financial interest in calculating cost in real time and it'd magically become more and more precise over releases.

Cutting off at the exact cent is difficult, but a hard limit that triggers within one dollar of the actual limit should really be possible

If for some resources you can't sample measurements fast enough you could weaken it to "triggers within one dollar or five minutes after cost overrun, whichever comes later". But LLM APIs are one of those cases where time isn't a factor, your only issue is that if you only check quota before each inference a given query might bring you over

Why would it be hard to calculate cost? Multiply a fixed price * requests/time ? It doesn't have to be exact in real time, it just has to report something approximately useful in realtime.

It's absolutely not fine to be at the mercy of other people, that's what we buy cloud products or really any products for: So that we are not at the mercy of hardware faults, bad weather, bad teeth, hunger, thirst, [insert anything]

> I think the logistics of calculating cost in real time is something that is extremely hard.

What makes you think that?

Ridiculous. They are clearly not trying at all. A hard wall preventing going over budget by 100x in a couple hours is not some devilishly complicated decentralized system problem.

Don't tote the party line.

Same reason why Azure AI only has easy rate limits by minute, not by day or week or month. Open source proxy projects do it easily tho. Think about the incentives.

Going over a hard cap by 3% would be a reasonable failure to make, not by 30000%.

> At best, I still use services that have hard-bounded usage limits, like EC2 from AWS, where one instance can never go beyond 24h/day usage and is always capped, with shutdowns when exceeded, and limited credit cards, too.

Is this possible on AWS today? I'm the same way, if I cannot set a hard-limit for the billing so I can know for a fact how much it'll maximum cost in a month, I'm not interested in using that service for anything. Which is one of the top reasons I've stayed clear of AWS, they used to have only billing-alerts, but you couldn't actually set limits, guess one step forward that they've finally implemented that now.

Not if its publicly called from Javascript, as your user's browser will make those requests. You neither know their IP addresses, nor is the referer or origin header a safe choice as it can be spoofed outside of a browser.

There's a brand-new, Gemini-specific feature for that (as new as March 23), but historically the answer has tended to be "no" from all the cloud providers. Most giants and indies alike have always been strongly opposed to implementing this feature for business reasons. (When you run across something that does let you do things that way, it's one of a handful of exceptions.) Their response is to tell you to set up budget alerts, which is not a solution, as described in this post.

<https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...>

No. I believe all major cloud providers are Pay As You Go. I think only Azure has a tier where you can run on free credits for a while.

Google doesn't allow disconnecting credit card from account unless you close it. That includes situation when you are just trying out free tier.

I doubt most cloud providers are even technically ready for true prepaid billing (which requires things such as estimating and reserving funds prior to paid operations, corresponding real-time two-way interfaces instead of just eventually consistent billing event aggregation etc).

In early mobile networks, the feature set for prepaid used to always lag behind, since real-time billing wasn't really a design consideration from the beginning.

I suppose rather than taking on that extra work or offering a reduced feature set or by building something best-effort and taking financial responsibility for its failures, if cloud providers can just get away with making this the user's problem, why wouldn't they?

> When your Prepay credit balance on the billing account hits $0, all API keys in all projects linked to that billing account will stop working simultaneously. Prepay credits apply only to Gemini API usage costs; you can't use them to pay for other Google Cloud services.

https://ai.google.dev/gemini-api/docs/billing#prepay

No GCP is not prepay.

OpenAI also worked like this last time I used it - not sure if that's changed.

This is GCP's revenue model, lol. Let's provide a (semi) generous free tier and trick people into accidentally going over it.

there is no way to cap your billing on gcp.

you can get notifications but that's it.

i don't want to get throttled below my quota but some type of spend limit would be good.

The company selling machine learning services would probably love a €54k bonus

It's not a security incident because it makes Google money. It's extra revenue. They are embarrassed all the way to the bank.

At some point, when it appeared 2 months ago on HN and they still did nothing about it, intentionality can be assumed.

This is only a little billing leakage, Operation Aurora in 2009 was 100x worse

What does this have to do with security?

And this is why we invented segmentation, and everybody that are still not doing that are paying now and this is fine

Google is not the only culprit here;

Prepaid only is a fantastic idea, especially for dumb-ass startups. Limiting your liability to $100 or so sound like a big-ass W.

Oh please no. And the "alternatives" to API keys aren't going to help much either, they'll just add friction to getting started (as reference: see the pain involved in writing a script that hits gmail or calendar API)

> and deploy to hardware you own or rent.

Because this part sucks. I grew up fiddling with Linux. I don't want to play devops anymore. I want to write code and run it.

That’s replacing Google with OpenAI/aanthropic/whatever. Same shit