Assuming you’re willing to drop a fat stack of cash on the upcoming Mac m5 ultra with 512 gb unified memory, you can even run it locally, quantized to 4 bit. Whether it’s even slightly reasonable, well, my wife would probably skin me alive but maybe yours is more understanding.
* LLM usage is new for the world
* Models are evolving quickly with high worldwide competition
* Hardware is evolving despite RAM shortages
Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what you’d spend otherwise on serverless or dedicated GPU spend…
However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
Furthermore we continue to follow the path that image gen neural networks took. In that domain hardware requirements reached a peak and then started sharply declining to where we are today where a plain old video card can rapidly generate images that took a supercomputer not that long ago. So it's reasonable to assume that performance of such a system could potentially even increase over time.
The concern would be future regulations that prohibit you from buying a hosted version of the model. Even that could be bypassed with a VPN to another country but it's more work to go through the payments.
As long as there is demand for a model, it will be hosted by multiple providers.
Also, on a more practical basis, what if the way it's served is bad. Maybe I want my specific KV setup, or ultra low quant for entertaining garbage at 200 tk/s
You can still find a lot of old and completely outdated models on OpenRouter. The providers can scale serving of models up and down as demand arrives, so models don't generally disappear. They're just kept in the mix and the clouds will allocate hardware to it if someone is willing to pay.
In the odd case that it disappears completely, buying the hardware 2 years from now is probably going to be a better deal. That wasn't true if you selectively check the time period before hardware got expensive, but as new hardware comes out we're going to start seeing Strix Halo and old Apple hardware hit the market as people upgrade. It's already happening.
There is a certain personality type that cannot tolerate any uncertainty and must lock everything in right now against all future possibilities. If you fit that description then there's nothing anyone can say to discourage you from buying your own hardware, but for everyone else I do not recommend buying hardware to self-host LLMs just to save money. I self-host and run a lot of tokens through my setup (non-coding work) but I'm not really saving money.
I thought HN banned personal attacks. I'm in this sentence and I don't like it. /s
I just buy the good apple hardware because it's good, and it also happens to run local models. It's not as good for the dollar, don't get me wrong, but I'm not going to develop iOS without a mac, that's even more questionable than buying a strix or whatever.
https://srinagar.nic.in/notice/immediate-suspension-of-virtu...
Phones are randomly searched on the streets and if VPN is found, arrested
https://www.medianama.com/2026/01/223-jammu-kashmir-vpn-ban-...
https://timesofindia.indiatimes.com/india/after-vpn-ban-in-k...
“Out of the 15 individuals identified, five were minors who were counselled and advised in the presence of their guardians, with emphasis on awareness, lawful digital conduct, and the consequences of violating lawful orders,” he added.
But currently cloud does beat costs of hardware ownership, particularly with ridiculously high RAM/GPU/SSD costs....again due to these same companies.
I think not giving them your complete trust is reasonable! I'm not saying zero trust, and ZDR is fine for most things but I understand the people who don't want to stream their whole codebase out token by token.
[1] Yes, I'm aware of experiments to operate on encrypted prompts, but these are only research attempts, not something that could actually be used with frontier models in production.
Anyway, yeah, even at the prices I spent on my local AI stuff (I bought before RAMpocalypse really kicked into gear, so I bought old server GPUs for about $350 each and the Strix Halo for a little over $2k) it was never going to pay for itself; I just like to tinker. But, I can't imagine spending today's prices for hardware for local AI.
When the memory shortage ends, I'll be down to the Apple Store (or, more likely, clicking refresh on the Apple outlet every few days). But, until then, there continues to be a glut of cheap and free models in the cloud that are better than anything I can run locally and they're faster, too.
*$1000? $14,000? Who knows but everything in the middle there has been claimed.
So, even though there are more models to run locally that can be useful for the stuff I do, it makes less sense now to do so than it did when I got it. There are more extremely cheap options, now, and it seems likely to continue to get cheaper and better and faster, while my local hardware will always be slow and hot and only gets better via software (which has roughly doubled the speed it can run stuff since I got it, but it seems like there's less room for improvement in software now, and even at twice the speed, it still kinda sucks to use local models interactively especially on the Strix Halo).
The cost of online services is also largely a result of the cost of training (though hard to say exactly what that number is). Assuming you are using open weight models at home, you aren't paying for the training - someone else is.
OpenRouter prices are somewhat simmilar to Antrhopic/OpenAI API prices. So I conclude that the hardware plus operating margin alone can genuinely produce prices way above what you'd pay if you had a subscription. Of course the primary unkown factor is average token use per subscription. Without that it's all wild speculation.
The models are currently improving much faster than the hardware and this doesn’t seem to have plateaued yet.
It varies by model, but I’m getting 50-60 t/s with Qwen 3.6 35B and Qwen 3 coder 30B.
I’ve also used Qwen 3.8 27B but I get 10t/s on it.
It’s useable in some use cases, but I rely mostly on my $20 Claude subscription.
Could be a long time till gets released
Cybersec is also hit and miss, depending on what provider you choose, verification systems and all that jazz. Also, running locally allows you 100% data privacy, in any situation and for whatever usecase you might have. ~100k for hardware for a small team of devs to code locally is not that expensive in the grand scheme of things.
Lastly, local models allow for training / finetuning on your own data and processes. $/tok is not everything for everyone. Sometimes you can take a hit on value / speed if you get something else that matters for you.
If you're paying Anthropic or OpenAI to use their models, harness, governance, etc., I could see the local inference potentially coming out ahead. They're already starting to ratchet down what your money gets you on their platforms, and that can be expected to continue as the leaders of those companies continue to seek the road to the El Dorado that is being a trillionaire.*
If you're looking to get into the guts of AI development instead of having it handed to you by a provider, that's where it gets murky. I'm wanting to write some sort of agent that does things and get into making outputs consistent in the like, and I'm not sure whether to host something on GCP or buy an M5 Mac.
*Note: El Dorado is a mythical city and many people died trying to find it.
The object permanence of not having to reinvent the world every time a model gets sunsetted has value.
with open models, there is ecosystem/market of providers, where you can easily switch to provider you like
There are several things I do in my life that only pay off in the event of a big disaster, like an extended internet outage, civil unrest, supply chain disruption, war, etc.
I like to be able to do the things I do even if offline for weeks.
I spent a lot of money for more flash in my iPad Pro so I can keep all of offline wikipedia and OSM in it, for example, along with tons of books. It’s sort of like being a digital prepper. (Being a prepper is a spectrum, from anyone who keeps food in their pantry to people building bunkers under their house - how much you invest is a personal prudence and threat modeling decision.)
Also, privacy. And when I got the Mac Studio the 512GB was only $15k, which is dirt cheap for that much VRAM.
Is that true though? Many of the core LLMs need to be retrained as languages evolve to incorporate changes (language specifics, compilers, tooling, etc.). To some degree this can be handled via context injection in a variety do forms (agents looking up documentation and so on) but inevitably it’s not stationary in time, just as your OSS stack (probably) isn’t (depending on the languages, technologies, and use cases).
So your hardware is to some degree dependent on the good merit of groups like Z or Alibaba or whomever pushing out updated open weight models that dumped loads of capital into to train. You can keep using the existing models but at some point I suspect they’ll start to have more friction due to dated specs in language and so on. Again there are tuning and ways of layering this information on, and in theory you can even do some training on your own but I don’t think it’s as stationary as being portrayed here.
Those updated open weight models may not always be there (updated on new data). The usability of them is probably fairly long to be fair, but I suspect you’re going to see explosion in everything from libraries to languages etc due to LLMs so even the rate of change across your OSS stack may cause these models to be dated quite quickly, at least in the core model which will require layering fixes.
To be clear I’m on the fence thinking about much of the same issues and as close as I am to pulling the trigger, I keep thinking of very valid counter arguments as to why it’s me just wanting this thing I own. Which may be enough.
I would say when this comes to pass, we are already 5 years along?
> Part of it is knowing that whatever sort of enshittification the cloud providers do, my local programming environment won’t ever be less effective than it is today locally.
I think this is quite understated. It basically is freedom from a growingly antagonistic relationship between you and some remotely hosted API managed by faceless corporates at the whims of their board, shareholders and governments.. It really is such a mental burden to need to constantly manage this relationship (watermarks, silent downgrades, random false refusals, downtimes, model sunsets, changing ToS's, fucking ads). These companies will need to squeeze you for every cent that they can before open-weight models are simply good enough for the valuable tasks we can throw at them.
To have your own hardware is to no longer have this mental burden.
That said, when I bought my pair of Sparks, the best model I could run on it was GPT OSS 120B. That has an AA score of 24.
Today, the best model I can run on them is GLM 5.3 Flash at Q4, AA score 57. Just still out on GLM 5.3 mixed quant.
So from that perspective, they are many times better value than when I bought them, and will likely continue to increase in value.
That AA score is for the original model only
The surprising thing for me is how much work you will need to cool the banks if you’re near your memory ceiling. My memory starts soft throttling at about 74C (dies may be hotter, that’s the bank temp) and will turn down speed to try to stay below 80.
Happy to send my llama.cpp config settings if you want it.
Temp wise, no throttling, surprisingly cool.
That said, I am guessing my problem is not enough RAM - but this poor consumer platform struggles to do memory training with 128 GiB as it is.
Now I surely regret not having gotten Threadripper and 256 GiB of RAM in the before-times.
For what it's worth, all of my hardware was used; I think, all-in, I'm probably at around 3k-4k USD? Not cheap, but also not the worst for something relatively versatile.
Which is completely insane for a ten year old configuration.
I'm happy with all of the competition in the APIs on openrouter... I watch that like I used to watch the stock markets, lol. It's great fun.
Does inference make full use of the memory bandwidth in a NUMA system?
Using an example of llama-server on a 256GB (each CPU is attached to 128GB) system running qwen 3.8-flash-next Q8, which takes up about 188GB with the model and context, it's not that the inference engine 'knows' about NUMA, but if you use --load-mode mlock to load the whole thing into memory at once at the time llama-server launches, the kernel will distribute it relatively evenly across the memory connected to both CPUs.
Though I appreciate how good it is at "solid" grunt work and at that price (in fact I am paying the grandfathered subscription price; mostly).
I am planning to let go for my Claude AI subscription which I now use only for "planning" and maybe use that via Open Router as PAYG (at to try how it ends up). But god glm is bad at "talking" and "responding" anything prose. Not only quality but it's almost impossible to tune it and make it let go of its habits and biases and enthusiasms which often result in too many too and fro.
So I sometimes wonder at what point that starts becoming the cost and mental hassle. Maybe it's not there for me yet.
But the cost difference between api and self hosted is so incredibly large now it's almost impossible to ignore the fact self hosting is a terrible deal. I'm waiting things out until the dust settles on what the required specs are and consumer hardware gets cheaper/more capable.
- It is trivial to extract samples of the training data that was used, which can bolster existing lawsuits/foster new ones.
- Older models are not as safety-hardened, so it is easier to coax unsafe behaviour out of them, which is a PR risk.
- It may be possible to divulge proprietary secrets from the model (e.g. architectural details that may still be relevant).
For these reasons, and more, it's unlikely that GPT-3/similar models will be released until these concerns are no longer relevant (e.g. when they become a purely historic concern, similar to the open-sourcing of other proprietary software from decades ago).
At least to this point, the legal teams could get the model via discovery. IDK that the lawfirms realize that they could get experts (or that they'd have contacts that could) to search the model weights.
The weights arent the truth tho, maybe a timecapsule-vhs but i wouldnt trust llm weights more than more hardcore deterministic media that might get preserved to infer facts from an era.
The companies doing the training are becoming the "winners" that are "rewriting history" as they train their models.
According to the book The Thinking Game, lower quality models at that time were considered less safe, because they could be easily tricked into doing harmful stuff. In the book, Dario (of Anthropic) was the head of safety at openAI and was responsible for pushing for 10x scaling in training to make the models safer.
It does make sense, a smart model is going to be way harder to jailbreak into telling me how to synthesize anthrax (or something).
In the times of GPT-3 I'd scoff at the idea of an LLM doing any hacking; today, I'm running several AIs on my code before publishing, and they are finding (and demonstrating!) RCEs on my localhost server.
For example, one found a missing check in a third party JWT library which allowed full account takeover, which I'd have never even looked at.
Hence I don't believe a single word coming out of these people's mouths. Their "beliefs" are just marketing.
I certainly think the labs have muddied the waters using safety for marketing, but that doesn't mean less capable models weren't more dangerous at one point.
Easy access to malicious information hasn't proven to be the disaster these anti-prophets proclaim. For the last ~3 decades of widespread internet and search engines, you could trivially find all sorts of information (drug synthesis, explosives, etc.), and we're just fine.
(Corollary: easy access to good information did not make non-intellectual non-curious people smarter. Easy access to information does not have the consequences people immediately assume.)
My guess is I’m unintentionally refusing implicit signal that you think it’s safe to release all models openly, because you have observed models finding a vulnerability in a JWT library. But that sounds like a straw man instead of a steel man. Idk. :/
According to me, this is nonsense.
What's the point of publishing it when it'll likely be outclassed by gpt-oss?
AI policy is being shaped somewhat by the things Sam and Dario say. So even if you're not feeling vindictive, it's probably good to keep a track record of the previous things they have said as a Bayesian prior. People who don't know better listen to these people, and maybe they shouldn't.
I think the release of kimi k3 is definitely arguably dangerous, we're already seeing consequences of elite-tier cyberoffense capabilities.
I cannot stand using gpt-oss, but I miss some of the creative spark of GPT-3 davinci dearly.
GPT-3 is probably one of the last models trained without synthetic data. Meanwhile gpt-oss is rumored to be only trained on synthetic data.
I also miss the 2020 GPT-3 based AI Dungeon fine tune.
Or did you just mean the weights? If so, the open models distilled from GPT3 should mostly suffice. That's what good "open weights" are.
We're nowhere near a Fable-class model IMO, but things are going to get interesting in this next year.
I'm wondering of you could clarify your thoughts on this. I've had a hard time evaluating what Fable-class actually is capable of that sets them (or really it) apart from other models in a very significant way.
The good news is it can only get better from here.
If it's significantly larger than GLM 5.3 (I've heard some insane guesstimates out there like upwards of 5T params or more), that would prove rather embarrassing for Anthropic.
I feel like most benchmarks cluster on a reasonably limited area of human knowledge
You do pay for the tokens, but in theory on a smaller model each token is cheaper.
They filter their results through their AI, though, so you get a sort of meta-summary of the top few results. It did well with geopolitical news stuff, but I've not tried a hard science sort of query.
I should do that myself, come think of it.
(And 4.5 was smaller than 4)
Not really, in that you just work with different constraints.
Anthropic and US labs in general has maybe 100s to 1000s of GPUs per person to experiment. Zai and Chinese labs in general have 1-10.
The priorities are different.
Rather embarrassing indeed.
You implying its better than opus 5?
https://openrouter.ai/docs/guides/routing/provider-selection
z-ai/glm-5.3: also Z.ai, Novita, Atlas Cloud, IO.NET
My visit to your website went like this:
1. Visit models page
2. Try to find GLM-5.3-Flash (which is among the ~5 models that 90% of people currently care about)
3. Give up scrolling (which would have taken OVER 50 SCROLLS!!!) and use Ctrl + F
4. Try to find input/output/cached price
5. Scroll all the way up to find out which column is what
6. Notice that output price is cut off
7. Notice that the scroll bar is over 100 scrolls further down the page
8. Use Shift + Wheel to scroll horizontally (most visitors probably won't know this trick)
9. Notice that cached price is missing
10. Conclude that this is probably not a serious offering and bounce
There are probably more issues later on, but this is how far I got.
I would suggest you to:
- Deslopify all pages that a user may visit before conversion
- List important models first (see OpenRouter rankings)
- Move the most important information (model name/input/output/cached price) to the left
- Disaggregate the prices per provider (maybe subtables per model? not sure)
- Measure cache hit rate and compute effective price per provider (see OpenRouter)
(- Optional: Fix the broken link on your HN profile page. Currently, the only way to get from this comment to your website is a search engine.)
we are doing billions of tokens a day and thousands of users
The first case of this I remember was Bloomberg thinking that their extensive private data about markets would give a home-trained model better performance in finance tasks. The next version of ChatGPT beat them.
With how often new versions of frontier models are released, you likely won’t finish your work before it’s obsolete. The labs have whole teams dedicated to specific getting training data for specific industries (finance is one), and more powerful generalized models make better decisions even without specialized knowledge.
Your best bet is to get really good at training for something and then sell your company to one of the frontier labs for their post-training efforts.
Apparently, Cami Clark was tight with Eric Schmidt. Per unsealed documents, she seems to have pursued Epstein to invest in her "luxury porn" businesses, after a divorce & going bankrupt? Wild: https://www.wsj.com/tech/ai/claude-dario-amodei-wife-anthrop... / https://archive.vn/MJI7q
The insufferable gatekeeping of the US companies is actively contributing to computer insecurity at this point.
I get how you feel, but it's too late to be concerned. The cat's out of the bag. It's like being scared of moving from the bronze age to the iron age... when everybody already knows how to make iron, and the raw materials are everywhere. People are already making iron spears. We need to make iron shields.
We need open-weight models that are good at finding security holes so we can apply them to all of our software by default, and close every possible security bug, before the attackers find them. Every piece of software in the world should be held for release until it's scanned by a high-powered security model.
This is the same debate we had in the 1990's when strong encryption was considered a munition and not allowed to be exported. This just made the world less secure. And it was pointless anyway, because you can't really stop it being developed and shared. Eventually good sense prevailed and now we all have strong encryption. The same thing applies to security bugs.
They have a new shiny data center with Chinese GPUs, so hopefully they will be able to handle the demand. Their subscriptions are meh, in particular the Flash model has not that much usage. The other thing I wanted to try is Groq Qwen 3.8-27B at 450 tps to see if it's able to get work done faster.
I feel it's very important to play outside of the walled gardens, I feel they will pull the rug very soon, and I need to get job done while waiting hardware prices to go down...
I've seen dozens of conversations about it in last 24 hours, and every major inference provided added in first 24 hours. I think it's gaining plenty of traction.
We saw the same issue with GLM 5.2 when they still published publicly who the providers are on their website. Most ran FP8 but one was doing FP4, so you had this issue where one moment you had the better FP8 and another session you had the FP4 provider.
You can check the internet archive, it was in the FAQ part before they hide/removed it. So if you looked up the providers, and the published quants, yea, ...
Given that a lot of complaints are coming from people that felt OpenCode Go Flash feel like a step down compared to old OpenCode Go/DeepSeek API directly, it smells of a quantized down provider is mixed in.
I use DSV4-Flash on Max through Deepseek's API. I have been using GLM-5.3-Flash on High through Openrouter which I thought had a 50% discount. I must be doing something wrong for the costs to be off this much.
There are other providers with much faster inference, like BaseTen at >100t/s: https://openrouter.ai/z-ai/glm-5.3-flash#performance
I'm happy to be proven wrong, but this makes me think that the region is where the servers are, not where the HQ is.
I hate how difficult it is to compare prices when looking at subscriptions.
Would $20 in open router, using models like GLM get me more or less?
Z.ai does have their own subscription, but I haven't used it because their privacy policy was pretty buns last time I checked.
I burn through my current Google AI pro sub for the week in about 2 and a half days so wanted something extra to add to it, but don’t want to buy any expensive ultra plan. Flash models have got me about 98% of what I need, but quotas are still a bit low.
What did you find objectionable? I looked at it when I subscribed almost a year ago and I was fine with it (e.g. they don't train on your API inputs).
"The Company do not store any of the content the Customer or its End Users provide or generate while using our Services. This includes any texts, or other data you input. This information is processed in real-time to provide the Customer and End Users with the API Service and is not saved on our servers."
tl;dr API cost (openrouter) is always more expensive than a subscription (for the same given model). you should always use a subscription first and only go to API pricing if you run out of your subscription.
In terms of which subscription is best, different ones provide different models, different amounts of tokens, different rate limits. So it depends on what model you want and how much you need to use it. The frontier ones are always more expensive than open weight ones, but a few subscriptions are starting to include frontier models like GPT-5.6 Luna (which is a great deal but not necessarily the best price-per-performance).
- price: DS4 wins... $0.0235 vs $0.0242 for ten tasks
- latency: GLM wins... 108s total against 154s
this is for a personal use-case where i'm detecting ads in a written transcript. sticking with ds4-flash for now since latency is not a critical factor
GLM-5.3 is now open-weight
https://huggingface.co/zai-org/GLM-5.3https://z.ai/blog/glm-5.3