This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
smeeth 12 hours ago [-]
Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.
It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
acchow 8 hours ago [-]
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
smeeth 6 hours ago [-]
1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model.
2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.
even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.
Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.
I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
nkozyra 7 hours ago [-]
No.
But do you need to run every small problem through a 10B-30B model?
We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.
Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
edg5000 3 hours ago [-]
I do about 10 google search queries for every 1 opus/gpt prompt. For google, I don't actually open pages anymore 9 out 10 times; I rely on the AI summary. It's fast and accurate; the trick is that you learn where the boundary is of what you can ask it. Querying information the small model is great at.
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
refulgentis 2 hours ago [-]
There’s a good eval floating around somewhere and tl;dr they’re awesome but the benchmarks are cooked, you’re better off with Qwen 8B Q4 than 27B 1b or ternary.
Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites
yieldcrv 7 hours ago [-]
Its somewhat good, the prism team’s webgpu demo gave it a couple dozen “kernels” written in Fable 5 and it calls them for almost everything procedural
I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
flockonus 7 hours ago [-]
There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat!
agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.
dTal 7 hours ago [-]
>in case they start to ban open source AI development
Is that likely, do you think?
submain 5 hours ago [-]
NVidia seems to want them. They have a lot of lobbying power.
dwa3592 6 hours ago [-]
dario has been saying open source models are dangerous. who knows who is listening to him.
mseri 38 minutes ago [-]
Nice!
I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
notnullorvoid 12 hours ago [-]
I mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).
reidrac 13 hours ago [-]
> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
LeifCarrotson 13 hours ago [-]
I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare!
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
RussianCow 12 hours ago [-]
Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.
nmfisher 6 hours ago [-]
Completely agree. Slow but smart models (Fable, Sol, GLM5.2 etc) are great, but they leave me with zero mental model of the code that's been written. Most of the time my mind wanders off and I go check social media or fire off a prompt for some other random project, it's a big productivity drain.
Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.
trollbridge 2 hours ago [-]
Glad to see another UltraSpeed addict here. I really hope they keep it around.
hx8 8 hours ago [-]
I'd rather have slower and better output than worse and faster output.
RussianCow 6 hours ago [-]
It depends. For something high stakes or inherently complex, sure, you don't want to have to clean up the agent's mess afterwards. But for many tasks like building web UIs, the difference in output quality is going to be small enough that iteration speed will win over quality.
With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.
ngcc_hk 9 hours ago [-]
In 1980s ibm has studied and said why sub-second response needed to maintain the mental flow. That time you send a whole screen unlike unix like character by character. This proves very true even when you deal with form processing. I think that we are dealing with the same issue here.
Keep your mental context in your brain is critical
dofm 8 hours ago [-]
> It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once.
At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.
allknowingfrog 12 hours ago [-]
We clearly have different goals. I want an LLM to review my code, not the other way around.
pmarreck 9 hours ago [-]
I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code"
Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.
Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.
But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
allknowingfrog 8 hours ago [-]
There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to.
The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language".
The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
pmarreck 4 hours ago [-]
> AI will replace humans because giant business believe they can do the same work, not because they will actually be able to.
This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe
In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though.
> The correctness of an application is limited by your ability to understand and describe what you need.
If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other.
We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?
nolok 12 hours ago [-]
It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back
Aurornis 9 hours ago [-]
> another task a lunch break whatever and it will be done when you're back
At 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.
allknowingfrog 11 hours ago [-]
How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.
vitally3643 10 hours ago [-]
This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws.
These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
allknowingfrog 8 hours ago [-]
This thread started with me saying "we clearly have different goals" and then being told that I just need to hold the screwdriver differently...
11 hours ago [-]
fwip 8 hours ago [-]
You don't "maintain flow." You eat lunch.
I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
allknowingfrog 8 hours ago [-]
This was a discussion about LLM usage patterns. I'm not opposed to lunch breaks. I'm opposed to being required to take the equivalent of 12 lunch breaks a day while I wait for slow responses.
ClikeX 8 hours ago [-]
Right? Tech should make my work easier. Not have me stressing out even more.
Let the machine do its work while I relax, I’ll check up on it later.
Aurornis 11 hours ago [-]
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
zoobab 11 hours ago [-]
You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.
Aurornis 11 hours ago [-]
It’s not a cost savings relative to paying API prices even if you’re suspending it.
This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
adastra22 12 hours ago [-]
We aren’t there yet. Not for frontier development work at least.
parineum 2 hours ago [-]
> there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
That Xeon is going to be devouring power.
bitpush 12 hours ago [-]
is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?
bredren 10 hours ago [-]
Except often queued agentic flows must be checked in on. Or to use the comparison, 3D printers are not immune to making spaghetti all night when something goes wrong. (I’m not a 3d printing expert so maybe that is solved now)
It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
varispeed 10 hours ago [-]
Filament snaps at 1am and then you have to run print again. 10 hours turn into many days potentially.
I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
ctoth 12 hours ago [-]
[dead]
dwa3592 13 hours ago [-]
i am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
jckahn 12 hours ago [-]
Ignore the haters. What you've done is incredible!
It says so right in the readme. They’re not hiding anything.
aziis98 12 hours ago [-]
For me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe).
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
kevincox 8 hours ago [-]
I think 10 t/s output is usable for many coding workflows if the input speed is much much higher (~100 t/s is a rough minimum line). The low output speed can really hurt for heavy reasoning output but it can still be used to get some jobs done as long as you don't expect highly interactive use.
ClikeX 8 hours ago [-]
The slower models seem fine for home lab usecases such as processing document transcriptions and tagging them, for example. I don’t need that to be live, it can just churn overnight.
Wowfunhappy 9 hours ago [-]
How fast does a human write code?
sigmoid10 9 hours ago [-]
It's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.
Wowfunhappy 6 hours ago [-]
If you were iterating on code with a human partner, how fast would they write?
Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code.
I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.
mh- 3 hours ago [-]
This isn't just measuring the speed of tokens that are written to a file (code). Your human partner is thinking a lot faster than 5 tok/s.
bigyabai 13 hours ago [-]
Yep, and we don't even know how long they spent on prefill. A typical 50-100k token session could take 10-20 minutes to prefill on a Mac.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
razster 7 hours ago [-]
I sense the same. Running Ornith 35b with Pi and hitting 50+toks, and now that I learned Pi can do search and fetch, I have not needed to visit any of the large models as a search function. Soon we will be seeing new models drop this month and next that will change some of the landscape. Exciting times. P.s. Try Ornith, well worth it.
its pretty good for building any kind of research report using search engines and it manages context corrextly.
VladVladikoff 5 hours ago [-]
Isn’t 9 tok/s unbearable? I frequently see Claude sessions hit 1Mtok in less than a day. It seems that 9tok/s would be really slow for actual work.
driverdan 4 hours ago [-]
You can't use it real time but you can have it run in the background and come back to it.
Aurornis 12 hours ago [-]
> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
dwa3592 9 hours ago [-]
>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
Aurornis 8 hours ago [-]
I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results.
Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.
It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
dwa3592 8 hours ago [-]
Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.
andrekandre 5 hours ago [-]
the local models open source harnesses are really improving quite fast; just a few months ago i couldn't get any tool calling to work, and responses were very slow (thinking going on too long etc) but now with some newer models on my macbook air, tool calling works and depending on the model, it returns from thinking fairly swiftly....
felineflock 9 hours ago [-]
That is awesome!
I am curious about the decision to not use GPU since this is for Apple Silicon.
Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
dwa3592 9 hours ago [-]
It will, but the process at this point is SSD bound rather than compute bound. On a bigger machine, Apple silicon must help but I don't have a bigger machine. I can think about this more and will make changes if that helps.
haute_cuisine 13 hours ago [-]
How are the thermals? I noticed that running any serious workload locally heats system fast.
dwa3592 12 hours ago [-]
i have been optimizing for that. for now samosa is capped at using half of the avaiable cores and switching between them, which keeps the system 'less hot' as it would have been. i will also release better thermal control in the next release. at this point its basically sacrificing about 20% of the speed to keep the hardware less stressed (and hot).
TYPE_FASTER 11 hours ago [-]
Downloading now just 'cause the repo name
typon 11 hours ago [-]
By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s
veber-alex 10 hours ago [-]
Literally the only way this is going to happen is if aliens come to earth and gift us some amazing technology.
kennywinker 10 hours ago [-]
Unless there are major improvements to how much hardware it takes to run a 1T model, this is deeply unrealistic. First because why release hardware that puts your biggest customers (data centers) out of business. Second because as I understand it the data centers have bought up all the high end chip production capacity for at least the next year and unless the bubble pops that'll continue for a while.
iwontberude 9 hours ago [-]
Because for the company that will actually do it, their biggest customers aren’t data centers they are iPhone owners.
kennywinker 8 hours ago [-]
First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT.
Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than op is suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.
Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but op’s scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
typon 3 hours ago [-]
There is ton of room for improvement "down there".
* Software inference optimizations
* Heavy quantization
* Chips with hardcoded transformer architecture
* Much cheaper HBM
* Much sparser models - 1T total with ~1-10B active params e.g.
* Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
andrekandre 5 hours ago [-]
> Datacenters are willing to pay $50k for a single high end GPU.
its true for now, because capital is flowing like a torrent, but how long will that last if returns start to be expected (aka the bubble pops)?
kennywinker 4 hours ago [-]
Even if the bubble pops and anthropic and openai et al implode - genie doesn’t go back in the bottle. The usefulness of LLMs for coding is proven, and a chip in a datacenter running 24/7 is always going to be more valuable than in a personal device running occasionally.
That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it eventually reaches phones and toasters and whatever. But not in two years.
Capricorn2481 12 hours ago [-]
I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.
segmondy 10 hours ago [-]
try q8, check your parameters. qwen3.6-35b-a3b should definitely be able to do so with no issues at all.
Azantys 12 hours ago [-]
You are comparing a 35B models to a 635B+ frontier model, of course thats not even close
Capricorn2481 11 hours ago [-]
I'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.
IsTom 11 hours ago [-]
To be fair, they don't have the text editor highlighting all the matching parens. I'd be lost too.
Capricorn2481 9 hours ago [-]
Yeah prediction models and many parentheses are probably not a good combination, but we're not talking about anything exceptionally complicated here. I have had syntax issues in Python as well.
Azantys 10 hours ago [-]
It really depends on the language, popular languages work pretty good
walrus01 5 hours ago [-]
what quantization? what temperature?
13 hours ago [-]
6 hours ago [-]
6 hours ago [-]
9 hours ago [-]
rvz 12 hours ago [-]
> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second.
That is no where near decent at all.
dwa3592 12 hours ago [-]
it's a 16GB machine. i am proud of this machine so far.
It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
smith7018 13 hours ago [-]
Nothing says they're using Qwen for local development. They could be using it to for conversations, knowledge, or "creative writing."
embedding-shape 8 hours ago [-]
> Nothing says they're using Qwen for local development
I know! That's my point! You're a poor salesman of a coding environment/tool if you don't even use it yourself for coding...
dwa3592 13 hours ago [-]
>>It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
embedding-shape 9 hours ago [-]
> that would have run into a race condition unfortunately ;)
Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)
hagen8 12 hours ago [-]
Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.
If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.
dlcarrier 16 minutes ago [-]
From what I've seen, most inference providers are running at a loss, so it wouldn't be at all surprising if using their services costs less that running the same software locally.
The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.
throwawayffffas 9 hours ago [-]
I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing.
In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.
For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and
close to 50 times more expensive for processing.
why_only_15 4 hours ago [-]
The correct comparison is not sonnet, but qwen3.5-27b on a cloud. Alibaba's pricing [0] is $0.20/m input $1.56/m output, so $0.72 for the prompt processing or $0.22 for the generation. Yours is still cheaper but the margins are less.
My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).
Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.
bischofs 12 hours ago [-]
I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.
hagen8 11 hours ago [-]
Inference costs will go down massively once they use the upcoming GPUs. I estimated that a model like GLM5.2 will be around 0.03USD/M output tokens in 2 years when the Feynman GPUs will be available in 2028. And this did not even consider architectural efficiency improvements. In mid 2027 we will already see a 10x reduction once everyone has switched to the Ruby architecture.
It will be feasible for everyone to have 20 different agents running at all times. A new world is coming
amanaplanacanal 12 hours ago [-]
Yeah I had to check, I'm paying 0.08usd per kwh. This is in the US PNW with quite a bit of local hydro power.
brewtide 12 hours ago [-]
It's all relative. On the opposite coast, Maine it is ~ $0.28 cents kwh including getting it there. (~ 50% energy, 50% delivery). It's too darn expensive here.
leobg 35 minutes ago [-]
I’m in DE, too. Same calculus. Excluding winter though. When it’s cold outside, a GPU heats my office and I get tokens free on the side. :)
sosodev 12 hours ago [-]
Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.
segmondy 10 hours ago [-]
don't care, and yeah i don't like to hear it. we don't run local because it's cheaper money wise. we do it for freedom, for privacy and having option makes it cheaper in the long run. if there was no local options, your cloud model would cost much more!
albrewer 12 hours ago [-]
Not OP, but your math is a bit off - I have solar panels :)
senordevnyc 46 minutes ago [-]
There is something genuinely beautiful to me that these goofy apes have managed to turn sunlight and sand into intelligence.
kylemaxwell 12 hours ago [-]
You pay 3x as much for electricity as I do, so the math here is going to work out very differently depending on a lot of factors.
carra 10 hours ago [-]
> Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.
Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...
apitman 11 hours ago [-]
It would be more efficient if you had multiple users (or agents) making parallel requests to take advantage of batching, right?
Waterluvian 12 hours ago [-]
Locally I’m looking at about CAD $0.05 per kWh when off peak.
newusertoday 12 hours ago [-]
its 18k not 180k
bigyabai 12 hours ago [-]
It's the "Race-to-Idle" situation all over again. It consumes less power to complete a task faster, whereas using "low power" hardware that draws max TDP for 30 minutes isn't very power efficient.
The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.
Aurornis 11 hours ago [-]
A dual Xeon of this era is probably pulling 300W or more when loaded.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
brailsafe 11 hours ago [-]
Reasonable analysis, especially because this person seems to have an actual house. In my case, I rent and don't pay for electricity directly, so the cost effectiveness threshold is whenever the landlord starts complaining
twsted 11 hours ago [-]
I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.
Aurornis 11 hours ago [-]
Anyone running LLMs at home will come to that realization quickly, if they’re looking at their power bills. Even feeling the heat output of a computer running at 100% in your office makes it clear.
I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.
euio757 10 hours ago [-]
> Even feeling the heat output of a computer running at 100% in your office makes it clear.
What does it make clear? That I can replace the space heater my wife runs 9 out of 12 months of the year with a home server? And effectively get $0.00 per token during those times?
In houses running A/C year round, sure there'd be some impact, but in all the places running heat, doesn't seem that it'd move the needle on power bills.
There are startups whose entire business model is "cloud server as a home space heater" (aka "data furnace") ...
Loocid 24 minutes ago [-]
There are bunch of bitcoin miner products now that are packaged to look like regular space heaters.
Aurornis 9 hours ago [-]
> What does it make clear?
That it’s pulling a lot of watts.
Good for you if it’s replacing electric space heaters.
hahahaa 3 hours ago [-]
It is like crypto currency - for most people more cost effective to buy than mine due to power prices.
bitexploder 10 hours ago [-]
It gets better in the cooler months when heating is running in a home :)
hahahaa 3 hours ago [-]
Sort of - heat pump could be 2x to 3x more efficient. Heat pumps cool outside air down to get some of the energy for heating inside.
hparadiz 14 hours ago [-]
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
trollbridge 13 hours ago [-]
I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
ryandrake 13 hours ago [-]
I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.
edg5000 3 hours ago [-]
Great speeds. So a small model on a lot of slow ram is fast. Now I wonder how larger models run on that thing.
neomindryan 13 hours ago [-]
Thank you for sharing!
throwaway2027 14 hours ago [-]
That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows)
I run the same setup Gemma 4 26B on a 2013 Mac Pro (dual graphics cards but they're useless for this). I also get about 5 t/s. It's perfectly serviceable for some tasks!
sjs382 9 hours ago [-]
I bought a trashcan Mac Pro on a whim last week ($120 in eBay!) and did some reading about them—it turns out people recently started using the GPUs to run models @ 20-30 tok/s.
I'm excited to get my mitts on it on Friday when it finally arrives.
Here's some of the resources I came across if you're interested in reading.
He's shown me his set up in his basement. It's sick! Talk about your 3d printer next!
thomasjb 12 hours ago [-]
I was inspired by that post also, got a Qwen Coder 1.5B up to 27tok/s prompt eval and 13tok/s decode on an e5-2650v2 inside a GNOME box
rrhjm53270 12 hours ago [-]
Good job. Thanks for sharing. I have a similar NAS server with 2 Intel(R) Xeon(R) CPU E5-2699 CPUs. I will test as well.
rhema 11 hours ago [-]
I love my little dual core X99 board with Xeon E5 2673 V3. It's not power efficient, but I just leave it in my basement for local Jupyter Notebook stuff. Much faster than everything cloud-based for a reasonably price at my scale.
throwawayffffas 9 hours ago [-]
The 5.2 tokens per second generation is not that bad, what kills it is the 16.2 prompt processing that makes this too slow to consider even if you have the hardware lying around.
mmastrac 13 hours ago [-]
Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around...
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
neomindryan 13 hours ago [-]
hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.
fouc 13 hours ago [-]
any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.
neomindryan 3 hours ago [-]
I think I was just following along with the previous post about running Gemma on a Xeon. Next I’m going to see which model can give the highest tokens/sec
superkuh 12 hours ago [-]
Such a system is RAM bandwidth limited and not compute limited Switching to q4 from q8 would decrease the amount of data needing to be loaded by half. The token generation rate would nearly double. But generally if you can do q6 or q8 and you have enough RAM you really should. Even if it's slower.
giantrobot 8 hours ago [-]
Token generation is nominally bandwidth limited. Prefill/prompt processing is nominally compute limited.
For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.
aniwalunj 15 hours ago [-]
Truly amazing. This gives a peek into the future for what's possible.
Yes and a 10 year old Xeon is going to be a v4 (not a v2 as in TFA) and it's going to have DDR4 ECC, not DDR3 ECC.
I've got a 14 cores / 28 threads Xeon from 2015 that I use as a server at home (ZFS / VMs).
It's really a sweet machine.
For ricing I've got a semi-recent AMD 7700X / DDR5 RAM (from 2023 ?) which is my main machine but the real deal is my old and trusty 10 years old Xeon server.
DDR4 ECC is pricey too atm but a 10 years old Xeon is basically free now.
A 20 cores / 40 threads costs maybe 20 USD (for just the CPU). Slap that in a $100 old HP Z440 workstation and you're good to go for quite a few workloads.
Mine is only on when I'm at my computer: it's not turned on 24/7 but more like 8/7 so the entire "but it consumes energy" point is moot.
okokwhatever 13 hours ago [-]
To me context means everything.
Tokens per second is a great metric but in the real world context window is the deal breaker when a real use case is on the table.
dofm 13 hours ago [-]
Gemma 4 26B is capable up to 256k or 262k, can't remember which.
Whether the writer's setup affects that choice I don't know.
rvz 12 hours ago [-]
...and how many servings can this do?
neomindryan 15 hours ago [-]
Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish.
The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.
why_only_15 4 hours ago [-]
Why is the post AI-generated? If you're going to make something for us to read you pay us the courtesy of actually writing it first.
The post is absolutely LLM-generated. “Punchy” short sentences, “… has quietly come to mean …”, “The optimized paths weren’t there to execute.”
logicallee 13 hours ago [-]
I think "this" refers to its parent comment. Part of it sounds like Claude wrote it. AI-generated comments aren't allowed on HN.
otherjason 11 hours ago [-]
Indeed, I was referring to the parent comment.
pkghost 13 hours ago [-]
Here's the thing: life also imitates art. If you invert your load-bearing assumption, it could be that he just reads too much slop. But my honest take? You might be right.
dofm 13 hours ago [-]
stopitgetsomehelp.gif ;-)
linncharm 5 hours ago [-]
[flagged]
0x457 10 hours ago [-]
[dead]
OsamaJaber 13 hours ago [-]
[dead]
CurbStomper 12 hours ago [-]
[dead]
rvba 13 hours ago [-]
Apologies for asking here but literally nobody knows:
Android studio connected to a local model disconnects automatically after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens.
I tried Googling, searching for settings in Android studio, even created a stackoverflow post - but zero information. Jetbrains mentions "remote agent timeout mechanism" - but after changing it, nothing happens.
NortySpock 12 hours ago [-]
If the local model is served via ollama, there's a default timeout of 10 minutes , which can be adjusted either per-call , or (as I did) in the systemd service environment variables
You didn't specify what was serving your local model.
rvba 12 hours ago [-]
Thank you for your reply. I use LM studio (local server), but can switch to a different tool.
Do you know how to switch it in LM studio?
What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected.
So I assumed it was a setting in android studio.
NortySpock 9 hours ago [-]
Dunno, I have not used either of those. (Had been using zed and ollama, and ollama had plenty of odd defaults that needed fixing)
Glancing through the docs, I would be digging down in the config of both Android studio and lm studio for either a TTL or jit auto evict setting, and if you find it, set it to some large number measured in hours?
Unfortunately, the post comes off as AI-written. Why not just write your own posts?
CookieCrisp 12 hours ago [-]
Not everyone is as confident in their writing as they are in their engineering
jeswin 12 hours ago [-]
The transformer architecture is fundamentally unsuitable for local inference, while being efficient at scale. It's a fun experiment to try, but that's about it.
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.
even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.
Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.
I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
But do you need to run every small problem through a 10B-30B model?
We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.
Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites
I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit
Is that likely, do you think?
I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.
With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.
Keep your mental context in your brain is critical
This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once.
At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.
Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.
Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.
But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language".
The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe
In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though.
> The correctness of an application is limited by your ability to understand and describe what you need.
If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other.
We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?
At 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.
These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
Let the machine do its work while I relax, I’ll check up on it later.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
That Xeon is going to be devouring power.
It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
It says so right in the readme. They’re not hiding anything.
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code.
I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
its pretty good for building any kind of research report using search engines and it manages context corrextly.
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.
It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
I am curious about the decision to not use GPU since this is for Apple Silicon.
Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than op is suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.
Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but op’s scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
* Software inference optimizations
* Heavy quantization
* Chips with hardcoded transformer architecture
* Much cheaper HBM
* Much sparser models - 1T total with ~1-10B active params e.g.
* Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it eventually reaches phones and toasters and whatever. But not in two years.
That is no where near decent at all.
It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
I know! That's my point! You're a poor salesman of a coding environment/tool if you don't even use it yourself for coding...
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)
If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.
The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.
In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.
For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and close to 50 times more expensive for processing.
My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).
[0]: https://openrouter.ai/qwen/qwen3.5-27b
Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.
It will be feasible for everyone to have 20 different agents running at all times. A new world is coming
Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...
The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.
What does it make clear? That I can replace the space heater my wife runs 9 out of 12 months of the year with a home server? And effectively get $0.00 per token during those times?
In houses running A/C year round, sure there'd be some impact, but in all the places running heat, doesn't seem that it'd move the needle on power bills.
There are startups whose entire business model is "cloud server as a home space heater" (aka "data furnace") ...
That it’s pulling a lot of watts.
Good for you if it’s replacing electric space heaters.
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
https://news.ycombinator.com/item?id=48354801
I'm excited to get my mitts on it on Friday when it finally arrives.
Here's some of the resources I came across if you're interested in reading.
https://echalupa.com/blog/mac-pro-6-1-llama-cpp-firepro-d300...
https://matthewgribben.com/blog/mac-pro-6-1-llama-cpp-firepr...
https://www.youtube.com/watch?v=ILV-eu90te8
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.
A 10 year old Xeon is all you need
https://news.ycombinator.com/item?id=48353348
I've got a 14 cores / 28 threads Xeon from 2015 that I use as a server at home (ZFS / VMs).
It's really a sweet machine.
For ricing I've got a semi-recent AMD 7700X / DDR5 RAM (from 2023 ?) which is my main machine but the real deal is my old and trusty 10 years old Xeon server.
DDR4 ECC is pricey too atm but a 10 years old Xeon is basically free now.
A 20 cores / 40 threads costs maybe 20 USD (for just the CPU). Slap that in a $100 old HP Z440 workstation and you're good to go for quite a few workloads.
Mine is only on when I'm at my computer: it's not turned on 24/7 but more like 8/7 so the entire "but it consumes energy" point is moot.
Whether the writer's setup affects that choice I don't know.
The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.
https://www.pangram.com/history/61fbc90e-180d-4d91-a85d-16ff...
I don't think the post itself reads like AI at all, but that's just me.
https://www.pangram.com/history/61fbc90e-180d-4d91-a85d-16ff...
Android studio connected to a local model disconnects automatically after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens.
I tried Googling, searching for settings in Android studio, even created a stackoverflow post - but zero information. Jetbrains mentions "remote agent timeout mechanism" - but after changing it, nothing happens.
https://docs.ollama.com/faq#how-do-i-keep-a-model-loaded-in-...
You didn't specify what was serving your local model.
Do you know how to switch it in LM studio?
What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected.
So I assumed it was a setting in android studio.
Glancing through the docs, I would be digging down in the config of both Android studio and lm studio for either a TTL or jit auto evict setting, and if you find it, set it to some large number measured in hours?
https://developer.android.com/studio/gemini/use-a-local-mode...
https://lmstudio.ai/docs/developer/core/ttl-and-auto-evict