Hacker News

> Are you buying from a provider that isn't listed on open router already

I am using openrouter with zdr guardrail which routes to any provider that supposedly doesnt train on user data. I also use fireworks (directly, not via openrouter) which is a provider promising zdr and has a bunch of open weights models. My issue is that these zdr providers dont transparently disclose caching/tokens etc and so they end up being far more expensive than directly using DS.

> They announced it already

I meant something I could download and run.

Two (linked) DGX Sparks would do it I guess. Though probably slowly (I'd guess 15-20 tok/sec for decode, but higher for prefill). So ~$8-9k USD at current RAM prices, substantially less if they ever (sigh) drop. Electricity use would actually be relatively modest.

But it makes little to no sense as long as API prices are what they are. Except for maybe privacy reasons.

thanks, corrected! I looked at Q3

It's less than $1/TB. If you have settlement-free peering it's like fractions of a penny at scale. Cloud provider DTO pricing is literally the biggest scam on Earth.

when trying to download models using browser, the DL domain for me us.aws.cdn.hf.co which points to bunch of singapore AWS IPv4 for me, despite it saying US.

It's not only cheap, it's actually free since if you are not using your 95th percentile you are losing money.

Download numbers are reported on the site, and they aren't low.

CDN costs are pennies compared to inference and training though, HuggingFace will just get another 100 million and be set.

I get where you're coming from, and the intent to make it easier for people to find examples and verses, but there's a fine line with LLMs giving you answers, is that it's interpreting it in some form. Doesn't that run counter to prevailing ideology, that you're meant to either struggle with the materials / seek understanding yourself, or have your religious leaders interpret/receive those insights?

Is the difference between this and a frontier model that the scripture is guaranteed to be real?

I'm on a team that develops a Bible study app, and we're all relatively content with how the basic models converse regarding scripture. Even as far back as GPT-4 was excellent. They occasionally have minor hallucinations (a dealbreaker for a production app), but they do an excellent job with theology and Bible scholarship, given reasonable guardrails.

I'll admit I'm coming from the perspective of "should we be implementing this?" It seems, on the surface, that a strong embedding-based verse retrieval covers the bases at a microfraction of the cost.

If you're interested, check out the development server where we're working on this. You navigate to the search (magnifying glass) and then hit "Meaning". Sorry for the confusing route; we're still deciding on back-end details and haven't focused on the front yet.

[1] https://ai.stepbible.org

The training use is probably responsible for this improvement tho.

For anything substantial, you'd want a bigger model anyway.

For simple tasks, they're already saturated, and you'd prefer the faster model, so that you can have a realtime/interactive-ish experience.

Or to put it bluntly, it's cheaper if you don't value your time. That goes for smaller models in general -- need more handholding, more correcting -- but the Chinese ones are slower on top of that.

As for speed, Sol on Low is faster than Luna on most settings.

>And can SOMEONE please tell artificialanalysis that using dark blue for both Deepseek AND OpenAI is an especially unfortunate choice of colors, especially today?

uhh openai is dark gray: `rgb(31, 31, 31)` and i'm pretty sure it always has been?

It’s not outdated at all to use tokens to estimate performance, it’s directly related.

Yes, but… more thinking tokens also means longer solution generation time. That said, v4 Flash is a fast model. I use it all the time because it’s very smart for the price. But it is verbose sometimes.

are you working in earplugs? :)) even with 128 gigs of ram it must be super noisy.

I've put Opus to it and it says it will take 3-4h to do the process to the new weights. hoping it works!

you don’t need new weight. try vllm-moet from github. it will autogenerate 2-bit plane.

What kind of tps are you getting?

same. been running it in a dgx spark and it slaps

Google TPUs are built around a 128×128 systolic array of multiply-accumulate (MAC) units. Trainium 1, Trainium 2, and Inferentia 2 also feature a 128x128 systolic array.

You learn something every day. Today, it was the term "systolic array": A systolic array is a specialized grid of simple, interconnected processing units designed to execute parallel data operations—like matrix multiplication—by rhythmically passing data directly from cell to neighboring cell without writing intermediate results back to main memory.

The term comes from the biological word systole (the contraction of the heart pumping blood through the body). In a systolic array, data "pulses" through a network of processing elements on every clock cycle, driven by a global clock beat.

Just because scales are grouped by 128x128 tiles, does not mean you need a single compute tile that large. It works completely fine to process it with multiple smaller tiles that get given the same scales, like how this works on Hopper and Blackwell today

Huawei Ascend NPU

Blackwell. You could do it with wgmma on hooper but then you'll only be able to run 1 CTA per SM. There are cases where this is OK, but most commonly 128x128 mmas are primitive in tcgen05.mma i.e blackwell. The fundamental reasons is that the systolic array accelerator (TMA) until blackwell wrote to the cuda core registers themselves so you were limited by the register file size of the SM[SP]. In blackwell there's a separate TMEM where the mma unit stores it's output. You can even do higher 256x256 and such using a hardware feature called 2-CTA MMA, this is essentially them letting neighbouring pairs of SMs co-operate and access each other's memory.

As for sibling comments, huawei ascend is more of an NPU-style architecture where you can easily have much bigger MMAs as primitive. But you usually don't anyways for many reasons.

Huawei Ascend presumably?

So is Fable, I’m asking who is the customer if the thing is free

Same. And I have come to use OMP (oh my pi) agent /advisor mode to put a 2nd model on the case (also mid-size one), reading everything. It can not block anything or change anything - just inserts comments in the text stream with 1 turn delay. Good portion of the time it's quiet. I'd say 1/2 of the time it's got something to say. About 2/3-rd of that the 'advice' is insubstantial or about something not-quite wrong. The good thing is the main model is confident - checks and then it stands its ground. Have not noticed it turning a right into a wrong b/c of advisor false alarm. And in 1/3-rd of the advice, it's a genuine defect teh advisor noticed, the main model works out a fix. This is my approximate feeling just observing the process, have not got collected the data. Afaik only OMP has advisor mode. Agent pi has plugin pi-omplike-advisor. For agent Hermes I had them code me an /advisor plugin (for now -0.1 old v0.18.x; yet to upgrade it to latest).

The problem is picking between models. I do not want to spend my time switching models and trying to decipher which model should be used for what. Maybe that's just a me problem that I need to figure out.

what plan are your 'fellow software engineers' using? I have a hard time even using up the Fable part of my allowance in a week of coding.

usually I am - Codex, make pi with deepseek to do something.

The issue for me is the data privacy if you're using their hosted prices, because you cannot opt out of data collection (and I'm not sure how you'd legally follow up on it if they did offer it, but didn't actually follow through). That means all of my code is being retained for training.

I've used it in some open source code though, and loved how fast it was.

My mind is changing on how valuable my code actually is though... it's the complete picture, how it's put together, the design, the UI, the attention to detail that's the real value.

I'm not a heavy user of agentic coding, but still use them quite a bit for some automation here and there. I've been going around shopping all the ~$10 subscriptions and I finally settled on openrouter + ds4 pro. The more intensive days cost me $1 and I set a $2 weekly limit which I've never hit the past 3 weeks, to me it's way cheaper than most subscriptions and I don't have to worry about maximizing my weekly quota/resets.

Chinese are pretty pragmatic. Even if they produce more expensive chips and memory, they are still going to focus on value. But who knows, maybe India will step up again like they did for Y2K 3 decades ago.

I agree. A meeting transcript posted recently from DeepSeek's founder suggests they're going to reach for larger models as soon as they can, and not look back. The model is already bigger than my machine can handle, though (I cannot put up with 25 t/s after getting used to over 100), so I'm not impacted.

We'll make sure there will be enough regulations to have neither the electricity to power it or the companies to build it.

Sorry we a focusing on leather handbags there.

The EU is not a state or some federation. It's like a NAFTA+ It unfortunately doesn't have the risk averse VC system (or pragmatic state driven like China) ecosystem to support the scaling required.

But if models will be commodities, maybe that won't be such a terrible thing longterm?

Seems incredibly unlikely, doesn't it? (I hope for the same thing but it feels almost like wishful thinking!)

Personally I don't. Wasting enormous amounts of resources on a technology that doesn't make money, can easily be copied, and eats up everyone's energy while at the same time can be downloaded over the internet is kind of stupid.

The fact that China could catch up in what, two years, shows that this is a commodity.

EU is out of game and can't be back anytime soon.

Incursions into other nations "just because"? Economic wars, and support for foreign governments or armed groups that caused harm or instability to the entire planet? You know? Planet Earth?

Sanctions and trade policies that hurt ordinary people? Cultural dominance through Hollywood (Holy Wood, again with the sick ruling elite and their witchcraft crap), social media, brands, the poisoning of food and water on global scale?

Etc? Etc?

Most people don't hate "Americans" or their companies, they are just against your ruling elite. Live one day outside the united states and understand how the planet works for the rest of mankind and you might start to grasp on reality.

And yeah, China will catch up with hardware pretty damn soon. Jesus, you guys bought your radios from Japan not 50 years ago. Sorry, can't dominate the world's economy forever. Never happened in history.

No matter how much black magic, dark rituals, sick crap and sacrifices people on the top throws at the wall.

We're humans, after all.

So yeah, it is not against America itself or American companies, but against the ruling class. So naturally , people get excited about China entering into the match!

3...2...1... Fight!

We'll never have fewer open-weight models than exist now. They won't suddenly disappear when labs stop publishing new ones. In fact, people will keep improving them and will keep distilling new frontier models into existing open-weight models.

Then non-Chinese labs can reopen, or they could offer lower prices right now not waiting for this event.

But imagine if Chinese labs would lose. Then there will be no open models, and the prices for closed models would be raised to the maximum.

My conspiracy theory is that this is the new space race, and the CCP encourages this to show the world what Chinese engineers are capable of, and tank the Anthropic/OpenAI valuation bubble as a desirable side effect.

As opposed to closed source American models where the federal government threatens your livelihood unless you build data centers to extend pax silica?

This will never happen btw.

Yes hit rate does vary by harness and by how you use the harness. If you use subagents, for instance, they will start with a whole new context created by the main agent, and this will not be cached. If you mostly use the main agent with Pi, you’ll have high hit rates and low costs. Sometimes agents do “cache busting” things where they’ll move around some of the text in the context to try to keep old instructions from being forgotten, thus keeping the agent on task, and this will bust the cache. I’ve heard, but not validated myself, that Open Code has some issues with this.

BTW, this is one of the things that I really like about Pi. It’s very simple and thus very predictable.

A benchmark website to benchmark benchmark websites?

Or a benchmark to benchmark benchmarks?

By making companies using them "toxic" to touch.

For example: no government contract to any company who uses even one vendor in it's entire chain of dependencies, who uses such open models.

They can extend this further by laying more conditions, such as: any company dealing in this-this field can only use models "officially" approved as "safe". Rest you can guess how easy it would be to get that "safe" rating for such open models.

They may reverse course in the future, but the current directive from the CPC is that Chinese AI labs should be open sourcing their models.

https://www.businessinsider.com/xi-jinping-open-source-ai-us...

I don't think it's necessarily "wiser" to go closed source. All of AI is built on mostly openness, at least on the software side. There are other ways to compete, it's just the model itself will be a commodity.

You’re describing the dump and pump strategy that china has historically used across a number of industries. Stands to reason that this is what is going on.

Because reddit unironically has better decorum around usage of their upvote/downvote system than HN does.

People on HN downvote objectively correct information because they don't like it 24/7. There's a reason the creator of Zig left and gave the computer version of a middle finger on the way out to HN!

@dang is that how we call you?

Surely you realize he or she is not literally looking for the answer to that but rather pointing out the censorship.

Oh? What are the American model censorship tells?

This is a straightforward false equivalency. “Western” models do not censor in the same way, nor for the same reasons, that the Chinese models do. “Just as much” is not remotely plausible, yet it’s doing all the heavy lifting.