A new Go API for JSON (go.dev)

329 points by olexsmir 18 hours ago

modulovalue 16 hours ago

I'm working on better SIMD support in Dart and one of the motivations for that work is specializing parser specifications based on schema specifications of that language.

Or more concretely, if we had a parser generator + json parser specification that can generate efficient simdjson style parsers, then we should be able to parse json restricted to a subset of json based on a schema at least as efficiently as simdjson can parse general purpose json.

That way we could have the best of both worlds, a (potentially lazy) statically type-safe model and simdjson performance!

I still think that this is definitely doable, it's just that a lot of infrastructure needs to be built for it.

I remember reading that Meta is doing something similar with binary formats. I find it weird that there's nothing like that for json yet.

weitendorf 16 hours ago

I was working on metaparsing earlier this year (specifically over grpc because of its better support for types and better serialization format).

I think most people who use JSON don't want strong typing or to use schemas. If you do, you just use a different serialization format: JSON is designed to be a generic, human-readable way to basically represent any kind of data structure (since it deserializes to JS objects). So everything about JSON is working against you if you do try to enforce type/schemas on it.

A grpc-like protocol with more prescriptive type registries/discovery (over the network) and better interop with http (like grpc-gateway) would be the best approach IMO. Definitely not JSON schema

tcfhgj an hour ago

> think most people who use JSON don't want strong typing or to use schemas. If you do, you just use a different serialization format: JSON is designed to be a generic, human-readable way to basically represent any kind of data structure (since it deserializes to JS objects). So everything about JSON is working against you if you do try to enforce type/schemas on it.

I have been using serde without issues, and never resorted to `serde::Value`[0], the untyped option, so far.

I have yet to experience anything working against me.

[0]: https://docs.rs/serde_json/latest/serde_json/enum.Value.html

dankobgd 42 minutes ago

I definitely want jsonschema, it's a standard and i can use it for many tools. openapi spec is also a standard and built on top of jsonschema. I can use jsonschema to generate models, types, servers, client sdks, automate forms etc. I think it's better than grpc for some of these things, plus protobufs and grpc have bunch of tooling that i can pretend to understand but in depth i don't really.

ahtihn an hour ago

> I think most people who use JSON don't want strong typing or to use schemas.

We use JSON because it's standard, human-readable, mostly self-described and works natively with Javascript.

You can read JSON object without having to know the schema even if one exists.

Protobuf is unusablr garbage without a lot of supporting tooling.

thayne 14 hours ago

> I think most people who use JSON don't want strong typing or to use schemas.

Not necessarily. It's biggest advantage is probably that it has very good interoperability. Basically every language has a readily accessible library for json, and even if you don't, it isn't that hard to write a parser and/or generator for it. And being human readable isn't something to sneeze at either.

lightandlight 16 hours ago

brunoborges 14 hours ago

> "JavaScript Object Notation (JSON) is a simple data interchange format [...]"

Clearly it isn't, otherwise every language would have sorted it out by... a long time ago.

yubblegum 12 hours ago

Meet the "Simple Object Access Protocol": https://www.w3.org/TR/2000/NOTE-SOAP-20000508/

kccqzy 13 hours ago

Some languages have a culture of doing things correctly and conforming strictly to the spec. These languages have indeed sorted it out years ago. Haskell for example. Any language that doesn’t have a special number type for JSON is a half-assed implementation, because 1e1000 is a perfectly valid JSON number that is beyond the typical IEEE754 double precision floating point numbers; does your language handle that correctly? Haskell does. This is but a small example of a strong culture bias towards correctness.

ncruces 13 hours ago

Ironically Go has(d) json.Number which almost no one used, so it got dropped from v2: https://pkg.go.dev/encoding/json#Number

You can still use it though, and I built my tiny decimal library around it: https://github.com/ncruces/decimal

layer8 12 hours ago

The spec (RFC 8259) leaves significant leeway to implementations:

A JSON parser MAY accept non-JSON forms or extensions.

An implementation may set limits on the size of texts that it accepts. An implementation may set limits on the maximum depth of nesting. An implementation may set limits on the range and precision of numbers. An implementation may set limits on the length and character contents of strings.

Add to that things like duplicate keys being allowed by the spec. There’s a lot of implementation choices to decide on.

And then things can get really complex with JSON Schema.

kccqzy 11 hours ago

DrBazza 3 hours ago

My company uses JSUN (/sarcasm) where O = object, and U = union.

Who knew the same object could have completely different definitions based on some flags?

ivanjermakov 12 hours ago

Simple data format does not imply that API to work with it is easy to get right.

DanielHB an hour ago

I mean, it is better than the disaster that is CSV...

jayd16 13 hours ago

Ok so nothing is simple then?

CamouflagedKiwi 17 hours ago

I generally like `encoding/json/v2` and I'm glad they did it.

I wish they'd left `MarshalIndent` in there though. The equivalent code is super complicated and non-obvious, so it's a pain to migrate, and it's a pain when you're banging out some quick code and want to dump some readable JSON. I suppose these days the LLMs can write the complicated version for us, of course.

athorax 17 hours ago

I'm not sure if I'm missing something, but isn't this functionality just moved into jsontext? https://pkg.go.dev/encoding/json/jsontext#Value.Indent

  v := jsontext.Value(b)
  v.Indent()
  fmt.Println(string(v))

irq-1 15 hours ago

    func WithIndent(indent string) Options
https://pkg.go.dev/encoding/json/jsontext#WithIndent

arccy 16 hours ago

    json.Marshal(v, jsontext.WithIndent("  "))

?

rednafi 17 minutes ago

This v2 idea in the stdlib should have started even earlier. So I am glad we are getting a better API for json/v2.

This pragmatic attitude towards software engineering is exactly why I still reach for Go for pretty much everything these days. I still read and review code.

So working with a language that doesn't throw a splash of sigil soup on my face is quite nice while reviewing clanker output. Also, being able to do all this without needing 5 crates is nice. Oh the fast build time helps too.

txomon 13 hours ago

I used the jsontext package at my company when we needed to make a parser for challenging clients (specific errors for each kind of mistake) while still keeping performance, and it works like a charm.

There is no introspection and speed is superb. I would definitely recommend jsontext as a parser for anything non-straightforward.

dofm 16 hours ago

I… think this is good?

I'm just getting the neuron-band back together where Golang is concerned. I have done some in the past for little system tasks, but I am switching away from interpreted languages entirely for web stuff, and initially to Go, as I switch away from hand-coding to at least LLM-supported coding.

So I am a bit out of date on the pain points (I never really felt pain points when I last used it, because the jobs I was doing had very small, quite Go-friendly shapes).

cbarrick 16 hours ago

The fact that this took 6 years to land stood out to me.

I'm not entirely surprised, and I'm glad the Go team is carefully considering their interfaces before launching. But 6 years seems like a lot of work to land a JSON API.

awesome_dude 16 hours ago

Sometimes I wonder about Go's velocity - generics (something that I still haven't used beyond the tutorials) took an age as well

JSON processing, quite markedly, is renown in Go for being slow, so I don't really understand why it took so long

Philip-J-Fry 13 hours ago

For like 99% of use cases encoding/json was fine. Of course improvements are welcomed. But personally I've lead development of the backend of part one of the biggest sites of its kind in the world, and we just used encoding/json and net/http. Most of our use cases are just building APIs, client facing ones, internal ones, trusted third-parties. These systems have pretty high throughput, more than most people will ever work on.

Within our company I saw people in other teams writing their own optimised JSON libraries for whatever hellish reason. I always reiterated the point that as soon as that app made a single IO request all those nanosecond savings are invisible. Yes, it affects your total CPU time for processing a request but compute was never ever a limitation for us and it's likely not a limitation for most people. And if the trade off is your app needs scale slightly larger versus writing custom JSON code or pulling in a third party dependency, then I'm accepting the larger scaling.

The biggest benefit of encoding/json/v2 isn't the speed, it's the improved API.

hnlmorg 16 hours ago

I think in the case of JSON, it was “good enough”. People who found this to be a problem would use one of the many 3rd party options.

I kind of like that the standard library doesn’t move fast. It makes it easier to fit the ecosystem into my head (unlike, for example, the JavaScript ecosystem where I find code doesn’t survive 2 years before failing to compile).

But I do agree that slow and steady produces its own problems. Your example of generics is a good one. But think the example of Go modules is another, perhaps even better, example of how sometimes the Go team really should have accelerated their efforts on.

awesome_dude 13 hours ago

win311fwg 15 hours ago

> generics (something that I still haven't used beyond the tutorials) took an age as well

Only about a year from when the Haskell guys lent their expertise in designing it and it landing in a release. That's actually fairly quick given that it was a pretty significant change under the hood and constrained by only two releases annually.

The Go team was always clear that they didn't have the necessary capabilities internally, despite a lot of failed efforts trying anyway, and were relying on the open source community to lend a hand. Unfortunately, open source doesn't mean open willingness to contribute. It turns out that for all the experts on HN, posting "go doesn't have generics lol" was preferable to helping out.

pjmlp 2 hours ago

enneff 15 hours ago

nonethewiser 9 hours ago

Go is an incomplete language masquerading as a simple one.

throwaway894345 8 hours ago

euroderf 4 hours ago

There is the superset of JSON known as JSON5. But A.I. tells me that nobody has (yet) built a JSON5 implementation that builds on encoding/json/v2 .

sneak an hour ago

Any modern redesigned json library in 2026 should have at least a flag to serialize and unserialize json5. Sad that they had this opportunity here and blew it.

asim 16 hours ago

At this stage to me Go is the new C compiler. Quite honestly I spent 10 years or more writing Go code. At times 18 hours a day. And now with AI, I get models to write my Go code. I STILL want it to be Go code. Is that strange? No because I understand the performance and safety of Go. But I rarely write it directly anymore. Probably not at all in 6-9 months. Yet I still occasionally look at the code to see what the output looks like.

So yea hopefully a nice new addition but only my prompts now matter. So need to make sure they pickup on the new API.

dofm 16 hours ago

This is mostly why I have come back to Go.

I am using local LLMs and Go has a bunch of strengths where smaller contexts are concerned; less complicated flow of control being one of them.

calvinmorrison 15 hours ago

not a lot of overloading complexity, very verbose with a small total language structure too helps.

dofm 15 hours ago

BigTTYGothGF 9 hours ago

> At times 18 hours a day

I hope you're not still doing this, it's nothing to brag about.

asim 5 hours ago

No not a brag. I haven't done this in a long time. It's really unhealthy but I was trying to get a startup off the ground and became obsessed. Don't recommend it.

karolist an hour ago

vikramkr 9 hours ago

As long as you do it because you want to and not because you're forced to for work or something who cares? Whatever gives people energy

jimbokun 8 hours ago

egnehots 15 hours ago

Truly the argument can be made either way.

If you code in Go, the code is more high level, prettier, simpler.

If you code in Rust, the code is more robust, efficient, faster.

Both are important for LLM, why can't we have a high level, simple language that is also robust, efficient and fast? =)

Vaushite 8 hours ago

Efficient code means code with a large amount of embedded assumptions. Code that's more generic is almost always less efficient.

Go is about as far as you can "go" to the efficiency end without becoming Rust. It's a low-overhead language (like C) but with a GC that ensures memory safety. If you want to manage memory yourself and still prove safety it gets very complicated, as we see in Rust.

satvikpendem 10 hours ago

OCaml fits your bill. Super fast compile times and also very fast runtime.

jimbokun 8 hours ago

andriy_koval 15 hours ago

> Both are important for LLM, why can't we have a high level, simple language that is also robust, efficient and fast? =)

I think many variants from Rust can be stripped to make it simple, especially in type system, which is way more complicated than go.

nonethewiser 15 hours ago

TheChaplain 15 hours ago

Goose?

rTX5CMRXIfFG 11 hours ago

Swift

jimbob45 14 hours ago

If I want simplicity and maintainability, I’d pick Kotlin over Go. If I want speed, I’m picking Rust over Go. Why would I ever pick Go? It’s the best at nothing.

coldtea 10 hours ago

procaryote 14 hours ago

jacobgold 15 hours ago

I still prefer Go over the alternatives because I'm reviewing the code carefully and trying to make it all look as good as if I'd written it very carefully by hand.

The few times I've needed a language without a GC and used Rust and C as target languages, I've reviewed the code less and enjoyed it less.

hoppp 12 hours ago

Go is lovely. Best language to generate backend services with.

pjmlp 2 hours ago

JVM and .NET platforms.

JIT and AOT compilers, REPL development, bleeding edge GC implementations, Java, Scala, Kotlin, Clojure, C#, F# to chose from, and modern IDEs inspired by Smalltalk/Lisp Machines experience, batteries included backed and frontend development frameworks.

wannabe44 an hour ago

biztos 15 hours ago

As I do more agentic coding, I am starting to feel strongly that I need to set aside a project -- and time -- for strictly hand-coding, using AI (if at all) only to explain things, never to write any code.

First, I like programming and don't want to stop doing it just because a machine can do it too. Second, I'm pretty sure atrophied coding skills will make me worse at directing the machines, thus worse at what seems to be "the job" these days.

nonethewiser 15 hours ago

>First, I like programming and don't want to stop doing it just because a machine can do it too. Second, I'm pretty sure atrophied coding skills will make me worse at directing the machines, thus worse at what seems to be "the job" these days.

In theory wouldn’t atrophied coding skills not matter if models are sufficiently good at coding? I mean isn’t whats left just design and architecture and testing?

Sure, if LLMs are producing shit code you need to have an understanding of what it should do instead. But I think that day is already gone.

Its hard to think of a skill that you’d need to build with an LLM that requires writing meaningful amounts of code by oneself. From my end all the complexity is business logic and system design.

cyphar 13 hours ago

swatcoder 14 hours ago

xgulfie 14 hours ago

nextaccountic 7 hours ago

queenkjuul 8 hours ago

lkirkwood 6 hours ago

Do it! I have the same feeling and the hardest part is picking something that you are interested in. I have just kept the idea of "thinking of ideas" on my mind for a few days once I finish something. I usually come up with a few ideas if I'm always considering any little thing for a project. It has been fun and I have learned a few things along the way - more than I feel I do letting AI write most of the code.

nilamo 14 hours ago

I haven't actually tried it yet, but I think it'd be good to write a dsl for whatever you're doing. Then have ai make an interpreter to run it. Sort of as a way to stay... interested? Engaged?

queenkjuul 8 hours ago

That's exactly how i do all my personal projects. No agents, AI just for planning/research/debugging and occasionally for small bits of code i am not super interested in. For example I'm currently working in Go, but need to use a C library, and don't know C much at all. So when i get stuck with the C interop parts, sometimes i let the AI write bits of C for me, or let it write scripts for cross-compiling or other stuff i just find to be a pain. But I'm doing all the writing and logic myself.

Agents make it too easy to gloss over too much. They're great at work where i care about productivity but I've zero interest at home where the point is learning and having fun

c0wb0yc0d3r 8 hours ago

This is sort of the deal with any language. I’m using go for personal projects and dotnet for paid projects.

Why? Because I don’t give time to wait for code to compile on my time and I don’t have time for people to learn a new language for review at work.

Life is good! Would recommend that split.

da02 12 hours ago

Which are your favorite models for writing Go? Any models for editing/improving written documentation?

citizenpaul 15 hours ago

People are sleeping on Go as a LLM language of choice. Everyone trying to rewrite in rust which is fine and rust is fine. However its compile times are several orders of magnitude greater than Go. You can have LLMs write and compile 20 apps in the time one rust app compiles. I've got a Go personal project that would take hours for rust to compile, I can't imagine making the same velocity in Rust. Though I do see the workplace appeal of bringing back the xkcd "compiling" meme.

However it kind of makes me feel like my professional life was wasted and I don't really know what kind of future I have. One of Go's main points was you can pick it up in a weekend. What career moat do people like us have?

simlevesque 15 hours ago

My only issue with it is that you can't ship a wasm go app in under 2mb. With rust you can make it under 100kb.

xarope 4 hours ago

groundzeros2015 9 hours ago

pjmlp 15 hours ago

I would rather have LLM directly generate machine code, and best of all, macro Assemblers are blazing fast for decades.

dgb23 2 hours ago

bayindirh 15 hours ago

kibwen 9 hours ago

politician 9 hours ago

hoppp 12 hours ago

I like it. I never really liked marshalling anyways, It was weird.

I'm happy it's getting fixed!

troupo 16 hours ago

> in any, out any

Completely unrelated: can anyone tell me why language authors introduce in/out/inout function parameters (see also C#).

It feels like this is some rather bad workaround for something that breaks assumptions about parameters being passed into the function.

hxtk 14 hours ago

In Go, it’s a question of who owns the allocation. If you return a pointer type allocated within the function, the answer is “no one,” so the pointer escapes and has to be managed by the garbage collector.

If the allocation is owned by the calling function which uses it internally and then doesn’t return it, it can be a stack variable. Or the same allocation can be reused many times in a loop. Or an array of allocations can be made all at once with foreknowledge of the total number.

throwaway894345 7 hours ago

If you pass a pointer into a function, the Go compiler can (and sometimes does) allocate it on the heap. Go has no allocation/ownership semantics, tragically, so the best we can do is depend on undefined behavior regarding the escape analyzer and hope the Go team doesn’t break it (which is a relatively safe assumption, considering that it’s unlikely that the Go team would start allocating things on the heap that were previously allocated on the stack—Go rarely regresses performance like this).

ahmedtd 13 hours ago

The parameters named "out" here are not "output parameters" in the way you seem to be thinking. They are "input parameters" named "out", because they are the buffer or pipe you want the data written to.

jayd16 13 hours ago

`out` params in C# are params that must be set within the function. They are way to implement multiple return values and tell the caller "this is written to, not read from."

But you might be conflating that with in/out for Contra/Co-variance when specifying a generic.

metaltyphoon 11 hours ago

> They are way to implement multiple return values

They are one way to implement multiple return values. For multiple returns, tuples can be used. In fact, async methods can’t declare out params

duskwuff 15 hours ago

"in" and "out" are the names of the parameters, not qualifiers.

CharlieDigital 15 hours ago

C# `out` is kinda nice because it can be `var`d and used directly.

troupo 14 hours ago

Erm... what do you mean?

metaltyphoon 11 hours ago

wiseowise 16 hours ago

This is how you roll out features, C++. Congrats to Go team and users.

SkiFire13 14 hours ago

> nil slices and maps are marshaled as an empty JSON array or object by default

I feel that is the kind of broken default that will come back and bite a bunch of people, and will eventually need to be fixed in a v3.

travisd 14 hours ago

I think it'll bite some people, but probably is the better default for JSON specifically. Most of the nil-maps-become-null thing is just an extension of Go defaulting to nil for empty arrays and maps, which is usually not what you expect when you're writing a JSON schema like

    type ListUsersResponse struct {
        Users []User `json:"users"`
    }
in Go. It's pretty common have code like

    userRows := db.Query(...)
    var users []api.User
    for _, user := range userRows {
        users = append(users, api.User{...})
    }
where you don't _really_ intend for the default nil-ness to leak out. (Apologies for any broken syntax, I'm realizing in real time that it's been a while since I've written Go)

SkiFire13 6 hours ago

> Most of the nil-maps-become-null thing is just an extension of Go defaulting to nil for empty arrays and maps, which is usually not what you expect when you're writing a JSON schema

This is definitely not what I expect in other language. I would argue this is a quick of Go though and not JSON's fault.

devmor 14 hours ago

Can you explain why? This seems like the appropriate expectation to me, given how JSON is constructed and consumed.

Someone 5 hours ago

Not the OP, but serializing both nil and empty arrays to the same json string means round tripping will not work for all objects.

That seems like a bad default to me.

throwaway894345 8 hours ago

How is this a broken default? In Go, I almost always want a nil slice to be treated exactly as an empty slice, and JSONv1 was the ~only place this didn’t work as expected, leading to frequent bugs.

the_gipsy 3 hours ago

The root cause is this idea of "zero values". Any de/serialization in go will always be crutchwork of guessing and incorrect validation.

Just look at the inane rules for map and slice access, what gets nil, what gets the zero value, which built-in mist be used. Even if that were consistent, then you'd still left to wonder if json.field was truly zero or not there.

gosen 14 hours ago

wohoo, good news indeed better later than never :D as an active Golong user I'm very happy

lolpanda 17 hours ago

the marshal api has been very annoying. why does marshal return an error? the only reason the marshal can return an error is that the input type is not marshalable. it's certainly a code bug. to me it's like int+str returning an error instead of compiler error

duskwuff 16 hours ago

> why does marshal return an error? the only reason the marshal can return an error is that the input type is not marshalable

Not necessarily. Among other things, MarshalJSON methods called during marshaling can return an error for whatever reason the developer sees fit - one possibility that comes to mind is that part of the object tree being marshalled is lazily loaded, and failed to do so.

And, as others have mentioned, there are types like "any" whose marshalability can't be determined at compile time. Those types aren't necessarily at the top level, either; they could appear anywhere in the object tree.

sgsjchs 16 hours ago

Types may be mostly marshallabble, have a subset of values that are unmarshallable. E.g. non-finite floats.

hnlmorg 17 hours ago

I wouldn’t class unmarshallable data as a compile time error.

Plus the point of the marshal interface is that it’s reusable.

thayne 17 hours ago

It could be if go had a more powerful type system.

hnlmorg 16 hours ago

bicarbonato 15 hours ago

AFAIK it's mostly because of floats. JSON doesn't support `NaN` and `infinity` for example.

Because they are common types, makes sense to precautionary have the error return.

0x696C6961 16 hours ago

Then write yourself a MustMarshal wrapper that panics.

jeremyloy_wt 16 hours ago

A generic Must[T] you mean ;)

hnlmorg 16 hours ago

The GP is talking about compile time errors. You’re discussing exceptions, which is still a runtime check

leetrout 17 hours ago

JSON is a runtime input of unknown type but in your example those are known types so the compiler can catch them.

lolpanda 16 hours ago

i meant i hoped it's panic as opposed to return an error. if there was an error, how am i supposed to handle it? can i tell the user hey sorry my bad it's likely a code error? crash is better than possible eating the error siliently

iaaan 16 hours ago

It is trivial to ignore an error in Go if you know your use case cannot result in one. But at the end of the day, json.Marshal takes an `any` as a parameter, so it must return an error.

NoDodgeQuestion 17 hours ago

do not know at compile time that input is not marshalable, every language marshal must return error for this reason, not only golang

slopnt 14 hours ago

Now guess what will happen when they add monadic error handling. Yes, correct, there will be another JSON Go API

rednafi 13 minutes ago

json/v3 ...so what? The v* idea in the stdlib allows things to move when needed.

muddi900 4 hours ago

"There only ONE way to do each thing! Only ONE! No exception"

"What if it sucks?"

"There are now TWO ways to do each thing! No more than TWO! I promise!"

It's like python again.

Mawr 2 hours ago

Sure, if you ignore the massive difference in the rate such changes are happening at and that this new api is strictly better than the previous one, so there's no two ways that need to be decided between but rather the new go-to and the old-to-be-moved-away-from option, so there's no cognitive load here at all.

So yeah, if you ignore most of what matters, everything is the same.

0xmattclark 12 hours ago

I think I can build a new web tools serving for this new API