Hacker News

SHOULD is a requirement. It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case. "I don't want to" is not a valid excuse, "I don't see a reason to" isn't either.

IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 The SHOULD lets those messages be valid. Low-entropy devices that can't generate a good random ID are rare these days, but old devices remain in service, so the workaround is IMO justified.

The official definition of SHOULD per RFC2119:

  3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
     may exist valid reasons in particular circumstances to ignore a
     particular item, but the full implications must be understood and
     carefully weighed before choosing a different course.

My take, as a postmaster for hosting company, who don't have any sympathy to gmail (that should be visible from my comments history): Message-ID is absolutely MUST in production e-mails. You can send your test stuff without it, but real messages always have it. Not having Message-ID's causes lot of fun things. All somewhat competent software is capable to add Message-ID's, so lack of it is good indication of poorly made custom (usually spamming) solution.

Rspamd and spamassassin have missing MID check in their default rules, I am sure that most antispam software is same.

Exactly. Message-ID is not required.

An unrelated frustration of mine is that Message-ID really should not be overridden but SES for instance throws away your Message-ID and replaces it with another one :(

I would read this as a requirement for email to be 'legit' and not classified as spam.

Sure, you can send email with whatever headers you want, use weird combos, IP addresses, reply-to, and it might be still a technically valid email, but not something that should land in people's inboxes.

Also, a payment processor not testing their email on the most popular email provider in the world is quite ridiculous.

As indicated in the RFC, it uses another RFC[0] to define those words. Here's the relevant excerpt from that one:

    3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
                may exist valid reasons in particular circumstances to ignore a
                particular item, but the full implications must be understood and
                carefully weighed before choosing a different course.

I know you're looking for "pedant points" but the specification generally take a backseat to implementation. If Message-ID is expected out here where the rubber meets the road, then you are the squeaky wheel in this scenario for not including it.

The only messages I receive without one are spam/phishing. I check because they're not recognised by notmuch, so I don't see them otherwise.

And the definition of "SHOULD" (from RFC 2119) is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."

Having said that, I regret my original characterization of the Message-ID header as a "requirement" and have updated the blogpost to be fair to all sides.

Thank you for bringing this up.

GMail SHOULD handle your messages, not MUST.

> That says SHOULD, not MUST, so how is it a requirement?

Battle with spam has been for long part just trying to algorithmically fingerprint the scam bots and reject the message if it looks like it wasn't sent by "real" mail server/client.

So a lot of things that are optional like SPF/DKIM are basically "implement this else your mail have good chance of being put into spam automatically".

The reason that European tech sucks is that people in Europe are open to such arguments. If an engineer in the US started talking about SHOULD vs MUST, some PM would just give them that "what the fuck did I just listen to" face, spend the next few minutes gently trying to convince them that the customer experience matters more than the spec, and if they fail, escalate and get the decision they want.

For example, why does Google handle this differently for consumer and enterprise accounts? Well it's Google so the answer could always just be "they are disorganized" but there's a good chance that in both cases, it was the pragmatic choice given the slightly different priorities of these types of customers.

The reason it’s recommended is that it’s useful for detecting when an email you receive is already in your mailbox, so that you don’t accumulate duplicates. Otherwise one would have to compare the complete email, which probably no MUA does. Another reason is that replies can include a reference to the original message, so that it can be properly threaded by MUAs.

So these are mostly quality-of-life reasons, it’s not a reason to reject an email.

You SHOULD follow the wording of the RFC, you MUST follow Google's interpretation of the RFC.

That is the difference.

You're totally right. I've updated the blog to reflect this. Thank you!

Standard RFC naming, optional field.

The HTTP User-Agent header is also optional, but if you omit it, something like half of all endpoints will respond with a 500 error code.

Also email as a protocol (SMTP) predates RFC5322 by 25 years or so.

Avoid SHALL, SHOULD and all other crap, use Elon MUST.

This is the one that gets me - sometimes you're forced to work with systems that do annoying things that you have to accommodate. It's annoying, but it's more important to do the thing that prevents your users from having issues than it is to be theoretically right about whether something's required by a standard.

I've dealt with many worse cases than this, where the systems I was integrating with were doing things that weren't even close to reasonable, but they had the market power so I sucked it up and dealt with it for the sake of my users. Maybe Google's wrong here, but how do you not just implement the solution anyway?

> I would not like to have business with such a payment provider.

Chances are that the decision-makers in most companies don't care about the technicalities (i.e. which email you used for registration) - they want to get up and running.

The reason that Viva doesn't care, I assume, is the reason Google workspace doesn't care: they're both too big to care for 5% of their clients won't do the extra work. They know that their, usually much smaller clients, will "figure it out" by i.e. using another setup that works™. So why bother?

> how can you ignore all of these bounces from a provider thats likeliest to be the major one you are sending to?

This is the major issue that most of the discussion is missing. It doesn't matter how you want to interpret the word SHOULD, if you want to send to google workspace, you MUST include a message-id. It's not like this is some fly-by-night server with 12 clients.

If you absolutely and completely don't want to include the message-id, then you need to have a warning that your service can't be used by Google Workspace customers. This used to be common practice, blocking communication to servers that behaved badly, and I sort of wish we'd bring it back.

I doubt Google Workspace is going to be the major provider for European businesses

I find the ones that try to be cute the most frustrating because these appear on the new message notifications so I can't just delete them straight from the notification.

We'd love to share this exciting announcement but you'll a different email app.

Although I guess the argument will be that email clients should use AI to summarise the HTML into a plain text summary.

I had one who sent me the booking details of another client in the plaintext part. I reported it to them nearly a year ago and they didn't reply, so screw anonymity, it was Avis.

So I'm wondering a bit here - I've seen an implementation where emails to send only have html versions, but as part of the sending process the html is run through a Lynx browser process with the -dump command to get the plain text, which is included as the text/plain part of the email.

Is there actual value to this? e.g. Is the output of Lynx's text dump better for plain-text email clients than whatever they'd display for html emails?

the best is when some put the same payload in the text/plain part as in the text/html part. yes. the html source. as text/plain.

With a broad statement like this, I would usually just suggest this is inflammatory and surely overstated.

However, I've also worked at a financial institution which used core systems by Harland Financial Systems. Their "encryption" for data in transit from teller workstations to the core system was just a two byte XOR, and they sent the key at the beginning of the connection!

Was so unbelievable to be able to crack this in under a half-hour after noticing patterns in a PCAP. Wouldn't have believed it if I hadn't seen it with my own eyes.

That fraud was good enough for our regulators and theirs, so I have no doubt the industry is filled with rotten incompetence through and through.

There is plenty amount of incompetence in FAAMG. Notepad ....

Do Europe financial institutions have the same level of corruption as the USA? Such as a credit card company authorizing credit card transactions with incorrect expiration date to maximum profit, Bank of America? Or opening new accounts without consumer consent, Wells Fargo?

Interesting that you mention this, as I also switched to Fastmail recently and got more spam than before, but after marking it as spam for some time it now died down I think. This may also be a symptom of changing providers, where the previous provider knew the kind of spam I tended to get from past years, while Fastmail needed some time to get up to speed.

Fingers crossed that the experience will be the same for you.

I don't think a single spam email has ever crossed into my Fastmail inbox. Granted, every service I sign up for gets it's own masked email. But while the @fastmail.nl email that I chuck on my website gets a fair amount of spam, it always gets categorized correctly.

Fastmail seems to go through periods where they're a little slower to adjust to new spam techniques, and they do rely on users filtering somewhat. About twice a year a few will slip through, but if I report them as spam they soon stop.

I've been a happy customer otherwise for years, for what its worth.

I've considered this switch. You're saying that previously gmail was dropping the emails, or they were landing in spam?

It's not necessarily the support person's fault. I worked in support way back when. There were some long-standing issues that needed only minor work to fix that engineering management didn't get, didn't care about or just wouldn't prioritize. Engineers weren't free to work on what wasn't prioritized. Acknowledging a problem and leaving the ticket open doomed to be unsolved forever negatively affected my performance rating. I had to respond with this sort of cheery everything's fine message and write it off as a customer issue, even though I didn't believe it.

Exactly. This minutiae is all so weird. Email as a formal specification does not work, and the industry as a whole has accepted that for decades now. It's not possible to filter spam from valid traffic without applying a truckload of heuristics and leveraging an ever growing set of auxiliary signals (SPF, DKIM, yada yada).

To wit: basically everything in this world is a "SHOULD", at best. The rules are a conversation.

They are testing it, every time someone signs up and it fails. We don't know that this wasn't something that changed on Google's side, so IMO it's a bigger indictment that no one is monitoring their live email deliverability

yeah, because the whole world uses Google workspaces, right /s

Interesting, but what do you gain to send an email which you know will not land?

While there is some truth in while you saying, I have to say that from my European perspective all the big American companies feel enormously bloated. For example, I've recently learned thats Atlassian has 13000 employees, and I have to ask myself, what do all those people do?

it's a greek company, if you buy from greek websites i guess you'll deal with it

i'm not greek but a greek ecommerce i buy from uses viva

My personal fork of ttrss, from 2005, is a dodgy patchwork of fixes for badly formatted RSS. I can't imagine trying to host a service that deals with RSS feeds from random sites at scale.

One that always irks me to no end, is every time I see someone ham fisting csv handling by hand instead of using an established, well-sourced library. They almost always fail at commas or newlines in quoted text... It's one of the more annoying things.

Currently working on replacing a couple decades old system, and my csv output is using a library that isn't quoting all the strings that don't require quotes... so I'm forced to do it (for compatibility) with the other system this csv is going to. (sigh).

Google's Postmaster Tools site has a "Report deliverabilty issue" link at the bottom left navigation column.

https://postmaster.google.com/v2/sender_compliance

Interestingly the MX record of via.com points to Google, but their verification emails could come from anywhere of course. The IP address in the log is also a Google IP, although that could also be the receiving IP.

Just did! My mac mini got pwned though and I wish I didnt give it SMTP accesss… sigh

I just hope my OpenClaw skills registry doesnt have malware anymore. I sure trust my supply chain of vibecoded software!

Thank you for the recommendation! That I couldn't sign up using a form and I had to "talk to their team" was a turn-off for my (extremely extroverted) self.

Google is at least less arbitrary than Microsoft. Microsoft will decide an email is spam today, and tomorrow the exact same email is perfectly fine. I think Google relies more and more on sending IP and domain reputation rather than content.

There's actually nothing that prevents that, if you craft the right headers you can reply to a thread you were not included in, and have it show up as a reply in the thread of common clients (tested Gmail and Outlook).

We added this feature at my $dayjob and I was quite surprised there is no authentication. But thinking about it, this is how mailing lists work (you aren't explicitly specified in "To:") so it makes sense you can do this.

Do you actually not receive their emails? Fidelity uses tracker dots to check email receipt, which drives me nuts, because like any sane person I don't allow emails to load images without damn good reason. My brokerage should not be sending me cute dog photos, thus I have no need of their images.

So they send me an email, send me another email saying they can't reach me by email, then mail me a letter with the same content as the original letter, and mail me an additional letter saying they can't reach me by email.

I usually reply with snark dialed up about 50% asking if anyone had actually read the original message. And include detail about how it is not, in fact, "ok"

The other day, I literally had trouble signing into a website... then I tried filling the contact us form, about the bug... only to have that fail... call in, have the person on the other end schedule my appointment, then almost drop the call without actually logging my bug report/complaint about the whole issue that had me calling in the first place.

SHIBBOLEET. I was seriously thinking of this when I contacted them :)

Unless you are running a more complex setup SPF, DKIM and DMARC really aren't that complicated. They are annoying and additional checkboxes you have to go trough that are hard to fully automate because they require access to DNS, but they are more busywork than difficult.

Domain and IP reputation and all the other quirks of deliverability are much more of a headache. DMARC is setup, test and done. But deliverability in praxis is something you cannot just test and can break at any time. The second worst are email providers that do whitelisting for email and require you to go through their process to even be allowed to send emails to their customers. The worst are providers that randomly decide to drop your emails without informing you or giving you a proper way to appeal as a small sender. If you're not a large email provider they have no problem just dropping you and the fault is on you because your service is the only one that is not working.

And then there are so many more intricacies of the ancient email protocol. Compatibility with horrendously outdated and misconfigured mail infrastructure is particular frustrating to me. I'm running a modern, properly configured, state of the art email server, but get ghosted by large email providers, yet have to deal with the broken mess, much bigger providers than myself are, which of course have no trouble delivering their broken spoofable email just because they are large enough.

Most of that can be mitigated, or at least centrally managed, using an ESP like Mailgun or Sendgrid.

It is a pain in the ass though, coming from someone that had to dig their domain out of "low" reputation with Google Postmaster.

Some of us selfhost email, like me for 30+ years, and have 100% deliverability.

Until the platform owner bans for whatever reason and if communication by way of the platform was your only means of communication with your customer base, that's the platform owner having the power to destroy your business. No different that businesses that rely on the neverending goodwill of the mobile app store owners. One misstep and your business is gone with no recourse whatsoever. Protocols > Platforms. Always.

Interesting, your take away is that Google is the one with the bug here?

If you read two paragraphs further than the Tl;Dr:

> To unblock myself, I switched to a personal @gmail.com address for the account. Gmail's own receiving infrastructure is apparently more lenient with messages, or perhaps routes them differently. The verification email came through.

1. Email didn’t arrive in his inbox of his Google workspace

2. He checked workspace email logs (with admin you can do this on gsuite)

3. It showed the intentional non-accept

4. Comprehending the problem, he switched to personal Gmail

5. The email arrived

6. He informed the sender of the original problem which he worked around

7. Sender is tech-illiterate and did not realize what the problem is. This is common with first line customer support so that happens.

The question to ask is whether you are literate in English or you skimmed too fast. Because I did a 30 s read of the article and got that.

It does seem unlikely that there are no customers on google workspace who have tried to use viva. I don't do payment processing, and my email is via zoho, so I've no idea how large either of those groups are.

I wonder what google workspace support said.

So, I've got to use my old account. I just found out via sources that he verified this account with the code from his main email. It was in his spam which is how he was able to see that the email headers weren't what he wanted. Which is why there was a log for a "bounced" email. I can't believe people don't realise bouncing means the mail server couldn't find an inbox.

And I think Viva is going to be pissed that I'm being stopped from pointing out the absolute lie here.

Lovingly yours that_guy_iain.

Dang, honestly, this is going to blow back big time because someone has clearly decided to stop me from editing my comments which means you're liable for damages and in breach of EU laws. YC is big enough and has enough interests in the EU to qualify. And it's the fact you've removed my ability to redress if your lawyers want to deal with it. And I'm pretty sure someone at YC is going to know how much money I'm going to get and who I'm getting it from which is the most impressive thing. And what my tagline is in certain circles.

The definition of SHOULD is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."

I suspect viva.com didn't consider the full implications, and I suspect Google did some hard math on hours saved for their customers

As a European (who spent 15 years in the US), I coudln't agree more. And while I agree, at the end of the day, I just want the better product for me.

Following Postel's law results in the normalisation and proliferation of defective implementations. The actual standard becomes irrelevant, and new implementations have to be coded against the defective ones.

My opinion is that Postel's law should be approached in the same way that Linus Torvalds did CVS when designing Git. If in doubt about an implementation decision, consider what Postel's law would recommend, and then do the exact opposite.

That “law” if from a different time, before protocols like SMTP became adversarial. It assumed everyone was acting in good faith.

The biggest war on small providers is waged by other small providers. They can be ancient and outdated or simply extremely picky. Which makes everything Google or others require a piece of cake, which it actually kinda is.