Hacker News

> In most strong statically typed languages, you wouldn't often pass strings and generic dictionaries around.

In 99% of the projects I worked on my professional life, anything that is coming from an human input is manipulated as a string and most of the time, it stays like this in all of the application layers (with more or less checks in the path).

On your precise exemple, I can even say that I never saw something like an "Email object".

My experience was that enterprise programmers burned out on things like WSDL at about the same time Rails became usable (or Django if you’re that way inclined). Rails had an excellent story for validating models which formed the basis for everything that followed, even in languages with static types - ASP.NET MVC was an attempt to win Rails programmers back without feeling too enterprisey. So you had these very convenient, very frameworky solutions that maybe looked like you were leaning on the type system but really it was all just reflection. That became the standard in every language, and nobody needed to remember “parse don’t validate” because heavy frameworks did the work. And why not? Very few error or result types in fancy typed languages are actually suited for showing multiple (internationalised) validation errors on a web page.

The bitter lesson of programming languages is that whatever clever, fast, safe, low-level features a language has, someone will come along and create a more productive framework in a much worse language.

Note, this framework - perhaps the very last one - is now ‘AI’.

> it feels like sometime after Java got popular [...] a large chunk of the collective programming community forgot why strong static type checking was invented and are now having to rediscover this.

I think you have a very rose-tinted view of the past: while on the academic side static types were intended for proof on the industrial side it was for efficiency. C didn't get static types in order to prove your code was correct, and it's really not great at doing that, it got static types so you could account for memory and optimise it.

Java didn't help either, when every type has to be a separate file the cost of individual types is humongous, even more so when every field then needs two methods.

> In most strong statically typed languages, you wouldn't often pass strings and generic dictionaries around.

In most strong statically typed languages you would not, but in most statically typed codebases you would. Just look at the Windows interfaces. In fact while Simonyi's original "apps hungarian" had dim echoes of static types that got completely washed out in system, which was used widely in C++, which is already a statically typed language.

> You'd naturally gravitate towards parsing/transforming raw data into typed data structures that have guaranteed properties instead to avoid writing defensive code everywhere e.g. a Date object that would throw an exception in the constructor if the string given didn't validate as a date

It's tricky because `class` conflates a lot of semantically-distinct ideas.

Some people might be making `Date` objects to avoid writing defensive code everywhere (since classes are types), but...

Other people might be making `Date` objects so they can keep all their date-related code in one place (since classes are modules/namespaces, and in Java classes even correspond to files).

Other people might be making `Date` objects so they can override the implementation (since classes are jump tables).

Other people might be making `Date` objects so they can overload a method for different sorts of inputs (since classes are tags).

I think the pragmatics of where code lives, and how the execution branches, probably have a larger impact on such decisions than safety concerns. After all, the most popular way to "avoid writing defensive code everywhere" is to.... write unsafe, brittle code :-(

And then clojure enters: let’s keep few data structures but with tons of method.

So things stay as maps or arrays all the way through.

> You'd naturally gravitate towards parsing/transforming raw data into typed data structures that have guaranteed properties instead to avoid writing defensive code everywhere e.g.

There's nothing natural about this. It's not like we're born knowing good object-oriented design. It's a pattern that has to be learned, and the linked article is one of the well-known pieces that helped a lot of people understand this idea.

In 2 out of 3 problematic bugs I've had in the last two years or so were in statically typed languages where previous developers didn't use the type system effectively.

One bug was in a system that had an Email type but didn't actually enforce the invariants of emails. The one that caused the problem was it didn't enforce case insensitive comparisons. Trivial to fix, but it was encased in layers of stuff that made tracking it down difficult.

The other was a home grown ORM that used the same optional / maybe type to represent both "leave this column as the default" and "set this column to null". It should be obvious how this could go wrong. Easy to fix but it fucked up some production data.

Both of these are failures to apply "parse, don't validate". The form didn't enforce the invariants it had supposedly parsed the data into. The latter didn't differentiate two different parsing.

In my experience that's pretty rare. Most people pass around string phone numbers instead of a phonenumber class.

Java makes it a pain though, so most code ends up primitive obsessed. Other languages make it easier, but unless the language and company has a strong culture around this, they still usually end up primitive obsessed.

This is an idea that is not ON or OFF

You can get ever so gradually stricter with your types which means that the operations you perform on on a narrow type is even more solid

It is also 100% possible to do in dynamic languages, it's a cultural thing

Strong static type checking is helpful when implementing the methodology described in this article, but it is besides its focus. You still need to use the most restrictive type. For example, uint, instead of int, when you want to exclude negative values; a non-empty list type, if your list should not be empty; etc.

When the type is more complex, specific contraints should be used. For a real live example: I designed a type for the occupation of a hotel booking application. The number of occupants of a room must be positiv and a child must be accompanied by at least one adult. My type Occupants has a constructor Occupants(int adults, int children) that varifies that condition on construction (and also some maximum values).

> Edit: Changed this from email because email validation is a can of worms as an example

Email honestly seems much more straightforward than dates... Sweden had a Feb 30 in 1712, and there's all sorts of date ranges that never existed in most countries (e.g. the American colonies skipped September 3-13 in 1752).

I think you're quite right that the idea of "parse don't validate" is (or can be) quite closely tied to OO-style programming.

Essentially the article says that each data type should have a single location in code where it is constructed, which is a very class-based way of thinking. If your Java class only has a constructor and getters, then you're already home free.

Also for the method to be efficient you need to be able to know where an object was constructed. Fortunately class instances already track this information.

this is very much a nitpick, but I wouldn't call throwing an exception in the constructor a good use of static typing. sure, it's using a separate type, but the guarantees are enforced at runtime

It's a design choice more than anything. Haskell's type safety is opt-in — the programmer has to actually choose to properly leverage the type system and design their program this way.

I'm not sure, maybe a little bit. My own journey started with BASIC and then C-like languages in the 80s, dabbling in other languages along the way, doing some Python, and then transitioning to more statically typed modern languages in the past 10 years or so.

C-like languages have this a little bit, in that you'll probably make a struct/class from whatever you're looking at and pass it around rather than a dictionary. But dates are probably just stored as untyped numbers with an implicit meaning, and optionals are a foreign concept (although implicit in pointers).

Now, I know that this stuff has been around for decades, but it wasn't something I'd actually use until relatively recently. I suspect that's true of a lot of other people too. It's not that we forgot why strong static type checking was invented, it's that we never really knew, or just didn't have a language we could work in that had it.

When I came to this idea on my own, I called it "translation at the edge." But for me it was more that just centralizing data validation, it also was about giving you access to all the tools your programming language has for manipulating data.

My main example was working with a co-worker whose application used a number of timestamps. They were passing them around as strings and parsing and doing math with them at the point of usage. But, by parsing the inputs into the language's timestamp representation, their internal interfaces were much cleaner and their purpose was much more obvious since that math could be exposed at the invocation and not the function logic, and thus necessarily, through complex function names.

I disagree. I think the key insight is to carry the proof with you in the structure of the type you 'parse' into.

I think that's an excellent way to build a defensive parsing system but... I still want to build that and then put a validator in front of it to run a lot of the common checks and make sure we can populate easy to understand (and voluminus) errors to the user/service/whatever. There is very little as miserable as loading a 20k CSV file into a system and receiving "Invalid value for name on line 3" knowing that there are likely a plethora of other issues that you'll need to discover one by one.

Not speaking for all Go programmers, but I think there is a lot of merit in the idea of "making zero a meaningful value". Zero Is Initialization (ZII) is a whole philosophy that uses this idea. Also, "nil-punning" in Clojure is worth looking at. Basically, if you make "zero" a valid state for all types (the number 0, an empty array, a null pointer) then you can avoid wrapping values in Option types and design your code for the case where a block of memory is initialized to zero or zeroed out.

> what's your take on "Parse, Don't Validate"

Always aspire to that. Translating that to Go conventions, the constructor has to have signature like:

    func NewT() (T, error) {
      ...
    }

In larger codebases, I've noticed an emergent phenomenon that usually the T{} itself (bypassing NewT constructor) tends to be unusable anyway, hence the constructor will enforce "parse, don't validate" just well enough. Only very trivial T{} won't have a nilable private field, such as a pointer, func, or chan.

I'd say that "making zero a meaningful value" does not scale well when codebase grows.

Systems tend to change over time (and distributed nodes of a system don’t cut over all at once). So what was valid when you serialized it may not be valid when you deserialize it later.

Suppose you're receiving bytes representing a User at the edge of your system. If you put json bytes into your parser and get back a User, then put your User through validation, that means you know there are both 'valid' Users and 'invalid' Users.

Instead, there should simply be no way to construct an invalid User. But this article pushes a little harder than that:

Does your business logic require a User to have exactly one last name, and one-or-more first names? Some people might go as far as having a private-constructor + static-factory-method create(..), which does the validation, e.g.

  class User {
    private List<String> names;
    private User(List<String> names) {..}
    public static User create(List<String> names) throws ValidationException {
       // Check for name rules here
    }
  }

If you encode your rules into your data-structure, it might look more like:

  class User {
      String lastName;
      NeList<String> firstNames;
      private User(List<String> names) throws ValidationException {..}
  }

Is this overkill? Do I wind up with too much code by being so pedantic? Well no! If I'm building valid types out of valid types, perhaps the overall validation logic just shrinks. The above class could be demoted to some kind of struct/record, e.g.

  record User(Name lastName, NeList<Name> firstNames);

  class Name {
      private String value;
      private Name (String name) throws ValidationException {..}
  }

> if I am reading input, I am always going to validate that input after parsing.

In the "parse, don't validate" mindset, your parsing step is validation but it produces something that doesn't require further validation. To stick with the non-empty list example, your parse step would be something like:

  parse [h|t] = Just h :| t
  parse []    = Nothing

  process data =
    do {
      Just valid <- parse data;
      ... further uses of valid that can assume parsing succeeded, if it didn't an error would already have occurred and you can handle it
    }

  validate [h|t] = true
  validate []    = false

  process data =
    if validate data then use data else fail "Well shit"

  use [h|t] = do_something_with h t
  use []    = fail "This shouldn't have happened, we validated it right? Must have been called without data being validated first."

Is this not `Result<Whatever, List<Error>>`? There's nothing enforcing that the error side needs to be the value-based equivalent of a single instance of an Exception class.

The important part is not to expose a "String -> Whatever" function publicly.

Definitely do not parse each row into eg pydantic models. You lose the entire performance benefit of pandas / polars by doing this.

If you need it, use a dataframe validation library to ensure that values are within certain ranges.

There are not yet good, fast implementations of proper types in Python dataframes (or databases for that matter) that I am aware of.

Pass as immutable values, and try to enforce schema (eg, arrow) to keep typed & predictable. This is generally easy by ensuring initial data loads get validated, and then basic testing of subsequent operations goes far.

If python had dependent types, that's how i'd think about them, and keeping them typed would be even easier, eg, nulls sneaking in unexpectedly and breaking numeric columns

When using something like dask, which forces stronger adherence to typings, this can get more painful

Speaking personally, I try not to write code that passes around dataframes at all. I only really want to interact with them when I have to in order to read/write parquet.

The circumstances where you would use one or the other are vastly different. A dataframe is an optimized datastructure for dealing with columnar data, filtering, sorting, aggregating, etc. So if that is what you are dealing with, use a dataframe.

The goal is more about cleaning and massaging data at the perimeter (coming in, and going out) versus what specific tool (a collection of objects vs a dataframe) is used.

I understand where you're coming from, but these terms seem fine to me:

This is exactly what, for example, Rust's str::parse method is for. The documentation gives the example:

    let four: u32 = "4".parse().unwrap();

If you spend your life talking about bool having two values, and then need to act as if it has three or 256 values or whatever, that's where the weirdness lives.

In C, true doesn't necessarily equal true.

In Java (myBool != TRUE) does not imply that (myBool == FALSE).

Maybe you could do with some weirdness!

In Haskell: Bool has two members: True & False. (If it's True, it's True. If it's not True, it's False). Unit has one members: () Void has zero members.

To be fair I'm not sure why Void was raised as an example in the article, and I've never used it. I didn't turn up any useful-looking implementations on hoogle[1] either.

[1] https://hoogle.haskell.org/?hoogle=a+-%3E+Void&scope=set%3As...

> no amount of static typing will save you from poorly defined or optimised-too-early types that encode business logic constraints into programmatic types.

That's not a fault of type systems, though.

> because business logic will move faster than whatever code you can write and fix, and exposing it as just "types" breaks the process for future programmers to extend your program

That's a problem with overly-tight coupling, poor design, and poor planning, not type systems

> In practice, I find that staunch static typing proponents are often middle or junior engineeers

I find people become enthusiastic about it around intermediate stages in their career, and they sometimes embrace it in ways that can be a bit rigid and over-zealous, but again it isn't a problem with type systems

how does this square with very senior people putting in a lot of effort to bolt fairly good type systems onto Python and JavaScript?

> business logic will move faster than whatever code you can write and fix, and exposing it as just "types" breaks the process for future programmers to extend your program.

I just don't understand how this is the case. Fields or methods or whatever are either there, or they are not. Type systems just expose that information. If you need to change the types later on, then change the types.

Example: Kotlin allows you to say "this field will never be null", and it also allows you to say "this field will either be null or not null". Java only allows the latter. If you want the latter in Kotlin, you can still just do that, and now you're able to communicate that (or the other option) to all of your callers.

Typed Python allows you to say "yeah this function returns Any, good luck!" and at least your callers know that. It also allows you to say "this function always returns a str".

> I find that staunch static typing proponents are often middle or junior engineeers

I wouldn't go this far as it depends on when the individual is at that phase of their career. The software world bounces between hype cycles for rigorous static typing and full on dynamic typing. Both options are painful.

I think what's more often the case is that engineers start off by experiencing one of these poles and then after getting burned by it they run to the other pole and become zealous. But at some point most engineers will come to realize that both options have their flaws and find their way to some middle ground between the two, and start to tune out the hype cycles.

This is such a tired take. The burden of using static types is incredibly minimal and makes it drastically simpler to redesign your program around changing business requirements while maintaining confidence in program behavior.

> staunch static typing proponents are often middle or junior engineeers

While we're sharing anecdotal data, I've experienced the opposite.

The older, more experienced fellows love static types and the new ones barely understand what they're missing in javascript and python.

You are yet another person who is misguided in exactly the way described in this article by the same author: https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-typ...

To amplify what yakshaving said, this may be the worst forum in the entire industry to title drop in. Half the people in any given article's comments are a CxO or Chief or Head or Director or Founder or whatever, or wrote the article, or invented the technology in the article, or are otherwise renowned for something or another.

See also: "Did you win the Putnam?"

> I don't like to title drop, but I am a Staff Data Engineer

I am a Chief Technology Officer[^1].

Your opinion here is common, and misguided.

Here is why: https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-typ...

---

[^1]: Literally nobody cares.