Hacker News

The problem is stats can actually do more with all the data including obvious errors. If you start filtering out data where they miss entered lat log you might introduce a new bias.

Sure we should indeed expect that they do that. But look at enough data and you'll learn that those expectations are a path towards never-ending frustration. I've been there, spending >100 hours cleaning data... that never got published because I was too damn focused on the dozens of years of errors that many, many people created.

To be clear, I'm not saying that we should accept messy data. Just, reality is messy and it's naive to think we can catch and remove all of reality's messiness -- which includes the bureaucratic slop that led to the data being published in the first place.

Yep, expect in real data actually confusing columns, NaNs casted to values like 1673, duplicates, etc, etc.

I prefer to get data with swapped lat/lng (a trivial fix), or prices said in dollars but being in cents, to no data.

"What does it mean to clean the data?"

This isn't possible to answer generally, but I'm sure you know that.

Look -- I've been in nonstop litigation for data through FOIA for the past ten years. During litigation I can definitely push back on messy data and I have, but if I were to do that on every little "obviously wrong" point, then my litigation will get thrown out for me being a twat of a litigant.

Again, I'd rather have the data and publish it with known gotchas.

Here's an example: https://mchap.io/using-foia-data-and-unix-to-halve-major-sou...

Should I have told the Department of Finance to fuck off with their messy data? No -- even if I want to. Instead, we learn to work with its awfulness and advocate for cleaner data. Which is exactly what happened here -- once me and others started publishing stuff about tickets data and more journalists got involved, the data became cleaner over time.

One of those people can republish their cleaned and validated version and the 999 others can compare it to the original to decide whether they agree with the way it was cleaned or not.

What...?

Agreed--and maybe they should have fixed it.

But sometimes the "provenance" of the data is important. I want to know whether I'm getting data straight from some source (even with errors) rather than having some intermediary make fixes that I don't know about.

For example, in the case where maybe they flipped the latitude and longitude, I don't want them to just automatically "fix" the data (especially not without disclosing that).

What they need to do is verify the outliers with the original gas station and fix the data from the source. But that's much more expensive.

It's not obvious to me that LLMs can't be made reliable.

Not sure, but I have worked a lot on stuff where the metrics were often very easily convertible to business decisions, like expenses or income. Things like how much do we need to pay for infrastructure, how much money each customer/products brings in and how etc.

If the company were an airplane essentially upper management were flying it by instrument. It would've been a scandal if the metrics had serious issues.

Some of the metrics less directly tied to business stuff were a bit more 'creative' - as in I could justify why I did them that way, but still not 100% solid.

Stuff like optimizing data pipelines, where data scientist experiments which tended to take 1hr, now only took 10 mins.

I could say that data people were 6x as productive, but it's just as well possible they were just more careless with what they ran, but whatever, a white lie.

However saying that stuff takes 1/6th the time, when in fact it doesn't, is an absolute no go. Neither is not knowing why is there a run that took 500 hours or 5 seconds, both of which should be impossible.

Doing that stuff destroys the confidence in the rest of the data.

Not fixed at this hour

> Sometimes the docs straight up lie, and it takes 5 minutes to figure that out. Should they also be ashamed?

Yes.

How is it fair to compare a Show HN project with official government datasets? People depend on government datasets, multi-billion dollar businesses are built on top of them. A show HN project is typically someone building it in a weekend. They’re not even remotely in the same league.

Sure it is expensive to check every number, but at least some of it can be automated and flagged for human review, no? Switching lat/long numbers. For example

This has become a spam site for AI shovelware projects that are nearly always posted by accounts with no activity here outside of self promotion.

If they publish a lie they should be ashamed, even if their lie is orders of magnitude less impactful.

And if someone publishes a flawless code but have no idea how it works, its not their code, quite clearly, AMD they should be ashamed if they lie it is.

It's just, like, my opinion, but I like it :)

>Sometimes the docs straight up lie, and it takes 5 minutes to figure that out. Should they also be ashamed?

Yes. Lying is bad, even if some people are trying hard to normalise it.

>What about people who don't know how their own code works? Despite it working flawlessly?

I think that is fine, as long as you aren't making untrue claims.