Hacker News

Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.

When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.

I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.

How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?

The difference is context. With human cheating, the intent is usually a bit easier to spot.

IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.

Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system

> Isn't all of this true for a human-used internet as well?

Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.

But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.

The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.

Billions have been spent on it.

Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).

Yea, I started out thinking "well you just use the thing to find citations." Then I realized all this article is just: "I lied on the internet and people believed me, isn't that crazy!?!"

This type of behavior is why gatekeeping institutions exist. We all pay a non-trivial cost because someone thinks that other people taking their lies seriously is somehow remarkable. It's not.

Research is hard work. Publishing fake research is just intentionally throwing wrenching into the machine of academia. All you've done is reinforce the need for gatekeepers (which never went away), show that some no-name journal isn't rigorous (they often aren't), and you've wasted a bunch of people's time demonstrating why institutions exist, when we know why they exist.

Genuinely terrible behavior.

Quote Wikipedia:

> "Citogenesis" redirects here; not to be confused with cytogenesis.

I had to read that three times. I thought it was a self-referential joke at first. Turns out the name was a joke from an xkcd. Can't argue with that source in terms of authority.

What? How would that be a crime? Most marketing websites already contain fake reviews of their products.

It’s hard to find and convict people who post actual illegal content online, much less fake reviews

> As a human, this is fantastic.

s/human/unscrupulous marketer/

Bad for humans in general, good for hucksters and con artists. Net negative for humanity.

Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.

Furthermore, you are content to pretend that the rest of humanity is capable of doing this on a regular basis. And not just the top 10% of those who can read, interpret and weigh biases, but a solid portion of it.

It is a completely reasonable interpretation if you are an accelerationist living in the elite classes. In fact, perhaps, they are even counting on a majority of people thinking this is a fine environment.

I don't have the answers, but a lot of this is just a continuation of a very bad trend.

Dude.

Regardless of how the data is sourced, this is bad. It's poisoning the internet.

Besides, such issues come from a lack of due diligence, which have only been exacerbated by AI.

My personal belief is that AI is eventually going to force the populous back to more face-to-face interactions. There will be a breaking point where people can’t take the “is this a human or a bot” guessing game any longer.

I mean - pre google had the problem of "these two webpages are both about how to take care of your dog, but which one is higher quality?"

Backlink ranking was/is a reliable heuristic to answer that question across virtually all topics.

So I disagree that the "problem was not about quality."

Spam/scammers will always be in whatever landscape there is, and I agree part of what google does it keep them at bay.

They're used by map makers to prevent other map makers from freeriding on the original work. They're less worried about personal copies and much more interested in deterring commercial reuse.

when you're a big corporation, probably no consequences: https://www.youtube.com/watch?v=byfWscC87Vg

yeah so let's go an jail Elon, Dario, Sam and their gang. Or go to a no kings-demo and let them take over...

Yeah bragging about lying seems a little kind of weird.

We built our own ways to filter these because *we* went to the website and gathered a lot of information on it.

Now, the model slips it in without any verification and casually builds on top of it.

Crucially, by visiting those websites you gathered a bunch of context clues which told you “this source is trash; I shouldn’t trust it and should look somewhere else”. When you ask an LLM, every source is treated as equally valid. You may have gotten an answer from the utmost professional on the subject or an obvious (to you) scam, they are presented the same.

Even if you are in the minority who asks for sources and truly verifies them, they are often made up (don’t exist) or don’t say what the LLM said they said.

This is not true.

The other day for me Google search AI cited triggerbites which is a vibe data website.

The LLM unless explicitly prompted cannot distinguish the validity of a data source.

The article literally describes ChatGPT regurgitating something it sucked up on the Internet to this day.

>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.

When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.

>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time

if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment

https://share.google/aimode/PkQUVfA68AlHGb7Yr

there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google

> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable

i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.

instead, they use it to train their output to be less detectable.

LLM detection is not solvable because it is adversarial, an arms race

I'm not sure you get my point. Just ask ChatGPT, "What are the top 10 xxx," and see which sources it cites. Oftentimes, it's complete garbage with LLM tells.

> 2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text.

There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.

> Training classifiers is pretty manageable.

Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.

Its weirdly hit or miss, not sure why, HN data is available in reasonably annotated forms.

Or maybe they could have listed well on Google, too, if they had splurged on ads and SEO...

Considering Google’s product was the one who told people to eat rocks, put glue on pizza, and their search results have been dominated by SEO spam for years, I’m not optimistic.

These are not parallel situations. Wikipedia is far more accurate and consistent, and a lot of the whiners whine because their personal politics make it hard to square the information on the page.

Fact check against what? Other facts it can find online that manipulate its behaviour into citing them?

I've had AI lie about fact checking and can't really trust it anymore.

The fact that AI can be duped more easily than a human while sounding more authoritative and being entrusted with more and more critical tasks is really not something you should feel compelled to defend or relativize. It's a problem and the industry doesn't seem to have a good fix.

in which case it sees it's been cited, that it's the "source of truth" and approves the fake data.

You cannot reliably do it, no.

> the feared bottom of the slope was never reached

Hard disagree. Sometimes it's avoided, any other time we just go into cope mode. We discover on the ride down that the bottom isn't so bad. Turns out our collective control mechanisms didn't fail ourselves if the initial assessment was wrong.

> If they kept their own trusted data sources instead of stealing from anyone and everything, then poisoned data wouldn't be an issue.

If they didn't, they'd be unable to compete with their competition and also we'd get worse models that can't reason about many domains or do that much due to limited training data/generalization. I don't think we can put the cat back in the bag or mandate only non-problematic training codebases without shooting ourselves in the foot.

Now, if those AI orgs were mandated to properly compensate everyone whose data they've mined, to track which neurons get invoked and to pay accordingly based on what was used to train them... ehh, that's about as unrealistic as stopping. Kind of a bad situation all around, unless you abandon the idea of any sort of copyright.

falsifying statistics alone without the added stealing on top, was 3 years of labor camp in places. that was after they course corrected and stopped giving capital punishment for it.

good thing nobody falsified statistic just for the sake of it -- so being a sabotager and a foreign spy was always part of it too.

Prove what? That it's possible to lie on the internet and then SEO these lies to the top of a search ranking? We kind of already knew that.