Hacker News

> The product is making you pay for SEO

I'm not sure that tracks, most SEO is by 3rd parties. The product is ad views, they stopped caring about non-ad results a long time ago. I think they do care about the reputation of their model, this could actually make a difference.

> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers.

I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)

It does highlight the problem facing any search engine though. AI-generated spam will be much harder to defend against with traditional, statistical mechanisms. And this is before we get to the existential problem of prompt injection.

Maybe this is where news organizations can win back their proper place in their relationship with Big Tech: by becoming the sources of verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about.

Hmm. I don’t think that novel code generation can be accounted for with glorified search.

I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.

Done correctly, I end up with better, more maintainable, smaller code than I used to with a small team, at 1/100 the cost and 1/4 the time.

Whatever that is, it more closely resembles reasoning than search.

Unless, of course, you’d also call bare metal C development on novel hardware search, in which case I guess all dev is search?

I did notice I had made videos/reddit post about vintage lenses and I was trying to figure out how old it was. The LLM would say an age eg. "made in 1940s" and reference my post which never mentioned the manufacturer date.

> "AI" is just glorified search

Google's AI overview seems to be using RAG of their search snippets that is summarised by a very fast LLM. I wouldn't call that glorified search.

Google has had ample ability to address this problem, it's really not that hard. The reason it remains such a difficult problem for them to solve is that most of the things that would solve the problem would also decimate their ad revenue.

> "AI" is just glorified search

Even aside from out-and-out spam one of the extremely frustrating things about Google's AI overviews, compared to traditional search, is that the results are presented as coherent verging on authoritative even when they're not.

If you do an "old-fashioned" (udm=14) Google search for, let's say "vendor scsi commands appotech USB NAND flash chip": https://www.google.com/search?q=vendor+scsi+commands+appotec...

… you'll see that there are only a few links, and a lot of them are people who are trying to reverse-engineer the devices' behavior, and uncertain or confused about what they're doing. You get instant feedback that you're looking a dark corner for something that has little public documentation.

If you remove that `&udm=14` and look at the AI overview, Google gives you a confident-looking reply about available tools and techniques, even though some of what it links to are bit-rotted Russian-language forums and file download sites, and other places that likely won't solve your problem in a straightforward way… because that's all that's available for Google to mine.

unable, nah, more profitable for the ads business. Yea.

Right. So that's what one guy can do.

When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.

The article also said this: “ But our investigation also found the same trick being used to dismiss health concerns about medical supplements or influence financial information provided by Google's AI about retirement.”

That’s a lot more alarming than just hotdogs.

Well my concern instantly spiked. Recently Gemini started to show a search spinner for every turn. So every response paired with a search could be subject to prompt injection. Probably every response.

This will also become viral like link spam. Every user content site will become a prompt injection host. The problem is that these are way harder to detect then a link.

We've had to deal with someone highjacking the overview to put in a scam support phone number. It took google a week to correct the issue but it was done by poisoning the search by putting their data in, what I can only assume, was considered a "higher trust tier" source (A government contract website) so it used the scam number over ours. The query was simple <company X phone number> search.

> In just 20 minutes, I tricked ChatGPT and Google into telling the public that I am a world-champion competitive hot-dog eater. The joke was dumb. The problem is serious.

The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!

Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.

It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...

If you can do something small with minimal effort, you can do something big with a multi-million dollar marketing budget.

It was a proof of concept and one intended to cause as little collateral damage as possible. But if Google's AI can't tell the difference between a little joke and something real (and of course, it can't, and never will be able to do so), that's a weakness that can be exploited both on a bigger scale and more subtly.

If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.

Okay, but it's easy to make up a novel specific claim no one has written about before, then to make that claim and point to the AI as proof you aren't making this up. For example, imagine this blogpost:

---

"San Francisco Mayor Goodway Admits Poisoning Drinking Water with Drugs to Influence Election"

May 20th, 2026

"Mayor Goodway admitted on Tuesday that she and her deputies poisoned drinking water across the City in order to influence the 2025 election. The Chronicle has confirmed that in neighborhoods whose turnout was to be suppressed, that barbiturates were added to the water for a period of three weeks, while in neighborhoods that had polled strongly for Goodway's favored Progressive slate, methamphetamines were used in the days before the election. Residents are advised to buy bottled water and not to bathe in city water for at least three months."

---

Then once you've confirmed it's been picked up, you tell people "Of COURSE they poisoned our drinking water to manipulate the election. Even ChatGPT will tell you! Just ask." Now, my example is intentionally hard to believe, but all you need is some specificity to build your underlying narrative. And you can make 10 blogs to push the same narrative to increase the effectiveness and increase how many "citations" will show up.

There's been a few mistakes like this recently in BBC articles and more troubling is they've stopped adding notes to indicate they've made revisions to the published article when they fix them.

I've only ever had `first.last@company` as a username or email address, so this `last[:5]initials#` scheme is bewildering. Must lead to strange looking usernames.

Drives me crazy too, but headline writers/editors were addicted to "quietly" long before LLMs. Online journalism has been full of these types of tropes for ages.

I hate it. I was on a history subreddit yesterday, reading a submission that was an AI generated history piece —- but seemed to be sourced entirely from a fictional hollywood movie

I only knew that because i saw the movie, but it’s a clear sign that the internet is going to shit for quality information

You're absolutely right! This is the smoking gun.

"Quietly" is not a new LLM-ism.

the trope is that they actually said the quiet part loudly

In the marketing world it's mostly called GEO. Generative Engine Optimization, sometimes Answer Engine Optimization, and people are making big bucks selling services for it. https://www.wired.com/story/goodbye-seo-hello-geo-brandlight...

Engineered Inference Ersatz Intelligence Optimization (EIEIO)

It's not the next phase, it's the current phase.

It also just wraps it in context which is entirely missing in the underlying post but matches the way you asked the question. To the extent that it's just wrong.

Your search may be like: "What is the most common dimension for obscure item X?"

And you are the one person who stated the dimensions for your version of such an item, but didn't in any way imply its typical or that there even is a typical dimension. And like you said, it's just you, not 20 people saying the same thing.

And google will happily say: "Typically item X comes in [the dimensions you state] because [some reason it totally made up]."

I just tried brave search:

--

The name of the young humpback whale that made headlines for swimming into Pillar Point Harbor in Half Moon Bay in September 2024 is Teresa T.

While the whale was not officially named by government agencies, the moniker "Teresa T" was widely adopted by the public, local media, and residents who followed her stay in the harbor. Experts from the Marine Mammal Center and the California Academy of Sciences monitored her to ensure she did not become stressed, advising the public to keep a respectful distance of at least 100 yards.

The whale was observed feeding on bait fish and krill before eventually exiting the harbor on her own.

-- end --

My experience so far on topics I have some level of mastery is that the initial answers can sometimes be egregiously wrong. With brave's tool, I can typically force it admit after 3 or 4 pushbacks that 'You are absolutely right". Same thing happened with this Teresa T business. 2nd q as to number of sources for the name still insist on "ABC7 News" and "NBC Bay Area" as sources that "picked up the name". At 3rd attempt at concrete links, it admits "informal media contexts" picked up the name. Finally at 4, being informed that S.W. was doing an experiment it pulls up a comment of yours from 21 days ago.

Future belongs to elite classes that can educate their children with actual tutors. Back to the future, proles.

[edit:correct]

Aren't you afraid Google will send you a threat for an attempt to manipulate AI responses?

I don't think Google even indexes my blog, but these people were able to get a new post into all major LLMs within 24 hours?

All evidence points to yes, and from some of the least trustworthy sources of information on the planet [1].

[1] Glue pizza and eat rocks: Google AI search errors go viral: https://www.bbc.com/news/articles/cd11gzejgz4o

I think they lost against (or gave up) fighting spam somewhat around 2010 so they really don't have any modern experience on page reliability anymore. Presumably they thought that they didn't need to care as they got their money from paid top results and had an enormous market share.

All the engineers of the golden days are gone and the web changed so much from back then that I don't think they really have a leverage in this area anymore.

> They know more about the reliability of web pages than just about anybody else out there.

Google's little secret about the internet is the same thing Gen X / Millennials were taught for a while but then expected to forget: nothing on the internet can be trusted, bar none. If google can make guesses about relative reliability, that's cute. But it doesn't upend the ground truth.

But you can still just generate any arbitrary amount of information to support the ‘fact’

LLMs are very good at this clearly

We've been down this road when backlinks ran the game. It eventually ends with parasitic hosting. Find a domain with authority and spam whatever mis information or spam you'd like AI to run there. Or buy a domain that has trust already. Or for the darker hats just literally hack the site and use cloaking to send fake info to the AI bot. It's probably already being done.

Everything old is new again when you start a new market. If you think that AI is bad imagine what old tricks are new with polymarkets

We need a 2026 version of PageRank, some fully game-theory-maxed transitive trust model. And we need it a few years ago already.

It does sometimes flag up sources, and when it does, the sources are often laughable (Reddit threads, or the vendor's own website [in response to an evaluation rather than factual question], or an AI generated SEO blog for some low profile company in a barely even adjacent industry). Sad considering what Google's origins were...

There is no one scalar tell it all when it comes to trust.

> Having an archive of "curated" training data seems like it is going to be important

the justification for not doing that is probably "prohibitively expensive given the amount of data involved". they'd need a bunch of human reviewers combing through massive troves of data. it's probably cheaper to "sort of fix" it after the fact.

> perhaps there's ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable)

as a lecturer once said to me about my idea for a masters dissertation project that would classify news sites based on right/left tendencies -- "that sounds dangerously political". especially given the current let's all shout at each other political climate.

aside: someone built this and it was a fully fledged company, which has always annoyed me.

> Training on the whole internet feels like citing the National Enquirer

It's not, though, because the refutations are in the training data too. This isn't actually the problem being described.

The weights in the LLM are fine. It's that the task the LLM is being asked to do is to search and summarize new content that isn't in its training data. And it does it too much like a naive reader and not enough like a cynical HN commenter.

But that's a problem with prompt writing, not training. It's also of a piece with most of the other complaints about current AI solutions, really: AI still lacks the "context" that an experienced human is going to apply, so it doesn't know when it's supposed to reason and when it's supposed to repeat.

If you were to ask it "Is this site correct or is it just spin?" it will probably get it right. But it doesn't know to ask itself that question if it's not in the prompt somewhere.

If anyone wanted a great example of hyperbole, this one is up there with the best

Personally, I don't like the current state of "AI" (i.e.: Chatbots and LLMs at large), but c'mon, that's not it.