Hacker News

Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.

Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.

e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]

[0]: https://github.com/EratoLab/erato

[1]: https://github.com/EratoLab/web-search-mcp

[2]: https://github.com/EratoLab/web-access-mcp

[3]: https://www.linkup.so/

I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.

searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.

Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data

Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last

I have built my own search engine, which is really great btw. I self-host it.

I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.

These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.

I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.

An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.

For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].

Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.

[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...

Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.

I've stopped using the browser and have my Pi/LocalLLM(Qwen3.8) do my searching for me. Made a nice interface that will display top results.

No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.

I ran into a thing running a local dense LLM on DSH (without DeepSeek API key) that doesn't use many MoE experts with vast knowledge. It ends up doing a lot of searching. In the end my local AI is as free as electricity but I pay for the searches.

The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.

How do you deal with fcking cloudflare blocking access to the websites ?

Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.

And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.

I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.

So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.

The only reason this exists today is because normal search providers haven’t been completely disintermediated.

As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this

Don't you put yourself at the mercy of bot banning with this approach? It's probably good for poc use cases, but I wouldn't count on it in production.

Notable that humans can't cite the source for a huge chunk of their knowledge. Eg. When/where and from who did you learn the meaning of the word "garden".

Your understanding of the world "garden" probably came from 5+ encounters with the word in different contexts, and you cannot separate them - instead your understanding is some kind of average of the knowledge from each encounter.

Interestingly, adjacent and overlapping-meaning words like park, meadow, yard, grassland etc often have disputed boundaries - ie. One person might call something a garden, whilst someone else calls it a meadow. Neither can explain quite why they have chosen their specific word for the object in question, but it might stem from the source of where the meaning of each word was initially learned.

Yes it’s maddening.

It also highlight that Google really isn’t about search, all they care about is advertising and making users profiles.

Have never understood this.

A paid search API seems like it would be easy money for Google. And maybe some lock-in for all sorts of start-ups and enterprises.

There are so many apps that would benefit from search.

I get that it means they can't serve ads.

But you've got to be willing to disrupt yourself or you'll lose eventually anyway.

Better to figure out how to make money from ads and an API, than to cower in fear.

That same fear may be why they failed to be first movers in LLMs.

Have to be willing to disrupt yourself.

At the same time, I've never built a billion dollar company. So I'm absolutely armchair quarterbacking here.

Google provides Vertex for search, but it’s quite slow and so probably only useful for async workloads.

Thought they'd be allover it. Dropping in cheeky ad results.

Ceramic.ai is $0.25/1000 searches. I've been using it as an MCP in OpenCode and it seems adequate. I don't see many people talk about it, and I have not benchmarked it against Exa/Tavilly.

Kagi MCP works really well imo also within DeepSeek Harness. Just looking at cost neglects the possible difference in quality. How good is that DeepSeek web search/fetch?

$4 with Exa, $1.50 with SERP API

Mojeek is £ 2 per thousand. They have their own index (not a SERP), but subjectively pretty close to Google (I prefer it to Bing/DDG).

$1 for LinkUp fetch, $5 for LinkUp search.

is the cost of search and scrape provider is impact the quality too? for the search is yes, because there it shows the quality and depth of the search. but for scrape, wht the cons?

>it’s a TOS violation

Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn

https://news.ycombinator.com/user?id=BrunoBernardino

Kagi has an official MCP server [1], where you set the API key. MCP use is always by an LLM, so this seems to be at odds with their TOS.

[1] https://github.com/kagisearch/kagimcp

Have you seen this https://sciencemadereadable.com/ it has free rate limited api end point too.

> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.

Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.

One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.

If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.

There is no single answer to this, it entirely depends on the exact service and the exact page

Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections

Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss

usually a combination of something like camoufox and a residential ip address

you can bypass those easily except Google Captcha v3, you need to have your own proxy swarm to make it at scale.

Source for those scraping providers?

It's absolutely not SOTA for vibe coding a whole app, but for guided work it is great.

It's really not neck and neck with Opus 5.5.

Or at a really good time, depending on who you ask.

Not everyone wants Anthropic to succeed.

i thought open models weren't SOTA. therefore not neck and neck... in your experience, do you think it depends on your type of work?

Models are comming every 4-6 months. Anthropic, OpenAI, all of them know this.

> bad time

Regardless of the performance metrics... why bad time?

Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.

It literally is not though... even DeepSeek doesn't claim that. It's just super cost-effective

The anthropic API was serving the deepseek search API?

Wouldn’t searching an index be faster? Unless you’re really digging through a site for some super deeply nested link?

I needed to find a word on a wayback archived website. Could not. So I scraped the waybackmachine pages to .md… and found it that way… sorry Archive.org

I like the idea, but its a surefire way of bloating the context. Although I guess you'd want a subagent to do the search.

Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.

The existing search machines that offer free APIs for local harnesses are super bad with low quality.

This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.

> I thought it was your locally-running harness doing it.

That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.

Your comment reminds me of this classic post: https://www.reddit.com/r/BestofRedditorUpdates/s/uiBsQd8r7f

In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.

That is a really good idea, in fact I’m doing something similar. I am charging anyone reading this comment 100 bucks per letter read, please tell me who I should send the bill to.

How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?

I understand if you pay is a different story.

I mean ... "how does this work?"

It is a service for AI agents, so that they can search the web and fetch pages. It is not a product for humans.

what do you mean by "Deepseeks browser platform"? Google isn't turning up much, and searching DeepSeek's docs for "browser" returns no results.

Not sure, is it worse than two years ago?

Think there will be two tiers as usual, one for domestic/mainland and one for international users.

It's for agentic use

The deepseek philosophy has always been to make a small profit or not do it at all.

I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.

Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.

Is Google paying? Or why do you ask.

I really wouldn't want search engines to pay me to crawl my websites, why would you? The point of publishing stuff online is for people to access it.

Do you?

Of course no, and that can disincentivize creating more content. You can copy anything and rehash it for next to nothing