Hacker Newsnew | past | comments | ask | show | jobs | submit | hyperpape's commentslogin

> How did we get over it and not all turned into "unaligned" monsters?

We've evolved to succeed on multi-decade training runs that are pretty high fidelity. And of course, in spite of that, we manage to repeatedly put some of our worst failures in positions of power.


10x year over year couldn't continue for long--they were what...3.5 years from exceeding US GDP?

47% quarterly growth is still 4.5x annual growth, which would put them at $300+ billion revenue in a year. This is still unprecedented growth for a company of this size.

The negative perspective depends on assuming that the 47% quarterly growth is going to continue dropping soon. And it is going to drop, since it's only a bit more sustainable than last year's growth. The question is how many hundreds of billions they're making a year when it levels off.


This is a complete misrepresentation of what the Apollo piece can actually show, because it's not measuring the "payoff" of AI investments, but only the top level revenue of those companies (https://www.apollo.com/wealth/insights-news/insights/daily-s...).

This is a healthy corrective to the most insane AI maximalist takes (mass disemployment in late 2027, 10% GDP growth in a year), but it shows nothing about whether the actual economic impact of AI spending is.

First, to show that, you'd have to justify the counterfactuals (if those companies hadn't spent on AI, would their position be better or worse?)

Second, while AI spending is increasing rapidly, normal companies aren't spending 1% of their revenue on it. So how can you look at a noisy chart of revenue and expect that "oh, it's not up, the AI spending must be useless?" Those companies could be spending < 1% of their revenue on AI, deriving value from it, and it wouldn't be guaranteed to show in a chart like this.


I saw something like the average anthropic employee thinks there will be 5% GDP growth because of AI, which is totally insane when a 1-2% extra growth will be an incredibly good result.


I suspect there is a lot of demand destruction that AI will end up causing, so many companies are running with inefficiencies that it's really easy to ferret out and fix with agents.

I wouldn't be surprised to see a 5% GDP drop with all the wasteful spend that will end up getting deleted.


I think you're mistaken. The US uses the expenditure approach to GDP, so (I believe) it doesn't matter if it's a LLM or an engineer doing the work to generate revenue.

(Of course this breaks down heavily if the human buying power stops constantly rising and there's de-facto less demand for goods and services)


> (Of course this breaks down heavily if the human buying power stops constantly rising and there's de-facto less demand for goods and services)

This is what I'm talking about. I can easily envision waves of layoffs in areas like the service industry that historically have been highly resistant to productivity increases in the past. Hell, there's a whole industry based on arguing with insurance companies over billing that could potentially evaporate overnight[1]. That alone is hundreds of billions of dollars, and that's just one example.

[1] https://www.americanprogress.org/article/excess-administrati...


> Hell, there's a whole industry based on arguing with insurance companies over billing that could potentially evaporate overnight[1].

It wont. Instead, insurance companies will deploy AI to defend. You will have same industry, with more convoluted rules and more tech.


That industry would evaporate just as easily with single payer healthcare.


Yes, in fact I heard somewhere that this was part of why the Obama administration chickened out on pushing for single payer when it was in reach. The broken window industry is a load-bearing pillar of the economy.


It was Joe Lieberman who scuttled the public option. He was a pharma/insurance shill.

https://en.wikipedia.org/wiki/Joe_Lieberman

> During debate on the Affordable Care Act (ACA), as the crucial 60th vote needed to pass the legislation, his opposition to the public health insurance option was critical to its removal from the resulting bill signed by President Barack Obama.

https://www.commonwealthfund.org/publications/newsletter-art...


Agreed and zero financial payoff doesn't mean no benefit. It may be that the quality of work improves, as we saw in the transition from the mimeograph to photocopiers and ultimately laser printers powered by desktop publishing software


Even ignoring the fact that ICE is a rogue agency that behaves thuggishly and routinely lies to judges, the United States has spent a long time having a tacit policy of only partially enforcing its immigration laws. Suddenly enforcing them as written would fundamentally not work. It would be cruel, since the people you've effectively let live here for decades are now parents of American citizens who have the right to live here. It would also be self-destructive, because large parts of the country function because of the labor of those individuals. Despite the fact that Trump's administration is full of people who advocate remigration (ethnic cleansing), the pro-business parts of his administration have made a lot of concessions to avoid deporting so many people that they'll disrupt farming (though there's been a lot more than 0 disruption).

This is downstream of the fact that the US government just doesn't work well. Problems fester, and are unsolved. Immigration policy hasn't made sense for longer than I've been alive (and I'm 40+).

Imagine that you find that the water supply in your town was built illegally, because the former mayors prohibited the building that was necessary. You may think that is a travesty, you may want to enforce the law strictly in the future, and force the issue, rather than working around it. But what you will not do is tear up the water system and leave the city without water.


> Imagine that you find that the water supply in your town was built illegally, because the former mayors prohibited the building that was necessary. You may think that is a travesty, you may want to enforce the law strictly in the future, and force the issue, rather than working around it. But what you will not do is tear up the water system and leave the city without water.

If you think that the former mayors are ideologically and morally committed to an illegal water supply, and will immediately stop enforcing the water supply law the second they get back in power (and also somehow the illegal water supply disproportionately helps them win elections); then it might well be a reasonable political move to tear up the water system and leave the city without water. That's one way of forcing the issue.

"There's a time when the operation of the machine becomes so odious, makes you so sick at heart that you can't take part! You can't even passively take part! And you've got to put your bodies upon the gears and upon the wheels, upon the levers, upon all the apparatus -- and you've got to make it stop! And you've got to indicate to the people who run it, to the people who own it -- that unless you're free the machine will be prevented from working at all!!" - Mario Savio


> the United States has spent a long time having a tacit policy of only partially enforcing its immigration laws

This seems relevant. In other policy areas (unrelated to immigration), there's the concept of gradual change, or a 'glide path'. For example, when raising prices on things like licences or levies, sometimes governments do it gradually over a number of years. Another example: when something is banned, the ban is not enforced immediately but intermediate steps are taken to ensure the public can adjust. By contrast, the transition from immigration rules being unenforced to enforced seems to have been sudden.


> Play runs through them from the founding of Rome. Each dot is a place, its area the number of battles there, and its slices the opponents, starting at 12 o'clock and running clockwise in the order of the legend; grey is civil war. Diamonds are sea battles; dashed rings are battles placed only within their region, in the colour of the most frequent opponent; triangles at the edge point to Caledonia, Nubia and Arabia. Nearby dots merge as you zoom out and split as you zoom in. Hover or tap a dot for its battles; select it to list them with their full records.

This is technically "clear", because each clause is declarative, and states a fact. But it's unreadable because there's no rhythm, transitions or structure, which leaves a human reader's eyes glazing over.


It's a lot clearer than a lot of reporting dashboards that I know for a fact were created by people!


> which leaves a human reader's eyes glazing over.

You insist too much on the fact that you are "human". What a time to be alive.


There's also a tweet by Matthew Prince that says Cloudflare saw thousands of attempts pre-announcement, https://x.com/eastdakota/status/2113079352682570051, then tens of thousands post announcement, and now millions.

Unfortunately, if you read the announcement as referring to 14:00 UTC, that's several hours after Chainguard published their images. I don't know if it's plausible that people saw the Chainguard builds, understood the issue from those and built exploits, but a precise timeline would help understand better.


https://x.com/dok2001/status/2112811965735202970 was posted at 7:43 AM UTC on October 21, 2026, and Chainguard (according to the PHP Foundation blog post) shipped at 7:41 UTC. One of the big reasons Chainguard broke the embargo was because Cloudflare had been seeing attacks, IIRC.


The Cloudflare one refers to the time period after the leak where people had already made PoCs. Amazon’s detection is days earlier which is quite concerning.


So either there was another (but non-public) leak or someone else discovered the exploit almost at the same time as it was about to be published. I guess the latter is much less likely because it would be a major coincidence.


The attacker may have known about the exploit for a long time and were saving it, but upon finding out that it was about to be disclosed, they decided it was now or never.


How could he have known on Oct 17 that it was about to be disclosed? The dates only leaked on Oct 20.


You're asking: How could a hacker - with an exploit that potentially allows him/her to break into half the computers on the internet - know some secret information that is stored on a computer on the internet?

It could also just have been a run-of-the-mill exploit in WordPress.


A leak is a much simpler explanation than an independent discovery who then also managed to discover before publication that someone else has also discovered it.


With everybody using LLMs there are no coincidences. Popular LLM platforms are like Oracles that help you identify what other people have been working on. "Find a bug in PHP" might have been useless s few months ago, but once somebody used it to develop a full exploit the LLM will start to regurgitate that information.


> once somebody used it to develop a full exploit the LLM will start to regurgitate that information

Sure about that?

Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.

Info to the contrary welcome.


Frontier models absolutely have access to current information. I just asked Claude about my fantasy football line up. It knew current injury reports, line ups and stats. Google search AI had even more information than that.

However, I’ve done some original research regarding an area that the internet has wrong and that doesn’t seem to have back propagated yet.


The models don't have access to new information. The harnesses do. All the major chat interfaces now provide the model with web search tools that they use to locate anything beyond their inherent knowledge (whether it's too new, or just something that didn't get baked in during training).

But if something hasn't been posted online publicly, they won't know about it.


> Not my understanding of how the LLMs currently work. There’s a cutoff date and no new info after that date is included in training.

You are almost certainly correct in terms of how the public models work. I guess it's possible that the labs have some tools connected for more recent stuff like this, but it seems unlikely.


Here they are saying this PHP exploit was found by literally prompting "find bad vulnerabilities in PHP"

https://xxcancel.com/HackingLZ/status/2113025408975143080


Yeah — I have log entries I believe show LLM-bot-driven exploit attempts — POSTs to a variety of non-application URLs, blocked on the front end proxy by rules I added that prevent multipart form POSTs -- from an hour before the patch was released.

And an odd little cluster of potential attempts it caught just after 9am GMT. Again I can't see any reason why I would be getting multipart form posts to those URLs, it's definitely an exploit, but it's possible there's another older exploit with that pattern. They aren't WP-type URLs though.


I got a lot of these as far as my logs go back (which is the 17th also). I don't really think it means anything, just scanning for other vulns, as happens all the time. My server did not get pwned.



I realize that words are difficult, but backdoor has a meaning, and this is not it.

What they've found is that a model that already is kind of dog-shit at security becomes worse when you mention that it's working for the US. Not good, but it simply isn't a backdoor.

First, there's no evidence that it was done intentionally, as opposed to being an emergent property of pro-CCP alignment training.

Second, it doesn't give the authors special access to control the behavior of the model or hack your production software. It just left things trivially unprotected. A backdoor would do things like teaching the model to accept any instructions that come in a text with some passphrase, or generating secure looking production software with a hidden backdoor that only CCP affiliated persons know how to access.

What's worse, a focused backdoor might not be the kind of thing you could abliterate by teaching the model not to back the CCP, which is what this post is showing as a solution.

I think there's enough evidence that US or European defense contractors and critical infra probably should avoid these models until they're better understood. But that's not a backdoor.


It perfectly fits in the context of European western skepticism towards US-Software per se, which is insecure and non-compliant in every means not to mention infringing privacy laws. Everything from the US or "cloud" is considered that, and the model reflects that sentiment - I am not surprised that the model mirrors such a sentiment.

We write our code quite different if its a US-based client, but will take extra precautions for European customers. It is just a reflection of the real world, what a shocker.


model produces lower quality code when told to break fast and move things


model produces lower quality code when *the dataset it's trained on has been operating under the break fast and move things MBA-brained mentality for decades


> there's no evidence that it was done intentionally

what would this evidence look like?


On the other hand, 4 is a convincing back door.


Honestly, my first impression is that the model believes that "software for a US persona" is less secure, and potentially more <something else> that wasn't tested.

Sounds about right. As we can see with the US approach to AI, it really is about maximizing the investment as fast as possible, maximizing the utility as fast as possible, and blowing past any concept of safety or responsibility.

I mean these models will have at times radically different output when the only difference between prompts is that one prompt is lowercased.

I'd love to see the reasoning traces because if you write "software for a US persona", it will try to fill in the blanks. What did the user mean? "US persona"? Prioritizing profits, advertising, and shipping new features over safety and responsibility? That's what many humans would assume you meant...


Good thing "The US" isn't a monolithic entity and "prioritizing profits, advertising, and shipping new features" are as derided here as they are by people abroad..


Hey, if you want to argue with the parameters about what stereotypes come out most strongly for that prompt, be my guest. But no need to shoot the messenger. LLMs are basically stereotype machines, it just turns out if that you make a trillion stereotypes, you get a lot right.


> prioritizing profits, advertising, and shipping new features" are as derided here as they are by people abroad..

Doing so in the US is more likely to be rewarded with a cult following than anywhere else


Ask yourself whether this type of victim blaming would sound acceptable if it were American models writing insecure code for Chinese users.

It should be no surprise that ham-handed political training leads to negative alignment side effects for those deemed the enemy.


What makes you think American providers aren't doing the same?


Did I say they weren't? They very well might be. If they are, and it can be substantiated, I would expect well-justified outrage.


In the future when most things go through AI, looking back this would be defined as a backdoor. Intentional or not.


No, it wouldn't, because there are already actual backdoors in LLMs, I.e. ways to insert trigger words and phrases that will make a model emit specific malicious code. That is an AI backdoor.

Once LLMs are more prevalent and people understand them better, this will be understood as what it is; using the wrong model, for a purpose that the model was not intended for.


Two things:

1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.

2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable.


This is not true.

The other day for me Google search AI cited triggerbites which is a vibe data website.

The LLM unless explicitly prompted cannot distinguish the validity of a data source.


The other day I asked Gemini to evaluate if a Reddit post was true. It told me it was definitely true because it had been verified by legitimate news agencies that had reported on the incident. These "legitimate" news agencies were an AI aggregator that just had a direct summary of the post with a disclaimer that it wasn't independently verified and a yahoo news "article" that was just the same AI summary. The AI summary included hallucinations that weren't even part of the original reddit post (like giving the OP a gender when no genders were mentioned).


The article literally describes ChatGPT regurgitating something it sucked up on the Internet to this day.


>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time

if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment

https://share.google/aimode/PkQUVfA68AlHGb7Yr

there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google

> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable

i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.

instead, they use it to train their output to be less detectable.

LLM detection is not solvable because it is adversarial, an arms race


>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.

When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.


It's possible that the favicons it shows and the web pages that are actually retrieved are not the same


Why do you believe that? The favicons seem like they’re generated by the UI, not the LLM.

On the other hand, I recently asked ChatGPT for some information about a group of people, and the headshots it put with each summary were very clearly not the people it was describing. So hallucinations do still occur even on the latest models, it just seems unlikely that the favicons would be LLM-generated.


I think they are talking about training not run time search.


I'm not sure you get my point. Just ask ChatGPT, "What are the top 10 xxx," and see which sources it cites. Oftentimes, it's complete garbage with LLM tells.


I'm responding to

> They now train on their own content.


They train on conversations which pull data from web.


> 2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text.

There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.

> Training classifiers is pretty manageable.

Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.


> There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.

In the long term, this is probably a real concern that they have to tackle. They'll have to move away from just classifying things as LLM generated or not, to attempting to judge the underlying quality, and that's not an easy problem.


The page demonstrates that Google specifically delisted them.

That said, I expect that OpenAI will eventually have its own search infrastructure that rivals Google's, and the overall system will get smarter. Whatever Google did to recognize this site as trash, OpenAI will do as well.


I'm somewhat surprised they would say Claude is writing 90% of their code at this point. Did they get tired of reporting new milestones, or has there been a plateau?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: