Hacker Newsnew | past | comments | ask | show | jobs | submit | cookiengineer's commentslogin

It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).

Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.

[1] https://github.com/cookiengineer/qwen-reasoning-trainer


And here is me, self hosting and self training Chinese models, thinking that Claude and Fable were useless since February against them...

I don't understand this, to be honest. Claude is such a stupid model compared to DeepSeek v4.1 or GLM 5.3 or even Qwen3.8 flash next, especially in cyber security related tasks.

I get their marketing strategy, but it's just that. And they don't deliver on their promise and haven't done so since November anymore. They had the lead with Opus 4.6 but it got gradually so much worse that pretty much all people working in cyber that I know of all switched to different models or started to selfhost.

They (including me) were happy customers back in January, though.


> Problem is tech is full of PMs and other managers that don’t know technology. Things that could be algorithms, functions, programs, regex filters, etc are now LLMs because they don’t know anything else to solve problems

Is this the right approach though?

Computers should be usable by humans, not the other way around. How CPUs are built, how assembler came to be, how programming languages came to be... ultimately, they were all just compromises along the way to make computers more usable by humans.

With LLMs, however, the threshold barrier for being able to modify and use a computer effectively is now zero, which in my opinion is a good thing.

I can recommend watching "the mother of all demos" talk from 1968 [1] which basically established the design patterns for modern computers. That's how old the concept of desktop environments is. There hasn't been any meaningful invention eversince, apart from touchscreens/smartphones, and now LLMs as a generalized interface design.

[1] https://www.youtube.com/watch?v=yJDv-zdhzMY


Well LLMs are more expensive and take longer to setup. If non tech people want to build and leverage APIs and so forth for something they could have done in 1 hr or less of coding I guess that’s up to them. But I’ve seen them planning thousands of dollars costs and a team of people working on something simple

For others:

> https://github.com/syntax-syndicate/onnxruntime

Check the web archive link for commit hashes, the syntax-syndicate fork seems to be the remaining legit fork.

[1] https://web.archive.org/web/20261113082932/https://github.co...


Okay, this is a pretty cool prompt bypass/injection challenge.

That is not how scale works.

Some (likely most) projects have just non-paid people working on it in their free time. That's the reality of open source.


You should not connect a service maintained by individuals in their free time to the public internet. Put it behind a VPN or local network only and then you don’t have to worry.

Delaying disclosures is a false sense of security when the attackers are running the same scans Google is. They will have exploits faster than 30 days.


> Put it behind a VPN or local network only and then you don’t have to worry.

This is not how software works whatsoever, to the extent that I have to wonder what relevant expertise do you have in the area. Like this is for all intents and purposes just nonsensical.


Welcome to the world of cybersecurity. /s

Jokes aside, this is how Blueteams or more specifically GRC people usually think like, because they're not owning the systems that they have to defend. Meaning that corporate politics comes in the way with "who is to blame" instead of "who can fix it" for unmaintained systems, and nobody in the company accepts responsibilities to actually patch/migrate their existing software debt towards a modernized infrastructure.

This can be as bad as being stuck in the pre-Citrix age because there's no top-down approved official budget to migrate towards container images.

So from their point of view (being strictly operational without ownership in those organizations) the best they are "allowed to do" is isolate those systems in segregated networks.

But we all know how that strategy turned out, this year, when LLMs finally got finally good enough to implement lateral movement exploits on their own.


This is the kind of thinking that in the 90s would have kept the internet exclusive to the domains of university and government. Every new technology has risks and people adapt. In the early part of the 20th century we might as well have kept cars away from private citizens and relegated transportation to buses, trains and professional drivers. Need private transportation? Use a horse. What with the number of people killed and maimed every year it's just not worth it. Take away our keys and mandate public transportation please.

People will adapt by updating stuff faster than 30 days, and by everything that doesn’t do that getting hacked.

No. People will stop caring because they do not wish to have the extra stress of this. Which kills open source and therefore a lot of collaboration and innovation. Not to mention killing the open internet.

That's like blaming the city for killing your golf course with [necessary] water rationing mandates.

I see only two possibilities. If it's a proprietary product that should either be rapidly fixed by the vendor or abandoned by the customer. Alternatively, if it's a FOSS project then if the maintainers are unresponsive the end user can simply request his LLM of choice to patch it for him. I honestly see no issue here. It isn't how things used to work even as recently as a year ago but it seems to be the only sensible way they can work going forward.

Well if an attacker starts attacking things in the wild, you better find a way to address it in faster than 30 days.

That would be the problem of the people running the software, not writing it.

Considering the direction software and their exploits are going with AI, if 30 days is insufficient in 2028 then that's useful signal to the people running that they should move elsewhere.


Still is their problem either way. Who else's would it be?

Only until cybersecurity liability legislation finally starts to kick in, then those writing it will have a solid reason to assign the necessary resources to make it happen.

>Only until cybersecurity liability legislation finally starts to kick in

In America software is protected by the First Amendment, so most software (and all open source for sure) cannot be subjected to "cybersecurity liability" of the sort you seem to be envisioning. Congress could have an angle when it comes to stuff that is sold, including in particular hardware and subscription services though there will be both plenty of lobbying and plenty of potential landmines involved in any such effort.

But with anything free there is no commercial angle involved, no contract. It's just speech that's put out there, identical to the posts we're writing here on HN right now. Software bugs aren't true threats, incitement, defamation, or obscene.

Other countries of course could take a different tack, but they'd then face "OSS becomes fully American" and I'm deeply skeptical that'd work politically.


There is this thing called soverignity where goveremnts can legally track down the citzens who fails to comply, and whose actions can endager the country in whatever means the forces in power decide it may be.

Also when US gets a saner goverment back, I am quite sure CISA will have yet another opinion on the matter, especially if national security is considered exposed in some media case.


>There is this thing called soverignity where goveremnts can legally track down the citzens who fails to comply, and whose actions can endager the country in whatever means the forces in power decide it may be.

Perhaps in your authoritarian fever-dreams or whatever shithole country you current reside in, but no in the US.

>Also when US gets a saner goverment back, I am quite sure CISA will have yet another opinion on the matter, especially if national security is considered exposed in some media case.

And I'm quite sure you're full of shit.


Any sensible legislation will unconditionally exempt the authors of FOSS software. If that is ever not the case the logical consequence is all FOSS contributors moving to pseudnonymous identities and collaborating exclusively via tor.

For example see the EU cyber resilience act (regulation 2024/2847). It sets out nuanced rules regarding FOSS.

If you simply run or contribute to a FOSS project then you don't have any obligations, those kick in when you distribute software for profit and your obligations scale with the size of your business. Companies that fail to promptly address security vulnerabilities will face strict liability for damages.


Or we go back to shareware and 30 days demos, most likely.

Then Google can just send a patch. What better marketing than that if you want to sell people on your periodic table AI.

great! so companies and individuals will get to learn this reality of open source.

a reality that has always been blatantly obvious, unless one only listened to its all too many zealots.


Oh the irony here.

AOSP still has a 6 months holding back patches delay, the reasoning was "improved security".

So, what is improving security now? Fast patches or slow patches? :)


You misunderstand - denying patches to downstream vendors improves financial security for Google.

But they actually did this to comply with the EU Cyber Resilience Act. When a patch is released, a legal clock starts ticking - anyone who doesn't apply it promptly is liable for any resulting damages.


I think you haven't realized yet that LLMs were trained on human output.

> And there has been strictly zero progress on this and IMO this isn't something LLMs are structurally capable of doing.

Totally agree with this.

I see the future of what an IDE is more like an "architecture designing interface" which designs tests, architectural decisions, type systems, and contracts between components. Tasks then should be delegated in their implementations to agents so they are just responsible for the "wiring parts together" tasks.

Currently I imagine it as something similar to how workflows are usually designed around strong types and schema validations in safety-critical areas like ASPICE, Ada Spark, etc.

That's mostly why I don't use existing agentic environments and started earlier this year to experiment with my own one, where I try to enforce the design and architecture related tasks through capabilities mapped on tools for each of the task-scoped roles involved.


Wanted to add that LLMs have become incredibly good at reverse engineering such devices, even when they don't have postmarketOS (now "Nura") or upstream kernel support yet. Been able to merge kernel changes and backport driver support to upstream kernels pretty easily by just giving them access to UART. They'll just go ahead and figure out all the partitioning and firmware quirks by themselves, and e.g. OpenWRT instructions seem to be well documented in the wiki so they'll understand the process on how to get there almost completely by themselves.

For my malware home lab, I decided to just reuse old phones and routers that nobody wanted anymore (quite literally got them from the trash bins). Now they're running an ARMv8 kernel as the host kernel and the rest is emulated or containerized, depending on the use case.

Works like a charm for most tasks I throw at it, and many old cheap Android phones have at least quad or octa cores in them, making them pretty awesome for podman/docker usage, too.

The only thing you have to look out for is to remove the battery, and that you use a separate power circuit with a step down converter to power it. But other than that, old smartphones have become amazing for selfhosting stuff now.


can you buy such circuit on ebay?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: