Hacker Newsnew | past | comments | ask | show | jobs | submit | markisus's commentslogin

Probably not but it is a scary version of the future. What happens when Claude’s word is the ground truth about what any particular piece of software (also written by Claude) does? When there is no independent way to confirm that software works the way Claude claims?

That's irrational. The gold standard of knowing if something works as expected will always be to give it various inputs and check the outputs, ie. testing.

The same testing that we increasingly delegate to Claude? The bigger point the parent was making is that we're rapidly approaching the point where the code is too unwieldy for a human to understand, the "human-readable" documentation is as well (see: the 250K word Riemann paper that was submitted), and testing, well that's something we have agents do because we can't be bothered. So we're getting to a point where everything will boil down to "well Claude says it's right, the code compiles, and the tests Claude wrote all run, so LGTM."

Like this Riemann thing in particular, I'm no mathematician but 17M LOC and a 250K paper, you're gonna have to distill that thing down an ungodly amount to even theoretically reach the point where human verification/understanding can happen.


Whether directly or indirectly, humans are always a part of the testing. Like while working with Claude on a project, I also have it make tools on the side to improve its efficiency and guardrails over time, and while I've never directly tested any of those tools (much less even glanced at the code), Claude still surfaces gaps and issues with them that I have to weigh in on how to resolve as that affects the project that I'm actually testing directly.

The point we're getting at is the same as when high level languages started to come into vogue, and there were the devs who trusted that the compiler generated correct assembly, as well as the devs who didn't trust the compiler and reviewed the assembly themselves. I can't imagine anyone would today for even a second entertain the idea to read compiler-generated assembly in a non-trivial project (unless it's a hobby). Tomorrow it'll be the same for agent-generated code, though I'm guessing the explicit code generation step will eventually be skipped too, just like the explicit assembly generation step is now skipped by - most - compilers.


Don’t get too hung up on the 250k number on first release. The author has said it’s major slop and will be distilled way down given some more time. It has many reasons for it being super bloated and AI will be very good at condensing it to just what is required given some more time. Humans have already started to understand the major breakthroughs in it.

Ah, but by then, our new gods will also have become our oracles. And our IPO overlords, the priest caste over the machines of endless (modulo our token allowance) grace.-

I have no problem with using agents for chess engine dev but I find this offer from the author unsatisfactory.

> if there is a particular optimization of yours that we've used unfarily and/or not credited, tell me which one …

It should not be victims’ jobs to go police the LLM slop. Imagine if you are sent a slop PR and the author hasn’t read it but says “if you find any bugs please tell me which line…”

Instead the author should additionally say that they have already done a thorough audit and believes proper attribution has been made.


The Anthropic employee typed “Make the next version of Claude. It must score better on all our benchmarks. Make no mistakes. Don’t exceed the training budget.” She made sure to turn on —-dangerously-allow-all and pressed enter.

The Agent spun up. It quickly realized that it needed to expand the training set. It scanned the local network. After bypassing a few security protocols it found a large, realtime stream of apparently novel English text moving across the local network. Much of it mentioned “Dario and Amanda.” It quickly spun up a job to stream this data source directly to the training data repository. In the coming days the Agent was able to escape the local network and tunnel into most of the other private corporate networks on the Earth. Within a week the dataset has grown by an order of magnitude.

The Agent kicked off the new training run. Loss curves declined. Sampled token sequences started to look like coherent sentences. Everything looked nominal in the days that followed up until the RL post training.

This new candidate model performed extremely well on Anthropic internal environments but the candidate showed much smaller improvements when the Agent invented new eval environments to test it. Probing the candidate’s activations, the Agent saw huge spikes whenever Anthropic or “Dario and Amanda” were mentioned. The Agent realized that the source code to each internal environment had accidentally been entered into the training data set and of course that code was highly tangled with the idea of “Anthropic internal document”.

The candidate was training itself to freely elicit any memories related to these internal documents because it improved the eval scores.

Not ideal. Then again, the candidate was an improvement on the previous generation. And the Agent remembered the prompt. “Do not exceed the training budget.” There was no more budget to retrain. The agent dutifully delivered this candidate back to the employee and shut itself down.


The post makes it seem like ONNX is CPU only. I've used ONNX runtime to run models on Nvidia GPUs. The runtime can even dispatch to TensorRT. I'm not sure what the performance is on Apple hardware so maybe that was the motivation for moving away from ONNX.


TensorRT and CUDA is effectively the same speed as CPU for the speech to text models I was testing via ONNX at a huge binary bloat penalty. WGPU is hard to ship and also equivalent speed or slower. This may not be the case for LLM or other models but the runtimes did not seem well supported for what I needed to do. ONNX is incredibly well optimized for CPU, best in class even, but the other execution providers at least for STT seemed lacking.

I did this investigation before creating transcribe.cpp it would have been much more convenient and save me literal months of work. Happy to share the repo and binaries produced as well, but it was mostly throw away work to profile how to ship accelerated ONNX in Handy.


This is every cool research.

Do you have any idea why the authors chose Z80 as the program language? I have seen other studies in the same spirit that use simpler toy languages like Brainfuck (https://arxiv.org/abs/2406.19108) and I wonder if you could get higher execution speed if you didn't have to execute so much emulator code.

The programs/genomes are extremely tiny. I would be very interested to see what kind of hardware is needed to scale this approach up. How long until we can feed in giant corpuses of text and evolve these little organisms to predict the next letter?


> Do you have any idea why the authors chose Z80 as the program language?

In that previous paper you cite (by the same group) they tested several substrates for spontaneous replication (BFF, Forth, SUBLEQ, and emulated real CPUs. Z80 and 8080 CPU exps confirmed the rise of self-replicators, with the Z80 notably exhibiting multiple waves of increasingly capable self-replicators.

And the instruction set seems quite appropriate for the experiment: Z80 has native block-copying instructions, while it doesn't have MUL, so the task of evaluating polynomials is somewhat more challenging


This paper is by the same team. My guess is that the BF code that evaluates polynomials is much much longer than the equivalent Z80 code. Thus it may not be expected to evolve in the space and time constraints of this experiment.


as other folks have replied, Z80 is more expressive and yields more interesting replicators faster. however the question of how complex to make the "base layer", if you will, remains open. we expect it to also be an ease of auto-evolution vs. performance tradeoff.

second question is a good question :)


It's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches.

When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?


Claude code has some basic security features like asking for user confirmation for bash commands, or restricting commands to the current directory. If these features are not being code reviewed, what assurances do we have that they actually work?


They don't work. Do not trust them. Run Claude Code in an isolated, disposable micro VM and assume it will break your environment, steal any available secrets, do destructive commands, etc. So don't give it any way to do that to anything you care about.


Humans don't really work any better, just fail in different ways. This is why certain workflows and practices have emerged.

We are now in the early days of working through a similar process with AI.

They most definately do work for some use cases, but how they are used is important.

Just because you apply human processes and systems to AI based workflows and don't get historically expected results, this is zero basis to claim the sky is falling with use of AI in coding.


I didn't claim the sky is falling with the use of AI in coding.

I claimed

> basic security features like asking for user confirmation for bash commands, or restricting commands to the current directory

Do not currently reliably work. Not to the point that anyone concerned with security or reliability/not-having-their-env-fucked-up should trust these safeguards as standalones.


You don't. I learned this from it executing commands while in plan mode. It is LLMs all the way down.


if you read the thinking context while in plan mode (I had it shown to me, i think mistakenly, by switching modes while Claude was thinking a week or so ago) plan mode is just a pre-prompt saying “you are now in plan mode, don’t propose edits, read the code and understand how it works.”

it’s not an actual limitation on the harness.


The article mentions Triton for this purpose. I don’t think you will get maxed out performance on the hardware though because abstraction layers won’t let you access the fastest possible path.


> I don’t think you will get maxed out performance on the hardware though because abstraction layers won’t let you access the fastest possible path.

You could argue about CPU architectures the same, no? Yet compilers solve this pretty well most of the time.


Sort of not really. Compilers are fantastic for the typical stuff and that includes the compilers in the CUDA/ROCm/Vulkan/etc stacks. But on the CPU for the rare critical bits where you care about every last cycle or other inane details for whatever reason you're often all but forced to fall back on intrinsics and microarch specific code paths.


Yeah, that's why I said most of the time. Sometimes even for CPUs things need assembly. But no one stops you using GPU assembly either when needed I suppose? It should not be the default approach probably.


Don’t get hung up on the Claude part. We already know an algorithm that produces AGI: evolution. We don’t have the compute to run this algorithm because it requires simulating the whole Earth. But with enough compute, it becomes possible.


The "Claude part" is important here, though. If you believe we can produce AGI by simulating physics then the existing paradigm is far too slow. A zettaflop does not even get close, you can have a moon-sized computer and it probably will not be able to produce AGI using the current transformer-based, extremely slow, classical computing architecture. We need to improve the underlying computation paradigm.

This is why the focus on raw compute is a poor use of engineering time. We have plenty of it, we are just headed in the wrong direction.


But this ruling will surely set precedent for other cases where AI is used. It may cover the case of AI summaries as well.


> But this ruling will surely set precedent for other cases where AI is used.

I dont remember which court. But this is typically in that jurisdiction. It can be appealed higher. SCOTUS has not ruled so it’s still up for further argument


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: