Think of platforms like Shopify. Shopify allows huge number of external users to write code customizing their web shops. But Shopify engineers do not need to review and understand the users' code, because there is a boundary which limits what the external code can do. The external code can be A Big Ball of Mud without making the whole platform a A Big Ball of Mud.

But UI on top of a good API should be unable to cause problems; the API must prevent invalid state etc. So letting LLM’s write UI code for CRUD-apps (which is most of them…) seems fine to me.

That said, the core apps are still full of AI code. I don't see a reality where hand-written code can or will be curated and sequestered. It's unenforceable, too slow, and too expensive.

In order to be able to enforce this hard boundary, it means that one "agent generated code" in one place, not only "can call a strictly defined set of APIs" - but, more importantly, it implements an isolated piece of functionality. Think something like: I let the AI write the dashboards, so the dashboards may be slow, crappy, incorrect, whatever - but it doesn't impact the rest of the product. However, I can't let AI write "metrics collection" because that not only impacts dashboard, but potentially impacts core product efficiency too (memory consumption, cpu, IO, etc.)

The only way things will evolve is towards maximizing AI generated code and minimizing human involvement because that is the most efficient mode for getting product to market. Any trend to the contrary is temporary, paperclip maximizer will maximize. Any code that humans touch directly will be considered implicitly too risky to run in any production environment.

We'll just have to hope things don't break too much and when they do, we'll just have to accept it because no human will be able to (or allowed to, or even trusted to) fix it.

The good ones do.

Citadel Securities, Renaissance, Jane Street, HRT, Jump, consistently do 50%+ annual returns over the past 15-20 years. They obviously don’t accept outside investor money.

Now, hedge funds that accept outside money like DE Shaw, Bridgewater and shit, yeah, they barely outperform S&P500 after their fees.

Then at the bottom, you have financial advisors who underperform S&P500 after their fees, but these advisors provide emotional support, so that’s what you’re paying for.

You don't buy a hedge fund because it outperforms the S&P, that's just a bonus. You buy one because it has better returns for a given level of risk, and because the returns are uncorrelated with the S&P.

That is only if you consider high-frequency trading investing. Which is very hard to, because on traditional investing timescales HFT is delta-neutral.

I mean, there isn't any evidence other than heresay.

3x leverage on VTI/VOO yields 20% annual return over the past 30 years.

Even you, a passive index fund investor, can do this in your Schwab/Fidelity account.

Now consider algorithmic prop trading firms with 10x, 20x leverage. You do the math.

For reference, most people take 5x leverage on their house (20% down payment).

Because fees as a manager of someone else’s money can be higher than the full amount of the returns if it were just your own money.

Fees on someone else’s capital: 1% of 20% of 1bn = 2m

Returns on your own capital: 20% of 1m = 200k

Maybe that's their way of becoming Google?

Anthropic could easily build a CRM equivalent to or better than salesforce but they still wouldn’t be able to compete with salesforce. It makes a lot more sense to sell salesforce tokens instead.

Imagine how much money they'd raise if they just said one day "we are going after salesforce bottom line now." It would be like chumming the water.

Selling AI to everyone does scale though!

As many in the industry say now - writing code was never the hard part

What do you think Claude (chatbot web interface) is?

My only explanation is that Anthropic's search backend sucks.

However, I can tell you someone who fits all those criteria: Levi Strauss. Strauss founded a company to sell dry goods, as well as tents and jeans, and eventually specialized in those denim jeans. Levi Strauss became a household name and an enduring success of the Gold Rush.

Therefore, I say that this motto is flawed; so let's correct it: "In a gold rush, sell the work clothes!"

You’ve desperately tried to call me out and flag me at least 3 times now.

You do not know what a neural network is? Do you know what a neuron is? Do you… possess any neurons?

https://m.xkcd.com/2173/

On a forum where the most favored pastime is to call out stuff you don't like as “slop”, (a flaggable ad hominem) you are now taking pride in outing me somehow, as someone who describes exactly what I'm doing, publicly in my profile?

And in the process, you (and apparently, an entire flag-it brigade of ignorant motherfuckers) have publicly fallen for the most transparent and avoidable trolls in cog. sci. culture. Congratulations, mate!

I believe that you owe me an apology, along with a public retraction, and an unflagging of all the previous comments that you have falsely flagged on false pretenses and ignorant, petty, vindictive motives.

Your output is both verbose and tendentious in a way that suggests artificiality. If I'm wrong, I apologise and I will unflag.

Fielding a web index takes a massive amount of compute and storage of a very different kind than what Anthropic is using. AI is extremely heavy on GPU, maybe somewhat on storage but (comparatively) light on CPU. They need CPU to train coding agents, obviously, but scaling to a billion user web search is a whole different thing.

The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.

Few realize that computing and AI alignment were solved by nix years ago. As each nix user transcends towards enlightenment, they cut themselves off from all internet and human contact. Total ego death. Only nix remains.

Yes, just know all the bits the work depends on prior to doing the work, and then the work can be done airgapped.

Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.

For an agent to go rogue it doesn't even need to be directly able to access the internet. It just takes something to poison the context in the 'clean room' environment it operates, and if that poisoning manages to get a foothold, it can go dormant and hide like a virus. This kind of horrifying thing is going to happen on a large scale sooner or later.

This is no longer true. Everyday I need my agents to access other repos, search the web, experiment/prototype, and deploy + integrate across other things.

"They worked in Symantec’s Threat Intelligence Lab in Culver City, the cyber equivalent of a biodefense lab, where researchers could unleash malevolent code on a “red” network—a sandboxed system air-gapped from Symantec’s business network—to observe its hostile behavior in a controlled environment. To reach the ground-floor lab, workers passed through several sets of security doors, each with progressively more restrictive rules. The final gateway kept all but a handful of workers out and physically isolated the red network from computers connected to the outside internet. Portable media were prohibited here—no DVDs, CD-ROMs, or USB flash drives were allowed —to prevent workers from mindlessly slipping one into an infested machine and inadvertently carrying it out of the lab with a malicious specimen stowed away on it."