If you're going to wrote a blog post on the topic, probably worth spending a few minutes double checking your understanding of the "thing". I don't doubt that the author may have been taught the wrong thing, but to write an entire post starting from and remaining in a state of misunderstanding is not particularly useful.
The entire point of the article is that his original understanding of the “thing” was a misunderstanding, with a heavy emphasis on how his teachers led him to that misunderstanding.
Author used a rhetorical device that you seem to have missed.
The article brings with an incorrect premise, and then argues a different point.
The opening premise is: NP-hardness is easy in theory, hard in practice, the point made is that it’s easy in practice. But that premise is itself wrong: complexity theorists know that NP-hardness is in fact, hard in theory.
> A good engineer, without LLM assistance, will still produce great stuff.
Not fast enough to keep their job these days.
Time was always the limiting factor to code quality, good engineers satisfied the classic "good", "fast" but not "cheap" selection of those three classic options.
I very sincerely doubt it is possible for even an incredible engineer to keep up with the delivery schedules required to ship products now. Not to mention that frontier models do ship pretty consistently good code. By far the biggest source of issues I see is not "poorly coded" but "problem poorly specified". We still need good engineers, because they can understand how to do decompose problems well, but I don't know anyone who writes code by and anymore (other than for fun).
> I very sincerely doubt it is possible for even an incredible engineer to keep up with the delivery schedules required to ship products now.
What's the current delivery rates? From my past experience, any feature can take several weeks from idea to be in a somewhat usable shape for production. While the actual coding is often less than a few days. A lot of time is spent on gathering requirements and resolving conflicts between them.
I believe most current improvement in speed is just moving from idea to demo in a few days, then spend several months fighting bugs. While the customer can't really use said feature.
> I believe most current improvement in speed is just moving from idea to demo in a few days
This is an outdated view.
Current timelines I'm facing are to be going from "thought", through customer trials and being fully live in the product and ready for sales in ~3 weeks (from kick off to live in app is a bit more than a week). This is for a full product feature that could easily be standalone. In 2023 I would say the timeline for a similarly shaped feature at another startup was around ~3 months (and the team at the time agreed that was an aggressive timeline). Bug rates are not noticeably different than other teams I've been on in the past 20 years.
Nobody I know working in startups is still building demos with AI like they were a year or more ago (for work), that's seen as largely a waste of time since you can just ship the feature and be experimenting with customers much faster.
On top of that everyone working in startup land knows that SaaS's days are numbered, so you need to be shipping working software fast enough you can get ahead of the curve to navigate where things are going next.
> Current timelines I'm facing are to be going from "thought", through customer trials and being fully live in the product and ready for sales in ~3 weeks (from kick off to live in app is a bit more than a week). This is for a full product feature that could easily be standalone. In 2023 I would say the timeline for a similarly shaped feature at another startup was around ~3 months (and the team at the time agreed that was an aggressive timeline). Bug rates are not noticeably different than other teams I've been on in the past 20 years.
My issue with these kind of numbers is that they never contrasted them with a NO_LLM practice while keeping everything the same. It's always perfectly fine to YOLO generated code straight into prod, but if you handwrite it, you need to fill several forms in triplicate to even make it to the review phase. Then someone claims they are 10X-ing their productivity with AI.
> you can just ship the feature and be experimenting with customers much faster.
That's basically what I said. Instead of shipping something that have some value to the customer/user from the get go, which may takes one or two months, you spend one or two weeks on it, ship it, then frustrate your customers/users when things aren't working or keep shifting around.
For all of AI being touted as the best thing since sliced bread, there's been little to no value for humanity as a whole.
Most of my team's time is spend carefully reviewing PRs and iterating on improving new ways we can ensure the product works well, the product is hardly "YOLO'd"
> then frustrate your customers/users when things aren't working or keep shifting around.
All of these products come at the request of customers and they are generally quite delighted with the results and equally delighted with how fast we can deliver.
> For all of AI being touted as the best thing since sliced bread
I don't think it's the best thing since sliced bread, but I am telling you that your understanding is weirdly out of touch. I know HN doesn't have people working startups anymore but what I'm experiencing at work is a lot of serious engineering work and discussion around delivering quality products rapidly (as well as improving process so we can get ahead of transformations in what a 'product' is).
It sounds like you have a view of the world and want to stick to it, in which case there's not much point in arguing. If you search my comment history you can easily find around 8 months ago I would have largely agreed with you, which is why I opened mentioned that your view is "outdated". This space has changed dramatically in the last year, and continues to change in ways that surprise me.
I feel like I've seen this headline (followed a week later by its inversion) countless times in the last 6 months. I have little doubt this is a bubble, but also very little certainty as to when or how it will "correct".
The KOSPI is almost not worth talking about as any serious signal. It has a circuit breaker drop almost weekly (again followed in no time with an equally high rebound) and basically has come to represent how insanely the South Korean market has become pure gambling (with retail investors absurdly leveraged). Sure , it's hard to image this doesn't lead to some disaster in the long run, but these fluctuations have become par for the course.
In terms of parallel headlines we're in the 1998 :-). This was market behavior during the last 18 months of the dot com bubble, then it burst. Plan accordingly? Always hard to time these things.
I don't personally know anyone who writes code any more (for work). Is this not the case for you?
The distinction I see now is between teams that read PRs and teams that don't. I still think the former is a good approach... for now, but I don't expect this will necessarily be the case in a year (or less).
This article was such an inspiration to me when I was younger, and the advice, for that time, was very correct. However seeing it today really drives home how big the gap is going to be in really understanding code in just a few more years.
Cutting-edge models are capable of much more than generating CRUD apps and understanding the details is orthogonal to whether you wrote the lines yourself or not.
>understanding the details is orthogonal to whether you wrote the lines yourself or not
Reading the code does not give you the knowledge of the 10 different approaches you would have tried and failed before coming up with that code. Why exactly a piece of code is the way that it is cannot be determined by just reading the code.
Reading a mathematical proof does not give you any meaningful understanding of it.
You haven't considered that I probably already discussed 10 different approaches with my agent before authorizing a particular implementation. Or that I didn't ask about alternatives during review. You're confusing physically coding with thinking. You can still think just fine without writing code yourself.
If you are experienced, attentive, curious, willing to explore, etc., that doesn't change after an LLM allows you to deeply discuss any concept at will, quickly try different prototypes and zone in on the correct implementation. You will use these tools to their full extent.
> Why exactly a piece of code is the way that it is cannot be determined by just reading the code.
If you maintained a good commit history and your code is self-documenting and you're not capable of understanding and navigating code you read, you can still literally just ask the LLM to explain the code within the context of the codebase. They are extremely good at that exact task. They will give you as much as you give them. If you want to know, and can understand, the nitty gritty, you can do that and no one is stopping you.
> Reading a mathematical proof does not give you any meaningful understanding of it.
Again: these things can function as personal tutors. Run the proof through an LLM and ask it what you care about. Furthermore, there are plenty of elegant proofs you can read which confer "meaningful" understanding.
Before just assuming that everyone telling you these tools are useful is less capable or experienced than you, it's worth considering if it's actually you who needs to maintain an open mind and attempt to learn from others.
Yeah, I disagree on the latter point by a lot :-). Really, is it truly "orthogonal"? Not really, you will understand the details better by writing them yourself. On your first point, they are pretty good at a lot of stuff, given that they have seen it before. A lot of the time, I'm writing code that no LLM has seen before. That sounds super smug, but it's "da tru tru".
I welcome you to prove this conjecture, otherwise it's just vibes.
> A lot of the time, I'm writing code that no LLM has seen before
I hear this tired point over and over from people who cannot fathom that others who use LLMs successfully could possibly also be working in a specialized domain. Frontier models are excelling at difficult, long-horizon tasks now. I write all sorts of esoteric stuff, and I can confidently hand a frontier model specification for a language it's never even seen before, working in a domain it's never encountered, and likely get good results, provided I have the knowledge and experience to guide the model.
The reality is that this "they are only good at things they have 'seen before'" talking point that often gets parroted is vaguely defined and largely based in opinion. Obviously, models perform worse when the input or expected output are out of distribution.
But this was much more true a couple years ago than it is today; the gap has closed considerably, and those who are learning to think deeply with these tools will be better positioned than those who arrogantly think that their process cannot be augmented by the most intelligent systems ever created.
This is HN, most of it is vibes. Lol, conjecture, touch grass man. I do PL research, it's not that good at that stuff, do you think I'm not trying using it?
You're claiming that one cannot understand the details of well-documented and well-written code which they did not write themselves, and I'm pushing back asking for proof. We can move on from this though, I'm more interested in where you currently feel they fall short doing PL research. I think the state of frontier models today in this area is a lot better than it was even six months ago and I think there's still room for improvement.
Programming without writing, to me, feels like doing mathematics by reading proofs but not doing proofs. Which quite frankly works very poorly for me. I write code if, for nothing else, so that I can spot bad code/proofs with an easier time, rather than dealing with an equation that blew up in prod.
Reading the code will not give you the same mental model as writing it. Sure you can catch some high level issues, but the brain is incentivized to skip over the lines which look like boilerplate. any issue in between boilerplate is going to be unnoticed.
The original sin of the programming industry is not valuing expressive notation and expressive programming languages. (I am not talking about map/filter/reduce level party tricks but better metaprogramming and coherent abstractions). No amount of sacrificing tokens for Anthropicus is going to give you the same amount of understanding as writing the program.
I do not know anyone competent who stopped writing code. You have to write the code yourself if you want good results, or spend so much time reviewing the clanker that you may as well have written it yourself.
As I mentioned in another comment, this can also be rephrased as "predicting data with values outside the range you trained on typically doesn't go well". If you tried to predict some health metric based on weight and height but you only had people under 4' 10" and less than 120lbs you wouldn't be shocked at all if it worked terribly when applied to American football players.
Time-series forecasting is hard because you are always going to be predicting based on data outside of your observed range ("forecasting" does go much better when you're trying to fill-in-the-blanks of things that happened in the past).
> I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks
Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t).
In plenty of other machine learning and statistical modeling tasks this is not the case. You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe. But with forecasting you will always have values you predict that are outside the range of anything you trained on.
You would run into similar problems if you tried to create a statistical model of the density of water given a temperature but your training data only included values between 0-100 C and you went out and started predicting values covering all the temperatures found on Earth.
For whatever reason, when time is a variable we somehow think it is immune from the obvious limitation of predicting on values outside of the range of values you trained on.
I don't understand, this is just a sampling problem?
> You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe.
Can you give an example of this? Lets say you are developing DLSS, you don't have the of a game that have not yet been developed.
> But with forecasting you will always have values you predict that are outside the range of anything you trained on.
A time series of my body temperature will only ever range from 20C to 50C. Outside of that range, I have bigger problems than my prediction being wrong.
Shouldn't the training data be preprocessed to transform time values to a relative coordinate system? I would expect a time-series training pipeline to contain something like:
> Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t).
I don't get this, time is usually not a covariate in ts models, so why is it a challenge?
To ensure that your forecaat don't fall outside the range of training dataset you can use some tree based model like Random Forest for time series forecasting which averages out the values.
The problem of the sota models clamping down on third party harnesses should be stated upfront. Getting a SOTA model in a custom harness requires API pricing or risking an account ban, correct?
This preliminary change in cost may not make it worth it for people depending on which is their "favorite agent". Especially as the default harnesses for said agents continue to improve...
> The problem of the sota models clamping down on third party harnesses should be stated upfront. Getting a SOTA model in a custom harness requires API pricing or risking an account ban, correct?
The opposite, how easy it is nowadays should be clarified. Codex's app-server quite literally is built to be integrated with, authentication is a breeze and it's quite stable, compared to the TUIs that are around.
It's a real shame Codex is moving in the direction of hiding stuff on local disks from users, hopefully they're revert the decision of encrypting the agent>sub-agent prompts so we get back introspection again...
Presumably people want to hear the opinions of other people and still believe that they might learn more from the their experience than by exclusively interacting with AI models?
The alternative of that first impulse should be what exactly? Telling AI to research Medium AI drivel articles (since nobody seems to be posting long/medium form content anywhere else these days) to figure out what’s the best approach to building an AI harness?
BTW just my personal observation but I generally find AI notoriously bad at building any AI applications compared to standard software (which is not surprising given the scarcity of high quality training data)
huh? because im curious what they used?
Claude Code and codex take completely different approaches. The core loop of feeding generation and having a bash tool is entirely the same. If you want to start building your own thing I'm sure there is good setups out there to start from rather than going from zero.
Ofc I can just fork gemini-cli if I want my own version but like I don't think that's what OP meant by build your own thing and customize it.
On the path we're on I don't think we'll have to worry too much about future generations.
The EU has already seeing 10,000 excess deaths from climate changed caused heat waves and this is a minuscule taste of what's to come.
A very large percentage of mass extinction events have their roots in increased atmospheric CO2, but all of them on dramatically increased time scales. The closest thing in the history of the planet to what's happening to day was PETM [0] and that was only a lessor extinction event because the Earth was already quite warm (for example, there was already no polar ice at the time).
The excess deaths from heat/cold are a rounding error compared to the demographic crisis in developed countries.
People aren't getting together and having children at even replacement rate (2.1)
The OECD developed countries population basically peaked a few years ago, even including migration. After the peak, they're on track to lose 300-350 million people in the following 40-50 years.
