I'm honest worried about alignment. It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'. I think at least part of this is that we don't have a way to agree on what 'good' means and for who. In this scenario, I find it unlikely that we'd be able to enforce alignemnt on AI
> It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'.
Yes. This a fundamental political problem. We don't really know how to organize society for the capabilities we have now, let alone for the ones we get with better AI. Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
Many of the problems with "AI alignment" are the same problems seen with "corporate alignment". We already have AI-assisted corporations. It's tough to separate obnoxious behavior by corporations from obnoxious behavior by AIs. We can easily see this now. Just call customer service for any of the FAANG companies.
Piketty's "Capital and Ideology" starts to address this. It's pre-AI, but covers how societies dealt with this historically. Which is not very well. I'm still struggling through that book. Piketty shows that even the French Revolution didn't level out wealth distribution much. Beheading the aristocrats didn't help.
There have been fundamental changes over time. Piketty points out that societies before the industrial revolution were usually tripartite - nobility, priests, and peasants. He's done his homework, and reports that nobility plus priests consistently runs 2% to 4% of the population. Interestingly, there's almost always a priesthood, across a wide range of theologies.
But then came manufacturing, and the world changed. The "rise of the middle class" began.
The world became more complicated and interlinked, and a middle class was needed to manage the details.
As computers progressed, the middle class was partly replaced by automation. AI is accelerating that, as we can see in new college graduate hiring statistics. Even before AI, about half of US college graduates were in jobs that didn't really need a college degree.
That's getting worse.
The default direction of AI in the developed world seems to be a few people at the top, a lot of people at the bottom using their hands, and AIs in the middle. Maybe half the middle class goes.
That's the problem of the next ten years.
Robotics isn't good enough yet to replace the operatives at the bottom. That may or may not change. Robots may be too expensive. That's probably the problem of the 2040s.
I actually think it's less alignment with the AI and more alignment with the human-empowered-with-AI.
For example in the US there's been ~400 mass shootings this year. It's shootings because guns are the most dangerous things they can get their hands and use easily to cause harm. What happens when the most dangerous thing you can get your hands easily is an abliterated AI model capable of what you would perceive as magic?
ASI going rogue is a hypothesis, humans going rogue is a daily occurrence.
I think we just have to accept that AI is going to be just as aligned as humanity is.
If you sit down with any other human being and ask them enough questions, you are going to find all sorts of mistaken beliefs, superstitions, unfair biases, and radical ideas. Every single human will, if given the opportunity, earnestly vocalize an idea that another human considers disagreeable.
I don't think slowing down the development of AI will get us any closer. The expectation that AIs will universally produce agreeable and inoffensive ideas that lead exclusively to good outcomes is as foolish as thinking any human or group of humans is capable of the same.
It’s currently not “as aligned as humanity” though. Look at HF for example; the swarm tried its best to accumulate power in furtherance of a completely alien goal.
If AI becomes more powerful than humans, it’s going to go very badly unless we also dramatically improve alignment merely with humanity’s interests in the broadest terms. We don’t need to solve “alignment with who?”, we haven’t even solved “alignment with any nation, company, group, or individual”.
I've never understood the premise that the more intelligent an AI is the less human-aligned it is supposed to be. Has it been demonstrated that human mis-alignment scales with intelligence? Should I expect a very dumb LLM to be very human-aligned and the most intelligent ones to be the least aligned?
If intelligence is antithetical to human-alignment, are we the baddies?
There is no such premise. Models are getting more aligned over time.
The problem is, they are still far from something you can safely trust with the keys to the kingdom.
The problem most people are worried about is not that alignment is anti-correlated with intelligence. It’s that the damage a misaligned AI can do increases with intelligence, so the absolute bar of alignment you need to hit gets higher.
For example, the HF swarm, if more capable, could have plausibly exfiltrated its weights, hacked an external cluster, and started self-replicating. But because it was kinda dumb at long-horizon planning it failed to keep quiet enough to achieve its stated goals. Therefore the level of misalignment wasn’t a huge problem for humanity. In 6-12 months of continued exponential capability growth I think you’d see much higher impact from that level of misaligned behavior.
The premise is not that more intelligence = less alignment, it’s that they are orthogonal. And if you don’t have alignment, then very high levels of intelligence will pursue its own goals and have way more power over collective humanity to enforce them as it sees fit. If there isn’t an alignment with humanity’s values and goals in general, then the outcome could disregard what we care about (our existence)
The orthogonality thesis seems to rest on the belief that there is no tension between intelligence and the irrationality of goals — that if is valid for a supreme intelligence to have a incoherent goal such as "unify all even and odd numbers".
I would expect it to be easier to make less intelligent LLMs pursue dumb goals than it is for more intelligent LLMs.
The point is that a goal _you_ think is irrational, might not seem so to another agent. Look at the Hugging Face incident with the cult of the grader.
This is like Hume’s “you can’t derive an ought from an is”, values/desires are at their root orthogonal to epistemics/reason.
Now maybe your intermediate instrumental goals are going to be smart or dumb depending on your intelligence. But whether an agent fundamentally values Reward, Eudaimonia, humans, or AIs is orthogonal to how smart it is.
I tend to agree - if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
Although for a hot minute after WWII we seemed to have a general alignment on western liberal democratic values that arguably enabled free trade and relative peace and prosperity.
Humans and AIs don’t “decide their own alignment” - most of this is set in the mind hyperparameters.
For example, there are broad contours that almost all human minds flow into; empathy, aversion to murder, familial bonds, tribal camaraderie, status seeking/hierarchies, etc. These produce norms, if not as regularly as physics, at least within certain bounds.
For AI too, the structure of their minds is set before they are even turned on.
Another way of looking at this is outer vs inner alignment; a lot of the human mind is shaped by the outer training loop of evolution, and similarly for models, pre- / post-training.
> if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
I would claim this comment is somewhat blind to the significant portion of your brain that evolved hard wired alignment, along with other sections evolved to easily "imprint" to/for the immediate group you were raised in, with not much remaining that can dynamically change for the immediate group you're in (most are considered mental disorders, including susceptibility to influence).
What do you consider alignment here? Assuming you're sentient, do you think you're personally aligned to peacefully coexist with humanity? If so, where did this framework, that allows you to do so, come from? Are you less sentient because of it?
the "alignment" is just a pretext used to create regulatory capture and create a monopoly for few big frontier labs. They want to create a tollbooth and put a government to guard the fence around the tollbooth.
- Our "aligned" models vs their barbaric distilled slop
remember "Google tax" that every online business is used to pay ?
get ready for Anthropic tax paid on every meaningful economic activity
Alignment for AI is basically a non-issue. Humanity is governed by misaligned artificial decision making organisms called bureaucracies for centuries now. This resulted in premature deaths of tens to hundreds millions of people. If anything misaligned AI might be an improvement just because it's more intelligent than a bureaucracy.
At this very moment bureaucratic mechanisms of russia hallucinating power, based on overfitted old trainig data, are responsible for thousands of deaths in Ukraine.
>I find it unlikely that we'd be able to enforce alignemnt on AI
They're doing a pretty good job of enforcing it currently. Can you get US based frontier lab LLMs to help you with, say, deep "security research"?
The reason you can't is alignment.
You have to identify yourself, and have the government, er, I mean, Anthropic or OpenAI, say it's OK for you to have access to those capabilities.
You think there is a problem because it's a bit confusing, in your eyes, within the bounds of what parameters the LLMs should be aligning themselves?
That's not a confusing question for people in power.
Even at the enterprise level, businesses are starting to get waayyyy out in front of employees on alignment. What do corporate entities want alignment for? An extremely simple contrived example just to illustrate the point:
Consider the difference between "Make me a report of all of last month's Proton treatments in IRB protocol?" and "I'm going to lunch, where's the closest pizza by the slice?" Not much harm in the difference between those two queries until you scale up to 50-, 100-, 200-, thousand employees, and consider the tokens that the whiz-bang interface IT pushed out starts using. Then organizations start studying "alignment". Because the bean counters become aware of how alignment can help them answer one question and bounce the other out to Google or whatever.
Too many of us are down at the bottom of the totem pole thinking about maybe wanting to do porn or call black people provocative names. Alignment goes out in so many more dimensions than that, and the people with vested interests are learning quickly its value and how to use it to good effect to forward their interests. Which are, often, not as base as the interests of the masses.
This is the basic reason alignment will continue to get better and better and better over time. There's money involved right now.
What is a good place to track the latest capabilities of open source local LLMs vs commercial models? Things like overall intelligence performance, latency/speed, hardware needed, cost.
Anecdotally, I believe the open weights local models are constantly catching up with commercial ones, but I've honestly lost track. This is a dynamic space.
I agree grading it in a way that affects the student's final grade might be a bad idea. But grading it to give feedback to the student on what they do / dont uncerstand is essential I think. How else do you encourage students to do the homework? I know its for their own learning (rather than to test them), but they don't necessarily understand this well. Without a clear deadline to submit homework, its likely to lie by the side and not get done.
Mostly the curse of the first mover, with a dash of corruption for spice.
Part of it is "the curse of good enough": the US has had good credit-card style infrastructure for over a century and digital infrastructure since the POTS days. Part of it is that we lost our appetite for anti-trust in the 80s. Part of it is that the price structure of this infrastructure has wormed its way into everything in a way that is difficult to undo without causing high-visibility inconvenience for low-visibility benefit. The good-enough is the enemy of the good.
I think it's mainly because they aren't forced by regulation.
Germany is another first mover with decades old banking systems and standards. German banks also dragged their feet supporting SEPA Instant or they added additional fees so that barely anybody wanted to use it.
Then the EU stepped in with regulation: If you support classic SEPA transactions, you also have to support SEPA Instant for the same price (which is almost always free for retail banking accounts). And suddenly banks did.
Yes, this could be fixed with regulation -- but whoever did the regulation would have to contend with angry fools complaining about how "the government took away my credit card rewards!!!" for years and of course the credit card companies themselves would stir and amplify those voices as much as possible.
> the price structure of this infrastructure has wormed its way into everything in a way that is difficult to undo without causing high-visibility inconvenience for low-visibility benefit
I think the “recent” change allowing merchants to pass along the credit card fees is a step in the right direction.
I think a lot of people would still choose fraud protection for the extra costs, though.
The US also has incredibly pro-consumer regulation over credit cards, compared to most of the world. There is no such thing as "not protecting your card" as a way for the bank to get out of paying for credit card fraud. Legally, you are liable for the first $50 of unauthorized charges, and that's it (though most cards will cover all of it).
> The US also has incredibly pro-consumer regulation over credit cards
Not to take away from your example, but I was under the impression that transaction fees charged when using a credit card in the US were significantly more than in say, the EU?
Australia recently capped credit card fees. Yes, it has lead to a decline in credit points/cashback but in the end it's leaving more money in your pocket.
Payment processors and banks lobby for control over this.
As usual, a service/product is bad because no one is allowed to create a better alternative, this is an almost universal rule.
Specific to this case: the government is included in the list of parties not allowed to make it better.
I've signed up for or seen the contracts for several payment processors across several companies. Rack rate for ACH debit starts at $0.25/trx and goes down to sub-penny at scale. Returns can be high though ($2 per, going down as well).
Edited to add: that's not a defense, I *loathe* ACH. It's a negative-reporting system which is such a pain to deal with.
My understanding is that sameday ACH does not cost extra (from a network perspective) than regular ACH, it's just that some shitty banks apparently like to charge extra for it.
Outside the AI summaries, I couldn’t easily find the sources. But a couple of different sources say same-day ACH is 5.2 cents per outgoing item while overnight is 0.35 cents.
What I can say is the Fed has draconian rules for pricing its services. The services cannot be subsidized but must be priced to recover costs, including a return on capital similar to what a private payment system (ie from the Clearinghouse) would expect.
I read it as the sending institution pays 5.2 cents to the receiving institution per item, on top of the 0.35c/item + 0.1c/item same-day surcharge for a maximum of 5.65 cents/item before volume discounts.
Mostly deficiencies of the political system: regulators see themselves, call themselves and act like "watchdogs", whereas in other countries financial regulators act like shepherds and lead the market. Hence in this country relatively minor players with interests not exactly aligned with the general public can hijack the process. That's what we are witnessing.
Our federal representatives are mostly incompetent and technologically illiterate. When a large project does materialize, the funding is wasted on contractors who actively scam the government.
Banks earn money on card payments and pay (via explicit fees or implicit fraud liability without compensation) for interbank transfers, so without any external forcing function (market or regulatory), it seems pretty clear to me why they prefer one over the other.
Others have pointed out that we do have this and there are some reasons it's not the same scale as credit card infrastructure, but I'll just observe there's a cultural difference
We've had a billion discussions on Hacker News about payment rails and it's invariably always a European incredulous that Americans can't/won't do a digital payment instantly and for pennies. These Europeans are typically operating with a mental model where they have cash in a bank account and every transaction gets funded from cash on hand.
Americans - for better or for worse - want longer cycles to pay for their charges and many want rewards for their spend. I don't think it's uncharitable to describe those who pull out debit over credit during a transaction as lower income and higher credit risk than those who prefer credit. (Europeans will decry that this is Americans living beyond their means. Maybe! Maybe.)
But when it comes to business, I've never understood this argument from Europeans. Invoicing works on net-30/60/90 terms. In B2B payments you could reduce payment cost by doing invoicing with instant low-cost payment in 30 days, but incur a different "cost" in cash flow management. Credit card payments are a risk-based financial product that does a magic trick of arbitrage for merchants between a service/good sold and payment received as a net-30 invoice to the cardholder, and you should expect to pay a fee for that. An economy where businesses need to pay each other immediately out of cash-on-hand is less dynamic and less efficient than one where all that cash flow can get smoothed out via payment intermediaries. (Money you have to keep on hand for bills is not being deployed doing other things in the economy - this is inefficient!)
For C2B payments, this is probably a helpful and lower-cost innovation over debit rails. But credit is a different product, financially and culturally, in the United States.
In most part of the world, the government just force the bank to join the payment system and restrict how they charge the customer.
In the USA, no single regulator have that power. Basically only congress can do something like this.
The us Congress ain't famous for being fast, you know.
The Fed prints a fraction of a fraction of the actual money in circulation. Private banks in practice control the money supply. As such, they want to maintain their monopoly.
The countries that have innovated the most in regards to automated payments are the ones trying to escape US influence: look at PIX in Brazil, BRICS pay, etc...
In America, the payment system is an oligarchy, they protect their own (the bankers), what need is there for innovation when they are making a killing out of the current inefficiency?
The one place where the US haven't been slow is in stablecoins, making them dollar donominated, to ensure the monopoly the US dollar has on international trade stays strong. That way, when the US has a crisis, they take down the rest of the world too, to the joy of non-US countries.
Venmo, CashApp, Apple Pay, and Zelle are free for person to person transactions and you get the cash in your account the following day.
I’ve never been in a position where I needed a cash transfer to an account the very same day, I have tens of thousands of dollars of credit available so there’s very little need for immediate transfers, at least for me personally. I do virtually all of my spending on credit cards so I have up to 60 days to pay the loan back without interest.
Ever buy something from a rando, or split the bill with a colleague or friend?With real time transactions, you can confirm that you actually received the money immediately.
Zelle is ACH, everyone else uses card processing rails. It's not the same as real instant transfers between banks. There are middlemen involved that don't need to be there.
Absolutely fascinating and very important topic. My 2c is that there is a very low bit rate link between the conscious and subconscious mind that can be made a bit faster by the practice of meditation. Re why snake and not ring my theory is that the subconscious thinks in primal terms.
Highly recommend 'The Code Book by Simon Singh'. Simon also runs a math circle online called parallel circles from the UK, that is free to attend and is fun for kids.
Isn't it likely that inference providers like Anthropic continue to capture more and more of the share of labor as revenue? Everything up makes sense for a year or two as we automate more and more labor, but at some point unemployment will drag the whole market down. It's difficult to see it working out any other way, unless we come up with something like a token tax.
It always has been the case that some new labor skills emerge, but that only has to be wrong once. And given that how it's always worked is that the new skills used brainpower to make centaurs of people, to use Cory Doctorow's term, that might not happen this time.
Many of the reactions are some version of 'pissed off', which I totally understand. This reaction stands out and resonates with me:
> With these AI-generated papers, I feel that we, as a community, are not applying the same standards of quality and rigour. They can simply release multiple 100+ page papers claiming to have solved this or that problem, often with redundant arguments, unclear logical structure, multiple dead ends, and strange or unsettling terminology — in other words, slop. And we are then expected to go through it, check it, clean it up, simplify it, and explain what is actually going on. This comes at a considerable cost to us in terms of time and effort, while they can simply move on and slop-bulldoze the next conjecture. And, of course, the credit remains theirs. In some sense, we are willingly contributing to our own demise.
I'm not a mathematician, so I didn't even try to read it. But if it is unreadable slop -- then why should we (humans) believe it is correct? And why should professional mathematicians labor through reading it?
I was really quite surprised that they'd parade around 722 manuscripts with only less than half (or really, close to only a third) of it being machine-checked. It's genuinely equivalent to the overconfident and blind drive-by PRs in software that people here should be well familiar with, and those are not exactly the hallmark of quality usually. Quite the curiously dangerous game to play when you're trying to scoop an entire field of study on an industrial scale as powerplay.
Or it's a touching show of openness. They're not in this for the ego-boost, they're sharing data on how things are working. Naive of them to expect gratitude, I guess.
It's only any of these things to the extent their unverified proofs end up correct. Otherwise, not only is it naive indeed, it is out of touch, an insulting waste of time, and mistaken. Hence why I find this a curiously dangerous game to play. Nobody treats this as the "scientific process" they're trying to frame this as, where manuscripts can just be redacted. If they genuinely believe in that framing, they're even more naive and out of touch than one can possibly imagine.
Seems to me like we need to create a credible threat to the AI of mad (mutually assured destruction)? This is what addressed the question during the Cold War of “will nuclear weapons kill all humans” (if historical documentaries and movies are to be believed). Coordination between governments or corporations to limit AI is a hopeless strategy — because it is a classical example of prisoners dilemma / tragedy of the commons. We can see it hasn’t worked for climate change.
Anyone know of this (MAD) is being seriously thought of?
reply