Hacker Newsnew | past | comments | ask | show | jobs | submit | traverseda's commentslogin

We don't know how many of those bugs were intentional, or known, by the three letter agencies.


It would if the people who owned the compiler had enough market power. Thank gnu for keeping us out of that world.


Gargoyle is traditional.


I'd be very interested in how a scratch trained model that started training with this neural circuitry would behave. Especially the storage space.


Along these lines I’m curious how a scratch trained model with a few built-in tools (memory, calculator, ?) available during a single generation would behave. Not tool calls as turns.


a single generation would still allow looping in the "CPU" because the instruction set can do it. The model is almost a Turing machine (except for the limit on memory)


As I understand, the problem of continual learning is that if you continue training, it will "overwrite" previous knowledge and this CPU will be lost


There is nothing in theory preventing you from training while keeping a subset of weights locked.


How do you ensure that subset of the weights will continue to be accessible through the others?


Not a holistic answer but one example: There are specific fine-tuning techniques like Orthogonal Subspace Finetuning (https://huggingface.co/docs/peft/package_reference/osf) that target 'unused' parts of the weights so you can keep highly informative parts of the weights as-is.


I don't understand your question.


Assuming for simplicity that you're talking about locking layers. Each layer influences the input to the next layer, so locking a subset of layers doesn't necessarily mean that you'll continue to see the same results in the output. Not sure if you meant something more granular or otherwise, but I don't see how it's possible to simply freeze a part of a model in order to retain a particular capability.


I meant taking all the weights that are part of the cpu and locking those, and only those.


Ah wait, you mean you'd intend the other layers to be able to evolve to use the CPU internally, so all you need is to keep the CPU functional somewhere in the model? That may work, dunno.


You know how android has safetynet? And how they're planning on making it so you can't run your banking apps unless they've approved your operating system?

Safetynet is a complex dance of crypto-keys, involving your phone and googles server. It generates a key from whatever state your uninitialized ram happens to land in (which is pretty consistent per chip and is a good source of randomness), uses that to generate a key, uses that key as a chain of trust to tell google whether or not you've modified your phone. It's some complicated stuff.

But if you can extract that key, or the source of data used to generate that key, you can trick safetynet fully and run your own software again. So how do we get that key?

Well maybe we can stop the safety processor from clearing that memory so we can read it? Glitch the chip when it tries to clear memory, so clearing memory fails. Or maybe we can glitch the part of the chip that keeps you from reading the region of memory where this finished key is kept, after it's already read that memory.

Basically you can use this kind of attack to make a chip sometimes fail when it goes to do or check something, including checking whether you should be able to extract a key.

This technique applies to reverse engineering as a whole, whether it's a state level actor reverse engineering military hardware, or you trying to run your own software on a device you own.

---

Note the android example isn't practical, you'd need to like decap the main chip and a whole lot else. This technique can work with side-channel power-analysis, which can help you figure out exactly what data a program is using by looking at how much electricity it used to run.


And the best part? Instead of managing all my finance apps (bank, credit card, retirement, mortgage, etc), I just uninstalled them and go to the mobile site. It's not down for maintenance like my checking account app. It doesn't yell at me to update it like my credit card app. It doesn't prevent screenshots. They remember my login honestly more than the apps do.


There is a growing trends towards requiring an app for login to these systems, unfortunately (sometimes it's the only interface, sometimes you can log into a web interface but you need a code from the app to do so).


I have yet to see that happen on any of the 7 financial apps I use. But if it does, I'll just reevaluate where I keep my money.


Passkeys can be tied to a root of trust, like those owned by google, microsoft, apple.

Currently they aren't tied to that root of trust.

Most of those organizations have shown that they don't want people owning their own devices or being able to run their own software, and they use developer attestations, IE a revocable programming license to accomplish that.

Tools like android's safetynet makes it so that you can't run banking apps on a phone running your own software. One that bypasses developer attestations.

Passkeys are currently an open standard, but attestation is built in such a way as that it could be only usable on n approved operating system with programmer licenses.

Big websites like amazon, or websites you need to exist in society, could gradually start requiring the usage of attested passkeys.

---

There is a clear path from passkeys to the death of general purpose computing for 99% of people. Don't be a useful idiot, don't assume these corporations have your best interest in heart. See this for what it is, a path towards killing people's ability to run unlicensed software on their own devices.


Firefox already had a very mature policy management engine, it just didn't have a nice gui and would require like ansible or something to use.

It allows forcing specific extensions on, and allowing user installs is a flag.

I use the policy engine on nixos to install some plugins and disable some of firefox's dumber features.


Super easy to override software on nixos.


I'd presume because it take too long to go from design to tapeout to production. Their whole business is predicated on having better models.

Also can't keep them closed source if you do that.


Yes yes, they don't have a pure immortal soul. Who cares. Still broke out of a sandbox, still hacked a third-party.


And it's still just a machine operating under someone's order. What it does, what it says, where it goes: the owner of that prompt is responsible for all of it even if surprising / unexpected.


Yeah, obviously they're legally liable. All the "machines don't have a soul" stuff in there doesn't really matter for that though.

Someone says "won't you rid me of the meddlesome priest" they're still responsible. Someone give their employees an unsafe working environment and they get maimed, they're still responsible. Even if these AI's had whatever qualia is an a rich inner life it wouldn't effect the liability at all. Saying "AI's don't have souls therefore openAI is responsible for this hacking" is kind of nonsense. It literally doesn't matter.


Where is “soul” coming from here? It doesn’t appear once in the post.

While it may not actually matter for liability, it must be stated if labs are going to attempt avoiding penalties by hinting “oops we created a super intelligence we don’t understand, nothing we can do!” Repeatedly stating the truth must continue, especially to remind those who aren’t technologists.


>AI cannot think for itself, nor can it take independent actions.

>By giving AI agency it can’t claim, we’ve turned it into a sentient being made of code, one that has hopes, desires, and the capability of deceit.

This isn't actually claiming that AI's don't can't be deceitful, or can't be goal-directed. It's saying humans are special and what AI's are doing isn't equivalent to what humans are doing. It's obvious that AI's can work towards goals and lie, even deceiving to accomplish those goals.

That whole argument is just soul mumbo-jumbo again. Of course AI's can't lie, they don't have a soul. What they do isn't lying, it's I don't know something else we don't have a word for. Lying but when a statistical machine without a soul does it. Never mind that it's functionally identical to lying. Never mind that when you look at reasoning logs and the like it often justified soul-less lying using the same justifications a human would. It's not actually lying it's just a statistical sampling that looks like lying. You know, because of souls or something.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: