> So I think they have an indirect responsibility of promoting this culture.
Interesting comparison to compare the "standard" software fraudulence of fudging your activity numbers to Theranos' fraud.
I think when it comes down to it, it's a false equivalence - one is a temporary measure done to overcome an inherently short-term critical mass limitation, the second is an indefinite fraud trying to paint an impossible company as real.
"Fake it till you make it" is a legitimate and almost necessary strategy for an early-stage startup, I wouldn't conflate it with Theranos' fraud.
Agreed. Even the non-governmental example in the article - Nikola Corporation - was uncovered as a fraud more than six years ago. And the governmental examples as described aren't even fraud, they're corruption.
Even in this day and age of somewhat exorbitant company valuations, it's infinitely more difficult to replicate the kind of fraud of Nikola or Theranos. If only because of those companies setting a precedent for abject fraudulence and making basic due diligence that much more of a necessity/expectation. A company like Instinct wouldn't manage to get away with its current valuation if their technology wasn't front-and-center for everyone to see and use.
Time will tell - for all we know there's currently a large company more fraudulent than either of those, and we'll only find out in the future - but trying to paint it as the norm isn't based in any recent real-world examples.
I feel like non-professional vibe coders are scapegoated too often. I'm a professional software engineer with extensive experience in CI/CD pipelines. And I heap on 30x more stress on GitHub than I did before AI-powered development took over - because (1) I'm much more efficient and running many development tasks in parallel, and (2) CI runs are one of my tools for ensuring that quality doesn't degrade with velocity.
I think that even if you stripped out non-professional vibe coders from the equation - the problem is still there, and will keep compounding.
Very fair point, and I'm absolutely in the same class. I suppose all I was pointing out was I wonder how many projects have CI that really don't need it but end up with it because AI added it without the user really understanding.
But yes, I also hammer GitHub a lot harder nowadays. I've had to move all of my private repos to self-hosted bare metal because I chew through 2k minutes in a few days.
Ive been spending a lot of time recently across all my projects optimizing ci because with this latest wave of models I can confidently make a lot more changes faster, and even if the pipelines are kinda optimized (I have rules like every pipeline fails if it exceeds 12 minute wall clock) the sheer amount of changes hitting ci is overloading it
Just subscribed, the graph of satellite launches in your video is mind-blowing - I didn't realize just how exponential the launch rate has become. Would be great if a future video could go into why this is happening, and if it's sustainable.
FYI VSCode's SSH Agent is a godsend for remote development - the "disadvantages" that Fly lists are part of its advantages. I've worked in several teams that have made extensive use of the extension, and it's never been an issue. You can restrict SSH access arbitrarily to ensure whatever security or access guardrails you need.
As a Linux user I've hated VSCode's ssh. There's lot of annoying things that make it harder to admin for. Like it doesn't pick up the MotD, preventing me from showing users important messages. I've found that it also doesn't reuse sessions (at least by default. TBF, neither does ssh) and I'll find that there's just dozens of open sessions over months from users. I literally had to write a script to boot people...
It would be one thing if the plugin was just a wrapper and people were still expected to know ssh but the plugin abstracts away all that and is intended to make it a "use VSCode on remote machine" tool. So it needs to do more than just handle creds, otherwise it creates a divergent experience while making people think it's just ssh
> There's lot of annoying things that make it harder to admin for.
It also (AIUI) tries to walk the entire file tree, so have fun with NFS (auto)mounts.
It also amounts to letting off fork bombs: we set up limits for a maximum of 256 process per UID, and regularly get folks asking "what does this 'cannot fork' message mean?": it mean you're trying to DoS the system.
> we set up limits for a maximum of 256 process per UID, and regularly get folks asking "what does this 'cannot fork' message mean?"
The max limit on 64 bit systems is what, 4,194,303? So if you have over 16,000 users per VM this limit makes sense, otherwise it just seems user-hostile.
Every process takes some memory and other resource, yes a stale process will pretty much all end up all paged out and not massively in the way of active processes, but they still aren't entirely free so it is more than a bean-counting number.
Yes, under Linux (and most unix-a-like systems) small processes are cheap to bring up and tear down which is why we create them so much, and it is not uncommon for complex interactive commands and bits of shell scripts to create several¹, but these are all likely to be short-lived so a limit of 256 certainly doesn't seem to be obscenely low to me.
What could it be doing that requires 256+ processes to be kept around for a prolonged time?
--------
[1] made up example: comparing filtered content of two gzipped files and sending the result through a script to send alerts by mail if certain things are found would be 7+ (2x gzip, 2x or more grep, diff, bash, mail or curl depending on what service you are sending alerts through)
> The max limit on 64 bit systems is what, 4,194,303?
What a weird framing... I'm not sure what you're even trying to argue. I mean a single process can overload the machine. Just because you can label 4m processes doesn't mean you can actually run that many programs. Just think about that for a minute. 256 processes is pretty generous
> The max limit on 64 bit systems is what, 4,194,303? So if you have over 16,000 users per VM this limit makes sense, otherwise it just seems user-hostile.
And yet we still regularly loads of >100 on our 64 core HPC login codes, and swap is regularly used even with 96G of system memory (we have per UID memory limits too).
What's hostile is the VSCode (and Codex and Claude) makers developing tools that basically DoS a system because they assume it will operate only on single-user machines.
(And WTF are you doing that you're forking 256 processes? We have quite a few expensive HPC nodes: use those to build, not the damn login nodes.)
What would truly be user hostile would be to allow so many processes per UID that a small handful (maybe using VSCode) make the system slow or unresponsive to everyone else.
Honestly every developer needs to increase those default limits, they are too low for modern development... So you are just crippling them and a proof of that is they keep getting this error while in their regular workflow
I don't even hit 256 on my desktop with a shitton of things open and 75 firefox sandbox processes, much less on a remote server. What in heaven's name are you doing to cross 256?
Honestly I think this thread has just devolved to HPC admins vs people who don't understand how shared multiuser sytems work cause they've been stuck on a laptop for too long to remember.
> Honestly every developer needs to increase those default limits, they are too low for modern development...
The users can develop on >100 compute nodes, but choose not to bother doing any kind of forwarding/proxying/jumping to them and just do stuff on the login nodes.
If they can't be bothered to do a "ssh -J …" then it's on them. The resources are there.
Ah, I see. Our admins typically understood that users were being given disk quotas precisely so that they could use that disk space, but that's probably not a universal stance.
Quotas mean "more than this is clearly too much" not "please use this space".
In good times nobody minds, but in bad times when you just can't extend the drive you have to tell people off. Part of the job of making sure the system can continue to work for what you need it, under _real_ constraints.
You may have to delete stuff, you may have to shut the server down to save power. You may have to limit clock speeds. It depends on the environment and "it really should work because it should be covered by next day on site warranty and you could download more ram" often doesn't apply.
> Quotas mean "more than this is clearly too much" not "please use this space".
Ah, memories of Uni, back when storage was fairly expensive, where we had both hard and soft quotas. The soft quota would allow for temporary growth of build artefacts and things¹ but you would get stern emails if you were over your soft quota for 24 hours, and if you persisted without good reason³ your hard quota would be reduced so you effectively have no soft quota any more.
--------
[1] some machines had no local storage that the user could touch so putting them there was not always possible, some people on Windows machines had local storage but didn't have the relevant tools locally so were actually running things on the shared server(s)² instead of that just being a storage resource
[2] via telnet/rsh/rlogin: yes, I am that old… SSH was a thing by that point, though OpenSSH wasn't, and I was using it where available, but the use of older plain-text protocols was still far far more common
[3] it wasn't actually difficult to justify a quota extension for project work, in fact people enrolled on certain modules got higher quotas automatically
Yes, let's hope it doesn't get even more expensive. Around here we are purposely not upgrading hardware unless we have to, hoping to ride through the price increases. I'm sure lots of people are doing the same thing, which probably won't help much once prices start to come back down... assuming they do.
Back in the day, because the architecture was shared memory and cooperative multitasking, this is how MacOS applications declared their memory constraints.
Apps had a (recommended and then user-configurable) "Minimum memory" and "Preferred memory." The app would not launch if the OS couldn't give it the minimum. It would then give the app up to the preferred amount, if available, exclusively... This was in the era before virtual memory and paging, so there was no easy way to share memory across an application boundary.
This mean that savvy users with high-resource tasks knew you had to launch your apps in a certain order to get the architecture into the configuration to do their work.
This is a fascinatingly "pets" approach to admin; it's been ages since I've been somewhere that used this approach. I've been in the "cattle fields" for decades now.
In my ecosystem, developers don't have time to glad-hand like this; if there are issues with DEV_NODE_CFG_1_29875, we might talk about it over Slack (because I'm the first one to know there's a problem as the end-user), and if we can't sort it out they'll give me some time to backup, blank the whole machine, and I get a brand-new image of DEV_NODE_CFG_1_29875.
I can't even tell you off the top of my head what territory the physical machine is running in or whether it's the only dev-node on that hardware.
(Broadly speaking, I think Microsoft is assuming cattle ecosystems; they're not openly-hostile to pets per se, but they have a strict ranking of the priorities because the "cattle ranches," as it were, bring in more money).
I had it seen on servers that were typically reserved for the team but there was no official booking system for those machines. When you start using the machine you would typically put some note to make sure somebody else does not overrun your long-running tests or performance measurements.
Sometimes you aren't really "the owner". For one example I was admining my group's server in grad school. I wanted to add quotas (we already had zfs) because people were abusing home directories but my advisor and a few members were very against it because I was "over complicating things". Their worry about me wasting time (20 minutes for all machines...) resulted in hours of yelling at people over the years. All because, surprise, a small number of people can't follow instructions and abuse systems, ruining it for everyone.
Another frequent problem we had was people using the systems while others were. They wouldn't check the machine's status. And of course you can guess that I wasn't allowed to add a scheduler.
A lot of groups do things in janky ways. Often not because they don't know any better but because leadership doesn't and is assertive
"The announcement was visible in a MOTD on every server in the fra3 location for the last two days."
"I have not touched any machines in fra3 for a week, how the hell would I know of it? Why do we even have #fra3-maintenance and #maintenance channels then, if that's your stance?"
> "I have not touched any machines in fra3 for a week, how the hell would I know of it? Why do we even have #fra3-maintenance and #maintenance channels then, if that's your stance?"
We put the MOTD up ≥7 days in advance and put it in relevant Slack channels.
We have the servers role, you can derive that from the name obviously, but we do have hosts which as the same naming scheme, but slightly different roles. There's when the last Puppet run happened and what it applied (and who authored it). Depending on the host type there's also active/standby, warnings for production hosts or information about increased log level on things like sudo.
It sounds like a lot, but it's fairly compact and really helps when you need to absolutely sure where you are and you have eight terminal windows open.
Things that should probably be an email. I think one time I’ve seen it work is to tell you stuff specifically about the host you’re on to avoid mistakes.
I miss fingering, it was such an easy way to get a log dump or status update from various daemons, I still think it has immense utility .. some of my fondest operator days were sat under the umbrella with a terminal while sleep 30 ; finger someone@all-the-things ; done .. watching the machines from afar.
Can still do it these days of course, but one with a seriously copious helping of ssh in the mix too ..
Trouble is, nobody else can do it. The only reason I have to use {social-media-blob} is because my friends don't know how to finger.
I don't find it that opaque. Even without trying to deobfuscate the obfuscated source code which Microsoft ships (I haven't tried but it wouldn't be hard) a lot of details about how it works become obvious just by reading its logs.
They will never open source it. For the same reason Pylance etc. aren't open source and MS tries hard to prevent them to be used in VSCodium. Every one of their open sourced projects contains a closed source plug that MS can pull at any time that is one of the features that gives the project its unique selling points.
Exactly! Like if I'm admining a server what am I supposed to do? Message on a big slack channel and have everyone ignore me? It's easy when people are just logging in through normal ssh as I can put a big bright warning message on their screen that they can't ignore.
I have looked for mosh support for a while and not found anything. It would drastically improve the connection experience in VSCode. My terminals never disconnect anymore, but the Code popups about your sessions needing to be restarted has drastically reduced my usage.
But so is the question. MOSH interprets all the escape sequences and uses them to decide what to send to the client. That way if you tail a log file and then get disconnected, you don’t have to download every line of text that was output to your terminal while you were away; it can just send you what is currently visible.
MOSH is strictly for interactive use; never ever for automated uses like TRAMP or VSCode or sshfs.
It's just another messaging. I used them to automate messages about current disk usage and warn if the machine was actively being used.
But then again, I had people who would run jobs without checking if the machine is already in use. Obviously these people didn't check email or slack either...
The one this that is better than just SSH+Tmux+Vim, is that if your latency is higher than 30-50ms, since VSCode's SSH agent streams the files to your computer, the typing experience feels snappier. When you work half a continent away from where the servers are, it makes life nicer.
Use a local vim, and use its ssh support. It will download the file to the local buffer then upload it when you save. This way your vim config remains on your local machine, too.
Related, netrw (vim) can do quite a lot of things that I think people don't realize. I mean not just that it supports file trees, but well... just open :h netrw
> I'll find that there's just dozens of open sessions over months from users
We've had the same issue with our local HPC; a few login nodes serving hundreds of users at a time, and each login node used to get swamped by these dangling SSH sessions/servers. They also wrote a script that shuts down all sessions once a day to save the login nodes.
I run the editor (and its extensions), my projects and any agent harnesses from inside a container and use that extension to get an editor.
This is mostly to protect my credentials and data from malicious extensions/dependencies/rogue-agents, bu it also lets me quickly port my dev environment to any machine (I use linux at home and macos at work). Just install podman, install VSCodium, add the SSH extension, build image, add my utility shellscripts (to quickly get in and out of the container in a shell) and done.
Apparently Microsoft keep some VSCode APIs proprietary so only its own extensions can use it (allegedly for security reasons), which is why this specific extension only works in VSCodium. I wonder if it is vulnerable to the same things the article points out.
I used VSCodium before and found it to be a massive waste of time for no benefit. A significant number of extensions either aren't in OpenVSIX (or whatever it's called), or don't work for some reason.
Just disable telemetry in VSCode and you're good.
Yeah, but to be honest on my current project the only extension that I needed, but wasn't available was the Typescript 8 support (the typescript with golang-based language-server) which should land eventually in the built-in typescript support.
Yeah this is the right architecture for remote editing with remote tools. It works really well. (There are longstanding bugs around reconnection when the SSH connection is broken but that's not the fault of the architecture.)
Also using it, and by now at least I see the reason why they did it. VSCode has a large plugin ecosystem, many which are essential for development. The problem is that those plugins don't know anything about remote development and expect to use the standard file system and OS APIs to interact with the workspace.
So how to make the plugins remote-capable? You could write a massive virtualization layer that captures all system calls and forwards them to the remote - or, you could run the plugin on the remote and just pass the user commands and UI updates over the connection.
VSCode does the latter, so the nodejs runtime is where all the plugins are running on the remote.
(I understood the reason, I didn't say it was a good reason...)
It also fills up the hard drives on shared server, where each user up to 5 GB of vscode nonsense.
Also, I realized that students are likely to use vscode to connect to embedded Linux machines and fill up their emmcs (and RAM), so I came up with a partial solution for our experiment's yocto image (https://github.com/RNO-G/meta-rno-g/blob/main/recipes-suppor...) , but a more general solution would be nice. Probably better to just not allow any node process to run via LSM.
It also has some unfortunate OS / glibc minimums which means that it's growing less useful as time goes on. I work on a lot of machines that are from centos 7 era (including amazon linux 2, which doesn't have an upgrade path: you just have to build a whole new instance and migrate). I have had to pin my vscode + extensions to old versions because it works on older machines. They just stopped supporting platforms (with lots of notice; to be fair). I would love if they had a binary tool that could be built which was much more agnostic to the vscode / extension version so that I could just keep using it everywhere I've been using it, while allowing me to keep current with the latest and greatest.
At least when Python minimum version changed there was a separate extension forked from the original that I can use when I need to work on old code alongside the new extension for more modern code bases. Sadly, no such thing exists for remote editing that I know of.
It turns out ssh is, well, a little awful. I use emacs as my primary dev tool, and I hardly ever tramp to a remote machine, preferring instead to ssh via terminal to the machine and run a local emacs instance on the target machine. Reason being that the ssh connection underlying Tramp can get downright creative in the ways it crashes or hangs emacs.
SSH, as a protocol, dates back to a time where you even though the network was built to be (ostensibly) robust against attack, there's a deeply-ingrained assumption that it's mostly up and persistent. That assumption doesn't work well in the era of laptops and wifi, as anyone who uses emacs regularly may be able to tell you. The defaults for SSH are wrong for the modern world, and once you tweak those the protocol reliability (against dropped or rerouted connection, not data corruption) is still rough.
Using SSH as a springboard to inject a better-designed protocol server is the right solution.
This is super good advice, thank you for sharing it.
It's good that the system can be tuned to work better, but the tricky bit is that the vscode solution just doesn't require that; for most people (it seems), It Just Works.
I’ve been using it daily since it came out. At first it was because I was tired of docker slowing my Mac down with some really heavy client projects. But now I use it as an easier ssh client w/ file editing. I really don’t like using vim/nano. Keeping everything in the same ide, huge for me.
I've been using VSCode Remote since it came out and it is indeed a godsend. I use it to develop from one trusted machine on my own LAN to another trusted machine on my own LAN and I want the experience to be as similar as possible to developing locally, which it is, and which is why I appreciate it.
I'm not surprised people trying to provide remote code editing to non-trusted clients are freaked out by it, but it is a great tool when operating in a trusted circle.
The ethos of VSCode was supposed to be lightweight, something like Notepad++ with a terminal. Developers have lost the plot. Please recommend Visual Studio if you would like a feature-rich SSH agent. I think developer trends have supercharged VSCode and it feels shinier and new with all the extensions but this is an anti-pattern; it defeats the whole purpose.
I don’t think it was ever intended to be lightweight like Notepad++ given the architectural designs from the outset (LSP, Chromium-base, etc).
It always felt to me more like a desperate attempt by Microsoft to regain the IDE market share as low end FOSS editors started taking over. So MS wanted to appeal to the open source community.
And it worked. Even if the primary build of VSCode which most people run isn’t technically open source.
Not really sure what low end FOSS editors you are referring to? Textmate?. The Monaco editor predated vscode by 4 years and it was an attempt to build a browser-based texteditor as a take at cloud9 which people thought was gonna be the “future of cloud development”. Vscode happened after Atom (which was not low end) made electron apps viable and cloud9 turned out not to be the future. Both were an attempt at SublimeText (which was not FOSS). SublimeText itself was a cross platform alternative for textmate.
The FOSS text editor space was always crowded between vim and emacs. FOSS alternatives would get measured against these 2 behemoths and it was a tall order to compete against. Now FOSS IDEs were a different story.
Java, C#, and C++ had many sophisticated and advanced IDEs, some FOSS some not. However, PHP, Python, Ruby, and JavaScript were quickly gaining huge mindshare and those developers didn’t want to install Visual Studio, Eclipse, NetBeans, or IntelliJ. It’s a tough proposition to tell a Ruby dev to install Java, then install Eclipse, then install an extension, then learn Eclipse nonesense just to edit your Ruby files vs “just open SublimeText and edit your files”. Most of what those developers wanted was just syntax highlighting and basic directory navigation to begin with. Atom/vscode were an attempt at SublimeText alternative. VScode was objectively better than Atom and TypeScript was also objectively better than CoffeeScript.
For all practical purposes VSCode is a vessel to sell AI, im using vscodium for now but expect a cycle reset for it this decade (someone makes a new lightweight IDE etc)
it's worse than that, last i looked into this - there's functionality in the protocol that allows the remote system to modify files and execute code on the local/frontend system. it really is bananas.
Edit: there's a security note (still) on the remote ssh extension page:
Security Note
Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
when slaves became workers and joined the union, the unions became "social partners". this is how one closes the laptop at 4 and doesn't have to suffer vibe-decrees mandating RTO
The ability to remotely run arbitrary code on a machine that intentionally gives SSH shell and write+execute access to the filesystem is not a vulnerability just because it's a productivity aid to users who want to leverage this access to do bad things on the remote machine.
A remote access protocol that gives a potentially untrustworthy remote system the ability to execute arbitrary code on the local machine is a serious problem in any scenario where the remote connection is presumed to be a one-way trust boundary.
Which of course includes any scenario where I myself deliberately run untrustworthy code on the remote, no matter how much I trust the remote itself and its owners.
There's also the risk that VS Code is a "trusted" application in many enterprises because developers force it to be. VS Code's node runtime (and plugin system) executing somewhat arbitrary Javascript means that any dev servers become vulnerable to exploits you otherwise might not have prepared to defend against [1].
Otherwise, you could see Javascript running on a server and instantly know something was odd, depending on the server.
From that same article - "The agency said it has identified nine crashes potentially linked to the issue, including one fatality and two involving injuries. It is also reviewing six additional crashes that may be related."
Fifteen crashes - though not to be trivialized - is not a damning number at all in this context. What's more, per the article it's unconfirmed that the crashes are related, so it's hardly fitting to dismiss Tesla's approach based on this.
I think it's great that serious efforts are being made in different approaches to autonomous driving - and in this thread's context, it seems possible that Tesla's approach might eventually be revealed as the optimal approach given modern AI.
Something a lot of model providers don't talk about: any time an engine uses speculative decoding the throughput will depend on how much your output token distribution matches what the draft model was trained on.
The DFlash2 draft model we're using was trained on a lot of code, so if you use it in a coding agent you'll probably notice it run a lot faster (we've seen it break 300 tok/s).
FYI, I might be missing something but I think your billing system might not be working well - I'm not seeing any indication in the UI that my usage is being deducted from the $5 of free credits.
Hey Daniel! It's a bit hidden, but at the bottom of the billing page there's a "Credits" section which should show usage of any active credits and the balance remaining. The usage/billing metrics are batched/handled async so it might take a minute or so for usage to be reflected. Let us know if it feels off.
This is a boggling level of disdain for customer security - even putting aside the insanely low levels of data security, it's mind boggling that the website remained up for months after the disclosure, and that even after being taken down the vulnerability remained open.
This is a boggling level of disdain for customer security
To be fair, this usually doesn't start as a boggling level of disdain. It usually starts out as 100% ignorance. It's how the people and the group respond to the negative feedback from experts and from reality, which brings in the disdain, even spiraling to boggling levels.
There are two deep lessons herein, rooted in game theory.
This is an "I have no idea what the hell I'm doing" level of incompetence.
Isn't it accepted security knowledge, that about 99% of everybody is at a "should not be doing it myself" level of security/crypto incompetence? I'm not saying that the example isn't particularly bad. It is.
Requiring competence would appear to be the wrong way to do it, here.
Given that the password hasn't changed, I'd assume that there are exactly 0 sysadmins or software engineers working at this insurance company. A web app was poorly hacked together a few years ago, and just ticks-over in the background. Nobody in the org knows about the exploit (and it's possible they don't have the capacity to understand the exploit).
There's a serious problem with human beings. A very loud, emotionally charged warning used to work perfectly for us. "SABERTOOTH TIGER!" is obvious and it's useful for the warning to be delivered with such emotional force.
However, there's a problem when the severe danger is disguised by layers of abstraction and complexity and obscured by time. Even emotionally neutral warnings will trigger our psychological attack defenses in these cases.
Note, I'm not saying op did anything wrong. What I am saying, is that delivering negative feedback about anything complex is itself a complex operation!
A security membrane which needs this kind of feedback to work correctly should be viewed as having a serious design flaw.
In case you ever decide to return to AWS, its Cost Explorer is far from perfect but it can show you where your expenses are coming from, especially if your costs are pennies. In the last re:invent they even released daily granularity when grouping by resources (https://aws.amazon.com/blogs/aws-cloud-financial-management/...).