That's not the problem here. Lovable uses the Supabase Auth layer; they just have their own interface on top of it.
Supabase is kind of similar to Firebase in that you usually communicate directly with the browser through a JavaScript lib. So you need to pay close attention to the security rules (RLS policy in Supabase's case) to restrict data per row based on the user auth.
It's easy to get it wrong; that's why so many Firebase and Supabase databases end up exposed. The platforms have improved the UX so that it's not so common anymore, but it's definitely a gun that's easy to shoot yourself with.
You don't have to expose it like that. You can disable that client and use a server to communicate with the database, which reduces the exposed surface in exchange for complexity (by even having a backend at all).
Mix that design with users with zero tech background who know nothing about systems design and LLMs, and you have a recipe for disaster.
We use Lovable as a prototyping tool and then eject it, review, and deploy on a different platform. It works great for that use case.
I'm in a similar situation, using Microsoft again after 13+ years. Outlook and teams I expected to be garbage, but Word and Excel was really surprising. I get it that everyone has that one feature of excel that is not available in gSuite so they need it. But UX in general is so much superior at Google it's baffling how they let it's workhorse fall behind so much.
So the servers are ok but many of the health checks are failing and so the servers are taken off rotation, increasing load on other servers. I'm sure it's more complicated than it seems but from my POV this looks like amateur hour at Microsoft.
Amateur hour? Not sure if you've lived through one of these cascading failures, but they are hard to design against, hard to load/chaos test for, and hard to recover from. The key is that this is a dynamic load, not a static load; and it's normally amplified by uncontrolled retry behavior by clients.
In retrospect, the cause and cure are obvious to everyone.
I have lived through and debugged cascading failures and large scale distributed systems.
The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.
Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).
A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.
We took some of the hardest problems in computer science: distributed systems, scale, concurrency and networking, and made them the standard for every To do app.
This is the inevitable consequence.
When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.
People expect software to be feature rich, complex and solve all the world’s problems. And if you don’t do that, the vibe coded competition will beat your cleanly architected product commercially.
It's not even better though. It doesn't have more features because it's a distributed system. It isn't faster because it's a distributed system. Maybe it has a bit higher uptime but nobody was paying too much attention to that anyway, except when it's down. And it's much better for your PR if, when it's down, it isn't down for everyone at once.
Managers and marketers are the ones asking for all those useless features. And god forbid they run out features to ask for, because then they start asking for architecture v2.0 projects.
I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.
My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.
I can't think of any other engineering discipline where demand might increase by a factor of a million in a few seconds because some DNS record was propagated incorrectly or something - regulation won't magically solve the issues unique to software. Maybe it's doable with vastly increased costs (a classic would be running fully redundant 1:1 warm failovers, which nobody could economically do at the cloud provider level.)
That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.
Electrical engineering of the power grid has the exact same failure mode: cascading failures of interconnected power grids can cause loads to increase by factor of a million in a few seconds.
I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.
Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, which would have provided ample time to intervene. However, a race condition caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.
Also note the power companies have clear procedures to reboot the grid, known as a "black start[up]". And they test these procedures, though without taking down the grid.
I found out they have prepared, in advance, a list of which disconnects have to be open at subatations in order for power stations to supply each other's startup needs (for stations that are not capable of unpowered startup) while supplying no or minimal actual load. There must be a software analogy to this. Does your company have a plan to bring up the system on a rackful of blank servers and a database backup? I doubt it.
Which is the kind of exceptionalism that really creates these problems. Software is the only industry that allows people untrained in running global infrastructure to run global infrastructure.
Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...
I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.
The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.
The other option is to not outsource everything to the cloud. Yeah, it is more convenient ... but in these times, being a bit more self reliant is not a bad thing.
In these times, paying more for that is not an option. Servers with lots of RAM are 5 times what they cost a couple of years ago. Cloud is a huge bargain today compared to running your own infrastructure.
The problem is that you are only as strong as your weakest link. At some point, something somewhere will be architected poorly and it will one day line up perfectly with 2 other bugs that were incredibly hard to predict and then you're stuck in Microsoft's current situation.
Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.
This is the same as what's been happening with entire industries: Build for the critical path. Eliminate redundancies. Eliminate resiliency. Lean staffing. Just-in-time manufacturing. Single-vendor mandates.
In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.
And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.
So what's your answer? Just ask the giant megacorporations siphoning billions of dollars a year into the pockets of oligarchs to pretty please stop ruining things for everyone else while they enrich themselves?
This attitude of "regulations don't work" "taxes will just cause wealth to flee" "increasing the minimum wage will just be inflated away" is all propaganda. Propaganda meant to make us believe all we can do is give in to the megacorporations and the oligarchs who lead them, and let them do whatever they want.
We learned the hard way that there's a very non-obvious inflection point in large distributed systems, where the your design needs to change from accommodating load resiliently via queueing, auto scaling, and such, to a posture of actively scaling back certain features and even shedding load when in a defensive situation.
It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.
How did that AWS DNS incident go recently? Unplanned outages are unfortunate but common, and hubris is the problem when everyone assumes that they (or their pet vendor) are immune.
Except for the fact that it seems like most Microsoft outages are self owns. I'm not sure how many times we've had outages due to bad deployments that somehow made it to their global infrastructure and only got noticed when customers notified.
Plenty of internal accounts from previous employees highlight this.
To be fair to Microsoft (and I'm having trouble believing I just wrote that), the vast majority of everybody's outages are "self owns". If you don't change anything, you're not going to break anything, and major external events (fires, cable cuts etc) are rare and can usually be worked around.
Many of us insist upon not building Rube Goldberg machines that cannot be reasoned about precisely for this reason.
Most of the time we get demonized by resume-driven developers job-hopping every few years (seems like the lack of deep expertise is working out). Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.
There are methods for building complex systems our of smaller pieces. Humanity has done this forever. An internal combustion engine is very complex. An airplane is very complex. You put a boundary around your system, you build it to a spec, you test it to a spec, you define the behaviour across anything you can see. When two engines fail on the airplane it can maybe still at least glide. Maybe there's an ejection seat. Because someone is doing the system thinking with some level of discipline.
A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.
In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.
Software systems are by nature more complex than physical systems. They also lack any tangible interfaces beyond code and diagrams and collective knowledge in the minds of the humans who build and maintain them.. The rapid evolution of software is no help either..if anything AI is accelerating that rapidity as well.
Newest update says that wasn't the issue after all. They say relatively small loads are actually causing servers to become unresponsive and have to be restarted.
They still don't know why this "core process" is maxing out the processor for no reason.
Could be something adjacent - the last time I majorly blew up a release was because something had changed (which was fine) but it caused a list of items to be broken over two pages, but only in prod. Me and a team mate were scratching our heads for the longest time because "what had changed" was quite mundane itself.
The classic 'database is smaller in testing than in prod' problem which is responsible for some outages IME.
Often the 'query was not optimised but that didn't show until prod' one.
They did look at recent changes but none seemed to be related, but it turned out recent network changes triggered a failure mode in a much older network change. Dependent or cascading issues remain hard to diagnose or find.
There was an article posted by a former MS / Azure person earlier this year that described the dumpster fire of node management and their cloud control plane
To be fair: I want the people who can speak most competently to a problem to be working on it versus writing commentary. Those highly-technical people can shine in the postmortem report.
I disagree; the communication aspects - including external - are a hugely important part of any incident response. IME the biggest challenges in these situations are the many levels & combinations of communication and coordination. You don't need your best DevOps person doing it, but there should be plenty of competent technical product managers or engineering managers who could do this well.
And how do those product or engineering managers get information on what’s being tried and what is being discovered without slowing down the process of fixing the issue?
Microsoft is not a small startup where the CTO is getting pulled into firefighting.
They should have established procedures for large-scale events, and they must include a coordinator whose entire job is to manage the external and internal communication.
And that is a good thing. Well, maybe not good. But the last thing you need in a crisis is to pull people away from fixing the problem and make them type up status updates. Admittedly, having tech knowledge would help the writers - a technical incident management team would be better. But ain't nothing wrong with letting the PR folks write up words while the tech folk work on problem solving.
This sounds suspiciously what occurred when I was working at a large motor sports company. We had a major release of new year models coming out on the website. The deployment went super smooth and then in about 30 mins the site was crashing. Since we were a huge MS client, almost everything we ran was all MS top to bottom.
Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.
CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.
This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.
I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.
The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.
I can't stand it when I see armchair analysts making claims like this.
I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour
What would it take for it to be professional hour?
Professional hour would be invisible. Outages don't happen because they were anticipated and prevented.
You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.
Of course keeping services online is professional hour.
Keeping important things working isn't not a high bar, however inevitably it will go down and the ability for a well experienced team to identify a complex issue and resolve it is also part of their job and is what professionals do.
I am worked with various technical incident response teams, some are people with their arms up, some of them are good displays of technical knowledge and communication.
Professional hour applies when things are online and when things go down.
I'm not thrilled with what they've done to Windows lately myself, but this is just rude.
You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.
Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.
Individual talent is irrelevant when you have a company culture that rewards the opposite of that. Any talented person working at Microsoft knows that they are dealing with structural problem that force MS products to be massive piles of shit. There isn't much an talented individual can do about that.
An organization can become fundamentally opposed to the purpose for which it was created. Organizations simply don't follow the same laws as the individuals inside it do. A company can be made of geniuses and be totally ineffective; it can be full of talented go-getters and be completely unable to fix problems.
Lots of nines means they have SLAs of no significant downtime. It does not guarantee that the SLAs can never be missed, just that there will be consequences when it is missed. Lawyers and accountants and sales folks will sort out the compensation payments that come back to the customers and try to negotiate relationships back to a healthy state.
That’s purely a choice by those claiming the nines. I’ve worked in companies selling software with more than 6 nines uptime (so that we can be more confident about the 5 nines). Not 4 nines uptime plus some money back for the other one.
My housemate texted me saying our internet is so slow because outlook wasn't loading. So I think lots of people associate email not working with entire internet not working..
With how many people are unable to associate a "broken" monitor with an unplugged HDMI/DisplayPort or power cable, it doesn't surprise me that most people would not make a disctinction between the "internet" and "Outlook".
It's another Jev copy, like we've seen so many over the last few weeks. But with no benchmarks or price comparison, which likely means it doesn't compare that well.
> But with no benchmarks or price comparison, which likely means it doesn't compare that well.
Doubt that's the case; this was only the announcement and the API isn't even released yet. As I noted, my own testing using a batching strategy (nothing more than "read 20 lines from this file instead of reading 1 at a time") yielded ~the same accuracy on an 800 row dataset we have with ~1.6x the throughput (at only batch_size=20) and 1.2x the cost. Didn't tune to find accuracy dropoff by batch size, but it's easy to see that the decision capability was there (understood that it's not the same without the probabilities, but even without it, the classifications were nearly identical).
IOS27 has a killer feature that will boost sales for duo. Iphone handoff. Don't underestimate how deep apple customers wallets are. You now can have multiple iPhones. A pro that you use day to day, an Air that you use when you go to the gym and a duo to use when at home. Now the crease lasts longer because you are not using it 24/7, you spread wear across multiple devices and pump the apple sales.
Why carry a phone at all? Just buy one and leave it at all the places you go. Have your personal security carry a backup in case you go somewhere your assistant hasn’t been able to get a phone delivered to yet.
Supabase is kind of similar to Firebase in that you usually communicate directly with the browser through a JavaScript lib. So you need to pay close attention to the security rules (RLS policy in Supabase's case) to restrict data per row based on the user auth.
It's easy to get it wrong; that's why so many Firebase and Supabase databases end up exposed. The platforms have improved the UX so that it's not so common anymore, but it's definitely a gun that's easy to shoot yourself with.
You don't have to expose it like that. You can disable that client and use a server to communicate with the database, which reduces the exposed surface in exchange for complexity (by even having a backend at all).
Mix that design with users with zero tech background who know nothing about systems design and LLMs, and you have a recipe for disaster.
We use Lovable as a prototyping tool and then eject it, review, and deploy on a different platform. It works great for that use case.
reply