Hacker Newsnew | past | comments | ask | show | jobs | submit | Kaedon's commentslogin

At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?

I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.


In my past experience in IT, the ticketing system basically doubled as an aggregated event log, and it was a legitimately good use case. The monitoring software we ran on our thousands of client workstations and servers was extremely powerful and could capture any conceivable metric to use as a trigger for auto-generating a ticket.

So, a lot of tickets generated would simply just be data reports with some info dumped in the ticket description, designed to be auto-closed but possibly useful later as an easily-accessible timestamped reference to some data.

Lots of other auto-generated tickets were "probably not an issue but it should pass over the eyes of someone knowledgable about this server/network" tickets. Like Windows Event Log error roundups, CPU usage spikes, or specific triggers that were set up in the aftermath of some catastrophe to help preemptively avoid a repeat incident.

Anyway, tickets are a great "first line of filtration" where you can take a massive ingest of all different kinds of data (hardware metrics, system logs, network activity), apply meaningful human context ("This pattern could be a sign of xyz"), and store it in a unified database. With that use case, it's easy to imagine a company with millions of customers and millions of servers producing millions of tickets.


I've never seen a ticket system used as just a dumping ground for data in auto-closed tickets. Seems like there would be better solutions for that, particulary for data you want to find later (in my experience, the search capability of ticket systems varies from poor to "don't bother, you won't find what you're looking for")

Anything else, that needs a human to interpret or take action on, yes, ticket systems get extremely wide usage.


Right, I didn't mean using tickets as a generic event log—the assumption is that someone would have set up triggers because they actually have a reason to have the data on hand, and it's a low-friction way of getting what they need without worrying about implementing new tools or systems. In a good human system, the trigger would be disabled once that info is no longer useful. (And in a bad human system it would just stick around until nobody knows what it's for anymore, haha.)

We also had tickets that would trigger scripts upon generation, e.g. an auto-generated "low disk space" ticket would then trigger a basic "disk cleanup" script, and close itself if enough space was freed to clear the threshold. Yes, it's comical that the ticket would do that—but hell, it worked great. It ran agnostically across thousands of differently-configured machines. And whether or not it was a human who clicked the "disk cleanup" button shouldn't determine the usefulness of having that event logged in the ticket database.

Point is, the whole system was a veritable Swiss Army knife that didn't necessarily have to equate a "ticket" (database entry) to "requires human intervention", provided there was a good human system in place to leverage the tool in a way that was productive for them.


That makes sense, but a part of me is wary of introducing noise at such a scale and drowning out whatever signal remains.

Also ticketing systems in my (very) limited experience tend to be bloated pieces of software, so that adds a fun dimension to it.


Indeed, I learned pretty quickly that maintaining the line between "underreported information" and "information overload" is an indefinite challenge that evolves over time as context changes.

But the human team is what understands the big-picture context and the way it changes over time, so it makes sense for them to manage the triggers for the "this may be important" tickets—for them to then ingest and discern whether it's important. Ultimately, ticketing software is just a tool designed to distribute context-stamped information to humans who then discern what to do with it.

So I think just like all other productivity software, it's only as good as the human system in place that drives its usage. That explains the bloat as well—it's not its own system, it's a big old suite of tools that are each available to be implemented into a human system if desired, or ignored if not.


Yes, easily. I work for a "small" org in Amazon and it has more than 100,000 tickets a month. And it's a small fish in terms of number of tickets. As another commenter pointed out, tickets are used for basically everything administrative.

I checked just for fun and there are actually a LOT LOT more than 92M tickets in the past year.


When you’re enterprise size, every request needs a ticket otherwise it didn’t happen

Exactly. For folks working at smaller companies, think of it as "if you'd Slack someone to ask them to do something, that's a ticket" just purely because Slack doesn't scale to dozens of people needing to figure out which person even needs to get messaged in a given minute of the day.

It's that, and you also want audit trail. But there's also processes for the sake of processes at Enterprise so a good % of these tickets aren't really needed but you need to file them to follow the process.

shared communications like slack also implies the same team, or at least the same company.

but i have 3 different support organizations, all of which are external. no ticket == no accountability, and that matters if and when things get ugly, like lawsuit territory level ugly


The amazon ticketing system used to run over email, and the exchange server team spent basically all their time trying to keep up with the scale...

yes, there's a ticket for EVERYTHING. people have tickets cut in their names even decades after they've left

In some big companied many core corporate teams which hate that level of transparency and build their own opaque systems. I've seen this for departments like IT, HR, legal, etc.

I'd rather have tickets for everything and full transparency except for confidential info.


Im a huge believer in transparency. But I also understand why, in a big company, people don’t like it. It’s very easy for metrics to be weaponised by political players who are out for themselves.

In our group, Jira exists because people were not reporting problems so they could use our team as a scapegoat for delays.

In my experience, corporate bureaucracy is a slowly grown defense mechanism created from an attempt to stop dumbasses and assholes.


That can happen anyway. Also if the other side is open, guess what, someone with a bit of patience can probably dig enough to find some dirt on them.

M.A.D.


At my company we also do the same every request to engineering (alarms, security, bug, customer support escalation) gets a ticket.

i mean there's even tickets cut if you need IT to fix a laptop or assign you a locker or unlock the code on your desk ped.

heck, there's even a ticket if the office toilet is clogged


I once had to raise a ticket for a double AA battery for my mouse. This sort of stuff frustrated me enough to start my own company.

What does your company use for ticketing?

Just spreadsheets for the time being. I think irrespective of the tool it's more the mindset for me. What is too small to have meticulously documented versus a general bucket. Also what becomes too big to not track on a ticket anymore but via project governance.

SqueakDesk

Ah! You must be that energy harvesting mouse company.

Were there tickets for tickets too?

Yes, plus let's close this ticket and open a new one so that the KPIs look solid.

this 100%

If it works it ain't stupid. Big companies desperately need traceability and ownership. Way too many bullshitters playing politics, being loud on meetings etc but very little actual output.

It really helps. Tickets are cheap and persistent. Breezing by my office or shooting me a DM is cheap and ephemeral. I’m just going to respond with “do me a favor and put that in a ticket thanks” anyway

Perhaps killing the need for tickets will be the real value unlock of llms. (Kidding! Kind of)

essentially servicenow's business model.

Well there is the UFO/Bigfoot/Poltergeist crowd.

If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded. If your Boss belongs to that club oh man god say the division.


it is a corporate mantra many of us want to forget. it also functions as a response to any request.

“submit a ticket!”


Nah, I like issue trackers. They solve a fundamental problem that can't be avoided: a searchable one source of truth for problems, their context, their work, need-by dates, and their resolution, and a record of all persons/branches/files/etc involved. Something like Slack is not a reasonable replacement for this, unless you make a bot that just duplicates all the functionality of an issue tracker (at which point it becomes one).

People that don't like issue trackers are people that like spending hours a week manually maintaining spreadsheets, keynotes, or kanban to do the same.


there’s a difference between tracking work and fueling bureaucracy.

People that don’t see the difference are substituting JIRA and confluence in for spreadsheets and keynotes.


Sorry, but the issue is one source of truth. A keynote cannot be a source of truth, and a spreadsheet can be a very poor one, both requiring manual updates from everyone involved, relying on "comment" fields/notes in both. I've seen it attempted tens of times and work exactly 0.

Jira can make reports, Gantt charts, etc, that you can just paste into your keynotes.

There's also just proper project management tools, like wrike. Luckily, these usually have Jira integrations, to maintain one source of truth.


You’re talking past the complaint.

I will admit that I may not have understand the comment, since it's fairly indirect.

Well if they use VPN point to point connections managed by AWS they'll get an email per week until they die for each VPN instance from AWS that they need to put in trash so I understand them.

based on my time at amazon it does seem low. Perhaps this is only for a subset, like a single VP-level org?

Overall feels right but started happening well before pandemic, back in 2012-2013, lots of shows in Romania, Atlanta, etc.


Yea the pandemic is overfit as a reason for most things. Production had already left over the preceding decades.

Hollywood didn't lose its relevance overnight. It's been a slow moving process as the film -> digital -> portable transition made production cheaper, the internet made talent cultivation/distribution easier, studios and audiences expanded internationally, unions kept ratcheting up rates, LA refused to build homes while population ballooned, QE/ZIRP/0% rates re-inflated a housing bubble, unions kept raising rates to keep up with housing in LA, internet/social content began competing with studio productions hurting demand, the incestuous hollywood-to-traditional media marketing pipeline died (much of old print/tv media existed to market Hollywood stars and products), etc. etc.


This is so AI coded that it’s almost impossible to get through the whole article. “Genuinely useful”, “the skill gap no one predicted”, “the gap wasn’t x it was y”.


You're right, I used AI to help organize the content but we have spoken to almost 50 companies over the last 5 months and the messages and pains they have conveyed are real.

How are you handing distributing your Agent Skills and MCP's? Is this a problem you're seeing?


This site is lousy with ads


DI probably also involves at that time the cost of scanning it into film. I worked in the industry a little after this and some of the DI theatres rent out for 1K+ per hour, plus there’s conform artist time, incorporating new cuts, etc. I could see it.


https://www.kevinlondon.com/

I write most commonly about experiences in big tech, code reviews, and navigating hiring processes


I don't know that I'd want to spend the energy to engage in this kind of behavior.


Why is it that in every post about a problem there's this what-about-ism? We can talk about the issue at hand and then the other ones separately.


At some point 'what-about-ism' must become a good faith comparison.


That’s fair. I’ve probably been reading too much Blind and miscalibrated.


The challenge would be that Prime Video is distributed on many client devices. Here’s a list of a few TV OSes: https://en.m.wikipedia.org/wiki/List_of_smart_TV_platforms. How would you manage a different native app for each of them?

Caveat: previously worked at Prime Video though not in this area, still at Amazon


I could be incorrect, but last time I did some reverse-engineering, Netflix had a Qt application that used native C++ bindings to knock out huge sections of that list.

That Qt application was only lightly modified for each platform, IIRC, and it appeared all over the place from those cheap Linux-based Blu-ray players to Smart TV clients. Anywhere that was embedded, generally Linux-based, and unlikely to get frequent updates.

As for some of the larger players like Roku or Android, those ones are more obvious where to hire. Java developers and BrightScript developers.


It all kind of undermines your original comment tho that they're trying to avoid native. It seems more likely that they're goal is to reach as many devices as they can and be able to do updates frequently. In the first paragraph of the linked article the guy says the trade-off is between "updatability and performance".


I am really sick of fast-churn and forced updates. Guys. Settle. Write a native app or three. It’ll be ok.


As someone who is writing a cross platform app, I'd love to, but I can't.

There are too many different stacks with their own languages (which means different library ecosystems), no good FFI/IPC options that work across stacks, massive test matrix for every little piece of the code etc. All of this, when the native differences that supposedly should make each platform unique are so few, and virtually all UI and application logic is identical. Our platform lords have basically not standardized a single thing.


> Our platform lords have basically not standardized a single thing.

And in addition, you have bugs in official native SDKs and then you end up writing your own text rendering engine, because some OS-level API didn't work properly.


I think it's important not to conflate churn with fast updates. In it's essence the ability to deliver updates quickly (in this case by downloading a blob of wasm+js each time the app is started) is a great thing imo, as you eliminate a huge chunk of problems around delivering updates and clients on old versions. For 99% of users this is a huge win.

You can obviously abuse this by redesigning the UI every 6 weeks but let's not throw the baby out with the bath water.


I'm of the strong opinion that there simply does not exist, as you say, 'a huge chunk of problems around delivering updates and clients on old versions.' It's certainly not a win for users, but for lazy developers. Good software engineers are much more than fast-churn developers. In fact, they are different people.


I'm 100% with you on this, but good software engineers are also much more expensive than fast-churn developers, and it seems to come down to how much spend these companies want to invest in their technology. There is just an overabundance of web developers out there, and the tech companies are taking advantage of that.


I guess we'll have to agree to disagree. I just don't see barriers in getting updates out to people as a good thing.

It seems your argument is that it encourages developers to be sloppy, but in practice I think it just ends up as a net loss for users as the average user just isn't interested in updating software, however critical the bugs in their current version. Maybe I'm different from the average user, but for me the fact that chrome has gone through 90+ versions over 10+ years without me having to think about it is great (and, I suspect, strongly correlated to the relative lack of security issues over the years despite it having a huge surface area).


if they did that then only the most popular platforms would ever be support, happen to use a niche product, or a older product, well your SOL.


... what are you saying?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: