This sounds more like a culture problem than a title problem though. Are you saying you can't even open PRs with a pipeline proposal? Your data engineering team is playing emperor and wants to protect their kingdom at all cost
You can't adequately clean some data though. That your psp had a hickup and only processed 50% of orders on Christmas Eve can't be modeled away. That's where you'd need a context layer to explain why it happened
I guess as long as it gets into the context layer it's all good! I'm excited, but skeptical, for those company OS agents that will be able to figure this out
I think there are big risks with just labelling it as "wild west" and let users do whatever they want there.
It's always been possible to massage your data to fit whatwver case you want to make. Previously you had to go through a data analyst and convince them to let go of their principles. Now you can just let an agent figure it out.
So how do you make sure insights are actually shareable (and trustable) with other parts of the company?
I think just exposing the metrics layer is probably the answer here.
It's kind of funny that the industry has tried to democratize data for so long, but now we scramble to restrict access to protect users from making incorrect decisions.
> I think just exposing the metrics layer is probably the answer here.
This is the short answer. Metadata has taken on a whole new level of importance and if your model isn't tight you're going to have a bad time in a lot of new and exciting ways that weren't exposed in a more traditional data stack.
I have actually spent a lot of time on this. But it was for an old version of my flowchart editor that did full relayout on every edit. I tried many variations, including a fork I made of dagrejs to shoehorn it into discrete coordinates.
In the end I decided to go a different route. The current version only does incremental relayout — it tries to move only a small number of nodes to make space for the current edit. And I wrote a custom algorithm that shifts nodes out of the way to make space on edits, and trys best effort to keep edges in the same direction by detecting which got bent and iteratively moving one of their nodes.
So far I don't think very much, except for it being a canvas and pasting images.
But I want this to become a "Bicycle for the mind" for data analysis, where you can move data by different gestures. Like circling an area in a scatter plot to examine those points, or dragging a bar from a bar chart onto the canvas.
My thesis is that humans are better than agents at pattern recognition, so I want to optimize for the "looking at data" part of analytics and make it as easy as and intuitive as possible
It is a full-featured reporting tool, and can be used without AI. If you do use AI it has a semantic layer to help AI understand your table and column names, and relationships between tables.
Pretty cool! I'm not swiss, but I've heard of a Nordic equivalent. I think its Hudd, uses BankID which I guess is the same as SwissID.
I personally would never sign up for a service like that though. Mostly a numbers problem. Smaller countries means fewer people with my niche interests. Better to join a discord community around it that allows a global audience
LLMs are also good for this; point them at your repo and ask them to do a fresh install and note all friction. One time when the lack of context really helps.
reply