This is certainly a great immediately useful tool but also a relatively small ROI, both the return and the investment. Big tech is aiming for a much bigger return on a clearly bigger investment. That’s going to potentially look like a lot of useless stuff in the meantime. Also, if it wasn’t for big tech and big investments, there wouldn’t even be these tools / models at this level of sophistication for others to be using for applications like this one.
While the press lumps it all together as "AI", you have to differentiate LLMs (driven by big tech and big money) from unrelated image/video types of generative models and approaches like diffusion, NeRF, Gaussian splatting, etc, which have their roots in academia.
Not at all - Transformer was invented by a bunch of former Google employees (while at Google), primarily Jakob Uszkoreit and Noam Shazeer. Of course as with anything it builds on what had gone before, but it's really quite a novel architecture.
The scientific impact of the transformer paper is large, but in my opinion the novelty is vastly overstated. The primary novelty is adapting the (already existing) dot-product attention mechanism to be multi-headed. And frankly, the single-head -> multi-head evolution wasn't particularly novel -- it's the same trick the computer vision community applied to convolutions 5 years earlier, yielding the widely-adopted grouped convolution. The lasting contribution of the Transformer paper is really just ordering the existing architectural primitives (attention layers, feedforward layers, normalization, residuals) in a nice, reusable block. In my opinion, the most impactful contributions in the lineage of modern attention-based LLMs are the introduction of dot-product attention (Bahdanau et al, 2015) and the first attention-based sequence-to-sequence model (Graves, 2013). Both of these are from academic labs.
As a side note, a similar phenomenon occurred with the Adam optimizer, where the ratio of public/scientific attribution to novelty is disproportionately large (the Adam optimizer is very minor modification of the RMSProp + momentum optimization algorithm presented in the same Graves, 2013 paper mentioned above)
I think the most novel part of it, and where a lot of the power comes from, is in the key based attention, which then operationally gives rise to the emergence of induction heads (whereby pair of adjacent layers coordinate to provide a powerful context lookup and copy mechanism).
The reusable/stackable block is of course a key part of the design since the key insight was that language is as much hierarchical as sequential, and can therefore be processed in parallel (not in sequence) with a hierarchical stack of layers that each use the key-based lookup mechanism to access other tokens whether based on position or not.
In any case, if you look at the seq2seq architectures than preceded it, it's hard to claim that the Transformer is really based-on/evolved-from any of them (especially prevailing recurrent approaches), notwithstanding that it obviously leveraged the concept of attention.
I find the developmental history of the Transformer interesting, and wish more had been documented about it. It seems from interview with Uszkoreit that the idea of parallel language processing based on an hierarchical design using self-attention was his, but that he was personally unable to realize this idea in a way that beat other contemporary approaches. Noam Shazeer was the one who then took the idea and realized it in the the form that would eventually become the Transformer, but it seems there was some degree of throw the kitchen sink at it and then a later ablation process to minimize the design. What would be interesting to know would be an honest assessment of how much of the final design was inspiration and how much experimentation. It's hard to imagine that Shazeer anticipated the emergence of induction heads when this model was trained at sufficient scale, so the architecture does seem to at least partly be an a accidental discovery, and more than the next generation seq2seq model that it seems to have been conceived as.
Key-based attention is not attributable to the Transformer paper. First paper I can find where keys, queries, and values are distinct matrices is https://arxiv.org/abs/1703.03906, described at the end of section 2. The authors of the Transformer paper are very clear in how they describe their contribution to the attention formulation, writing "Dot-product attention is identical to our algorithm, except for the scaling factor". I think it's fair to state that multi-head is the paper's only substantial contribution to the design of attention mechanisms.
I think you're overestimating the degree to which this type of research is motivated by big-picture, top-down thinking. In reality, it's a bunch of empirically-driven, in-the-weeds experiments that guide a very local search in a intractably large search space. I can just about guarantee the process went something like this:
- The authors begin with an architecture similar to the current SOTA, which was a mix of recurrent layers and attention
- The authors realize that they can replace some of the recurrent layers with attention layers, and performance is equal or better. It's also way faster, so they try to replace as many recurrent layers as possible.
- They realize that if they remove all the recurrent layers, the model sucks. They're smart people and they quickly realize this is because the attention-only model is invariant to sequence order. They add positional encodings to compensate for this.
- They keep iterating on the architecture design, incorporating best-practices from the computer vision community such as normalization and residual connections, resulting in the now-famous Transformer block.
At no point is any stroke of genius required to get from the prior SOTA to the Transformer. It's the type of discovery that follows so naturally from an empirically-driven approach to research that it feels all but inevitable.
At least, the Transformer didn't. The abstract idea of a language model goes way back though within the field of linguistics, and people were building simplistic "N-gram" models before ever using neural nets, then using other types of neural net such as LSTMs and CNNs(!) before Google invented the Transformer (primarily with the goal of fully utilizing the parallelism available from GPUs - which couldn't be done with a recurrent model like LSTM).
On the plus side, for Adobe, is that they have a fairly stable & predictable SaaS revenue stream so as long as their R&D and product hosting costs don't exceed their subscription base, they're ok. This is wildly different from -- for example -- the hyperscalers, who have to build and invest far in advance of a market [for new services especially].
I'm unable to read the article due to the paywall, but if the first paragraph is indicative of the thesis, then it is "startups that build incremental functionality on top of existing products are upset when the company improves their product with the same functionality". I'd say there are two issues at play here:
1. The large Tech companies are too big and do too many things. They should be broken up for no other reason than to protect the public from their eventual collapse or abuse of power.
2. As a startup, don't build an incremental product that can easily be scooped by a competitor, especially if that competitor is a mega corporation.
While hemoglobin in blood has magnetic properties there’s no evidence that it leads to anyone being able to feel magnetic fields. At least one study has shown evidence for humans sensing magnetic fields but the author's do not make a case for blood being the biophysical mechanism https://authors.library.caltech.edu/90480/. The earths magnetic field is very weak and the force felt on our blood is even weaker.
It’s still a debate how animals sense the earths magnetic field. However, all theories rely on special adaptions and not on the extremely weak magnetic properties of blood.
Particles of magnetite in or between cells are a much more likely mechanism.
There are languages where grammar requires all directions are absolute, and even three-year-olds speaking those languages never get them wrong. I don't know of any studies to show they really are referenced to magnetic north, or are misled by artificial magnetic fields.
There is probably an easy PhD in showing such an effect.
SparkMeter is hiring! We're looking for a Systems Engineer, a Customer Operations Associate, a Commercial Operations Associate, a Business Administration Intern, and a Commercial Operations Intern.
We developed our low-cost, pre-pay enabled, smart metering system as a solution for electric utilities in developing countries to serve low-income customers. Our metering system is being widely adopted by utilities serving customers in rural parts of Asia, Sub-Saharan Africa, and Latin America and the Caribbean.
SparkMeter is a great group of smart people with a mission. Message me if you want to chat about these roles!
The article mentions that it's a problem to ride the elevator past the top or bottom floor, but it is unclear why. I don't see why this should be an issue.
I once accidentally took a tour over the top floor. I read warning signs on the wall that told me to keep all limbs and especially hands in the cabin. Then it became rather dark since there was no light bulb in the cabin. After a short while, when my eyes got adjusted to the darkness and a bit further up I could see why the warnings were there: I could see the whole mechanism that drags the chain of cabins around. Big cogwheels only centimeters away from the door. I kept my distance from the door opening and I arrived at the top floor again where I got off.
Thanks to the warning signs I wasn't really afraid. They made me realize that this sort of thing likely happens quite regularly.
Riding over the top allows a form of queue jumping. If you are near the top and there is a queue to go down, take the ride up and over the top. You'll already be in the car when it gets back to the same level to service the down-queue.
I used to do this in the Paternoster in the Roger Stevens building at the Uni Of Leeds in the 1980s.
Imagine you’re in a building with 22 floors, and your event with a few hundred people just ended on the 18th floor. The elevator has three bays with cars that only carry about 20 people at a time.
While you’re in the crowd waiting to go down, you hear elevator cars race to the top of the building, then when they come back down, the doors open and you’re met by a car that is already packed with people. A lot of the passengers are avoiding making eye contact, and wait a second, some of them look like people you just saw leaving by the stairs in the fire escape!
Apparently, the elevator algorithm is trying to be efficient by starting at the highest floor with a request. Do you wait where you are, take the fire escape up one floor, two floors, or just walk down 18 floors moderately disgusted with the situation?
You probably don't even need to take the stairs. Just press the up button and get on the next up elevator while everyone else waits for down. Then wait for it to reverse direction and ride it down.
You are right, but in practice you have to deal with the repercussions of inspiring leaders among the waiting crowd to police your actions. Going down to the 17th or 16th floor and catching the “up” elevator might not get as many glares.
Where I’ve seen this happen is after speaking events in the university clubs near Grand Central Station in New York. The complete dynamics of the situation probably involve a minor risk to social reputation, and some people playfully trying out game theory to amuse their friends.
The one I know has several signs on the exposed inside walls above the topmost floor all assuring you that you will not be turned upside down, or folded, or run through a meat grinder.
Based on the adage that every (non-)warning sign is there for a reason, I can only assume that there have been incidents of people panicking after they realized they had gone to far.
Just to be clear: it’s absolutely safe to go around. Any other stories you may hear are the inside joke of those five beaurocrats on the top floor enjoying their tiny fiefdom of chaos.
Edit: could people downvoting this please give a reason? I’m really not complaining, just terribly curious what could possibly stroke people the wrong way in this benign post. Or is this the Paternoster conspiracy coordinating to keep their secret?
Just to be clear: it’s absolutely safe to go around.
There is a variation which isn't - the "man lift".[1] Those are dangerous, and sometimes seen in parking garages for staff only.
Those really do turn upside down at the top. They're usually equipped with a switch to shut them down if someone rides past the top landing.
I rode one the other weekend in Prague with my girlfriend, and we agreed to ride it over the top together. Despite doing the same thing 20 years ago when last there, it’s pretty scary - on the last floor there is a big sign saying ‘get out now!!!’ in big red letters, and it is much noiser and shaky as you are close to all the machinery. We held on, proper dose of adrenaline, as the cabin is plunched into rattling darnkess, and the clanking gets even louder. You then get a bit of light and see the big exposed cog wheel and other machinery, move sideways and eventually down. It’s a major relief when you arrive at the top floor again. Great fun!
Apparently troublemakers like to wait until their friends are in the space past the last floor and trip the safety switch, leaving them trapped in darkness. Also, some of the more temperamental installations may not like the extra weight shifting around.
The cabin becomes upside down as it loops around at the top or bottom, i.e, if some one fails to get out at the top floor, they will be tossed around as the the lift loops around to the bottom floor, so I guess.
Even in the care-free days of the early 20th century I don't think a device that would casually throw you around like that would be open to the public!
I remember reading that as a practical joke, people would ride over the top, then stand on their heads before the cabin came back down to the top floor, so bystanders would think it had turned over.
When working in Deutsche Bank in the '90s I frequently used one of these in the building and had the same question (but no Wikipedia). We put a piece of tape on the floor of a car that was going up and saw it on the floor when that car was going down. (Note this was necessary because the cabins were completely carpeted and there was no discernable wear).
There were also no signs stating that you shouldn't ride over the top ... so we did and probably would have even if they flipped.
Yes. But true in the sense that anything computable can be computed by a Turing machine. sets are universal building blocks but most of the time you want a higher level abstraction.
It isn't so simple, unfortunately. The idea behind set theory was to describe some universal building blocks of mathematics. The problem is that - unlike in the case of computable functions - such a universal building block can't exist.
There's a lot of mathematics which can't be formalized in classical first-order logic with the ZFC axioms. For most things you can get around this by adding enough Grothendieck universes, but this is still incomplete and always will be.
According to who? I don't think that's a very good analogy. I think it's better to say type theory is just another language in a 'family of assembly languages'. Its superiority is very debatable and not commonly accepted as a foundation.
I think it's a nice analogy. It's a bit tongue in cheek, but it does give you the right idea when you compare it to programming languages.
Set theory is a reductionistic system. It's supposed to give a foundation which is as "small" as possible.
In set theory you start with almost nothing and you have to encode all useful concepts into sets before you can start proving/programming.
Compare this to Turing machines, where you start with a very primitive model of computation. The first step in most treatments of Turing machines is similar to set theory, you show that some useful concepts (associative memory, high-level control flow, etc.) can be encoded into Turing machines. You then use this encoding to construct arbitrary programs.
Type theory starts out with a functional programming language and you derive new concepts from free constructions. In type theory you shouldn't be encoding things at all. You should be describing concepts (~adding an API) and possibly extending your model (~the compiler).
As analogies go, I'd say that this one is pretty accurate.
Unquestioning acceptance of science is definitely not a conclusion of this piece nor is it alluded to.
The main argument of this piece is how "an understanding of what real truth-seeking looks like" can allow you to reject claims when they are scientifically proven to be false. While he does not directly state that one should be skeptical of the scientific community he also never states that there should be unquestioning acceptance of it either. It is however made clear that skepticism is a key trait of being a scientist.
He begins by pointing out the seeming contradiction of being a scientist who is "supposed to have skepticism and imagination, but not too much" while "gathering facts and testing your predictions" before you "either affirm or reject the ideas at hand". Even then you still must "accept that nothing is ever completely settled, that all knowledge is just probable knowledge." Establishing early on that a scientist must be willing to accept that "a contradictory piece of evidence can always emerge" while still advancing our collective understanding.
"Knowledge has become too vast and complex for any one person [...] to convincingly master more than corners of it". You therefore must rely on the collective of scientific knowledge and those who practice it, the "scientific community". He points out the difference between this group and one of pseudoscientific thought is that the claims of the latter can be demonstrably rejected using the scientific method.
A scientist must remain skeptical, but in order to be productive you also need to rely on your community, no one person can verify all claims. Being skeptical is inherit in being a scientist, and therefore part of the scientific community. Relying on the scientific community is not akin to unquestioning acceptance. Questioning established beliefs while backing it with scientific evidence is the key difference between the scientific community and the pseudoscientific one.
Speculation: It appears they are using this for the overarching structure of projects. Rather than managing individual appointments it manages overall time, e.g. on this day BobbySue should be spending 50% of her time on project A and 50% on project B. Then you can make the calendar read only from the digital side.
That being said, I would still like to see a pick-n-place managed version of this that stays in sync with google calendar.
99.6% is the method-effectiveness (i.e. the efficacy if used properly) or 0.4 unintended pregnancies per 100 "women years" (13 cycles)
Other interesting results from the study:
- the method-effectiveness rate found for this method is comparable to oral contraceptive
- 9.2% stopped using the method due to dissatisfaction
- Couples that had intercourse during the fertile period had an increased pregnancy rate of 7.5%
- This study was done in European countries and the pregnancy rate was lower than similar studies performed in developing countries
