Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.
Somewhat unclear how particularly novel this is vs a way to save compute.
Iroh's discovery is genuinely neat, and node-IDs-as-public-keys is similar to bitbang.
A browser can't join a DHT though because of no UDP sockets, so the connecting side would need a native client, which is the thing bitbang avoids. And DHT lets you locate the peer, but you still need somewhere to trade SDP and ICE candidates, so the rendezvous stays either way I think.
What I can do is keep the broker small enough that self-hosting is realistic, which is where it is now -- no accounts, no registry, nothing that grants access, only a simple broker.
This tracks with what I’ve heard around as well. What changed in the financial instruments?
My understanding is a lot of the loans have gone PIK or otherwise essentially aren’t serviceable at current prices. Do you think that’s resolvable somehow or just lagging implosion?
Nothing changed in financial instruments - we just don't usually have a lot of office tenants decide to walk away like we have with the advent of remote work. The loans will work themselves out. We've seen a couple of buildings trade for way less than pre-Covid numbers, and we'll see more.
In another 10 years downtown Seattle will be aligned with the rest of the market again.
Anisotropy in word embeddings dates back to at least 2017 with word2vec - where there were zero layers.
The cone-shaped anisotropy in transformers is known since at least Gao et al. 2019. That lineage explained it fairly intuitively as an artifact of word frequency and softmax geometry (so a training dynamic).
A variety of papers followed up by adding post-hoc ‘whitening’ steps (from classical statistics/NLP), then adding regularizers to the loss to penalize the anisotropy, eventually penalizing the covariance matrix (a la VICReg), and then the SIGReg method as a computationally much cheaper way to approximate the full covariance.
As another commenter pointed out it’s also similar to the InfoNCE/contrastive learning objectives. Where terms to increase uniformity (spread out evenly) on the hyper sphere were added. Like the SimCSE (Gao 2021) paper or the excellent alignment/uniformity breakdown from Wang & Isola 2020.
This proposed dispersion loss seems to be similar in that it pushes things apart by penalizing cosine similarity. Although this one works on the tokens within one sequence. Usually contrastive methods mean pool the sequences and then contrast against the other pooled sequences in the batch.
> 1 … they will often chose the more compressed/louder one
I’ve always been curious - but presumably that’s true even after volume matching?
> 3 Compressed sound works better in noisy environments and as background music
I’ve heard this is also why film and video game soundtracks are often very compressed, even when orchestral, because they have to fit in the background with dialog/sfx
> I've always been curious - but presumably that’s true even after volume matching?
Yes. Many musicians want their stuff to sound like the music of their heroes they grew up with, and that music is often compressed to a block as well. So compression isn't just about making things sound louder, it also has its own aesthetical value. Whether that is good or bad aesthetics can be argued about, but some people also like to distort their instruments which was also a thing people frowned upon in the past.
> I’ve heard this is also why film and video game soundtracks are often very compressed, even when orchestral, because they have to fit in the background with dialog/sfx
The official themes often are quite compact, but there is often also highly dynamic orchestral work used that is way less recognizable and used with more dynamics (think about te soon creating orchestral atmospheres). Cinema mixes are a thing btw. where many consumers complain about too high dynamic ranges. They complain that the dialog is low and the explosion loud. Cinemas being among the few spaces we mixing engineers have where we have a bit more control over the presumed levels, especially if we are talking about Dolby certified venues.
Ah, I hadn’t thought about the generational aspect that’s interesting. The aesthetic totally makes sense to me when the music is intended for it / designed with it in mind, which I guess quite a lot of music is.
I particularly dislike when old intentionally-dynamic music is remastered to be “modernized” into a brick, which is sort of the opposite direction.
> Cinema mixes
I didn’t know about these, that’s neat! Makes sense that the levels can’t really be the same in my living room as a theater. Is it really a whole separate mix or just some compression in mastering?
I really hope that’s not another masterings collection rabbit hole I’m about to fall down haha. I’ll look out for some Dolby certified venues in my area too
I've been to IMAX cinemas where the volume was so loud my ears physically hurt
I understand them, they want to shake you in the seat, to make it an experience (unlike watching at home), but it's ridiculous I have to consider bringing earplugs
ReplayGain is nice - but note it doesn’t ’fix the compression’. Compression and dynamic range is about loud/quiet _within_ the track. ReplayGain just turns the volume up and down for the entire track, the point being so all your tracks play back at about the same level. It saves a preset on the volume knob for you essentially.
If you remember making a playlist where one song is suddenly much louder than the last, and you’re riding the volume knob on every other song, you’ll see why this is nice!
It really does on some records, if you’re interested check out some comparisons on YouTube. Many times it’s subtle eq tweaks, granted, and that won’t much matter. But a lot of older rock and pop records for example go from being super dynamic and well produced to completely crushed with boosted bass and treble to ‘modernize’ the sound.
You can see some examples of how dynamic range (they don’t track ‘mastering’ overall) varies across releases on this site: https://dr.loudness-war.info/
Sorry, but you don’t know what you’re talking about. Many releases get mastered separately for digital and for vinyl, and one or both of them often does “change much”. Usually the brickwall limiting (among other things) on the digital master.
I absolutely know what I'm talking about and have released multiple records. I don't think you know what mastering is and how subtle it is. You're thinking of a separate mix which, yes, does sound quite different.
Just because you chose to have subtle mastering applied doesn’t mean that the commercial records we’re discussing have done the same, particularly for their digital releases, which is why they sound different. If you listen to vinyl versus digital releases of certain albums, it is obvious. Stop spreading misinformation.
Somewhat unclear how particularly novel this is vs a way to save compute.
reply