If you don't care about max current performance (frankly you probably don't), then picking up a used M1 pro or M1 max is a much better value proposition. It doesn't seem like the other upgrades are very substantial, maybe Wifi 6E but displays, design, otherwise it's likely to feel essentially the same.
Honestly, unless there is any specific feature added that you find important (the wifi update or whatnot), a refurbushed/discounted m1 pro seems like a no-brainer choice under a more restricted budget.
It's a spec bump, possibly meaningfully so but if you are on an Intel machine or want two monitors and can find a M1 14inch for ~$1500-$1700 that's a killer upgrade no matter what.
Shoot you are right, they are port limited on the fourth display. (as the M1 Max has double the hardware from the M1 Pro so one can assume it can support quad 5K, but it has only three Thunderbolt 4.)