They have to. A phone number is not a key, so there's no way to avoid that.

However I've seen leaked screenshots that suggest at least the Android client is getting the ability to see what the keys are and pin them. Hence, TOFU encryption. It's about the best you can do in a consumer product.