Every Signal account is represented by the phone number the user provided in order to receive their SMS activation code, and messages are not sent directly between users' clients/apps but relayed through Signal's systems.

Your original assertion "signal knows who is talking to whom" does not follow from "Signal relays the data".