> Would that be horribly insecure?

Yes.

> I guess it would be impossible to salt those hashes, and it probably risks defeating the whole crypto.

Exactly. (There are other problems too, but that one by itself is a show-stopper.)