It ultimately does depend on how you choose to define "sex". This turns out to be a hard question, with different definitions making sense in different contexts. Is it gametes, chromosomes, genitals, hormones, something else, or some mix of all of that? What happens when these characteristics that can change, do? How do we classify people who have mismatches between some of these categories of characteristics? What assumptions are safe to make about people of different sexes?
My point is, our interpretation of these observable biological processes and facts, and our response to them, is socially constructed. There are many Doctors and Scientists who do very much agree with this complex view, just as I'm sure there are those who believe in a strong binary like you. I see the complex view here as much more useful, especially in navigating things like human dynamics and healthcare needs in ways that benefit us all
