Yeah, but to a certain extent that's true of things that no one doubts are science. Evolutionary biology is limited by the fact that evolution is non-repeatable, for example.
I used to be an archaeologist. Definitely a field where you can't experiment; you're studying the evidence left by the past. But good methodology in archaeology is making hypotheses about what you expect to find in a new excavation or an inadequately studied old collection, and then testing those hypotheses. Yes, there's a problem with controls, but that's not unique to social science.
Archaeology isn't a science either, even though I've studied it myself and ancient civilisations are a particular favourite area of study for me. Archaeologists and historians in general frequently do things that wouldn't match the usual understanding of the scientific process: that doesn't mean we shouldn't study history, but comparing the fields credibility to that of physics is really problematic.
But to clarify, when I said "social science" I was referring to people who explicitly call themselves that. Archaeology and even psychology wouldn't count under that definition - but if someone actually says they are a professor of social science, to me that's an immediate red flag because the "research" produced by such people is in my experience usually just a complicated and expensive form of personal politics blogging.
Jonathan Haidt has done a lot of good talks on this topic. There's a paper here:
