The phenomenon also results in huge costs that the individual ends up imposing on society. I'm not saying that means the substance must be banned. But clearly there's a difference between freedom of speech/association, and freedom to consume addictive substances, and maybe that freedom ought to be modulated according to how addictive the substance is.
The huge costs are close to 100% related to policing something extremely difficult to police. And of course these costs are somoene’s profit. The DEA still insists that cannabis should be a schedule I drug, but to them these costs are profit and job security, another example how measuring success via profit corrupts then original purpose of an institution.
Similarly, if we were to outlaw red underwear or parsley, individuals who chose these things would incur similar costs.
The only difference is society’s attitude toward it.