I think you might have just had a bad class on the topic. For what it's worth I watched Erik Olson's artistic linear perspective class lectures on New Masters Academy and it definitely didn't teach perspective working the way you describe.

https://www.nma.art/courses/a-complete-guide-to-perspective-...

The first lesson is free to watch too: https://www.nma.art/videolessons/perspective-1-an-introducti...