Yeah. If you have a large enough GPU you can use vanilla pytorch to load as many models as required. Docker is a good option if you have isolated services. Triton, TensorRT, TorchServe, RAY are also good services to checkout, especially when you want to load multiple adapters for the same LLM/VLM backbone. Is there anything specific you are looking to serve?

Papers like OneVision have started looking into this. But most of the research is still in nascent stages, answering in one word/phrase for simple questions. I don't even think there's a good enough benchmark dataset to evaluate such models.