Haven't studied the SvelteJS documentation but as far as I can reason there is nothing in their idea that prevents them from shielding the DOM behind a lightweight, virtual DOM, ReactJS style. In fact, I'd be slightly disappointed if they don't shield the DOM somehow. It's the #1 bottleneck, after all.
The virtual DOM isn't a replacement for the DOM, it's just a string diffing algorithm. It is only needed because of React's "render everywhere" approach, which is a tradeoff sacrificing efficiency for simplicity. If you don't re-render the entire page any time the model changes, you don't need a virtual DOM.
I'd say your mostly right with the one caveat that a virtual DOM also helps out with batching changes to the DOM, but I suppose there are other ways to make that happen.