There are engines for this. I started out with Fabric.JS but it turned very slow with hundreds of repos. Then I moved to PixiJS (a game engine) which is super fast. I feel like I'll need to move to WASM / OffScreenCanvas and implement a custom engine, like Figma is doing.