Bundler has always been the best package manager of any language that I've used, but dealing with gem extensions can still be a pain. I've had lots of fun bugs where an extension worked in dev but not prod because of differences in library versions. I ended up creating a docker image for development that matched our production environment and that pretty much solved those problems.
That's one of the reason I prefer a dev environment (either physical install or VM) that matches prod. Barring that I would go with with a build system (container-based?) that can be local. Otherwise it's painful.