Original question said: when moving near the comet. So ISTM that it's not so straightforward. Some months ago someone posted a gravity simulator here and it was very instructive playing with it and seeing how unstable a small object's orbit is around two bigger objects rotating around their center of gravity, situation that seems similar to the "duck".
Try creating two big objects in a circular orbit and experiment launching smaller objects with the same direction and speed. Some of them will crash, some others will escape.
My impression is that although it is not spherical, the core of the comet is a single object. It may therefore be treated as a point mass for these purposes.
You can treat anything as a point mass as long as you're far away enough ;) And even with point masses only, orbital mechanics can get quite complex.
When you are close enough that a significant portion of the comet's mass is "besides" yourself (somewhere off to the side) it will pull you to the side.
This even holds true for spherical objects (say, you standing on earth), where you get gravitational pull not only towards the center but also towards the sides. For a perfectly symmetrical sphere, these sideway forces cancel each other out tough.
But earth isn't a perfect sphere, so even on our own planet you get (very small) variations in gravitational pull (stronger pull at the poles or near mountains (where the crust is thicker)): https://en.wikipedia.org/wiki/Gravity_of_Earth
I encourage you to think a bit more about what is meant by center of mass. By definition, it takes into account all the mass of the body. If the force of gravity is pulling you "somewhere off to the side", then the center of mass is also "somewhere off to the side". Think about the extreme case of two identical-size, uniform-density spheres attached at a single point. That point will be the center of mass, despite the fact that little of the mass of the entire object is located near there.
The center of mass is the point where the force of gravity would pull you if the force increased proportionally with distance. In reality the force decreases as 1/r^2, which changes things. Think about the more extreme case of a barbell -- a single object. If you're 2/3 of the way on the bar, you'll be attracted towards the closest endpoint, away from the center of mass (midpoint).
Edit: http://www.nowykurier.com/toys/gravity/gravity.html
Try creating two big objects in a circular orbit and experiment launching smaller objects with the same direction and speed. Some of them will crash, some others will escape.