IIRC, this is not a bug, it's by design:
Assuming you have "clickToScroll text", if your file is big enough that the entire minimap will not fit in the screen, when you click outside the box, the box will not move at the same place as the mouse cursor, since the minimap has to scroll.
Being able to drag the box afterwards would be akward since the mouse is not over the box.
A possible solution that was discussed would be to force the mouse cursor to jump to the new box position.
See this (dead) old thread: https://forum.sublimetext.com/t/file-change-detection/21/1#p2967
With enough votes maybe Jon will consider implementing it