I've had that same exact problem when I tried to fish an XD-5. A drift sock could help but I think the best option is to change baits. Carolina rig, jigs, flutter spoons, slow rolling spinnerbaits won't pull you to the bait as bad. If you insist on using a crankbait, some medium divers with lower drag could work depending on where the fish are - Bandit 300, Norman N series, Bomber Fat Free Shad, etc.
Another option is to change your retrieve. Crank the bait deep and then pause every couple of seconds throughout the retrieve. This should reduce the pull over a steady retrieve.