I use worms about 95 percent of the time and have always had barbs on my hooks. For most of the time my fish have been released without any problems at all.
There will always be a few exceptions to the rule.
A good solid hook set and a very early feel of the bite play a important fact on where a fish gets hooked. Just my two cents.
Side Note: Rios are the best fish by far to take a hook out of. I hook just about all of them in the upper front of the mouth. They seem to hold the bait in their mouths and not swallow it so quickly.