You can swim in one, but it's kind of bulky up front when it's inflated. It might be a little cumbersome on a re-entry. Plus it'll cost ya' like 30 bucks for a recharge kit after taking a dunk.
I have one and used to wear it all the time. Still do in a bass boat. In my yak, though, I just got used to wearing my regular PFD. I like the option of being able to get it wet, especially in the summer. I've been known to beach the boat and take an impromptu swim when it's hot, and I leave my vest on when I do it.