So here is how I approach it
A) you WILL get wet... and you WILL turtle at some point - have a change of clothes available somewhere. Agree with the above though - staying as dry as you can is critical... but impossible to stay entirely dry
B) Know how to get back in your kayak... possibly practice it in your cold weather gear.
C) when/where possible - kayak/fish w/ a buddy... it can be the difference for both of you in many situations.
1) Base Layer - Wool/Llama/artificials that stay warm even when wet - socks, underwear (long if its really cold, undershirt if it's really cold)
2) Mid Layer - Artificials that shed water or stay warm - Pants, shirt
3) Top Layer - gloves (i actually use bike gloves), ski jacket (w/ side vents & gortex shell - i LOVE those side vents for paddling - cools me off nicely on a long paddle), lightweight waders, Hat of choice, ear muffs.
4) Misc - I actually use my life jacket as additional insulation - usually i wear it under my ski jacket.
Good quality gloves do WONDERS to how you feel - and the right jacket can make a big difference as well. I personally love the side/under-arm vents as they give me a way to regulate the insulation easily w/o having to take the jacket off.
Dry suit is probably overkill down here in Texas... but i could totally see using it if i was kayaking up on any body of water that actually can ice over during the winter.