This is a picture of my son when he was 3 1/2, but I had him in the boat since he was old enough to walk. We always used a life a jacket that had a loop that went between there legs. Also my wife started swimming lessons with our children when they were 6 months old.
As far as the boat, if its safe for you, its safe for them. My wife always made a statement before I took my kids fishing : " Remember your kids are not going fishing with you, you are going fishing with them. " This means make it very enjoyable for them, don't take them out too long, make sure the weather is comfortable and they are having fun. It must of work, because my son is 23 now and goes fishing with me almost every trip. Right now is a perfect time, you can find perch any where and you can beach the boat and take them swimming. Bring your wife and make it a family outing!!