I'll try to to keep this succinct as possible, as long winded statements tend to lose the audience.
You CAN swim rivers with waders, providing you prevent water from entering them. I have done experiments. When you enter a river with waders on, the water will compress the waders to your body and even with a tight wading belt air will be expelled. This will create a TEMPORARY seal, water will slowly leak in and once the seal is broken, water will enter your waders rapidly. This is were a wading belt or belts, greatly reduce water penetration. Wearing a Goretex jacket, drytop and PFD will further reduce water penetration.
If your waders fill will water, you won't sink because water is neutral in water. What can happen if your not wearing a wading belt, is your waders become like a parachute and catch currents and they can pull you under.
When I came out of a pool after filling my waders, I had 70 pounds of water in them! It is extremely difficult to swim with your waders full of water, it's like having a full length cast on both legs. It is extremely difficult to kick with your legs as a result. This also makes it very hard to safely climb up to shore to safety, due to the weight of water in your waders.
Use a wading belt at all time!
Rivers can be dangerous, you need to ask yourself if it's worth the risk wading above your waist in faster currents.
As someone else pointed out, you don't suffer if you die, your family, kids, friends do! No fish is worth a risky river crossing. I'm not saying not to wade above your knee, I'm talking about a high likelihood that you can be swept downstream. If you plan on doing high risk wading I highly recommend wearing a whitewater style PFD, as inflatable PFD's ARE NOT designed for rivers.