I think what some times happens is that people find the feeling of bobbing about on the water so relaxing that they shut their eyes for a little while and don't pay attention to how far away from the shore they are - until it's too late. Clifton's beach is very protected from the wind, but once you drift a little further out there's sharp change and the wind is easily able to drag you far away from the beach - too far for the average person to swim back.
Fortunately, the Clifton and Camps Bay beaches normally have lifeguards on duty who would probably warn you when you drift too far out. That said, it's not worth the risk, so be careful and keep your wits about you!