I always find the best thing to do is to wait for them to dry. If you're down on the beach and under the sunshine this shouldn't take too long. Once they are dry you can gently brush it off with a soft towel or your fingers and it won't scratch the skin.
When the skin is wet all it does it allow the sand to stick to it. When it's dry there's nothing holding the sand off and it makes it much easier to brush away.