Try lemon juice to remove the stain from the PVC. As it's vertical it will need a reapplication every 10 minutes or so for an hour. Gently, with a blade try and see if the rust will lift.