That looks like you've got porosity beneath your gelcoat surface and you're sanding into it, then the compound gets into the porosity and it becomes visible.
Porosity can be a bit of an issue with polyester gelcoats. I would suggest trying a vinyl ester if you can get one.