Never had any problems like that on my moulds UNLESS I didn't use the 2 cup system; first mixing in one cup, then pouring over in a second cup to mix again to make sure everything is well mixed. Sounds like your silicone is sweating... I'm not 100% sure but  warm soapy water shouldn't cause to much problems, maybe wait for Warren to confirm this... Just make sure that before use you put your mould in the oven to make sure no moist is left.

What I also like to do to clean moulds is use a clean cloth and some easylease... Best case you'll remove some excess residue, worst case you'll add some more release. So It's a win win sitaution...

As an extra tip: I always do a first pour with xencast P4 which is relatively cheap and fast curing... that way while demoulding you'll remove any residue or dust that was still on you mould after demoulding it from the plug. I see it's a big mould but you could do a small pour just on the bottom where you have some residue like you say, or pour it in 4 times so you have all the sides