I've recently cured the issue of fish rank in the boot.
Wire wheel or cup brush on a drill or grinder should work well, and you could do them in-situ quite easily and effectively but it would probably be better to remove them. That way, you can check the condition of the brakes inside the drum. Also, cleaning and painting would be easier, if you want to go that far.