I'd go with the brushes. As you have a separate solenoid the tap on the starter will only affect the brushes contact on the armature face.
From memory, and its a long while ago, the old inertia starters have four small screws holding the back plate on. Pop it off..... If they are dirty just...