rattle from steering bearing

I am getting closer to being able to put some body panels back on, having just about done the mechanicals.
However, when i started the engine today (which did run nice and smoothly) there was a rattle, which i tracked down to the steering. The outer skin of the steering rod is firmly in place, so this was the inner steering rod that leads from the steering wheel to the box.
When i refurbished the steering box, i did sort out everything in the box (new bearings etc), but i didn't touch the bearing at the top of the steering rod, just below the steering wheel. At the time, i had read some descriptions which put me off rebuilding this bearing.

I have a choice now. Either I leave it with the slight rattle or rebuild the bearing (ideally without removing it from the car, as it took a long time to get the steering geometry right, and getting the arm on the steering box off and on again leaves me with dread).
Any advice?

From what I recall it has to be off the vehicle to do.

I've changed both types with the column in place, the bearing one is pressed steel and I dimly recall peeling its remains out.

That's good to know @boguing.

Could be mot failure , so best get it sorted either way , one less annoying noise to suffer

i pulled the outer off today (undid the bolts onto the box, let the oil flow out and un did the various bolts holding the brackets in). Having got it off, i then used the broom handle to push the (very worn) bearing out. I will get a replacement, fit that, and then refit all the pieces. Best to get the job done now whilst i have got reasonable access.