Fixed it!
First I tried to prise it out with a plastic shim, but that wasn't strong enough. So, with the ignition off and a very small flat-head screwdriver I managed to lever it out quite nicely. No damage done to the car or the lighter.
I've swapped the lighter with the plastic stopper...