Why is it that everybody I talk to tells me that they could open my defender with various items (screwdriver, spoon, 10p, big toenail and so on) and how easy it is to steal, yet the actual key will work about one day a month and only when I'm not in a hurry at all? Should I just start carrying loose change and bin the key?

Yep.

Ideally you want a disklok, pedal lock and replace the locks with deadbolts. Start to worry (more) when you end up being able to start the Landy with a spoon....