I've just had a simular problem. I couldn't hear the starter clicking, but if I could then I would be thinking about getting the starter repair kit to change the contacts. The clicking should be the starter. I would check the battery connections are tight, the positive connection to the starter is secure, the control wire to the starter is tight. Also the 0volt return wire from the starter/engine or where ever it is.
My problem was the electrical contacts in the switch on the rear of the ignition barrel being dirty/carbonned up. I took the plastic off the rear of the switch to reveal the contacts through small holes. While operating the switch manually with a screw driver I wiped the contacts with sand paper to clean them a bit. May not be the same fault as yours but something easy to check. Just be careful when taking the switch off. Disconnect the battery power first. Also if it's an auto theres tricky plastic and metal bits which are primed by the auto inhibit mechanisum, behimd the switch when you pull it off. Make sure you see how they fit in just incase they jump out.
What colour = what engine/gearbox/year.