Just a tip, AC dries the air that runs trough it, so it helps demist your screen more quickly. It doesn't need to be set on cold for that. Aside from that, some people on the Toyota Owners Club forum had been using the belt from the non-AC equipped versions of the Avensis to bypass running the AC compressor. I don't know if the TD4 has a non-AC equipped version, or if there is a suitable route for a shorter belt to miss out the compressor without fouling on something else, though.
*edit - Re-reading your post gives the impression that the AC runs on a different belt to the other belt-driven accessories?