I can't add much more, except that I also use vienna lime, but I also have an unsewn soft cotton wheel that is only used for final polishing, so there will be no heavy grease deposits on it.
Like Giles, I also use a felt wheel, but only for getting into very sharp corners, I find it rounds out a part a bit too much for my liking, I like nice sharp edges, but they do cut the metal back to a very fine finish.
I have recently purchased a mop for plastic polishing that is made up of thousands of very soft loose strand cotton fibres. I think that would be just great, with a bit of vienna lime, for getting the workpiece to it's final sheen.
Your problem, I think is just a bit of technique and wheel selection problems. The higher the sheen, the softer the touch and cleaner and softer the wheel has to be. Plus your hands, they are covered in natural grease, wash them prior to final polishing. But never, ever, use gloves or a bit of cloth for holding you parts, it is better to lose the part rather than your fingers. Professional polishers do use special polishing gloves, but that is purely for speed reasons, to allow them to hold hot friction heated parts much longer, and so allow more production. We should never be in a rush.
You might find that a manual polish with a bit of soft cotton (new t-shirt) and no polish just might rub the haze off. I am lucky in that I can obtain some very special paper cleaning cloths that are perfect for the job.
I also find that a proprietary brand of furniture cleaner and polish will bring up the surface and keep it shiny for much longer. In the UK I use Mr. Sheen, again with a super soft cloth.