As far as my previous experience of travelling to UK is concerned, I remember that when I exited from UK they did not stamp my passport (I used visit visa), but only when I entered. If it is right, then I hope there would not be any problem as it may lead them to conclude that you were in UK for you...