Yeah, thats what happens to most people. Haruyuki fools them into thinking he'll be a whiny little runt the entire time. His character progression is all the better for it though. The first episode almost did that to me too, but luckily i told myself to give it one more chance.