I'm dating a guy I don't really want?

Ok, so... I don't know if I like him. I mean, I love hanging out, spending time, the things we do together and our outings. Problem is I'm having a really hard time seeing any kind of a future with this guy. We clicked so fast, played around, and are sweet to each other. I'm not sure this will all pan out, but there's nothing wrong with this guy! Sometimes I think I'm crazy for not being into him because he's respectful, sweet and a perfect gentleman. He treats me right and isn't in it for . I just don't feel the way he feels about me. He thinks I'm the greatest thing in his life and the luckiest guy but I don't think I can love him the way he deserves. He just broke it off with his ex and is pretty down about himself because of the verbal slander of this girl. I've told him that everything happened so fast and that I didn't know if I liked what I started but he feels that I'll "grow" on him. I'm gonna give it some more time to be sure but how do I go about breaking it off with this guy without completely hating me, or more importantly, himself?