First of all, it seems it's harder for you to stick it out here a bit longer because of the distance/effort you have to make to go out again. However, keep in mind that it's a mitzvah and hishtadlus towards finding your zivug, even if it isn't him.
That being said, I wasn't clear on whether this was a first date? You can't really expect to be attracted to someone the first time you meet him, even if you spend five hours with him. Oftentimes when there is that first-sight blinding attraction, it's not a good thing. Real attraction comes from mutual respect.
You speak about seeing some sort of potential. I'm wondering what you mean by this? Are you being fair in its interpretation? Potential doesn't mean "sparks;" it means a sense that you two share core values, could get along with each other, etc., which could potentially lead to a relationship.
I think rather than judging on an initial feeling that could easily change, you need to ask yourself other questions. Was there something wrong? Do you share similar hashkafos/values? Are you repulsed by him? If yes, try to pinpoint why.
I'm not saying that attraction isn't important, and that you should marry someone if there isn't anything wrong. But, agreeing to a second or third date isn't a commitment of marriage. You're not saying you'll marry him, you're saying you'll go out again. I know you're worried about wasting your time (and his), but sometimes you have to make that effort, and it isn't a waste. Agreeing to another date isn't by any means stringing him along; he understands what a second or third date is. It's not a commitment in any way! You don't want to wonder a year from now if you should have continued.
I would say that if you go out again (another one or two times) and still feel like you do, like you just don't like him, then I'd say we have something more to talk about. However, at this point, I would suggest going out again.