As an aspiring author, I will frequently hear "Not right for my list," or "the characters weren't compelling to me." What do you as an agent do if an author you've signed writes a second book that doesn't resonate with you? How do you pitch that book to publishers?
I think the important thing to remember is that these phrases you're hearing are form letter phrases. It's the wording agents have honed over the years because it's vague enough not to open ourselves to back-and-forth communication and yet truthful because there's something about this book that isn't right for our list. Maybe it was your voice, maybe it was the genre, maybe it was simply the story.
When an agent signs an author, usually there's a meeting of minds (doesn't that sound grand). In other words, usually the agent feels passionate enough about the author's voice and the general direction of the author's writing to want this author on her list and everything else this author might bring to the table. This is why it's so important that you sign with an agent who sees clients as a long-term endeavor and not just a book-by-book project.
I think the fear that authors have that an agent will love the first book but nothing after that is overblown. Sure it can happen, and I'm sure many have horror stories of it happening, but for the most part I think once an agent feels passion for an author's work, they feel passion about that author in general.