How to get refund from Travel Agent

I bought tickets from travel agent. And they over charged me about one hundred dollars. When I called the agent, he told me that I can send my statement to him or go directly to credit card company. And he prefer contacting with my credit card company. I am surprised why he prefer credit card company, he will lose nothing if credit card company ask the money back for me? Anybody has such experiences? Thanks.