Slow SQL connection when using listener DNS name

Hi,

I am observing an a slowness in SQL server connection when we are using SQL Server listener name in the connection string.   However when we use the listener IP in the connection string instead of listener name the connection seems to be faster. 

We are using SQL server 2012 SP4, and using Windows authentication. 

Any hints what I should be  looking at?
BTW, Its not a multi subnet settings

Satheesh