I listen to podcasts all the time, ones by American, British, and Australia speakers, and I think my aural comprehension has actually improved from that. I have a friend just like you though. Has to turn on subtitles for everything.