When you can repeat real English at real speed, you can follow it too. That's the whole secret, really. Your ear learns what comes next, so films and podcasts stop rushing past you.
And here's the thing. Real English doesn't come in levels. Films and books don't have "intermediate words" or "advanced words" - they just have the words that this or that situation needs. So yes, learning new words matters, even the ones you wouldn't use in your everyday conversations.
The more words you know - even just passively - the more you understand. And the more of those passive words become active, the easier it gets to follow the English around you.
And from that point on, learning gets easier - and more fun. Because you're not studying anymore. You're just watching a film or listening to a podcast. And you're still learning.