The best answer I can give is that they're trying to keep up conversation. It makes sense from the interpretation that conversation is about constant reciprocation. That it bothers you is rational because they're indicating that they didn't understand why you made the reference.

Here's an autistic rebuttal, though: what are you hoping to gain by making a reference? I get a lot of awkward situations due to other people trying to make references, and the only motives I can discern are either manipulation or simplicity. So making references "grinds my gears".

Are there at least three acceptable responses to being confronted with a reference? One is laughing, that's accepted. Second is to carry the reference with another reference to the same scene/character/media. But what's a third acceptable response?