It is better to walk away to ensure what you say does not go back to that person which could be disastrous to your working relationship. No one should intimidate you into listening or give you a hard time about it if you do not want to join in.
Consider this if the “office gossip” is talking about everyone else what might they be saying about you when you walk away?
What are your feelings about office gossip? Let me know!