Thanks to Bob for this question.

Silence always works if you have the mental and physical presence to back it up.

However, in my declining years I choose not to antagonize those who do not require it. I will be courteous to down and out types rater than walking over them or looking through them. I now reserve silence for grifters, set-up men, muggers sizing me up for a hit or drug dealers. A lot depends on your personality.

A polite decline must be spoken while taking action, executing some coming and going act, and said with confidence, not as a note of apology.