Calling younger deer

Often if I see a younger 2 or 3 year old I will grunt at it just to mess with it and see if it will come in (which they usually do). It makes for a fun hunt and is awesome when they are literally right under your stand. It got me thinking though, maybe I shouldn't do this at deer I don't intend to shoot. Maybe they will retain this info as they get older and become call shy/educated? Also they could work down wind of you and spook, thus scaring other/future deer that may have came in as well.

The other thought is that if you do call them, another larger buck that you didn't see could come in as well. Or the younger one could act as a live decoy.

Do you guys call at younger bucks on purpose or leave them be?