A week or so after Yom Tov, Danny argues against walking out during Yizkor.
I sometimes used to daven in a shul in Ra'anana where every year the rabbi would ask people who are not saying Yizkor for close relatives not to leave for the exact reasons Danny gives; each year, everyone would walk out anyway, so strong is this superstition.