I suspect most of us understand on a gut level that any "joke" that devalues others because of their race, sex and/or gender, nationality, sexuality, class, disability etc. contributes to their oppression. Speaking up when we hear these so-called "jokes" works to disrupt oppression. In fact, being an ally depends on us intervening when we witness a situation in which oppression and exploitation is taking place. But if we remain silent in these situations we allow oppression to continue. Not only are those targeted by the "joke" harmed, but so are we by the damage done to our own integrity.
Often when we intervene in these situations we can expect some kind of defensive reaction or backlash from those making the "joke." They may use denial by claiming that the "joke" doesn't devalue, exploit, or oppress others. Or they may try to minimize it by saying, "It's only a joke." They may blame the target, perhaps saying, "Well, if those people weren't so... ." They might attempt to redefine the situation by claiming that some people in the target group make the same "jokes" and therefore it's not oppressive. Or they might claim it was unintentional, that nothing oppressive was meant by it. They could claim that all that is in the past or that it's over now, as if oppression is no longer an issue. They could claim that oppression is an issue of only a few people and we should be concerned about the "real" oppressors. Or they might just counterattack and suggest a competing victimization by claiming everyone oppresses someone so why should they have to change.*
If we intervene we're likely to be seen as being "extreme," "radical," or "difficult." So we're told "it's only a joke" and that we should just "lighten up." We're asked to be complicit in the oppression of others, if not willing collaborators. Our adherence to anti-oppression principles is being tested when we're asked to compromise our personal integrity and ignore oppression.
While "jokes" that devalue other animals are common, for vegans, our adherence to anti-speciesism is more often tested with the products of nonhuman exploitation. We're often offered something that contains ingredients derived from nonhuman animals and told that these ingredients are only "trace" or "insignificant." Most of the defensive tactics listed above are used against us. For instance, saying that the ingredients are "insignificant" is a form of minimization, and claiming that the ingredients are "humane" or "cruelty-free" is a form of redefinition.
We may also experience counterattacks, this can include attacking our personal integrity by calling it "personal purity" or a "holier-than-thou attitude" that turns people off of veganism. This last tactic suggests that our integrity is something we should be ashamed of. It's very odd, because usually integrity is attacked when we fail to stick by our principles, but here our integrity is attacked for adhering to anti-oppresive practices. In each of these situations where backlash tactics are used against our veganism we are being pressured be complicit with the oppression of other animals, and often to become collaborators with that oppression.
Just as no "joke" devaluing others is "insignificant," neither is any amount of exploitation of nonhuman animals that goes into food, products, or events. As with "jokes" that devalue other people, we can intervene in the oppression of other animals by asking about ingredients and refusing to accept anything derived wholly or in part from the exploitation of other animals.
* See the chapter on "Retaining Benefits, Avoiding Responsiblity" in Uprooting Racism: How White People Can Work for Racial Justice, by Paul Kivel, for more on the backlash tactics listed here.