IMO the victims are the members of these communities that feel unsafe walking down their own streets because a glorified klanbake just took place in their hometown. "Go back to your country" sounds like some dumb redneck shit that any of us can laugh at, but that can very easily be perceived as a threat when you are the minority being targeted. I would argue that the safe play would be to assume it is a threat, just in case these people are a threat to you/your family. Kinda like assuming a gun is always loaded, so you never point it at something you wouldn't want to shoot.
Verbal assault can definitely meet the legal definition of harm. But I'm in economics not law. I'm not sure where that line is, but I do know you don't have to actually threaten someone with violence to get arrested for yelling obscenities at someone in a bar. I view it as somewhat analogous to a case like that, where a female victim is being verbally abused. She has the right to exist in public, without being attacked in that manner. I would argue that the individuals within minority ethnic groups have the same rights when their race is attacked as a collective.
To be clear, I'm not claiming that the average trump rally meets these criteria. But some of the "alt-right" organized rallies have.