How do you know that? Maybe it did bother him back then, too. In fact, I'd be willing to bet that it did. And, why does it matter? It bothers him now. That deserves consideration. Can you empathize with his perspective? If not, then you're probably going to be considered a racist on some level, and it would be a rightfully earned label.
As I've said before, Confederate monuments neither offend or inspire me. I have direct lineage to at least three Confederate soldiers, one was a famous cavalry general. I'm not proud of what they fought for. But, I understand the context of "then and now." The statues were usually erected to try and find a way to honor the "Lost Cause" white-washing of the Confederacy. That wasn't very honorable, either. SIlent Sam, to me, is a little different. I think Silent Sam is pretty benign and actually a symbol of defeat while trying to show compassion for the loss of life. The loss of life is equally felt on all sides of any war. The earth gets filled with bodies and it doesn't care what uniform was on the body of the new resident.
It should be abundantly obvious why it bothers black people.