I wouldn't wear letters like a jersey, but I do still wear tee shirts that have letters. I also have a fraternity tie and this shirt:
that I will wear when appropriate, including to more casual alumni functions. Beyond that, I have lots of things that are red and black without letters.
