Kind of a reach. We think it's cool because of our experience, but would you go way out of your way to visit?
I think it depends on what you're looking for in a long weekend excursion. If you like visiting a city for the weekend, prefer higher end dining and pro sports, then yeah, it's not going to be for a lot of people.
On the other hand, if you like college towns because they offer a fun vibe, have some college sports & music, and you like a laid back town with a couple wineries, breweries, and outdoor activities, then Athens is a great weekend stop. Plus, you can park the car for the whole weekend and walk everywhere you want to go.