Racism is part of the problem but it's not the main issue.
First, a lot of people crave for the way the game was played in the 80's which today's game can't provide. In my opinion, today's game is far more entertaining to watch but I totally get that some people gave up in the early 00's after they didn't have Jordan to watch anymore because the game wasn't the same.
Second, the league had a serious image problem up until about 2005-2006 and people haven't come back to realize the league has done a 180. That image problem wasn't based on race it was based on players flashing gang signs, the whole Pacers/Pistons brawl and everything else that went with that gang culture. It was prevalent in the NBA for at least five years. The common fan doesn't want any part of that and that doesn't have anything to do with race. It was hard for even a hardcore fan to love the NBA during those years.
And then lastly race. I get so mad when a couple of punches are thrown on the court and it becomes the top story on Sportscenter and makes local news when it's not even local. Meanwhile over at the hockey arena across the street people are cheering on two goons dropping gloves on the ice. Or, a brawl erupts at the baseball stadium and barely anybody flinches. That's directly related to race issues and that's where you get a lot of people who haven't come back to the NBA the chance to say, "They're thugs." No, they are pro athletes in a competitive enviroment. If you don't like it, don't watch sports. But, at least be consistent.