In my view, the disconnect is that architecture isn't merely art. Unlike a sculpture or a painting which doesn't actually have to *do* anything, it has a function. If it fails at that function it is a failure. It's insane that this position has to be defended.
Know what else shouldn't be avant-garde? Food. I'd rather eat Cheerios than a bowl of steel cubes.
As far as the more aesthetic concerns go, I'm sorry, but if you think you can just put up whatever you want on your land without consideration for what other people think, or for how the neighborhood as a whole ends up feeling when you're done, you might be within your "rights," but you're still an asshole. Also, tastes change. Since you should be building things that last for hundreds of years, you should stick with styles that not just you, but your great-grandchildren might like. Thus the need for a modicum of restraint.