“Everyone knows that while plenty of new buildings are fine, any large
area of new buildings will be lifeless and unpleasant, and no one has
any idea what to do about it. That’s why it’s basically impossible to
demolish anything old — because it’s nearly inevitable that it will be
replaced with something worse — and why people come up with hair-brained
regulatory schemes.”

Once architects and builders prove they can do better, then the public’s desperate attempt to save everything old will fade away. We were happy to demolish beautiful architecture in the past because we instinctively knew that something even more beautiful would come along. Look at Grand Central Station, for example. Here was the first station:
http://upload.wikimedia.org/wikipedia/commons/f/fb/1880_Grand_Central.jpg

It was happily razed for a more beautiful second version…
http://farm4.static.flickr.com/3227/3128933345_02a2f097e0.jpg

The second version was happily razed for the even more beautiful third version (still extant)…
http://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Grand_Central_Terminal_Exterior_42nd_St_at_Park_Ave_New_York_City.jpg/800px-Grand_Central_Terminal_Exterior_42nd_St_at_Park_Ave_New_York_City.jpg

Not a peep was raised over the numerous demolitions and replacements! We were a confident society back then: we knew our architects/builders could produce a beauty that would resonate with the public, so we eagerly took any opportunity at redevelopment. Could we expect the same thing today?! If some fatuous contemporary starchitect proposed razing GCS and replacing it with some disgusting pile of blank-walled crap (as was the case in 1968 with the proposed Breuer tower, in which the Supreme Court even upheld the station’s historic preservation status), what would the public say? There’d be ferocious outrage!

We’ve over-regulated development and aesthetics because we lack confidence that new development will be as good as any surviving old development. And the crap churned out by most contemporary architects, builders, and developers constantly re-affirms our lack of confidence. (“See, we knew it would turn out awful!”)

“There are no good examples of un-regulated and un-planned beautiful cities that I am aware of.”

There are several towns along the coast of Italy that were basically shantytowns when first founded, but now are famous tourist attractions:
http://en.wikipedia.org/wiki/Cinque_terre

These beautiful shantytowns were mostly devoid of top-down planning: there was actually plenty of “planning” here, except it was undertaken by individuals aggregating into larger voluntary groups to get things done a certain way (a public consensus).

I would argue that the unified architecture and urban design in prewar cities was driven mostly by public consensus. Sure, styles and fashions went in and out, but when it came to attractive urban form there was probably a broad-based agreement on how to do most things a certain way: you didn’t have to create codes to tell builders not to put up blank walls near the sidewalk, for example, because it was self-evident that these were optimal locations for shopfronts and other eye-catching activities in a society in which most people walked.

Everyone knows that while plenty of new buildings are fine, any large area of new buildings will be lifeless and unpleasant, and no one has any idea what to do about it. That’s why it’s basically impossible to demolish anything old — because it’s nearly inevitable that it will be replaced with something worse — and why people come up with hair-brained regulatory schemes.

Imagine how much nicer a place the world would be if everything erected since WWI was as nice as that which preceded it. It would make everyone’s life better, every single day. And we should be to do it. Hell, we should be able to do much better. We’re richer. We have more knowledge to build on. But we build crap instead. Everyone knows it. And, as I said, no one knows how to fix it, even though it seems like it should be simple. So we get people saying it can be fixed with one decent code. (Actually, it probably could be fixed with the right code, but we’ve regressed so far that the chances of enacting that right code are about zero.)

Certainly bad bureaucrats and bad regulation  exist, as do plenty of bad developers, architects, and building owners. Democracy provides a feed-back mechanism to ensure that a city’s development delivers the results that the majority of residents consider “best”. Without that feedback mechanism, there is nothing to stop the Donald Trumps or Ray Crocs from destroying the urban fabric, except the finite amount of money they have to spend.

The incentive structure for individuals is certainly not all “positive”, as greed, selfishness, and corruption deliver plenty of “incentive” to individuals as they impoverish the community. Crack dealers get very good returns and have strong monetary “incentive structure”, but few would argue that they benefit the community as they enrich themselves. I would argue the same about McDonalds and Burger King, though some would disagree.

There are no good examples of un-regulated and un-planned beautiful cities that I am aware of.
Certainly planning, especially US-style sprawl mandates can also result in ugly places.

The complicated truth is that it is not the presence or absence of planning and regulation that determines if  a city is beautiful, but the quality of the planning and regulation has a huge impact, and all the real world cities that we consider beautiful had very substantial planning, public investment in infrastructure and regulation during their development. Not the simple story that anti-government zealots want to hear.

 London and Paris would be literally un-inhabitable without the public planning and investment in their water and sewer systems, and of course death by disease was a major reason that those investments were made. The model of privately owned sewers, water systems, roads, and transit may be a nice pipe dream for libertarian utopias (as the unicorns gallivant among the rainbows), but it does not and will not function at scale in the real world.

 Unplanned and un-regulated cities quickly implement both planning and regulation as soon as they have the resources. (Check out what is happening in Kigali right now).

To defend the anti-regulation argument, there is one restriction that Houston has which Paris didn’t– parking minimums which require roughly equal amounts of parking as building floor space. The street layout is also centrally planned, moreso in Houston’s case than Paris’s, resulting in a much larger fraction of Houston’s land area being used for streets. These two things alone are pretty effective at determining the layout of the city.

Form and aesthetics matter. For too long relativists have been dismissing these as personal “taste” issues (“beauty is in the eye of the beholder”), but it turns out there are real consensuses over these things. Few people want to live in and among something like this:
http://www.slate.com/content/dam/slate/articles/life/walking/Walkingpart1_magnum.jpg.CROP.article568-large.jpg
http://www.kunstler.com/Grunt_Atlanta%20Tour.html

Save for a few (and not-surprisingly expensive!) exceptions, most American cities look like that. Why go for a half-assed suburb (a city trying to act like a suburb) when you can buy the real thing for less? Why go for a half-assed city when you can buy the real thing too? Most US cities are an unappealing, muddled compromise between two different markets (half city/half suburb) and they fail at serving both markets.

Still, I think your points are great and generally agree that a bottom-up, fine-grained urbanism aggregates into far more of a cohesive and beautiful urban form than a top-down vision (Haussmann’s renovation of Paris being an exception).

“He suggests that the ugliness of glassy towers plays a part in driving
people from center cities to suburbs. This doesn’t make much sense to
me, as the rising popularity of glass facades correlates with increasing demand to live in city centers.”
This is the only point that irked me. Correlation is not causation! Duany argues that in expensive cities, people are willing to sacrifice beauty for the sheer ability to find a shoebox to live in:
“SOLOMON: If people don’t like modern architecture, then why are certain
units selling so well?

DUANY: I’ll tell you why. There are victims. We misunderstand each other
because you operate in a world where there is a scarcity of housing, where
people have little choice. They are so grateful to find a dwelling in
San Francisco (or Manhattan) that they put up with housing that they may
not like. The world that I operate in — the suburban Sunbelt — has the
opposite: enormous choice. Once you qualify for an $80,000 mortgage you
enter the threshold of choice. There are 10 projects to choose from with
four models each. I am referring to unconstrained markets where there
is good old American choice. When one of our projects doesn’t meet their
expectations the customer just drives off to buy some shitty Colonial
or Mediterranean, and that’s a big difference. That’s the difference.
One of the reasons that we can do modernist buildings in Aqua is that
Miami Beach is a victim situation. We’re doing modernist high-rises across
from Manhattan, and it’s no problem. But anywhere else out there in the
‘burbs — all those the places that you said you drove by — try to put
modernist houses out there and you will bankrupt the community builder.”
http://www.tndtownpaper.com/council/SolomonDuany.htm

I like ornate architecture, but I’d happily snap up a cheap apartment in a soulless glass box just for the ability to live in New York and participate in its street life. But in more affordable places where people *do* have a housing choice (the rest of the country outside SF, Seattle, Boston, Chicago, LA, DC, and NYC), rich, exuberant aesthetics prevail. Aesthetics are not as much of a relativist, easily-dismissed, subjective issue as the “urban theorists” would think.

Look at the historic preservation issue, for example. I get the impression that a lot of market urbanists disdain preservation. But if there is an unconscious consensus among the general public that almost everything new is palpably inferior to (and uglier than) the surviving prewar stuff, is it any surprise that people throw up roadlbocks to the market’s attempt at replacing richly-detailed, fine-grained urbanism with soullessly-slick, disposable contemporary crap?