Three factors are involved in how 9/11 came to happen:
FACTOR ALPHA
Liberal courts. In the desire of a free and fair court system, much of the west, the US included, seems to have a fairly incompetent justice system, in that a high portion of people guilty of crimes are found not guilty because of technicalities, flaws in law, failure by prosecutors to gather evidence in precisely the right way, etc. The end result being, no matter how strong the evidence arrayed against a suspect, there can never be any guarantee that they will be convicted, and if convicted, will receive an appropriate sentence.
FACTOR BRAVO
Arrogance. This is perhaps too strong a word, but perhaps not. Essentially an over-confidence in the abilities of its own systems and people for achieving their objectives. This covers both an exaggerated opinion of intelligence abilities and of military/defence
capabilities.
FACTOR CHARLIE
Hesitancy. What I mean is a reluctance to commit to military actions in order to address a situation. In particular, a reluctance to commit questionable strikes and pre-emptive strikes.
Okay, now how is this relevant?
Over the course of the decades prior to 9/11 the US has had an enormous number of opportunities to take direct action against various members of different terrorist networks, most notably members of Al Qaeda, and in particular those involved in 9/11. These opportunities varied, from actually capturing known terrorists (but letting them go), to being offered terrorists by other countries and refusing them, to knowing the location of terrorists in the USA and not acting.
Now, the ALPHA, BRAVO, and CHARLIE factors mentioned caused problems for any three of the potential actions in each case.
Firstly, the obvious action to take when you know the location of a significant terrorist is to send people to kill them. However CHARLIE steps into play here. The US has had policy of not assassinating people. While admirable, it is also, I think, unwise. I'm not advocating assassination as a way to solve political problems, but it should always be there as an option.
This is an example of a wider American reluctance to become involved in military actions. This is not new. America was incredibly reluctant to enter both world wars, and countless actions post-World War 2. In Somalia, for example, thousands of troops sat in ships off the coast of Mogadishu with heavy armour doing nothing while a hundred lightly armoured and armed infantry were caught in an enormous gun fight. The government even pulled support such as gunships as this was considered too high profile.
Even Post 9/11, when it is clear that America's strategy has failed miserably, they sent only 150,000 troops into Iraq after military commanders said a minimum of 300,000 were needed.
Countless opportunities to snatch or kill identified enemies of the US have been missed because of a reluctance to deploy clandestine operations on foreign soil. I am guessing the much quoted fear of upsetting political allies or creating a crisis are responses to the failed Operation Eagle Claw (attempt to rescue US hostages in Iranian Embassy). This response itself is unfair I believe. It is too easy to write off Eagle Claw as a "dismal failure". I disagree. The gutsy and risky plan had ever chance of succeeding. It only had a single flaw in its plan (though a big one) which was immediately addressed through extension of US capabilities.
Given that US reluctance to execute military operations, the response of killing terrorists must be ruled out.
That leaves two options.
One - arrest the terrorists and put them on trial
Two - allow the terrorist to go free and continue to monitor them
One would, of course, be preferred by the FBI, who are primarily law enforcers - they want to catch bad guys.
Two would be preferred by the CIA, who are in intelligence and are interested in gathering information about targets so the military can act (which, of course, they won't, due to factor CHARLIE)
ALPHA causes major problems for One. I believe factor ALPHA is primarily responsible for the enormous number of times the FBI either failed to act on evidence and arrest a terrorist, or let a terrorist go that they had in their custody.
There are a number of problems with putting a terrorist on trial, aside from the simple truth that in the west you can never guarantee a conviction, regardless of your evidence.
In these cases, one of the big problems was the nature of the evidence. Western courts have very vigorous conditions for what is acceptable as evidence and what isn't. It does not favour convicting terrorists (a large number of those caught are actually convicted on minor crimes such as passport forgery).
There are three basic ways in which intelligence (evidence) is gathered. Open gathering involves anything where the target is aware of what you are doing. An interview, for example.
Covert gathering is when you are open about what you are doing (official) but the target is not aware of it. For example getting a court order to tap someone's phone and using conversations recorded.
The final one is clandestine gathering. This involves actions that officially never even happened. In other words spying. Most evidence gathered by intelligence agencies in a clandestine manner would not be acceptable as evidence in a court. Unfortunately, virtually all information gathered on terrorists is gained in this way. Further more, revelation of such evidence at trial will require exposure of significant intelligence assets. These are likely to undermine ongoing intelligence operations and potentially risk lives. So you can't reveal this information until you've got someone significant on trial (like Osama bin Laden).
This is essentially the great flaw in using spy systems to apprehend criminals for trial. That is not what they are for. They are for gathering intelligence so that military operations can be executed. The evidence gathered by them is never intended to stand up to the rigours of civil court, and nor should it. Military intelligence is not about investigating (or even preventing) crimes. It is about assessing threats.
So, there are obvious risks in taking a known terrorist to trial. At best, the guy walks free, at worst anything in place to gather intelligence on the wider terrorist network is exposed and the entire infrastructure has to be rebuilt (which can take decades).
The only other option open is to let them go and continue to monitor them, and I suspect in the majority of cases this is the action that the US decided to take.
And herein comes factor BRAVO. The major reason people believe the US government had something to do with 9/11 is that they can't accept that Osama Bin Laden and associates managed to run circles around the US intelligence community for several decades, including having double (or even triple) agents working with the CIA. It is inconceivable that the US system for protecting the country was simply beaten by better opponents (especially given those opponents were a bunch of "cave-dwelling towel-heads"). This is extreme arrogance.
From what I can see, Al Qaeda have an exceptional understanding of the American political mindset and their judicial system, and I believe they quite effectively used these factors against the US efforts to stop them. They exploited CIA involvement in Afghanistan and the prevailing US fear of the "Red Terror", they exploited the US' weak immigration policies, and they exploited the US' reliance on Mid-east oil resources. They did these things brilliantly. Not because of the incompetence of US officials (or even compliance).
Lastly, I believe US over confidence also led to officials at policy level to grossly underestimate the threat these terrorists possessed. Thus, with policy the way it was, even when the few agents at the lower levels identified threats and could have acted, policy prevented it (through budget restraints, etc...) such as with the Phoenix FBI Office Counter-Terrorist investigations.
In considering these three factors, ALPHA, BRAVO, and CHARLIE, I have found them evident not only throughout US history, but throughout world history.
With the exception of the liberal courts (only relevant in the modern age where democratic societies have the naive notion that you can simply arrest and trial enemies of the state) these factors (BRAVO and CHARLIE) occur throughout history.
