The Secret Sentry
Authors: Matthew M. Aid
The Secret Sentry
THE SECRET SENTRY
The Untold History of the
National Security Agency
MATTHEW M. AID
Copyright © 2009 by Matthew M. Aid
All rights reserved. No part of this book may be used or reproduced in any manner whatsoever without written permission from
the publisher except in the case of brief quotations embodied in critical articles or reviews. For information address Bloomsbury
Press, 175 Fifth Avenue, New York, NY 10010.
Published by Bloomsbury Press, New York
All papers used by Bloomsbury Press are natural, recyclable products made from wood grown in well-managed forests. The manufacturing
processes conform to the environmental regulations of the country of origin.
LIBRARY OF CONGRESS CATALOGING-IN-PUBLICATION DATA
Aid, Matthew M., 1958–
The secret sentry : the untold history of the
National Security Agency / Matthew M. Aid.
p. cm.
Includes bibliographical references and index.
eISBN: 978-1-60819-179-6
1. United States. National Security Agency— History. 2. Intelligence service— United States. 3. Electronic surveillance—United
States. 4. United States— History—1945– I. Title. II. Title: Secret sentry, the untold history of the NSA. III. Title: Untold
history of the National Security Agency.
UB256.U6A53 2009
327. 1273—dc22
2008037442
First U.S. Edition 2009
1 3 5 7 9 10 8 6 4 2
Typeset by Westchester Book Group
Printed in the United States of America by Quebecor World Fairfield
To Harry, Rita, and Jonathan Aid
My Family, My Best Friends, and My Staunchest Supporters
Gratis eternum
Know your enemy and know yourself, find
naught in fear for 100 battles.
Know yourself but not your enemy, find level of
loss and victory.
Know thy enemy but not yourself, wallow in
defeat every time.
—SUN TZU
There are no secrets except the secrets that keep
themselves.
—GEORG BERNARD SHAW,
BACK TO METHUSELAH
Contents
The Origins of the American Cryptologic
The Travails of American Communications
SIGINT and the Korean War: 1950–1951
The Creation of the National Security
4. The Inventory of Ignorance:
SIGINT and the Kennedy Administration:
6. Errors of Fact and Judgment:
SIGINT and the Gulf of Tonkin Incidents
NSA and the Vietnam War: 1964–1969
NSA During the Johnson Administration:
NSA During the Nixon, Ford, and Carter
10. Dancing on the Edge of a Volcano:
NSA During the Reagan and Bush
From Desert Storm to the War on Terrorism
12. Snatching Defeat from the Jaws of Victory:
9/11 and the Invasion of Afghanistan
13. A Mountain out of a Molehill:
NSA and the Iraqi Weapons of Mass
15. The Good, the Bad, and the Ugly:
SIGINT and Combating the Insurgencies
The Origins of the American Cryptologic Effort Against Russia
Another man’s soul is darkness. Does anybody
ever really know anybody else?
—RUSSIAN PROVERB
The consensus of historians (and the overwhelming burden of evidence) dates the initial stages of the Cold War to well before
the end of World War II. The United States would emerge from the war as a superpower with arguably the world’s strongest armed
forces, sole possession of the atomic bomb, a vastly expanded industrial base, and an infrastructure untouched by the ravages
of war. But on the negative side, the country had at best a rocky relationship with one of its war time allies, the Soviet
Union. By the time Nazi Germany and Japan had surrendered, Russia was on a collision course with both the United States and
Britain. It was not long before the Soviet Union was regarded as “the main enemy” by the Western nations. Since it remained
a rigidly closed society under Joseph Stalin’s regime, the lack of transparency was a major factor driving the Cold War. Because
the United States had only a very limited idea of what was going on in the Soviet Union, its satellite countries in Eastern
Eu rope, and communist China, the emerging confrontation became all the more dangerous. But one of the most secret resources
that had greatly contributed to the victory of the Allied Powers—the United States and Britain’s ability to intercept and
read the communications of our former enemies Germany, Japan, and Italy, both in the clear and encoded—would be quickly redirected
to the task of gathering communications intelligence about the new Sino-Soviet threat.
It is difficult to imagine, many decades later, just how mortal that threat was perceived to be, particularly after the Soviet
Union detonated its first atomic device in the summer of 1949. The prospect of a “nuclear Pearl Harbor” meant that the United
States would rely heavily on an increasingly large and expensive communications intelligence effort.
Carter Clarke Declares War on Russia
In a certain sense, Brigadier General Carter Clarke was the founding father of the National Security Agency (NSA). A blunt,
often profane, hard-drinking, and demanding individual, Clarke lacked the polish of his fellow officers who had gone to West
Point. He began his career as an enlisted man and worked his way up through the ranks. Despite a lack of previous intelligence
experience and a file drawer full of bad fitness reports (Clarke was a real maverick), he was the man the U.S. Army selected
to run the analytic side of SIGINT Army G-2, the Special Branch. A college dropout (he joined the army and served under General
John Pershing chasing Pancho Villa in Mexico), he was a highly intelligent man and an autodidact.
Clarke was described by many who worked with him as being a tough, impatient, no-nonsense workaholic who abhorred conformity
and was intolerant of bureaucracy. When things did not get done to his satisfaction, Clarke’s volatile temperament usually
took over. Former colleagues recall that his temper tantrums were legendary. A former army officer said, “I knew that Clarke
had an explosive temper. Although quite a decent person, he laced his language with frequent bursts of profanity.” His detractors,
who were many, described him as loud, uncouth, brash, and argumentative, with a tendency toward overstatement when trying
to make a point or win an argument. And yet, despite his brashness, gruff talk, and stern demeanor, Clarke earned the respect
(and fear) of virtually all the U.S. Army intelligence officials he dealt with. A former senior NSA official, Frank B. Rowlett,
described Clarke as “a very unconventional man and a man of considerable moral courage [who] would spit in your face and laugh
at you.”
1
Clarke’s Special Branch was a component of Army G-2 in the Pentagon created after Pearl Harbor, the unit to which all intercepts
were sent for analysis and reporting to consumers. It only worked on SIGINT materials, while the rest of Army G-2 worked on
more mundane materials, like military attaché reports. The army’s SIGINT organization, the Signal Security Agency (SSA), commanded
by Brigadier General W. Preston Corderman, was a separate field agency that was (until 1944) part of the Army Signal Corps.
As noted above, all its intercept material went to Clarke’s G-2 Special Branch.
When Clarke took command of the Special Branch of Army G-2 (intelligence) in May 1942, the United States was able to read
the top Japanese diplomatic and military encoded communications (which enabled U.S. forces to win the Battle of Midway in
1942, the turning point of the war in the Pacific) and the British were reading the German codes generated by the Enigma machine.
Despite his rough edges, Clarke worked well with his British counterparts in the Bletchley Park code-breaking center. Deep
down, however, he trusted no man and no nation. According to Rowlett, “Clarke was a good man to have in the intelligence business
in our line of command [the communications intelligence, or COMINT, field] because he didn’t trust any nation. He just said,
‘They’re your friends today and they’re your enemies tomorrow, and when they’re on your side find out as much as you can about
them because you can’t when they become your enemy.’ ”
2
The United States was not only reading the codes of the three Axis Powers; it was reading the encrypted diplomatic and military
traffic of more than forty other countries—including our allies and neutral states. Well before the end of the war, Clarke,
like many in the American military and government, decided that the Soviet Union would become our next “main enemy” after
the war, and he issued an order in January 1943 to begin cracking Russian codes. So secret and delicate was this operation
that very few people were allowed to even know it existed, and since virtually nothing was put in writing, the paper trail
today is virtually non existent. The U.S. Navy had its own code-breaking operation headquartered in Washington. Though the
two cryptanalytic organizations shared code-breaking responsibilities, cooperation was the exception rather than the rule.
3
The army code-breaking operation was headquartered in a former girls’ preparatory school named Arlington Hall, located in
Arlington, Virginia. The main building on its large and beautifully landscaped campus housed the administrative offices. Tacked
onto it, once the army took over and fenced it off from the world, were two wings that housed large open bays crammed with
code breakers, linguists, and analysts, crowded together and forced to endure the scorching and humid Washington summers before
the widespread use of air-conditioning. Hundreds of fans provided some relief— but unfortunately they blew working papers
all over the place. The sole air-conditioning was reserved for the noisy and noxious IBM tabulating machines.
4
Clarke had some supervisory authority over Arlington Hall Station (its official designation), but he largely worked out of
a high-security area in the Pentagon. The intercepts of enemy communications that were picked up by a far-flung network of
listening posts, some of them in remote areas like Ethiopia and Alaska, went to Arlington Hall, where they were decrypted
and translated. Then they were sent on to Clarke’s analytic organization. The intelligence product derived from intercepts
was so sensitive that its distribution was extremely limited, reaching only a few hundred people with the highest security
clearances. The paradox here is that in order to protect the sources and methods used to gather this invaluable signals intelligence
(SIGINT) and not tip off the enemy that the United States was reading virtually all of its communications, the intelligence
product often had to be “sanitized” (i.e., put in a form that would not disclose the source of the intelligence reporting)
and sometimes did not reach those who needed it most. (Both Admiral Husband Kimmel and General Walter Short, who took the
burden of blame for Pearl Harbor, were arguably deprived of information that could have made the events of December 7, 1941,
a very different story.) Throughout the war, commanders in the field below a certain level of rank and responsibility were
not furnished with this critical information, or got it in a very watered-down form, which tended to make the material not
as useful as it should have been, particularly because these officers could not know just how definitive and reliable it was.
The same complaints that were voiced back then are still heard today.
Because the British had developed a formidable code-breaking operation that was in many ways superior to the Americans’, once
the United States entered the war there was an almost complete sharing of information and coordination of efforts. But the
British were not apprised of the U.S. attack on Russian codes. In any event, they were undertaking their own effort, which
they also did not disclose to the United States.
5
Well before Germany, Japan, and Italy surrendered, the Cold War was under way, setting our quondam ally, the Soviet Union,
on a collision course with the United States, Great Britain, and, in time, the other nations that would become the North Atlantic
Treaty Organization (NATO). Accordingly, before Germany surrendered, the United States and the United Kingdom decided that
everybody’s cards had to be put on the table. Prime Minister Winston Churchill and his commanders (particularly Brigadier
General Sir Stewart Menzies, the head of the British spy agency MI-6) firmly believed that a concerted effort had to be made
to penetrate what Churchill described as a “riddle wrapped up inside an enigma”—the essentially closed society of the Soviet
Union. This belief was shared by General George Marshall, Admiral Ernest King, and just about everybody at senior levels of
the U.S. government and military— with one exception, President Franklin Delano Roosevelt. FDR wistfully believed that the
United States and Russia could “peacefully coexist” after the Allied victory. So it was decided that he not be informed that
we were spying on our Russianally. The Russians, of course, were doing the same thing to the United States and Britain and,
unfortunately, as we know now, doing a much better job. The full extent of Russian espionage was made clear when we began
to read their enciphered messages. One key early break-through came in October 1943, when a thirty-seven-year-old lieutenant
named Richard Hallock, who before the war had been an archaeologist at the University of Chicago, made the first break into
the Russian ciphers. Incredibly, the Soviets had reused the pages of their one-time pad cipher keys on a number of occasions
in different kinds of message traffic.
6
(A “one-time pad” used to encipher messages is a bound set of sheets, each one printed with randomly generated numbers—representing
both words and numbers— organized as additive “keys” and a certain number of lines of numbers in separate “groups.” No one
sheet in a pad and no pad or set of sheets duplicates any other, except for the matching pad’s sheets used for deciphering
the encoded message. The sheets are to be used once only and then destroyed. If used properly, the pad provides a virtually
unbreakable code.)
The German invasion of Russia in June 1941 and the chaos that followed had created a severe shortage of cipher materials at
Russian overseas diplomatic establishments, leading the NKVD’s* cryptographic department in Moscow, which produced all code
and cipher materials, to take shortcuts to fill the increasing demand for cryptographic materials. As the German army drew
ever closer to Moscow in the winter of 1941, the Russians apparently panicked, printing duplicates of twenty-five thousand
pages of one-time pad keys during the first couple of months of 1942, then binding them into onetime pad books and sending
them not only to their diplomatic and commercial establishments, but also to the various NKVD
rezidenturas
(or “stations”) around the world, thus unwittingly compromising the security of all messages encrypted with these duplicated
pads. Then, to make matters worse, the Russians could not get new cipher materials to their diplomatic establishments in the
United States and elsewhere because of German U-boat activity in the North Atlantic, which hampered Soviet merchant shipping
traffic between Murmansk and the United States.
7
SIGINT Comes of Age
Beginning in early 1943, the U.S. Army’s SIGINT collection effort slowly began to shift from Axis military communications
targets to the pre–Pearl Harbor focus on foreign diplomatic communications traffic, largely because of dramatic changes taking
place in the global geopolitical balance of power, with the United States rapidly emerging as the world’s top superpower.
Senior U.S. government and military policy makers and intelligence officers alike fully understood that while military decrypts
(Ultra) might be helping win World War II on the battlefield, diplomatic COMINT (Magic) would be essential to help the U.S.
government “win the peace.” There was a determination within the U.S. government that this time around America would not be
bullied or manipulated by its now less powerful European allies or the Russians at the peace talks that would inevitably follow
the end of the war. It would soon become clear that Clarke’s suspicions about Soviet long-term intentions were not only widely
shared by others in the military and the government—they would also become key factors in how the nations of the West would
respond to and then counter Russia’s postwar strategy.
8
To achieve these goals, however, the United States had to become as self-sufficient as possible in the realm of SIGINT. This
meant that it had to put some distance between itself and Great Britain and begin spying on those countries or organizations
that might conceivably constitute a threat in the future. The secrecy of the Russian effort was particularly intense. When
Corder-man inquired whether Russian traffic had been deliberately omitted from a target list just received by his agency,
he was told that “[reference to] Russian traffic was intentionally omitted with Clarke’s approval.”
9
But the accumulating intercepts of Russian traffic from 1943 on would yield one of the greatest U.S. COMINT harvests ever—the
program code-named Venona. Begun immediately after the end of World War II, the decoding and analysis would stretch over many,
many years (until the program formally ended in 1980). Venona material gradually and retrospectively revealed the astounding
extent of Soviet intelligence activity in America and Mexico. (Among other things, it made clear why Stalin was not surprised
by Truman’s carefully vague reference to the atomic bomb at Potsdam.) As we will see, the ultimate irony was that Venona’s
access was so valuable that it could not be compromised by using the material gathered as evidence (or even for counterintelligence
measures) against those Soviet sources (and methods) revealed by decryption over many years.