NASA STS Recordation
Oral History Project
Edited Oral History Transcript
Interviewed by Rebecca Wright
Huntsville, Alabama – 20 July 2010
is July 20th, 2010. This interview is being conducted with Robert
J. Schwinghamer in Huntsville, Alabama as part of the NASA STS [Space
Transportation System] Recordation Oral History Project. Interviewer
is Rebecca Wright. Thanks so much for coming in this afternoon and
offering all the information for the history. You’ve been involved
in aerospace history for more than 50 years, and you were here when
they were talking about some of the concepts. One of them that we
talked about was the metallic reradiating heat shield.
In the early days of [Space] Shuttle, we looked at the prospect of
using some metal plates applied like shingles overlapped on a roof.
We had a columbium alloy that was good up to about 3,500 [degrees]
Fahrenheit, but it eroded badly during testing so we had to come up
with a coating to protect the columbium. We did develop a ceramic
coating, but in order to apply it to these plates we had to go to
Fansteel Company [Fansteel Inc.] in Canada. That was a disadvantage,
but they did the coating of those shingles.
The advantage of a metallic reradiating shield certainly would be
you could fly through rain—it was much more robust than the
nonmetallic tiles [and was] not very subject to damage in any way.
It would have required a whole lot less maintenance had [one] done
it this way. There were some significant disadvantages that developed.
Because it was so much heavier than tile, we’d have had to beef
up the structure. We needed some more thrust from the engines because
it was heavier. Then, not to be considered lightly, we got a lot of
complaints from [NASA] Headquarters [Washington, DC] about depending
on [foreign] help [to] build the Space Shuttle. I think the extra
weight and the [foreign] dependence killed the metallic reradiating
shingles heat shield.
You wanted to talk a little bit about thermal protection material
for the solid rocket boosters. In the 1970s the normal [thermal] protection
material used was cork, [relatively] expensive ablation [material].
The material was laboriously applied by hand; it was hand labor. We
knew this would never meet the Shuttle schedule, at least as anticipated
at that time, so we developed an excellent thermal protection material
in the lab [laboratory] for use on the solid rocket boosters. We then
learned how to spray it instead of putting it on laboriously by hand,
and we used some just developed robotics to do the job and we got
wonderful consistency and super adherence.
Then the design guys said, “But we have to take it off for reuse.”
So I said, “What do you want? Make up your mind. You want it
on or you want it off?”
They said, “Both.” [Well] we had to come up with a way
to take it off, and it was really sticking on there good. Luckily
right about that time we had something in the lab called a hydrolaser,
which is a very high pressure water jet. We had bought one, so we
roboticized it, and it took the thermal protection material off like
gangbusters, no problem. Because it was automated, we were able to
use precise control both during the application and the removal of
the thermal protection material. That really worked out great.
A little bit on the spray-on foam for the external tank. I’m
going to let you in on a little episode here. I wouldn’t have
done this 20 years ago but I’ll tell you today. When we looked
at the external tank we had the same huge acreage problem as we had
in the solid rocket boosters. We had to cover acres of the thing with
the foam [applied] with very close control. We had completely characterized
about three of the most promising materials in house at the [NASA]
Marshall [Space Flight] Center [Huntsville (MSFC)] in the materials
lab. We did that for the [candidate] materials we could find. They
were three pretty good [ones], but we picked one.
Well, from an obscure technical paper I found out that the foam we
selected was highly toxic when combusted—so said the paper.
I thought maybe the [author] was a hot dog, I didn’t know. We
had already given the go-ahead to the contractor on that foam so it
was a serious thing to consider changing at that point. Because I
didn’t totally buy the paper’s warning we got some white
mice and we gave them a foam combustion products breathing test. The
mice kicked the bucket in an alarmingly short time so we switched
foams in midstream and by now we were robotics experts, [so] we devised
an automated system to spray the large areas very successfully. After
this though, we always did extensive toxicity testing on the foam
we received for use on the Shuttle.
I remember when we did the first one down at the Michoud plant [Michoud
Assembly Facility] in New Orleans [Louisiana]. I was standing there
with my boss and he looked at it and he said, “Bob, that thing
looks like it was done by a drunken plasterer.” I said, “We
can fix that, doc, it’s just the software.” And that’s
what it was. We fixed the software and the second one was beautiful.
On the first Shuttle flight we had some foam coming off of the tank
as it sat on the launch pad. I took a crew to KSC, the [NASA] Kennedy
Space Center [Florida], and after days and nights we found the trouble.
There had been inadequate activator application. It wasn’t uniform
enough over the surface. That brings us to the subject of foam bonding
on the external tank. That foam incident for the first Shuttle flight
was disturbing to me because we learned for the first time—and
we were material types, [and] should have [guessed] this—there
was [really] no adequate quantitative method of determining cleanliness
of the tank before spraying it.
They had that old-time water test where you poured water on it and
if the water globuled up it was dirty and if it didn’t globule
up it was clean. But that was sure not quantitative; that was a qualitative
test so it wasn’t good enough. We joined with a small instrument
company that we liked very much, and we decided we were going to develop
a quantitative diagnostic device that could tell us the cleanliness
level before bonding or spraying the hardware. We called the device
the optically stimulated electron emission [OSEE] instrument. We put
[the scanner] on a robotic system and we scanned over the surfaces
to be bonded, or sprayed, at fairly high speed and it did a great
job. You could [quantitively] tell if it was clean.
This was also of great significance to the guys building the solid
rocket motor. The manufacturers there had concerns always about safety
and reliability, because it’s extremely dependent upon superior
bonding of the propellant to-liner-to insulation internally in the
rocket. If that stuff comes loose you’re going to have a bad
day. As a result by using this OSEE, there was a great improvement
in reliability and safety for the solid motors themselves. This wonderful
little device was warmly received, and it’s now widely used
throughout the aerospace industry.
Our squabble with the Air Force. For a while we flew some Air Force
payloads in the Shuttle payload bay, and they were called IUS, it
was the inertial upper stage. Well, when we examined the design of
the IUS proposed to be flown aboard the orbiter, we noted that the
critical payload bay IUS support structure used a material that we
knew had a critical flaw size so small there was no means of detecting
it before flight. That means we could have a flaw that would fail
in flight because we couldn’t [find] it.
We immediately insisted that until we could do better than that, every
time the Air Force flew an IUS structure there had to be a proof test
before flight to make sure it wouldn’t break in flight. There
was still [an exceedingly] small risk that [after proof testing, the
flaw could possibly] grow to critical dimensions before you got it
flown, but that was a small risk. Since one couldn’t detect
the flaw that could fail in flight, this was just too big a risk for
NASA to take [without a proof test]. That made the United States Air
Force a bunch of unhappy campers but we stuck to our guns.
It happens that in the lab at that time we had a fully characterized
new alloy. We developed [and characterized] alloys in the materials
lab at that time, and this one was called Custom 455. It was an excellent
substitute material, but turned out that nothing the size of that
IUS support structure had ever been forged before. They couldn’t
forge anything that big so we worked with a contractor to build such
a large forging. It was accomplished and it became the material of
use for the IUS in the payload bay, so it worked out pretty [well].
This proved completely satisfactory and improved safety without a
proof test before flight. That brought peace in the family.
took a few years ago to get that, right? Did this take you a while
to determine this?
Yes. At least months, approaching a year. In the meantime they still
had to proof to fly, but after that no more proof testing. We were
comfortable and they were too, so that was an internal peace between
One of the really difficult challenges was to weld the big external
tank. We had developed a technique called variable polarity plasma
arc welding. The external tank at that time was made from 2219 aluminum
alloy, and it was a fairly new development back then. Not much was
known about its weldability, especially in such large structures.
Nobody ever made structures [of that size] with that alloy. At first
the welding process that we selected just couldn’t cut the mustard,
it wouldn’t do the job, but we did several things.
We added additional diagnostics to better control the welding arc,
we beefed up some of the [welding machine] components because they
actually failed during the extremely long run times—it took
us three and a half hours continuously running the machine to make
a circumferential weld around this tank. The guys making the welding
machines hadn’t counted on that. They were talking about 30-minute
duty cycles or something. So we went in the welding machines and altered
them ourselves to make them able to run three and a half hours without
stopping. Then we used some design-of-experiments smarts to optimize
the weld current wave shape. When we did those things we finally had
a process that worked great.
We were able to perfect the welding at the Marshall Center because
we had a large tower and we tested welding techniques on full-scale
hardware in that tower, so we could handle the 33-foot diameter tanks
when we tested. But even at that, at one point we couldn’t weld
at all. We were down three weeks one time, we could not make a weld.
I didn’t get too much involved at first and then it got worse
and worse so I went over there myself and we examined that thing and
we found that the cable that took the [welding] current from the weld
back to the [welding] machine was continuously moving as the tank
was rotated to weld. So we designed a fitting to allow that cable
to stay in a fixed position at the center, at the bottom and it never
moved and that solved a nasty problem. No trouble after that.
simple, but what a difference it made.
Simple, but you had to understand the physics involved.
I got much involved in the decision of how we would prevent the components
in the solid rocket booster from being destroyed on recovery. The
parachutes of course brought it down but [it was] still moving pretty
fast, and every time that thing splashed in the ocean it buggered
up [a lot of components] in the aft skirt. So we investigated several
foams and we finally found one that was very good at cushioning and
we sprayed the components with this cushioning foam and after that
we could reuse the hardware.
In the area of the engines, the turbine blades and bearings were problems
at the start. Serious problems. I think what caused the problems were
the enormous temperatures and pressures that [the] engine [had to
sustain]. Other engines were not running at those temperatures and
pressures but to get [both] the kind of specific impulse we [needed]
and low weight [too], that’s how the design ended up. So now
it’s up to the materials guys. “How come you can’t
get me something that can stand these temperatures and pressures?”
With the turbine blades it became immediately apparent we had to do
something there, and I started an in-house, in the materials lab,
metallic materials division, a program to develop a suitable turbine
blade material. We had one which looked promising, and then we tweaked
that thing over a period of many months. Tweaked and tested, and we
finally tweaked it into the kind of durability and usability that
we expected. We did that pretty much in house. In the bearings, they
were another special problem. Suffice to say, again, we had something
going in house that proved to be the ultimate solution. Materials
got so important that management decided that the materials people
had to sign off on all the MSFC drawings for Shuttle hardware, so
they never got issued until the materials people signed off.
I guess you could say I was a prime mover in the decision to go to
a better aluminum alloy for the external tank. We called that one
the super lightweight tank because it used a new alloy, 2195. That
was a very successful program. We had some problems, but it was finally
successful and saved about 7,000 pounds of payload by switching to
that new alloy. Along the way we also developed and applied some hydrogen
environment embrittlement data and stress corrosion cracking data,
and we made two major specifications in those areas. They’re
[still] used by the aerospace companies.
In 1990 I headed a leak team for NASA when the Shuttle fleet was grounded.
That was a tough problem and lasted several months. I [also] headed
a NASA failure investigating team on the big casting pit fire and
the mix pit fire we had at a contractor in Utah. Huge fires, big damages.
We found out what they were however, and we used a fault tree to do
that. I was [a dedicated] user of the fault tree.
I was also the solid rocket motor contingency team leader [of the]
Marshall Center. When the Shuttle came along, [MSFC] named contingency
teams and contingency team leaders for each of the major elements;
the engine, the tank, the solid rocket motors, payloads, etcetera.
[Those team leaders] were committed to take action immediately, [one]
didn’t have to call and ask, “Boss, should I go?”
When anything happened the team leader convened the contingency team.
I was the solid rocket motor contingency team leader, and so I did
what we were supposed to do and I set up the fault tree and we worked
four months on the [Space Shuttle] Challenger [STS 51-L] accident
Back in the Saturn [rocket] days, [the] Apollo program had a definitive
components development and test program, that ran right along with
the major program. In fact it had to precede it a little bit because
what came out of the components development and test program was what
you then put in your [final] design. Sometimes we had as many as three
contractors doing in-house work. It’d be ongoing at one time,
and we’d pick the best of the three. [One] really had a pedigree
in the system when that [was done]. That [meant] that the final selection
had a significant development and test pedigree, and when the whole
system was complete and put together, all the elements played together,
largely because of that pedigree. I think it was very beneficial.
Of course the key to that kind of an approach just [depends on] funding.
[One] cannot do that without [adequate] funding. I guess in Shuttle
we never really had the funding that Apollo had. It never was there,
but [on the other hand] there were some advantages that the Shuttle
had in hand. We learned for instance a lot from Apollo, and that proved
very useful for Shuttle so it just carried on into [that] program.
At the same time, new analytical methods were being devised in areas
like computational fluid dynamics, we had new stress analyses and
improved materials and processes. Even so I remember at one point
we finally decided we really did need a fracture control program.
We were trying to get by without it but we needed it after all. We
got a late start on that one but we caught up.
Early-on the turbine blades were already a problem, so we began, as
I mentioned, an in-house project on the turbine blade materials and
we tweaked the alloy several times over a period of [many] months
to get the durability we needed for Shuttle. We finally made it and
we passed that along then to the contractors. As I indicated earlier,
it was the very high temperatures and pressures that drove us to these
extraordinary efforts. [One] just couldn’t take something off
the shelf, no alloy off the shelf—it just wouldn’t hack
it. [One] had to tweak it and make it fit. We ran many tests until
we finally hit pay dirt. Then of course, as I indicated, we passed
that along to the contractor.
Turbopump bearings were an even bigger problem. Sometimes we had to
change out bearings after only one full duration run on the test stand,
and we burned up several engines. Changing after one full duration
run is like your car is good to drive to Birmingham [Alabama], but
that’s as far as it goes. It just wasn’t acceptable. In
order to test the bearings we had to use whole engines at the [NASA]
Stennis [Space Flight Center, Mississippi], and that was dangerous
and expensive because here you have something in very much a development
stage, and now you’re risking a full all-up engine on that test.
So we built a bearing tester. It was just a simple system that sat
aside and didn’t have to have the whole engine attached to run
the test, and that speeded everything up. We would run tests in a
couple days and change the material and put them in and run them again.
Boy, it saved a lot of money and certainly expedited the solution
of the problem.
When we got that thing going, we did find immediately that the bearings
as they were in the design were undercooled. They were running too
hot, they needed cooling. Well, we supplied additional cooling, and
it got better, but not good enough. Fortunately, we did a lot of anticipating
in those days, “What might we get into?” [We] had a lot
of experimental stuff going in the lab to head off possible problems.
In this case we had anticipated this to some extent and we had a lab
project going on bearing balls made of silicon nitride, a ceramic.
They were superb.
There was no interest at all from the United States bearings makers,
none whatsoever about the silicon nitride. I couldn’t get them
interested to provide what I needed, so we had to get the best silicon
nitride we could find, and it came from Japan. Then we sent that silicon
nitride material to Germany, to of all places Schweinfurt. [Now you
may not be] old enough to remember Schweinfurt in World War II, but
Schweinfurt was where the Germans made all their bearings. We [the
US Army Air Force] bombed Schweinfurt and that ended the war months
earlier because they had no more ball bearings. I thought [our bearing
arrangement was kind of] ironic. Anyway, the first user of silicon
nitride in cryomachinery was the Space Shuttle main engine. That was
a first. Nobody in the world had ever done that before.
I put my reputation on the line in the beginning to sell the silicon
nitride. I was sold on it. We had test data out the kazoo; it looked
good. I went to the Center Director and he said, “Oh you [surely
wouldn’t] put glass balls in these turbines.” I said,
“It’s not glass.” We took a piece of steel, put
a silicon nitride ball [on it], and hit it with a steel hammer. Nothing.
That convinced him. The stuff worked so well that they’re now
used widely in high performance aircraft. In fact one aircraft company
put those silicon nitride balls in their bearings, they flew the airplane,
they shut the coolant flow off to the bearings with the plane in flight,
and [the bearings] were still good enough at high temperatures [so
that the pilot could] land the airplane without incident. That’s
how good they were.
It’s still a little tough for me to talk about Challenger. I
mentioned we had contingency teams for each of the elements on the
Shuttle. I had the contingency team for the solid rocket motor, so
[setting up the fault tree] fell in my lap when we had Challenger.
We went into action immediately, which we were allowed to do. We set
up the construction of the fault tree to guide our part of the investigation,
but that investigation, the whole thing, I still have bad thoughts
about it. It was especially disheartening because of the loss of the
astronauts. [None of my previous] problems were of that nature. It
was an exhaustive grueling investigation. It started in January 
for us and it lasted until June, day and night.
At least half of my team members came down with flu, colds, depression.
I learned you can’t go four months day and night without suffering
the consequences. I believe our efficiency must have dropped to about
40 percent after four months or so but we just kept plodding on. During
one of the many 10:00 evening dinners I was having [at home] while
that was going on, my mother, who then lived with us, promised to
make two big pans of rolls [every weekend] for the team [as a morale
lifter]. And they were big pans. She did until one weekend she got
sick. I’ll never forget, as I came to work that Saturday I was
personally roundly, castigated. Heads popped out of doors and shouted,
“Where are the rolls, Schwinghamer?” It was a morale booster
to say the least.
to look forward to.
I mentioned earlier about the hydrogen leak grounding the Shuttle
fleet in ’90. After an unsuccessful tanking test and three aborted
attempts to launch the [Space Shuttle] Columbia [OV-102] in ’90
the entire Shuttle fleet was grounded. Since we didn’t know
what it was or where it was, we couldn’t risk flying any of
the other Shuttles on the premise that only the Columbia has it, the
others don’t. So the fleet was grounded, which was a good decision.
The NASA director for Space Shuttle named me as a team leader, and
I picked up the investigation team myself. I named the team members.
J. R. [John R.] Thompson, who was [NASA] Deputy Administrator at that
time, sent me to Kennedy Space Center with a one-way ticket. He says,
“You’ll get a ticket back when you solve the problem.”
That was a little bit of an incentive. We were down there from September
to December. We set up a fault tree immediately, with instructions
to everybody in the system there that there would be no confirmatory
tanking test until the complete gamut had been run on the fault tree
master plan. Project managers, they get antsy. They can’t wait,
they want the answer, and I kept saying, “No, no, no, not until
the fault tree gamut has been run.”
This was a big operation. It involved three NASA Centers and many
many contractors. After about six weeks the fault tree was complete,
the tanking test was successful, the major leak was found, and even
other leaks were found because we used the fault tree. Those were
repaired, and Columbia was the soundest leak-free orbiter at that
time in the fleet. We [all developed a special affinity for] Columbia
[due to] the time we spent down there with our team. The eventual
loss of Columbia [STS-107 accident] was really a blow because we had
become [so] intimately acquainted with that bird during the leak investigation.
That was the end of the kind of things I was used to doing.
have a couple questions, I made notes as you were going through. You
said you went out to Utah, about the fires. Can you share more information
about what you went out to investigate with those fires and problems?
It was early in the Shuttle Program; it was late ’70s or early
’80s. I’d just have to go pick a team and go and we’d
go and stay there till we found [the cause of the failure—the
smoking gun]. Most of those things took several weeks typically. You’re
away from home that long, and I think what I learned about that was
[that one] can’t just go in with your investigating team and
[dominate the investigation]. I made sure when we set up the fault
tree, at every block or element in the fault tree the contractor [also]
supplied who he wanted to be in that block. It ends up being a team
effort when you do it that way.
If [one] just sends an investigating team into an existing organization
and trys to work it that way it’s very difficult. I learned
early-on that every block on that fault tree [needed a person’s]
name, and it had two names, it had a NASA guy and a contractor guy.
That encouraged their cooperation, also contributed knowledge that
you wouldn’t have been able to dig out, in every instance on
every subject, from the contractor. That’s why [the] fault trees
were successful I think.
guess you start in a way with objectivity because of the way that
the tree is laid out, so that you have to supply facts?
Yes. The top block is always: this is what happened. [It must be comprehensive]—this
might have been the problem, [or that] might have been the problem,
or this other thing—all conceivable things that might have influence.
Then there were subheads that would be further broken down. You’d
name an expert—if it was structures you’d pick the smartest
structures guy you had and so would the contractor, then they’d
wade down through, “Well, could it have been stress relief,
could it have been fatigue?” All those things, each one of those
questions get either exonerated or indicted. “Oh, couldn’t
have been that.”
“Prove to me it couldn’t have been that.”
“Here’s my test report and here’s the test data.”
“Okay, wasn’t that.” [One finally] gets down [near
the end] and here’s the one that’s indicted. “Well,
how do you know that that’s the case?”
“We did an analysis and it looked bad, we ran a confirmatory
test and the test showed it was bad.” [One] just [has to] wade
down through the whole system. If they’d been able to do this
with the oil leak in the Gulf [of Mexico, BP 2010 oil spill], they’d
have [gotten to the culprit earlier].
believe when you were doing the investigation of the solid rocket
motor that some of what you discovered was that they didn’t
need to change the materials.
I think you’re talking about the O-ring situation on the solid
rocket booster failure. We had a material that we had selected because
it was the toughest O-ring material we could find, which meant that
when you put the segments together you were not very likely to damage
the O-ring, and it had extremely high temperature capabilities. In
our opinion, after months of testing of many materials, silicones,
all kinds of rubbers, we finally decided that this one material [Viton]
was far superior.
After the Challenger failure the O-ring was involved. Really wasn’t
the O-ring’s fault, it was a deficient mechanical design, but
because the O-ring was involved there was a hue and a cry, “Let’s
get rid of that O-ring.” I kept saying, “You can’t
find a better one. There is not a better one. I’ll show you
the data. I’ll show you the rigidity, the toughness, the temperature
capability. You can’t get a better O-ring material.” I
finally prevailed, and they finally listened to reason.
But you know how there’s [always] an overreaction when there’s
an accident, “Let’s do this, don’t want to see any
of that anymore.” There was a pretty strong contingent at very
high level in NASA, and I just kept lying down in the flame bucket
and saying, “No, no, no, don’t change this O-ring material.”
That struggle went on for several months too. Early-on the people
at the Marshall Center supported me. I had very good luck with that,
they always supported me in whatever argument we got into. The Center
Directors and all the rest of management were fine. [But it] went
on and on, and then finally—it was even suggested by the NASA
Administrator that we change the O-ring material. That’s kind
of impelling when you get that kind of suggestion, but I kept saying,
“No, no, no, don’t do this.”
I think also it became more evident as we analyzed our design. We
redesigned that whole thing, and we had a test facility where we could
test those O-rings during a hot firing at Marshall. After we saw that
the clevis joint was mechanically [deficient], then the heat came
off a little bit on changing the O-ring. Initially everybody thought
it was a lousy O-ring. There were some people too willing to want
to bet on that. But had we only changed that and not changed the clevis
design we’d have had trouble again later, although I doubt if
we’d ever launch at 27° [degrees] Fahrenheit anymore after
also mentioned something about—I think you called it a fracture
control problem. Can you talk more about those?
Yes. Fracture mechanics is a method of examining your materials, in
their design, to see whether they will meet the endurance requirements—will
they fail before [reaching the design life] or not? In order to do
that you’ve got to get off to the side and get extensive materials
data [so one can] plug these material data into these fracture mechanics
calculations to determine whether fatigue [life] is going to be adequate.
It’s a [comprehensive] method, and you have to gather the data
through the whole system both for the materials and for the structures.
That’s time-consuming, it’s expensive. We had it in Apollo
and we decided at first not to have it in Shuttle.
Then as we went on, from a materials perspective it became more and
more apparent we were stretching the limits in some [situations].
“How far could [one] go?” Well, fracture control program
would tell [one] how far [one] could go. [At first] we weren’t
going to do it and then it wasn’t too long before it became
evident this is not [just] a nice thing to have, it’s a necessary
thing to do. So we did that, we had a fracture control program for
the Space Shuttle.
of what you had mentioned was the need for automation of processes.
How were you able to, during the process of trying to meet schedules,
find better and newer ways to automate and structure those processes
where they were superior?
I’ll tell you what the secret is. Anticipation. [One has] to
seriously sit down and consider what [will be necessary]. What your
structure should do, what [one’s] materials should stand. Then,
at that moment, [one does] the best [one] can with what [one has],
whatever is in the system. Then [one] looks at it and says, “The
margins here are less than I’d like to see,” so [one]
starts a program off to the side to improve that situation. If nothing
happens you can elect either to leave it as it is, good enough, or
you can still change it and get a higher factor of safety. [One] can
make a choice. A lot of it has to do with scrutiny and anticipation.
If [one delays] it falls on [one] like a ton of bricks, it [may be]
pretty late to start a program to improve that situation. You will
start a program but you’re late, you’re a day late and
a dollar short. I always prided the Marshall Space Flight Center—and
all the [center] elements [anticipated] that. We urged people to use
ingenuity and think outside of the box. “If so-and-so happens
what am I going to do? What’s the backup position?” Of
course funding constraints put a damper on that, because there’s
a limit to how far you can go outside the box if you don’t have
the money to do it. I attribute that ability at the Marshall Center
to everybody thinking out of the box and anticipating what might happen.
Sometimes it did, and sometimes we headed it off. I know we headed
it off on bearings and turbine blades.
I knew the bearings were going to be a problem from day one. If we
hadn’t had that ongoing program in the lab and brought the silicon
nitride along at the same time it wouldn’t have been there.
And not everybody can do that. It takes ingenuity and most people
can’t or don’t want to think outside the box. It’s
uncomfortable. When [one] finds people like that in the organization
[one] cultures those, you treat them nice, because not everybody can
do it, but we had a bunch of people who knew how.
were able to start your materials lab. Are these the type of people
that you pulled to be in that, your system of finding those answers?
[Let me say that I was very fortunate in that my two predecessors
had already created a disciplined, capable and smooth-running organization
when I took over.] And yes, we had a great esprit de corps. Part of
it comes from being under the heat of a requirement. That makes you
all pull together. But I always went out of my way to cultivate the
mavericks. I got a lot of good stuff out of mavericks. Sometimes [people
would] tell me, “What do you want to hire that guy for? What
do you want to transfer him over to you for? Are you kidding?”
But they had [special] abilities, and [one has] to treat them accordingly.
[One] can’t treat everybody the same. Some people are satisfied
to say, “Go fix it, Joe,” [Joe’s] tickled to death
with that. Other people have to be coddled a little bit—they
got to be stroked. [One has] to recognize when [one has] people like
that. Then the other thing is, I was always a walk-around manager.
I didn’t stay in the office very long. At 4:00 I went to the
office and then I stayed till 6:00 or 6:30, but during the day I’m
walking around. “How’s it going here? What does that tensile
test look like on that material we’re testing?” Walk around.
I’m really proud of one thing. I was in the materials lab for
20 years before I went to Associate Director’s job, and I never
had one union complaint out of anybody. [There were always some complaints,
and the legal group had one department that spent a lot of time arbitrating]
those complaints. [But] I didn’t have one in 20 years. I think
it’s because you have to recognize people are different. And
[one has] to be honest, upfront.
I thought the worst thing that ever came along was the leaving [of
important interpersonal] messages on the telephone. [One has] to talk
to people [eye to eye]. It’s especially so if it’s a sensitive
interpersonal relations thing, like “I wish you’d stop
doing so-and-so.” I always tried to tell them why I thought
[the action taken] was necessary, and then, “[Or] do you disagree
with that?” and then we’d discuss it. But it’s so
easy to pick up the phone and leave them a message and say, “Joe,
I don’t like what you’re doing, [do better].” No
good. It’s one on one, that’s how [one does] that. We
really had it [going that way at the center].
That was something that was always prevalent at the Marshall Center.
[Wernher] von Braun was a hands-on walk-around guy, and he talked
to [one]. He did a lot of that. It was a team from day one and it
stayed that way all the way through Shuttle. I [think] it’s
would have to think that with your background there were a lot of
important priority issues all going on at the same time. Was there
one that was more prevalent? Or was there one that sticks out in your
brain as more time-consuming, maybe just more important for you to
The whole thing was a kind of a problem du jour. Whatever was hottest
is what we did. I must confess the one that was potentially the most
damaging to [my career] was that leak investigation. What I haven’t
mentioned [before] is that shortly before the hydrogen leaks were
discovered [in Columbia], they had moved an orbiter and some of the
people responsible for working in the aft, where the engines were,
had neglected to do something with a beam and it came loose, and rattled
around and did damage in the engine compartment.
I [was aware of] that, and [I knew] they got a seven-day unpaid vacation.
Here we are a couple weeks later, I’m down here with a tiger
team trying to find out why we can’t find this leak. I lay awake
nights in the motel thinking suppose there’s a hardnose in there
that says, “I’ll fix you for that seven weeks’ [un]paid
vacation,” and he takes his wrench and he cracks a valve and
lets it leak at the right time and shuts it off so you can’t
find the leak afterwards. I thought, “Oh my, it could be sabotage,
we’ll never find it if it’s that.” I spent days
out at the pad with those guys working in the aft, and I crawled around
in the aft with them, on Saturdays and Sundays I was out there with
them. I finally just intuitively decided that these people’s
heart was in the right place. That is not what is happening.
Of course I had the fault tree going down all the blocks at the same
time, but had it been a sabotage the fault tree wouldn’t find
it. They were all contrite and they didn’t have any hard feelings
about their seven-day unpaid vacation. It took a while for me to be
with them and get [with] them on a first name basis. They’d
come up to me and we’d eat lunch together. [One] learns a lot,
a lot you don’t know otherwise when you eat lunch with them.
That was the one big concern. I thought, “Oh my, my career is
dead.” I had a pretty well established career for a troubleshooter
by then. I thought, “Oh my, if this has been sabotaged I’m
dead.” But it wasn’t, fortunately.
mentioned you’d been to the Cape [Canaveral, Florida] and that
you’d gone to Utah to do some investigations there. Did you
find yourself at the Michoud facility or any other places? Did you
go down to Stennis for any of the tests that were there as well during
the time that you worked in the Shuttle program?
For instance at Utah when we had those terrible fires, the first day
I got there I said, “Okay, I need the name of everybody that
was working when this occurred.” Then I took the time to take
each individual that was working the day the incident occurred. I
took them in a room privately just one on one—no overseers,
no nothing. I talked to them, and tried to find out what they might
know about the incident because when we made the fault tree you had
a whole bunch of possibilities that either had to be exonerated or
indicted, and I thought if I talk to the people I may get enough information
that some of those can automatically be removed [as potential causes].
They [might] remove my concerns therefore.
I’ll never forget. [I] got this one guy in there, and he came
in and he sat down, and he was obviously nervous to begin with. He
was really nervous. I started to question him, I said, “Just
tell me. I’m not looking for an opportunity to blame somebody
here. Please tell me what you saw that day, the day it happened.”
He said, “I was afraid to say this to anybody. As I was working
there I could see that this cart that was carrying propellant was
running on tracks and every time this wheel came around on that track
it made sparks.”
I said, “Were the sparks falling into the pit with the propellant?”
He said, “Yes.” There it was. I didn’t have to run
the fault tree very far after that. If I hadn’t talked to him,
hard telling how many hours we would have spent. I guess eventually
we would have [come to the proper conclusion] after two or three more
months. But he said, “Yes.” And he cried, I felt so sorry
I patted him on the back until he moved on and said, “This is
between you and me. Don’t worry about it, nobody’s going
to come down on you for this. I’ll see to it that they don’t.”
Then I talked to [the company] management, I said, “Come on.
This guy saved us weeks and weeks. Leave him alone.” It wasn’t
his fault anyway. It was the design of the system that allowed this
propellant to come out of the hopper, fall on a steel rail, and then
comes the steel wheel—that wasn’t this guy’s fault.
He could have been criticized a little bit for not coming forward
earlier. He was afraid, he was afraid they’d blame him.
I really had [invigorating] times on those failure investigations,
were able to learn and investigate and probe all different types of
areas within the Shuttle.
One of the guys at the Aerospace Safety Advisory Committee in Washington
said, “Schwinghamer, you’re a technical gadfly.”
I said, “Okay. I don’t mind that a bit.”
of the other things that you mentioned was computer-aided design,
computer-aided manufacturing techniques in house. Can you talk the
impact of that?
I had both [the] materials and processes lab. The guys in processes
lab did their best to keep up with the latest state-of-the-art in
things like automated controls, numerically controlled machinery,
high speed machining, things that were just new and developing—and
new welding techniques. We had the money to spend to do that. We did
processes research and materials research. Very frequently then because
of that capability we would find things that fit very well.
Sometimes it allowed us to head off a problem. Many times it helped
us to improve the process. We also became very knowledgeable about
two things that are awfully important if you’re going to manufacture
something. We did process sensitivity analysis. That’s very
important, process sensitivity analysis. [One] determines the limits
within which your process can operate and the limits so that you don’t
get outside the processing limits. [One does] process sensitivity
analysis that tells you where the limits on the process are, then
[one does] statistical process control. [One] sets controls based
on what [one] knows those limits are and [one] sets those controls,
and data is extracted [continuously] where these various operations
are going on at these various machines, and it’s looked at statistically
to guarantee [the process] stays within those limits.
We got into that early, we did that here. We insisted that all our
contractors do statistical process control and process sensitivity
analysis. Those are two kingpins if you’re going to build anything.
[One must] do that. Otherwise, if [one doesn’t] do the process
sensitivity analysis, [the process will] be making widgets and all
of a sudden the [lousy] widgets, they don’t work anymore. Why?
Because [the process] got outside the limits.
The parts themselves [one was] putting together had variability, and
sometimes that variability stacked up against [one]. Most of the time
it’s random but sometimes it stacks up against [one]. By doing
statistical analysis and setting statistical analysis on the machines
that are making the project, [one] can tell whether [the process is]
in or out. [One] can tell [the process] made a bad one even before
you test it. The process guys got very fluent at that. Those were
the two things that we hammered with every contractor, process sensitivity
analysis and then [use] that data to do statistical process control.
Now, they don’t like that. That’s like Big Brother looking
over [one’s] shoulder while doing [the] job, but all the successful
companies today use that technique. The only thing wrong with it is
it’s expensive. When [one] starts doing the process sensitivity
analysis, that comes way before [the process] starts cutting chips
and making parts, and that takes time. [One does] things like design-of-experiments,
which are statistical methods to determine which of these variables
are the important variable. [Finally one has the important variables
under control.] It all plays together. It’s all time-consuming,
it’s all expensive but when [one does] that what [one] builds
will be a quality product.
do you think was the biggest challenge that you had to face during
the time you were working on the Shuttle program?
It’s hard to say. I told you before, it’s the problem
du jour, whatever [it is] that [day]. The leaks stand out, bearings
[too]. It just all runs together for me, and nothing really irritated
me to the point that every day it bothered me. Never had anything
you spend much time at Michoud with the external tank?
Yes, especially on the external tank welding and then the implementation
of the new 2195 alloy. I spent a lot of time at Michoud, and also
had two really good engineers that saved that program. He called me
one time. I’ll never forget, he says, “You got to come
I said, “My goodness, it’s two days before Christmas.”
“You got to come.” So I went down there. He said, “I
wanted to show you something.” They had welded up the first
bulkhead and we ducked under it and looked up there and here was the
weld and I could see daylight for 12 inches. He said, “We’re
in trouble.” I said, “Yeah, we are.” So there went
the Christmas vacation. But like I said, it was really the problem
to identify it and solve it.
Yes, identify it and solve it. Fault tree if necessary. I always told
the [project] guys about fault trees. They’d say, “It
[delays] the project [schedule]. I don’t want to do that.”
I said, “Look, if [one is] driving [one’s] car and the
wheel comes off [everyone] knows why you came to a stop. [One doesn’t]
need a fault tree for that. If there’s a subtlety going on inside
[one’s] engine [one doesn’t] understand, [one] needs a
fault tree.” I tried to give [project managers an] analogy why
[a fault tree] was necessary, because they are time-consuming.
The worst of it is, I kept saying, “I [must] have stress analysis
Project manager says, “I can’t afford to let him work
on your fault tree for two months.”
Then I’d have to go to higher management and say, “I can’t
do the fault tree if I don’t get the right people.” The
fault tree is only as good as the people that populate it, the people
that are doing it. But if you pick the right people, you’ll
get the right answer.
of the statements that I read that you had mentioned was the reusability
element plus durability impacted a whole different way you put the
Shuttle together based on how you did the Apollo. Did that keep coming
up during all the factors when you were solving these problems, the
fact that it had to be durable and it had to be reusable? Did that
continue on through the program?
Yes, there was really a hard drive on reusability and durability because
before [Shuttle], we threw [our stages of the rockets] away. So [reusability]
was an ever present consideration. There wasn’t a day [gone
by that reusability wasn’t considered]—and of course [this
aspect] manifested itself in having to have materials that could be
used longer, could be used at higher temperatures, that had to have
higher strength without sacrificing the toughness. Usually when you
get strength in a material you sacrifice toughness, but in the case
of the engine we had to develop things that had both the toughness
and the strength. And durability and reusability, it was just there
every day. [One] knew [one] had to do better because we had to reuse
it. It was a very evident aspect of the whole problem, and it drove
a lot of tests and studies that were required because of that aspect,
you were going to make 400 tanks. In your mind did you believe that
was an achievable requirement based on that plus how you were designing
the rest of the components?
Well, 20/20 hindsight is better on Monday morning. We were all young
guys, we thought we could do anything. And we pretty much did everything,
except for a few things, few you don’t want to remember. But
at that age nothing’s too tough. Everybody was all about that
same age. I couldn’t do that anymore, I’m 82 now. Man,
I couldn’t do it.
mentioned one time that flying the Shuttle is very scientific. Could
you expand on that, why you feel it’s scientific?
After having had some insight into the amount of intelligence required
and even transcending engineering into science, it almost became a
science aspect because of the sophistication of the machine and the
way it was put together and the high performance that was driven from
the machinery. To me, that smacked almost of science rather than bridge
building, and that’s why I alluded to it that way.
It was an absolutely outstanding accomplishment. The horsepower-to-weight
[ratio] on the engines—it’s just enormous. It’s
like 0.5 for a high performance racer, and ours was over 100 or 200
to one, horsepower-to-weight ratio. All of that doesn’t come
for free. You’re working in a terrible environment, and you’re
stressing things beyond what you normally would stress them to. So
it almost became science more than engineering in many respects.
remember reading that you had said, “It’s not a truck,
it was a race car.”
you have some other notes of contributions that you wanted to talk
about today? I think we went over the list that we had prepared, but
I didn’t know if there were some other areas that you can think
About Shuttle specifically you mean? I had to spend a little of the
time on the other things going on, Skylab and a few other things.
had developed a hammer, could you share with us about that tool?
I didn’t say too much about it because [although] it is still
used, the most extensive use was in Apollo, Saturn V. I think those
bulkheads were 33 feet in diameter, and thick. When [one] makes a
weld the weld metal becomes molten. Then when it solidifies it shrinks
and that puts tension across that weld joint. So when they were welding
up the bulkhead on the first Saturn V, the first stage, the S-IC stage,
they made the measurements and the bulkhead was down two and a half
inches from where it was supposed to be. The stress guys were having
a fit about, “Oh my, our stress calculations, they’re
based on this, not that.”
I had made contact with a plasma physicist in California, and his
name was Dr. Waniak [phonetic]. He worked for Atomic Energy Commission
for a while and then he had his own company. I read an article of
his—he was trying to design coils to maintain a high magnetic
field inside, and [he was blowing his] coils up, just with one pulse
of electricity through the coil. So I thought, “I wonder what
we could do with that about moving metal. If he could blow these coils
apart, I can move aluminum I’m sure.” So I got on a plane,
I went and I talked to him. I got enthused about it, and he got enthused
about it. I finally gave him a contract and he did some of the theoretical
analyses [more science]. Then I built the coils, based on his calculations
largely, and I learned that I could stretch [electrically conductive]
materials in every dimension isometrically.
Normally when [one] makes a weld and it shrinks like that, [one] comes
along with a big roller and [squeezes] it. [But] that’s only
two-dimensional stretching, [and] my coil stretched it in three dimensions,
thickness, crossways, and every way. Back in those days if you had
an idea you were allowed to pursue it. So I put some guys on it and
we developed these coils and tested [them] on the metal. Then we tested
[them] on some welds, and we found out that it stretched the welds
in three dimensions instead of just two. [Conventional approach is
to] roll planish it—that’s what [one] calls that—it
does it in two dimensions but [not three, as the “hammer”
So I said, “Look, I believe we can chase these meridian welds
on this big bulkhead and stretch that thing.” We had a big capacitor
bank. It was about as big as two [large] tables, with capacitors in
there. We designed electrical stuff to charge those capacitors, then
we would discharge those charged capacitors through this coil and
we’d get 20,000 or 30,000 amperes of current that would last
only one and a half milliseconds, one and a half thousandths of a
second. But boy, it would really stretch that material.
[But] I had a selling job. Here’s this big thing sitting, and
the guys are all done with it, and they’re standing there and
I come in there and say, “I think I can fix that.” There
was a sound and round discussion about, “You’re going
to let him get on that thing with that crazy tool that’s never
been used before?” They let me get on it, so we started chasing
one after the other, one after the other. I never will forget, it
was so surprising. We got about three quarters of the way up on the
last meridian weld, and that big 33-foot diameter thing went ka-ching,
and it popped right up there where it was supposed to, because we
took all that [shrinkage out of the welds]. We did that on all of
[the bulkheads] after that. In fact when we made the gore segments
[which make up the bulkheads], if they weren’t in contour we
used to hit them [too] with this magnetic hammer. The guys called
it the swinghammer.
love that. That’s great.
I got that thing patented, and they used it in the ship factories
at Pascagoula, Mississippi and they used it someplace up on the northwest
coast building their hydrofoils. We had all kinds of stuff going on.
But that preceded Shuttle; that was during Apollo, Saturn. It’s
been a great time.
My dad used to come down for vacation. Sometimes he didn’t have
a very good one, because I remember during Skylab I was out [at the
center] all day, and several nights; I never spent any time with him.
[Earlier] I took him fishing; he liked to fish. We went down the Tennessee
River to fish. On the way home I remember—I think he must have
been about 76 then—he said, “It’s getting harder
and harder for me to go to work every day, it’s really a job.
But you seem to get up and go out there like you’re anxious.”
I said, “Man, I’d like nothing better than to get up in
the morning and go to work.”
He says, “You don’t know how lucky you are.” I never
forgot that. That’s one of the benefits associated with it that
you can’t put a price tag on.
must have been an exciting time to be through with ASTP [Apollo-Soyuz
Test Project], knowing that Shuttle was coming along. Not as fast
as maybe everybody would like, but knowing you were working through
those problems and a whole new program.
We had goals, objectives, we had schedules, and we knew where we were
going. That’s a big difference. I don’t know if we know
where we’re going today, but we did then. We absolutely knew
where we were going, and believed we could.
back now, do you think the flyback booster would have been the better
way to go?
I was really sorry we didn’t get the flyback booster. But again,
it was money. A flyback booster, we estimated its cost at that time
would have been I think around $12 billion, maybe $14 billion. They
gave us $5.5 billion to do the Shuttle, so you did what you could
do with $5.5 billion. But oh, it would have been so neat to have that
first stage separate, fly back and land, tank up and be ready to go
again two days later. I think it was within the state of the art,
or was close enough [so] that it could have been developed into the
state of the art. We all wanted that. But the money just wasn’t
there, so as engineers you do the best you can do with what you [have].
retired in ’99.
Yes, and then they kept calling me back. So I guess I spent most of
my time on Shuttle [after I retired]. I worked on several of the problems
that they were having with Shuttle. Then of course when we lost Columbia
I spent a lot of time working again with them on Shuttle and that
program. I spent a lot of time at Michoud with them. They got back
into some foam adhesion problems on the external tank. We were losing
foam in flight, so we constructed a fault tree for that and then we
found some very significant improvements. A little bit still comes
off, always will, but nothing of any significance anymore. I worked
a long time with them on that and then several other engine problems.
This year in January I decided it’s about enough, “I’m
82, I think I’ll quit.” So I haven’t done much of
anything with Shuttle anymore since January.
you like to talk any more about the work that you did with Columbia?
Were you specific on the foam or were you doing something else for
them in the Columbia accident?
No, it was all aspects of the accident, what did happen. There was
a fault tree for that too, but after we looked at those high speed
movies it became apparent what hit the wing leading edge. That was
[suspicious]. For two weeks I kept saying, “You can’t
let that foam do that.” You know what they kept telling me?
It was a refrain, “But Bob, it’s only foam. Gee, foam
is soft enough.” I said, “Yeah, going 700 miles an hour,
kinetic energy, 22,000 foot-pounds. You know what my deer rifle kinetic
energy is? 3,000 foot-pounds. Now does that give you some idea the
difference, how important it was for that foam hitting that leading
They finally ran some tests. “Yeah, yeah, yeah, it could do
it.” It did do it. But I looked at that and I said, “Man,
I don’t need a fault tree for this. Look at that.” They
had the pictures. As the vehicle ascended you could see that foam
come down and strike that leading edge of the wing. Then after that,
how to prevent that from happening, we went through all kinds of changes
in the design to preclude that the foam should come off. Then we put
some materials where the foam had been and changed those around.
That went on for a year or more I guess, but [frequently] they’d
call me, and I’d pack up and go. Immediately after I retired
they asked me to go around to the Centers and I went around and gave
a pitch on fault tree analysis, how [one] sets up to do one and what
the critical elements of it are and then gave a couple of examples—one
of how we did it on the turbopumps on the blades. I gave them a little
help for about a year there. I hope I helped some of them understand
that process better. They thought it was valuable enough to all the
Centers to know how to do the fault tree.
It’s interesting where [our fault tree] came from. I was at
MIT [Massachusetts Institute of Technology, Cambridge] on an [Alfred
P.] Sloan [Foundation] fellowship, and while I was there I read this
article called “Decision Trees for Business Decisions.”
It was written by some guy from Harvard [University, Cambridge]. It
was very methodical; it was impressive in how methodical it was. For
example if you want to build a new Walmart [store] someplace, this
is how you put a fault tree on it and see if it feasible. I thought,
“You know what, I believe that would work on [failure of other]
So I brought that process back to Marshall, that was about 1968. I
started using it in the materials lab for different problems that
required that kind of an approach. Then finally it pretty much got
universally accepted. It turns out that we weren’t unique in
that. There were other people other places unknown to us that had
also discovered that idea and were [also] using it. It’s pretty
much an accepted process and procedure now, but it had a tough row
to hoe in the beginning. People didn’t want to take the time,
spend the effort, contribute their best experts to something that
might run one, two, three months, [or more].
showed accountability, that’s the scary part of it.
The [experts’] names were on the [blocks on our] trees. The
technical reports supporting the blocks had to come in, there’s
a name on everybody’s and then when I was in one of those we’d
quit at 4:00 and then we’d have a progress report. “Okay,
Block 3, who’s the blockhead?”—I called them blockheads—“Block
3, how’d it go today? Did you find anything out?” Forcing
everybody else that had blocks on the fault tree to listen to him
helped them solve their blocks on the fault tree [also]. We had a
lot of fun with those fault trees.
a great transfer of information that never would have been exchanged.
Exactly. But again, it takes a little bit of ingenuity. You have to
think a little bit out of the box. Then each one [of those blocks
on the tree] has to be exonerated or indicted, every one, until you’re
through. That gets way down to the lower level. Somebody may have
to, on [a particular task], run a fatigue test and prove that the
material would stand that cyclic fatigue up to the point where [failure
occurred]. It involved all the engineering techniques and analytical
approaches. It was a great thing for us and we used it successfully.
like it works well. I thank you so much for coming in and sharing
all this information. I really appreciate you coming in.
I owe Marshall and the Space Shuttle a whole lot, I owe them a lot.
I was happy as a clam, and I couldn’t have been more pleased
anyplace else. I would not have wanted to do anything other than what
I did do. So any time somebody said, “Can you help with this?”
even after retirement, I [was] ready.