In 1933, he gave a talk at a meeting of the British Association for the Advancement of Science, from which he was quoted in The London Times of September 12, 1933 about the possibility of energy-efficient nuclear fission as follows:
We might in these processes obtain very much more energy than the proton supplied, but on the average we could not expect to obtain energy in this way. It was a very poor and inefficient way of producing energy, and anyone who looked for a source of power in the transformation of the atoms was talking moonshine. But the subject was scientifically interesting because it gave insight into the atoms.
The 16 September 1933 issue of Nature tells its readers about the talk:
One timely word of warning was issued to those who look for sources of power in atomic transmutations ‒ such expecations are the merest moonshine.
It is easy to invoke this statement as a further example for a severe scientific misjudgement by a senior scientist. But in context, it was perfectly reasonable: Rutherford was discussing nuclear reactions tiggered by the proton beam from the then brand-new accelerator of Cockroft and Walton. Trying to gain nuclear energy that way is about as efficient as producing antimatter at CERN to fuel a matter-antimatter-annihilation engine.
Moreover, in his paper Atomic Energy is "Moonshine": What did Rutherford Really Mean?, historian of science John G. Jenkin argues that Rutherford was well aware that there might be ways to harness nuclear energy, especially using neutrons as tools to induce reactions. He suggests that Rutherford "in all of his later negative pronouncements regarding the possibility of atomic energy, was adopting a quite deliberate policy to disguise and postpone, for as long as possible, the awful prospect that he saw looming over the horizon: a new and dreadful war, a new and devastating weapon, and unprecedented destruction."
On a lighter side, Rutherford also allegedly warned (quoted for example in "The Strangest Man: The hidden life of Paul Dirac" by Graham Farmelo):
"Don't let me catch anyone talking about the universe in my department!"
and said about special relativity (quoted for example in "The Nobel Prize: A History of Genius, Controversy, and Prestige" by Burton Feldman):
"Oh, that stuff. We never bother with that in our work."
Though I am not sure about the origin of this latter quotation.
Rutherford was reportedly skeptic about special relativity in its early days, and for most of atomic physics it can be safely neglected and one does indeed not have to bother. But when in 1930 he prepared a new and updated edition of his book "Radiation from Radioactive Substances", he did add a discussion of the mass defect based on E = mc².