The Scottish champions went out 4-3 on aggregate after a 2-0 defeat in Sweden.
Deila said: "The hype about the Champions League is big, everybody wants it so badly. You have to deal with it but we looked very stressed.
"We didn't look like we wanted to go out and play football. We have a lot of young players coming through but this was a hard lesson for us."
The Norwegian was new to the Celtic manager's job when his team was knocked out in the play-off round last season by Slovenian side Maribor.
But having reshaped the team on his way to winning the Scottish Premiership in his debut season and across this summer, he was confident he could guide them into the group stage of the Champions League at the expense of the Swedish champions.
And with a 3-2 lead heading to Malmo's 20,000 capacity stadium his side were in a strong position to qualify.
Malmo are fifth in the Swedish league, seven points off the leaders with only nine matches to play of their summer season.
Media playback is not supported on this device
But Celtic will now compete in the Europa League and find out their opponents in Friday's group stage draw. Among the teams in Europe's second-tier competition are Liverpool and Tottenham.
"We looked very uncomfortable on the ball. We looked frightened and scared and that is very disappointing," he said in his media conference afterwards.
"It was a team performance that was not at the level we should expect at these games. We have to learn from this, some very hard learning.
"We knew if we were at our best we had a good chance to go through but we were not even close.
"We have shown that the Europa League is the level we have to compete at and it is better than nothing. It is a good competition."
Asked about the goal Nir Bitton had disallowed for a foul - replays showing that Bitton's marker, Kari Arnason, had handled the ball in the aerial challenge - Deila felt Serbian referee Milorad Mazic had made "a big mistake".
"It could have changed the game - it is irritating that we lost that," he said.

The partially-dismembered body of David Miller, 56, was found at his flat in Patterdale Walk in Boothville, Northampton, on 14 June 2016.
He had been stabbed multiple times and "crude attempts" were made to destroy his body, a court heard.
Ian Cuthbertson, 49, Michael Hallett, 37, Joseph Catlin, 30, and Zena Kane, 35, of no fixed address, deny murder.
At the start of their trial, Northampton Crown Court heard police officers had forced entry into the flat and found Mr Miller's remains in the bedroom.
Several items were found at the property including knives, scissors and a rolling pin which had been used as weapons during the attack on 10 June, and an angle grinder which had been borrowed to try and destroy the body.
A post-mortem examination found he had substantial facial injuries and had been stabbed fifteen times in the back and in the chest.
Peter Joyce QC, prosecuting, said that Mr Catlin had been living with Mr Miller in the two months before his death and that the pair had known each other for years.
The court heard that Mr Miller was "well known and well liked" in the area but had long-standing issues with drink.
Mr Joyce said the four defendants were part of the "Northampton street drinking fraternity".
"They lived chaotic lives with long periods of homelessness and regularly used Mr Miller's flat as a place to drink, " he said.
"These four defendants all took part in the murder of a vulnerable man in his own home and then spent days trying to cover up what they had done," he added.
The trial, expected to last for four weeks, continues.
Celtic were "scared and frightened" in their Champions League exit to Malmo, said boss Ronny Deila.
[NEXT_CONCEPT]
A jury has heard how a "vulnerable" man died after a "sustained attack" with knives and scissors in his own home.