Liam Laverick, 25, from Beeford Grove in Hull has pleaded not guilty to the manslaughter of his four-week old son, Tommy Lee.
The baby was taken to Hull Royal Infirmary on 23 September 2014 and died later at Leeds General Infirmary.
Tommy Lee had been left in the sole care of Mr Laverick when the baby became ill, Hull Crown Court heard.
Prosecutor Nicholas Lumley QC said in his opening statement Tommy Lee was left in the care of Mr Laverick at his flat on Linnaeus Street, Hull.
He also told the court Mr Laverick had killed his son by shaking or throwing him.
The jury was shown CCTV footage of Mr Laverick taking Tommy Lee to a relative's house to find the baby's mother, Kelly Whitworth, before the couple ran with the child to the nearby Hull Royal Infirmary.
Mr Laverick gave different accounts of what had happened, including saying that he had fallen over while carrying the child, the prosecution said.
A recording of Mr Laverick speaking to Tommy Lee's mother at a police station and made without their knowledge was also played in court.
In it Mr Laverick described falling over in the hallway of their flat while carrying the baby.
The forensic pathologist who carried out the post mortem examination on the body of Tommy Lee said he had suffered serious bleeding around his brain and the back of his eyes.
The case continues.

Dunlop, 27, lapped the course at 132.36mph on his BMW superbike and was only four seconds outside John McGuinness' absolute lap record.
The Ballymoney man's speed of 131.34 in the Superstocks improved his own previous best lap of 131.22.
Ian Hutchinson was second fastest in the Superbike class with 131.66.
Eleven-time TT winner Hutchinson, 36, produced his time on his Tyco BMW Superbike from a standing start.
Honda-mounted John McGuinness made up the top three in the blue riband class with 131.12, followed by his Honda team-mate Conor Cummins (129.99) and Yorkshire's Dean Harrison on a Kawasaki (129.97).
Fermanagh rider Lee Johnston, riding a Triumph, clocked the fastest Supersport speed of the week at 124.84mph to lie sixth overall in the leaderboard.
Bruce Anstey, the quickest Supersport rider to date with 124.508, came off his Superbike machine at Keppel Gate on Thursday evening, but walked away with a suspected minor ankle injury.
Michael Rutter moved to the top of the Lightweight Supertwins leaderboard with a lap of 118.054mph, just bettering Ivan Lintin's speed of 118.000mph set on Tuesday night.
Meanwhile, Tim Reeves and Patrick Farrance posted a lap of 114.83mph to send them to the top of the sidecar leaderboard ahead of Dave Molyneux and Dan Sayle.
The final qualifying session is scheduled to take place on Friday night, with the first race, the Superbikes, due off at 11:00 BST on Saturday.
A father killed his son by shaking or throwing him when he was just four weeks old, a jury has been told.
[NEXT_CONCEPT]
Michael Dunlop continued his Isle of Man TT practice week dominance as he set the fastest Superbike lap and set an unofficial Superstock lap record.