I understand that the set up of a battery consists of very good, but not ideal conductor, and therefore, some internal resistance exists. Also, I get that emf is the PD that would exist if the internal resistance is zero. What I don't get, however is why EMF and PD are equal in case of an open circuit (circuit with infinite resistance). I'm sure I'm missing something here. Help is appreciated.
As you stated, we can think of a real battery as an ideal one with an internal resistance $R_i$. This battery is then connected to an external circuit with resistance $R$. Those 2 resistors form a voltage divider. If the EMF has a value of $V$ then the voltage measured across the external resistance is $V*R/(R+R_i)$.
This voltage is equal to the EMF of the battery when $R_i = 0$ (because $R/R=1$), and also when $R=\infty$ (because in that case $R_i$ can be ignored in comparison to $R$).
The electromotive force of a battery is equal to the potential difference between its terminals in an "open circuit", when there is no current being drawn. The potential difference between the terminals generally drops when the current is being drawn.
let me know if you need more explanation.
When a circuit is open (infinite resistance) then there is still work per unit charge being done by the battery but it just builds up charge imbalance inside the battery (and on the ends) until equilibrium is achieved when electrostatic forces can counter the battery. So the charge imbalance grows until the electrostatic force can stop the battery's force. The charge buildup occurs only inside the battery because of the open circuit.
When the circuit is connected, then the charge balance redistributes the charge around the circuit, and in equilibrium there are electrostatic forces due to the now-not-changing charge density and that's an electrostatic force, so has zero line integral, so the line integral of all the force per charge (battery and electrostatic) is just that due to the battery, i.e. the emf.