"If you know calculus" ... that's the nub isn't it! It moves the mysterious appearance of one half to somewhere else i.e. the integration of the variable v being v^2/2
I had the experience a few years ago of trying to explain this to a young person who was confused about this very issue. I too came at it from the mathematical end, which is entirely natural to me. Nevertheless we weren't making any headway and I realised that the maths was obscuring rather than revealing. It's often not helpful to explain something using explanations that are at least if not more complicated, especially for people who are not mathematically inclined. It turned out that we had to go back to basics and explain things which my young tutee had forgotten or not taken in. Things like work done is defined as the force applied in moving (or accelerating) an object over a certain distance, the equivalance of potential and kinetic energy, Newton's second law, the equations of motion etc. No calculus was involved. We got there!
You have to explain to the level of your audience. But when you don't know that level, you have to guess. We never got any feedback that we were talking above or were at Vay's level.