It can certainly be done, but it requires a small amount of linear algebra - really all any one needs to know is a little bit about Linear algebra. HSV and RGB are both bases for the three dimensional vector space of colors. RGB is a cartesian coordinate system where as HSV is cylindrical. Each of the three (Red, Green, Blue, or Hue, Saturation, Value) is a component of the vector. Using this, the conversion between the two is pretty easy. The first thing to note is that the two coordinate systems are not oriented in the same way. In HSV, the longitudinal coordinate (Value) is oriented along the line (a,a,a) in the RGB coordinate system. This is to say that if you took all of the colors where red=blue=green, they would lie along the axis defined by the value coordinate. Geometrically, we can think of this as rotating our RGB cube. Taking this to be an axis requires that we give up RGB and have to choose two other axes in the plane perpendicular to V. For Hue, what we do is project the points on our RGB cube representing R, G, and B into the plane, and then measure hue as the angle formed by the line between the origin and our red point and the line between the origin and whatever color we're talking about. Now all we need is saturation, which just turns out to be the distance between our origin, and the point representing our chosen color projected on to our plane with the RGB points.
Using this, we can translate between one or the other at will and talk about colors using whatever is appropriate at the time.