Whey in general is problematic:
With the increased demand for liquid whey, the dairy industry has achieved alternative methods of recapturing whey. Some of these methods may present serious halachic issues. Today whey is retrieved also from Swiss cheese and Mozzarella cheese productions. The milk curd in Swiss cheese productions is cooked in temperatures exceeding yad soledes, 120 F. That means that the whey has absorbed the taam, taste, of the non kosher Swiss cheese.
In order to give elasticity to mozzarella cheese, the cheese curd undergoes a second process. The cheese curds are cooked and stretched in a hot water stretcher (140 F-160 F). This gives mozzarella its elasticity. Although smaller mozzarella cheese companies discard the cook water, larger companies recognizing whey's value have started to retrieve the whey out of the mozzarella cook water. In these cases, where whey is retrieved from hot curd or hot cheese cook water productions, the whey has absorbed the taste from non-kosher cheese and would possibly be subject to the restrictions of gevinas akum.
However, there are potential leniencies:
Interestingly, other Rabbinic authorities postulate a more liberal position regarding whey produced from cheese "cooked" at a temperature that had exceeded yad soledes, 120 F. They maintain that in the U.S., cheese is produced using all kosher ingredients. The government maintains strict restrictions and imposes stiff penalties if manufacturers deviate from using cows' milk. The rennet used is largely microbial and even if non-kosher rennet was used, it has undergone so many processes that it is no longer fit for human consumption, eino ra'uy l'achilas kelev. Since the milk and other ingredients were never prohibited, the whey, a milk by-product that was separated from the cheese, has absorbed taste from kosher ingredients and would be permissible. The gezaira d'rabonon for gevinas akum only addresses the cheese curds.
That last step in the leniency is definitely the most controversial. It is saying that gevinas akum doesn't forbid what it cooks in. I don't know of many who would pasken that way, even for cheese made from microbial rennet. (Check with your own Rabbi).
Cholov Yisroel may still be required here, depending on if you say that cheese has to be made from Cholov Yisroel milk, which the Star-K article takes as obvious that it doesn't, but that is not universal (in fact Star-K's own policy is to require the milk to be Cholov Yisroel for its cheese).
So given all of the above it would be a good idea to only drink such Vodka with certification. If it has such certification, I would expect it to be dairy. The CRC says:
Unflavored vodka is acceptable unless the label states that it is produced from grapes, wine, milk, lactose, or whey. Special attention should be paid to vodka from France. Flavored vodka requires certification.
So you can expect to be labeled appropriately if it has a problematic source of alcohol, and according to them that can be relied upon.
A kosher whey based Vodka would be dairy vis-a-vis basar bchalav. Whey is מי חלב and only מי מי חלב is Pareve. I don't know if it is even possible to make alcohol out of that. On the one hand the lactose remaining is minimal, on the other hand, it is hard to dismiss the ingenuity of modern food processing.
I see you added a comment about it being specifically about non-commercial milk. I mentioned above that this is a matter of disagreement if whey in principle needs to be Cholov Yisroel, but the result is dairy at least.