According to Perform:
Working Set is the current size, in bytes, of the Working Set of this process. The Working Set is the set of memory pages touched recently by the threads in the process. If free memory in the computer is above a threshold, pages are left in the Working Set of a process even if they are not in use. When free memory falls below a threshold, pages are trimmed from Working Sets. If they are needed they will then be soft-faulted back into the Working Set before leaving main memory.
Total Server Memory:
Total amount of dynamic memory the server is currently consuming
Keep in mind that Total Server Memory is a SQL Server counter (SQLServer:Memory Manager performance object), while working set is a system counter (Process performance object). So one is being measured from within the database while the other from outside the database. Also the counters could have been taken at different moments in time, or averaged differently which would give you a strange looking number.