Is there a difference between voiced and unvoiced consonants when whispering, which as I understand it, does not use the vocal cords? I know it sounds silly to ask because we can all understand whispered speech, but I wonder how much is psychological (ie brain-determined using context) and how much is physically different between the sounds? If someone could explain how we differentiate an 's' from a whispered 'z' for instance, that would be greatly appreciated.
Assuming you mean phonologically voiced (because if you're whispering there can't be any phonetic voicing), this indeed has to be a language-specific question, because how phonological voicing contrasts are realized phonetically differs from language to language.
For English, there are often several other phonetic manifestations of phonological voicing other than phonetic voicing (in fact, quite often phonologically voiced consonants are not phonetically voiced at all). What these manifestations are depends on the consonants in question and on their position in an utterance.
For example, word-initial voiceless stops (/p t k/) tend to have stronger bursts than their voiced counterparts (/b d g/), where "stronger bursts" means longer and more intense periods of noise. They are also characterized by a relatively long period of aspiration (especially if they are in a stressed syllable) whereas the voiced stops lack such aspiration or only show a bit of it. Aspiration may be distinguished from whispered phonation by being more intense (because the air coming out is not as impeded by the vocal folds). Also, the period of aspiration usually adds duration to the total duration of a syllable (this relative duration difference is usually not perceptually relevant in my experience, but it can be seen in spectrograms when comparing minimal pairs side by side).
As for syllable-final stops and fricatives, the relative duration of the nucleus is dependent on the voicing of the coda. All else being equal, a nucleus before a voiced coda will be longer than a nucleus before a voiceless coda. Try recording yourself whispering minimal pairs like seat and seed to convince yourself of this (or have a native speaker do it if you are not one yourself).
For phrase-final stops, released voiceless stops often display a stronger burst than released voiced stops.
Finally, if the consonants in question are in positions where their closure durations can be measured (stops in utterance-initial position and final unreleased stops do not fall under this category), then all else being equal, the closure duration of a voiceless consonant tends to be longer than that of its voiced counterpart. This is true for fricatives, too, where the period of frication is longer for voiceless fricatives (like /s/) than for their voiced counterparts (/z/ in the case of /s/). Try recording yourself whispering E.C. and E.Z. to confirm this.
The differences might be visible in a spectrogram. You could do the experiment yourself: record yourself saying /sasas/ /zazaz/ both voiced and whispered and make spectrograms with for instance Praat, and compare. You'll need to increase the size of the y-axis though, since you're comparing fricatives. If you're comparing other consonants the standard setup should be fine. If you're not comfortable with reading spectrograms you could add the images to your question and let us have a look at it.
A little quick experimentation of my own makes me think the whispered voiced fricatives are more tense than the unvoiced ones: less space between the articulators, and less air pressure. An interesting follow-up question to this though: is whispering language dependent? That is: is a whispered /z/ the same sound acoustically speaking in, say, French and English?