In the C API, system calls return a negative value to indicate an error, and the error code in
errno gives more information on the nature of the error. Your man page should explain the possible errors on your system. There are two standard error codes:
EAGAIN indicates that the new process cannot be created due to a lack of available resources, either insufficient memory of some kind or a limit that has been reached such as the maximum number of processes per user or overall.
ENOMEM indicates that the new process cannot be created due to a lack of memory of some kind. Under Linux,
ENOMEM indicates a lack of kernel memory, while a lack of userland memory is reported as
EAGAIN. Under OpenBSD,
ENOMEM is used for a lack of userland memory as well.
fork can fail due to a lack of available resources (possibly in the form of an artificial limit rather than a simple lack of memory).
The behavior of shells when
fork fails is not specified by POSIX. In practice, they tend to return different error statuses (1 in pdksh, 2 in ash, 254 in bash, 258 in ksh93, 1 in tcsh; zsh returns 0 which is a bug). If you're implementing a shell for production use (as opposed to a learning exercise), it might be worth discussing this on the Austin Group mailing list.