WhatsApp use Erlang Programming Language. Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability.

http://www.erlang.org/

For OS, they use FreeBSD.

http://www.freebsd.org

FreeBSD is like Linux, but is more like Unix/Mac OS. Yahoo use FreeBSD for its servers.