How many Context Switches is “normal”?

This depends very much on the type of application you run. If you've got applications which are very trigger-happy WRT syscalls you can expect to see high amounts of context switching. If most of your applications idle around and only wake up when there's stuff happening on a socket, you can expect to see low context switch rates.

System calls

System calls cause context switches by their very own nature. When a process does a system call, it basically tells the kernel to take over from it's current point in time and memory to do stuff the process isn't privileged to do, and return to the same spot when it's done.

When we look at the definition of the write(2) syscall from Linux, this becomes very clear:

NAME

       write - write to a file descriptor

SYNOPSIS

       #include 

       ssize_t write(int fd, const void *buf, size_t count);

DESCRIPTION

       write() writes up to count bytes from the buffer pointed buf to the file

       referred to by the file descriptor fd. [..]

RETURN VALUE

       On success, the  number of bytes written is returned (zero indicates

       nothing was written). On error, -1 is returned, and errno is set

       appropriately.

       [..]

This basically tells the kernel to take over operation from the process, move up to count bytes, starting from the memory address pointed at by *buf to file descriptor fd of the current process and then return back to the process and tell him how it went.

A nice example to show this is the dedicated game server for Valve Source based games, hlds.http://nopaste.narf.at/f1b22dbc9 shows one second worth of syscalls done by a single instance of a game server which had no players on it. This process takes about 3% CPU time on a Xeon X3220 (2.4Ghz), just to give you a feeling for how expensive this is.

Multi-Tasking

Another source of context switching might be processes which don't do syscalls, but need to get moved off a given CPU to make room for other processes.

A nice way to visualize this is cpuburn. cpuburn doesn't do any syscalls itself, it just iterates over it's own memory, so it shouldn't cause any context switching.

Take an idle machine, start vmstat and then run a burnMMX (or any different test from the cpuburn package) for every CPU core the system has. You should have full system utilization by then but hardly any increased context switching. Then try to start a few more processes. You'll see that the context switching rate increases as the processes begin to compete over CPU cores. The amount of switching depends on the processes/core ratio and the multitasking resolution of your kernel.

Further reading

linfo.org has a nice writeup on what context switches and system calls are. Wikipedia has generic information and a nice link collection on System calls.

Here is a blog abount how long will constext switches do: http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html