Nginx: the High-Performance Web Server and Reverse Proxy

Source: http://www.linuxjournal.com/magazine/nginx-high-performance-web-server-and-reverse-proxy?page=0,0

Having performance issues with your Web server? Maybe the Russians can help.

Apache is the most popular Web server and one of the most successful open-source projects of all time. Since April 1996, Apache has served more Web sites than any other Web server. Many of the world's largest Web sites, including YouTube, Facebook, Wikipedia and Craigslist, use Apache to serve billions of page views per month. Over the years, Apache has proven itself to be a very stable, secure and configurable Web server. Although Apache is an excellent Web server, what if there were an alternative with the same functionality, a simpler configuration and better performance? That Web server exists, and it's called Nginx.
Nginx, pronounced “Engine X”, is a high-performance Web server and reverse proxy. It was created by Igor Sysoev for www.rambler.ru, Russia's second-largest Web site. Rambler has used Nginx since summer 2004, and it's currently serving about 500 million requests per day. Like Apache, Nginx is used by some of the largest Web sites in the US, including WordPress (#26), YouPorn (#27), Hulu and MochiMedia. As of May 2008, Nginx is the fourth-most-popular Web server, and it is currently serving more than two million Web sites. As it is only trailing behind Apache, IIS and GFE, it is effectively the second-most-popular Web server available for Linux.

Why Use Nginx?

Like Apache, Nginx has all the features you would expect from a leading Web server:

• Static file serving.
• SSL/TLS support.
• Virtual hosts.
• Reverse proxying.
• Load balancing.
• Compression.
• Access controls.
• URL rewriting.
• Custom logging.
• Server-side includes.
• WebDAV.
• FLV streaming.
• FastCGI.

It is stable, secure and very easy to configure, as you will see later in the article. However, the main advantages of Nginx over Apache are performance and efficiency.
I ran a simple test against Nginx v0.5.22 and Apache v2.2.8 using ab (Apache's benchmarking tool). During the tests, I monitored the system with vmstat and top. The results indicate that Nginx outperforms Apache when serving static content. Both servers performed best with a concurrency of 100. Apache used four worker processes (threaded mode), 30% CPU and 17MB of memory to serve 6,500 requests per second. Nginx used one worker, 15% CPU and 1MB of memory to serve 11,500 requests per second.
Nginx is able to serve more requests per second with less resources because of its architecture. It consists of a master process, which delegates work to one or more worker processes. Each worker handles multiple requests in an event-driven or asynchronous manner using special functionality from the Linux kernel (epoll/select/poll). This allows Nginx to handle a large number of concurrent requests quickly with very little overhead. Apache can be configured to use either a process per request (pre-fork) or a thread for each request (worker). Although Apache's threaded mode performs much better than its pre-fork mode, it still uses more memory and CPU than Nginx's event-driven architecture.

Installation and Basic Usage

Nginx is available in most Linux distributions. For this article, I use Ubuntu 8.04 (Hardy), which includes Nginx version 0.5.33. If your distro does not have Nginx, or if you want to run a newer version, you always can download the latest stable version (v0.6.31 at the time of this writing) and install from source.
Run the following command as root to install Nginx:

# apt-get install nginx

Now that Nginx is installed, you can use the startup script to start, stop or restart the Web server:

# /etc/init.d/nginx start
# /etc/init.d/nginx stop
# /etc/init.d/nginx restart 

Most configuration changes do not require a restart, in which case you can use the reload command. It is generally a good idea to test the Nginx configuration file for errors before reloading:

# nginx -t
# /etc/init.d/nginx reload

Let's go ahead and start the server:

# /etc/init.d/nginx start

Nginx now should be running on your machine. If you open http://127.0.0.1/ in your browser, you should see a page with “Welcome to nginx!”.

Main Configuration File

Now that Nginx is installed, let's take a look at its config file, located at /etc/nginx/nginx.conf. This file contains the server-wide settings for Nginx, and it should look similar to this:

user www-data;
worker_processes  1;
error_log  /var/log/nginx/error.log;
pid  /var/run/nginx.pid;
events {
  worker_connections  1024;
}
http {
  include  /etc/nginx/mime.types;
  default_type  application/octet-stream;
  access_log  /var/log/nginx/access.log;
  sendfile  on;
  keepalive_timeout  65;
  tcp_nodelay  on;
  gzip  on;
  include /etc/nginx/sites-enabled/*;
}

We are not going to change any of these settings, but let's talk about some of them to help us understand how Nginx works. The worker_processes setting tells Nginx how many child processes to start. If your server has more than one processor or is performing large amounts of disk IO, you might want to try increasing this number to see if you get better performance. The worker_connections setting limits the number of concurrent connections per worker process. To determine the maximum number of concurrent requests, you simply multiply worker_processes by worker_connections.
The error_log and access_log settings indicate the default logging locations. You also can configure these settings on a per-site basis, as you will see later in the article. Like Apache, Nginx is configured to run as the www-data user, but you easily can change this with the user setting. The startup script for Nginx needs to know the process ID for the master process, which is stored in /var/run/nginx.pid, as indicated by the pid setting.
The sendfile setting allows Nginx to use a special Linux system call to send a file over the network in a very efficient manner. The gzip option instructs Nginx to compress each response, which uses more CPU but saves bandwidth and decreases response time. Additionally, Nginx provides another compression module called gzip precompression (available as of version 0.6.24). This module looks for a compressed copy of the file with a .gz extension in the same location and serves it to gzip-enabled clients. This prevents having to compress the file each time it's requested.
The last setting we are concerned with is the include directive for the sites-enabled directory. Inside /etc/nginx, you'll see two other directories, /etc/nginx/sites-available and /etc/nginx/sites-enabled. For each Web site you want to host with Nginx, you should create a config file in /etc/nginx/sites-available, then create a symlink in /etc/nginx/sites-enabled that points to the config file you created. The main Nginx config file includes all the files in /etc/nginx/sites-enabled. This helps organize your configuration files and makes it very easy to enable and disable specific Web sites.

Static Web Server
Now that we covered the main configuration file, let's create a config file for a basic Web site. Before we begin, we need to disable the default site that Ubuntu created for us:

# rm -f /etc/nginx/sites-enabled/default

Now, create a new configuration file called /etc/nginx/sites-available/basic with the following contents:

server {
  listen  127.0.0.1:80;
  server_name  basic;
  access_log  /var/log/nginx/basic.access.log;
  error_log  /var/log/nginx/basic.error.log;
  location  / {
    root  /var/www/basic;
    index  index.html index.htm;
  }
}

Create the root directory and index.html file:

# mkdir /var/www/basic
# cd /var/www/basic
# echo "Basic Web Site" > index.html

Enable the site and restart Nginx:

# cd /etc/nginx/sites-enabled
# ln -s ../sites-available/basic . 
# /etc/init.d/nginx restart

If you open http://127.0.0.1/ in your browser, you should see a page with “Basic Web Site”. As you can see, it is very easy to create a new site using Nginx.
Let's go over the new configuration file we created. The server directive is used to define a new virtual server, and all of its settings are enclosed in braces. The listen directive indicates the IP and port on which this server will accept requests, and server_name sets the hostname for your virtual server. As I mentioned earlier, the access_log and error_log settings can be set on a per-site basis. It is usually a good idea to provide each site with its own set of log files.
Next is the location directive, which allows you to modify the settings for different parts of your site. In our case, we have only one location for the entire site. However, you can have multiple location directives, and you can use regular expressions to define them. We have two other directives inside our location block: root and index. The root directive is used to define the document root for this location. This means a request for /img/test.gif would look for the file /var/www/localhost/img/test.gif. Finally, the index directive tells Nginx what files to use as the default file for this location.

Static Web Server with SSL

Some Web sites, such as on-line stores, require secure communication (HTTPS) to protect credit-card transactions and customer information. Like Apache, Nginx supports HTTPS via an SSL module, and it's very easy to set up.
First, you need to generate an SSL certificate. The openssl command will ask you a bunch of questions, but you simply can press Enter for each one:

# apt-get install openssl
# mkdir /etc/nginx/ssl
# cd /etc/nginx/ssl
# openssl req -new -x509 -nodes -out server.crt -keyout server.key

Create a new config file called /etc/nginx/sites-available/secure, which contains the following:

server {
  listen   127.0.0.1:443;
  server_name  secure;
  access_log  /var/log/nginx/secure.access.log;
  error_log  /var/log/nginx/secure.error.log;
  ssl on;
  ssl_certificate /etc/nginx/ssl/server.crt;  
  ssl_certificate_key /etc/nginx/ssl/server.key;  
  location / {
    root   /var/www/secure;
    index  index.html index.htm;
  }
}

Create the root directory and index.html file:

# mkdir /var/www/secure
# cd /var/www/secure
# echo "Secure Web Site" > index.html

Enable the site and restart Nginx:

# cd /etc/nginx/sites-enabled
# ln -s ../sites-available/secure . 
# /etc/init.d/nginx restart

If you open https://127.0.0.1/ in your browser (note the https), you probably will get a warning about not being able to verify the certificate. That's because we are using a self-signed certificate for this example. Go ahead and tell your browser to accept the certificate, and you should see a page with “Secure Web Site”.
This config file is very similar to our previous config, but there are a few differences. First, notice that this new server is listening on port 443, which is the standard port for HTTPS. Second, we enabled the SSL module with the line ssl on;. If you compiled Nginx yourself instead of using the Ubuntu package, you need to make sure you specified --with-http_ssl_module when you ran ./configure; otherwise, the SSL module will not be available. Third, we used the ssl_certificate and ssl_certificate_key directives to point to the certificate and key we created earlier.

Virtual Hosting

In many cases, you will want to run multiple Web sites from a single server. This is called virtual hosting, and Nginx supports both IP- and name-based vhosts.
Let's create two virtual hosts: one.example.com and two.example.com. First, we need to add a line to our /etc/hosts file, so that one.example.com and two.example.com point to our server (normally you would do this using DNS):

# echo "127.0.0.1 one.example.com two.example.com" >> /etc/hosts

Now, we need to create a configuration file for each site. First, create a file called /etc/nginx/sites-available/one with the following contents:

server {
  listen   127.0.0.1:80;
  server_name  one.example.com;
  access_log  /var/log/nginx/one.access.log;
  error_log /var/log/nginx/one.error.log;
  location / {
    root   /var/www/one;
    index  index.html index.htm;
  }
}

Then, make a copy of that file called /etc/nginx/sites-available/two, and replace each occurrence of “one” with “two”:

# cd /etc/nginx/sites-available
# cp one two
# sed -i "s/one/two/" two

Create the root directories and index.html files:

# mkdir /var/www/{one,two}
# echo "Site 1" > /var/www/one/index.html
# echo "Site 2" > /var/www/two/index.html

Enable the sites and restart Nginx:

# cd /etc/nginx/sites-enabled
# ln -s ../sites-available/one .
# ln -s ../sites-available/two .
# /etc/init.d/nginx restart

If you open http://one.example.com/ in your browser, you should see a page with “Site 1”. For http://two.example.com/, you should see “Site 2”.
We just created two name-based virtual hosts running on 127.0.0.1 by changing the server_name directive. For IP-based virtual hosts, simply change the listen directive to use a different IP for each site.
Now, go ahead and disable these two virtual hosts:

# rm -f /etc/nginx/sites-enabled/one
# rm -f /etc/nginx/sites-enabled/two
# /etc/init.d/nginx restart

Don't forget to remove the line we added to /etc/hosts when you are done.

Reverse Proxy and Load Balancer
In addition to being an extremely fast static Web server, Nginx also is a load balancer and reverse proxy. A load balancer is a device used to spread work out across multiple servers or processes, and a reverse proxy is a server that transparently hands off requests to another server. Among other things, this allows Nginx to handle requests for static content and to load-balance requests for dynamic content across many different back-end servers or processes.
For this example, let's create a very simple Python Web server to serve up some dynamic content. Don't worry if you are not familiar with Python; we're just using it to display a Web page that indicates on which port the server is running. Save the following to a file called /tmp/server.py:

import sys,BaseHTTPServer as B
class Handler(B.BaseHTTPRequestHandler):
  def do_GET(self):
    self.wfile.write("Served from port %s" % port)
  def log_message(self, *args):
    pass
if __name__ == '__main__':
  host,port = sys.argv[1:3]
  server = B.HTTPServer((host,int(port)), Handler)
  server.serve_forever()

Now we can start two of these local servers, each on a different port:

# python /tmp/server.py 127.0.0.1 8001 &
# python /tmp/server.py 127.0.0.1 8002 &

If you open http://127.0.0.1:8001/ in your browser, you should see “Served from port 8001”, and if you open http://127.0.0.1:8002/, you should see “Served from port 8002”.
Now, create a new configuration file called /etc/nginx/sites-available/proxy with the following contents:

upstream python_servers {
  server 127.0.0.1:8001;
  server 127.0.0.1:8002;
}
server {
  listen   127.0.0.1:8000;
  server_name  proxy;
  access_log  /var/log/nginx/proxy.access.log;
  error_log /var/log/nginx/proxy.error.log;
  location / {
    proxy_pass http://python_servers;
  }
}

Enable the site and restart Nginx:

# cd /etc/nginx/sites-enabled
# ln -s ../sites-available/proxy .
# /etc/init.d/nginx restart

If you open http://127.0.0.1:8000/ in your browser, you should see a page with either “Served from port 8001” or “Served from port 8002”, and it should alternate each time you refresh the page.
Let's go over some of these new settings. The upstream block defines a name for a group of back-end servers. In our case, we defined a group named python_servers, which contains the two local Python servers we started on port 8001 and 8002. We then configured Nginx to hand off all requests to our back-end servers with the line proxy_pass http://python_servers;. Nginx automatically load-balances the requests to each Python server in a round-robin manner. You also can set weights for each back end, so you can direct more or fewer requests to specific servers.
Nginx handles back-end failures automatically and will stop sending requests to a failed back-end server until it starts responding again. To demonstrate this, we can kill off the Python process that's running on port 8001. Use the jobs command to find the job number for the Python process running on port 8001, and use kill %<job number> to kill the process:

# jobs
# kill %1

Open http://127.0.0.1:8000/ in your browser and keep refreshing the page, you should see only the “Served from port 8002” page. Nginx detected that the back-end server from port 8001 was not responding, so it stopped sending requests to that server. Now, restart the Python process for port 8001:

# python /tmp/server.py 127.0.0.1 8001 &

Keep refreshing the page and you should see your browser start alternating between “Served from port 8001” and “Served from port 8002” again. Nginx detected that the port 8001 back end was responding and began sending requests to it.

Conclusion

Whether you are looking to get the most out of your VPS or are attempting to scale one of the largest Web sites in the world, Nginx may be the best tool for the job. It's fast, stable and easy to use. Thanks to Igor Sysoev for creating this excellent piece of software.

Resources

Nginx Web Site: wiki.codemongers.com/Main
Module Comparison Index: wiki.codemongers.com/NginxModuleComparisonMatrix
Testimonials: wiki.codemongers.com/NginxWhyUseIt
Nginx at WordPress: barry.wordpress.com/2008/04/28/load-balancer-update
Facebook App Using Nginx: highscalability.com/friends-sale-architecture-300-million-page-view-month-facebook-ror-app

Will Reese has worked with Linux for the past ten years, primarily scaling Web applications running on Apache, Python and PostgreSQL. He enjoys beating Cory Wright at foosball and Wii Tennis.

Perl: send email script

#/usr/bin/perl

use Net::SMTP;

my $SUBJECT = "[warn] Punch INET";
my $FROM = "inet-checker\@dena.jp";
my $TO = "punchvn\@gmail.com";
my $SMTP_SERVER = "SVNserver.localdomain";

my $speed=`php speedtest.php`;
$speed =~ s/\n//g;
print "$speed\n";
if ($speed < 1) {
        &sendmail("$speed Mbps");
}

sub sendmail {
        my ($sp) = @_;
        my $smtp = Net::SMTP->new($SMTP_SERVER, Hello => $SMTP_SERVER, Timeout => 300);
        $smtp->mail($ENV{USER});
        $smtp->to($TO);

        my $data = "To:$TO\n";
        $data .= "From:$FROM\n";
        $data .= "Subject:$SUBJECT\n\n";
        $data .= "INET speed is $sp";

        $smtp->data($data);
        $smtp->quit;
        print "\nSent mail to $TO\n";
}

Examining Load Average

Source: http://www.linuxjournal.com/article/9001?page=0,0

Dec 01, 2006  By Ray Walker

 in

• SysAdmin

Understanding work-load averages as opposed to CPU usage.

Many Linux administrators and support technicians regularly use the top utility for real-time monitoring of their system state. In some shops, it is very typical to check top first when there is any sign of trouble. In that case, top becomes the de facto critical measurement of the machine's health. If top looks good, there must not be any system problems. top is rich with information—memory usage, kernel states, process priorities, process owner and so forth all can be obtained from top. But, what is the purpose of those three curious load averages, and what exactly are they trying to tell me? To answer those questions, an intuitive as well as a detailed understanding of how the values are formed are necessary. Let's start with intuition.

The Intuitive Interpretation

The three load-average values in the first line of top output are the 1-minute, 5-minute and 15-minute average. (These values also are displayed by other commands, such as uptime, not only top.) That means, reading from left to right, one can examine the aging trend and/or duration of the particular system state. The state in question is CPU load—not to be confused with CPU percentage. In fact, it is precisely the CPU load that is measured, because load averages do not include any processes or threads waiting on I/O, networking, databases or anything else not demanding the CPU. It narrowly focuses on what is actively demanding CPU time. This differs greatly from the CPU percentage. The CPU percentage is the amount of a time interval (that is, the sampling interval) that the system's processes were found to be active on the CPU. If top reports that your program is taking 45% CPU, 45% of the samples taken by top found your process active on the CPU. The rest of the time your application was in a wait. (It is important to remember that a CPU is a discrete state machine. It really can be at only 100%, executing an instruction, or at 0%, waiting for something to do. There is no such thing as using 45% of a CPU. The CPU percentage is a function of time.) However, it is likely that your application's rest periods include waiting to be dispatched on a CPU and not on external devices. That part of the wait percentage is then very relevant to understanding your overall CPU usage pattern.
The load averages differ from CPU percentage in two significant ways: 1) load averages measure the trend in CPU utilization not only an instantaneous snapshot, as does percentage, and 2) load averages include all demand for the CPU not only how much was active at the time of measurement.
Authors tend to overuse analogies and sometimes run the risk of either insulting the reader's intelligence or oversimplifying the topic to the point of losing important details. However, freeway traffic patterns are a perfect analogy for this topic, because this model encapsulates the essence of resource contention and is also the chosen metaphor by many authors of queuing theory books. Not surprisingly, CPU contention is a queuing theory problem, and the concepts of arrival rates, Poisson theory and service rates all apply. A four-processor machine can be visualized as a four-lane freeway. Each lane provides the path on which instructions can execute. A vehicle can represent those instructions. Additionally, there are vehicles on the entrance lanes ready to travel down the freeway, and the four lanes either are ready to accommodate that demand or they're not. If all freeway lanes are jammed, the cars entering have to wait for an opening. If we now apply the CPU percentage and CPU load-average measurements to this situation, percentage examines the relative amount of time each vehicle was found occupying a freeway lane, which inherently ignores the pent-up demand for the freeway—that is, the cars lined up on the entrances. So, for example, vehicle license XYZ 123 was found on the freeway 30% of the sampling time. Vehicle license ABC 987 was found on the freeway 14% of the time. That gives a picture of how each vehicle is utilizing the freeway, but it does not indicate demand for the freeway.
Moreover, the percentage of time these vehicles are found on the freeway tells us nothing about the overall traffic pattern except, perhaps, that they are taking longer to get to their destination than they would like. Thus, we probably would suspect some sort of a jam, but the CPU percentage would not tell us for sure. The load averages, on the other hand, would.
This brings us to the point. It is the overall traffic pattern of the freeway itself that gives us the best picture of the traffic situation, not merely how often cars are found occupying lanes. The load average gives us that view because it includes the cars that are queuing up to get on the freeway. It could be the case that it is a nonrush-hour time of day, and there is little demand for the freeway, but there just happens to be a lot of cars on the road. The CPU percentage shows us how much the cars are using the freeway, but the load averages show us the whole picture, including pent-up demand. Even more interesting, the more recent that pent-up demand is, the more the load-average value reflects it.
Taking the discussion back to the machinery at hand, the load averages tell us by increasing duration whether our physical CPUs are over- or under-utilized. The point of perfect utilization, meaning that the CPUs are always busy and, yet, no process ever waits for one, is the average matching the number of CPUs. If there are four CPUs on a machine and the reported one-minute load average is 4.00, the machine has been utilizing its processors perfectly for the last 60 seconds. This understanding can be extrapolated to the 5- and 15-minute averages.
In general, the intuitive idea of load averages is the higher they rise above the number of processors, the more demand there is for the CPUs, and the lower they fall below the number of processors, the more untapped CPU capacity there is. But all is not as it appears.

The Wizard behind the Curtain

The load-average calculation is best thought of as a moving average of processes in Linux's run queue marked running or uninterruptible. The words “thought of” were chosen for a reason: that is how the measurements are meant to be interpreted, but not exactly what happens behind the curtain. It is at this juncture in our journey when the reality of it all, like quantum mechanics, seems not to fit the intuitive way as it presents itself.
The load averages that the top and uptime commands display are obtained directly from /proc. If you are running Linux kernel 2.4 or later, you can read those values yourself with the command cat /proc/loadavg. However, it is the Linux kernel that produces those values in /proc. Specifically, timer.c and sched.h work together to do the computation. To understand what timer.c does for a living, the concept of time slicing and the jiffy counter help round out the picture.
In the Linux kernel, each dispatchable process is given a fixed amount of time on the CPU per dispatch. By default, this amount is 10 milliseconds, or 1/100th of a second. For that short time span, the process is assigned a physical CPU on which to run its instructions and allowed to take over that processor. More often than not, the process will give up control before the 10ms are up through socket calls, I/O calls or calls back to the kernel. (On an Intel 2.6GHz processor, 10ms is enough time for approximately 50-million instructions to occur. That's more than enough processing time for most application cycles.) If the process uses its fully allotted CPU time of 10ms, an interrupt is raised by the hardware, and the kernel regains control from the process. The kernel then promptly penalizes the process for being such a hog. As you can see, that time slicing is an important design concept for making your system seem to run smoothly on the outside. It also is the vehicle that produces the load-average values.
The 10ms time slice is an important enough concept to warrant a name for itself: quantum value. There is not necessarily anything inherently special about 10ms, but there is about the quantum value in general, because whatever value it is set to (it is configurable, but 10ms is the default), it controls how often at a minimum the kernel takes control of the system back from the applications. One of the many chores the kernel performs when it takes back control is to increment its jiffies counter. The jiffies counter measures the number of quantum ticks that have occurred since the system was booted. When the quantum timer pops, timer.c is entered at a function in the kernel called timer.c:do_timer(). Here, all interrupts are disabled so the code is not working with moving targets. The jiffies counter is incremented by 1, and the load-average calculation is checked to see if it should be computed. In actuality, the load-average computation is not truly calculated on each quantum tick, but driven by a variable value that is based on the HZ frequency setting and tested on each quantum tick. (HZ is not to be confused with the processor's MHz rating. This variable sets the pulse rate of particular Linux kernel activity and 1HZ equals one quantum or 10ms by default.) Although the HZ value can be configured in some versions of the kernel, it is normally set to 100. The calculation code uses the HZ value to determine the calculation frequency. Specifically, the timer.c:calc_load() function will run the averaging algorithm every 5 * HZ, or roughly every five seconds. Following is that function in its entirety:

   unsigned long avenrun[3];

   static inline void calc_load(unsigned long ticks)
   {
      unsigned long active_tasks; /* fixed-point */
      static int count = LOAD_FREQ;

      count -= ticks;
      if (count < 0) {
         count += LOAD_FREQ;
         active_tasks = count_active_tasks();
         CALC_LOAD(avenrun[0], EXP_1, active_tasks);
         CALC_LOAD(avenrun[1], EXP_5, active_tasks);
         CALC_LOAD(avenrun[2], EXP_15, active_tasks);
      }
   }

The avenrun array contains the three averages we have been discussing. The calc_load() function is called by update_times(), also found in timer.c, and is the code responsible for supplying the calc_load() function with the ticks parameter. Unfortunately, this function does not reveal its most interesting aspect: the computation itself. However, that can be located easily in sched.h, a header used by much of the kernel code. In there, the CALC_LOAD macro and its associated values are available:

   extern unsigned long avenrun[]; /* Load averages */

   #define FSHIFT   11  /* nr of bits of precision */
   #define FIXED_1  (1<<FSHIFT) /* 1.0 as fixed-point */
   #define LOAD_FREQ (5*HZ) /* 5 sec intervals */
   #define EXP_1  1884  /* 1/exp(5sec/1min) as fixed-point */
   #define EXP_5  2014  /* 1/exp(5sec/5min) */
   #define EXP_15 2037  /* 1/exp(5sec/15min) */

   #define CALC_LOAD(load,exp,n) \
      load *= exp; \
      load += n*(FIXED_1-exp); \
      load >>= FSHIFT;

Here is where the tires meet the pavement. It should now be evident that reality does not appear to match the illusion. At least, this is certainly not the type of averaging most of us are taught in grade school. But it is an average nonetheless. Technically, it is an exponential decay function and is the moving average of choice for most UNIX systems as well as Linux. Let's examine its details.
The macro takes in three parameters: the load-average bucket (one of the three elements in avenrun[]), a constant exponent and the number of running/uninterruptible processes currently on the run queue. The possible exponent constants are listed above: EXP_1 for the 1-minute average, EXP_5 for the 5-minute average and EXP_15 for the 15-minute average. The important point to notice is that the value decreases with age. The constants are magic numbers that are calculated by the mathematical function shown below:

When x=1, then y=1884; when x=5, then y=2014; and when x=15, then y=2037. The purpose of the magical numbers is that it allows the CALC_LOAD macro to use precision fixed-point representation of fractions. The magic numbers are then nothing more than multipliers used against the running load average to make it a moving average. (The mathematics of fixed-point representation are beyond the scope of this article, so I will not attempt an explanation.) The purpose of the exponential decay function is that it not only smooths the dips and spikes by maintaining a useful trend line, but it accurately decreases the quality of what it measures as activity ages. As time moves forward, successive CPU events increase their significance on the load average. This is what we want, because more recent CPU activity probably has more of an impact on the current state than ancient events. In the end, the load averages give a smooth trend from 15 minutes through the current minute and give us a window into not only the CPU usage but also the average demand for the CPUs. As the load average goes above the number of physical CPUs, the more the CPU is being used and the more demand there is for it. And, as it recedes, the less of a demand there is. With this understanding, the load average can be used with the CPU percentage to obtain a more accurate view of CPU activity.
It is my hope that this serves not only as a practical interpretation of Linux's load averages but also illuminates some of the dark mathematical shadows behind them. For more information, a study of the exponential decay function and its applications would shed more light on the subject. But for the more practical-minded, plotting the load average vs. a controlled number of processes (that is, modeling the effects of the CALC_LOAD algorithm in a controlled loop) would give you a feel for the actual relationship and how the decaying filter applies.

Ray Walker is a consultant specializing in UNIX kernel-level code. He has been a software developer for more than 25 years, working with Linux since 1995. He can be contacted at ray.rwalk2730@gmail.com.