Ubuntu Server 18.04 – Setting up failover with keepalived

How to Configure Network Static IP Address on Ubuntu 19.10

Using keepalived is a great way to add high availability to an application or even a hosted website. keepalived allows you to configure a floating IP (also known as a Virtual IP or VIP) for a pool of servers, with this special IP being applied to a single server at a time. Each installation of keepalived in the same group will be able to detect when another server isn’t available, and claim ownership of the floating IP whenever the master server isn’t responding. This allows you to run a service on multiple servers, with a server taking over in the event another becomes unavailable.

keepalived is by no means specific to Apache. You can use it with many different applications and services, NGINX being another example. keepalived also allows you to create a load balanced environment as well.

Let’s talk a little bit about how keepalived can work with Apache in theory. Once keepalived is installed on two or more servers, it can be configured such that a master server will have ownership of the floating IP, and other servers will continually check the availability of the master, claiming the floating IP for themselves whenever the current master becomes unreachable. For this to work, each server would need to contain the same Apache configuration and site files. I’ll leave it up to you in order to set up multiple Apache servers. If you’ve followed along so far, you should have at least one already. If it’s a virtual machine, feel free to create a clone of it and use that for the secondary server. If not, all you should have to do is set up another server, following the instructions earlier in this chapter. If you’re able to view a website on both servers, you’re good to go.

To get started, we’ll need to declare a floating IP. This can be any IP address that’s not currently in use on your network. For this to work, you’ll need to set aside an IP address for the purposes of keepalived. If you don’t already have one, pick an IP address that’s not being used by any device on your network. If you’re at all unsure, you can use an IP scanner on your network to find an available IP address. There are several scanners that can accomplish this, such as nmap on Linux or the Angry IP Scanner for Windows (I have no idea what made that IP scanner so angry, but it does work well).

Be careful when scanning networks for free IP addresses, as scanning a network may show up in intrusion detection systems as a threat. If you’re scanning any network other than one you own, always make sure you have permission from both the network administrator as well as management before you start scanning. Also, if you’re scanning a company network, keep in mind that any hardware load balancers in use may not respond to pings, so you may want to also look at an IP layout from your network administrator as well and compare the results.

To save you the trouble of a Google search, the following nmap syntax will scan a subnet and report back regarding which IP addresses are in use. You’ll need the nmap package installed first. Just replace the network address with yours:

nmap -sP 192.168.1.0/24 

Next, we’ll need to install keepalived. Simply run the following command on both of your servers:

sudo apt install keepalived 

If you check the status of the keepalived daemon, you’ll see that it attempted to start as soon as it was installed, and then immediately failed. If you check the status of keepalived with the systemctl command, you’ll see an error similar to the following:

Condition: start condition failed 

Since we haven’t actually configured keepalived, it’s fine for it to have failed. After all, we haven’t given it anything to check, nor have we assigned our floating IP. By default, there is no sample configuration file created once you’ve installed the package, we’ll have to create that on our own. Configuration for keepalived is stored in /etc/keepalived, which at this point should just be an empty directory on your system. If, for some reason, this directory doesn’t exist, create it:

sudo mkdir /etc/keepalived

Let’s open a text editor on our primary server and edit the /etc/keepalived/keepalived.conf file, which should be empty. Insert the following code into the file. There are some changes you’ll need to make in order for this configuration to work in your environment. As we continue, I’ll explain what the code is doing and what you’ll need to change. Keep your text editor open after you paste the code, and keep reading for some tips on how to ensure that this config file will work for you:

global_defs { 
    notification_email { 
    myemail@mycompany.com 
    } 
    notification_email_from keepalived@mycompany.com 
    smtp_server 192.168.1.150 
    smtp_connect_timeout 30 
    router_id mycompany_web_prod 
} 
vrrp_instance VI_1 { 
    smtp_alert 
    interface enp0s3 
    virtual_router_id 51 
    priority 100 
 
    advert_int 5 
    virtual_ipaddress { 
    192.168.1.200 
    } 
} 

There’s quite a bit going on in this file, so I’ll explain the configuration section by section, so you can better understand what’s happening.

global_defs { 
    notification_email { 
    myemail@mycompany.com 
    } 
    notification_email_from keepalived@mycompany.com 
    smtp_server 192.168.1.150 
    smtp_connect_timeout 30 
    router_id mycompany_web_prod 
} 

In the global_defs section, we’re specifically configuring an email host to use in order to send out alert messages. When there’s an issue and keepalived switches to a new master server, it can send you an email to let you know that there was a problem. You’ll want to change each of these values to match that of your mail server. If you don’t have a mail server, keepalived will still function properly. You can remove this section if you don’t have a mail server, or comment it out. However, it’s highly recommended that you set a mail server for keepalived to use.

vrrp_instance VI_1 { 
    smtp_alert 
    interface enp0s3 
    virtual_router_id 51 
    priority 100 
 
    advert_int 5 

Here, we’re assigning some configuration options regarding how our virtual IP assignment will function. The first thing you’ll want to change here is the interface. In my sample config, I have enp0s3 as the interface name. This will need to match the interface on your server where you would like keepalived to be bound to. If in doubt, execute ip a at your shell prompt to get a list of interfaces.

The virtual_router_id is a very special number that can be anything from 0-255. This number is used to differentiate multiple instances of keepalived running on the same subnet. Each member of each keepalived cluster should have the same virtual_router_id, but if you’re running multiple pairs or groups of servers on the same subnet, each group should have their own virtual_router_id. I used 51 in my example, but you can use whatever number you’d like. You’ll just have to make sure both web servers have the same virtual_router_id. I’ll go over some tips setting up the other server shortly, so don’t worry if you haven’t configured keepalived on your second server yet.

Another option in this block that’s important is the priority. This number must absolutely be different on each server. The keepalived instance with the highest priority is always considered the master. In my example, I set this number to 100. Using 100 for the priority is fine, so long as no other server is using that number. Other servers should have a lower priority. For example, you can set the second web server’s priority to 80. If you set up a third or fourth one, you could set their priority to 60 and 40, respectively. Those are just arbitrary numbers I picked off the top of my head. As long as each server has a different priority, and the server you’d like to be the master has the highest priority, you’re in good shape.

    virtual_ipaddress { 
    192.168.1.200 
} 

In the final block, we’re setting the virtual IP address. I chose 192.168.1.200 as a hypothetical example; you should choose an IP address outside of your DHCP range that’s not being used by any device.

Now that you’ve configured keepalived, let’s test it out and see whether it’s working. On your master server, start keepalived:

sudo systemctl start keepalived 

After you start the daemon, take a look at its status to see how it’s doing:

sudo systemctl status -l keepalived 

If there are no errors, the status of the daemon should be active (running). If for some reason the status is something else, take a look at the log entries shown after you execute the status command, as keepalived is fairly descriptive regarding any errors it finds in your config file. If all went well, you should see the IP address you chose for the floating IP listed in your interfaces. Execute ip a at your prompt to see them. Do you see the floating IP? If not, keep in mind it can take a handful of seconds for it to show up. Wait 30 seconds or so and check your interface list again.

If all went well on the first server, we should set up keepalived on the second web server. Install the keepalived package on the other server if you haven’t already done so, and then copy the keepalived.conf file from your working server to your new one. Here are some things to keep in mind regarding the second server’s keepalived.conf file:

  • Be sure to change the priority on the slave server to a lower number than what you used on the master server
  • The virtual_ipaddress should be the same on both
  • The virtual_router_id should be the same on both
  • Start or restart keepalived on your slave server, and verify there were no errors when the service started up

Now, let’s have some fun and see keepalived in action. What follows is an extremely simple HTML file:

<html> 
    <title>keepalived test</title> 
    <body> 
        <p>This is server #1!</p> 
    </body> 
</html> 

Copy this same HTML file to both servers and serve it via Apache or NGINX. We’ve gone over setting up both in this chapter. The easiest and quickest way to implement this is to replace Apache’s sample HTML file with the preceding one. Make sure you change the text This is server #1! to This is server #2 on the second server. You should be able to use your browser and visit the IP address for each of your two web servers and see this page on both. The only difference between them should be the text inside the page.

Now, in your browser, change the URL you’re viewing to that of your floating IP. You should see the same page as you did when you visited the IP address for server #1.

Let’s now simulate a failover. On the master server, stop the keepalived service:

sudo systemctl stop keepalived 

In our config, we set the advert_int option to 5, so the second server should take over within 5 seconds of you stopping keepalived on the master. When you refresh the page, you should see the web page for server #2 instead of server #1. If you execute ip a on the shell to view a list of interfaces on your slave server, you should also see that the slave is now holding your floating IP address. Neat, isn’t it? To see your master reclaim this IP, start the keepalived daemon on your master server and wait a few seconds.

Congratulations, you set up a cluster for your Apache implementation. If the master server encounters an issue and drops off the network, your slave server will take over. As soon as you fix your master server, it will immediately take over hosting your web page. Although we made our site slightly different on each server, the idea is showing how hosting a website can survive a single server failure. You can certainly host more than just Apache with keepalived. Almost any service you need to be highly available that doesn’t have its own clustering options is a good candidate for keepalived. For additional experimentation, try adding a third server.

Comments are closed.