Product Documentation

    Multiple-node redundant deployment.


    SKFS can be clustered with multiple nodes to deliver high availability (HA) across a local area network (LAN) and/or disaster recovery (DR) on a wide area network (WAN). No additional software is required to enable these features because StrongKey has enabled this capability as a standard feature in its FIDO2 server. Furthermore, with multiple nodes processing FIDO2 transactions at the same time, the SKFS cluster can deliver higher throughput to multiple web applications that use this server. This document guides you through the setup of a cluster with two nodes, as depicted in the image below.

    The clustering capability in SKFS only applies to the FIDO2 capability. Web applications that use SKFS must make their own arrangements to deliver HA and/or DR independent of SKFS. The sample application used here to demonstrate FIDO2 clustering will, itself, not be highly available, but demonstrates that the web application can use either or both FIDO2 servers in this HA configuration.

    While it is possible to add more than two nodes to the cluster, IT architects will recognize that there is a trade-off with N-way replication designs—the more nodes, the higher the resource requirements on each node to manage fail-safe replication, which can reduce the overall throughput after a certain point. Each site will have to do its own testing to discern where the throughput curve flattens out. However, if you have truly large-scale deployments in mind, please contact us to see how we can help.

     

    See also Install HAProxy Load Balancer.

     

    Sample Cluster Configuration

    Prerequisites

    • Two virtual machines (VMs) for the FIDO2 servers, running the current version of CentOS Linux 7.x, with fully qualified domain names (FQDN) and internet protocol (IP) addresses.

    • One VM for the load balancer, running HAProxy version 1.5.18 software on the current version of CentOS Linux 7.x with an FQDN and an IP address

    • One VM for the StrongKey sample Proof-of-Concept web application from this GitHub repository, also running on the current version of CentOS Linux 7.x with an FQDN and an IP address.

    NOTE: This document assumes you are setting up this cluster with all nodes connected to a single Ethernet switch. If your intent is to do a more realistic test, you should plan on using VMs with multiple network interfaces connected to different switches to isolate traffic to the appropriate segments, as you might expect in a real-world environment.

    Cluster Setup

    1. Using the Standalone Installation steps, install and configure the two FIDO2 server VMs as if they were individual FIDO2 servers, but do NOT install any web applications to test the FIDO2 server at this point; we will do this later.

    2. For each server determine the FQDN and assign it a unique Server ID. A Server ID (SID) is a numeric value that uniquely identifies a node within the cluster. Conventionally, StrongKey cluster SIDs begin with the numeral 1 and continue incrementally for each node in the cluster.

      • SID 1 is fidoserver1.strongkey.com

      • SID 2 is fidoserver2.strongkey.com 

    3. As the root user, perform the following sub-tasks on every node to be in the cluster:

      1. Login as root.

      2. If DNS is configured, make sure it is configured for forward and reverse lookups—meaning that it should be possible to resolve the IP address using the FQDN, as well as resolve the FQDN using the IP address doing a reverse lookup. Without the reverse resolution, services in the Payara application server configuration will not work correctly.

        If Domain Name Service (DNS) is not configured, add the following entries to the /etc/hosts file to identify the cluster nodes. Use a text editor such as vi to modify the /etc/hosts file. For the two-node cluster, add the following to the end of the hosts file, substituting the strongkey.com domain name for your own environment:

        <ip-fidoserver1>	fidoserver1.strongkey.com fidoserver1
<ip-fidoserver2>	fidoserver2.strongkey.com fidoserver2

      3. Modify the firewall configuration to open ports 7001, 7002, and 7003 to accept connections between just the FIDO2 servers to enable multi-way replication. Run the following command once for each cluster node's IP address (substituting for <ip-target-fidoserver>).

        Do not execute this command for the IP address of the cluster node on which you are executing the command itself. It is not necessary to open the node's ports on the firewall for itself, since the replication module in SKFS does not need to replicate to itself.

        shell> firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address='<ip-target-fidoserver>' port port=7001-7003 protocol=tcp accept'

      4. After adding the new rule, restart the firewall:

        shell> systemctl restart firewalld

      5. Logout from the root account.

    4. As the strongkey user (the default password is ShaZam123), perform the following on every SKFS node to be clustered:

        1. Using a text editor, edit the configuration properties of the SKFS node; if the specified file is empty add these properties:

          shell> vi /usr/local/strongkey/appliance/etc/appliance-configuration.properties
appliance.cfg.property.serverid=<server-id> (set the value to the corresponding SID of the current node)
appliance.cfg.property.replicate=true (should be set to true)

        2. Using the MySQL client, login to the MariaDB database that was installed as part of the SKFS installation. The default password for the skfsdbuser is AbracaDabra.

          shell> mysql -u skfsdbuser -p skfs

        3. Truncate the existing SERVERS table—this deletes all contents of the SERVERS table:

          mysql> truncate SERVERS;

        4. Insert the following entries into the SERVERS table, ensuring the SID and FQDN match the values used in Step 2. For example, if there are two nodes in the cluster, add the following entries:

          mysql> insert into SERVERS values (1, 'fidoserver1.strongkey.com', 'Active', 'Both', 'Active', null, null);
mysql> insert into SERVERS values (2, 'fidoserver2.strongkey.com', 'Active', 'Both', 'Active', null, null);

        5. Logout of the MySQL client:

          mysql> exit

        6. Import the self-signed certificates generated as part of the FIDO2 server installation into the Payara Application Server's truststore—this is necessary to ensure that replication between the FIDO2 server nodes occurs over a trusted Transport Layer Security (TLS) connection. Execute the certimport.sh script included in the /usr/local/strongkey/bin directory to import the certificate.

          shell> /usr/local/strongkey/bin/certimport.sh fidoserver1.strongkey.com -kGLASSFISH
shell> /usr/local/strongkey/bin/certimport.sh fidoserver2.strongkey.com -kGLASSFISH
        7. All SKFS instances are using the same JWT and SAML keys, copy the JWT and SAML keys from the first server to replace the keys on the other servers. These keys can be found in the /usr/local/strongkey/skfs/keystores directory 
          shell> scp -r /usr/local/strongkey/skfs/keystores strongkey@<hostname-NewSID>:/usr/local/strongkey/skfs

        8. Restart the Payara Application Server (even though we refer to it as the Payara Application Server, the startup script is named glassfishd for legacy reasons, given Payara's origins from the open-source GlassFish Application Server):

          shell> sudo service glassfishd restart

        9. Repeat Steps 3 and 4 on the remaining SKFS nodes of this cluster.