Addition of a new server starts by updating the Existing nodes to reflect changes to the cluster. Simple modifications will be made to the database and the Payara application server will be restarted for these changes to take effect.

1. Log in as 'strongkey' into the SKFS
2. Before performing any updates to the database, create a database backup

   shell> mariadb-dump -u skfsdbuser -p skfs > /usr/local/strongkey/dbdumps/skfsdata-MM-DD-YYYY.db

3. Log into mysql database 'skfs' as the 'skfsdbuser' user

   shell> mysql -u skfsdbuser -p skfs

4. The new appliance to be added into the cluster must be assigned a sequential Server ID (SID). If this is to be the third appliance in the cluster, it should have an SID = 3. Add an entry to the servers table for the new server

   mysql> insert into servers values (SID, 'FQDN', 'Active', 'Both', 'Active',null,null);

   SID must be the numeric value of the new SID to be added to the cluster.
   
   FQDN must be the Fully Qualified Domain Name of the new appliance that will be added to the cluster.
   
   Adding a new server with the Fully Qualified Domain Name (FQDN) 'skfs03.<domain-name>' with SID = 3, the command would be

   mysql> insert into servers values (3,'fidoserver03.<domain-name>','Active','Both','Active',null,null);

5. Using a text editor, list the configuration properties of the SKFS node. Since this is server ID 1 (the primary node), the replication should be set to true. if the specified file is empty add these properties: 
   

    shell> vi /usr/local/strongkey/appliance/etc/appliance-configuration.properties
   
   appliance.cfg.property.serverid=1
   appliance.cfg.property.replicate=true
   

6. Log out of mysql and change to the 'root' linux account

   shell> su -

7. Modify the firewall configuration to open ports 7001, 7002, and 7003 to accept connections between just the FIDO2 servers to enable multi-way replication. Run the following command once for each cluster node's IP address (substituting for <ip-target-fidoserver>). 

   Do not execute this command for the IP address of the cluster node on which you are executing the command itself. It is not necessary to open the node's ports on the firewall for itself, since the replication module in SKFS does not need to replicate to itself.

   shell> firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address=''<ip-target-fidoserver>' port port=7001-7003 protocol=tcp accept'
   shell> firewall-cmd --reload
   

8. Logout of the 'root' account

   shell> logout

9. In order for the new configurations to take effect, the application on the SKFS must be restarted. All customer traffic to this appliance should be routed to one of the other appliances. Once the appliance has stopped receiving new transactions, payara can be restarted

   shell> sudo systemctl restart payara

10. The above steps should be repeated on all other existing nodes.
11. A database dump from one of the existing servers must be created to be imported into the new appliance. To ensure no records are lost during this process, it is recommended that all traffic be routed through a single appliance. Before the database dump is created, verify that any other processing appliances have completed replication to the others.
    With replication up to date, create the database dump on the 'primary' appliance using the --ignore-table=skfs.replication, --no-create-info, and --replace options

    shell> mariadb-dump -u skfsdbuser -p skfs --ignore-table=skfs.replication --no-create-info --insert-ignore > /usr/local/strongkey/dbdumps/skfs-newserver.db

    Once this file has been successfully generated, transactions can be sent to all appliances again.