Addition of a new server starts by updating the Existing nodes to reflect changes to the cluster. Simple modifications will be made to the database and the Glassfish application server will be restarted for these changes to take effect.

1. Log in as 'strongauth' into the SAKA
2. Before performing any updates to the database, create a database backup

   shell> mysqldump -u skles -p strongkeylite > /usr/local/strongauth/dbdumps/strongkeylite-MM-DD-YYYY.db

3. Log into mysql database 'strongkeylite' as the 'skles' user

   shell> mysql -u skles -p strongkeylite

4. The new appliance to be added into the cluster must be assigned a sequential Server ID (SID). If this is to be the third appliance in the cluster, it should have an SID = 3. Add an entry to the servers table for the new server

   mysql> insert into servers values (SID, 'FQDN', 'Active', 'Both', 'Active',null,null);

   SID must be the numeric value of the new SID to be added to the cluster.
   
   FQDN must be the Fully Qualified Domain Name of the new appliance that will be added to the cluster.
   
   Adding a new server with the Fully Qualified Domain Name (FQDN) 'saka03.<domain-name>' with SID = 3, the command would be

   mysql> insert into servers values (3,'saka03.<domain-name>','Active','Both','Active',null,null

5. A new entry must be added to the server_domains table for each Domain ID (DID) that is present in the cluster. For instance, if there exists 3 domains in the cluster, there must be a new record in the server_domains table for SID=3 DID=1, SID=3 DID=2, and SID=3 DID=3. Add an entry in the server_domains table for each domain
   

   mysql> insert into server_domains values (SID, DID, 'STARTING_PSEUDONUMBER','Active',null,null); 

   SID must be the numeric value of the new SID to be added to the cluster.
   
   DID must be the value of one domain already existing in the cluster. You can see what domains currently exist with the mysql command
   

   mysql> select * from domains\G

   STARTING_PSEUDONUMBER is the first token to be used by the new server. This value can be any number that is the same length as the appliance configured token length (default 16 digits). This value can be reused between multiple domains. A value of '3000000000000001' is the suggested format for SID 3.
   
   Adding a new server with SID = 3 to a cluster with DID 1 and 2, the commands would be

   mysql> insert into server_domains values (3,1, '3000000000000001','Active',null,null); 
   mysql> insert into server_domains values (3,2, '3000000000000001','Active',null,null); 

6. Log out of mysql and change to the 'root' linux account

   shell> su -

7. Using a text editor (gedit or vi), edit the firewall settings to unblock replication between the existing appliances and the new appliance. There should be a set of rules for port 7001, 7002, and 7003 that need to the IPAddress Range modified to incorporate the new appliance's IP Address. Once the changes have been saved, restart the firewall

   shell> service iptables restart

8. Logout of the 'root' account

   shell> logout

9. In order for the new configurations to take effect, the application on the SAKA appliance must be restarted. All customer traffic to this appliance should be routed to one of the other appliances. Once the appliance has stopped receiving new transactions, payara can be restarted

   If using payara6, use the following command:
   shell> sudo systemctl restart payara
   
   If using payara5, use the following command:
   shell> sudo service glassfishd restart
   

10. The three KeyCustodians must activate cryptographic services using their USB Tokens by running the KC-SetPINTool either directly on the appliance or remotely from their own laptops/workstations. Once the KeyCustodians have completed their task, appliance functionality should be verified using the ping webservice.

    shell> java -cp /usr/local/strongauth/topaz/sakaclient.jar ping https://saka01.<domain-name>:8181 <DID> <'pinguser' password>

11. Steps E-1-1 through E-1-10 should be repeated on all other existing nodes.
12. A database dump from one of the existing servers must be created to be imported into the new appliance. To ensure no records are lost during this process, it is recommended that all traffic be routed through a single appliance. Before the database dump is created, verify that any other processing appliances have completed replication to the others. An empty replication table on any appliance would indicate that it is up to date with replication

    mysql> select * from replication where tsid!=SID;

    SID must be the numeric value of the new SID to be added to the cluster.
    
    With replication up to date, create the database dump on the 'primary' appliance using the --ignore-table=strongkeylite.replication, --no-create-info, and --replace options

    shell> mysqldump -u skles -p strongkeylite --ignore-table=strongkeylite.replication --no-create-info –-insert-ignore > /usr/local/strongauth/dbdumps/strongkeylite-newserver.db

    Once this file has been successfully generated, transactions can be sent to all appliances again.