--- canonical: https://safekit.evidian.com/wp-content/uploads/downloads_safekit/version-82/safekituserguidehtml/documentation/safekituserguideen.htm --- ## 10.9          SafeKit web service settings SafeKit comes with a web service, safewebserver, which runs on each SafeKit server. It is a standard Apache web service that is **mandatory** for running: ·         the web console (see section 3) ·         the distributed command line interface (see section 9.8) ·         the checkers (see section 13.17) safewebserver starts automatically at the end of SafeKit package install and on server reboot. If you do not need the SafeKit web service and want to remove the automatic boot start, refer to section 9.1.2. The default configuration is HTTP with file-based authentication, initialized with: ·         a single admin user that got the Admin role for the web console. The role can be changed via configuration files ·         a private rcmdadmin user to execute the distribute safekit command The usernames and passwords for these users can be changed, if necessary, as described in section 9.1.2. ### 10.9.1      Configuration files The configuration of an instance of safewebserver on a SafeKit server is contained in the SAFE/web/conf directory. It consists in standard Apache configuration files (see https://httpd.apache.org/). The configuration is split into many files, but for most common configurations, only the main configuration file httpd.conf need to be modified.   | | | | --- | --- | | Commentaire important contour | ·         After changes, you must restart the service with the command: safekit webserver restart (see section 9.1). ·         Do not edit .default files since they are backups of delivered configuration files. |   The httpd.conf file consists essentially in a set of Define statements. Comment character # disables the definition. The mains Define are: | | | --- | | Connection port definition: Define httpport 9010 Define httpsport 9453 Set the listening port in http and https mode. (See section 10.9.2  for usage). | | User authentication definition: Define usefile # Define useldap # Define useopenid … Select which user authentication to use. At most one must be defined. usefile is the default. (See section 11.4 for details.) | | Apache logging definition: #Define Loglevel info #Define accesslog Uncomment these lines to enable the logging for debug purposes. Logging files httpd.log and access.log are in SAFEVAR. | | Session validity period definition: Define SessionMaxAge 28800 Since SafeKit 8.2.1, the user is automatically logged out after 8 hours of inactivity (28800 seconds). If necessary, adjust this value. |   Other Define are self-documented in the httpd.conf file.   The other configuration files are listed below. Modifying one of them may cause problems when upgrading SafeKit: | | | | --- | --- | | Global configuration | **httpd\_main.conf** | | File based authentication and role mapping | **httpd.webconsolefileauth.conf** | | Form authentication configuration | **httpd.webconsoleformauth.conf** | | LDAP/AD authentication configuration | **httpd.webconsoleldap.conf** using a LDAP/AD server | | OpenID Connect authentication configuration | **httpd.webconsoleopenidauth.conf** using an OpenID connect identity provider | | HTTPS configuration | **httpd.webconsolessl.conf** in SAFE/web/conf/ssl | User authentication configurations may optionally use group.conf (for HTTP) or sslgroup.conf (for HTTPS) files in SAFE/web/conf for user to role mapping. ### 10.9.2      Connection ports configuration By default, connect the web console with the URL http://host:9010. The SafeKit web server will redirect to the appropriate section according to your security settings. If you need to change the default value: 1.    Edit SAFE/web/conf/httpd.conf and change the value of httpport or httpsport variables. 2.    Restart the service using the command safekit webserver restart.   The HTTP and HTTPS configurations cannot be active simultaneously. See section 11.3 for how to configure HTTPS. The port value 9010(HTTP)/9453(HTTPS) is also used by the module checker. Therefore, if the configuration of an application module defines a checker: 1.    Edit the module configuration file userconfig.xml 2.    Edit the port attribute and assign it to the new port value                3.    Apply the new configuration of the module ### 10.9.3      HTTP/HTTPS and user authentication configuration ·         The default configuration is for HTTP. The default configuration is also set with file-based authentication, initialized with a single admin user that got the Admin role. ·         The HTTPS configuration requires the installation of certificates and the definition of user authentication. For a detailed description, see section 11. To re-enable the HTTP configuration if it has been changed to HTTPS see section 11.2.1.1. ### 10.9.4      SafeKit API Use Swagger UI to visualize and interact with the SafeKit API provided by the SafeKit web service. To access it, open a browser at the following URL: http://localhost:9010/swagger-ui The endpoints presented are used by the SafeKit web console, but they can also be invoked from scripts or external management tools to automate operations on the SafeKit cluster. From the local node, the API of any node in the cluster can be accessed in proxy mode. In this case, the API entry point is prefixed with: /nodes/ where corresponds to the node name as defined in the cluster configuration. Requests to the API use the same authentication mechanism as the one configured for the SafeKit web console (see section 11.4). Since SafeKit 8.2.6, to facilitate cluster management automation, it is also possible to authenticate using a Bearer token, but only for direct API calls (not from the SafeKit web console). Examples of API access using the curl command are provided below. On Windows, this command is provided in SAFE/private/bin. #### 10.9.4.1  Accessing the API using the default authentication The default configuration uses HTTP and includes file-based authentication, initialized with a single user admin having the Admin role. The following examples show how to authenticate to the SafeKit web service, retrieve the session cookie, and then call SafeKit API requests using this cookie. | | | | --- | --- | | Commentaire important contour | The following examples are for HTTP. When configured for HTTPS, change the URL to https://localhost:9453 and add the option --cacert $SAFE/web/conf/cacert.crt to the curl command. | Authentication The following request sends the credentials of the admin user with password passwd to the /login endpoint.   curl -c cookies.txt -d "httpd\_username=admin&httpd\_password=passwd&login=Login" http://localhost:9010/login The server returns a session cookie (sessionSK), which is stored in the cookies.txt file. Calling API The following request retrieves information about the local node by passing the previously obtained cookie stored in cookies.txt. curl -b cookies.txt http://localhost:9010/api/v1.0/level The following request retrieves information about the node node2 in the cluster. curl -b cookies.txt http://localhost:9010/api/v1.0/nodes/node2/level If preferred, you can explicitly pass the cookie value in the request header as follows: curl -H "Cookie: sessionSK=" http://localhost:9010/api/v1.0/level Similar calls can be made for any API endpoint. #### 10.9.4.2  Accessing the API using a bearer token The following examples show how to manage a Bearer token and then call SafeKit API requests using it. The management of Bearer tokens is itself exposed through the apikeys API in Swagger. To access it, you must first authenticate using a user with the Admin role. This API allows you to: ·         Create a new key by specifying its role (Admin, Control, or Monitor) and its logical name. Once created, the associated Bearer token value is returned and must be stored in a secure location. There is no way to retrieve this token if its value is lost. ·         Push the new key to all nodes in the cluster. This allows a single Bearer token to be used to access the SafeKit API locally or in proxy mode on other nodes. ·         Delete a key. These features can be accessed either directly, for example using the curl command, or through a dedicated web interface. | | | | --- | --- | | Commentaire important contour | The following examples are for HTTP. When configured for HTTPS, change the URL to https://localhost:9453 and add the option --cacert $SAFE/web/conf/cacert.crt to the curl command. | Managing bearer tokens using the web interface The web interface dedicated to bearer token management is not accessible by default for security reasons. To enable access: 1.    Edit the file SAFE/web/conf/httpd.conf 2.    Uncomment ApiKeyManagement Define ApiKeyManagement 3.    Restart the web service with safekit webserver restart 4.    Open a browser and navigate to the following URL: http://localhost:9010/keymgmt/keymgmt.html On the login page, authenticate using a user with the Admin role. Managing bearer tokens using curl 1.    Authenticate with a user having the Admin role curl -c cookies.txt -d "httpd\_username=admin&httpd\_password=passwd&login=Login" http://localhost:9010/login The server returns a session cookie (sessionSK), which is stored in the cookies.txt file. 2.    Create a new key with the Admin role and for example the logical name administrator ·         In PowerShell on Windows $token=curl -b cookies.txt http://localhost:9010/api/v1.0/apikey/Admin/administrator -d " " ·         In bash on Linux token=`curl -b cookies.txt http://localhost:9010/api/v1.0/apikey/Admin/administrator -d " "` The server returns the associated bearer token, which is stored in the variable token for the remainder of this example. 3.    Push the new key to all nodes in the cluster curl -b cookies.txt http://localhost:9010/api/v1.0/apikey/push/administrator -d " " Calling the API using the bearer token The following request retrieves information about the local node by passing the bearer token previously stored in the token variable. curl -H "Authorization: Bearer $token" http://localhost:9010/api/v1.0/level The following request retrieves information about the node node2 in the cluster. curl -H "Authorization: Bearer $token" http://localhost:9010/api/v1.0/nodes/node2/level Similar calls can be made for any API endpoint. ## 10.10      SafeKit email notification agent Since SafeKit 8.2.4, SafeKit offers a notification agent that sends emails for major events on modules. These events are extracted from the system log (see section 10.13), which is populated by the log messages of modules configured on the SafeKit server. Using this feature requires that your company's IT team has set up an SMTP server that can be accessed by the agent running on SafeKit nodes. During the SafeKit installation, the notification agent is installed but disabled by default. The following procedure is required to configure and enable it. Apply it on all SafeKit nodes: ![*](safekituserguideen_fichiers/image001.png)       SafeKit notification agent configuration described in section 10.10.1 It allows you to define the SMTP server, the email recipients, the selection of events to be sent… ![*](safekituserguideen_fichiers/image001.png)       Optional SMTP client credentials setup described in section 10.10.2 It allows you to define the username and password needed to send an email if authentication is required by the SMTP server. ![*](safekituserguideen_fichiers/image001.png)       Email sending test described in section 10.10.3 This allows you to verify that your configuration is functional for sending emails. ![*](safekituserguideen_fichiers/image001.png)       SafeKit notification agent activation described in section 10.10.4 Once activated, major events from modules on this server will be automatically sent by email to the configured recipients.   | | | | --- | --- | | Commentaire important contour | In Windows, PowerShell 5 is required, which is the default version, as it is not compatible with PowerShell 7. |   Below is an example of an email sent by the SafeKit notification agent set up on node1: ![](safekituserguideen_fichiers/image315.png) With the default agent configuration, when an application module critical or state change event occurs on node1, the agent gathers all other events within the following minute and sends them in a single email. In case of issues, refer to the section 7.21 for assistance. ### 10.10.1  SafeKit notification agent configuration Follow the steps below to configure the SafeKit notification agent: 1.    Open a PowerShell/shell window as administrator/root 2.    Change directory to SAFE/notifications where SAFE=C:\safekit in Windows (if %SYSTEMDRIVE%=C:), and SAFE=/opt/safekit in Linux 3.    Copy safenotif\_conf.json.default to safenotif\_conf.json 4.    Edit safenotif\_conf.json to set up your configuration safenotif\_conf.json  is a self-documented configuration file in which you must at least configure the fields in the following excerpt: {     // ...     "emailNotifications": {         // ...         "sender": "noreply@it-smtp-server.my.company.com",         "recipients": [             "my.name@my.company.com"         ],         // ...     },     "smtp": {         "host": "it-smtp-server.my.company.com",         "port": ***25***,         "protocol": ***"smtp+starttls"***         // ...     } } Where: ·         "sender" is the email address to send emails from ·         "recipients" is the list of email addresses to send emails to ·         "host" is the hostname or IP address of the SMTP server ·         "port" is the port the SMTP server listens on ·         "protocol" is the protocol to use to connect to the SMTP server. In this example, it is set to an encrypted connection, initiated from a STARTTLS command.   By default, only critical messages and local state changes of the module are sent by emails. See safenotif\_conf.json  to select other messages, if necessary, change the sending delay or for other configuration options. | | | | --- | --- | | Commentaire important contour | After upgrading SafeKit, you may need to reconfigure the notification agent if its configuration file format has changed between versions. |   You can check the safenotif\_conf.json  for common logic errors with: SAFE/private/bin/safenotif -testconfiguration It returns a failure if an error is detected. Fix it before proceeding to the next step.   | | | | --- | --- | | Commentaire important contour | On Linux, once the notification agent is enabled, any modification to the safenotif\_conf.json file requires restarting the service for the changes to be applied, using the following command: systemctl restart safenotif.service. | ### 10.10.2  SMTP client credentials setup for authentication SMTP servers are usually configured to require SMTP clients to specify credentials (username, password). The SafeKit notification agent (which embeds a SMTP client) stores such credentials in the file system in a secure, encrypted way. For this follow the steps below: 1.    Open a PowerShell/shell window as administrator/root | | | | --- | --- | | Commentaire important contour | For certain versions of Windows, a Windows Terminal (wt) is required instead of a PowerShell window. | 2.    Change directory to SAFE where SAFE=C:\safekit in Windows (if %SYSTEMDRIVE%=C:), and SAFE=/opt/safekit in Linux 3.    Run ./private/bin/smtpcfg credentials set This command prompts you to enter the username and password that correspond to the email sender account. These credentials will then be used by the SafeKit notification agent to send mail. To reset the credentials, and consequently disable the authentication, run: SAFE/private/bin/smtpcfg credentials none ### 10.10.3  Email sending test Once the SafeKit notification agent configured (see section 10.10.1), and the SMTP credentials set if authentication is required (see section 10.10.2), the following procedure can be used to send a test e-mail, using this configuration: 1.    Open a PowerShell/shell window as administrator/root 2.    Change directory to SAFE where SAFE=C:\safekit in Windows (if %SYSTEMDRIVE%=C:), and SAFE=/opt/safekit in Linux 3.    Run ./private/bin/safenotif -testemail Here is a sample output of this command: Sending email from noreply@it-smtp-server.my.company.com to my.name@my.company.com with SMTP account noreply on server it-smtp-server.my.company.com... Email sending successful, check your mailbox(es). Although the command is successful, it’s advisable to check the recipient’s mailbox that you configured to ensure the email was properly received. If the email test fails, resolve the issue before proceeding to the next step. Refer to the section 7.21 for assistance. ### 10.10.4  SafeKit notification agent activation Once you have ensured that your configuration is functional, apply the following procedure to enable the notification agent: 1.    Open a PowerShell/shell window as administrator/root 2.    Change directory to SAFE where SAFE=C:\safekit in Windows (if %SYSTEMDRIVE%=C:), and SAFE=/opt/safekit in Linux 3.    Run ./safekit notification enable Once activated, major events from modules on this server will be automatically sent by email to the configured recipients. To deactivate the notification, run: SAFE/safekit notification disable To check the status of the notification agent, run: SAFE/safekit notification status ## 10.11      SNMP monitoring SafeKit could be monitored by snmp. Since version 8, snmp monitoring implementation differs in Windows and Linux: In Windows, SafeKit use its own snmp agent service, when in Linux, the operating system’s snmp agent is used. ### 10.11.1  SNMP monitoring in Windows For using the SafeKit SNMP agent, you must: 1.    configure it to start on boot, with the command | | | | --- | --- | | safekit boot [snmpon | snmpoff | snmpstatus] | Controls the automatic start at boot of the Net-SNMP Agent service ("on" or "off"; by default, "off") |   2.    add the corresponding firewall rule When using the operating system firewall, the firewall has already been configured for Net-SNMP Agent if you have applied the command: SAFE/private/bin/firewallcfg add 3.    start it with the command | | | | --- | --- | | safekit safeagent [start | stop | restart | check] | Controls start/stop of the Net-SNMP Agent service that implements the SafeKit SNMP agent. |   The configuration of the Net-SNMP Agent is defined in the self-documented **SAFE/snmp/conf/snmpd.conf** file. It is a standard net-snmp configuration file as described in http://net-snmp.sourceforge.net. By default, the service is listening on UDP **agentaddress** port 3600 and accepts read request from the public community and write requests from the private community. Read requests are used to get module status and write requests to run actions on the module. You can change the default configuration according to your needs. When you modify snmpd.conf, you must manually change the firewall rule and restart the service to load the new configuration with: safekit safeagent restart. | | | | --- | --- | | Commentaire important contour | Since SafeKit 8, the service name is Net-SNMP Agent instead of safeagent in previous releases. | ### 10.11.2  SNMP monitoring in Linux Since SafeKit 8, Safekit did not come with its own snmp agent anymore, so the following safekit commands are obsoleted in Linux**:**safeagent install, safeagent start, safeagent stop, boot snmpon, boot snmpoff, boot snmpstatus. Instead, it is possible to configure the standard snmpd Linux agent to access safekit mib: 1.    Install net-snmp dnf install net-snmp net-snmp-utils 2.    If selinux is in enforced mode, you have to set snmpd in permissive mode for snmp by: semanage permissive -a snmpd\_t 3.    If firewall is active, you have to open the snmp ports with: firewall-cmd --permanent --add-service snmp firewall-cmd --reload 4.    Edit /etc/snmp/snmpd.conf Add the following lines: pass .1.3.6.1.4.1.107.175.10 /opt/safekit/snmp/bin/snmpsafekit view systemview included .1.3.6.1.4.1.107.175.10 Note: the “view systemview” line set the access rights. You could have to adapt it to your general snmpd configuration. 5.    Enable and start the snmp agent systemctl enable snmpd systemctl start snmpd ### 10.11.3  The SafeKit MIB The SafeKit MIB is common to Windows and Linux implementation. It is delivered in **SAFE/snmp/mibs/safekit.mib** . The SafeKit MIB is accessed with the following identifier (OID, prefix of SafeKit SNMP variables): **=** **enterprises.bull.safe.safekit (1.3.6.1.4.1.107.175.10)**. The SafeKit MIB defines: ·         The module table: **skModuleTable** The index on the module table is the ID of the application module as returned by the command safekit module listid. Through the MIB, you can read and display the status of an application module on a server (STOP, WAIT, ALONE, UP, PRIM, SECOND) or you can take an action on the module (start, stop, restart, swap, stopstart, prim, second). For example, the status of the module with ID 1 is read by an SNMP get to the variable: enterprises.bull.safe.safekit.skModuleTable.skModuleEntry.skModuleCurrentState.1 = stop (0) Use the snmpwalk command to check all MIB entries. ·         The resource table: **skResourceTable** Each element defines a resource as for instance the one corresponding to the network interface checker "intf.192.168.0.0" and its status (unknown, init, up, down). Example: SNMP get request to enterprises.bull.safe.safekit.skResourceTable.skResourceEntry.skResourceName.1.2 means name of resource 2 in application module 1. ## 10.12      Commands log of the SafeKit server There is a log of the safekit commands ran on the server. It allows auditing the actions performed on the server to help support for instance. The log records all the safekit commands that are run and that modify the system such as an application module install and configuration, an application module start/stop, the safekit webserver start/stop, … The command log is stored in the SAFEVAR/log.db file in SQLite3 format. For viewing its content: ·         run the command safekit cmdlog or ·         click on the commands log tab into the web console Below is the raw extract of this log: | 2021-07-27 14:37:33.205122 |   safekit |   mirror |  6883 | START | config -m mirror | 2021-07-27 14:37:33.400513 |   cluster |   mirror |     0 |     I | update cluster state | 2021-07-27 14:37:33.405597 |   cluster |   mirror |     0 |     I | module state change on node centos7-test3 | 2021-07-27 14:37:34.193280 |           |          |  6883 |   END | 0 | 2021-07-27 14:37:34.718292 |   cluster |   mirror |     0 |     I | update cluster state | 2021-07-27 14:37:34.722080 |   cluster |   mirror |     0 |     I | module state change on node centos7-test4 | 2021-07-27 14:37:37.510971 |           |          |  6871 |   END | 0 | 2021-07-27 14:38:05.092924 |   safekit |   mirror |  7017 | START | prim -m mirror -u admin@10.0.0.103 | 2021-07-27 14:38:05.109368 |           |          |  7017 |   END | 0 Each field has the following meaning: ·         The 1st field in the logentry is the date and time of the message ·         The next one is the type of the action ·         The next one is the module name when the action is not global ·         The next one is the pid of the process that runs the command. It is used as the identifier of the log entry ·         The next ones are START when the command starts and the command’s arguments; or END when the command has finished with the return value. ## 10.13      SafeKit log messages in system log Since SafeKit 8, SafeKit modules log messages are sent to system log too. To view them: ·         In Windows, open a PowerShell window and run     Get-EventLog -Logname Application -Source Evidian.SafeKit that returns:    47086 Nov 23 11:27  Information Evidian.SafeKit        1073873154 mirror | heart | Remote state UNKNOWN Unknown...    47085 Nov 23 11:27  Information Evidian.SafeKit        1073873154 mirror | heart | Resource heartbeat.flow set to down by heart...    47084 Nov 23 11:26  Information Evidian.SafeKit        1073873154 mirror | heart | Local state ALONE Ready...    47082 Nov 23 11:26  Warning     Evidian.SafeKit        2147614977 mirror | heartplug | Action alone called by heart: remote stop...    47081 Nov 23 11:25  Information Evidian.SafeKit        1073873154 mirror | heart | Remote state PRIM Ready...    47080 Nov 23 11:25  Information Evidian.SafeKit        1073873154 mirror | heart | Local state SECOND Ready...    47079 Nov 23 11:25  Information Evidian.SafeKit        1073873154 mirror | rfsplug | Reintegration ended (default)... ·         In Linux, open a shell window and run journalctl -r -t safekit that returns: Nov 23 15:22:43 localhost.localdomain safekit[3689940]: mirror | heart | Local state ALONE Ready Nov 23 15:22:43 localhost.localdomain safekit[3689940]: mirror | heart | Local state PRIM Ready Nov 23 15:16:48 localhost.localdomain safekit[3689940]: mirror | heart | Local state ALONE Ready Nov 23 15:16:48 localhost.localdomain safekit[3690096]: mirror | userplug | Script start\_prim > userlog\_2023-11-23T151648\_start\_prim.ulog Nov 23 15:16:48 localhost.localdomain safekit[3690066]: mirror | rfsplug | Uptodate replicated file system Nov 23 15:16:24 localhost.localdomain safekit[3689940]: mirror | heart | Remote state UNKNOWN Unknown