On the fly Reconfiguration

On-the-fly reconfiguration, or live reconfiguration, or in-service configuration update, can be used to change Tempesta FW configuration without stopping, unnecessary backend server reconnects, client disconnects, HTTP sessions losses.

Tempesta FW allows to update load balancing rules on the fly. It is allowed to:

• listening address, port and proto
• add a new server group;
• add a new server to a server group;
• add more connections to a server;
• remove server from a server group;
• remove entire server group;
• change scheduler attached to a server group;
• modify various server group options;
• modify virtual hosts, locations and related options;
• modify HTTPtables rules to change load distribution across virtual hosts;
• modify traffic filtering by fingerprints rules;
• change count of allowed control frames per second for a HTTP/2 connections;
• TF hash filters for online analytics and bots blocking

The list of available on-the-fly reconfiguration actions may be extended in future versions.

Using on-the-fly reconfiguration is simple:

• update configuration file;
• request for on-the-fly reconfiguration using startup script or systemd unit file:

$ ${tempesta_src_dir}/scripts/tempesta.sh --reload
systemctl reload tempesta-fw.service

After reconfiguration is requested Tempesta FW will parse the updated configuration file and apply it. If the updated configuration is not valid it won’t be applied and Tempesta FW will continue to use last valid configuration.

Make sure there is enough free memory space to use the updated configuration. Otherwise critical errors may happen during applying of the updated configuration and Tempesta FW will be fully stopped since it’s effective configuration will be far away from both original and updated configurations.

Tempesta FW supports graceful server removal. After a server is removed from its server group, the server connections are moved from active to grace-shutdown state, and only after the grace shutdown timeout runs out the connections are permanently closed and removed from the configuration.

The grace-shutdown state is a special state of a backend server connection. Similarly to active state connections, Tempesta FW will try to reopen failed or closed by backend server connections. But unlike active connections, the grace-shutdown are used only to serve either already scheduled (and forwarded) requests or to serve any amount of new requests from already pinned HTTP sessions.

HTTP sessions can be almost infinite, thus the grace_shutdown_time configuration option specifies maximum time limit in seconds to wait for sessions to close before all connections to the server are terminated. If the grace timeout is not set (defaults), then all the connections to the removed server are terminated immediately on reconfiguration.

Grace shutdown process is started only if the original server group is still used in the configuration. If the whole server group is fully removed from the configuration, including its vhost and corresponding http_chain rules, then no rules in the new configuration will match new requests to existing HTTP sessions and the removed server groups. The newly reconfigured http_chains are matched to new requests before the HTTP sessions are identified. Usually the new rules blocks the requests to removed server groups and vhosts.

On-the-fly reconfiguration is not applicable for some cases and the full restart is required.

Not all configuration options may be changed during on-the-fly reconfiguration. All configuration directives that can be changed during reconfiguration are listed in On-the-fly Configuration possibilities section.

Sometimes there is no logically correct transition from original to updated configuration even if both configurations are semantically valid and correct. E.g. The same vhost name or is configured to handle completely different HTTP application after reconfiguration. Or HTTP tables has changed significantly so the same requests will be forwarded to different vhosts. Since Tempesta FW doesn’t break existent user sessions, clients may receive unexpected content. Such changes defines application logic, which Tempesta FW is not aware of, so such transition issues can’t be spotted and reported.