SW that should support high availabity (up time including planned maintenance 99.999% or better).
Problem:
Imagine that you write SW that must not be down for more than few minutes per year, where this time includes the time of SW upgrades that might occur from time to time. Imagine also that any down time period causes "a headache" for the user of the SW and such scenario should really be avoided. An example for such system can be a network switch that in case the switch is down (let's say over 50 milliseconds) it could lead to network topolgy changes and preventing user from getting the service the operator is committed to. While usually in such switches firmware upgrade (FPGA, network processer, ...) is relatively fast (up to few hundreds milliseconds and sometimes even less than 50 milliseconds) this is far from being the case about SW upgrade, that can take up to several minutes in some cases.
As always, let's try to make things complicated and I will limit the discussion to a system without a redundant CPU or another machine that can "take over" in case our system is down during the upgrade. In addition let's assume that new SW has a very different data scheme.
Solution:
The proposed solution is based on the ability of the OS to run several processes at the same time (for most of view this sounds obvious but for instance VxWorks 5.X and below are not capable to do so). The solution is composed of the following steps:
1. Running SW downloads the new SW and launches it (upon download completion or at some time later on).
2. New SW has a protocol to communicate with the old SW and to recieve its persistent and transient data. Note: the new SW should be aware of the data scheme of the current scheme in order to convert is to the new scheme.
3. New SW performs a data conversion from the old and new data schemes. Relatively simple conversions (such as adding a new column in a table or types expansions (e.g. int to long)) can usually be done "automatically" or by centralized mechanism. However, when a change is complex (e.g. split of a single table to several tables) then a custom upgrade code should be written.
4. When the new SW completes this stage, it signals the old SW that it is ready to "take over", the old is terminated (gracefully) and the new SW is becoming active and takes control. This step can be performed very fast (usually few tens milliseconds) which satisfies even systems with demanding requirements for high availability.
Implications:
Performance: during the entire procedure the "old" SW is still processing data and modifies its persistent and transient data. It should be assumed that during the upgrade procedure, the old SW is not fully busy and allows enough available CPU to synchronize the new SW and enough CPU to allow the new SW performing its data convertion procedure. In addition we need to assume that the rate of data sychronization and conversion is faster than data modifications in the "old" SW so the new SW can "catch up" with the data of the old SW.
Resources Usgae: in order to perform the proposed scheme the new SW should have enough available memory, persistent storage (flash), OS resources (such as file descriptors and more),... In cases there is not enough memory to allocate the needed memory for both old and new SW, the procedure described below is becoming much more complex and requires simulatanous synchronization to the new SW and deletion from the old SW (assuming at least some of the data is rarely changed and is not needed at least for the time of the upgrade procedure). Another option for case that some data remained unchanged is to copy this data to a shared memory, thus preventing the need to have a "duplication" of it.
Variations:
- Systems with redundant contoller/CPU can launch the new SW on the redundant Controller reducing CPU load on the active CPU.
- SW that is composed of several processes might upgrade independent modules one at a time (or even some of the processes only), but that will be discussed in another post, stay tuned :).
No comments:
Post a Comment