My SW Architecture Blog

Simulator Usage in Embedded Environments

2009-11-15T10:27:00.000-08:00

Context:
Embedded system where the HW is not always available, difficult to debug and to test.

Problem:
Imagine that you develop in a complex embedded environment when the SW development is done in parallel to the HW development. Basically there is no option for you to test the SW till the HW is ready. Even when the HW is ready it is usually very slow to debug on the actual HW and in some cases only few HW devices are available for testing.
To make things even more problematic imagine that the code you develop requires several devices to perform a test, for instance in case you develop some network protocol that needs to interface with one or more devices to perform its operation.

Solution:
The proposed solution is based on development of HW simulator that can run on PC environment. Compilers of embedded OS (e.g. VxWorks) has an option to compile the code for X86 processors and have a version of the code running on PC environment.

Ha ha ha, well you think it's that easy? Imagine that part of your code (as often occurs in embedded systems) is writing data to FPGA, ASIC or other HW devices. Of course such code will fail on the PC environment since the actual HW does not exist.

So what can you do in order to test your code on PC environment? The answer resides in my 1st post: http://ranlevy-swarchitecture.blogspot.com/2009/10/decoupling-sw-from-platform.html. All you need to is to code plugins for PC environment that can simulate for you access for the HW and can simulate for you various events that are generated by the HW. A major issue to notice is that basically the application code remains unchanged.

Implications:

Development Efforts: well indeed there is some development overhead in creating a project for PC environment, writing plugins for PC env. and so forth. However, the efforts really worth the extra development since the total development cost is dramatically reduced since debugging and testing are becoming much easier.
Testing: SW that works well on the simulator environment will be easier to work well on the target, but of course it MUST be tested on the actual system. A common mistake in implementing "dummy" PC plugins that always return success. Such behavior might mislead application to developer from coding and testing the "bad" path of each flow. Beware!!!
Debugging: debugging in simulator mode is much easier than the actual embedded system which is usually much more complex.
Performance: the relatively good performance that might be achieved in PC environment are usually much slower in the actual embedded system since it uses slower CPU, memory, disk, etc. Therefore performance must be tested as soon as the HW is available and an extrapolation of the performance from the PC env. to the HW environment must be available for early performance prediction.
Compilation: compilation of the code for several architectures usually improves the code stability since sometimes the compiler detects potential errors that are not detected for other architecture.

In Service SW Upgrade (ISSU)

2009-10-31T05:21:00.000-07:00

Context:

SW that should support high availabity (up time including planned maintenance 99.999% or better).

Problem:

Imagine that you write SW that must not be down for more than few minutes per year, where this time includes the time of SW upgrades that might occur from time to time. Imagine also that any down time period causes "a headache" for the user of the SW and such scenario should really be avoided. An example for such system can be a network switch that in case the switch is down (let's say over 50 milliseconds) it could lead to network topolgy changes and preventing user from getting the service the operator is committed to. While usually in such switches firmware upgrade (FPGA, network processer, ...) is relatively fast (up to few hundreds milliseconds and sometimes even less than 50 milliseconds) this is far from being the case about SW upgrade, that can take up to several minutes in some cases.

As always, let's try to make things complicated and I will limit the discussion to a system without a redundant CPU or another machine that can "take over" in case our system is down during the upgrade. In addition let's assume that new SW has a very different data scheme.

Solution:

The proposed solution is based on the ability of the OS to run several processes at the same time (for most of view this sounds obvious but for instance VxWorks 5.X and below are not capable to do so). The solution is composed of the following steps:

1. Running SW downloads the new SW and launches it (upon download completion or at some time later on).

2. New SW has a protocol to communicate with the old SW and to recieve its persistent and transient data. Note: the new SW should be aware of the data scheme of the current scheme in order to convert is to the new scheme.

3. New SW performs a data conversion from the old and new data schemes. Relatively simple conversions (such as adding a new column in a table or types expansions (e.g. int to long)) can usually be done "automatically" or by centralized mechanism. However, when a change is complex (e.g. split of a single table to several tables) then a custom upgrade code should be written.

4. When the new SW completes this stage, it signals the old SW that it is ready to "take over", the old is terminated (gracefully) and the new SW is becoming active and takes control. This step can be performed very fast (usually few tens milliseconds) which satisfies even systems with demanding requirements for high availability.

Implications:

Performance: during the entire procedure the "old" SW is still processing data and modifies its persistent and transient data. It should be assumed that during the upgrade procedure, the old SW is not fully busy and allows enough available CPU to synchronize the new SW and enough CPU to allow the new SW performing its data convertion procedure. In addition we need to assume that the rate of data sychronization and conversion is faster than data modifications in the "old" SW so the new SW can "catch up" with the data of the old SW.

Resources Usgae: in order to perform the proposed scheme the new SW should have enough available memory, persistent storage (flash), OS resources (such as file descriptors and more),... In cases there is not enough memory to allocate the needed memory for both old and new SW, the procedure described below is becoming much more complex and requires simulatanous synchronization to the new SW and deletion from the old SW (assuming at least some of the data is rarely changed and is not needed at least for the time of the upgrade procedure). Another option for case that some data remained unchanged is to copy this data to a shared memory, thus preventing the need to have a "duplication" of it.

Variations:

Systems with redundant contoller/CPU can launch the new SW on the redundant Controller reducing CPU load on the active CPU.
SW that is composed of several processes might upgrade independent modules one at a time (or even some of the processes only), but that will be discussed in another post, stay tuned :).

Decoupling SW from Platform

2009-10-25T22:46:00.000-07:00

Context:

SW that is aimed to run on different platforms.

Problem:

Imagine that you need to write a SW that needs to run on completely different types of HW. An example for such SW can be a SW that runs in Telco products. Some of these products contain FPGAs , some of them contains Network Processors (NPU) or dedicated chip. These HW entities arrive with API that is usually wrapped by some platform code.

In addition different products might not support the same functionality - e.g. some products supports bridging functionality only, while other supports MPLS or even both.

To make things even more complicated, imagine that this SW is off the shelf product and cannot apriori tell what are the capabilities of the platform in terms of functionality.

Other considerations:

Since the SW and the platform code might be developed by completely different companies, we would like to decouple as much as possible between the two.
Since in some systems the acceptance tests of customers are quite long we would like to enable the replacement of SW appplication or platform code independently and allowing the customer to test only the affected code (e.g. platform code).
Since some embedded systems have major requirements for high availability, the solution should consider an option to replace smaller modules fast enough without affecting the entire system.
Such decoupling allows the SW to focus on the logic it should provide rather the exact details of the HW platform.

Solution:

The proposed solution is based on Plug-ins Architecture (see overview in Wikipedia http://en.wikipedia.org/wiki/Plug-in_(computing)). The binding between the host application and the Plug-ins is done on runtime allowing maximal decoupling between the host and the platform. The Plug-in (platform code) needs to know only the interface of the host (e.g. the SW application) implement it (or part of it) and the host is in-charge of dynamically locating the plug-ins and detecting what functionality it supports.

Structure:

Run-time behavior:
Initialization phase: during initialization the host maps the supported plug-ins (and potentially queries the supported functionality of each plug-in - depending on technologies).

Runtime phase: when the host needs to invoke an operation, it seeks in its internal map and invokes the Plug-in for those operations if a suitable entry is found.

Implementation:
Plugin: impelements as a dll(s) (windows) or as shared-objects (Linux) its required functionality. Let's assume one of the files is called ManageCreation.dll and it exports a function called CreateObjA(...).

Host:During initialization the host scans a well known directory (or a directory that is provided to him by configuration file for instance), and queries each DLL for its exported functions (e.g. using GetProcAddress). The host finds during the scan the ManageCreation.dll implements the interface "CreateObjA".

The host creates a map between its interfaces to the actual function that implements the interface in the plug-in.

When the host needs to invoke the creation of ObjA, it searches its internal map and invokes the corresponding function that it found in the plugin.

Implications:

Performance: the solution is obviously slower than static binding, however the overhead of the dynamic binding is negligible.
High Availability: a system can replace dynamic libraries easily without affecting the entire system. It requires the host and plugins to agree on a scheme that changes the mapping to from an old implementation of the dll to the new one.

References:
"Patterns of Enterprise Application Architecture", Martin Fowler, 2002. Addision Wesley publication. p. 499.