Excelero

Improving kernel uptime via hot upgrade of modules (part I)

By May 28, 2019 No Comments

What is a hot upgrade?

Upgrades are generally a big pain for developers. Yes, it is satisfying to keep improving the code – deep down we all dig that. But when backwards compatibility is broken by the introduction of a cool new feature or optimization, R&D have to dedicate costly time and efforts for generating this compatibility. And while working on upgrade compatibility, engineers cannot work on the next cool new feature.

Hot upgrades can be particularly tricky, but really pay off for the users when it’s done right. By “hot” I mean that the product keeps delivering the same service (responding to the designed API) while it is being upgraded. Ideally, users will not even notice that upgrade took place.

“Hot upgrade” is sometimes also referred to as “Live update”.

In this short blog post series, I want to discuss a couple of approaches to hot upgrades, take a look at the specifics and the challenges, and eventually elaborate on how we apply this for NVMesh.

It’s important to note that not every software update can be done via live updates. When that is the case, doing a warm/cold upgrade is common practice. For example, if a product is not forward-compatible, it is almost impossible to hot-upgrade it, even if the newer software version is backward-compatible.

The substitution method

Most hot upgrades are done with the substitution method: the services/APIs of an upgraded component are temporarily performed by a different instance of the upgraded program. This might be another thread, a different server, a parallel route or any other solution. Substitution can be done using the proxy design pattern (link). A proxy always exists and directs calls to specific instances. During the upgrade, the proxy then redirects the calls. Upgrading the proxy itself is much more difficult.

This approach can perfectly be used for hot upgrading kernel modules. At first sight, it would seem that a hot upgrade is impossible due to basic contradiction between 3 constraints:

  1. User space applications can still use system calls and services given by the module.
  2. The module can be removed from the operating system only when it is not in use.
  3. Typically, a new driver cannot be introduced before the old one is removed.

The latter challenge could be compared to hot-upgrading the driver for your laptop’s camera while having a video chat.  Sounds absurd, doesn’t it? When the old driver is removed, how will the computer decipher the camera pixels? Isn’t it so that operating system fixes always require a restart of the computer? Well, not really. It is indeed hard to hot upgrade via the substitution method because maintaining a constant proxy would always slow down the system calls. And without proxy, no one can reroute the calls to bypass the module. But there is another way: the 2-step method – instead of a proxy we use inheritance.

The parent class is very small and not expected to be changed often. The inherited child is hot upgraded rather frequently. In the upgrade process a type of polymorphism is used where the child is removed, but parent keeps operating until a new version of a child is introduced. This method relies on the Liskov substitution principle (more info here).

The 2-step upgrade

Let’s look into the 2-step approach to hot upgrades with an analogy: Imagine you want to upgrade lightbulbs in a traffic light on a busy street. The naive approach would be to bring in an electrician with a long ladder, a thick mustache and brand new lightbulbs. This approach would quite probably clog the road for a while and drivers will not appreciate this as a “hot upgrade”. A smarter approach would probably be to call for a local policeman to signal and direct the cars during the upgrade. This is called the 2-step method, which guarantees a hot upgrade as long as the traffic light and the policeman are not upgraded simultaneously.

The same method can be used for kernel modules. However, a hot upgrade cannot replace the entire module at once: the module needs to be split into 2 or more components each of which can be hot upgraded individually.

In the second part of this blog series, I will discuss how we apply all this wisdom to our own software.


Side note about Kpatch

The kpatch feature of the Linux kernel generally follows the same goal to replace code transparently at runtime. However, kpatch can only be used to substitute function pointers and not to update internal data structures. This makes it suitable to patch security vulnerabilities, but unfortunately insufficient for larger upgrades (more info on kpatch here).


Side note about Cold upgrade

Another option is to perform a cold upgrade of a single component in a fault tolerant system. For example: shutting down for upgrade, 1 server from a quorum of 3, with majority voting. Another example would be a Raid-1 or Raid-6 backup system which can operate in degraded mode. But the disadvantage is that this impacts the fault tolerance of the system, which is why the hot upgrade is much cooler (pun intended 😉 ).

Daniel Shmulyan

Author Daniel Shmulyan

Data Service Team Leader

More posts by Daniel Shmulyan