Supervisors By Hand

I just got home from Goatmire, and I saw an interesting talk by Frank Hunleth: “Nerves by hand”. The talk was about demystifying the Nerves toolchain and how you get an Elixir application compiled with a Linux kernel and run that on bare metal. I kind of knew how most of it worked at a very high level, but Frank made it clear that there is no black magic happening. This way of learning things is something that my brain is wired for. So, with that in mind, I wanted to give a shot at doing OTP supervisors by hand. They might seem like black magic to some people, but they are absolutely not. In this article I intend to show this.

The Four Primitives

There are four primitives you need to know to build your own supervisor: send/receive, spawn, monitor, and link. I will briefly explain them here, and then with just those four things we will build our own supervisor.

Spawn

On the BEAM, a unit of concurrency is a “process.” They are independent computations that run from start to finish, and have their own memory. Nobody can fiddle in it besides the process itself. A process can be created using the spawn function.

spawn(fn -> IO.puts "I am #{inspect(self())}" end)

The code above starts a process, makes it print “I am pid”, and then the process terminates. That is it. You can put as complex of a function in there as you like. The “pid” of the process is its unique address. There will never be another pid like it in the whole runtime of the BEAM. So no other process can ever have the same pid.

Note: There can be, when the counter wraps around, but there will never be two processes alive with the same pid.

Send/Receive

Since each process has its own memory, and only it has access to it, you need a way to let processes synchronize (i.e., communicate with each other). The primitive for this is send. Sending a message puts an arbitrary value in the process’ mailbox. Think of your actual mailbox outside. People put stuff in, and you take it out when you want to.

Taking out a message is done with the receive primitive. The receive expression will look at the processes’ mailbox and see if there is a message. If there is, the clause in the expression is executed, and the message is dropped from the mailbox.

In the code below a process is spawned and it will wait for 1 message. If the process receives the message it will print it, and terminate. So if you were to send 2 letters, only one will be read. The other one will disappear.

my_mailbox =
  spawn(fn ->
    receive do
      {:letter, subject, content, from} ->
        IO.puts("I got a letter from #{from}, saying #{content}")
    end
  end)

# Send a letter to my_mailbox
send(my_mailbox, {:letter, "Invoice", "Gimme 100 bucks", "Scammer"})

Monitor

If you are sending letters, you might want to keep an eye on the mailbox you send it to. If that mailbox is gone for some reason, you should stop sending letters. This is where the monitor primitive comes in. You can monitor any pid and get notified as soon as it terminates. The notification, of course, is a message sent to the process that called monitor. The result of calling monitor on a process is a reference. These are random unique values, much like process ids. They can be generated cheaply, and they are unique in your virtual machine. They are useful for figuring out which monitor sent a message about a process dying, as we’ll see later.

my_mailbox =
  spawn(fn ->
    receive do
      {:letter, subject, content, from} ->
        IO.puts("I got a letter from #{from}, saying #{content}")
    end
  end)

# Get notified about a termination.
ref = Process.monitor(my_mailbox)

# Send a letter to my_mailbox
send(my_mailbox, {:letter, "Invoice", "Gimme 100 bucks", "Scammer"})

# Wait for the termination message.
receive do
  {:DOWN, _ref, :process, pid, reason} ->
    IO.puts("My mailbox died, with reason #{inspect(reason)}")
end

Link

A more powerful variant of monitoring is linking. Let’s say a process is sending letters to a specific mailbox, but as soon as it’s gone, the sender has no purpose anymore. We should therefore tell the sender to terminate when this happens. The mechanism to do this is called linking, done with the link primitive. When two processes are linked, and one of them dies, the linked process will die as well. This is a transitive mechanism, meaning that if you link A to B, B to C, and C to D, all these processes will terminate as soon as one of them dies.

Note: Processes exit with a reason. If this reason is :normal, the linked processes do not terminate. This makes sense, since you want to have linking in place for failure handling, not orchestration.

iex(1)> pid = spawn(fn -> Process.sleep(3000); exit(:oopsie)  end)
#PID<0.120.0>
iex(2)> Process.link(pid)
true
** (EXIT from #PID<0.119.0>)
  shell process exited with reason: :oopsie

Interactive Elixir (1.20.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(3)>

In the terminal snippet above a process is created that will exit after 3 seconds with the reason :oopsie. The linked process is the process of the repl, and that will terminate as soon as pid terminates.

Building a Supervisor

We can create processes, we can communicate with them, and we can keep an eye on them to see if they’re still alive. Those are the only ingredients you need for a supervisor. Because I’m writing this in the Varberg train station, we’ll use the name Goatherder for supervision, and all our processes in the tree are going to be Goats.

Goat’s Behavior

In a GenServer you have a lot of callbacks: handle_info, handle_call, handle_cast and so on. We’ll keep it simple and only have one: message. The function should accept the message and the current state of the process, and then return {:ok, state}. A simple Goat that refuses to do anything could look like this.

defmodule Goatherder.HelloGoat do
  def message(message, state) do
    IO.puts("Nobody tells the goat to '#{inspect(message)}'")
    {:ok, state}
  end
end

This is just the behavior of our goat; it defines what our HelloGoat does when it receives a message. It does not actually create a process for it. For that, we have the generic Goat process. For each goat we create in our herd, we need a process that will represent it. The code will be explained below, but given our HelloGoat module we can create a goat process as follows.

iex(1)> {:ok, pid} = Goatherder.Goat.create_goat(Goatherder.HelloGoat)
{:ok, #PID<0.164.0>}
iex(2)> Process.whereis(Goatherder.HelloGoat)
#PID<0.164.0>
iex(3)> send(pid, "faint!")
Nobody tells the goat to '"faint!"
"faint!"

We now have a process that is representing the behavior of the HelloGoat. Let’s look into how the Goat module actually works.

Goat

In the Goat module, we have 2 functions. A create_goat function, and a goat_loop function. When we call create_goat with a module, a process is spawned that will register itself under the name of the module. Then, the goat_loop function is called to wait for the first incoming message for our goat.

The goat_loop function keeps track of the goat’s state. Each goat’s initial state is nil. In a real GenServer, this can be changed in the init callback.

defmodule Goatherder.Goat do
  require Logger

  @doc """
  Given a goat module, creates a process that executes that goat's logic. The
  name of the goat process is always the name of the module (i.e., we can only
  have one of each goat running).
  """
  def create_goat(goat) do
    goat_pid =
      spawn(fn ->
        Process.register(self(), goat)
        goat_loop(goat, nil)
      end)

    {:ok, goat_pid}
  end

  # ...
end

The goat_loop function makes use of the receive primitive. For each message that arrives in the goat’s mailbox, the module’s message/2 function is called. The message and the state are passed in, and we expect the module to either return {:ok, state} or {:error, reason}. Anything else will cause the goat process to crash. If the message was well received, the loop waits for the next incoming message.

  # listen for messages, and relay them to the goat implementation.
  # Terminate on all values except `{:ok, term()}`.
  defp goat_loop(goat, state) do
    receive do
      message ->
        case goat.message(message, state) do
          {:ok, state} ->
            goat_loop(goat, state)

          {:error, error} ->
            Logger.error("Goat error: #{inspect(error)}")
        end
    end
  end

Goatherder

We have discussed the mechanics to create a goat process, but we still have no supervisor for them. The main task of a supervisor is to monitor the goats, and recreate them as soon as they fall over.

Starting from the top again, we would like to write code as follows. Notice that this is not too different from the real Supervisor implementation. We define a list of modules that contain the behavior we want. We then use Goatherder to create our herd. The Goatherder is a process that will do all the work for us.

defmodule Goatherder.Example do
  def init do
    goats = [Goatherder.HelloGoat]
    Goatherder.create(goats)
  end
end

iex(1)> Goatherder.Example.init
13:12:32.292 [info] Creating goat Goatherder.HelloGoat
{:ok, #PID<0.159.0>}

The mechanism behind this boils down to the following. For each goat we passed in, the following happens:

The Goatherder process will then be notified as soon as one of its goats crashes by means of a {:DOWN...} message we saw before. If such a message arrives, the Goatherder process will do the following.

There are some bookkeeping details we need to do, however. Let’s look at the creation of a herd, first. The create function will call spawn_herd on the goats it has been given, and then monitor the entire herd in an infinite loop.

A goat is spawned by creating a Goat process for the module, and subsequently calling monitor on it, making sure the herder is notified if the goat faints.

The return value from spawn_goat carries with it all the information we need about our goats. The process id of the goat we just spawned, the module that it is running, and a reference.

defmodule Goatherder do
  require Logger

  def create(goats) do
    herd_pid = spawn(fn ->
      herd = spawn_herd(goats)
      monitor(herd)
    end)

    {:ok, herd_pid}
  end

  defp spawn_herd(goats) do
    for goat <- goats, into: %{} do
      spawn_goat(goat)
    end
  end

  defp spawn_goat(goat) do
    Logger.info "Creating goat #{inspect goat}"
    {:ok, goat_pid} = Goat.create_goat(goat)
    goat_ref = Process.monitor(goat_pid)
    {goat_ref, {goat_pid, goat}}
  end
end

The final step is now monitoring the goats, and acting on a crashed goat. If a goat crashes, we just create it again.

defp monitor(herd) do
  herd =
    receive do
      {:DOWN, ref, :process, pid, reason} when is_map_key(herd, ref) ->
        Logger.warning("Goat #{inspect(pid)} exited: #{inspect(reason)}")
        {_, goat} = Map.get(herd, ref)
        {goat_ref, {goat_pid, goat}} = spawn_goat(goat)

        herd
        |> Map.delete(ref)
        |> Map.put(goat_ref, {goat_pid, goat})

      _ignored ->
        herd
    end

  monitor(herd)
end

The monitor function keeps calling itself, and waits for it to receive a message about a fainted goat. Remember that monitor returns a reference, so we can use that reference here to determine which goat it was that fainted.

The reference is looked up in the herd state, and we fetch the module that this goat was running. Using that, the herder simply spawns a new copy of that goat and stores it in its state, waiting for the next goat to faint.

Missing Features

This is by no means an equivalent implementation of the real Supervisor, but it gives you an idea of what it takes to build something. If you want to play around with this, a few things that are still missing, but fundamental to supervisors are the following.