..  Copyright Authors of Cilium

===================================
Tail Call Hooks for Custom Programs
===================================

The files in this directory implement a sample byte-counter program which can
be attached to dedicated hooks in the datapath.

Note: This is a beta feature. Please provide feedback and file a GitHub issue
if you experience any problems.

Datapath Hooks
==============

Implementation
--------------

When compiled with ``ENABLE_CUSTOM_CALLS`` set, Cilium's datapath contains some
tail call hooks to insert custom programs in the datapath. These custom
programs should be particularly careful *not* to interfere with the logics of
the datapath, at the risk of altering or breaking the network connectivity in
the cluster.

Before the tail call happens, the datapath encodes two elements into the ``cb``
field of the socket buffer for the packet:

- The return value for that packet, so that the custom program can return it on
  exit. This is because a tail call is not like a function call, and the custom
  program does not *return* into the core datapath logics. Therefore, the
  custom program must be called as the very last step of the processing of the
  packet, and must preserve the return value selected by the datapath.

- The remote security identity associated with the packet, that is, the Cilium
  identity associated with the sender for the ingress hook, and with the
  receiver when on the egress hook. This identity is not strictly necessary to
  the custom program, but it proved useful for some use cases. Given that the
  datapath already did the work for retrieving the identity for the packet, it
  just passes it down to the custom programs, in case it saves them the trouble
  to extract it a second time.

Limitations
-----------

- There is currently no hook on the egress path for socket-level
  load-balancing.

Sample Byte-Counter Program
============================

This directory contains a sample byte-counter program, to help count how many
bytes a given endpoint sends or receives, and to understand how the tail call
hooks work.

The following steps indicate how to use this example program.

1. Compile the program. This should be easy thanks to the provided Makefile.
   From the root of the Cilium repository::

       $ make -C bpf/custom

2. Load and pin the byte-counter program. Bpftool can do it::

       # bpftool prog load bpf/custom/bytecount.o \
           /sys/fs/bpf/tc/globals/bytecounter \
           type classifier \
           pinmaps /sys/fs/bpf/tc/globals/bytecounter_maps

3. Pick the endpoint for which the program should count the bytes. Update the
   tail call map for custom programs for that endpoint, with a reference to the
   pinned byte-counter program::

       $ EP=<endpoint_id>
       # bpftool map update \
           pinned /sys/fs/bpf/tc/globals/cilium_calls_custom_$(printf '%05d' ${EP}) \
           key 0 0 0 0 \
           value pinned /sys/fs/bpf/tc/globals/bytecounter

   The key indicate the hook to use. It is possible to attach the program at
   multiple hooks, although the counter currently makes no distinction and will
   condensate the values collected from all hooks into a single counter. The
   available hooks and their keys are the following:

   - IPv4, ingress: ``key 0 0 0 0``
   - IPv4, egress:  ``key 1 0 0 0``
   - IPv6, ingress: ``key 2 0 0 0``
   - IPv6, egress:  ``key 3 0 0 0``

4. After some traffic has flown, dump the content of the byte-counter map to
   read the statistics::

       # bpftool map dump \
           pinned /sys/fs/bpf/tc/globals/bytecounter_maps/bytecount_map

5. When the byte-counter is no longer necessary, it is possible to clean it up
   by deleting the entry in the tail call map, and removing the pinned
   objects::

       # bpftool map delete \
           pinned /sys/fs/bpf/tc/globals/cilium_calls_custom_$(printf '%05d' ${EP}) \
           key 0 0 0 0
       # rm /sys/fs/bpf/tc/globals/custom_prog
       # rm -r /sys/fs/bpf/tc/globals/bytecounter_maps

Using Other Custom Programs
===========================

Other custom programs can be hooked into the datapath, just like the byte
counter program. The workflow is similar.

First, write a custom program, and make sure it does not interfere with the
logics of the datapath. It is recommended to define a custom function and to
include it in the “landing point” program from bpf_custom.c.

Then, compile your program. If reusing bpf_custom.c, this is easy to do. The
Makefile should use bpf_custom.c as a basis for compiling each .h header file
found in the directory::

    $ make -C bpf/custom

If you want to build a single program, just pass the name of the file
containing the ``custom_prog`` function when calling make::

    $ BPF_CUSTOM_PROG_FILE=custom_function.h make -C bpf/custom

The environment variable ``BPF_CUSTOM_PROG_NAME`` is similarly available to set
the name of the ELF section where the resulting program will be located. This
is often used to set either the program name, or, depending on the loading
method, to indicate the type of the program to the loader. It will default to
the base name (without the extension) of the .h file used to compile the
program.