Newer kernels ship with separate BTF for kernel modules (kmod):

$ ls /sys/kernel/btf/ | head -n5
aes_ce_blk
aes_ce_cipher
aes_neon_blk
aes_neon_bs
async_memcpy

The library is currently oblivious to their existence. This means that kmod BTF is not taken into account for CO-RE relocations and it's not possible to attach fentry / fexit (others?) programs to kernel modules. Both should be supported without the user needing to take additional steps in the happy case.

• LoadSplitSpecFromReader
• CO-RE / Merge BTF
• Load kmod BTF
• fentry / fexit

Proposed API

// LoadSplitSpecFromReader loads split BTF from a reader.
//
// Types from base are used to resolve references in the split BTF.
// The returned Spec only contains types from the split BTF, not from the base.
func LoadSplitSpecFromReader(r io.ReaderAt, base *Spec) (*Spec, error)

// Merge the types from other into the current Spec.
//
// Merged types don't have a canonical type ID, and therefore TypeID() will
// return an error for them.
func (*Spec) Merge(other *Spec) error

// Spec retrieves the types associated with the Handle from the kernel.
//
// base is used to decode split BTF and may be nil.
func (h *Handle) Spec(base *Spec) (*Spec, error) // BREAKING CHANGE

CO-RE

CO-RE is currently done entirely in the library, by matching types used by the BPF program to types found in the kernel.

To extend CO-RE to kmods, we should merge the set of vmlinux and all kmod types and search this superset. Even though a kprobe is attached to a particular kmod's function, it
might still want to pull types from other modules, for example to cast void* to
appropriate types.
I believe this is what libbpf does.

Creating the superset of types will be done by Spec.Merge(). Merged types won't
have a canonical type ID, so Spec.TypeID() will return a well known error.

Problem: bpf_core_type_id_kernel is supposed to return the ID of a type in the kernel.
According to the documentation we could probably return 0 and be "compatible":

/*
 * Convenience macro to get BTF type ID of a target kernel's type that matches
 * specified local type.
 * Returns:
 *    - valid 32-bit unsigned type ID in kernel BTF;
 *    - 0, if no matching type was found in a target kernel BTF.
 */
#define bpf_core_type_id_kernel(type)					    \
	__builtin_btf_type_id(*(typeof(type) *)0, BPF_TYPE_ID_TARGET)

The most interesting use case of the target type ID is probably bpf_snprintf_btf.

Builtin vs. external BTF

Newer upstream kernels ship BTF for vmlinux and kmods in /sys/kernel/btf.
On older kernels and when distros disable BTF the user needs to provide this data out of band.
To use external BTF, users load a *btf.Spec and pass it via ProgramOptions.KernelTypes.

To support kmods for external BTF, LoadSplitSpecFromReader allows reading split BTF from
disk, which is followed by repeated calls to Spec.Merge.

To support kmods for builtin BTF, changes to NewHandleFromID and Handle.Spec are necessary.
To avoid parsing vmlinux on each call to NewHandleFromID we move inflating into
Handle.Spec.
Additionally Handle.Spec will allow specifying a base Spec.

fentry, fexit, etc.

fentry, etc. programs require BTF information in the syscall boundary:

	struct { /* anonymous struct used by BPF_PROG_LOAD command */
...
		__u32		attach_btf_id;	/* in-kernel BTF type id to attach to */
		union {
...
			/* or valid module BTF object fd or 0 to attach to vmlinux */
			__u32		attach_btf_obj_fd;
		};
...
	};

To support kmods, we need to set attach_btf_obj_fd and attach_btf_id. The
obj_fd has to come from a call to NewHandleFromID, it can't come from
a file on disk. The library currently has a code path that allows using external
BTF via ProgramOptions.KernelTypes to find attach_btf_id for vmlinux targets,
but that is most likely broken, since a kernel without builtin BTF will refuse attaching fentry, etc.

Going forward, ProgramOptions.KernelTypes is ignored for the purpose of finding an attach target.
Instead we will first look for the attach target in /sys/kernel/btf/vmlinux.
If successful, we use attach_obj_fd == 0.

Otherwise we will iterate BTF IDs using BPF_BTF_GET_NEXT_ID to find kmod BTF.
Via NewHandleFromID and Handle.Spec we then search the kmod.
If the target is found we use the handle and retrieve the (aliased) type ID via
Spec.TypeID.

Problem: this requires re-inflating kmods for each program, which
is very expensive. Need to cache this via handleCache somehow?