eBPF on WSL2 [kernel version 6.x] [Ubuntu] [x64] [Arm64] [2024]

Introduction

A few days ago, I wanted to develop eBPF application on WSL2 but turned out i can't develop eBPF application on ordinary WSL2 kernel, so i started to research how can i accomplish this task. i found lots of tutorials but they didn't quite handy cause they were outdated or not worked so i decided to write blog post to help others getting started with developing and running eBPF programs.

Step 0

Update all packages on your system:
sudo apt update && sudo apt upgrade

Step 1

Clone Microsoft Linux kernel repository from GitHub:
git clone https://github.com/microsoft/WSL2-Linux-Kernel.git --depth=1 -b linux-msft-wsl-6.6.36.3

step 2

open WSL kernel config using gedit so you can easily edit file with GUI:
gedit WSL2-Linux-Kernel/Microsoft/config-wsl

step 3

add these flags (override them with new value if flag exist):
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_IKHEADERS=y
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_DUMMY=m
CONFIG_VXLAN=m

step 4

install packages to build kernel:
sudo apt update && sudo apt install build-essential flex bison libssl-dev libelf-dev bc python3 pahole

step 5

Change directory to the kernel source code:
cd WSL2-Linux-Kernel

step 6

Build the kernel:
make -j$(nproc) KCONFIG_CONFIG=Microsoft/config-wsl

step 7

Install the kernel modules and headers:
sudo make modules_install headers_install

step 8

Copy the kernel image to the Windows file system:
cp arch/x86/boot/bzImage /mnt/c/
I was building kernel on my Surface Pro X and noticed the kernel image name and directory is different based on each architecture so if you have Arm processor use below command to copy the kernel image:
cp arch/arm64/boot/Image /mnt/c/

step 9

Create or edit the file %USERPROFILE%\.wslconfig with the following content:
[wsl2]
kernel=C:\\bzImage
if you have Arm processor:
[wsl2]
kernel=C:\\Image
notice the file extension if you are creating the file with notepad and delete .txt extension.

step 10

Open a terminal window as Administrator and stop the WSL instance:
wsl --shutdown
now verify the kernel version:
uname -r
now you must see 6.6.36.3-microsoft-standard-WSL2.

step 11

now we must install these packages to start developing eBPF apps:
sudo apt install linux-libc-dev clang llvm libelf-dev libpcap-dev libbpf-dev build-essential libc6-dev-i386
if you have Arm processor replace libc6-dev-i386 with libc6-dev.

step 12

install linux generic tools and headers:
sudo apt install linux-headers-generic linux-tools-generic linux-tools-common


step 13

let's write simple XDP program to drop all packets. create new file drop.c:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("xdp_drop")
int xdp_drop_prog(struct xdp_md *ctx)
{
    return XDP_DROP;
}

char _license[] SEC("license") = "GPL";
now compile with below command:
clang -O2 -g -Wall -target bpf -c drop.c -o drop.o
if you encounter asm types not found error enter command below:
ln -s /usr/include/asm-generic /usr/include/asm

step 14

load program into the kernel using ip command:
sudo ip link set dev eth0 xdpgeneric obj drop.o sec xdp_drop
if you got operation not allowed error disable LRO before running XDP program:
ethtool -K eth0 lro off

now enter ping 1.1.1.1 all packets must be dropped.

to unload program, use command below:
sudo ip link set dev eth0 xdpgeneric off

step 15

if you want to install BCC enter command below to get dependencies (for ubuntu jammy 22.04):
sudo apt install -y zip bison build-essential cmake flex git libedit-dev \
  libllvm14 llvm-14-dev libclang-14-dev python3 zlib1g-dev libelf-dev libfl-dev python3-setuptools \
  liblzma-dev libdebuginfod-dev arping netperf iperf
Install and compile BCC:
git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd

step 16

run xdp_drop_count.py example program:
sudo python3 bcc/examples/networking/xdp/xdp_drop_count.py eth0
if pyroute2 is not installed then enter:
sudo apt-get install python3-pip
sudo -H pip install pyroute2






if you have any question don't hesitate to ask!

Comments

  1. Thanks for getting all this info together! One note - there is a problem with a recent Docker Desktop version when the 6.6 kernel is used - https://github.com/docker/for-win/issues/13638. I've tried the suggestions to downgrade Docker Desktop mentioned there, but they didn't work for me. I tried with the 6.1 kernel, and it works, so it might be worth mentioning.

    ReplyDelete
    Replies
    1. I've dived deeper into the problem and it looks worse now.

      So the thing is that according to this post/comment https://github.com/microsoft/WSL/issues/11742#issuecomment-2207883742
      since recent/upcoming WSL release and 6.6 kernel, WSL is supporting modprobe autoloading when module is required by application

      the problem is that:
      - in 6.6 kernel configuration they moved a lot of previously builtin features to modules (as autoloading now support, they suppose)
      - BUT, autoloading _is not_ supported when you do custom kernel build

      Docker Desktop for Windows expecting that number of features are builtin, and they were moved to modules in 6.6 config (see discussion in the link above)

      So doing custom build with the default config will lead to issues with the software (such as Docker Desktop) that relied on 5.* builtin features

      What I'm doing right now is, as per this github issue recommendation, I'm manually comparing 5.15 and 6.6 microsoft kernel configs and update =m to =y in 6.6 config, where it was =y in 5.15 config (probably a better idea would be to create a script for this, but so far I want to compare configs manually)

      Will update soon if it helps.

      Generally, I think that the problem is that now, building customer kernels is unreliable, it would be better if they would keep builtin features with =y, and previously disabled features with =m, so it would be 100% compatible, but they optimized and this optimization won't work for custom builds :(

      Delete
    2. I've merged "=y" flags from 5.15 to 6.6 config, in places where they became "=m"
      gonna try it the next week
      gist with the config is here
      https://github.com/microsoft/WSL/issues/11742

      Delete
    3. I think you should publish blog post to help other people.

      Delete
  2. Do you know how to fix: cp arch/x86/boot/bzImage /mnt/c/
    cp: cannot create regular file '/mnt/c/bzImage': Permission denied

    ReplyDelete
    Replies
    1. Same issue with sudo

      Delete
    2. Enter "\\wsl$" in the File Explorer address bar and then you can access file and copy it

      Delete
    3. cp arch/x86/boot/bzImage /mnt/c/%USERPROFILE%/bzImage
      Now get wsl: Invalid escaped character: 'U' in C:\Users\\.wslconfig:2 after wsl --shutdown and restarting wsl
      Followed this link, still same error ... frustrating
      https://learn.microsoft.com/en-us/answers/questions/1305746/how-to-escape-spaces-of-kernel-path-in-wslconfig-f

      Delete
    4. Working ... needed double forward slash C:\\Users\\account\\bzImage
      uname -r
      6.1.21.2-microsoft-standard-WSL2+
      Thanks Massoud for the blog and responses

      Delete
    5. You are copying kernel image into your user profile so you must edit kernel image path inside .wslconfig file. If your problem exist i can help you remotely if tou want

      Delete
    6. Don't mention it. Glad your problem solved. If you have problem running docker you can move config file to elsewhere to load vanilla kernel and when you are finished just move back config file.

      Delete
  3. when i run python examples: show the follow error: Can u help me solve this problem, thanks
    ❯ python3 examples/networking/xdp/xdp_drop_count.py eht0
    Traceback (most recent call last):
    File "/root/bcc/examples/networking/xdp/xdp_drop_count.py", line 10, in
    from bcc import BPF
    File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 27, in
    from .libbcc import lib, bcc_symbol, bcc_symbol_option, bcc_stacktrace_build_id, _SYM_CB_TYPE
    File "/usr/lib/python3/dist-packages/bcc/libbcc.py", line 17, in
    lib = ct.CDLL("libbcc.so.0", use_errno=True)
    File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
    OSError: /lib/x86_64-linux-gnu/libbcc.so.0: undefined symbol: _ZSt28__throw_bad_array_new_lengthv

    ReplyDelete
    Replies
    1. Did you install all dependencies correctly?

      May be gcc version difference in python and compiled code.

      Check link below
      https://github.com/iovisor/bcc/issues/4610#issuecomment-1780698518

      Delete
    2. I use gcc 10.5.0; may i try gcc 11.x.0 again; thx

      Delete
    3. Yes install gcc 11 and test again. This issue is about version difference in dependencies.
      Hope this helps.

      Delete
    4. worked, so thanks!

      Delete
    5. Don't mention it. Glad your problem solved.

      Delete

Post a Comment