eBPF on WSL2 [kernel version 6.x] [Ubuntu] [x64] [Arm64] [2024]
Introduction
Step 0
Update all packages on your system:
sudo apt update && sudo apt upgrade
Step 1
Clone Microsoft Linux kernel repository from GitHub:
git clone https://github.com/microsoft/WSL2-Linux-Kernel.git --depth=1 -b linux-msft-wsl-6.6.36.3
step 2
open WSL kernel config using gedit so you can easily edit file with GUI:
gedit WSL2-Linux-Kernel/Microsoft/config-wsl
step 3
add these flags (override them with new value if flag exist):
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_IKHEADERS=y
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_DUMMY=m
CONFIG_VXLAN=m
step 4
install packages to build kernel:
sudo apt update && sudo apt install build-essential flex bison libssl-dev libelf-dev bc python3 pahole
step 5
Change directory to the kernel source code:
cd WSL2-Linux-Kernel
step 6
Build the kernel:
make -j$(nproc) KCONFIG_CONFIG=Microsoft/config-wsl
step 7
Install the kernel modules and headers:
sudo make modules_install headers_install
step 8
Copy the kernel image to the Windows file system:
cp arch/x86/boot/bzImage /mnt/c/
I was building kernel on my Surface Pro X and noticed the kernel image
name and directory is different based on each architecture so if you have
Arm processor use below command to copy the kernel image:
cp arch/arm64/boot/Image /mnt/c/
step 9
Create or edit the file
%USERPROFILE%\.wslconfig with
the following content:
[wsl2] kernel=C:\\bzImage
if you have Arm processor:
[wsl2] kernel=C:\\Image
notice the file extension if you are creating the file with notepad and delete
.txt extension.
step 10
Open a terminal window as Administrator and stop the WSL instance:
wsl --shutdown
now verify the kernel version:
uname -r
now you must see 6.6.36.3-microsoft-standard-WSL2.
step 11
now we must install these packages to start developing eBPF apps:
sudo apt install linux-libc-dev clang llvm libelf-dev libpcap-dev libbpf-dev build-essential libc6-dev-i386
if you have Arm processor replace
libc6-dev-i386 with libc6-dev.
step 12
install linux generic tools and headers:
sudo apt install linux-headers-generic linux-tools-generic linux-tools-common
step 13
let's write simple XDP program to drop all packets. create new file drop.c:
#include <linux/bpf.h> #include <bpf/bpf_helpers.h> SEC("xdp_drop") int xdp_drop_prog(struct xdp_md *ctx) { return XDP_DROP; } char _license[] SEC("license") = "GPL";
now compile with below command:
clang -O2 -g -Wall -target bpf -c drop.c -o drop.o
if you encounter
asm types not found
error enter command below:
ln -s /usr/include/asm-generic /usr/include/asm
step 14
load program into the kernel using
ip command:
sudo ip link set dev eth0 xdpgeneric obj drop.o sec xdp_drop
if you got operation not allowed error disable LRO before running XDP program:
ethtool -K eth0 lro off
now enter ping 1.1.1.1 all packets
must be dropped.
to unload program, use command below:
sudo ip link set dev eth0 xdpgeneric off
step 15
if you want to install BCC enter command below to get dependencies (for ubuntu
jammy 22.04):
sudo apt install -y zip bison build-essential cmake flex git libedit-dev \ libllvm14 llvm-14-dev libclang-14-dev python3 zlib1g-dev libelf-dev libfl-dev python3-setuptools \ liblzma-dev libdebuginfod-dev arping netperf iperf
Install and compile BCC:
git clone https://github.com/iovisor/bcc.git mkdir bcc/build; cd bcc/build cmake .. make sudo make install cmake -DPYTHON_CMD=python3 .. # build python3 binding pushd src/python/ make sudo make install popd
step 16
run xdp_drop_count.py example program:
sudo python3 bcc/examples/networking/xdp/xdp_drop_count.py eth0
if pyroute2 is not installed then
enter:
sudo apt-get install python3-pip sudo -H pip install pyroute2
if you have any question don't hesitate to ask!
Thanks for getting all this info together! One note - there is a problem with a recent Docker Desktop version when the 6.6 kernel is used - https://github.com/docker/for-win/issues/13638. I've tried the suggestions to downgrade Docker Desktop mentioned there, but they didn't work for me. I tried with the 6.1 kernel, and it works, so it might be worth mentioning.
ReplyDeleteThanks for the heads-up
DeleteI've dived deeper into the problem and it looks worse now.
DeleteSo the thing is that according to this post/comment https://github.com/microsoft/WSL/issues/11742#issuecomment-2207883742
since recent/upcoming WSL release and 6.6 kernel, WSL is supporting modprobe autoloading when module is required by application
the problem is that:
- in 6.6 kernel configuration they moved a lot of previously builtin features to modules (as autoloading now support, they suppose)
- BUT, autoloading _is not_ supported when you do custom kernel build
Docker Desktop for Windows expecting that number of features are builtin, and they were moved to modules in 6.6 config (see discussion in the link above)
So doing custom build with the default config will lead to issues with the software (such as Docker Desktop) that relied on 5.* builtin features
What I'm doing right now is, as per this github issue recommendation, I'm manually comparing 5.15 and 6.6 microsoft kernel configs and update =m to =y in 6.6 config, where it was =y in 5.15 config (probably a better idea would be to create a script for this, but so far I want to compare configs manually)
Will update soon if it helps.
Generally, I think that the problem is that now, building customer kernels is unreliable, it would be better if they would keep builtin features with =y, and previously disabled features with =m, so it would be 100% compatible, but they optimized and this optimization won't work for custom builds :(
I've merged "=y" flags from 5.15 to 6.6 config, in places where they became "=m"
Deletegonna try it the next week
gist with the config is here
https://github.com/microsoft/WSL/issues/11742
I think you should publish blog post to help other people.
DeleteDo you know how to fix: cp arch/x86/boot/bzImage /mnt/c/
ReplyDeletecp: cannot create regular file '/mnt/c/bzImage': Permission denied
Did you try 'sudo cp' ?
DeleteSame issue with sudo
DeleteEnter "\\wsl$" in the File Explorer address bar and then you can access file and copy it
Deletecp arch/x86/boot/bzImage /mnt/c/%USERPROFILE%/bzImage
DeleteNow get wsl: Invalid escaped character: 'U' in C:\Users\\.wslconfig:2 after wsl --shutdown and restarting wsl
Followed this link, still same error ... frustrating
https://learn.microsoft.com/en-us/answers/questions/1305746/how-to-escape-spaces-of-kernel-path-in-wslconfig-f
Working ... needed double forward slash C:\\Users\\account\\bzImage
Deleteuname -r
6.1.21.2-microsoft-standard-WSL2+
Thanks Massoud for the blog and responses
You are copying kernel image into your user profile so you must edit kernel image path inside .wslconfig file. If your problem exist i can help you remotely if tou want
DeleteDon't mention it. Glad your problem solved. If you have problem running docker you can move config file to elsewhere to load vanilla kernel and when you are finished just move back config file.
Deletewhen i run python examples: show the follow error: Can u help me solve this problem, thanks
ReplyDelete❯ python3 examples/networking/xdp/xdp_drop_count.py eht0
Traceback (most recent call last):
File "/root/bcc/examples/networking/xdp/xdp_drop_count.py", line 10, in
from bcc import BPF
File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 27, in
from .libbcc import lib, bcc_symbol, bcc_symbol_option, bcc_stacktrace_build_id, _SYM_CB_TYPE
File "/usr/lib/python3/dist-packages/bcc/libbcc.py", line 17, in
lib = ct.CDLL("libbcc.so.0", use_errno=True)
File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /lib/x86_64-linux-gnu/libbcc.so.0: undefined symbol: _ZSt28__throw_bad_array_new_lengthv
Did you install all dependencies correctly?
DeleteMay be gcc version difference in python and compiled code.
Check link below
https://github.com/iovisor/bcc/issues/4610#issuecomment-1780698518
I use gcc 10.5.0; may i try gcc 11.x.0 again; thx
DeleteYes install gcc 11 and test again. This issue is about version difference in dependencies.
DeleteHope this helps.
worked, so thanks!
DeleteDon't mention it. Glad your problem solved.
Delete