Microkernel architecture pretty much requires that processes communicate via explicitly defined interfaces, usually message passing. And it becomes much more difficult for subsystems to coordinate low-level activities, which is crucial for performance optimizations. And yes, context switches matter a lot too.
1. A sel4 context switch does significantly less than a Linux ditto. It saves fewer hardware registers etc
2. You will need many more context switches to achieve the same thing in a microkernel due to its distributed architecture.
3. A lot of "complexity" in the kernel is simply performance optimization that does not exist in sel4. Back in the days Minix and Hurd promised to achieve optimization by replacing generic servers with custom ones but I haven't seen this actually working in 20 years
