[ci] Fix linux wheel builds

See commit messages.

Also bumps minimum gcc to 11.4 (which is the oldest currently tested on ubuntu1 runner). Does not compile anymore (!4992 (merged)) with 10.2.

1. Feature description was added to the changelog
2. Tests covering your feature were added?