Stop using oldest-supported-numpy #113

Job	Run time
Tests with external liberfa	47s
[ ubuntu-latest ]	56s
[ macos-latest ]	1m 49s
[ windows-latest ]	1m 22s
Test with oldest supported versions of our dependencies [ ubuntu-20.04 ]	2m 4s
Test with medium old supported versions of our dependencies [ ubuntu-22.04 ]	2m 45s
Test with development versions of our dependencies [ ubuntu-latest ]	1m 5s
Code style checks [ ubuntu-latest ]	10s
Documentation build [ ubuntu-latest ]	1m 22s
build_and_publish / Build source distribution	1m 22s
build_and_publish / Load build targets	7s
build_and_publish / Build cp*-manylinux_i686 wheels	18m 54s
build_and_publish / Build cp*-manylinux_x86_64 wheels	3m 35s
build_and_publish / Build cp*-musllinux_x86_64 wheels	2m 49s
build_and_publish / Build cp*-manylinux_aarch64 wheels	19m 5s
build_and_publish / Build cp*-macosx_x86_64 wheels	7m 28s
build_and_publish / Build cp*-macosx_arm64 wheels	8m 20s
build_and_publish / Build cp-win wheels	13m 34s
build_and_publish / Upload	1s
	1h 27m 35s

Provide feedback