ci: uninstall matplotlib #1806

Job	Run time
Build wheels on Linux (aarch64/manylinux)	10m 38s
Build wheels on Linux (x86_64)	19m 49s
Build wheels on Linux (aarch64/musllinux)	6m 35s
Build sdist and test extra dependencies	2m 15s
Build wheels on macOS (x86_64)	15m 51s
Build wheels on Windows (x64)	28m 36s
Build wheels on Windows (ARM64)	15m 40s
Build wheels on macOS (arm64)	8m 15s
Build wheels on Windows (Win32)	9m 17s
Build with sanitizers for debugging purposes	3m 47s
Build wheels for WebAssembly	6m 46s
	2h 7m 29s

Provide feedback