#152 Fix crashes in tests on x86 systems without sufficient ISA support

[apache-hb]

Resolves #152. I opted to use the compiler provided extension support rather than modify codec_choose_x86 just to be sure I didn't introduce any bugs into the autodetection logic. I've tested these changes on zen4 Genoa windows & linux, as well as qemu without kvm enabled. Changes were tested on clang, gcc, and msvc.

[aklomp]

This might not be very portable and it seems to be breaking CI. Perhaps it can be wrapped in an #ifdef __has_builtin?

[aklomp]

While I agree that it's bad form for the tests to crash without warning, I'm not totally sure about this solution yet. Two reasons:

• There is already some "mature" runtime feature detection code in lib/codec_choose.c, and this PR feels like it's treading the same ground. I'd prefer to unify those things if possible. (I'm not happy with the runtime detection code there by the way - it's a wart that grew out of necessity and not a problem I want to have.)
• Catching SIGILL seems like possibly a reasonable way to deal with the issue, since these are just tests.
  • Catching signals portably across platforms is another can of worms.

[apache-hb]

Sorry I couldn't respond sooner, I've been very busy at my day job. Your comments make sense, I was initialy hesitant to use the internals of codec_choose_x86 because I didn't want to accidentally export a symbol from it that wasnt previously visible. If I can figure out how to ensure no symbols leak I'll push a new revision using that function to detect feature support rather than using compiler builtins.

[mcourteaux]

@aklomp Is this PR ready to land? Can you approve CI?

[aklomp]

@aklomp Is this PR ready to land? Can you approve CI?

CI is approved, sorry for the delay. Unfortunately it's failing.

I did a quick first review on the revised code and I like the approach. It feels clean to separate the feature detection code from the codec choosing code.

There are some things that I'm not fond of, like removing some of the #if HAVE_xxx guards. Those are functional. They prevent code from being compiled for platforms where the code is not needed, and they also prevent auto-selecting a codec that is not present in the build.

What I think can happen now is the following. Let's say that the library is compiled at SSE4 support level, but run on an AVX2 machine. The new codec chooser picks the AVX2 codec, but that one won't work because it's a no-op in that build. The right behavior would be to fall back to SSE4. The if HAVE_xxx logic was there to limit the handled support level to just the codecs that are built in.

Ironically, I think this is similar to the #152 issue that started the discussion?

[mcourteaux]

Thanks for taking a look! I think there are two options indeed:

1. Always compile in all the versions, and let the runtime detect which version is appropriate. Potentially a less lean binary, but simple.
2. Let the user (or CMake) decide which versions to compile, and have the runtime not select what's not available. This is what you're saying, and offers maximal control to the user

I'd also be in favor of option 2. However, I'd like to argue that HAVE_AVX2 is too generic of a name, and can cause bugs in larger projects who use this project as a subproject in CMake. A generic HAVE_AVX2 leaking into the compilation flags of this project can happen easily. To that end, I propose getting rid of those aliases defined here:
https://github.com/aklomp/base64/blob/9e8ed65048ff0f703fad3deb03bf66ac7f78a4d7/cmake/config.h.in#L4C1-L27C1
and adjusting the necessary code in the project.

I could do that and make my own PR to replace this one? Or you could do it? Any preferences?

[BurningEnlightenment]

A generic HAVE_AVX2 leaking into the compilation flags of this project can happen easily.

Can you describe a realistic scenario for this? I can only think of global unity builds (where you very likely also get in trouble due to non-namespaced static globals like codec) or someone doing bad practice stuff like abusing CMAKE_C_FLAGS by explicitly adding -DHAVE_AVX=1.

[mcourteaux]

A generic HAVE_AVX2 leaking into the compilation flags of this project can happen easily.

Can you describe a realistic scenario for this? I can only think of global unity builds (where you very likely also get in trouble due to non-namespaced static globals like codec) or someone doing bad practice stuff like abusing CMAKE_C_FLAGS by explicitly adding -DHAVE_AVX=1.

I think a simple and commonly overused add_definition(HAVE_AVX) will already do the trick.

[BurningEnlightenment]

I think a simple and commonly overused add_definition(HAVE_AVX) will already do the trick.

Which does only matter if this project is part of the directory scope calling add_definition. ExternalProject_Add opens a new independent directory scope. This leaves people who call add_subdirectory on external projects (i.e. this project) which is just asking for trouble in any case.