Merge pull request OpenMS#6978 from cbielow/base64_simd

[FEATURE] Base64 encoding/decoding using SIMDe
cbielow · Aug 11, 2023 · 69a0ee6 · 69a0ee6
2 parents aba1cd5 + 69a1a31
commit 69a0ee6
Show file tree

Hide file tree

Showing 383 changed files with 194,995 additions and 538 deletions.
diff --git a/CHANGELOG b/CHANGELOG
@@ -17,6 +17,9 @@ PR - Pull Request (on GitHub), i.e. integration of a new feature or bugfix
 ----                                OpenMS 3.1.0     (under development)              ----
 ------------------------------------------------------------------------------------------
 
+- require some advanced instruction sets for x64 CPUs: SSE3 (g++/clang) or AVX (MSVC); and NEON for ARM64 CPUs (#6978)
+- Base64 encoding/decoding using the SIMDe library (#6978)
+
 Cleanup of old/unused tools and code:
 - Removed old tools and associated code in the library for InclusionExclusionListCreator, SvmTheoreticalSpectrumGenerator, PrecursorIonSelector, and MSSimulator
   Note: general SVC and SVR is still supported with the SimpleSVM class.
@@ -29,33 +32,71 @@ Cleanup of old/unused tools and code:
 ----                                OpenMS 3.0     (released 7/2023)                  ----
 ------------------------------------------------------------------------------------------
 
-- full release
-- OpenSwath: Add support for diaPASEF data with overlapping m/z and IM windows, and add new outputs on ion mobility features (delta_im)
-- OpenSwath: Fix duplicated transition error when multiple genes map to a single peptide (#5653)
-- TOPPView: TheoreticalSpectrumGenerationDialog now supports generation of isotope patterns for metabolites
-- pyopenms: pyopenms-extra is renamed to pyopenms-docs.
-- Add extractSpectra to TargetedSpectraExtractor with support of MS1 features
-- TargetedSpectraExtractor::searchSpectrum added an option to add unknown features to the result featureMap
-- Fixed race condition when logging messages.
-- Added the support for 'no cleavage' for XTandemAdapter and CometAdapter (#6133).
-- ParamEditor with more convenient StringList editing  (#5135)
-- TOPPAS: add a `recent files` submenu
-- removed InspectAdapter
-- removed OMSSAAdapter
-- removed MyriMatchAdapter
-- removed Crux
-- OpenMS documentation is moved to openms.readthedocs.io/en/latest. OpenMS API
-reference and advanced developer documentation remains inside OpenMS doxygen
-documentation.
-- Add support for user defined ribonucleotides with phosphorothioate linkages
-- Add support for JSON based ribonucleotides and updated to latest Modomics database
-- fix bug in GaussFilter when using ppm as width (#6830)
-- Add support to load parameter values from a JSON formatted .json file. (Accessible via -ini. This will be
-  helpful for Common Workflow Language users and others)
-
 New Tools:
-- FLASHDeconv -- Ultra-fast high-quality deconvolution enables online processing of top-down MS data (TOPP)
-- FLASHDeconvWizard -- An GUI assistant for FLASHDeconv execution.
+
+    FLASHDeconv -- Ultra-fast high-quality deconvolution enables online processing of top-down MS data (TOPP)
+    FLASHDeconvWizard -- A GUI assistant for FLASHDeconv execution.
+
+New Features:
+
+e.g.
+
+    TMT18plex support (#6390)
+    ProteinQuantifier supports iBAQ (#6107)
+    OpenSwath: Add support for diaPASEF data with overlapping m/z and IM windows, and add new outputs on ion mobility features (delta_im), IM calibration (#5911, #6234, #6268)
+    OpenSwathDecoyGenerator speed improvement and remove duplicates (#6054)
+    NucleicAcidSearchEngine (NASE): user defined ribonucleotides with phosphorothioate linkages (#6337), JSON based ribonucleotides and updated to latest Modomics database (#6482)
+    TargetedSpectraExtractor: more features (#6106)
+    TOPPView: TheoreticalSpectrumGenerationDialog now supports generation of isotope patterns for metabolites (#6023); faster loading of external drag'n'drop data (#6837)
+    colored commandline/console on all platforms (#6275)
+    support for 'no cleavage' for XTandemAdapter and CometAdapter (#6133).
+    Percolator pin file reader (#6824)
+    JSON export for OMS files(SQLite) (#6114)
+    ParamEditor with more convenient StringList editing (#5135)
+    load parameter values from a JSON formatted .json file. (Accessible via -ini. This will be
+    helpful for Common Workflow Language users and others)
+    FileFilter can remove convex hulls of features and consensusFeatures to reduce file size (#6140)
+    Faster compile time (#6618)
+    Improving code quality by fixing lots of linting warnings and leaks (e.g. #6839, #6831, #6829)
+
+Documentation
+
+    website redesign (visit openms.org)
+    OpenMS user documentation is moved to openms.readthedocs.io/en/latest.
+    OpenMS API reference and advanced developer documentation remains inside OpenMS doxygen
+    documentation (https://abibuilder.cs.uni-tuebingen.de/archive/openms/Documentation/release/)
+    pyopenms: pyopenms-extra is renamed to pyopenms-docs.
+
+Bug fixes
+
+e.g.
+
+    GaussFilter when using ppm as width (#6830)
+    NASE a-B ion masses (#6718), ID-Mapper for TMT data (#6758)
+    FeatureFinderMetaboliteIdentification speed improvements (#6619)
+    IDRipper speed improvements (#6584)
+    Honor MissedCleavages in SimpleSearchEngine (#6889)
+    TOPPView: fixed lots of display glitches, e.g. axis labels, goto dialog and easier re-use of components, etc (#6673, #6616, #6592, #6703, #6793)
+    mzTab fixes for empty IDs (#6445)
+    Fix GNPS error for empty scans in Bruker files (#6898)
+    PrecursorPurity: handle unknown charge (#6283)
+    OpenSwath: Fix duplicated transition error when multiple genes map to a single peptide (#5653)
+    Fixed race condition when logging messages.
+
+Removed tools:
+
+    InspectAdapter
+    OMSSAAdapter
+    MyriMatchAdapter
+    CruxAdapter
+
+Supported compilers (when building from source):
+
+    g++ (7.0 or later, tested up to v13.0)
+    clang (?, tested up to v16)
+    Visual Studio (2019(v16.8.4) or later)
+
+Full changelog: [OpenMS 2.8 &rarr; 3.0](https://github.com/OpenMS/OpenMS/compare/Release2.8.0...Release3.0.0)
 
 ------------------------------------------------------------------------------------------
 ----                                OpenMS 2.8     (released 2/2022)                  ----

diff --git a/cmake/Modules/FindSIMDe.cmake b/cmake/Modules/FindSIMDe.cmake
@@ -0,0 +1,15 @@
+find_path(SIMDe_INCLUDE_DIR simde/simde-align.h)
+
+include(FindPackageHandleStandardArgs)
+
+find_package_handle_standard_args(
+  SIMDe
+  REQUIRED_VARS SIMDe_INCLUDE_DIR
+)
+
+if(NOT TARGET SIMDe)
+  add_library(SIMDe INTERFACE)
+  target_include_directories(SIMDe SYSTEM INTERFACE ${SIMDe_INCLUDE_DIR})
+endif()
+
+mark_as_advanced(SIMDe_INCLUDE_DIR)
diff --git a/cmake/compiler_flags.cmake b/cmake/compiler_flags.cmake
@@ -48,6 +48,34 @@ if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
   add_definitions(-D_LIBCPP_DISABLE_AVAILABILITY)  
 endif()
 
+########
+########    deal with SSE4/AVX flags
+########
+set(x64_CPU "x86|AMD64") ## CMake returns 'x86-64' on Linux and 'AMD64' on Windows..
+message(STATUS "Processor is : ${CMAKE_SYSTEM_PROCESSOR}")
+if (MSVC)
+  ## enable 'AVX' on x86-64, and 'neon' on arm, to achive faster base64 en-/decoding via SIMDe
+  if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "${x64_CPU}") 
+    ## for SIMDe we need to use explicit compiler flags, which in turn define macros (like '#define __AVX__'), which SIMDe will check for and only then create vectorized code
+    ## Disabling AVX will actually make the SIMDe code slower compared to the non-SSE version (for Base64 encoding/decoding at least)
+    add_compile_options(/arch:AVX)  ## note: MSVC lacks flags for SSE3/SSE4 (only unofficial ones like /d2archSSE42 are available, but SIMDe does not care about them)
+  elseif (${CMAKE_SYSTEM_PROCESSOR} MATCHES "arm")
+    add_compile_options(/arch:neon)
+  endif()
+else()  ## GCC/Clang/AppleClang
+  ## enable SSE3 on x86, 'neon' on arm to achive faster base64 en-/decoding
+  if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "${x64_CPU}") 
+    add_compile_options(-mssse3)
+  elseif (${CMAKE_SYSTEM_PROCESSOR} MATCHES "arm")
+    add_compile_options(-neon)
+  endif()
+endif()
+
+
+####
+####  more flags...
+####
+
 if (CMAKE_COMPILER_IS_GNUCXX)
 
   add_compile_options(-Wall -Wextra
@@ -56,9 +84,10 @@ if (CMAKE_COMPILER_IS_GNUCXX)
     -Wno-long-long 
     -Wno-unknown-pragmas
     -Wno-unused-function
-    -Wno-variadic-macros)
-
-  option(ENABLE_GCC_WERROR "Enable -WError on gcc compilers" OFF)
+    -Wno-variadic-macros
+    )
+
+  option(ENABLE_GCC_WERROR "Enable -Werror on gcc compilers" OFF)
   if (ENABLE_GCC_WERROR)
     add_compile_options(-Werror)
     message(STATUS "Enable -Werror for gcc - note that this may not work on all compilers and system settings!")
@@ -69,7 +98,6 @@ if (CMAKE_COMPILER_IS_GNUCXX)
   if (CMAKE_GENERATOR STREQUAL "Eclipse CDT4 - Unix Makefiles")
     add_compile_options(-fmessage-length=0)
   endif()
-
 
 elseif (MSVC)
 	# do not use add_definitions
@@ -110,11 +138,6 @@ elseif (MSVC)
 	## use multiple CPU cores (if available)
 	set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /MP")
 
-  if (NOT OPENMS_64BIT_ARCHITECTURE)
-    ## enable SSE1 on 32bit, on 64bit the compiler flag does not exist
-    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /arch:SSE")
-  endif()
-
 elseif (CMAKE_CXX_COMPILER_ID MATCHES "Clang") # using regular Clang or AppleClang
 
   set(CMAKE_COMPILER_IS_CLANG true CACHE INTERNAL "Is CLang compiler (clang++)")
@@ -179,5 +202,4 @@ if (CXX_WARN_CONVERSION)
     add_compile_options(-Wconversion)
   endif()
 endif()
-message(STATUS "Compiler checks for conversion: ${CXX_WARN_CONVERSION}")
-
+message(STATUS "Compiler checks for conversion: ${CXX_WARN_CONVERSION}")
diff --git a/src/openms/CMakeLists.txt b/src/openms/CMakeLists.txt
@@ -90,17 +90,21 @@ set(OPENMS_DEP_LIBRARIES Evergreen LibSVM::LibSVM XercesC::XercesC Qt5::Core Qt5
 
 ## setup the argumentes to 'target_link_libraries(OpenMS PRIVATE ${OPENMS_DEP_PRIVATE_LIBRARIES})'
 set(OPENMS_DEP_PRIVATE_LIBRARIES 
-  GTE Quadtree ${LPTARGET} 
-  Eigen3::Eigen
   $<$<BOOL:${WITH_HDF5}>:HDF5::HDF5>
-  Boost::iostreams
-  Boost::date_time
+  ${LPTARGET} 
+  BZip2::BZip2
   Boost::boost
+  Boost::date_time
+  Boost::iostreams
   Boost::regex
-  BZip2::BZip2
-  ZLIB::ZLIB
+  Eigen3::Eigen
+  GTE 
+  Quadtree 
+  SIMDe
   SQLiteCpp
-  nlohmann_json::nlohmann_json)
+  ZLIB::ZLIB
+  nlohmann_json::nlohmann_json
+  )
 
 ## ----------
 ## No need to for this at the moment: no Qt plugins required anymore. Left here for reference.

diff --git a/src/openms/include/OpenMS/CONCEPT/ClassTest.h b/src/openms/include/OpenMS/CONCEPT/ClassTest.h
@@ -980,8 +980,8 @@ namespace TEST = OpenMS::Internal::ClassTest;
           stdcout << " +  line " << TEST::test_line <<                                    \
             ":  TEST_EXCEPTION_WITH_MESSAGE(" # exception_type "," # command ", " # message   \
             "): OK\n";                                                                      \
-          break;                                                                            \
         }                                                                                 \
+        break;                                                                            \
       case 2:                                                                             \
         stdcout << " -  line " << TEST::test_line <<                                    \
           ":  TEST_EXCEPTION_WITH_MESSAGE(" # exception_type "," # command ", " # message   \