Conflicts resolved with rel-v7r2

.github/workflows/basic-python3.yml

@@ -19,14 +19,7 @@ jobs:
           # TODO These three tests fail on Python 3:
           #   * `test_BaseType_Unicode` and `test_nestedStructure` fail due to
           #     DISET's string and unicode types being poorly defined
-          #   * `testLockedClass` deadlocks frequently due to the use of
-          #     self.__processThread._Thread__stop(). Officially "threads
-          #     cannot be destroyed, stopped, suspended, resumed, or
-          #     interrupted" so the code should be rewritten to follow the
-          #     supported alternative: "If you want your threads to stop
-          #     gracefully, make them non-daemonic and use a suitable
-          #     signalling mechanism such as an Event."
-          - pytest --runslow -k 'not test_BaseType_Unicode and not test_nestedStructure and not testLockedClass'
+          - pytest --runslow -k 'not test_BaseType_Unicode and not test_nestedStructure'
           - pylint -E src/
 
     steps:

dirac.cfg

@@ -665,6 +665,8 @@ Operations
         FTS3
         {
           ServerPolicy = Random # http://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/DataManagement/fts.html#ftsserver-policy
+        # Plugin to alter default TPC selection list
+          FTS3Plugin = Default # http://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/DataManagement/fts.html#fts3-plugins
         }
       }
     }

docs/source/AdministratorGuide/Resources/storage.rst

@@ -307,6 +307,11 @@ External services like FTS requires pair of URLs to perform third party copy.
 This is implemented using the same logic as described above. There is however an extra step: once the common protocols between 2 SEs have been filtered, an extra loop filter is done to make sure that the selected protocol can be used as read from the source and as write to the destination. Finally, the URLs which are returned are not necessarily the url of the common protocol, but are the native urls of the plugin that can accept/generate the common protocol. For example, if the common protocol is gsiftp but one of the SE has only an SRM plugin, then you will get an srm URL (which is compatible with gsiftp).
 
 
+.. versionadded:: v7r1p37
+    The FTS3Agent can now use plugins to influence the list of TPC protocols used. See :ref:`fts3`
+
+
+
 Protocol matrix
 ^^^^^^^^^^^^^^^
 

docs/source/AdministratorGuide/Systems/DataManagement/fts3.rst

@@ -50,6 +50,7 @@ Operations configuration
 
   * DataManagement/FTSVersion: FTS2/FTS3. Set it to FTS3...
   * DataManagement/FTSPlacement/FTS3/ServerPolicy: Policy to choose the FTS server see `FTSServer policy`_.
+  * DataManagement/FTSPlacement/FTS3/FTS3Plugin: Plugin to alter the behavior of the FTS3Agent
 
 
 ======================
@@ -92,11 +93,11 @@ The RMS will create one FTS3TransferOperation per RMS Operation, and one FTS3Fil
 The grouping into jobs is done following this logic:
     * Group by target SE
     * Group by source SE. If not specified, we take the active replicas as returned by the DataManager
-    * Since their might be several possible source SE, we need to pick one only. The choice is to select the SE where there is the most files of the operation present. This increases the likely hood to pick a good old Tier1
+    * Since there might be several possible source SEs, we need to pick one only. By default, the choice is random, but this can be changed (see FTS3Plugins)
     * Divide all that according to the maximum number of files we want per job
 
 Once the FTS jobs have been executed, and all the operation is completed, the callback takes place. The callback consists in fetching the RMS request which submitted the FTS3Operation, update the status of the RMS files, and insert a Registration Operation.
-Note that since the multiple targets are grouped in a single RMS operation, failing to transfer one file t one destination will result in the failure of the Operation. However, there is one Registration operation per target, and hence correctly transferred files will be registered.
+Note that since the multiple targets are grouped in a single RMS operation, failing to transfer one file to one destination will result in the failure of the Operation. However, there is one Registration operation per target, and hence correctly transferred files will be registered.
 
 ====================
 FTS3StagingOperation
@@ -170,3 +171,13 @@ States from the FTS3Job::
 The status of the FTS3Jobs and FTSFiles are updated every time we monitor the matching job.
 
 The FTS3Operation goes to Processed when all the files are in a final state, and to Finished when the callback has been called successfully
+
+
+FTS3 Plugins
+------------
+
+.. versionadded:: v7r1p37
+    The ``FTS3Plugin`` option
+
+
+The ``FTS3Plugin`` option allows one to specify a plugin to alter some default choices made by the FTS3 system. These choices concern the list of third party protocols used, as well as the selection of a source storage element. This can be useful if you want to implement a matrix-like selection of protocols, or if some links require specific protocols, etc. The plugins must be placed in :py:mod:`DIRAC.DataManagementSystem.private.FTS3Plugins`. The default behaviors, as well as the documentation on how to implement your own plugin can be found in :py:mod:`DIRAC.DataManagementSystem.private.FTS3Plugins.DefaultFTS3Plugin`

docs/source/DeveloperGuide/AddingNewComponents/DevelopingCommands/index.rst

@@ -37,7 +37,7 @@ which will set the interpreter directive to the python on the environment.
 
 **2.** The next is the documentation line which is describing the command. This same documentation line will be used also the command help information available with the *-h* command switch.
 
-**3.** The majority of the code should be contained with a function, often called ``main`` though this is not required. This function should be wrapped with the ``@DiracScript()`` decorator to allow the DIRAC plugin mechanism to override the script with the function from the highest priority extension.
+**3.** The majority of the code should be contained with a function, often called ``main`` though this is not required. This function should be wrapped with the ``@DIRACScript()`` decorator to allow the DIRAC plugin mechanism to override the script with the function from the highest priority extension.
 
 .. code-block:: python
 
@@ -46,8 +46,8 @@ which will set the interpreter directive to the python on the environment.
    from DIRAC.Interfaces.API.DIRAC import DIRAC
    from DIRAC import gLogger
 
-   @DiracScript()
-   def main()
+   @DIRACScript()
+   def main():
      # Do stuff
 
    if __name__ == "__main__":

environment-py3.yml

@@ -9,7 +9,7 @@ dependencies:
   # Temporary workarounds
   - astroid !=2.5  # https://github.com/PyCQA/astroid/issues/903
   # runtime
-  - python =3.8
+  - python =3.9
   - pip
   - boto3
   - certifi

release.notes

@@ -17,6 +17,32 @@ CHANGE: (#4937) removed StatesMonitoringAgent (use StatesAccountingAgent agent i
 *tests
 CHANGE: (#5046) don't use mail in the self generated certificates
 
+[v7r2p3]
+
+FIX: Fixes from v7r054 and v7r1p37
+FIX: (#5088) Fix various Python 3.9 deprecations
+
+*Core
+FIX: (#5095) Avoid KeyError when calling parseCommandLine
+
+*Framework
+CHANGE: (#5092) Use String(64) for DIRACVersion column in InstalledComponentsDB.HostLogging
+CHANGE: (#5098) in the run files created by ComponentInstaller, execute directly the deployed scripts
+
+*WMS
+NEW: (#5073) JobManager service now exposes a "RemoveJob" RPC call
+NEW: (#5073) implementing bulk indexing for jobs parameters
+CHANGE: (#5073) JobCleaningAgent will not remove those jobs that, while marked "DELETED", have still Operations to do in RMS
+
+*RequestManagement
+FIX: (#5091) Ensure BLOB columns are always passed bytes data
+
+*Resources
+CHANGE: (#5050) StompMQConnector uses a single connection
+
+*tests
+NEW: (#5086) Integration tests can be re-ran locally without re-installing the entire setup
+
 [v7r2p2]
 
 FIX: Fixes from v7r053 and v7r1p36
@@ -154,6 +180,11 @@ NEW: (#4910) --runslow option on unit tests to allow faster local tests
 NEW: (#4938) added a helloworld test for the (yet to be implemented) cloud testing in certification
 CHANGE: (#4968) Change the defaults for tests (to MySQL 8 and ES 7)
 
+[v7r1p37]
+
+*DataManagement
+NEW: (#5096) FTS3 plugins to alter TPC and source SE preferences
+
 [v7r1p36]
 
 *DMS
@@ -769,6 +800,16 @@ FIX: (#4551) align ProxyDB test to current changes
 NEW: (#4289) Document how to run integration tests in docker
 NEW: (#4551) add DNProperties description to Registry/Users subsection
 
+[v7r0p54]
+
+*Core
+FIX: (#5087) Undefined variable error when RabbitMQAdmin fails to connect
+
+*CS
+FIX: (#5090) VOMS2CSSynchronizer: add email information about a user joining groups for an additional VO, fix for #5037
+FIX: (#5090) VOMS2CSSynchronizer: When a user leaves a second VO, only the membership in these groups is removed
+CHANGE: (#5090) VOMS2CSSynchronizer: If email of user is known, do not replace it with None
+
 [v7r0p53]
 
 *Interface

src/DIRAC/ConfigurationSystem/Agent/VOMS2CSAgent.py

@@ -180,6 +180,7 @@ def execute(self):
             mailMsg += result['Message']
         if self.dryRun:
           self.log.info("Dry Run: mail won't be sent")
+          self.log.info(mailMsg)
         else:
           NotificationClient().sendMail(self.am_getOption('MailTo', voAdminMail),
                                         "VOMS2CSAgent run log", mailMsg,

src/DIRAC/ConfigurationSystem/Client/VOMS2CSSynchronizer.py

@@ -316,9 +316,11 @@ def syncCSWithVOMS(self):
       modified = False
       suspendedInVOMS = self.vomsUserDict[dn]['suspended'] or self.vomsUserDict[dn]['certSuspended']
       suspendedVOList = getUserOption(diracName, 'Suspended', [])
+      knownEmail = getUserOption(diracName, 'Email', None)
       userDict = {"DN": dn,
                   "CA": self.vomsUserDict[dn]['CA'],
-                  "Email": self.vomsUserDict[dn].get('mail', self.vomsUserDict[dn].get('emailAddress'))}
+                  "Email": self.vomsUserDict[dn].get('mail', self.vomsUserDict[dn].get('emailAddress')) or knownEmail,
+                  }
 
       # Set Suspended status for the user for this particular VO
       if suspendedInVOMS and self.vo not in suspendedVOList:
@@ -376,7 +378,12 @@ def syncCSWithVOMS(self):
         if groups['OK']:
           self.log.info('Found groups for user %s %s' % (diracName, groups['Value']))
           userDict['Groups'] = list(set(groups['Value'] + keepGroups))
+          addedGroups = list(set(userDict['Groups']) - set(groups['Value']))
           modified = True
+          message = "\n  Modified user %s:\n" % diracName
+          message += "    Added to group(s) %s\n" % ','.join(addedGroups)
+          self.adminMsgs['Info'].append(message)
+
 
       # Check if something changed before asking CSAPI to modify
       if diracName in diracUserDict:
@@ -415,15 +422,37 @@ def syncCSWithVOMS(self):
     for user in diracUserDict:
       dnSet = set(fromChar(diracUserDict[user]['DN']))
       if not dnSet.intersection(set(self.vomsUserDict)) and user not in nonVOUserDict:
-        for group in diracUserDict[user]['Groups']:
-          if group not in noVOMSGroups:
-            oldUsers.add(user)
+        existingGroups = diracUserDict.get(user, {}).get('Groups', [])
+        nonVOGroups = list(set(existingGroups) - set(diracVOMSMapping))
+        removedGroups = list(set(existingGroups) - set(nonVOGroups))
+        if removedGroups:
+          self.log.info("Checking user for deletion", "%s: %s" % (user, existingGroups))
+          self.log.info("User has groups in other VOs", "%s: %s" % (user, nonVOGroups))
+          userDict = diracUserDict[user]
+          userDict['Groups'] = nonVOGroups
+          if self.autoModifyUsers:
+            result = self.csapi.modifyUser(user, userDict)
+            if result['OK'] and result['Value']:
+              self.log.info("Modified user %s: %s" % (user, str(userDict)))
+              self.voChanged = True
+              message = "\n  Modified user %s:\n" % user
+              modMsg = "    Removed from group(s) %s\n" % ','.join(removedGroups)
+              self.adminMsgs['Info'].append(message + modMsg)
+              resultDict['ModifiedUsers'].append(user)
+          continue
+        if not any(group in noVOMSGroups for group in existingGroups):
+          oldUsers.add(user)
 
     # Check for obsoleted DNs
     for user in diracUserDict:
       dnSet = set(fromChar(diracUserDict[user]['DN']))
       for dn in dnSet:
         if dn in obsoletedDNs and user not in oldUsers:
+          existingGroups = diracUserDict.get(user, {}).get('Groups', [])
+          nonVOGroups = list(set(existingGroups) - set(diracVOMSMapping))
+          if nonVOGroups:
+            self.log.verbose("User has groups in other VOs", "%s: %s" % (user, nonVOGroups))
+            continue
           self.log.verbose("Modified user %s: dropped DN %s" % (user, dn))
           if self.autoModifyUsers:
             userDict = diracUserDict[user]
@@ -453,7 +482,7 @@ def syncCSWithVOMS(self):
       else:
         self.adminMsgs['Info'].append('The following users to be checked for deletion:\n\t%s' %
                                       "\n\t".join(sorted(oldUsers)))
-        self.log.info('The following users to be checked for deletion: %s' % str(oldUsers))
+        self.log.info('The following users to be checked for deletion:', "\n\t".join(sorted(oldUsers)))
 
     resultDict['CSAPI'] = self.csapi
     resultDict['AdminMessages'] = self.adminMsgs

src/DIRAC/Core/Base/Script.py

@@ -60,7 +60,10 @@ def parseCommandLine(script=False, ignoreErrors=False, initializeMonitor=False):
 
   # Read and parse the script __doc__ to create a draft help message
   if not gIsAlreadySetUsageMsg:
-    localCfg.setUsageMessage(inspect.currentframe().f_back.f_globals['__doc__'])
+    try:
+      localCfg.setUsageMessage(inspect.currentframe().f_back.f_globals['__doc__'])
+    except KeyError:
+      pass
     gIsAlreadySetUsageMsg = True
 
   if gIsAlreadyInitialized:
@@ -134,7 +137,10 @@ def addDefaultOptionValue(option, value):
 def setUsageMessage(usageMessage):
   global gIsAlreadySetUsageMsg
   gIsAlreadySetUsageMsg = True
-  localCfg.setUsageMessage(inspect.currentframe().f_back.f_globals['__doc__'])
+  try:
+    localCfg.setUsageMessage(inspect.currentframe().f_back.f_globals['__doc__'])
+  except KeyError:
+    pass
   localCfg.setUsageMessage(usageMessage)