Requests get stuck in Scheduled Status

[arrabito]

Dear all,

I'm performing 'Moving' transformations and times to times transformations do not progress anymore.
I found that for some reasons the associated requests get stuck in Scheduled status.
Do you have any idea in which conditions this may happen and how to fix it?

Looking at the reqdb summary (at the end of the message), I see that there are 760 ReplicateAndRegister operations in Waiting Status. Is this the reason why Requests get stuck as Scheduled?

If yes, I then don't really understand why Operations get stuck in Waiting status.

Thank you for your help,

Luisa

dirac-rms-reqdb-summary 
Requests:
- 'Assigned' 4
- 'Canceled' 402206
- 'Done' 3280
- 'Failed' 205783
- 'Scheduled' 4396
Operations:
- 'CheckMigration':
  - 'Failed' 3
  - 'Queued' 5
- 'ForwardDISET':
  - 'Done' 5734
  - 'Failed' 58729
- 'MoveFile':
  - 'Failed' 2
  - 'Waiting' 1
- 'MoveReplica':
  - 'Failed' 1
- 'RegisterFile':
  - 'Done' 5
  - 'Failed' 144
  - 'Queued' 1
- 'RegisterReplica':
  - 'Done' 402322
  - 'Failed' 2
- 'RemoveFile':
  - 'Done' 13
  - 'Failed' 2294
- 'RemoveReplica':
  - 'Done' 401854
  - 'Failed' 126410
  - 'Queued' 26187
- 'ReplicateAndRegister':
  - 'Done' 528306
  - 'Failed' 20559
  - 'Queued' 91
  - 'Scheduled' 4832
  - 'Waiting' 760
Files:
- 'Done' 3433328
- 'Failed' 268870
- 'Scheduled' 439837
- 'Waiting' 538186

[arrabito]

Looking directly in the ReqDB I found that all Requests associated to these Waiting Operations are in Canceled Status.

I thought that cancelling a request would have also cancelled the associated operations, but it does not seem so.
Here below an example:

dirac-rms-request 1350007
Request name='00002313_00000001' ID=1350007 Status='Waiting' ('Canceled' in DB)
Created 2020-12-01 11:04:03, Updated 2020-12-03 11:26:14, NotBefore 2020-12-01 11:04:03
Owner: '/O=GRID-FR/C=FR/O=CNRS/OU=LUPM/CN=Luisa Arrabito', Group: cta_prod
  [0] Operation Type='ReplicateAndRegister' ID=1534045 Order=0 Status='Waiting'
      SourceSE: DESY-ZN-Disk - TargetSE: CC-IN2P3-Tape - Created 2020-12-01 11:04:03, Updated 2020-12-01 11:04:03
    [01] ID=28658730 LFN='/vo.cta.in2p3.fr/MC/PROD3/Paranal/electron/merge_simtel/738/Data/003xxx/electron_20deg_180deg_run3144___cta-prod3-merged_desert-2150m-Paranal-3HB8-NG.simtel.gz.tar' Status='Waiting'
  [1] Operation Type='RemoveReplica' ID=1534046 Order=1 Status='Queued'
      TargetSE: DESY-ZN-Disk - Created 2020-12-01 11:04:03, Updated 2020-12-01 11:04:03
    [01] ID=28658731 LFN='/vo.cta.in2p3.fr/MC/PROD3/Paranal/electron/merge_simtel/738/Data/003xxx/electron_20deg_180deg_run3144___cta-prod3-merged_desert-2150m-Paranal-3HB8-NG.simtel.gz.tar' Status='Waiting'

Could you suggest a proper way to get rid of these requests?
Thank you.

[chaen]

Hi. If a Request is in Scheduled status, it means that there should be an Operation in Scheduled status too, and an FTS3Operation created for it. You should look into the FTS sytem what happened to your FTS3Operation

[chaen]

the Canceled status only acts on the Request. But that's the only status that matters for the TransformationSystem so I am not sure what exactly is your issue