404 error in report-staged channel

[natalieda]

When I delete files in a directory that a report-staged channel is listening, I get this error in the channel:

curl: (22) The requested URL returned error: 404 Not Found ./ada: line 978: [: =: unary operator expected

Could you please check this bug?

[onnozweers]

4e6e865
I think this fixes the "line 978: [: =: unary operator expected".

[onnozweers]

As for the other part of the issue:

"curl: (22) The requested URL returned error: 404 Not Found"

This happens because Ada collects file info for the file, but it doesn't exist anymore.

Our options:

• Ignore the 404 error.
• List an event for files that have been removed.
  • But what to do when a file is moved? Should we list staging events for the new location of the file? Or only when it falls inside the scope of the channel? Do we actually list stage events for new files? Should we? This can become very complicated.

[natalieda]

Hi Onno,

Since the error "line 978: [: =: unary operator expected" is fixed I think the 404 error can be ignored. Returning "not found" makes sense for deleted files.

[onnozweers]

But you don't see the file name in the error, do you? It would make sense if it included the file name.

[natalieda]

I think we should provide information inside the scope of the channel. If a file is removed, the error "Not found " should be returned and if possible the filename too. If the file is moved to a location outside the scope of the channel then I would say not to display information on the new location.

We need to test the behavior when new files are created within the scope of the channel as it would be handy to display their locality for consistency

[onnozweers]

Fixed in c52bf90

Example output (while renaming bloemen.jpg to bloemen2.jpg and back):

[surfadvisors-ozweers@wn-ui-01 ~]$ SpiderScripts/ada/ada --tokenfile dolphin12-onno.conf --report-staged mychannel1 /users/onno/tape --recursive
Channel: https://dolphin12.grid.surfsara.nl:20443/api/v1/events/channels/4cddfGM65YuBKPIz9Q_2Hw
path=/users/onno/tape
...
Listing initial file status...
...
ONLINE_AND_NEARLINE  disk+tape  /users/onno/tape/bloemen.jpg
...
Listening for file status changes...
Resuming from 31

WARNING: could not get object type of /users/onno/tape/bloemen.jpg. It may have been deleted or moved.
ONLINE_AND_NEARLINE  tape  /users/onno/tape/bloemen2.jpg
NEARLINE  tape  /users/onno/tape/bloemen2.jpg
WARNING: could not get object type of /users/onno/tape/bloemen2.jpg. It may have been deleted or moved.
NEARLINE  tape→disk+tape  /users/onno/tape/bloemen.jpg

Leaving issue open for: "We need to test the behavior when new files are created within the scope of the channel as it would be handy to display their locality for consistency".

We also may need to check what happens when you rename a subdir.

[onnozweers]

For new files there already is an IN_ATTRIB event, so they are listed:

ONLINE  volatile→tape  /users/onno/tape/mylist
ONLINE_AND_NEARLINE  tape  /users/onno/tape/mylist

[onnozweers]

For moved/renamed files, I added -e IN_MOVED_TO to the grep. That will list the new location, including locality and QoS.

Renaming a file and renaming it back (nothing else):

WARNING: could not get object type of /users/onno/tape/bloemen.jpg. It may have been deleted or moved.
ONLINE_AND_NEARLINE  disk+tape  /users/onno/tape/bloemen2.jpg
WARNING: could not get object type of /users/onno/tape/bloemen2.jpg. It may have been deleted or moved.
ONLINE_AND_NEARLINE  disk+tape  /users/onno/tape/bloemen.jpg

[onnozweers]

Listing new location of renamed file: d6e91d2