Pilots/ARC "proxy not found"

[marianne013]

Hi All,

We've been having problems on ARC-CEs where the pilot jobs fail with a proxy not found error on the server Janusz is using to develop his pilot logging code. (HTCondorCE works fine.)
Now I have just noticed that the same problem has appeared on the certification server, but only for RALPP and Glasgow (and even in Glasgow it's ce01 & 02 returning the error and not ce04). Initially I though it may be because we use single core queues on those CEs and I didn't set "LocalCEType = Pool" for the single core queues (and that was about the only difference I can see), but it turns out, adding this makes no difference (and by the looks of it Janusz has always used PoolCE anyway).
I'm still suspicious as ce04.gla.scotgrid.ac.uk on the certification server was auto configured, and ce01 and ce02 modified, but I can't see any difference that would explain this.
We do not see this on our v7r3 production server.
Does anyone have a creative idea ?

Daniela

[andresailer]

Have you tried adding (runTimeEnvironment="ENV/PROXY") to the XRLSString thing?

[marianne013]

No, but I will. Still doesn't explain why some (pilot) jobs at Glasgow worked and most of them didn't.

[marianne013]

I stuck it in CEDefaults on the certification server. Now the Glasgow jobs seem to work, but RALPP still fails.

[marianne013]

I've stuck it in CEDefaults. That certainly seems to have fixed Glasgow, but RALPP still fails.

[marianne013]

So I managed to get it "working" at RALPP by deleting one of the queues. It's almost like ARC cannot handle multiple queues with one CE.
This even seems to hold for the Janusz instance where one queue has a RequiredTag, i.e. jobs get sent unambiguously to one or the other queue.

[marianne013]

I was too quick. All the jobs on the certification server have now run, though I still see ARC pilots with a proxy not found error.
I haven't managed to make Janusz' machine to work at all.
Both RALPP and Glasgow work fine with multiple queues and without the extra string on the production server, so there is definitely something off. Also I don't see this on the HTCondorCEs at all, even with multiple queues.

[marianne013]

@martynia I tried ....

[martynia]

And…

…

On Thu, 27 Oct 2022 at 16:40, Daniela Bauer ***@***.***> wrote: @martynia <https://github.com/martynia> I tried .... — Reply to this email directly, view it on GitHub <#6465 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2KE7M7JKMWM2W34SUWR43WFKO5BANCNFSM6AAAAAARQAKKVA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[marianne013]

You are special. I got jobs running on the certification machine, but not on yours, even though the error is the same.

[marianne013]

I think we can close this, we'll change to AREX and see if the problem reoccurs.