pipe and integration creation based on YAML file fails ,

[pguptajsq]

What happened?

i have this yaml and applied , it created pipe and integration but it showing error

kubectl logs pipe/test
error: no kind "Pipe" is registered for version "camel.apache.org/v1" in scheme "pkg/scheme/scheme.go:28"

and also Integration

kubectl get it
NAME                            PHASE   READY   RUNTIME PROVIDER   RUNTIME VERSION   CATALOG VERSION   KIT                        REPLICAS
test                       Error   False   quarkus            3.15.0            3.15.0            kit-ct1up2gro8bs73dchil0   1

kind: Pipe
metadata:
  name: test
  namespace: camel-k
spec:
  sink:
    properties:
      channel: channelname
      destinationName: destinan
      password: '{{pass}}'
      queueManager: test.UAT.BSSB
      serverName: serverip
      serverPort: "1414"
      username: app
    ref:
      apiVersion: camel.apache.org/v1
      kind: Kamelet
      name: jms-ibm-mq-sink
  source:
    properties:
      queueNameOrArn: queue
      region: us-west-2
      accessKey: "dummy"
      secretKey: "dummy/sk45"
    ref:
      apiVersion: camel.apache.org/v1
      kind: Kamelet
      name: aws-sqs-source

Any idea what's missing here?

Another Query how we can pass ServiceAccount to pipe yaml instead of aws accesskey and secretkey

Steps to reproduce

No response

Relevant log output

No response

Camel K version

No response

[squakez]

Hello. It seems the installation did not proceed correctly. I suggest you to clean the previous installation and proceed from scratch following the official Camel K installation guides.

[squakez]

You may also want to check the Camel K troubleshooting guide.

[pguptajsq]

how we can pass ServiceAccount to pipe yaml instead of aws accesskey and secretkey to access aws sqs ?

[pguptajsq]

this is sample of pipe yaml . i am passing service account in pipe but still getting beloow error
Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-sqs://ququ?accessKey=xxxxxx&amazonAWSHost=amazonaws.com&autoCreateQueue=false&delay=500&deleteAfterRead=true&greedy=false&overrideEndpoint=false&protocol=https&region=us-west-2&secretKey=xxxxxx&useDefaultCredentialsProvider=false due to: useDefaultCredentialsProvider is set to false, useProfileCredentialsProvider is set to false, useSessionCredentials is set to false, AmazonSQSClient or accessKey and secretKey must be specified

`apiVersion: camel.apache.org/v1
kind: Pipe
metadata:

spec:
serviceAccountName: camel-k-routes
sink:
properties:
channel:pavan.TO.BSSB.UAT.CHAN
destinationName: pavanUAT.BSSB.MT.INPUT
password: '{{pavan-backend/mq-app-password}}'
queueManager:paban.UAT.BSSB
serverName: ip
serverPort: "1414"
username: app
ref:
apiVersion: camel.apache.org/v1
kind: Kamelet
name: jms-ibm-mq-sink
source:
properties:
queueNameOrArn: queuearn
region: us-west-2
ref:
apiVersion: camel.apache.org/v1
kind: Kamelet
name: aws-sqs-source`

[squakez]

The usage of ServiceAccount has nothing to do with the Kamelet or component configuration. You still need to provide secret, likely in the form of Camel properties which are backed by a Secret config.

[pguptajsq]

Okay. thank you .
now i am getting error on integration , these integration automatically created when apply pipe yaml .
is there way we can pass traits to integration from pipe.yaml
kubectl logs payments-backend-sqs-to-ibmmq-pavan-8454c5755c-rngns exec /opt/java/openjdk/bin/java: exec format erro

from pipe.yaml how we can set
traits: affinity: node-affinity-labels: "kubernetes.io/arch in(amd64)"

when i am applying yaml with above traits setting , i am getting
Error from server (BadRequest): error when creating "pipe.yaml": Pipe in version "v1" cannot be handled as a Pipe: strict decoding error: unknown field "spec.traits"

[squakez]

Yes, the trait configuration in Pipes is different.

[pguptajsq]

Hi Team , could you please help me to leverage k8s secrets in pipe.yaml . here i am trying but its not working. how pipe.yaml will read secrets from k8s . could you please give me proper yaml

`apiVersion: camel.apache.org/v1
kind: pipe
metadata:
name: payments-backend-sqs-to-ibmmq-pavan
namespace: camel-k

spec:
traits:
mount:
configs:
- name: camel-sqs-cred
mountPath: /mnt/secrets
optional: false
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-sqs-source
properties:
queueNameOrArn: "queyename"
region: "us-west-2"
accessKeyId: "{{ env:aws-access-key-id }}"
secretAccessKey: "{{ env:aws-secret-access-key }}"
sink:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: jms-ibm-mq-sink
properties:
channel: "test.TO.BSSB.UAT.CHAN"
destinationName: "test.UAT.BSSB.MT.INPUT"
password: "{testbackend/mq-app-password}}"
queueManager: "test.UAT.BSSB"
serverName: "ip4"
serverPort: "1414"
username: "app"`

[squakez]

Hello. All the information is provided in the links in the documentation posted in previous comments, please, go through it.

1. You must specify traits via annotation
2. Once you've specified the secret to use, you can use any camel properties normally as {{my.camel.property}}

[pguptajsq]

@squakez : I tried using the access key and secret, but it's not working. Could you please provide a sample pipe.yaml file that uses secrets from Kubernetes secrets? Alternatively, is there a way to use an IAM role through a service account in the pipe.yaml

[squakez]

Hello, there should be enough documentation about this matter as shared in the previous comments. This is how to configure on an Integration: https://camel.apache.org/camel-k/2.5.x/configuration/runtime-config.html#runtime-config-props - the mechanisms for the Pipe is identical.

[pguptajsq]

"I followed the documentation, but it's not picking up the secrets and is giving the error: Caused by: j: java.lang.IllegalArgumentException: useDefaultCredentialsProvider is set to false, useProfileCredentialsProvider is set to false, useSessionCredentials is set to false, AmazonSQSClient or accessKey and secretKey must be specified

In my pipe.yaml, it works when I directly use the AWS accessKey and secretKey, but I cannot commit these keys to GitHub for security reasons. Instead, I want to use a Kubernetes ServiceAccount with IRSA roles in the kind: pipe Kubernetes configuration. How can I achieve this, or how can I reference Kubernetes secrets inside the pipe.yaml file

kid: pipe
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-sqs-source
properties:
queueNameOrArn: "queue-name"
region: "us-west-2"
accessKey: "keyname"
secretKey: "secrets"

[oscerd]

Use a secret, you cannot refer a service account.

[pguptajsq]

Okay , `apiVersion:`` camel.apache.org/v1
kind: Pipe
metadata:
name: sqs-to-ibmmq
namespace: camel-k
labels:

annotations:
trait.camel.apache.org/kubernetes.volumes: |
volumes:
- name: aws-credentials-volume
secret:
secretName: camel-sqs-cred # Reference the secret where AWS credentials are stored
volumeMounts:
- mountPath: /etc/camel/resources.d/_secrets/camel-sqs-cred # Mount path for the secret
name: aws-credentials-volume
trait.camel.apache.org/kubernetes.env: |
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: camel-sqs-cred
key: aws-access-key-id # AWS access key from the secret
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: camel-sqs-cred
key: aws-secret-access-key # AWS secret access key from the secret

spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-sqs-source
properties:
queueNameOrArn: "queuename"
region: "us-west-2"
accessKey: "{{ env:AWS_ACCESS_KEY_ID }}"
secretKey: "{{ env:AWS_SECRET_ACCESS_KEY }}"
sink:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: jms-ibm-mq-sink
properties:
channel: "test.TO.BSSB.UAT.CHAN"
destinationName: "test.UAT.BSSB.MT.INPUT"
password: "{{mq-app-password}}"
queueManager: "reat.UAT.BSSB"
serverName: "IP"
serverPort: "1414"
username: "app" , i tried this yaml also. but

Caused by: org.apache.camel.FailedToCreateRouteException: Failed to create route source: Route(source)[From[aws2-sqs:{{queueNameOrArn}}?accessKey=xxxxxx because of Failed to resolve endpoint: aws2-sqs://queue name?accessKey=xxxxxx&amazonAWSHost=amazonaws.com&autoCreateQueue=false&delay=500&deleteAfterRead=true&greedy=false&maxMessagesPerPoll=1&overrideEndpoint=false&protocol=https&region=us-west-2&secretKey=xxxxxx&sessionToken=xxxxxx&useDefaultCredentialsProvider=false&useProfileCredentialsProvider=false&useSessionCredentials=false due to: The security token included in the request is invalid. (Service: Sqs, Status Code: 403, Request ID: 877ef9f1-9e7d-5bf6-9e7f-a6392799a81d)

[oscerd]

Surround the secret key parameter with RAW().

Do you have special char in the key like + or /?

Then use RAW({{env:..}})

[pguptajsq]

Now getting aused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-sqs://queue-jaeaccessKey=xxxxxx)&amazonAWSHost=amazonaws.com&autoCreateQueue=false&delay=500&deleteAfterRead=true&greedy=false&maxMessagesPerPoll=1&overrideEndpoint=false&protocol=https&region=us-west-2&secretKey=xxxxxx)&sessionToken=xxxxxx&useDefaultCredentialsProvider=false&useProfileCredentialsProvider=false&useSessionCredentials=false due to: Invalid key=value pair (missing equal-sign) in Authorization header (hashed with SHA-256 and encoded with Base64): 'n89f/dLinHLU1RT5+GtK0ZOBXg7Ov6gUs8MgPGbLap4='.

[oscerd]

Can you post the pipe now? It seems you syntax is wrong

[pguptajsq]

`apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
name: test-backend-sqs-to-ibmmq-pavan
namespace: camel-k
labels:
helm.sh/chart: universal-chart_v2.48.4
app: test-backend
junipersquare.com/initiating_pull_request: pr-1427
junipersquare.com/initiating_repo: test-backend
junipersquare.com/requestor: euanlau
tags.datadoghq.com/env: prpay1427
tags.datadoghq.com/version: pr-1427
annotations:
app: payments-backend1
department: eng
junipersquare.com/initiating_pull_request: pr-1427
junipersquare.com/initiating_repo: test-backend1
junipersquare.com/requestor: euanlau
product: gpx fa
tags.datadoghq.com/env: prpay1427
tags.datadoghq.com/version: pr-1427
team: payments
annotations:
app: payments-backend1
trait.camel.apache.org/kubernetes.volumes: |
volumes:
- name: aws-credentials-volume
secret:
secretName: camel-sqs-cred # Reference the secret where AWS credentials are stored
volumeMounts:
- mountPath: /etc/camel/resources.d/_secrets/camel-sqs-cred # Mount path for the secret
name: aws-credentials-volume
trait.camel.apache.org/kubernetes.env: |
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: camel-sqs-cred
key: aws-access-key-id # AWS access key from the secret
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: camel-sqs-cred
key: aws-secret-access-key # AWS secret access key from the secret

spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-sqs-source
properties:
queueNameOrArn: "queuename"
region: "us-west-2"
accessKey: "({{ env:AWS_ACCESS_KEY_ID }})"
secretKey: "({{ env:AWS_SECRET_ACCESS_KEY }})"
sink:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: jms-ibm-mq-sink
properties:
channel: "test.TO.BSSB.UAT.CHAN"
destinationName: "test.UAT.BSSB.MT.INPUT"
password: "{{/mq-app-password}}"
queueManager: "JSQ.UAT.BSSB"
serverName: "ip"
serverPort: "1414"
username: "app"`

[oscerd]

This

({{ env:AWS_ACCESS_KEY_ID }})

Should be RAW({{ env:AWS_ACCESS_KEY_ID }})

Same for the secret key. I wrote it above

[pguptajsq]

Still same issue Caused by: software.amazon.awssdk.services.sqs.model.SqsException: Invalid key=value pair (missing equal-sign) in Authorization header (hashed with SHA-256 and encoded with Base64): 'hMg4ZI09YH7RNqzswja69tK20cHVncx0RApiU0fJLrU='. (Service: Sqs, Status Code: 400, Request ID: b4c352cb-e952-5053-b9e8-0d0317df63a2)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)

[oscerd]

That component, with that authentication mechanism,have been around for many years, so I can only suppose it is something on your side. We cannot help more than this.

[squakez]

@pguptajsq I think you can take as a reference this chat thread: https://camel.zulipchat.com/#narrow/channel/257299-camel-k/topic/Hashicorp.20Token.20in.20modeline/near/480962291

Ideally you should be able to put in your Pipe or Integration specification a Camel property, such as {{my.aws.key}}. Then, you need to provide a secret containing such variable, eg, kubectl create secret generic my-secret --from-literal my.aws.key=123456. Finally you can run your Pipe or Integration mounting the secret via mount.config=secret:my-secret. At runtime, the variable will be using the secret value.

[pguptajsq]

getting kubectl apply -f pipe1.yaml
Error from server (BadRequest): error when creating "pipe1.yaml": Pipe in version "v1" cannot be handled as a Pipe: strict decoding error: unknown field "mounts" getting this

[squakez]

Please, you need to carefully read and understand the solutions we have already proposed. This problem was previously discussed in this thread [1]. The configuration of annotations in Pipes is different than in Integrations.

[1] #5963 (comment)