调用自定义组件失败 #486
Comments
|
1.确认下双方路由是正常的，kubectl get dr -A |
|
1.在其中一方运行kubectl get dr -A，能看到另一方是认证过的； |
|
kubectl describe node看下 |
|
Name: idata-kuscia-autonomy-com2023011620063473637 NetworkUnavailable False Tue, 19 Nov 2024 14:11:58 +0800 Tue, 19 Nov 2024 14:11:58 +0800 RouteCreated RouteController created a route com2023011620063473637 serving-2024111117494129039-7499b7d4f4-twmqc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 55d cpu 100m (0%) 32 (200%) |
|
通过api得到的任务报错信息为 |
|
清理下内部的pod，kubectl delete pod ‘ pod名’，然后任务重试下 |
|
您好，清理pod之后问题仍没有解决。目前的状态是假设有a，b，c三方，我在a上发起任务，a的状态是running，b和c的状态都是AwaitingApproval。调用approve接口去解决AwaitingApproval，返回的结果是success，但是状态没有改变。调用query接口查看a的报错信息仍是domain [com2023011620060497797,com2023011620072311738] can not reserve resources for pods |
|
而且a b c任意两方的任务是可以跑通的，不会出现等待验证，只有三方会出现这一状态。 |
|
三方的日志方便提供一下吗？ 【/home/kuscia/var/logs/envoy】kuscia流量出入口日志信息 |
|
k3s.log I0108 10:57:49.963055 27 pathrecorder.go:248] kube-aggregator: "/apis/kuscia.secretflow/v1alpha1/domains/com2023011620072311738" satisfied by prefix /apis/kuscia.secretflow/v1alpha1/ |
|
kuscia.log |
|
可以提供一下你任务发起方的路由配置吗？ |
|
这个11739不在abc三个之中，是一个已经停掉的节点。 |
|
2025-01-08 14:18:16.315 INFO queue/queue.go:124 Finish processing item: queue id[domain-controller], key[com2023011620072311738] (102.389µs) |
|
我仔细查看了协作方的日志，发现报错的原因是找不到要执行的task。两个协作方都是同样的原因。 |
|
2025-01-08 14:36:40.394 INFO queue/queue.go:124 Finish processing item: queue id[taskresourcegroup-controller], key[jianew-job-psi] (380.575µs) |
|
然后这是发起方的日志，一直在等待协作方保留pod。 |
|
|
原因找到了，是kuscia版本的问题。从0.8 升级到0.13就可以了，看来低版本跑多方任务存在问题。 |
Issue Type
Api Usage
Search for existing issues similar to yours
Yes
Kuscia Version
0.8.0b0
Link to Relevant Documentation
No response
Question Details
您好，我们在原有psi组件的基础上做了一个多方psi组件。在中心化组网模式下是可以成功运行的，但是在p2p模式下失败。报错信息是： {"status":{"code":0,"message":"success","details":[]},"data":{"job_id":"multi-psi","initiator":"com2023011620063473637","max_parallelism":4,"tasks":[{"app_image":"secretflow-image","parties":[{"domain_id":"com2023011620063473637","role":""},{"domain_id":"com2023011620072311738","role":""}],"alias":"job-psi","task_id":"job-psi","dependencies":[],"task_input_config":"{\"sf_datasource_config\":{\"com2023011620063473637\":{\"id\":\"default-data-source\"},\"com2023011620072311738\":{\"id\":\"default-data-source\"}},\"sf_cluster_desc\":{\"parties\":[\"com2023011620063473637\",\"com2023011620072311738\"],\"devices\":[{\"name\":\"spu\",\"type\":\"spu\",\"parties\":[\"com2023011620063473637\",\"com2023011620072311738\"],\"config\":\"{\\\"runtime_config\\\":{\\\"protocol\\\":\\\"REF2K\\\",\\\"field\\\":\\\"FM64\\\"},\\\"party_address\\\":{\\\"com2023011620063473637\\\":\\\"172.16.16.115\\\",\\\"com2023011620072311738\\\":\\\"172.16.16.120\\\"},\\\"link_desc\\\":{\\\"connect_retry_times\\\":60,\\\"connect_retry_interval_ms\\\":1000,\\\"brpc_channel_protocol\\\":\\\"http\\\",\\\"brpc_channel_connection_type\\\":\\\"pooled\\\",\\\"recv_timeout_ms\\\":1200000,\\\"http_timeout_ms\\\":1200000}}\"},{\"name\":\"heu\",\"type\":\"heu\",\"parties\":[\"com2023011620063473637\",\"com2023011620072311738\"],\"config\":\"{\\\"mode\\\": \\\"PHEU\\\", \\\"schema\\\": \\\"paillier\\\", \\\"key_size\\\": 2048}\"}],\"ray_fed_config\":{\"cross_silo_comm_backend\":\"grpc\"}},\"sf_node_eval_param\":{\"domain\":\"data_prep\",\"name\":\"multi_psi\",\"version\":\"1.0.0\",\"attr_paths\":[\"protocol\",\"sort_result\",\"allow_duplicate_keys\",\"allow_duplicate_keys/yes/join_type\",\"allow_duplicate_keys/yes/join_type/left_join/left_side\",\"multiparty_info\"],\"attrs\":[{\"s\":\"PROTOCOL_RR22\"},{\"b\":true},{\"s\":\"no\"},{\"s\":\"left_join\"},{\"ss\":[\"com2023011620063473637\"]}, {\"s\":\"{\\\"party_nums\\\":2,\\\"data_refs\\\":[{\\\"uri\\\":{\\\"datasource_kind\\\":5,\\\"server_file_path\\\":\\\"/home/kuscia/var/storage/data/part1.csv\\\"},\\\"party\\\":\\\"com2023011620063473637\\\",\\\"format\\\":\\\"csv\\\",\\\"keys\\\":[\\\"id\\\"],\\\"key_types\\\": [\\\"str\\\"],\\\"features\\\": [\\\"area error\\\",\\\"compactness error\\\",\\\"compactness error\\\",\\\"concavity error\\\",\\\"fractal dimension error\\\",\\\"mean area\\\",\\\"mean compactness\\\",\\\"mean concave points\\\",\\\"mean concavity\\\",\\\"mean fractal dimension\\\",\\\"target\\\"],\\\"feature_types\\\": [\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\"]},{\\\"uri\\\":{\\\"datasource_kind\\\":5,\\\"server_file_path\\\":\\\"/home/kuscia/var/storage/data/part3.csv\\\"},\\\"party\\\":\\\"com2023011620060497797\\\",\\\"format\\\":\\\"csv\\\",\\\"keys\\\":[\\\"id\\\"],\\\"key_types\\\": [\\\"str\\\"],\\\"features\\\": [\\\"worst area\\\",\\\"worst compactness\\\",\\\"worst concave points\\\",\\\"worst concavity\\\",\\\"worst fractal dimension\\\",\\\"worst perimeter\\\",\\\"worst radius\\\",\\\"worst smoothness\\\",\\\"worst symmetry\\\",\\\"worst texture\\\"],\\\"feature_types\\\": [\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\",\\\"float\\\"]}]}\"}]},\"sf_output_ids\":[\"psi-output\"],\"sf_output_uris\":[\"psi-output.csv\"]}","priority":100}],"status":{"state":"Running","err_msg":"","create_time":"2025-01-06T02:29:02Z","start_time":"2025-01-06T02:29:02Z","end_time":"","tasks":[{"task_id":"job-psi","state":"Pending","err_msg":"","create_time":"2025-01-06T02:29:02Z","start_time":"2025-01-06T02:29:02Z","end_time":"","parties":[{"domain_id":"com2023011620072311738","state":"Pending","err_msg":"0/1 nodes are available: task resource com2023011620072311738/job-psi-46436ddb2a1e status phase is Failed, skip scheduling pod. last failed scheduling result: . preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling., reject the pod job-psi-0 even after PostFilter.;","endpoints":[{"port_name":"fed","scope":"Cluster","endpoint":"job-psi-0-fed.com2023011620072311738.svc"},{"port_name":"global","scope":"Domain","endpoint":"job-psi-0-global.com2023011620072311738.svc:23311"},{"port_name":"spu","scope":"Cluster","endpoint":"job-psi-0-spu.com2023011620072311738.svc"}]}],"alias":"job-psi"}],"stage_status_list":[{"domain_id":"com2023011620063473637","state":"JobCreateStageSucceeded"},{"domain_id":"com2023011620072311738","state":"JobCreateStageSucceeded"}],"approve_status_list":[{"domain_id":"com2023011620072311738","state":"JobAccepted"},{"domain_id":"com2023011620063473637","state":"JobAccepted"}]},"custom_fields":{}}} 请问这是什么原因导致的呢？感谢帮助。The text was updated successfully, but these errors were encountered: