Bulk Merge v1.5 WebConsole Changes. See #362 (#386)

* Add job/update functionality

* fix

* add migration

* fix

* fix

* fix

* fix

* fix

* fix

* add heartbeat webconsole

* fix

* trigger

* Add migration create table

* Add client side job validation

* Fix create/update job

* fix

* Feat/datasource (#328)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* Feat/datasource (#330)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* ds bug fix (#331)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* change delete job logic

* feat: use full params to set default (#338)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* Fix form defaults

* Fix client_params & server_param inversion

* feat: config replica type (#339)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

* feat: select datasource in training job

* feat: config replica type:
* feat update form value when update

* fix: hide env value in training

* Update form-default.js

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* add replica workers (#340)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

* feat: select datasource in training job

* feat: config replica type:
* feat update form value when update

* fix: hide env value in training

* Update form-default.js

* fix: replica num

* feat: add replica workers

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* bug fix (#346)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

* feat: select datasource in training job

* feat: config replica type:
* feat update form value when update

* fix: hide env value in training

* Update form-default.js

* fix: replica num

* feat: add replica workers

* fix: data source and worker nums fill

* fix: replicas is null

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* fix pod-log pod-shell (#351)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

* feat: select datasource in training job

* feat: config replica type:
* feat update form value when update

* fix: hide env value in training

* Update form-default.js

* fix: replica num

* feat: add replica workers

* fix: data source and worker nums fill

* fix: replicas is null

* fix: pod-log pod-shell

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* Update server.js

* remove replicas from data join jobs default

* move DataSource select to ticket (#354)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

* feat: select datasource in training job

* feat: config replica type:
* feat update form value when update

* fix: hide env value in training

* Update form-default.js

* fix: replica num

* feat: add replica workers

* fix: data source and worker nums fill

* fix: replicas is null

* fix: pod-log pod-shell

* fix: move datasource select from job to ticket

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* feat: federation config (#356)

* feat(web_console): form json

* chore(web_console): make form fields immutable
* mask sensitive data

* fix: lock version

* feat(web_console): support popover on nav tab

* feat(WIP): datasource

* feat(web_console): use form meta to implement form convert

* chore(web_console): update doc

* feat(web_console): raw data form:
* form utils unit test
* fix logic of `fillJSON`

* fix: raw data submit:
* fix federation value validator

* fix: image should be written into context

* fix: async response on submit

* fix: ticket not clone job

* chore: use `useMemo` with form fields

* feat(web_console): rewrite `fiilJSON`:
* support two array mode

* feat(WIP): datasource job form
* form item `name-value` support ignore keys
* incremental update form value in group type change
* TODO: form submit

* feat(web_console): boolean select in form

* feat(WIP): datasource ticket
* fix header active tab

* feat(web_console): datasource and trainning

* fix(web_console): ticket
* update form
* create ticket
* job link
* add `filterArrayValue`

* fix(web_console): clone job

* feat(web_console): stop and start job

* fix: update job status

* feat: job federation id filter

* fix: job status

* fix: package-lock

* fix: prerender error

* fix: prerender error

* fix: boolean select;
* update router of charts

* fix: bool-select

* feat: header dropdown icon

* feat: federation heartbeat

* fix: clone job

* fix: heartbeat

* fix: server ticket select

* fix: raw data in clone

* fix: delete job rpc

* fix: clear formMeta on cancel

* chore: remove raw_data in job form

* feat: show total info of server ticket:
* fix create jon derect in the index
* fix server ticket pull in cloning

* fix: useEffect

* fix: server ticket info

* chore: remove mock data

* chore: fix typo

* feat: disable federation with filter

* fix: raw data fill:
* fix raw data appState

* fix: num_partitions

* fix: raw_data fill:
* TODO: figure out why not working in some conditions

* feat: use full params to set default:
* change default value when  modify job type

* feat: select datasource in training job

* feat: config replica type:
* feat update form value when update

* fix: hide env value in training

* Update form-default.js

* fix: replica num

* feat: add replica workers

* fix: data source and worker nums fill

* fix: replicas is null

* fix: pod-log pod-shell

* fix: move datasource select from job to ticket

* feat: federation config:
* fix init value of boolean

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* Add k8s settings default

* Add datasource_meta migrations

* chore: remove useless default

* Pass DB env args to pod

* Add new psi data join entry script

* update psi data join default json

* Fix bug in webconsole create job

* fix: env not update (#364)

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* Allow injecting form default from env var

* Add +x for scripts

* Turn off cmd display

* fix: v1.5 bugs (#376)

* fix: v1.5 bugs
* ticket raw data name
* form data not updated when editing ticket
* federation form checker

* chore: remove console.log

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* Fix form-default

* fix: job can not submit (#378)

* fix: job can not submit

* Update CommonJobList.jsx

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* Fix delete job error

* change Role assignment for PSI data join

* feat: code key (#381)

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* fix:  edit code key (#382)

* feat: code key

* fix: edit code key

* feat: search param data

* Update CommonTicket.jsx

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: Eric Junyuan Xie <piiswrong@users.noreply.github.com>

* fix: search exit condition (#383)

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* feat: clone ticket and rawdata (#384)

Co-authored-by: shenshihao <shenshihao.20@bytedance.com>

* build DATA_PATH from DATA_SOURCE for tree worker

* Full integrated test (#385)

* add csv data

* fix file_type for tree model

* add field 'label' to tfrecord_raw_data

* fix num_parallel

* add num_parallel for compatibility.

* Revert and add new fake feature in tfrecord raw data for tree model.
This reverts commit 8ca6061.

* modified tree trainer for debug purpose, will be reverted

* Revert debug, add send metrics in tree trainer, shrink raw data, full integrated test supported

This reverts commit 977cd45.

* add loss_type for run_tree_worker.sh

* eliminated data path

* add num_parallel for compatibility.

* Revert and add new fake feature in tfrecord raw data for tree model.
This reverts commit 8ca6061.

* modified tree trainer for debug purpose, will be reverted

* Revert debug, add send metrics in tree trainer, shrink raw data, full integrated test supported

This reverts commit 977cd45.

* rebased and modified defaults

* deleted csv data

Co-authored-by: zhangzihui <zhangzihui@bytedance.com>

Co-authored-by: Allen Shen <932142511@qq.com>
Co-authored-by: shenshihao <shenshihao.20@bytedance.com>
Co-authored-by: ADeltanp <45765741+ZhZhang711@users.noreply.github.com>
Co-authored-by: zhangzihui <zhangzihui@bytedance.com>

.gitignore

@@ -93,3 +93,4 @@ config.mk
 server.config.js
 es.match_phrase.js
 es.match_phrase.json
+web_console/config

deploy/charts/fedlearner/charts/fedlearner-web-console/templates/deployment.yaml

@@ -34,9 +34,9 @@ spec:
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           command:
-            - node
+            - sh
           args:
-            - bootstrap.js
+            - startup.sh
           env:
             - name: NODE_ENV
               value: production

deploy/charts/fedlearner/charts/fedlearner-web-console/values.yaml

@@ -80,9 +80,11 @@ cluster:
     DB_PASSWORD: fedlearner
     DB_HOST: fedlearner-stack-mariadb
     DB_PORT: 3306
-    DB_SYNC: true
+    DB_SYNC: false
     GRPC_AUTHORITY: ""
     KIBANA_HOST: fedlearner-stack-kibana
     KIBANA_PORT: 443
     ES_HOST: fedlearner-stack-elasticsearch-client
     ES_PORT: 9200
+    ETCD_ADDR: "fedlearner-stack-etcd.default.svc.cluster.local:2379"
+    KVSTORE_TYPE: "mysql"

deploy/integrated_test/client_integrated_test.py

@@ -2,7 +2,8 @@
 import argparse
 import json
 import requests
-from tools import login, request_and_response, build_raw_data, build_data_join_ticket, build_train_ticket
+from tools import login, request_and_response, build_raw_data, \
+    build_data_join_ticket, build_nn_ticket, build_tree_ticket
 
 
 def build_federation_json(args):
@@ -18,17 +19,61 @@ def build_federation_json(args):
 
 if __name__ == '__main__':
     parser = argparse.ArgumentParser()
-    parser.add_argument('--name', type=str)
-    parser.add_argument('--x-federation', type=str)
-    parser.add_argument('--image', type=str)
-    parser.add_argument('--url', type=str)
-    parser.add_argument('--username', type=str)
-    parser.add_argument('--password', type=str)
-    parser.add_argument('--api-version')
+    parser.add_argument('--name',
+                        type=str,
+                        help='Name for peer federation.')
+    parser.add_argument('--x-federation',
+                        type=str,
+                        help='Name for local federation.')
+    parser.add_argument('--image',
+                        type=str,
+                        help='Image address.')
+    parser.add_argument('--data-portal-type',
+                        type=str,
+                        help='Type of raw data, Streaming(default) or PSI.',
+                        choices=['Streaming', 'PSI'],
+                        default='Streaming')
+    parser.add_argument('--model-type',
+                        type=str,
+                        help='Type of train model, (nn model) or tree model.',
+                        choices=['nn_model', 'tree_model'],
+                        default='nn_model')
+    parser.add_argument('--rsa-key-path',
+                        type=str,
+                        help='Path to RSA public key.')
+    parser.add_argument('--rsa-key-pem',
+                        type=str,
+                        help='Either rsa key path or rsa key pem must be given.')
+    parser.add_argument('--url',
+                        type=str,
+                        help='URL to webconsole.',
+                        default='127.0.0.1:1989')
+    parser.add_argument('--username',
+                        type=str,
+                        help='Username of webconsole.',
+                        default='ada')
+    parser.add_argument('--password',
+                        type=str,
+                        help='Password of webconsole.',
+                        default='ada')
+    parser.add_argument('--api-version',
+                        help='API version of webconsole.',
+                        default=1)
     args = parser.parse_args()
+    args.streaming = args.data_portal_type == 'Streaming'
+    if not args.streaming:
+        args.cmd_args = {'Master': ["/app/deploy/scripts/rsa_psi/run_psi_data_join_master.sh"],
+                         'Worker': ["/app/deploy/scripts/rsa_psi/run_psi_data_join_worker.sh"]}
+        if args.rsa_key_pem is not None:
+            args.psi_extras = [{"name": "RSA_KEY_PEM", "value": args.rsa_key_pem}]
+        elif args.rsa_key_path is not None:
+            args.psi_extras = [{"name": "RSA_KEY_PATH", "value": args.rsa_key_path}]
+        else:
+            raise Exception('Either RSA_KEY_PEN or RSA_KEY_PATH must be provided when using PSI.')
+        args.psi_extras.append({"name": "SIGN_RPC_TIMEOUT_MS", "value": "128000"})
+
     args.url = args.url.strip().rstrip('/') + '/api/v' + str(args.api_version)
     cookie = login(args)
-
     federation_json, suffix = build_federation_json(args)
     federation_id, federation_name = request_and_response(args=args,
                                                           url=args.url + '/federations',
@@ -45,15 +90,22 @@ def build_federation_json(args):
     requests.post(url=args.url + '/raw_data/' + str(raw_data_id) + '/submit', cookies=cookie)
 
     join_ticket_json, suffix = build_data_join_ticket(args, federation_id, raw_data_name,
-                                                      'template_json/template_join_ticket.json', 'Leader')
+                                                      'template_json/template_streaming_join_ticket.json'
+                                                      if args.streaming
+                                                      else 'template_json/template_psi_join_ticket.json',
+                                                      'Leader' if args.streaming else 'Follower')
     join_ticket_id, join_ticket_name = request_and_response(args=args,
                                                             url=args.url + '/tickets',
                                                             json_data=join_ticket_json,
                                                             cookies=cookie,
                                                             name_suffix=suffix)
 
-    train_ticket_json, suffix = build_train_ticket(args, federation_id,
-                                                   'template_json/template_train_ticket.json', 'Follower')
+    if args.model_type == 'nn_model':
+        train_ticket_json, suffix = build_nn_ticket(args, federation_id,
+                                                    'template_json/template_nn_ticket.json', 'Follower')
+    else:
+        train_ticket_json, suffix = build_tree_ticket(args, federation_id,
+                                                      'template_json/template_tree_ticket.json', 'Follower')
     train_ticket_id, train_ticket_name = request_and_response(args=args,
                                                               url=args.url + '/tickets',
                                                               json_data=train_ticket_json,

deploy/integrated_test/template_json/template_psi_join_ticket.json

@@ -0,0 +1,178 @@
+{
+  "public_params": {
+    "spec": {
+      "flReplicaSpecs": {
+        "Master": {
+          "pair": true,
+          "replicas": 1,
+          "template": {
+            "spec": {
+              "containers": [
+                {
+                  "env": [
+                    {
+                      "name": "PARTITION_NUM",
+                      "value": "4"
+                    },
+                    {
+                      "name": "START_TIME",
+                      "value": "0"
+                    },
+                    {
+                      "name": "END_TIME",
+                      "value": "999999999999"
+                    },
+                    {
+                      "name": "NEGATIVE_SAMPLING_RATE",
+                      "value": "1.0"
+                    },
+                    {
+                      "name": "RAW_DATA_SUB_DIR",
+                      "value": "portal_publish_dir/"
+                    }
+                  ],
+                  "image": "!image",
+                  "ports": [
+                    {
+                      "containerPort": 50051,
+                      "name": "flapp-port"
+                    }
+                  ],
+                  "command": [
+                    "/app/deploy/scripts/wait4pair_wrapper.sh"
+                  ],
+                  "args": []
+                }
+              ]
+            }
+          }
+        },
+        "Worker": {
+          "pair": true,
+          "replicas": 4,
+          "template": {
+            "spec": {
+              "containers": [
+                {
+                  "env": [
+                    {
+                      "name": "PARTITION_NUM",
+                      "value": "4"
+                    },
+                    {
+                      "name": "RAW_DATA_SUB_DIR",
+                      "value": "portal_publish_dir/"
+                    },
+                    {
+                      "name": "DATA_BLOCK_DUMP_INTERVAL",
+                      "value": "600"
+                    },
+                    {
+                      "name": "DATA_BLOCK_DUMP_THRESHOLD",
+                      "value": "65536"
+                    },
+                    {
+                      "name": "EXAMPLE_ID_DUMP_INTERVAL",
+                      "value": "600"
+                    },
+                    {
+                      "name": "EXAMPLE_ID_DUMP_THRESHOLD",
+                      "value": "65536"
+                    },
+                    {
+                      "name": "PSI_RAW_DATA_ITER",
+                      "value": "TF_RECORD"
+                    },
+                    {
+                      "name": "PSI_OUTPUT_BUILDER",
+                      "value": "TF_RECORD"
+                    },
+                    {
+                      "name": "DATA_BLOCK_BUILDER",
+                      "value": "TF_RECORD"
+                    },
+                    {
+                      "name": "EXAMPLE_JOINER",
+                      "value": "SORT_RUN_JOINER"
+                    },
+                    {
+                      "name": "SIGN_RPC_TIMEOUT_MS",
+                      "value": "128000"
+                    }
+                  ],
+                  "image": "!image",
+                  "ports": [
+                    {
+                      "containerPort": 50051,
+                      "name": "flapp-port"
+                    }
+                  ],
+                  "command": [
+                    "/app/deploy/scripts/wait4pair_wrapper.sh"
+                  ],
+                  "args": []
+                }
+              ]
+            }
+          }
+        }
+      }
+    }
+  },
+  "private_params": {
+    "spec": {
+      "flReplicaSpecs": {
+        "Master": {
+          "template": {
+            "spec": {
+              "containers": [
+                {
+                  "image": "!image",
+                  "env": [
+                    {
+                      "name": "RAW_DATA_SUB_DIR",
+                      "value": "portal_publish_dir/"
+                    },
+                    {
+                      "name": "PARTITION_NUM",
+                      "value": "4"
+                    }
+                  ]
+                }
+              ]
+            }
+          },
+          "replicas": 1
+        },
+        "Worker": {
+          "template": {
+            "spec": {
+              "containers": [
+                {
+                  "image": "!image",
+                  "env": [
+                    {
+                      "name": "RAW_DATA_SUB_DIR",
+                      "value": "portal_publish_dir/"
+                    },
+                    {
+                      "name": "PARTITION_NUM",
+                      "value": "4"
+                    }
+                  ]
+                }
+              ]
+            }
+          },
+          "replicas": 4
+        }
+      }
+    }
+  },
+  "name": "!name",
+  "federation_id": "!federation_id",
+  "job_type": "psi_data_join",
+  "role": "!role",
+  "expire_time": "!expire_time",
+  "remark": "Built by integrated test."
+}

deploy/integrated_test/template_json/template_raw_data.json

@@ -2,7 +2,7 @@
   "name": "!name",
   "federation_id": "!federation_id",
   "output_partition_num": 4,
-  "data_portal_type": "Streaming",
+  "data_portal_type": "!data_portal_type",
   "input": "/app/deploy/integrated_test/tfrecord_raw_data",
   "image": "!image",
   "context": {

deploy/integrated_test/template_json/template_join_ticket.json → deploy/integrated_test/template_json/template_streaming_join_ticket.json

@@ -61,21 +61,29 @@
               "containers": [
                 {
                   "env": [
+                    {
+                      "name": "PARTITION_NUM",
+                      "value": "4"
+                    },
+                    {
+                      "name": "RAW_DATA_SUB_DIR",
+                      "value": "portal_publish_dir/"
+                    },
                     {
                       "name": "DATA_BLOCK_DUMP_INTERVAL",
                       "value": "600"
                     },
                     {
                       "name": "DATA_BLOCK_DUMP_THRESHOLD",
-                      "value": "262144"
+                      "value": "65536"
                     },
                     {
                       "name": "EXAMPLE_ID_DUMP_INTERVAL",
                       "value": "600"
                     },
                     {
                       "name": "EXAMPLE_ID_DUMP_THRESHOLD",
-                      "value": "262144"
+                      "value": "65536"
                     },
                     {
                       "name": "EXAMPLE_ID_BATCH_SIZE",
@@ -96,14 +104,6 @@
                     {
                       "name": "RAW_DATA_ITER",
                       "value": "TF_RECORD"
-                    },
-                    {
-                      "name": "RAW_DATA_SUB_DIR",
-                      "value": "portal_publish_dir/"
-                    },
-                    {
-                      "name": "PARTITION_NUM",
-                      "value": "4"
                     }
                   ],
                   "image": "!image",
@@ -182,5 +182,5 @@
   "job_type": "data_join",
   "role": "!role",
   "expire_time": "!expire_time",
-  "remark": "Build by integrated test."
+  "remark": "Built by integrated test."
 }