Kubeflow istio init issue

[vamshikrishna9999]

Bug Description
I was trying to install kubeflow version v1.5.0 on OpenShift cluster. I have deployed all the components of Kubeflow then pods should be in running state so some're up and regarding istio init When istio starts it fails with
iptables-restore --noflush /tmp/iptables-rules-1670170358102470740.txt3948297396
error Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'
Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2022-12-04T16:12:38.103748Z error Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1670170358102470740.txt3948297396, exit status 2

Version
kubeflow version: v1.5.0
Kubernetes platform: OpenShift
version:4.10
Additional Information
istio-init startup log:

2022-12-04T16:17:48.125777Z info Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=

2022-12-04T16:17:48.125806Z info Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
OUTPUT_PATH=
NETWORK_NAMESPACE=
CNI_MODE=false
HOST_NSENTER_EXEC=false
EXCLUDE_INTERFACES=

2022-12-04T16:17:48.125937Z info Writing following contents to rules file: /tmp/iptables-rules-1670170668125834078.txt958303642

nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2022-12-04T16:17:48.125961Z info Running command: iptables-restore --noflush /tmp/iptables-rules-1670170668125834078.txt958303642
2022-12-04T16:17:48.127206Z error Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'
Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2022-12-04T16:17:48.127224Z error Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1670170668125834078.txt958303642, exit status 2

[RakeshRaj97]

Any updates on this issue?

[RakeshRaj97]

What is the operating system are you using? I had this problem on RHEL 9. I fixed it by installing istio-cni daemonset before installing Kubeflow. Set cni to be true in istio profile.yaml

istioctl install -f profile.yaml
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

My environment:
K8s: v1.24.7
OS: RHEL 9

Moreover, I am not installing KNative and KServe -- commented out those lines in kustomization.yaml

[vamshikrishna9999]

Hi @RakeshRaj97

Finally All pods are in running state but i am unable get the dashboard while accessing from istio-ingressgateway route and i also tried with port-forwarding to access in locally but not worked.

[RakeshRaj97]

Can you share some logs please?

[juliusvonkohout]

Please take a look at #2455

/close

There has been no activity for a long time. Please reopen if necessary.

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

Please take a look at #2455

/close

There has been no activity for a long time. Please reopen if necessary.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mpaulgreen]

The issue persist #2455 on Openshift 4.12.x and 4.13.x. Please reopen the issue.

/open

[juliusvonkohout]

/reopen

, but this is probably misconfiguration on your end. I know for myself that Kubeflow works on Openshift, with istio and istio-cni. If help is needed consulting is available.

[google-oss-prow]

@juliusvonkohout: Reopened this issue.

In response to this:

/reopen

, but this is probably misconfiguration on your end. I know for myself that Kubeflow works on Openshift, with istio and istio-cni. If help is needed consulting is available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[juliusvonkohout]

/close

since #2455 is merged and redhat is using it quite intensively nowaydays on openshift.

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

/close

since https://github.com/kubeflow/manifests/pull/2455is merged

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.