Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with KRaft Configuration: ZooKeeper Settings Appear in server.properties #1765

Open
masoudtayefeh opened this issue Aug 19, 2024 · 1 comment

Comments

@masoudtayefeh
Copy link

masoudtayefeh commented Aug 19, 2024

I'm encountering an issue with configuring Kafka in KRaft mode using the Ansible role for Confluent Platform. Despite specifying KRaft mode in my hosts.yml file, the generated server.properties file still includes ZooKeeper-related settings.

Debian Version: 12.6
cp-ansible: 7.7.0-post

hosts.yml:

kafka_controller:
  hosts:
    kafka-1:
    kafka-2:
    kafka-3:

kafka_broker:
  hosts:
    kafka-1:
    kafka-2:
    kafka-3:

schema_registry:
  hosts:
    kafka-miscellaneous:

kafka_rest:
  hosts:
    kafka-3:

kafka_connect:
  hosts:
    kafka-miscellaneous:

all:
  vars:
    ansible_connection: ssh
    ansible_user: kafka
    ansible_become: true
    ansible_ssh_private_key_file: /home/kafka/.ssh/id_rsa
    jmxexporter_enabled: true
    jolokia_enabled: true
    proxy_env:
       http_proxy: http://proxy:8118
       https_proxy: http://proxy:8118
    kafka_broker_custom_properties:
       num.io.threads: 15

After running the Ansible playbook with the above configuration, the server.properties file on Kafka brokers still contains ZooKeeper-related configurations.

Error that i get after running ansible:

TASK [confluent.platform.kafka_controller : Check Kafka Metadata Quorum]
fatal: [kafka-1]: FAILED! => {"changed": false, "cmd": "/usr/bin/kafka-metadata-quorum --bootstrap-server kafka-1:9091  --command-config /etc/controller/client.properties describe --replication\n", "delta": "0:01:01.149704", "end": "2024-08-19 10:20:01.372824", "msg": "non-zero return code", "rc": 1, "start": "2024-08-19 10:19:00.223120", "stderr": "[2024-08-19 10:19:01,068] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:01,172] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:01,274] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:01,576] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:19:02,079] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\n[2024-08-19 10:20:00,070] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)\norg.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum\njava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)\n\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)\n\tat org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.handleDescribeReplication(MetadataQuorumCommand.java:158)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:106)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:62)\n\tat org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:57)\nCaused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum", "stderr_lines": ["[2024-08-19 10:19:01,068] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-1/172.16.104.41:9091) could not be established. Node may not be available. ((org.apache.kafka.clients.NetworkClient)", "org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum", "java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum", "\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)", "\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)", "\tat org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.handleDescribeReplication(MetadataQuorumCommand.java:158)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:106)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:62)", "\tat org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:57)", "Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeMetadataQuorum"], "stdout": "", "stdout_lines": []}

And here is confluent-server service logs:

Aug 05 17:28:34 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:34 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:35,693] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:35,694] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:35 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:36,795] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:36,797] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:36 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:37,898] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:37,899] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:37 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:39,000] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:39,002] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:39 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:40,103] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:40,105] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:40 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:41,206] INFO Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:41,207] WARN Session 0x0 for server localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:41 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:42,308] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:42,309] WARN Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: java.net.ConnectException: Connection refused
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnect(Native Method)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]:         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)
Aug 05 17:28:42 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:42,662] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,410] INFO Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181. (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,514] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,515] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,516] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,518] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:216)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:116)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2853)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2959)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:1065)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.startup(KafkaServer.scala:355)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka$.main(Kafka.scala:112)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka.main(Kafka.scala)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,519] INFO shutting down (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,537] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,538] INFO shut down completed (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,538] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:216)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:116)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2853)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2959)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:1065)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.server.KafkaServer.startup(KafkaServer.scala:355)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka$.main(Kafka.scala:112)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]:         at kafka.Kafka.main(Kafka.scala)
Aug 05 17:28:43 kafka-1 kafka-server-start[105343]: [2024-08-05 17:28:43,539] INFO shutting down (kafka.server.KafkaServer)
Aug 05 17:28:43 kafka-1 systemd[1]: confluent-server.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 17:28:43 kafka-1 systemd[1]: confluent-server.service: Failed with result 'exit-code'.
Aug 05 17:28:43 kafka-1 systemd[1]: confluent-server.service: Consumed 3.602s CPU time.
@julnow
Copy link

julnow commented Sep 26, 2024

@masoudtayefeh did you manage to solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants