Name node unable to obtain password from user after enabling kerberos

[whiskerch]

Hi,

I am running the hdfs-k8s chart with the following command:

helm install -n my-hdfs charts/hdfs-k8s \
    --set global.kerberosEnabled=true  \
    --set global.kerberosRealm=EXAMPLE.COM  \
    --set tags.kerberos=true

and then follow all the steps in the Readme to enable kerberos on the cluster.

After running the last command in the instructions ($ $_SECRET_CMD) I check the nodes, but can see that the name nodes are unable to come up.

They are showing the following errors in the logs:

19/05/28 09:28:41 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
19/05/28 09:28:41 INFO namenode.NameNode: createNameNode [-bootstrapStandby, -nonInteractive]
19/05/28 09:28:42 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Login failure for hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM from keytab /etc/security/hdfs.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:246)
	at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:107)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:420)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1454)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

	at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
	... 7 more
19/05/28 09:28:42 INFO util.ExitUtil: Exiting with status 1
19/05/28 09:28:42 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-172-20-38-253.eu-west-2.compute.internal/127.0.1.1
************************************************************/
+ rm -rf /hadoop/dfs/name/current
+ exit 1

As far as I can see when setting up the keytab, everything executes as expected. This is the output from the

tar: removing leading '/' from member names
Authenticating as principal root/admin@EXAMPLE.COM with password.
Principal "HTTP/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM" created.
WARNING: no policy specified for HTTP/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM; defaulting to no policy
WARNING: no policy specified for hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM; defaulting to no policy
Authenticating as principal root/admin@EXAMPLE.COM with password.
Principal "hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM" created.
Authenticating as principal root/admin@EXAMPLE.COM with password.
Entry for principal hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local.keytab.
Entry for principal hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local.keytab.
Entry for principal HTTP/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local.keytab.
Entry for principal HTTP/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local.keytab.

I am using kubernetes version 1.11.9, and running the commands from a max running osx 10.14

[shockdm]

I've ran into the same issue. The workaround that helped was to use hostname based keytab instead of the service based keytab on the namenode. The error that you're getting is basically saying that the keytab you are using does not contain the credentials for user hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM, because since hostNetwork is enabled by default - and you are using the keytab that has hdfs/<your_node_hostname>@EXAMPLE.COM. What I've tried - is adjusting the yaml to mount keytab containing hdfs/my-hdfs-namenode-1.my-hdfs-namenode.hdfs.svc.cluster.local@EXAMPLE.COM. But that presents another problem. When zkfc starts up - it tries to connect to the name node, and it uses physical hostname. Since we got the wrong keytab for that - it fails, rendering the HA cluster useless. I've yet to find a solution to that problem.

[whiskerch]

I think I ended up using a similar workaround - I set the value hostNetworkEnabled to false in order to get the keytab for the namenodes to be chosen by the init container.

[shockdm]

@whiskerch Right, but that will cost you the data locality, since datanodes will identify themselves using internal addresses.