feat(keycloak): add seeding data to configuration #800
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ntruchsess
changed the title
ntruchsess
force-pushed
the
feature/438-keycloak-seeding
branch
from
aa5807f
to
b11b589
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
ntruchsess
force-pushed
the
feature/438-keycloak-seeding
branch
from
b11b589
to
a853d02
Compare
evegufy
requested changes
Jun 24, 2024
There was a problem hiding this comment.
Hi @ntruchsess I just tried to import this realm from scratch (not only at upgrade / updating existing realm configuration) and I'm encountering the following error:
{"Timestamp":"2024-06-24T16:12:24.1533063+00:00","Level":"Information","MessageTemplate":"Start seeding","RenderedMessage":"Start seeding"}
{"Timestamp":"2024-06-24T16:12:25.0499479+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 404 (Not Found): GET http://centralidp-keycloak/auth/admin/realms/CX-Central\n at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Not Found: http://centralidp-keycloak/auth/admin/realms/CX-Central","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7938875+00:00","Level":"Error","MessageTemplate":"{Message}","RenderedMessage":"\"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\"","Exception":"Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)","Properties":{"Message":"Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e","SourceContext":"Program"}}
{"Timestamp":"2024-06-24T16:12:27.7992279+00:00","Level":"Fatal","MessageTemplate":"Unhandled exception","RenderedMessage":"Unhandled exception","Exception":"System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n ---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e\n at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n --- End of inner exception stack trace ---\n at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51\n at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)\n at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)\n at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)\n at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54\n at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100\n at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71\n at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58\n at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86"}
Unhandled exception. System.ArgumentException: Bad Request: http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
---> Flurl.Http.FlurlHttpException: Call failed with status code 400 (Bad Request): DELETE http://centralidp-keycloak/auth/admin/realms/CX-Central/roles-by-id/64dd11f1-0d9a-4da3-a475-ca3fcbc8f90e
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
--- End of inner exception stack trace ---
at Org.Eclipse.TractusX.Portal.Backend.Keycloak.ErrorHandling.FlurlErrorHandler.<>c__DisplayClass0_0.<ConfigureErrorHandler>b__1(FlurlCall call) in /src/keycloak/Keycloak.ErrorHandling/FlurlErrorHandler.cs:line 51
at Flurl.Http.FlurlRequest.RaiseEventAsync(Action`1 syncHandler, Func`2 asyncHandler, FlurlCall call)
at Flurl.Http.FlurlRequest.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
at Flurl.Http.FlurlRequest.SendAsync(HttpMethod verb, HttpContent content, CancellationToken cancellationToken, HttpCompletionOption completionOption)
at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Library.KeycloakClient.DeleteRoleByIdAsync(String realm, String roleId, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Library/RolesById/KeycloakClient.cs:line 54
at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateAndDeleteRoles(KeycloakClient keycloak, String realm, IEnumerable`1 roles, IEnumerable`1 updateRoles, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 100
at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.RolesUpdater.UpdateRealmRoles(String keycloakInstanceName, CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/RolesUpdater.cs:line 71
at Org.Eclipse.TractusX.Portal.Backend.Keycloak.Seeding.BusinessLogic.KeycloakSeeder.Seed(CancellationToken cancellationToken) in /src/keycloak/Keycloak.Seeding/BusinessLogic/KeecloakSeeder.cs:line 58
at Program.<Main>$(String[] args) in /src/keycloak/Keycloak.Seeding/Program.cs:line 86
at Program.<Main>(String[] args)
The issue is not specific to your change as I tested it also with the previous state but it's relevant now because we want to use the job to setup entire realms not just updating existing ones.
evegufy
requested changes
Jun 24, 2024
I guess (at least one) role that keycloak creates by default is missing in the json. Whenever a realm is being created keycloak implicitly creates roles to manage administrative privileges of that realm. Those realm-management-roles cannot be deleted. What happens is that the seeding-job creates the realm, keycloak creates the management-roles, the seeding-job now queries the realm and calculates the delta to what is defined in json. Whenever a role is existing in keycloak that is not also defined in json the seeding-job tries to delete it. Unfortunally those realm-management-roles are not marked as such so the seeder cannot destinguish in between manually and implicitly created roles. Therefore for a successful import all implicitly created roles must also be defined in the json. |
ntruchsess
force-pushed
the
feature/438-keycloak-seeding
branch
from
7779e05
to
6937651
Compare
ntruchsess
force-pushed
the
feature/438-keycloak-seeding
branch
2 times, most recently
from
329439f
to
7340289
Compare
ntruchsess
force-pushed
the
feature/438-keycloak-seeding
branch
2 times, most recently
from
949a35c
to
43f1688
Compare
done |
As discussed, renaming the default role to the default pattern did solve the issue |
evegufy
requested changes
Jun 27, 2024
There was a problem hiding this comment.
I noticed that seeding of the (bpn) user attribute via seeding config fails (with error, see logs-from-centralidp-keycloak-realm-seeding-in-centralidp-keycloak-realm-seeding-1-5qw8s.log) for service account users:
used config:
- name: "KEYCLOAKSEEDING__REALMS__0__USERS__0__Attributes__0__NAME"
value: "bpn"
- name: KEYCLOAKSEEDING__REALMS__0__USERS__1__USERNAME
value: "service-account-sa-test-2"
- name: KEYCLOAKSEEDING__REALMS__0__USERS__1__Attributes__0__VALUES__0
value: TEST2
I tested with a normal user as well, there it worked.
evegufy
added a commit
to eclipse-tractusx/portal-iam
that referenced
this pull request
Jun 27, 2024
to enable import of realm via seeding job, without proper naming of the default role the seeding job runs into error eclipse-tractusx/portal-backend#800 (comment)
this is missing: |
evegufy
approved these changes
Jun 27, 2024
evegufy
requested changes
Jun 28, 2024
There was a problem hiding this comment.
Need to revert the approval as c5b4a10 is causing this error
logs-from-centralidp-keycloak-realm-seeding-in-centralidp-keycloak-realm-seeding-1-wvnxh.log
As discussed - this error was due to invalid configuration that was not compliant with the startup-configuration-checks. |
Phil91
previously approved these changes
Jul 18, 2024
|
Please wait for my approval before merge, I hope I get the final tests soon |
update framework version
ntruchsess
force-pushed
the
feature/438-keycloak-seeding
branch
from
5a67a80
to
91b082d
Compare
evegufy
added a commit
to eclipse-tractusx/portal-iam
that referenced
this pull request
Jul 29, 2024
to enable import of realm via seeding job, without proper naming of the default role the seeding job runs into error eclipse-tractusx/portal-backend#800 (comment)
evegufy
added a commit
to eclipse-tractusx/portal-iam
that referenced
this pull request
Jul 30, 2024
to enable import of realm via seeding job, without proper naming of the default role the seeding job runs into error eclipse-tractusx/portal-backend#800 (comment)
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The keycloak-seeder has been changed to (optionaly) read multiple json-files per realm and seeding-data from it's settings and merge this additional data with the one that is read from json before writing changes to keycloak. This allows to both separate seeding-data of different domains and override values in the keycloak-json by settings being passed as environment-variables (which can be defined via helm-chart).
The following client-properties can now be updated: RootUrl, BaseUrl, AdminUrl, RedirectUrls, Secret. (Before the change those values could only be initially set on creation of the respective entity)
KeecloakSeederSettings have been adjusted accordingly:
Why
seeding e.g. secrets from json-files would uncover confidential data. It also is unflexible - settings defined by environment-variables can e.g. be defined in kubernetes secrets.
Issue
#438
Checklist
Please delete options that are not relevant.