Empty Headers from Hydrator #942

tschlaepfer · 2022-03-23T10:39:25Z

tschlaepfer
Mar 23, 2022

Hi @vinckr, I have set up a hydrator mutator within Oathkeeper, the goal is forward a custom header (X-TENANT-ID) to the backend application. I can confirm the presents of the header in the backend application. However, the header is always empty and it seems that I'm not the only one experiencing this issue, #380 (comment). Do you have an idea what I'm doing wrong or if this is a known issue?

Answered by alan994

Apr 1, 2022

Hi all,

I think we found the problem. I was sending headers all in lower case, but Ory Oath Keeper expects headers with capital letters. So I changed my hydrator endpoint from this.

app.MapPost("/hydrator", async (HttpContext httpContext, [FromBody] OryPayload payload, [FromServices] ILogger<Program> logger) =>
{
    logger.LogInformation("Payload from Ory: {@Payload}", payload);

    httpContext.Response.StatusCode = 200;

    if (payload.Header is null)
    {
        payload.Header = new Dictionary<string, object>();
    }

    payload.Header.Add("x-tl-user-id", new string[] { "alan994" });
    payload.Header.Add("x-tl-company-id", new string[] { "aj-solutions", "talentlyft" });

    lo…

View full answer

alan994 · 2022-03-23T15:00:25Z

alan994
Mar 23, 2022

Have the same problem myself. Here is my configs.

Ory Oath Keeper Access Rule

- id: "alan:api:auth:old"
  upstream:
    preserve_host: false # this will ensure we use host and port of `url` property bellow
    url: "http://host.docker.internal:5292" #5291
    strip_path: "/api/old"
  match:
    url: "http://127.0.0.1:4455/api/old/secure"
    methods:
      - GET
  authenticators:
    - handler: cookie_session
  authorizer:
    handler: allow
  mutators:
    - handler: hydrator

Ory Oath Keeper config - section mutators. Only Hydrator is important

mutators:
  noop:
    enabled: true

  hydrator:
    enabled: true
    config:
      api:
        url: http://host.docker.internal:5292/ory

  id_token:
    enabled: true
    config:
      issuer_url: http://127.0.0.1:4455/
      jwks_url: file:///etc/config/oathkeeper/id_token.jwks.json
      claims: |
        {
          "session": {{ .Extra | toJson }}
        }

  header:
    enabled: true
    config:
      headers:
        X-UserId: "{{ print .Subject }}"
        X-CompanyName: "{{ print .Extra.CompaynId }}"

My .NET service that hydrator mutator calls

app.MapPost("/ory", async (HttpContext httpContext, [FromBody] OryPayload payload, [FromServices] ILogger<Program> logger) =>
{
    logger.LogInformation("Payload from Ory: {@Payload}", payload);

    httpContext.Response.StatusCode = 200;

    if (payload.Header is null)
    {
        payload.Header = new Dictionary<string, object>();
    }

    payload.Header.Add("x-tl-user-id", new string[] { "alan994" });
    payload.Header.Add("x-tl-company-id", new string[] { "aj-solutions", "talentlyft" });


    return payload;
});

0 replies

tschlaepfer · 2022-03-23T16:46:30Z

tschlaepfer
Mar 23, 2022
Author

This is my configuration.

The oathkeeper config is:

mutators:
  noop:
    enabled: true
  header:
    enabled: false
  hydrator:
    enabled: true
    config:
      api:
        url: http://demo.ms.svc.cluster.local:8080/as/device-info
        retry:
          give_up_after: 1s
          max_delay: 200ms
      cache:
        ttl: 60s

and the rule configuration is:

kind: ConfigMap
apiVersion: v1
metadata:
  name: demo-rules
data:
  token_rule.yaml: |
    - id: demo-token-rule
      upstream:
        url: http://demo.idp.svc.cluster.local:8080/idp/token
        preserve_host: false
        strip_path: /as/token
      match:
        url: http://demo.apps.dev/as/token
        methods:
          - POST
      authenticators:
        - handler: jwt
      authorizer:
        handler: remote
        config:
          remote: http://demo.ms.svc.cluster.local:8080/as/authz
      mutators:
        - handler: hydrator
          config:
            api:
              url: http://demo.ms.svc.cluster.local:8080/as/device-info
        - handler: id_token
          config:
            issuer_url: demo-client
            ttl: 10s
            claims: '{"aud": ["https://demo.apps.dev/oauth2/token"]}'
      errors:
        - handler: json

In the logs of the mircoservice that provides the information for the hydrator I can see that the X-TENANT-ID header is set correctly:

{"match_context":{"regexp_capture_groups":[],"url":{"Path":"/as/token","Scheme":"http","User":null,"Opaque":"","ForceQuery":false,"RawQuery":"","RawFragment":"","Host":"demo.apps.dev","RawPath":"","Fragment":""}},"subject":"demo-client","extra":{"scp":[],"sub":"demo-client"},"header":{"X-TENANT-ID":["demo"]}}

0 replies

vinckr · 2022-03-23T16:51:00Z

vinckr
Mar 23, 2022
Maintainer

Hey folks, thanks for opening the discussion and the extra info.
I am currently doing some exploration, not being super familiar with hydrator.

Here is another issue with the same problem:
#582

I found these earlier discussions with a similar issues:

#901
#851
#793

0 replies

alan994 · 2022-03-28T11:12:49Z

alan994
Mar 28, 2022

Hi @vinckr,
I looked at the links you attached. It seems to me that all of them are having issues with header mutator, not with the hydrator. In our case, we don't need a header mutator because hydrator should change AuthenticationSession object directly and whatever is in header map should be propagated to upstream API. And it is but without value :(

Did you try to reproduce this issue yourself? Can we help somehow?

5 replies

vinckr Mar 28, 2022
Maintainer

Hey Alan,
you are right,

Here is also an older relevant issue: #299 , and it seems the headers should be forwarded correctly? (see also this PR)

I did not get around to reproduce it, it would help greatly if you could share a repo with a simple config that reproduces the error.
Also tagging @gen1us2k, did you run into this before by any chance?

gen1us2k Mar 29, 2022

Hello there.
I built a simple example with Go, oathkeeper, and Kratos.
I have a service hello and I want to proxy only authenticated and mutated requests to this service. The code is simple and you can find it here

An example of hydrator that sets a custom and hardcoded header.

Oathkeeper config

log:
  level: debug
  format: json

serve:
  proxy:
    cors:
      enabled: true
      allowed_origins:
        - "*"
      allowed_methods:
        - POST
        - GET
        - PUT
        - PATCH
        - DELETE
      allowed_headers:
        - Authorization
        - Content-Type
      exposed_headers:
        - Content-Type
      allow_credentials: true
      debug: true

errors:
  fallback:
    - json

  handlers:
    redirect:
      enabled: true
      config:
        to: http://127.0.0.1:4455/login
        when:
          -
            error:
              - unauthorized
              - forbidden
            request:
              header:
                accept:
                  - text/html
    json:
      enabled: true
      config:
        verbose: true

access_rules:
  matching_strategy: regexp
  repositories:
    - file:///etc/config/oathkeeper/access-rules.yml

authenticators:
  anonymous:
    enabled: true
    config:
      subject: guest

  cookie_session:
    enabled: true
    config:
      check_session_url: http://kratos:4433/sessions/whoami
      preserve_path: true
      extra_from: "@this"
      subject_from: "identity.id"
      only:
        - ory_kratos_session

  noop:
    enabled: true

authorizers:
  allow:
    enabled: true

mutators:
  noop:
    enabled: true
  hydrator:
    enabled: true
    config:
      api:
        url: http://hydrator:8090/hydrate

Access rules

-
  id: "api:service:protected"
  upstream:
    preserve_host: true
    url: "http://service:8090"
  match:
    url: "http://127.0.0.1:8080/hello"
    methods:
      - GET
  authenticators:
    -
      handler: cookie_session
  mutators:
    - handler: hydrator
  authorizer:
    handler: allow
  errors:
    - handler: redirect
      config:
        to: http://127.0.0.1:4455/login

You can check an example for now, but my next steps are to have a similar configuration that you have

alan994 Mar 30, 2022

Sorry, for the late response. I will create a repo with a sample today or tomorrow

alan994 Mar 31, 2022

Hi @vinckr,
Here is my sample repo that contains API with the target endpoint (/secure) and hydrator endpoint (/hydrator). In readme, you will find instructions to reproduce.

gen1us2k Apr 1, 2022

@alan994 Hello. I found the root cause of your issue. You use a simple dict<string, object>. That's okay, but oathkeeper uses http.Header type, which is map[string][]string with additional sugar for Get/Add/Set headers. The difference between our examples is that I use the same http.Header data type, and when I set a custom header it changes its name to a canonical version. For instance, I use x-user-id and I have X-User-Id inside a map and have no issues with that.

I changed two things in your demo repo. First one is a data type of Header

 public Dictionary<string, string[]> Header {get; set;}

and the second thing is

       payload.Header.Add("X-Tl-User-Id", new string[] { "alan994" });
       payload.Header.Add("X-Tl-Company-Id", new string[] { "aj-solutions", "talentlyft" });

I can find in logs custom headers.

[11:22:29 INF] Header arrived: X-Tl-Company-Id - ["aj-solutions"]
[11:22:29 INF] Header arrived: X-Tl-User-Id - ["alan994"]

I suggest using the canonical version of headers. Oathkeeper gets custom headers and has them in header object with key x-tl-user-id but tries to get by different key X-Tl-User-Id and hence you have empty headers.
I hope this helps!

alan994 · 2022-04-01T10:49:47Z

alan994
Apr 1, 2022

Hi all,

I think we found the problem. I was sending headers all in lower case, but Ory Oath Keeper expects headers with capital letters. So I changed my hydrator endpoint from this.

app.MapPost("/hydrator", async (HttpContext httpContext, [FromBody] OryPayload payload, [FromServices] ILogger<Program> logger) =>
{
    logger.LogInformation("Payload from Ory: {@Payload}", payload);

    httpContext.Response.StatusCode = 200;

    if (payload.Header is null)
    {
        payload.Header = new Dictionary<string, object>();
    }

    payload.Header.Add("x-tl-user-id", new string[] { "alan994" });
    payload.Header.Add("x-tl-company-id", new string[] { "aj-solutions", "talentlyft" });

    logger.LogInformation("Returning payload to Ory: {@Payload}", payload);
    
    return payload;
});

to this

app.MapPost("/hydrator", async (HttpContext httpContext, [FromBody] OryPayload payload, [FromServices] ILogger<Program> logger) =>
{
    logger.LogInformation("Payload from Ory: {@Payload}", payload);

    httpContext.Response.StatusCode = 200;

    if (payload.Header is null)
    {
        payload.Header = new Dictionary<string, object>();
    }

    payload.Header.Add("X-Tl-User-Id", new string[] { "alan994" });
    payload.Header.Add("X-Tl-Company-Id", new string[] { "aj-solutions", "talentlyft" });

    logger.LogInformation("Returning payload to Ory: {@Payload}", payload);
    
    return payload;
});

And it works 🥳

We found that this code changed headers to capital letters

oathkeeper/proxy/proxy.go

Line 146 in 2610d2c

r.Header.Set(h, s.Header.Get(h))

6 replies

alan994 Apr 1, 2022

Thanks for the help and effort. Keep up the good work with Ory 👏👏👏

vinckr Apr 1, 2022
Maintainer

@tschlaepfer
Does this also resolve your problem?

Thanks to all in this discussion 🙏

tschlaepfer Apr 4, 2022
Author

Thanks @alan994 for finding a solution for this issue. In contrast to your all lower case headers I was using all upper case letters, after switching my header to X-Tenant-Id I now also see the value of the header in my backend application :)

aeneasr Apr 5, 2022
Maintainer

Can we add this as a warning to the documentation? I didnt know lower cases were a problem! Or alternatively fix it in a PR :)

vinckr Apr 7, 2022
Maintainer

i made some issues in oathkeeper & docs
#950
ory/docs#745

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Empty Headers from Hydrator #942

{{title}}

Replies: 5 comments 11 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Empty Headers from Hydrator #942

Replies: 5 comments · 11 replies

tschlaepfer Mar 23, 2022 Author

vinckr Mar 23, 2022 Maintainer

vinckr Mar 28, 2022 Maintainer

vinckr Apr 1, 2022 Maintainer

tschlaepfer Apr 4, 2022 Author

aeneasr Apr 5, 2022 Maintainer

vinckr Apr 7, 2022 Maintainer

Replies: 5 comments 11 replies

tschlaepfer
Mar 23, 2022
Author

vinckr
Mar 23, 2022
Maintainer

vinckr Mar 28, 2022
Maintainer

vinckr Apr 1, 2022
Maintainer

tschlaepfer Apr 4, 2022
Author

aeneasr Apr 5, 2022
Maintainer

vinckr Apr 7, 2022
Maintainer