Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal Server Error: /cleanup #53

Open
mcarvalhor opened this issue Apr 10, 2024 · 7 comments
Open

Internal Server Error: /cleanup #53

mcarvalhor opened this issue Apr 10, 2024 · 7 comments
Assignees
@arturo-seijas

Comments

@mcarvalhor
Copy link
Member

Bug Description

During the certificate request flow, we received an error on the client (httprequest-lego-k8s):

2024-04-10T13:02:41.188Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     2024/04/10 13:02:40 [WARN] [internal-redacted-url.canonical.com] acme: cleaning up failed: httpreq: unexpected status code: [status code: 500] body: <!doctype html>
2024-04-10T13:02:41.190Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     <html lang="en">
2024-04-10T13:02:41.193Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     <head>
2024-04-10T13:02:41.195Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log       <title>Server Error (500)</title>
2024-04-10T13:02:41.197Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     </head>
2024-04-10T13:02:41.200Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     <body>
2024-04-10T13:02:41.203Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log       <h1>Server Error (500)</h1><p></p>
2024-04-10T13:02:41.206Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     </body>
2024-04-10T13:02:41.208Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     </html> 
2024-04-10T13:02:41.212Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     2024/04/10 13:02:40 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/REDACTED
2024-04-10T13:02:41.214Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     2024/04/10 13:02:41 Could not obtain certificates:
2024-04-10T13:02:41.217Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log         error: one or more domains had a problem:
2024-04-10T13:02:41.220Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log     [internal-redacted-url.canonical.com] [internal-redacted-url.canonical.com] acme: error presenting token: httpreq: unable to communicate with the API server: error: Post "https://internal-redacted-lego-url.canonical.com/present": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

To Reproduce

  1. kubectl delete pod traefik-ingress-0 # Deleting the tls_certificates pod retriggers the flow.

Environment

Canonical Prodstack 6:

  1. httprequest-lego-provider: prod-lego-certs-k8s@is-bastion-ps6 (latest/edge 22)
  2. httprequest-lego-k8s: prod-is-cos@is-bastion-ps6 (latest/edge 68)

Relevant log output

https://pastebin.canonical.com/p/ss2zHv8xnX/

Additional context

Per DNS repository log, it looks like the DNS entry already existed previously and was not properly cleaned up:
https://git.launchpad.net/canonical-is-dns-configs/log/canonical.com.domain?h=lego
@cbartz
Copy link
Contributor

cbartz commented Nov 5, 2024

@arturo-seijas Can you take a look and create a Jira ticket if necessary?

@mcarvalhor
Copy link
Member Author

I think we have a few not-cleaned up domains btw:
https://git.launchpad.net/canonical-is-dns-configs/tree/canonical.com.domain?h=lego

Should we clean these manually with a MP?

@arturo-seijas
Copy link
Collaborator

I think we have a few not-cleaned up domains btw: https://git.launchpad.net/canonical-is-dns-configs/tree/canonical.com.domain?h=lego

Should we clean these manually with a MP?

The /cleanup doesn't work for any of those? Do you get the same error?

@mcarvalhor
Copy link
Member Author

Hi @arturo-seijas ,

how can we submit the request manually? Which parameters are needed?

@arturo-seijas
Copy link
Collaborator

Hi @arturo-seijas ,

how can we submit the request manually? Which parameters are needed?

The charm exposes REST endpoints as defined by LEGO requiring Basic authentication. For the specific case of /cleanup you should be able to reach that endpoint with the Basic auth header corresponding to your user and password and a body like the following

{
    "fqdn": "[fqdn]",
    "value": "[token]"
}

@cbartz
Copy link
Contributor

cbartz commented Nov 15, 2024

@arturo-seijas Can you close the issue if its no longer relevant, please?

@mcarvalhor
Copy link
Member Author

I think this issue is still relevant as there are entries on the DNS repository that are never cleaned up:

But we're not sure yet how to replicate it, or when it happens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arturo-seijas arturo-seijas

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mcarvalhor @cbartz @arturo-seijas