Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can loki.write exceed the endpoint.block_size under certain conditions? #2299

Open
martinjaegerdk opened this issue Dec 18, 2024 · 0 comments
Open

Can loki.write exceed the endpoint.block_size under certain conditions? #2299

martinjaegerdk opened this issue Dec 18, 2024 · 0 comments

Comments

@martinjaegerdk
Copy link

martinjaegerdk commented Dec 18, 2024 •
edited
Loading

Running Alloy on Linux, with version v1.4.3.

We expect that the agent will never send batch sizes larger than 1MiB as this is the default behaviour for loki.write. But still, we have noticed getting back HTTP 413 Request Entity Too Large from the Loki write endpoint (fronted by NGINX). Nginx allows up to 8MB of payload.

ts=2024-12-18T07:11:39.097614506Z level=error msg="final error sending batch" component_path=/ component_id=loki.write.logs_default component=client host=REDACTED status=413 tenant="" error="server returned HTTP status 413 Request Entity Too Large (413): <html>"

We have seen this behaviour when the agent spins up the first time, and has to parse and send a lot of data, in case the positions file is not there, and it's configured to read from the beginning.

We have experimented with configuring the agent to not read from the beginning, to try and reduce the data. But still, with the endpoint.block_size at it's default value (1MiB), i'm expecting that it should never have happened in the first place.

Maybe there is something i'm missing?
Appreciate any guidance you can give for troubleshooting this further. Perhaps any agent logs/metrics i can look it, to validate if it actually exceeds the 1MiB, or if this is related to the ingestion route somehow having a limit i'm not aware of.

local.file_match "logs_default_varlogs" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/var/log/**/*log",
                component   = "varlog",
        },{
                __address__ = "localhost",
                __path__    = "/var/log/messages",
                component   = "messages_log",
        },{
                __address__ = "localhost",
                __path__    = "/var/log/secure",
                component   = "secure_log",
        },{
                __address__ = "localhost",
                __path__    = "/var/log/cron",
                component   = "cron_log",
        }]
}

loki.process "logs_default_varlogs" {
        forward_to = [loki.write.logs_default.receiver]

        stage.label_drop {
                values = ["filename", "traces"]
        }
}

loki.source.file "logs_default_varlogs" {
        targets               = local.file_match.logs_default_varlogs.targets
        forward_to            = [loki.process.logs_default_varlogs.receiver]
        legacy_positions_file = "/opt/alloy/data/position/default.yml"
}

loki.write "logs_default" {
        endpoint {
                url = "REDACTED"

                oauth2 {
                        client_id     = format("%s",env("CLIENT_ID"))
                        client_secret = format("%s",env("CLIENT_SECRET"))
                        scopes        = ["REDACTED"]
                        token_url     = "REDACTED"
                                                proxy_url     = "REDACTED"

                }
                proxy_url = "REDACTED"
         }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@martinjaegerdk