Self hosted crashing sometimes with mem spike

[brajbou]

We have deployed shieldsio container into our eks cluster. Every once in a while a pod crashes (replicaset=2) and I notice a spike in memory each time. I have set resource mem request min 256 mb and limit of 1gb. It still crashes once in a while. How can I enable debugging to get more information in the log? So far I haven't been able to capture any errors or warnings.

I've enabked prometheus metrics from setting env var. I don't see any option for enabling more logging. https://github.com/badges/shields/blob/master/config/custom-environment-variables.yml

[chris48s]

Our docker container doesn't output any logs. If you want to log each request, you'll need to do that in front of the shields container. The prometheus metrics are all the container itself will output.

FWIW, we used to run on instances with 256Mb in production. A little while back we found this just wasn't enough for an instance serving production traffic and upped them to 512Mb per instance.

[calebcartwright]

There's an awful lot of configuration knobs within the kubernetes ecosystem, and as such it's exceptionally difficult for us to offer much in the way troubleshooting/diagnostic support given what's largely a black box for us (not to mention that we don't really have the bandwidth for in-depth self-hosting support anyway).

The Shields.io instance handles a very large volume of traffic, and I suspect that volume likely would've surfaced any memory leaks. That's not to say it's impossible for that to be the root cause of what you are observing, just that I think it's more likely related to the configured limits and bids.

If you were to run the container directly on a virtual server (even one with just 1 Gb) and observe the memory steadily ticking up under load till it ran OOM then I'd definitely be interested to hear more about kind of badges/load you were serving from your instance

[chris48s]

One thing that occurs to me sine I posted last night: What have you got fetchLimit set to? Is it set to the default (10Mb) or have you increased it? One way you can cause a shields instance to consume a lot of memory is ask it to consume a really large response from an upstream endpoint, but the default limit of 10Mb on the size of a response should provide protection against that.

[brajbou]

I noticed that and it is the default of 10mb. At one point I had set min 512mb ram and limit of 1gb ram but noticed in datadog that it was running steady just under 256 so I reduced the two numbers to 256mb and 1gb respectively. I have a graph in datadog that shows the spike in memory corresponding with a pod crash. It has been about a week since last crash but I am still holding out at this point. What could I set the fetchLimit to as another option to try? 20mb?

[chris48s]

fetchLimit is a cap on the maximum response size shields will accept from an upstream API. Increasing that wouldn't help you decrease memory usage.

I brought it up because I wanted to eliminate this as a factor: If you had fetchLimit set to some very large number, that would be one way a lot of memory could be consumed.