Skip to content

Commit

Permalink
fix(filebeat): Send fewer events in each bulk api call
Browse files Browse the repository at this point in the history
Large log lines grouped together (1000 at a time) create a very large payload that log server rejects
Filebeat keeps retrying this huge payload over and over again.
This causes excess bandwidth usage between db and log servers.

References:
- https://gameplan.frappe.cloud/g/space/237/discussion/3340/frappe-cloud-metrics-review-14th-january-2024?comment=12478
- https://gameplan.frappe.cloud/g/space/17/discussion/2743/aws-costs-and-blunders
  • Loading branch information
adityahase committed Jan 22, 2025
1 parent 6b962e0 commit a90f30c
Show file tree
Hide file tree
Showing 4 changed files with 8 additions and 0 deletions.
4 changes: 4 additions & 0 deletions press/playbooks/roles/filebeat/templates/filebeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ output.elasticsearch:
ssl.verification_mode: strict
ssl.supported_protocols: [TLSv1.3]

{% if server_type is defined and server_type == "Datbase Server" %}
bulk_max_size: 10
{% endif %}

# ================================== Logging ===================================

logging.to_files: true
Expand Down
2 changes: 2 additions & 0 deletions press/press/doctype/database_server/database_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,7 @@ def _setup_server(self):
user=self.ssh_user or "root",
port=self.ssh_port or 22,
variables={
"server_type": self.doctype,
"server": self.name,
"workers": "2",
"agent_password": config.agent_password,
Expand Down Expand Up @@ -833,6 +834,7 @@ def _rename_server(self):
playbook="database_rename.yml",
server=self,
variables={
"server_type": self.doctype,
"server": self.name,
"workers": "2",
"agent_password": agent_password,
Expand Down
1 change: 1 addition & 0 deletions press/press/doctype/server/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,7 @@ def _install_filebeat(self):
user=self._ssh_user(),
port=self._ssh_port(),
variables={
"server_type": self.doctype,
"server": self.name,
"log_server": log_server,
"kibana_password": kibana_password,
Expand Down
1 change: 1 addition & 0 deletions press/press/doctype/virtual_machine/virtual_machine.py
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,7 @@ def get_cloud_init(self):
"filebeat_config": frappe.render_template(
"press/playbooks/roles/filebeat/templates/filebeat.yml",
{
"server_type": server.doctype,
"server": self.name,
"log_server": log_server,
"kibana_password": kibana_password,
Expand Down

0 comments on commit a90f30c

Please sign in to comment.