v11.4.0

Quick on the heels of our previous release, we've got two new features.

• syslog.filter_legacy_vcap_messages, a off-by-default feature that addresses log duplication caused by Cloud Foundry Application Runtime releases that use tee to send logs directly to rsyslog and to a file on the disk, where blackbox now picks them up. This should eliminate the need for a custom rule that manifest maintainers have needed to maintain to avoid massive log volume. We may switch this to on-by-default at some point in the future, but that will be a breaking change with attendant major version bump if/when we do.
• syslog.heuristically_filter_debug_messages, another off-by-default feature to address log volume. This one targets debug logs, but since the PRI field does not actually reflect the priority in the logs we're interested in, we have to use a text-matching heuristic. Manifest maintainers have historically needed to either maintain a rule doing much the same thing, or manually set, for instance, UAA's logging level to INFO from its default of DEBUG. For those that cannot or do not wish to change UAA's logging behavior, this property should be helpful, though it will not eliminate all DEBUG-like messages from all components, and will drop some DEBUG logs from any other components that happen to emit them, as well.

See the spec for the syslog_forwarder job for details on these new properties.

v11.3.3

No functional or code changes to the release, though it now explicitly supports Ubuntu Xenial stemcells. No changes to the release itself were necessary to achieve this support. The release has worked with Xenial for some time.

There are changes in the test and CI code to make the tests work across multiple stemcells (and work with Xenial at all). The release is now continuously tested against both Trusty and Xenial.

In particular, it's now possible to set STEMCELL_OS as an environment variable,
and run the acceptance tests with an arbitrary stemcell.
It gets interpolated into the test manifests.
Any value that's valid for os in the manifest should work.

Also, the version of Go used is now 1.11.

v11.3.2

No functional changes, the "fix" the patch version implies is just that this was shipped via CI as normal, rather than having been manually "fixed" like v11.3.2 was.

v11.3.1

This release contains a small template refactor to use local variables instead of string literals in the definition of the structured data format. This should enable custom rules that override the message template to make use of these variables, though this is not a feature as such.

We had some CI hiccups as we published this release. We intend to ship another patch release with no changes using our CI, once we fix it.

v11.3.0

This release finishes fixing a problem with ephemeral disk usage. It also adds an option to respect file permissions when forwarding files. And, we switched to a modern Golang version. For details, read on!

Fix: loglines forwarded to blackbox were still being written to disk multiple times, as we were unable to filter them with the necessary precision. We've taken over maintenance of blackbox and extended it so that we can filter these messages out. As a side-effect, messages forwarded from files now carry a PROCID of rs2 for the remote_syslog2 syslog library blackbox uses to forward them.

Experimental Feature: The syslog.respect_file_permissions property is a preview of how we'd like to behave by default in the future. Right now, blackbox runs with the capability to read every file on the disk, and forwards every log file it can read. We'd like to only read files that we're meant to, as represented using linux filesystem permissions. So, with this property, we only forward .log files in /var/vcap/sys/log if they're readable to the syslog user or the vcap group.

We encourage CF component teams to try turning this on and see if they're getting all the logs they care about! More communication about this will be coming once it's easily available in cf-deployment.

Fix: Go 1.7 was too old. Now we use Go 1.10!

Docs Note: We relocated our custom rule examples doc to the examples directory, along with our example logstash config. This will break external links to master. Sorry! (Our examples are not part of our semver'd API, but we'll still at least apologize for breaking links.)

v11.2.0

This release has several improvements.

Feature the release can now be disabled by setting syslog.migration.disabled to true. This allows the deprecation of the syslog-migration-release. Please do not use this feature if you don't have to; it should be considered deprecated from its very introduction.

Docs changes/improvements We've improved our example rules so that they use stop instead of ~, since the latter is nominally deprecated and might stop working someday. We've also heard there are somehow performance benefits?

Fix: We'd seen some issues with the syslog_storer job shutting down properly due to issues with un-mounting the persistent disk. While this job is just intended for test purposes, this was an important issue to fix. It now has a drain script, so it stops using the disk during upgrades.

Fix: Invalid custom rules used to prevent syslog forwarding. We now test the syntactic validity of custom rules before putting them in place, so forwarding can still operate, albeit without the custom rule. Note that you will not necessarily have any indication your rule is not working, as the only place we're able to report this without disrupting platform availability is in the logs.

v11.1.1

This release contains a single change, meant to mitigate a problem with rsyslog disk usage reported by production operators of cf-deployment who were running at scale. In short, each log line was hitting the disk six extra times as a side effect of processing it.

The config syslog-release writes to /etc/rsyslog.d/ is now called 20-syslog-release.conf rather than rsyslog.conf.

This allows our rules to run before those in 50-default.conf, which allows our already existing filtering rules to prevent logs from being written to /var/log/{messages,syslog,user.log} when they are written by bosh jobs using the logger utility.

Unfortunately, this is only a partial fix, as logs forwarded by blackbox are still being written to disk three times as a side-effect of processing them. This can be avoided by setting syslog.forward_files: false. We are exploring options to more fully address this issue.

Please note that with forward_files turned off, some logs that are only written to disk will not be forwarded.

v11.1.0

This release contains a single new feature, which is intended to mitigate an issue with the syslog.file_forwarding feature truncating logs before transmitting them to local rsyslog.

See issue #34 for discussion and details.

We've added the option to switch to TCP for local log ingest from files by setting syslog.use_tcp_for_file_forwarding_local_transport to true. This prevents the truncation, but may have negative performance impact. The default behavior still uses UDP, which respects the older RFC 3164 log length limit and truncates longer messages.

Also, please note that enabling this option will cause rsyslog to listen for both UDP and TCP messages on port 514. Otherwise, it listens only for UDP messages.

Logs not forwarded by the syslog.file_forwarding feature will be unaffected by this change.

v11.0.2

This release contains no code changes. There are documentation and spec property description updates. In short, the ca_cert property is much less required for most TLS-enabled circumstances than previously advertised, because if it's not set, the stemcell's cert store will be used.

v11.0.1

This release contains no code changes, but does update the spec and documentation around the custom_rule property.

This is also the first release by the new Platform Logging Improvements team, also known as the syslog team. The README has been updated to reflect our contact information.