Connecting Fluentd to Honeycomb

Fluentd is a widely-used data router. If you’re using Fluentd to aggregate structured logs, Honeycomb’s Fluentd output plugin makes it easy to forward data to Honeycomb.

Getting Started

Install the Honeycomb plugin by running:

fluent-gem install fluent-plugin-honeycomb

If you’re using the stable distribution package of Fluentd, td-agent, run the following instead:

td-agent-gem install fluent-plugin-honeycomb

To set up the plugin, first grab your team write key from your account page, then update your Fluentd configuration file (usually found in /etc/fluentd/fluentd.conf or /etc/td-agent/td-agent.conf). A basic configuration to forward events with the my.logs tag to Honeycomb looks like this:

<match my.logs>
  @type honeycomb
  writekey "YOUR_WRITEKEY"
  dataset "fluentd_test_dataset"
</match>

More Advanced Configuration

Fluentd and Honeycomb support a wide variety of logging setups. Here are some things you might want to do.

Manage Multiple Data Sources

If you want to combine multiple related log sources into one dataset, you can add the option include_tag_key true to augment each event with its Fluentd tag before sending it along.

On the other hand, if you have completely unrelated log sources, you can forward them to separate datasets by using two separate configuration blocks:

<match a.*>
  @type honeycomb
  writekey "YOUR_WRITEKEY"
  dataset "dataset_b"
</match>

<match b.*>
  @type honeycomb
  writekey "YOUR_WRITEKEY"
  dataset "dataset_a"
</match>

Fork Output

To send output to both Honeycomb and another destination simultaneously, we recommend using the fluent-plugin-copy_ex plugin. For example, a configuration to send nginx access logs to both Honeycomb and Elasticsearch might look like:

<match myapp.*>
  @type copy_ex
  <store ignore_error>
    @type honeycomb
    writekey "YOUR_WRITEKEY"
    dataset "fluentd_test_dataset"
  <store ignore_error>
    @type elasticsearch
    ...
  </store>
</match>

Sample Traffic

If you have a high volume of traffic, you can use the sample_rate option to send a sampled fraction of your traffic to Honeycomb. For example, to forward one out of every five events:

<match your_match>
  @type honeycomb
  writekey "YOUR_WRITEKEY"
  dataset "fluentd_test_dataset"
  sample_rate 5
</match>

Configuration Reference

fluent-plugin-honeycomb supports the following options, in addition to Fluentd’s standard options for controlling output buffering:

Parameter	Type	Required?	Description
writekey	string	yes	Your Honeycomb write key.
dataset	string	yes	The name of the destination dataset in your Honeycomb account.
sample_rate	integer	no	Sample your event stream by sending 1 out of every N events.
include_tag_key	bool	no	Whether to include the Fluentd tag in the submitted event.
tag_key	string	no	If include_tag_key is true, the tag key name in the event (default: fluentd_tag).

Open Source

fluent-plugin-honeycomb is open source, Apache 2.0 licensed. Its source is on GitHub.