Shippers

Operational Insight makes use of agents, which run on edge servers to continuously ingest and process logs, metrics and other event data of interest. Application logs, OS stats and performance metrics are written locally on application servers in a variety of formats and are usually on different file-systems and servers. This data must be moved to a central location, cleansed and transformed before it can be analyzed.

Shippers are made up of forwarders and transformers, which form part of the Operational Insight data pipeline, allowing efficiently movement of large amount of data to a central index store in near real time.

Filebeat

Filebeat is a native log shipper, offering a lightweight way to forward and centralize logs and files. It can be installed as an agent on Linux and windows servers, where it monitors the log directories or specific log files, tails the files, and forwards them on to a central log parsing engine.

Filebeat log shipper monitors log directories or specific log files, keeps track of changes to the file contents, and forwards the new entries to Logstash for indexing.

When you start Filebeat, it starts one or more prospectors that look in the local paths you’ve specified for log files. For each log file located, Filebeat aggregates the events and sends the aggregated data to the configured output.

Before starting, please collect the connection information:

• Location of log files

• Fields to be mapped – app, tier, appsvr, lpar and logtype

Java Log Forwarder

The log Forwarder is a Java based log data shipper. It can be installed as an agent on any server where you can run Java. Once started, it monitors the log directories or specific log files, tails the files, and forwards them on to Logstash using the Lumberjack protocol. Use the java log forwarder if you have to forward logs from an operating systems which do not support golang.

Nmonrelay

Nmonrelay collects nmon data from the servers and send the data periodically to processor nodes for parsing. Nmon is a popular performance metrics data capture program for UNIX servers. Kafka is the only supported output target for nmonrelay.

Journalbeat

Journalbeat is a lightweight shipper based on libbeat framework, which can be used to forward logs from systemd journals. Installed as an agent on your servers, Journalbeat monitors the journal locations that you specify, collects log events, and forwards them to either to Elasticsearch or Logstash.

For more information, see the Beats Platform Reference.