metropolis/pkg/logtree/doc.go

// Copyright 2020 The Monogon Project Authors.
//
// SPDX-License-Identifier: Apache-2.0
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/*
Package logtree implements a tree-shaped logger for debug events. It provides log publishers (ie. Go code) with a
glog-like API and io.Writer API, with loggers placed in a hierarchical structure defined by a dot-delimited path
(called a DN, short for Distinguished Name).

    tree.MustLeveledFor("foo.bar.baz").Warningf("Houston, we have a problem: %v", err)
    fmt.Fprintf(tree.MustRawFor("foo.bar.baz"), "some\nunstructured\ndata\n")

Logs in this context are unstructured, operational and developer-centric human readable text messages presented as lines
of text to consumers, with some attached metadata. Logtree does not deal with 'structured' logs as some parts of the
industry do, and instead defers any machine-readable logs to either be handled by metrics systems like Prometheus or
event sourcing systems like Kafka.

Tree Structure

As an example, consider an application that produces logs with the following DNs:

    listener.http
    listener.grpc
    svc
    svc.cache
    svc.cache.gc

This would correspond to a tree as follows:

                          .------.
                         |   ""   |
                         | (root) |
                          '------'
           .----------------'   '------.
    .--------------.           .---------------.
    |     svc      |           |    listener   |
    '--------------'           '---------------'
           |                   .----'      '----.
    .--------------.  .---------------.  .---------------.
    |  svc.cache   |  | listener.http |  | listener.grpc |
    '--------------'  '---------------'  '---------------'
           |
    .--------------.
    | svc.cache.gc |
    '--------------'

In this setup, every DN acts as a separate logging target, each with its own retention policy and quota. Logging to a DN
under foo.bar does NOT automatically log to foo - all tree mechanisms are applied on log access by consumers. Loggers
are automatically created on first use, and importantly, can be created at any time, and will automatically be created
if a sub-DN is created that requires a parent DN to exist first. Note, for instance, that a `listener` logging node was
created even though the example application only logged to `listener.http` and `listener.grpc`.

An implicit root node is always present in the tree, accessed by DN "" (an empty string). All other logger nodes are
children (or transitive children) of the root node.

Log consumers (application code that reads the log and passes them on to operators, or ships them off for aggregation in
other systems) to select subtrees of logs for readout. In the example tree, a consumer could select to either read all
logs of the entire tree, just a single DN (like svc), or a subtree (like everything under listener, ie. messages emitted
to listener.http and listener.grpc).

Leveled Log Producer API

As part of the glog-like logging API available to producers, the following metadata is attached to emitted logs in
addition to the DN of the logger to which the log entry was emitted:

 - timestamp at which the entry was emitted
 - a severity level (one of FATAL, ERROR, WARN or INFO)
 - a source of the message (file name and line number)

In addition, the logger mechanism supports a variable verbosity level (so-called 'V-logging') that can be set at every
node of the tree. For more information about the producer-facing logging API, see the documentation of the LeveledLogger
interface, which is the main interface exposed to log producers.

If the submitted message contains newlines, it will be split accordingly into a single log entry that contains multiple
string lines. This allows for log producers to submit long, multi-line messages that are guaranteed to be non-interleaved
with other entries, and allows for access API consumers to maintain semantic linking between multiple lines being emitted
as a single atomic entry.

Raw Log Producer API

In addition to leveled, glog-like logging, LogTree supports 'raw logging'. This is implemented as an io.Writer that will
split incoming bytes into newline-delimited lines, and log them into that logtree's DN. This mechanism is primarily
intended to support storage of unstructured log data from external processes - for example binaries running with redirected
stdout/stderr.

Log Access API

The Log Access API is mostly exposed via a single function on the LogTree struct: Read. It allows access to log entries
that have been already buffered inside LogTree and to subscribe to receive future entries over a channel. As outlined
earlier, any access can specify whether it is just interested in a single logger (addressed by DN), or a subtree of
loggers.

Due to the current implementation of the logtree, subtree accesses of backlogged data is significantly slower than
accessing data of just one DN, or the whole tree (as every subtree backlog access performs a scan on all logged data).
Thus, log consumers should be aware that it is much better to stream and buffer logs specific to some long-standing
logging request on their own, rather than repeatedly perform reads of a subtree backlog.

The data returned from the log access API is a LogEntry, which itself can contain either a raw logging entry, or a leveled
logging entry. Helper functions are available on LogEntry that allow canonical string representations to be returned, for
easy use in consuming tools/interfaces. Alternatively, the consumer can itself access the internal raw/leveled entries and
print them according to their own preferred format.

*/
package logtree