Carbonate

"瓶瓶罐罐。" -- 鸟人

Graphite集群很酷。这里有一些原始工具来帮助您管理您的Graphite集群。

所有工具都支持两个常见参数；配置文件路径和集群名称。使用这些工具和描述您石墨集群的配置文件，您可以构建脚本来管理您的指标。一些工具可以很容易地用shell中的一行代码替换，但在这里提供是为了方便和可读性。目标是提供快速、可预测的实用工具，这些工具可以轻松地组合成更高级的工具。

安装

Carbonate可以从Python官方第三方仓库获取（即PyPi），因此可以通过常规Python包管理器进行安装。请注意，您可能需要在Ubuntu主机上安装Python包管理器（例如，apt-get install python-setuptools）

pip install carbonate

配置

Carbonate期望一个配置文件，该文件定义了您环境中的集群。默认配置文件位于/opt/graphite/conf/carbonate.conf或可以提供在命令行上。默认集群名称为'main'。这两个默认值可以通过在环境中设置CARBONATE_CONFIG和CARBONATE_CLUSTER来覆盖。

[main]
DESTINATIONS = 192.168.9.13:2004:carbon01, 192.168.9.15:2004:carbon02, 192.168.6.20:2004:carbon03
REPLICATION_FACTOR = 2
SSH_USER = carbon

[agg]
DESTINATIONS = 192.168.9.13:2004:carbon01, 192.168.9.15:2004:carbon02, 192.168.6.20:2004:carbon03
RELAY_METHOD = aggregated-consistent-hashing
REPLICATION_FACTOR = 2
SSH_USER = carbon

[fnv]
DESTINATIONS = 192.168.9.13:2004:ba603c36342304ed77953f84ac4d357b, 192.168.9.15:2004:5dd63865534f84899c6e5594dba6749a, 192.168.6.20:2004:866a18b81f2dc4649517a1df13e26f28
REPLICATION_FACTOR = 2
SSH_USER = carbonate
HASHING_TYPE = fnv1a_ch

您应该确保目标IP地址或主机名列表与集群中的节点匹配（即应与carbon relay的路由配置相匹配）。顺序很重要，因为这是创建一致性哈希环的方式。

您可以配置中继方法为“一致性哈希”或“聚合一致性哈希”。如果省略，则默认使用“一致性哈希”。使用“聚合一致性哈希”通常需要提供规则文件给相关命令。

复制因子应与集群的复制因子匹配。

此外，您可以选择提供在Carbonate需要连接到集群中的另一个节点以执行操作时将使用的SSH用户。如果没有提供，则将选择执行命令的当前用户。

最后，您可以提供集群的HASHING_TYPE。默认为carbon_ch，也支持fnv1a_ch。请注意，要使用fnv1a_ch哈希，您需要安装1.0.2或更高版本的carbon（或者您需要使用carbon-c-relay中继）。

工具

carbon-hosts

usage: carbon-hosts [-h] [-c CONFIG_FILE] [-C CLUSTER]

Return the addresses for all nodes in a cluster

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (default: main)

carbon-lookup

usage: carbon-lookup [-h] [-c CONFIG_FILE] [-C CLUSTER] [-s] METRIC

Lookup where a metric lives in a carbon cluster

positional arguments:
  METRIC                Full metric name to search for

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (default: main)
  -a AGGREGATION_RULES, --aggregation-rules AGGREGATION_RULES
                        File containing rules used in conjunction with the
                        "aggregated-consistent-hashing" relay method (default:
                        /opt/graphite/conf/aggregation-rules.conf)
  -s, --short           Only display the address, without port and cluster
                        name (default: False)

carbon-list

usage: carbon-list [-h] [-c CONFIG_FILE] [-C CLUSTER] [-d STORAGE_DIR]

List the metrics this carbon node contains

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (default: main)
  -d STORAGE_DIR, --storage-dir STORAGE_DIR
                        Storage dir (default: /opt/graphite/storage/whisper)

carbon-sieve

usage: carbon-sieve [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE]
                    [-n NODE] [-I]

Given a list of metrics, output those that belong to a node

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (default: main)
  -a AGGREGATION_RULES, --aggregation-rules AGGREGATION_RULES
                        File containing rules used in conjunction with the
                        "aggregated-consistent-hashing" relay method (default:
                        /opt/graphite/conf/aggregation-rules.conf)
  -f METRICS_FILE, --metrics-file METRICS_FILE
                        File containing metric names to filter, or '-' to read
                        from STDIN (default: -)
  -n NODE, --node NODE  Filter for metrics belonging to this node (default:
                        self)
  -I, --invert          Invert the sieve, match metrics that do NOT belong to
                        a node (default: False)

carbon-sync

usage: carbon-sync [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] -s
                   SOURCE_NODE [-d STORAGE_DIR] [-b BATCH_SIZE]
                   [--source-storage-dir SOURCE_STORAGE_DIR]
                   [--rsync-options RSYNC_OPTIONS] [--rsync-disable-copy-dest]
                   [--tmpdir TMP_STAGING_DIR] [--rsync-max-retries MAX_RETRIES]
                   [--rsync-retries-interval SECONDS] [--dirty] [-l] [-o]

Sync local metrics using remote nodes in the cluster

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (env: CARBONATE_CONFIG) (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (env: CARBONATE_CLUSTER) (default: main)
  -f METRICS_FILE, --metrics-file METRICS_FILE
                        File containing metric names to filter, or '-' to read
                        from STDIN (default: -)
  -s SOURCE_NODE, --source-node SOURCE_NODE
                        Override the source for metrics data (default: None)
  -d STORAGE_DIR, --storage-dir STORAGE_DIR
                        Storage dir (default: /opt/graphite/storage/whisper)
  -b BATCH_SIZE, --batch-size BATCH_SIZE
                        Batch size for the rsync job (default: 1000)
  --source-storage-dir SOURCE_STORAGE_DIR
                        Source storage dir (default:
                        /opt/graphite/storage/whisper)
  --rsync-options RSYNC_OPTIONS
                        Pass option(s) to rsync. Make sure to use "--rsync-
                        options=" if option starts with '-' (default: -azpS)
  --rsync-disable-copy-dest
                        Avoid --copy-dest, transfer all whisper data between
                        nodes. (default: False)
  --rsync-max-retries RETRIES
                        Number of times rsync will attempt to copy each batch
                        of metrics before moving on. If all retry attempts are
                        unsuccessful, carbon-sync will write a file containing
                        the name of each metric in the failed batch so they can
                        be easily retried at a later time. (Default: 3)
  --rsync-retries-interval SECONDS
                        How long to wait in between each rsync retry attempt
                        (see --rsync-max-retries). (default: 5)
  -t TMP_STAGING_DIR, --tmpdir TMP_STAGING_DIR
                        Specify an alternate location in which the temporary
                        rsync staging dirs will be created. This can be useful
                        for large syncs where the default location (as chosen
                        by mkdtemp) resides on a filesystem that's too small
                        to store all the metrics being copied from the remote
                        host.
  --dirty               If set, don't clean temporary rsync directory
                        (default: False)
  -l, --lock            Lock whisper files during filling (default: False)
  -o, --overwrite       Write all non nullpoints from src to dst (default:
                        False)

carbon-path

usage: carbon-path [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] [-r]
                   [-p] [-d STORAGE_DIR]

Transform metric paths to (or from) filesystem paths

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (default: main)
  -f METRICS_FILE, --metrics-file METRICS_FILE
                        File containing metric names to transform to file
                        paths, or '-' to read from STDIN (default: -)
  -r, --reverse         Transform from file paths to metric paths (default:
                        False)
  -p, --prepend         Prepend storage dir to file paths (default: False)
  -d STORAGE_DIR, --storage-dir STORAGE_DIR
                        Whisper storage directory to prepend when -p given
                        (default: /opt/graphite/storage/whisper)

carbon-stale

usage: carbon-stale [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] [-r]
                    [-d STORAGE_DIR] [-l HOURS] [-o HOURS] [-w] [-p]

Find and list potentially stale metrics.

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config file to use (default:
                        /opt/graphite/conf/carbonate.conf)
  -C CLUSTER, --cluster CLUSTER
                        Cluster name (default: main)
  -f METRICS_FILE, --metrics-file METRICS_FILE
                        File containing metric names to transform to file
                        paths, or '-' to read from STDIN (default: -)
  -r, --reverse         Output metrics which are not stale instead (default:
                        False)
  -d STORAGE_DIR, --storage-dir STORAGE_DIR
                        Whisper storage directory to prepend when -p given
                        (default: /opt/graphite/storage/whisper)
  -l HOURS, --limit HOURS
                        Definition of staleness, in hours (default: 24)
  -o HOURS, --offset HOURS
                        Use a whisper data window ending HOURS ago (implies
                        -w) (default: 0)
  -w, --whisper         Use whisper data instead of filesystem stat() call
                        (default: False)
  -p, --paths           Print filesystem paths instead of metric names
                        (default: False)

whisper-aggregate

usage: whisper-aggregate [-h] [-f METRICS_FILE] [-d STORAGE_DIR]

Set aggregation for whisper-backed metrics this carbon instance contains

optional arguments:
  -h, --help            show this help message and exit
  -f METRICS_FILE, --metrics-file METRICS_FILE
                        File containing metric names and aggregation modes, or
                        '-' to read from STDIN (default: -)
  -d STORAGE_DIR, --storage-dir STORAGE_DIR
                        Whisper storage directory (default:
                        /opt/graphite/storage/whisper)

whisper-fill

usage: whisper-fill [-h] [-l] [-o] SRC DST

Backfill datapoints from one whisper file into another

positional arguments:
  SRC             Whisper source file
  DST             Whisper destination file

optional arguments:
  -h, --help      show this help message and exit
  -l, --lock      Lock whisper files during filling (default: False)
  -o, --overwrite  Write all non nullpoints from src to dst (default: False)

示例用法

重新同步集群中的一个节点

#!/bin/sh
#
# Resync a node from other nodes in the cluster
#

LOCAL_IP="$1"

for h in $(carbon-hosts) ; do
  (
    ssh $h -- carbon-list |
    carbon-sieve -n $LOCAL_IP |
    carbon-sync -s $h
  ) &
done

重新平衡集群

#!/bin/sh
#
# Rebalance a cluster from one size to another. Remember to cleanup metrics
# that no longer belong when all nodes are rebalanced!
#

LOCAL_IP="$1"
OLD_CLUSTER="old"
NEW_CLUSTER="main"

for h in $(carbon-hosts -C "$OLD_CLUSTER") ; do
  ssh $h -- carbon-list |
  carbon-sieve -C "$NEW_CLUSTER" -n $LOCAL_IP |
  carbon-sync -s $h
done

列出不属于的指标

#!/bin/sh
#
# List metrics from disk that don't belong
#

LOCAL_IP="$1"

carbon-list | carbon-sieve -I -n $LOCAL_IP

列出停止更新的指标

过去2小时内完全为空的whisper数据的指标（如果您怀疑fs时间戳或carbon客户端写入“未来”的问题，这可能很有用）

carbon-list | carbon-stale --whisper --limit=2

48小时或更长时间内看起来未更改的指标文件（功能上等同于find /your/data/dir -type f -mtime +2）

carbon-list | carbon-stale --limit=48

更有趣的是，如果您使用carbon-stale，然后筛选以识别不属于这里的陈旧指标（而不是不属于这里的未陈旧指标，这些指标由于路径中的重复周期（例如损坏的收集器）而错误地报告为carbon-sieve中的陈旧指标。这是一个问题。）

carbon-list | carbon-stale --limit=48 | carbon-sieve -I -n $LOCAL_IP

要打印文件路径以供例如xargs rm等使用，请使用-p

carbon-list | carbon-stale -p | xargs -n 100 rm

许可协议

代码可在MIT许可证下使用。