ZooKeeper协议分析器和统计收集守护进程

导航

验证详情

这些详情已由PyPI 验证
维护者
Avatar for rgs from gravatar.com rgs Avatar for Yasumoto from gravatar.com Yasumoto

未验证详情

这些详情尚未由PyPI验证
项目链接
元数据
  • 许可： Apache软件许可协议（Apache）
  • 作者： Raul Gutierrez Segales
  • 标签 ZooKeeper, Sniffer, Stats
分类器

项目描述

# ZKTraffic [![构建状态](https://travis-ci.org/twitter/zktraffic.svg?branch=master)](https://travis-ci.org/twitter/zktraffic) [![覆盖率状态](https://coveralls.io/repos/twitter/zktraffic/badge.png)](https://coveralls.io/r/twitter/zktraffic) [![PyPI版本](https://badge.fury.io/py/zktraffic.svg)](http://badge.fury.io/py/zktraffic)

目录

  • [tl;dr](#tldr)

  • [安装](#installing)

  • [ZKTraffic是什么？](#what-is-zktraffic)

  • [贡献和测试](#contributing-and-testing)

  • [更多工具！](#more-tools)

  • [OS X](#os-x)

  • [依赖项](#dependencies)

### tl;dr ###

ZooKeeper协议分析器和统计收集守护进程

### 安装 ###

您可以通过pip安装ZKTraffic

$ pip install zktraffic

或者从源代码运行它（如果您已安装依赖项，请参阅以下内容）

$ git clone https://github.com/twitter/zktraffic.git
$ cd zktraffic
$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --iface=eth0

以快速计数请求的路径

$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --iface=eth0 --count-requests 10000 --sort-by path
/ 1749
/services/prod/search 846
/configs/teleportation/features 843

或类型

$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --iface=eth0 --count-requests 10000 --sort-by type
GetChildrenRequest 9044
ExistsRequest 958

您还可以按路径（平均、p95和p99）测量延迟

$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --measure-latency 1000 --group-by path --aggregation-depth 2 --sort-by p99
path                     avg         p95         p99
---------------  -----------  ----------  ----------
/party/services  0.000199077  0.00048846  0.00267805
/party           0.000349498  0.00136839  0.00201204
/party/configs   0.000157728  0.00036664  0.00122663

或类型

$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --measure-latency 1000 --group-by type --sort-by p99
type                            avg          p95          p99
----------------------  -----------  -----------  -----------
CreateEphemeralRequest  0.000735009  0.000978041  0.0032404
GetChildrenRequest      0.000182547  0.000453258  0.00220628
ExistsRequest           0.000162728  0.000430155  0.000862937

或客户端

$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-dump --measure-latency 1000 --group-by client --sort-by p99
client                          avg          p95          p99
----------------------  -----------  -----------  -----------
10.0.1.3:44308          0.000735009  0.000978041  0.0032404
10.0.1.6:34305          0.000182547  0.000453258  0.00220628
10.0.1.9:36110          0.000162728  0.000430155  0.000862937

或使用统计收集守护进程

$ sudo ZKTRAFFIC_SOURCE=1 bin/zk-stats-daemon --iface=eth0 --http-port=9090

或者您可以从源代码构建 PEX 文件，适用于任何可用的工具

$ pip install pex

# zk-dump
$ pex -v -e zktraffic.cli.zk -o zk-dump.pex .

# zk-stats-daemon
$ pex -v -e zktraffic.cli.stats_daemon -o stats-daemon.pex .

# zab-dump
$ pex -v -e zktraffic.cli.zab -o zab-dump.pex .

# fle-dump
$ pex -v -e zktraffic.cli.fle -o fle-dump.pex .

关于 PEX 的更多信息[在此](https://pex.readthedocs.org “PEX”)

### 什么是 ZKTraffic？###

ZKTraffic 是一个类似 iptraf 的 ZooKeeper 流量监控工具。目前它导出每个路径（和全局）的统计数据。最终它将导出每个用户的统计数据。

它有一个前端 zk-dump，可以用于交互模式以转储流量

` # 需要 root 或 CAP_NET_ADMIN & CAP_NET_RAW $ sudo zk-dump --iface eth0 21:08:05:991542 ConnectRequest(ver=0, zxid=0, timeout=10000, session=0x0, readonly=False, client=127.0.0.1:50049) ————————►21:08:06:013513 ConnectReply(ver=0, timeout=10000, session=0x148cf0aedc60000, readonly=False, client=127.0.0.1:50049) 21:08:07:432361 ExistsRequest(xid=1, path=/, watch=False, size=14, client=127.0.0.1:50049) ————————►21:08:07:447353 ExistsReply(xid=1, zxid=31, error=0, client=127.0.0.1:50049) 21:08:07:448033 GetChildrenRequest(xid=2, path=/, watch=False, size=14, client=127.0.0.1:50049) ————————►21:08:07:456169 GetChildrenReply(xid=2, zxid=31, error=0, count=1, client=127.0.0.1:50049) ... `

或者，它可以在守护进程模式下工作，从而公开 HTTP/JSON 端点，并提供可以输入您喜欢的数据收集系统的统计数据

$ sudo zk-stats-daemon.pex --app_daemonize --aggregation-depth=5

# Wait for 1 min and:

$ sleep 60 && curl http://localhost:7070/json/paths | python -mjson.tool
{
 "ConnectRequest": 2,
 "ConnectRequestBytes": 90,
 "CreateRequest/configs": 2,
 "CreateRequest/configs/server": 2,
 "CreateRequest/discovery": 2,
 "CreateRequest/discovery/hosts": 2,
 "CreateRequest/discovery/services": 2,
 "CreateRequestBytes/configs": 110,
 "CreateRequestBytes/configs/server": 124,
 "CreateRequestBytes/discovery": 114,
 "CreateRequestBytes/discovery/hosts": 126,
 "CreateRequestBytes/discovery/services": 132,
 "ExistsRequest/": 1574,
 "ExistsRequest/configs": 3,
 "ExistsRequest/configs/server": 2,
 "ExistsRequest/discovery": 4,
 "ExistsRequest/discovery/hosts": 2,
 "ExistsRequest/discovery/services": 2,
 "ExistsRequestBytes/": 22036,
 "ExistsRequestBytes/configs": 63,
 "ExistsRequestBytes/configs/server": 56,
 "ExistsRequestBytes/discovery": 92,
 "ExistsRequestBytes/discovery/hosts": 58,
 "ExistsRequestBytes/discovery/services": 64,
 "GetChildrenRequest/configs": 1285,
 "GetChildrenRequest/configs/server": 1242,
 "GetChildrenRequest/discovery": 1223,
 "GetChildrenRequest/discovery/hosts": 1250,
 "GetChildrenRequest/discovery/services": 1222,
 "GetChildrenRequest/zookeeper/config": 1285,
 "GetChildrenRequest/zookeeper/quota/limits": 1228,
 "GetChildrenRequest/zookeeper/quota/limits/by-path": 1269,
 "GetChildrenRequest/zookeeper/quota/limits/global": 1230,
 "GetChildrenRequest/zookeeper/quota/stats/by-path": 1222,
 "GetChildrenRequestBytes/discovery/hosts": 36250,
 "GetChildrenRequestBytes/discovery/services": 39104,
 "GetChildrenRequestBytes/zookeeper/config": 38550,
 "GetChildrenRequestBytes/zookeeper/quota/limits": 44208,
 "GetChildrenRequestBytes/zookeeper/quota/limits/by-path": 55836,
 "GetChildrenRequestBytes/zookeeper/quota/limits/global": 52890,
 "GetChildrenRequestBytes/zookeeper/quota/limits/slices": 51815,
 "GetChildrenRequestBytes/zookeeper/quota/stats": 42630,
 "GetChildrenRequestBytes/zookeeper/quota/stats/by-path": 52546,
 "GetChildrenRequestBytes/zookeeper/quota/stats/global": 50568,
 "reads/": 2761,
 "reads/configs": 1288,
 "reads/configs/server": 1244,
 "reads/discovery": 1227,
 "reads/discovery/hosts": 1252,
 "reads/discovery/services": 1224,
 "reads/zookeeper/config": 1285,
 "reads/zookeeper/quota/limits": 1228,
 "reads/zookeeper/quota/limits/by-path": 1269,
 "reads/zookeeper/quota/limits/global": 1230,
 "readsBytes/": 38654,
 "readsBytes/discovery/services": 39168,
 "readsBytes/zookeeper/config": 38550,
 "readsBytes/zookeeper/quota/limits": 44208,
 "readsBytes/zookeeper/quota/limits/by-path": 55836,
 "readsBytes/zookeeper/quota/limits/global": 52890,
 "readsBytes/zookeeper/quota/limits/slices": 51815,
 "readsBytes/zookeeper/quota/stats": 42630,
 "readsBytes/zookeeper/quota/stats/by-path": 52546,
 "readsBytes/zookeeper/quota/stats/global": 50568,
 "total/readBytes": 655586,
 "total/reads": 21251,
 "total/writeBytes": 606,
 "total/writes": 10,
 "writes/": 0,
 "writes/configs": 2,
 "writes/configs/server": 2,
 "writes/discovery": 2,
 "writes/discovery/hosts": 2,
 "writes/discovery/services": 2,
 "writesBytes/": 0,
 "writesBytes/configs": 110,
 "writesBytes/configs/server": 124,
 "writesBytes/discovery": 114,
 "writesBytes/discovery/hosts": 126,
 "writesBytes/discovery/services": 132
}

其他相关的统计数据端点包括

  • /json/ips: 每个 IP 的 top-N 统计数据

  • /json/auths: 每个 auth 的统计数据

  • /json/auths-dump: 已知 auth 的完整转储

  • /json/info: 进程运行时间和内省信息

  • /threads: 所有线程的堆栈

### 贡献和测试###

请参阅 [CONTRIBUTING.md](CONTRIBUTING.md)。

### 更多工具!###

除了 zk-dump 和 zk-stats-daemon，您还可以找到 fle-dump，它允许您检查 FastLeaderElection 流量（即 ZooKeeper 决定谁将领导以及随后的领导者发现机制）

$ sudo fle-dump --iface eth0 -c
Notification(
  timestamp=00:57:12:593254,
  src=10.0.0.1:32938,
  dst=10.0.0.2:3888,
  state=following,
  leader=3,
  zxid=0,
  election_epoch=0,
  peer_epoch=0,
  config=
       server.0=10.0.0.1:2889:3888:participant;0.0.0.0:2181
       server.1=10.0.0.2:2889:3888:participant;0.0.0.0:2181
       server.2=10.0.0.3:2889:3888:participant;0.0.0.0:2181
       server.3=10.0.0.4:2889:3888:participant;0.0.0.0:2181
       server.4=10.0.0.5:2889:3888:participant;0.0.0.0:2181
       version=10010d4d6
)
Notification(
  timestamp=00:57:12:595525,
  src=10.0.0.2:3888,
  dst=10.0.0.1:32938,
  state=looking,
  leader=1,
  zxid=4296326153,
  election_epoch=1,
  peer_epoch=1,
  config=
       server.0=10.0.0.1:2889:3888:participant;0.0.0.0:2181
       server.1=10.0.0.2:2889:3888:participant;0.0.0.0:2181
       server.2=10.0.0.3:2889:3888:participant;0.0.0.0:2181
       server.3=10.0.0.4:2889:3888:participant;0.0.0.0:2181
       server.4=10.0.0.5:2889:3888:participant;0.0.0.0:2181
       version=10010d4d6
)
...

注意：如果您使用的是 ZooKeeper 3.5.1-rc2 之前的 ZooKeeper，则需要使用可用的补丁 [ZOOKEEPER-2098](https://issues.apache.org/jira/browse/ZOOKEEPER-2098 “ZOOKEEPER-2098”）才能使初始消息可见。

注意：如果您使用的是 Linux 3.14 或更高版本，则需要通过运行 echo 0 > /proc/sys/net/ipv4/tcp_autocorking 来禁用 [TCP 自动封包](http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f54b311142a92ea2e42598e347b84e1655caf8e3）

如果您对调试 ZAB（ZooKeeper 原子广播协议）感兴趣，则可以使用 zab-dump

$ sudo zab-dump --iface eth0

Request(
 cxid=6,
 dst=10.0.0.1:2889,
 length=112,
 req_type=CreateRequest,
 session_id=0x34e4d23b0d70001,
 src=10.0.0.2:48604,
 timestr=22:54:31:995353,
 zxid=-1,
)
Proposal(
 cxid=6,
 dst=10.0.0.2:48603,
 length=110,
 session_id=0x34e4d23b0d70001,
 src=10.0.0.1:2889,
 timestr=22:54:31:995753,
 txn_time=1435816471995,
 txn_type=CreateRequest,
 txn_zxid=8589934619,
 zxid=8589934619,
)
Proposal(
 cxid=6,
 dst=10.0.0.1:48604,
 length=110,
 session_id=0x34e4d23b0d70001,
 src=10.0.0.1:2889,
 timestr=22:54:31:995755,
 txn_time=1435816471995,
 txn_type=CreateRequest,
 txn_zxid=8589934619,
 zxid=8589934619,
)
Proposal(
 cxid=6,
 dst=10.0.0.3:48605,
 length=110,
 session_id=0x34e4d23b0d70001,
 src=10.0.0.1:2889,
 timestr=22:54:31:995770,
 txn_time=1435816471995,
 txn_type=CreateRequest,
 txn_zxid=8589934619,
 zxid=8589934619,
)
Ack(
 dst=10.0.0.1:2889,
 length=20,
 src=10.0.0.1:48603,
 timestr=22:54:31:996068,
 zxid=8589934619,
)
Ack(
 dst=10.0.0.1:2889,
 length=20,
 src=10.0.0.1:48604,
 timestr=22:54:31:996316,
 zxid=8589934619,
)
Ack(
 dst=10.0.0.1:2889,
 length=20,
 src=10.0.0.1:48604,
 timestr=22:54:31:996318,
 zxid=8589934619,
)
Commit(
 dst=10.0.0.1:48603,
 length=20,
 src=10.0.0.1:2889,
 timestr=22:54:31:996193,
 zxid=8589934619,
)
Commit(
 dst=10.0.0.2:48604,
 length=20,
 src=10.0.0.1:2889,
 timestr=22:54:31:996195,
 zxid=8589934619,
)
Commit(
 dst=10.0.0.2:48605,
 length=20,
 src=10.0.0.1:2889,
 timestr=22:54:31:996442,
 zxid=8589934619,
)

### OS X ### 虽然没有人尝试在生产环境中运行此软件，但它可以用于开发的一些部分以及单元测试。如果您正在 OS X 上运行，请运行以下命令来安装正确的依赖项

$ pip install -r ./osx_requirements.txt

### 依赖项### * Python 2.7 (Py3K 即将推出) * ansicolors * dpkt-fix * hexdump * psutil>=2.1.0 * scapy==2.4.2 * six * twitter.common.app * twitter.common.collections * twitter.common.exceptions * twitter.common.http * twitter.common.log

项目详情

验证详情

这些详情已由PyPI 验证
维护者
Avatar for rgs from gravatar.com rgs Avatar for Yasumoto from gravatar.com Yasumoto

未验证详情

这些详情尚未由PyPI验证
项目链接
元数据
  • 许可： Apache软件许可协议（Apache）
  • 作者： Raul Gutierrez Segales
  • 标签 ZooKeeper, Sniffer, Stats
分类器

发布历史 发布通知 | RSS 源

此版本

0.2.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

下载文件

下载适合您的平台文件。如果您不确定选择哪个，请了解有关 安装软件包 的更多信息。

源代码发行版

zktraffic-0.2.0.tar.gz (50.9 kB 查看哈希值)

上传时间 源代码

构建发行版

zktraffic-0.2.0-py3-none-any.whl (82.3 kB 查看哈希值)

上传时间 Python 3

由以下机构支持

AWS AWS 云计算和安全赞助商 Datadog Datadog 监控 Fastly Fastly CDN Google Google 下载分析 Microsoft Microsoft PSF 赞助商 Pingdom Pingdom 监控 Sentry Sentry 错误日志 StatusPage StatusPage 状态页面