为Kubernetes pod提供相对简便的JSON日志记录

导航

验证详情

这些详情已由PyPI验证
维护者
Avatar for curella.org from gravatar.com curella.org Avatar for fwiles from gravatar.com fwiles

未验证详情

这些详情尚未由PyPI验证
项目链接
元数据

项目描述

jslog4kube (python/JSON日志记录工具，用于Kubernetes pod/容器)

为什么？

  • 为python提供JSON-to-stdout的配置
  • 为gunicorn提供相同的JSON-to-stdout配置
  • 因为创建复杂的日志收集器配置来处理当时其他人认为的好主意是徒劳的。

我们希望使我们的客户和我们自己更容易启动发出良好日志的新项目。

让事情变得简单

两个环境变量配置此模块

  • KUBE_META：指定Kubernetes downward-API 卷的挂载点（默认：/etc/meta）
  • KUBE_META_ENV_PREFIX：任何针对此pod日志记录包括在内的环境变量的目标文本前缀（默认：X）

这些位已在python 2.7.13、3.5.3和3.6.1上经过验证，可以正常工作

from logging.config import dictConfig
from jslog4kube import LOGGING

dictConfig(LOGGING)

从Django

项目 settings.py

from jslog4kube import LOGGING

gunicorn

gunicorn.conf

access_log_format = 'remote!%({X-Forwarded-For}i)s|method!%(m)s|url-path!%(U)s|query!%(q)s|username!%(u)s|protocol!%(H)s|status!%(s)s|response-length!%(b)s|referrer!%(f)s|user-agent!%(a)s|request-time!%(L)s'
accesslog = '-'
logger_class = 'jslog4kube.GunicornLogger'

gunicorn CLI

gunicorn -c /path/to/gunicorn.conf [rest of your options here]

这将产生以下类型的输出

{
  "asctime": "2017-07-12 16:07:34,624",
  "message": "Booting worker with pid: 6801",
  "name": "gunicorn.error",
  "created": 1499893654.6243172,
  "filename": "glogging.py",
  "module": "glogging",
  "funcName": "info",
  "lineno": 247,
  "msecs": 624.3171691894531,
  "pathname": "/home/gladiatr/.virtualenvs/json-logs-for-kube/lib/python3.6/site-packages/gunicorn/glogging.py",
  "process": 6801,
  "processName": "MainProcess",
  "relativeCreated": 70.62673568725586,
  "thread": 140275859264576,
  "threadName": "MainThread",
  "levelname": "INFO",
  "x_node_name": "ip-10-70-59-190.eu-central-1.compute.internal",
  "x_sa_name": "default",
  "x_pod_ip": "100.96.1.11",
  "build": "5000",
  "builder": "Stephen Spencer",
  "image": "gladiatr72/kube-demo",
  "version": "1.0.2",
  "app": "kube-demo",
  "env": "dev",
  "pod-template-hash": "2802633501",
  "something": "else"
}
{
  "asctime": "2017-07-12 21:08:16,354",
  "message": "in view: Chameleon",
  "name": "efk.views",
  "created": 1499893696.3544216,
  "filename": "views.py",
  "module": "views",
  "funcName": "Chameleon",
  "lineno": 12,
  "msecs": 354.42161560058594,
  "pathname": "/home/gladiatr/git/json-logs-for-kube/demo/efk/views.py",
  "process": 6800,
  "processName": "MainProcess",
  "relativeCreated": 41800.73118209839,
  "thread": 140275726399232,
  "threadName": "<concurrent.futures.thread.ThreadPoolExecutor object at 0x7f947f4b0828>_0",
  "levelname": "INFO",
  "x_node_name": "ip-10-70-59-190.eu-central-1.compute.internal",
  "x_sa_name": "default",
  "x_pod_ip": "100.96.1.11",
  "build": "5000",
  "builder": "Stephen Spencer",
  "image": "gladiatr72/kube-demo",
  "version": "1.0.2",
  "app": "kube-demo",
  "env": "dev",
  "pod-template-hash": "2802633501",
  "something": "else",
  "additional data": "whee"
}
{
  "asctime": "2017-07-12 21:08:16,369",
  "message": "(access record)",
  "name": "gunicorn.access",
  "created": 1499893696.3695881,
  "filename": "glogging.py",
  "module": "glogging",
  "funcName": "access",
  "lineno": 327,
  "msecs": 369.58813667297363,
  "pathname": "/home/gladiatr/.virtualenvs/json-logs-for-kube/lib/python3.6/site-packages/gunicorn/glogging.py",
  "process": 6800,
  "processName": "MainProcess",
  "relativeCreated": 41815.89770317078,
  "thread": 140275726399232,
  "threadName": "<concurrent.futures.thread.ThreadPoolExecutor object at 0x7f947f4b0828>_0",
  "levelname": "INFO",
  "x_node_name": "ip-10-70-59-190.eu-central-1.compute.internal",
  "x_sa_name": "default",
  "x_pod_ip": "100.96.1.11",
  "build": "5000",
  "builder": "Stephen Spencer",
  "image": "gladiatr72/kube-demo",
  "version": "1.0.2",
  "app": "kube-demo",
  "env": "dev",
  "pod-template-hash": "2802633501",
  "something": "else",
  "access": {
    "remote": "10.0.1.195",
    "method": "GET",
    "url-path": "/",
    "query": "",
    "username": "-",
    "protocol": "HTTP/1.0",
    "status": "200",
    "response-length": "140",
    "referrer": "-",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
    "request-time": "0.019269"
  }
}

示例Kubernetes部署

apiVersion: extensions/v1beta1
kind: Deployment
metadata:         <<-- This is not the metadata you are looking for
  name: kube-demo
  labels:
    project: kube-demo
    environment: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-demo
      env: dev
  template:
    metadata:
      labels:
        app: kube-demo
        env: dev
        something: else
      annotations:
        build: "5000"
        builder: "Stephen Spencer"
        image: "gladiatr72/kube-demo"
        version: "1.0.2"

目前仅支持通过卷暴露metadata.(labels|annotations)

(原因，对吧？)

    spec:
      volumes:
          name: podinfo
          downwardAPI:
            items:
              - path: labels
                fieldRef:
                  fieldPath: metadata.labels
              - path: annotations
                fieldRef:
                  fieldPath: metadata.annotations
      containers:
        - name: kube-demo
          image: gladiatr72/kube-demo:1.0.2
          volumeMounts:
            -
              name: run
              mountPath: /run
            -
              name: podinfo
              readOnly: true
              mountPath: /etc/meta  <<-- KUBE_META must equal this

环境变量前缀

(或者：如何在将数据存储密码垃圾邮件到您的日志系统时避免看起来像个白痴)

您可以使用任何字母或序列作为前缀，只要它与KUBE_META_ENV_PREFIX的值匹配即可。

          env:
            - name: MEMCACHE_HOST
              value: unix:/run/memcache.sock
            - name: DJANGO_SETTINGS_MODULE
              value: "revsys.settings.dev"
            - name: DJANGO_FQDN
              value: kube-demo.dev.revsys.com
            - name: *X_NODE_NAME
              valueFrom:
                fieldRef:
                 fieldPath: spec.nodeName
            - name: *X_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: *X_SA_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.serviceAccountName
            - name: REDIS_PASSWORD  (oh, nos!)
              valueFrom:
                secretKeyRef:
                  name: redis
                  key: pass
          ports:
            - containerPort: 8000

    * unmagical prefix set in KUBE_META_ENV_PREFIX

日志配置

它只是一个标准的python字典。最明显要更改的是处理器定义。

from jslog4kube import LOGGING

LOGGING_HANDLERS = {
    'mypackage': {
        'handlers': ['json-stdout'],
        'formatters': ['json'],
        'propagate’: False,
        'level’: 'ERROR',
    }
}

LOGGING['handlers'].update(LOGGING_HANDLERS)

设置python/django应用以正确记录日志

这个提供的 LOGGING 字典设置了一些要记录的“通用”事项，但如果你想要包括自己的 Python 库或 Django 应用，则需要指定它们。要指定一个名为 'foo' 的 Django 应用，你只需调整 LOGGING 字典，如下所示

LOGGERS = {
    'foo': {
        'handlers': ['json-stdout'],
        'formatters': ['json'],
        'propagate': True,
        'level': 'INFO',
    }
}

LOGGING['loggers'].update(LOGGERS)

dictConfig(LOGGING)

或者，如果你想以 DEBUG 级别记录所有内容，你可以设置一个空（也称为默认）记录器

LOGGERS = {
    '': {
        'handlers': ['json-stdout'],
        'formatters': ['json'],
        'propagate': True,
        'level': 'DEBUG',
    }
}

LOGGING['loggers'].update(LOGGERS)

dictConfig(LOGGING)

用法

这是正常的 Python 日志记录，所以你可以做一些简单的事情，比如下面的 info 调用，或者更复杂的事情，比如 debug 调用，并添加额外的数据到日志信息中

import logging

logger = logging.getLogger(__name__)

logger.info("Simple log message")

foo = 12
bar = 'something else'
logger.debug("More complicated message", extra={
  "foo": foo,
  "bar": bar,
})

需要帮助？

REVSYS 可以帮助您进行 Python、Django 和基础设施项目。如果您对这个项目有问题，请打开一个 GitHub 问题。如果您喜欢我们，并想跟踪我们的动态，这里可以找到我们在网络上的位置

项目详情

验证详情

这些详情已由PyPI验证
维护者
Avatar for curella.org from gravatar.com curella.org Avatar for fwiles from gravatar.com fwiles

未验证详情

这些详情尚未由PyPI验证
项目链接
元数据

发布历史 发布通知 | RSS 源

本版本

1.0.6

1.0.5

1.0.4

下载文件

下载适用于您的平台的文件。如果您不确定选择哪个，请了解更多关于 安装包 的信息。

源分布

jslog4kube-1.0.6.tar.gz (11.0 kB 查看散列)

上传时间 源

构建分布

jslog4kube-1.0.6-py2.py3-none-any.whl (10.9 kB 查看散列)

上传时间 Python 2 Python 3

由以下赞助