logmerger

logmerger是一个TUI，用于查看多个日志文件的合并显示，按时间戳合并。

给定以下两个日志文件

# log1.txt
2023-07-14 08:00:01 WARN   Connection lost due to timeout
2023-07-14 08:00:04 ERROR  Request processed unsuccessfully
Something went wrong
Traceback (last line is latest):
    sample.py: line 32
        divide(100, 0)
    sample.py: line 8
        return a / b
ZeroDivisionError: division by zero
2023-07-14 08:00:06 INFO   User authentication failed
2023-07-14 08:00:08 DEBUG  Starting data synchronization
2023-07-14 08:00:11 INFO   Processing incoming request

# log2.txt
2023-07-14 08:00:01 INFO   Request processed successfully
2023-07-14 08:00:03 INFO   User authentication succeeded
2023-07-14 08:00:06 DEBUG  Starting data synchronization
2023-07-14 08:00:08 INFO   Processing incoming request
2023-07-14 08:00:11 DEBUG  Performing database backup
2023-07-14 08:00:14 WARN   Invalid input received: missing required field

此命令

logmerger log1.txt log2.txt

显示以下可浏览的合并显示（使用textual Python库启用）

按'h'键获取交互式显示中所有快捷键的帮助。

使用--output -将合并后的日志发送到stdout

  Timestamp                 Files/Log1.Txt                        Files/Log2.Txt
 ────────────────────────────────────────────────────────────────────────────────────────────────────
  2023-07-14 08:00:01.000   WARN   Connection lost due to         INFO   Request processed
                            timeout                               successfully
  2023-07-14 08:00:03.000                                         INFO   User authentication
                                                                  succeeded
  2023-07-14 08:00:04.000   ERROR  Request processed
                            unsuccessfully
                             Something went wrong
                             Traceback (last line is latest):
                                 sample.py: line 32
                                     divide(100, 0)
                                 sample.py: line 8
                                     return a / b
                             ZeroDivisionError: division by zero                           
  2023-07-14 08:00:06.000   INFO   User authentication            DEBUG  Starting data
                            failed                                synchronization
  2023-07-14 08:00:08.000   DEBUG  Starting data                  INFO   Processing incoming request
                            synchronization
  2023-07-14 08:00:11.000   INFO   Processing incoming request    DEBUG  Performing database backup
                            INFO   Processing incoming request
                            (a little more...)
  2023-07-14 08:00:14.000   DEBUG  Performing database backup     WARN   Invalid input received:
                                                                  missing required field

安装

从PyPI安装logmerger

pip install logmerger

这将安装logmerger作为shell控制台命令，然后您可以直接运行它而无需调用python。

要添加对合并pcap文件的支持，请使用以下命令安装

pip install logmerger[pcap]

命令行参数

logmerger -h将显示以下帮助

usage: logmerger [-h] [--interactive] [--inline] [--output OUTPUT]
                 [--start START] [--end END] [--autoclip]
                 [--ignore_non_timestamped] [--width WIDTH]
                 [--line_numbers] [--show_clock]
                 [--csv CSV] [--encoding ENCODING]
                 [--timestamp_format [TIMESTAMP_FORMATS ...]]
                 [--demo]
                 [files ...]

positional arguments:
  files                 log files to be merged

options:
  -h, --help            show this help message and exit
  --interactive, -i     show merged output using interactive TUI browser (default)
  --inline              show merged log data as inline merge
  --output OUTPUT, -o OUTPUT
                        save merged output to file ('-' for stdout; files ending in '.md' are saved
                        using Markdown)
  --start START, -s START
                        start time to select time window for merging logs
  --end END, -e END     end time to select time window for merging logs
  --autoclip, -ac       clip merging to time range of logs in first log file
  --ignore_non_timestamped
                        ignore log lines that do not have a timestamp
  --width WIDTH, -w WIDTH
                        total screen width to use for interactive mode (defaults to current screen
                        width)
  --line_numbers, -ln   add line number column
  --show_clock, -clock  show running clock in header
  --csv CSV, -csv CSV   save merged logs to CSV file
  --encoding ENCODING, -enc ENCODING
                        encoding to use when reading log files (defaults to the system default encoding)
  --timestamp_format [TIMESTAMP_FORMATS ...]
                        custom timestamp format
  --demo                Run interactive demo
  
Start and end timestamps to clip the given files to a particular time window can be
given in `YYYY-MM-DD HH:MM:SS.SSS` format, with trailing milliseconds and seconds
optional, and "," permissible for the decimal point. A "T" can be included between
the date and time to simplify entering the timestamp on a command line (otherwise
would require enclosing in quotes because of the intervening space). These command
line values do not need to match the timestamp formats in the log files.

These values may also be given as relative times, such as "15m" for "15 minutes ago".
Valid units are "s", "m", "h", and "d".

支持的文件类型

日志数据通常从文本日志文件中提取，但也可以从其他相关日志文件中提取。

合并

日志文件通过交错每个日志文件的日志行（基于每行日志的时间戳）进行合并。 logmerger 尝试使用不同的时间戳格式，直到为每个输入文件找到匹配的格式。支持的格式有：

未标记时间戳的包含多个行（如跟踪堆栈）的日志行将与前一个标记时间戳的行合并（见上面的示例）。

安全联系方式

要报告安全漏洞，请使用 Tidelift 安全联系方式。 Tidelift 将协调修复和披露。