Python How to add custom fields to logging formatter

eye-catch Python

logging module is a useful module for logging. The log format somehow needs to be defined to make it more readable. The format can be defined easily by using logging.Formatter(format) with pre-defined fields. However, those pre-defined fields might not be enough for our format. How can we add custom fields there?

Sponsored links

How to define formatter only with pre-defined fields

Let’s check how to set formatter with pre-defined fields. This is the code.

import logging
import logging.handlers
import os
from datetime import datetime
from path import Path

def generate_logger_without_custom_formatter(log_path: str):
    logger = logging.getLogger("ABC_APP")

    log_path_dir = Path(log_path).parent
    os.makedirs(log_path_dir, exist_ok=True)

    rotatingHandler = logging.handlers.RotatingFileHandler(
    formatter = "%(asctime)s <%(levelno)d>[%(levelname)s] %(name)s %(filename)s line: %(lineno)d %(message)s"

    return logger

def write_log(logger: logging.Logger):

log_path_dir = Path(__file__).parent.joinpath("temp")
log_path = Path(log_path_dir).joinpath("test.log")
logger_without = generate_logger_without_custom_formatter(str(log_path))

The formatter can be set by setFormatter(). Since the pre-defined fields can be used, we can easily create our preferred format.

The following log is written as a result.

2023-11-21 19:30:02,099 <10>[DEBUG] ABC_APP line: 125 message
2023-11-21 19:30:02,099 <20>[INFO] ABC_APP line: 126 message
2023-11-21 19:30:02,099 <30>[WARNING] ABC_APP line: 127 message
2023-11-21 19:30:02,099 <40>[ERROR] ABC_APP line: 128 message
2023-11-21 19:30:02,099 <50>[CRITICAL] ABC_APP line: 129 message

It is enough to use only pre-defined fields in many cases but I needed to use custom fields because a log analysis application couldn’t handle this format. Another application supports the format but my application doesn’t. Then, I had to adapt it.

Sponsored links

Define custom fields in formatter

What I needed is the following.

  • Change the time format to DD-MM-YYYY HH:MM:SS.FFF
  • Change Log level number
  • Show the parent directory for the module

I tried to use LoggerAdapter at first but it’s impossible to map the log level to another value. So what we can do is to extend logging.Formatter.

    logging.CRITICAL: 5,
    logging.FATAL: 5,
    logging.ERROR: 4,
    logging.WARNING: 3,
    logging.WARN: 3,
    logging.INFO: 2,
    logging.DEBUG: 1,
    logging.NOTSET: 0,

class LoggingFormatter(logging.Formatter):
    CustomFormatter provides a format that supports TNCanalyzer. The format is same as advanced-touch.

    def __init__(self) -> None:
        formatter = "%(custom_time)s <%(custom_log_level)s>[%(levelname)s] %(name)s %(source_path)s line: %(lineno)d %(message)s"

    def format(self, record: logging.LogRecord) -> str:
        record.custom_time ="%d.%m.%Y %H:%M:%S.%f")[:-3]
        record.custom_log_level = _LEVEL_MAP.get(record.levelno, 0)
        record.source_path = self.__get_source_path(record)

        return super().format(record)

    def __get_source_path(self, record: logging.LogRecord):
            parent_dir_name = os.path.basename(os.path.dirname(record.pathname))
            return os.path.join(parent_dir_name, record.filename)
        except BaseException:  # pylint: disable=:broad-exception-caught
            return ""

We can access the pre-defined fields in format(). Therefore, we can define custom fields by using those values. Assign a value to record.variable_name. Then, it can be used in formatter.

This is the result.

21.11.2023 19:30:10.301 <1>[DEBUG] NICE_APP src/ line: 125 message
21.11.2023 19:30:10.301 <2>[INFO] NICE_APP src/ line: 126 message
21.11.2023 19:30:10.301 <3>[WARNING] NICE_APP src/ line: 127 message
21.11.2023 19:30:10.301 <4>[ERROR] NICE_APP src/ line: 128 message
21.11.2023 19:30:10.301 <5>[CRITICAL] NICE_APP src/ line: 129 message

The format is different from the previous one.


Copied title and URL