Python logging: levels, file output, and separate console and file handlers

print() is fine for a script’s normal output, but it isn’t a good way to report what the program is doing: you can’t turn it down, send it to a file, or tell an error from a progress message. Python’s standard logging module solves that. The Python docs put it this way:

This module defines functions and classes which implement a flexible event logging system for applications and libraries.

The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log can include your own messages integrated with messages from third-party modules.

This guide goes from a one-line warning to a logger that writes everything to a file and only the important messages to the console.

When to log and when to print

The logging HOWTO maps each task to the right tool:

For a longer argument against print debugging, including a logging config kept in a YAML file, see Joe Freeman’s Log! Don’t Print! Use the Python logging library and the accompanying video.

Your first log message

import logging
logging.warning("Ey, this is a logging warning")

WARNING:root:Ey, this is a logging warning

The output shows the level, the logger name (root) and the message.

Logging levels

Each message has a level, and handlers and loggers drop messages below their configured level (logging levels):

The root logger’s default level is WARNING, which is why logging.info() prints nothing until you configure logging. Applications commonly set INFO and above for normal runs and DEBUG when investigating.

To color the console output by level, see the Stack Overflow thread How can I color Python logging output?

Log to a file with basicConfig

The quickest way to send everything to a file is logging.basicConfig (Logging to a file):

import logging

logging.basicConfig(filename='example.log', encoding='utf-8', level=logging.DEBUG)

logging.warning("Ey, this is a logging warning")
logging.error("Ey, this is a logging error")
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
logging.error('And non-ASCII stuff, too, like Øresund and Malmö')

With level=logging.DEBUG, all six messages go to example.log, and nothing goes to the console. The encoding argument requires Python 3.9 or later.

Different levels for console and file

basicConfig configures one destination. When you want detail in a file but a quiet console, create a named logger with one handler per destination:

• A file handler that records DEBUG and above.
• A console handler that shows INFO and above (INFO, WARNING, ERROR, CRITICAL).

The example uses yfinance to check which tickers trade below a price threshold:

pip install yfinance

import yfinance as yf
import json
import logging

# Configure logging
logger = logging.getLogger("tickers_below_five")
logger.setLevel(logging.DEBUG)  # Capture all messages

# File handler for logging DEBUG and above messages
file_handler = logging.FileHandler('logs\\below_five.log', encoding='utf-8')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))

# Console handler for logging INFO and above messages
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(logging.Formatter('%(levelname)s - %(message)s'))

# Adding handlers to the logger
logger.addHandler(file_handler)
logger.addHandler(console_handler)

# Main logic
json_file_path = "sources\\company_tickers.json"
price_threshold = 5
low_price_stocks = []

try:
    with open(json_file_path, 'r') as j:
        contents = json.loads(j.read())

        for key, value in contents.items():
            cik_str = value.get('cik_str')
            ticker = value.get('ticker')
            title = value.get('title')

            try:
                stock = yf.Ticker(ticker)
                hist = stock.history(period="1d")
                
                if hist.empty:
                    logger.warning(f"Key:{key}, Ticker: {ticker}, Title: {title}, Price: Data not available")
                    continue
                
                price = hist['Close'].values[0]
                logger.info(f"Key:{key}, Ticker: {ticker}, Title: {title}, Price: {price}")

                if price < price_threshold:
                    low_price_stocks.append((title, ticker, price))

            except Exception as e:
                logger.error(f"Key:{key}, Ticker: {ticker}, Title: {title}, Error: {e}")

except Exception as e:
    logger.error(f"Error reading JSON file: {e}")

# Final log output
logger.info(f"\nLow price stocks: {low_price_stocks}")

How it works:

1. The logger is set to DEBUG, so it passes every message on to its handlers; each handler then applies its own level.
2. The file handler writes DEBUG and above to logs\below_five.log with timestamps.
3. The console handler prints INFO and above without timestamps.
4. The main loop reads a JSON file of company tickers, fetches the last day of prices for each with yfinance, logs the price (INFO), a missing-data WARNING, or an ERROR if the fetch fails, and collects stocks below price_threshold.

Console output:

INFO - Key:123, Ticker: XYZ, Title: Example, Price: 4.75
WARNING - Key:456, Ticker: ABC, Title: Another, Price: Data not available
ERROR - Key:789, Ticker: DEF, Title: Another Example, Error: Data fetch failed
INFO - 
Low price stocks: [('Example', 'XYZ', 4.75)]

Log file (below_five.log), with timestamps:

2024-09-29 12:34:56,789 - INFO - Key:123, Ticker: XYZ, Title: Example, Price: 4.75
2024-09-29 12:34:57,000 - WARNING - Key:456, Ticker: ABC, Title: Another, Price: Data not available
2024-09-29 12:34:58,123 - ERROR - Key:789, Ticker: DEF, Title: Another Example, Error: Data fetch failed

Any logger.debug(...) calls you add would appear only in the file.

Gotchas

• A handler’s level is a minimum, not a filter for specific levels. setLevel(logging.INFO) on the console handler shows ERROR and CRITICAL too. To show only some levels, attach a logging.Filter.
• The log directory must exist. FileHandler doesn’t create folders; a missing logs directory raises FileNotFoundError when the handler is created.
• Backslash paths aren’t portable. On Linux and macOS, 'logs\\below_five.log' creates a file literally named logs\below_five.log in the current directory. Use os.path.join("logs", "below_five.log") or pathlib.Path("logs") / "below_five.log".
• Duplicate lines from root propagation. Calling logging.warning() (or basicConfig) adds a handler to the root logger. Messages from a named logger then propagate to it and print twice, once in each format, and root’s handler prints DEBUG too. Set logger.propagate = False on the named logger, or don’t mix module-level logging.* calls with your own logger.
• basicConfig runs once. If the root logger already has handlers, a later basicConfig call does nothing.

Related

• Can you perform data grouping directly with the yfinance API? covers what to do with the prices once they’re fetched.
• Automating SQL script execution and logging with batch scripts logs sqlcmd results from a Windows batch file.
• How to set CloudWatch log retention policies with Terraform covers what happens to logs once they leave your machine.