Handling NoneType Errors When Extending Lists in Python

When working with Python, especially with functions that return lists or other iterable objects, you might encounter a TypeError that says something like:

TypeError: 'NoneType' object is not iterable

This error occurs when you try to iterate over or extend a list using a value that turns out to be None. In Python, NoneType represents a null value, and it is not iterable, meaning you cannot loop over it or treat it like a list.

Let’s walk through an example that demonstrates this error and how to handle it properly.

Example Scenario: Extending Lists with NoneType Error

Consider the following Python function:

def create_consolidated_csv(bucket_name, folders, year, month, day, output_file):
    all_object_details = []  # A list to hold object details

    for folder in folders:
        # Get object details for each folder and specific date
        object_details = list_s3_objects_by_date(bucket_name, folder, year, month, day)
        
        # Try to extend the list with object details (error happens if object_details is None)
        all_object_details.extend(object_details)

    # Save details to a CSV file
    with open(output_file, mode='w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(["Object Name", "Last Modified"])  # Header
        writer.writerows(all_object_details)

This function attempts to fetch details about objects stored in an AWS S3 bucket, filter them by folder and date, and then write the data into a CSV file. The potential problem arises in this line:

all_object_details.extend(object_details)

If list_s3_objects_by_date returns None (instead of a list), you’ll get a TypeError because NoneType is not iterable.

How to Fix the NoneType Error

There are two common ways to handle this issue:

1. Check if the value is None before extending the list.
2. Ensure that the function always returns an empty list instead of None.

Option 1: Checking for None

Before extending the list, you can check if the object_details is None. If it is, skip that step:

def create_consolidated_csv(bucket_name, folders, year, month, day, output_file):
    all_object_details = []

    for folder in folders:
        object_details = list_s3_objects_by_date(bucket_name, folder, year, month, day)
        
        if object_details:  # Ensure object_details is not None
            all_object_details.extend(object_details)

    with open(output_file, mode='w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(["Object Name", "Last Modified"])
        writer.writerows(all_object_details)

Option 2: Returning an Empty List Instead of None

Alternatively, you can modify the list_s3_objects_by_date function to always return an empty list, even if no objects are found. This approach avoids the need for extra conditionals in the main function:

def list_s3_objects_by_date(bucket_name, folder, year, month, day):
    s3 = boto3.client('s3')
    prefix = f"{folder}/{year}/{month}/{day}/"
    
    try:
        response = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)
        if 'Contents' in response:
            return [(obj['Key'], obj['LastModified']) for obj in response['Contents']]
        else:
            return []  # Return an empty list if no contents are found
    except Exception as e:
        print(f"Error fetching objects: {e}")
        return []  # Ensure an empty list is returned even in case of an error

Conclusion

When handling iterable objects in Python, it’s essential to safeguard against situations where a function might return None. You can either check for None before attempting to extend or iterate over a list, or modify your functions to always return an empty list, ensuring your code doesn’t throw a TypeError. This small adjustment can make your code more robust and error-resistant when dealing with dynamic data sources like AWS S3.