AWS Glue Job Fails Due to Dependency Version Conflict: How to Fix It

The script runs fine locally. You deploy it to Glue, add one library, and the job dies before your code does anything useful:

ValueError: numpy.dtype size changed, may indicate binary incompatibility.
Expected 96 from C header, got 88 from PyObject

Or a plain ImportError, or PythonModuleInstaller failing at launch. The cause is the same: you’re not installing into an empty environment. You’re installing on top of a runtime that already has dozens of preinstalled packages, and pip is happy to upgrade one of them underneath the others.

Why it happens

Each Glue version ships a fixed Python version and a fixed set of preinstalled libraries (pandas, numpy, pyarrow, boto3, and more). When you pass --additional-python-modules, Glue runs pip3 install on each node at startup.

If your library declares numpy>=2, pip upgrades numpy. The preinstalled pandas or pyarrow was compiled against the old numpy ABI, and the import fails. The same applies to any package with C extensions.

The library you added isn’t broken. The combination is.

Step 1: know your runtime

Pin the Glue version and look up its preinstalled packages in the docs before choosing versions. Glue 4.0 runs Python 3.10, Glue 5.0 runs Python 3.11, and the package lists differ.

The fastest way to see the real environment is to print it from a throwaway job:

import subprocess
print(subprocess.run(["pip3", "list"], capture_output=True, text=True).stdout)

Or reproduce locally with the official Glue container image:

$ docker run -it --rm public.ecr.aws/glue/aws-glue-libs:5 bash
$ pip3 list | grep -Ei 'numpy|pandas|pyarrow'

Step 2: use the right parameter

There are three ways to ship Python code to Glue, and they’re not equivalent:

--extra-py-files is for your own pure-Python modules. Using it for third-party packages with transitive dependencies or compiled code is a common source of ImportError.

Step 3: pin everything that matters

Pin your library and the shared dependencies it would otherwise upgrade:

--additional-python-modules  my-lib==1.4.2,numpy==1.26.4

AWS recommends pinned versions here for a reproducible environment. In practice, pin to the versions already present in the runtime unless you have a reason not to.

To find a compatible version of your library, test in the Glue container:

$ pip3 install "my-lib==1.4.2" --dry-run
$ pip3 check

pip check reports broken requirements across the whole environment. If it’s clean in the container, it’s very likely clean in the job.

Step 4 (Glue 5.0+): use requirements.txt

Glue 5.0 accepts a requirements.txt from S3:

--python-modules-installer-option  -r
--additional-python-modules        s3://my-artifacts/glue/requirements.txt

# requirements.txt
my-lib==1.4.2
numpy==1.26.4
s3://my-artifacts/wheels/internal_pkg-0.3.0-py3-none-any.whl

This is cleaner than a comma-separated string once you have more than a few packages. It doesn’t work on Glue 4.0, where the job fails at launch.

Step 5: build wheels for the target platform

If you ship your own wheels with compiled code, they must match the Glue runtime: Linux x86_64 (manylinux) and the right CPython ABI (cp310 for Glue 4.0, cp311 for Glue 5.0). A wheel built on an M-series Mac won’t install.

Build inside the Glue container, or use:

$ pip3 download my-lib==1.4.2 \
    --platform manylinux2014_x86_64 \
    --python-version 3.11 \
    --only-binary=:all: \
    -d wheels/

Pure-Python wheels (py3-none-any) avoid this problem entirely.

Related failure: no network access

If the job runs inside a VPC (Glue connection) without a NAT gateway or a PyPI mirror, pip3 install from PyPI fails with a timeout, not a version error. Ship wheels from S3 (with an S3 VPC endpoint) or add NAT.

Checklist

• Pin the Glue version; check its preinstalled packages
• Use --additional-python-modules, not --extra-py-files, for third-party libs
• Pin your library and the shared deps it touches (numpy, pandas, pyarrow)
• Validate with pip3 check in the official Glue image
• On Glue 5.0+, move to requirements.txt
• Build compiled wheels for manylinux x86_64 and the right CPython version

References

• Using Python libraries with AWS Glue
• AWS Glue job parameters
• AWS Glue versions
• Developing and testing AWS Glue jobs locally using a Docker image
• pip check