Python Package Installation Method Comparison

How to use pip install

The ASU Supercomputers offers a python-environment manager, mamba, which should help improve the reliability of python installations and sidestep version conflicts.

This page describes the inner-workings of installing packages via various python methods (pip and mamba install) and the locations it installs files to.

info

Improper python package installation can result in Jupyter stopping working with "Bad State", as they introduce conflicts with user-installed python packages. See further below for the fix.

Differences between installation methods

Environment Activated?	Installer	Command	Default Install Location	Note
No mamba/no env	pip	pip install package_name	/usr/local/lib/pythonX.X/site-packages ...then ~/.local/lib/pythonX.X/site-packages	This location is not user-writable, so python may default to installing to your $HOME. Avoid this route.
No mamba/no env	pip (with --user)	pip install --user package_name	~/.local/lib/pythonX.X/site-packages	This location is user-writable, but installs packages in a location Jupyter Notebook/Lab uses on the Web Portal. Avoid this route.
source activate YYYY	mamba	mamba install package_name	~/.conda/envs/<envname>/lib/pythonX.X/site-packages	This location is user-writable and will be installed properly via mambas safeguards. PREFERRED route.
source activate YYYY	pip	pip install package_name	~/.conda/envs/<envname>/lib/pythonX.X/site-packages	This location is user-writable and will be installed properly via mambas safeguards. This is an ACCEPTABLE route for packages that cannot be installed via mamba.
source activate YYYY	pip (with --user)	pip install --user package_name	~/.local/lib/pythonX.X/site-packages	This location is user-writable, but installs packages in a location Jupyter Notebook/Lab uses on the Web Portal. Avoid this route.

MAIN TAKEAWAYS:

info

1. Prefer mamba install everywhere, where possible.
2. Never use pip --user install
3. Only use pip install after source activating your environment.

Deleting conflicting packages

When using Python3.11, the pip installed packages will be located in ~/.local/lib/python3.11/site-packages. When using Python3.9, they will be located at ~/.local/lib/python3.9/site-packages and every version will follow this pattern.

Therefore, if you have installed pip packages using python, all other environments you have created that share the same python version will be able to use those pip packages....but conversely, all the other environments that share the same python version might be broken by those pip packages.

This is the primary reason for mamba environments and the installation in to mamba via mamba install.

Fixing this involves deleting these site-packages directories using rm -rf.

Fixing Jupyter Notebook in a "Bad State"

Fixing Jupyter Notebook that is having trouble starting or reporting "Bad State" can be done by removing (or renaming) the packages located in your ~/.local/lib directory.

By removing files that are universally installed to your $HOME directory, both Jupyter and conda environments will no longer have access to these files.

For packages installed here, they should be installed directly into the environment using the guidance above.

[asurite@login01:~]$ cd ~/.local/lib/
[asurite@login01:~/.local/lib]$ ll
total 80
drwxr-x--- 3 asurite grp_usergrp 31 Jan 13 12:20 python3.11/
[asurite@login01:~/.local/lib]$ mv python3.11/ python3.11.old
[asurite@login01:~/.local/lib]$ ll
total 80
drwxr-x--- 3 asurite grp_usergrp 31 Jan 13 12:20 python3.11.old/

Steps to Fix Jupyter Issues

1. Go to the $HOME package installation dir: cd ~/.local/lib/
2. Identify any python packages directory--you may see multiple versions here, e.g., python3.11 or python2.7
3. Rename the directories to have a new suffix, making them not get loaded automatically, e.g., .old or .bak.
4. Install these packages to a mamba environment using the table above.

Only SBATCH submissions seem impacted

If only SBATCH submissions seem impacted regarding use of your mamba environment, one likely cause is environment cross-pollination of variables. That means that the session you use to submit your job has variables set that are incompatible with your jobscript that may have run successfully before.

To ensure this is not the case--and is often a useful and positive change, anyway--ensure that variables do not cross over with the following line added to your SBATCH script:

#SBATCH --export=None

Technical details

Background about Python Libraries Locations

When running python, python2, or python3, additional libraries are identified and python attempts to load these into the current session from /home/<asurite>/.local/lib/python<version>/site-libraries.

This is the default behavior of python, and in many cases for workstations, a dependable and time-saving functionality.

However, this default behavior is potentially disruptive when using Jupyter, which fully contains all the necessary packages in a curated, version-locked manner.

Background about Mamba-contained libraries

When using mamba create and mamba install, libraries are placed in to /home/<asurite>/.conda/envs/<envname>. This means you can have multiple environments that are completely independent of eachother which maybe both include the same packages, but do not interfere with eachother and can have different (and potentially conflicting) versions.

The selection of the environment is done with source activate and this ensures all mamba installand pip install remain safely contained within.

Additional Help

If you require further assistance, contact the Research Computing Team:

• Ticket-based support via RTO Request Help Portal.
• Slack support via the #rc-support channel in the ASU Research Computing workspace.
• Weekly office hours for one-on-one assistance.

We also offer Educational Opportunities and Workshops.