Did You Know ... Python Edition

Did you know official support for Python 2 is over?

That said, we have Python 3 available on Memex by loading the module, "python/3.6.7". This Python version includes conda, R, Jupyter, IntelMPI, and many other packages. Most Python packages can be installed by sending your request to memexsupport@carnegiescience.edu.

Did you know you can install your own Python packages using conda?

For example, if you’d like the latest python version available, 3.8.1, first search for it here first (mainly conda-forge channel) or directly from the command line with:

module purge \
#only needed to ensure a clean environment
module load python/3.6.7
conda search python \
#latest should be 3.8.1 as of last week

and use it to set up your own conda environment (both anaconda.org and the command line shows the proper channel, "-c", to use below).

conda create -p /lustre/scratch/$USER/.envs/py38 \
-c conda-forge python=3.8.1

The above command assumes /lustre/scratch/$USER/.envs is where you want the conda environment, py38, to reside. It has to be in a directory you own.

To activate any conda environment (using example, py38, above):

source activate py38

To deactivate any conda environment:

source deactivate

To remove an entire conda environment:

conda-env remove -n py38

Did you know you can list existing conda environments already on Memex?

To list the available conda environments (still using the module, python/3.6.7), simply type:

conda env list

if you can see them, you can activate them but if you don’t have write permission, you cannot install, remove, or modify them. You can however, clone them (make the new name unique–> "foo", not the clone name --> “foo2”):

conda create -p /lustre/scratch/$USER/.envs/foo --clone foo2

Did you know you can list any installed Python package, in the base conda environment (activated by module load python/3.6.7) or a custom conda environment (activated by source activate foo)?

conda list packagename #replace packagename with real package name

where “packagename” is a placeholder for a real package (like python, or r-base). Of course,

conda list

lists all packages installed in your current environment.

Did you know there are other useful commands (#replace packagename and X with real values):

conda search packagename=X.X.X --info #where X.X.X=version_number of packagename
conda clean #removes downloaded packages and caches
conda remove packagename #to remove package
conda list --revisions #to identify a revision to rollback to
conda install --revision X #rollback to revision number, X

Type conda -h and/or conda-env -h for more options.

Note: To manage where your cache directory is located (includes tar balls and downloaded packages), you can set the following in your bashrc:

echo CONDA_PKGS_DIRS=/lustre/scratch/$USER/.envs/pkgs >> \
$HOME/.bashrc
source $HOME/.bashrc

Or in your current environment only with:

export CONDA_PKGS_DIRS=/lustre/scratch/$USER/.envs/pkgs