Upgrading All Your Pip Packages
Pip, the package manager (and its package repository PyPI) is one of the most important parts of the Python ecosystem. Arguably, it’s the most important part of why Python is so popular. The fact that it’s so easy to import a module into Python and easily use its features would be greatly hindered if the process of actually getting all those cool modules wasn’t easy. Pip is one of the foundational tools in the Python ecosystem and is really cool to use.
Which makes it even harder to understand why it’s missing some very key and - I would argue - basic features.
Take, for instance, upgrading all your packages.
Let’s look at brew, the awesome “missing package manager for Mac”. It is another way cool package manager, that also has a really cool and taken for granted feature:
brew upgrade
BOOM
Now all your packages managed by brew are updated. Simple.
Ok, that’s a package manager for an OS, so not quite the same (although I’m not sure if it matters or not). So let’s look at another example, the node.js package manager, npm:
npm update
WOW, yet another fine example of using only 2 commands strung together to easily update all your npm-managed node.js packages. Quite an amazing feat of technology. Since we’re talking about pip, let’s take a look at how to upgrade all your modules at once:
pip help
Usage:
pip <command> [options]
Commands:
install Install packages.
download Download packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
search Search PyPI for packages.
wheel Build wheels from your requirements.
hash Compute hashes of package archives.
completion A helper command used for command completion
help Show help for commands.
Ok…I don’t see an update or upgrade command. Let’s try install and see if it has any subcommands that could work.
pip help install
Install Options:
-c, --constraint <file> Constrain versions using the given constraints file. This option can be used multiple times.
-e, --editable <path/url> Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
-r, --requirement <file> Install from the given requirements file. This option can be used multiple times.
-b, --build <dir> Directory to unpack packages into and build in.
-t, --target <dir> Install packages into <dir>. By default this will not replace existing files/folders in <dir>. Use --upgrade to
replace existing packages in <dir> with new versions.
-d, --download <dir> Download packages into <dir> instead of installing them, regardless of what's already installed.
--src <dir> Directory to check out editable projects into. The default in a virtualenv is "<venv path>/src". The default for
global installs is "<current dir>/src".
-U, --upgrade Upgrade all specified packages to the newest available version. This process is recursive regardless of whether a
dependency is already satisfied.
--force-reinstall When upgrading, reinstall all packages even if they are already up-to-date.
-I, --ignore-installed Ignore the installed packages (reinstalling instead).
--no-deps Don't install package dependencies.
--install-option <options> Extra arguments to be supplied to the setup.py install command (use like --install-option="--install-
scripts=/usr/local/bin"). Use multiple --install-option options to pass multiple options to setup.py install. If you
are using an option with a directory path, be sure to use absolute path.
--global-option <options> Extra global options to be supplied to the setup.py call before the install command.
Wow, that’s a lot of stuff, I had to cut off some of the output to keep this post from being unnecessarily long. Oh look! There’s an --upgrade option. Alright, it looks like you have to specify what package to update. Maybe it will accept an asterisk as a wildcard
pip install -U *
Invalid requirement: '__pycache__'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 78, in __init__
req = Requirement(req)
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "'__pycach'"
Doesn’t look like that will work. After doing some Googling, the most concise way I found was
pip list -o -l | cut -d ' ' -f 1 | sudo -H xargs pip install -U
Wowza. That’s a far cry from a two word command. pip list -o -l will show you all outdated packages in your local environment (that way if you’re in a virtualenv, it’s not going to try and update global packages - which could be very bad in some circumstances) in the format of requests (2.10.0) - Latest: 2.11.1 [wheel]. So that gets you the list of oudated packages. We then have to use cut -d ' ' -f 1 to trim off the fat. -d ' ' changes the field delimiter to a space, then -f 1 tells cut to only spit out the first field, which in our case would be the actual name of each package. Finally, we’ll pipe everything back into pip via a pip install -U. This is done via xargs, and I’m using sudo -H with it for times when I’m updating system level packages, with the -H setting the HOME variable to root’s home directory.
That’s not bad, just have to do a little command line-fu, but it’s nowhere near the simplicity of node’s or other package managers. Oh and on a final note, it’s a good idea to create an alias out of this, so that you don’t actually have to remember that chain of commands or worse, type it out
alias pipgrade="pip list -o -l | cut -d ' ' -f 1 | sudo -H xargs pip install -U"
Bonus Points
For extra compatibility with systems that might not have the cut command, here’s a version using awk.
pip list -o -l | awk -F' ' '{print $1}' | sudo -H xargs pip install -U