Upgrading All Your Pip Packages

Posted on Fri 19 August 2016 in articles

Pip, the package manager (and its package repository PyPI) is one of the most important parts of the Python ecosystem. Arguably, it’s the most important part of why Python is so popular. The fact that it’s so easy to import a module into Python and easily use its features would be greatly hindered if the process of actually getting all those cool modules wasn’t easy. Pip is one of the foundational tools in the Python ecosystem and is really cool to use.

Which makes it even harder to understand why it’s missing some very key and - I would argue - basic features.

Take, for instance, upgrading all your packages.

Let’s look at brew, the awesome “missing package manager for Mac”. It is another way cool package manager, that also has a really cool and taken for granted feature:

brew upgrade

BOOM

Now all your packages managed by brew are updated. Simple.

Ok, that’s a package manager for an OS, so not quite the same (although I’m not sure if it matters or not). So let’s look at another example, the node.js package manager, npm:

npm update

WOW, yet another fine example of using only 2 commands strung together to easily update all your npm-managed node.js packages. Quite an amazing feat of technology. Since we’re talking about pip, let’s take a look at how to upgrade all your modules at once:

pip help

Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion
  help                        Show help for commands.

Ok…I don’t see an update or upgrade command. Let’s try install and see if it has any subcommands that could work.

pip help install

Install Options:
  -c, --constraint <file>     Constrain versions using the given constraints file. This option can be used multiple times.
  -e, --editable <path/url>   Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
  -r, --requirement <file>    Install from the given requirements file. This option can be used multiple times.
  -b, --build <dir>           Directory to unpack packages into and build in.
  -t, --target <dir>          Install packages into <dir>. By default this will not replace existing files/folders in <dir>. Use --upgrade to
                              replace existing packages in <dir> with new versions.
  -d, --download <dir>        Download packages into <dir> instead of installing them, regardless of what's already installed.
  --src <dir>                 Directory to check out editable projects into. The default in a virtualenv is "<venv path>/src". The default for
                              global installs is "<current dir>/src".
  -U, --upgrade               Upgrade all specified packages to the newest available version. This process is recursive regardless of whether a
                              dependency is already satisfied.
  --force-reinstall           When upgrading, reinstall all packages even if they are already up-to-date.
  -I, --ignore-installed      Ignore the installed packages (reinstalling instead).
  --no-deps                   Don't install package dependencies.
  --install-option <options>  Extra arguments to be supplied to the setup.py install command (use like --install-option="--install-
                              scripts=/usr/local/bin"). Use multiple --install-option options to pass multiple options to setup.py install. If you
                              are using an option with a directory path, be sure to use absolute path.
  --global-option <options>   Extra global options to be supplied to the setup.py call before the install command.

Wow, that’s a lot of stuff, I had to cut off some of the output to keep this post from being unnecessarily long. Oh look! There’s an --upgrade option. Alright, it looks like you have to specify what package to update. Maybe it will accept an asterisk as a wildcard

pip install -U *

Invalid requirement: '__pycache__'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 78, in __init__
    req = Requirement(req)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "'__pycach'"

Doesn’t look like that will work. After doing some Googling, the most concise way I found was

pip list -o -l | cut -d ' ' -f 1 | sudo -H xargs pip install -U

Wowza. That’s a far cry from a two word command. pip list -o -l will show you all outdated packages in your local environment (that way if you’re in a virtualenv, it’s not going to try and update global packages - which could be very bad in some circumstances) in the format of requests (2.10.0) - Latest: 2.11.1 [wheel]. So that gets you the list of oudated packages. We then have to use cut -d ' ' -f 1 to trim off the fat. -d ' ' changes the field delimiter to a space, then -f 1 tells cut to only spit out the first field, which in our case would be the actual name of each package. Finally, we’ll pipe everything back into pip via a pip install -U. This is done via xargs, and I’m using sudo -H with it for times when I’m updating system level packages, with the -H setting the HOME variable to root’s home directory.

That’s not bad, just have to do a little command line-fu, but it’s nowhere near the simplicity of node’s or other package managers. Oh and on a final note, it’s a good idea to create an alias out of this, so that you don’t actually have to remember that chain of commands or worse, type it out

alias pipgrade="pip list -o -l | cut -d ' ' -f 1 | sudo -H xargs pip install -U"
Bonus Points

For extra compatibility with systems that might not have the cut command, here’s a version using awk.

pip list -o -l | awk -F' ' '{print $1}' | sudo -H xargs pip install -U