Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.
curl can probably do this too, and supports authentication, but it wasn't obvious.
So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.
Eventually, I came up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):
wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m -k http://domain.to.mirror/
Don't use it carelessly on someone else's website, as they might get angry...