townx - Using wget to mirror a website - Comments

I put a short (and probably

elliot — Sat, 29 May 2010 06:49:24 -0500

I put a short (and probably not very helpful) suggestion in reply to your other comment. Always happy to get a genuine comment from someone who found my blog useful!

Nifty use of wget

John — Wed, 19 May 2010 09:26:51 -0500

Nifty use of wget, Seems so simple and useful I, not only bookmarked it but also cut & pasted your article in my personal linux help document.. Dont want to risk a page not found some time off in the distant future. ;)
Consequentially I stumbled onto your blog searching for an easy way to implement spam filtering. The how to was very helpful, but left me with one question when training my inbox (see my comment in that blog entry, if you have any insight)
Thanks, John

Using wget to mirror a website

elliot — Wed, 12 May 2010 04:29:46 -0500

Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.

curl can probably do this too, and supports authentication, but it wasn't obvious.

So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.

Eventually, I can up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):

wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m http://domain.to.mirror/

where:

-e robots=off obviously disables robots
--wait 1 forces a pause between gets (so the site doesn't get hammered)
--user and --password: self-evident
-x creates a local directory structure which "mirrors" (see what I did there?) the directory structure on the site you're mirroring
-m turns on mirror mode: "turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings" (from the man page)

Don't use it carelessly on someone else's website, as they might get angry...