Tuesday, 23 September 2008

Harness the power of wget

I finally invested enough time in reading wget's man page to be able to download file hierarchies exposed through http file server. The magic command is:
wget -r -np -nH --cut-dirs=2 http://example.com/some/nested/directory
The switches mean:

turns of recursive download
no-parent - only links to files below in the hierarchy are followed
no-host - doesn't dump the files into directory named after host name
in addition to skipping host name skips also 2 topmost directories. In the example instead of some/nestes/directory the files are put straight into directory