recursively download and crawl web pages using wget

in case downloading hangs due to missing/incorrect robots.txt, just use “-erobots=off”.
This skips downloading robots.txt altogether.

Multiple URLs

References

1. http://skeena.net/kb/wget%20ignore%20robots.txt

About Gugulethu Ncube

IT enthusiast getting things done.