Terminal 1:
python3 -m http.server 8000 --bind 127.0.0.1 -d ./demo_server_filesFor run test server.
Terminal 2:
python demo.py "http://localhost:8000/" | tee downloaded.list | xargs wget -q -P ./downloaded
# OR
echo "http://localhost:8000/" | python demo.py --urls - | tee downloaded.list | xargs wget -q -P ./downloaded
# OR
cat crawl.list
# Output:
# http://localhost:8000
python demo.py --urls crawl.list | tee downloaded.list | xargs wget -q -P ./downloadedExplanation:
teefor save new urls for future download, use for breakpoints and can be used for skip downloeded files.xargsrun wget for each line from pipewget -q -P ./downloadedfor download url.-qfor no output,-Pfor download files into./downloadeddirectory.