wget spider

wget --mirror --convert-links --page-requisites --adjust-extension <site-url>

• --mirror: make the download recursive
• --convert-links: convert links to work offline ex. ./ instead of /
• --page-requisites: make sure to download js and css
• --adjust-extension: append missing extensions (mostly for html)

WARC

If I'm crawling a site for an offline backup I probably need a zim file. Wget can't output a zim file but it can output a warc file which can be converted to a zim file using warc2zim

• --warc-file=filename: enables warc export, standard output will also happen

Cross domain

• --span-hosts: crawl cross domain links
• --domains: make sure you only crawl the domains you want to (don't want to accidentally crawl the whole internet)

Netcat Webserver

while true; do echo -e "HTTP/1.1 200 OK\n\n" $(cat index.html) | nc -lp 8080 -q 1; done

• "HTTP/1.1 200 OK\n\n" can be put in the html file. Just be sure to use ^M^M instead of the literal \n\n

Strip everything except ascii from Text files

A really quick way of doing this is to use strings. However this will strip whitespace characters.

If you need whitespace characters intact you can use this:

cat <file> | tr -d "[:cntrl:]" | iconv -c -f utf-8 -t ascii -

Python One-Liners

Takes a list of urls and formats them in markdown python -c "import sys; from urllib.parse import urlsplit; print(''.join([f\"[{urlsplit(line.strip()).path.lstrip('/')}]({line.strip()})\n\" for line in sys.stdin]))"