Using wget to Automate Logging Into Websites
March 21, 2017
Posted by on
The open-source wget tool is useful for automating website access/scraping. In particular because it can store/retrieve cookies from a file.
# create a name for the cookie jar/file
# save cookies from homepage access
wget --spider --save-cookies $COOKIE_JAR --keep-session-cookies http://www.smrt.com.sg/
# now submit request using saved cookies
wget -O - \
--load-cookies $COOKIE_JAR \
--save-cookies $COOKIE_JAR \
--header "Referer: http://journey.smrt.com.sg/" \
Note that –spider performs a HEAD request and does not download the response. Options useful for debugging and seeing what is sent/received are -d and -S. For cookies the –keep-session-cookies option is essential to save session cookies (with no expiry time set) to the cookie file.