Cai Feng’s Blog

Recently, I’m interesting with android, but they only have online documents now. I can’t access the internet freely when I’m at home, so I want to download the documents and read them offline.

I searched the internet, and found wget can deal with this task very well.


wget --mirror --no-parent --page-requisites --convert-links --no-host-directories http://code.google.com/intl/zh/android/documentation.html

--mirror : Mirror is equivalent to "-r -N -l inf --no-remove-listing" (basically, infinite recursion).
--no-parent : Do not follow links that ascend to the parent directory. Only follow links that are under the given URL.
--page-requisites : Download all page requisites necessary to display the page (images, CSS, javascript, etc.).
--convert-links : Convert links in the pages so that they work locally relative to the OUTPUT_DIR.
--no-host-directories : Don't create host name directories.


Two additional useful options:
-A: Accept list, define which files you need.
-R：Reject list, define which files you don’t need.

I got that from http://www.noah.org/wiki/Offline_mirror_with_wget

Linux

I’m confused by different endline charactors for different systems recently. I organized some information here.

LF(n) — new line
CR(r) — enter
For windows system, it used CRLF(rn) at the end of each line.
For Mac, it used CR(r) at the end of each line.
For Linux, it used LF(n) at the end of each line.

So if you open a windows file in linux or mac system, there is a character ‘^M’ and the end of each line. On the contrary, you will see there is no line break if you open a mac/linux file in windows system.

I used Linux, so I just collect the way to convert the file from CRLF to LF.

1. Used vim

vim somefile
:set ff=unix
:wq


2. used commond tr
tr can delete characters from standard input, writing to standard output. See detail, please man tr

tr -d 'r' < "inputfile" > "outputfile"


3. If you use archlinux, install hd2u package, it includes a tool named dos2unix
With dos2unix, you can convert the file from mac, windows, unix freely.

dos2unix --d2u file


Linux

1. Write a script to sync the tickets from trac to platform(Platform is a internal project.). I don’t need to create the ticket in the trac and copy it to the platform. It’s a waste.

2. Improve google code: getsong project. I want to get the top 500 song list from http://www.9sky.com/ and download the song with getsong script.

Developer