Update MD files

advanced.md

@@ -0,0 +1,80 @@
+# Advanced options
+
+[Get started](./) | **Advanced**
+
+## By photoset vs. collection
+
+Flixport scans all photos in 2 different approaches, by photoset or by
+collection. User makes the choice with `-w` or `--by_collection` and while
+photoset is the default approach. By collection means Flixport traverses
+all collections first, then iterates photoset in each leaf collection.
+By photoset simply means it iterates all photosets. Why does it matter to user?
+
+### Destination path
+
+The way Flixport figures out the exact destination of a particular photo is
+based on 3 options:
+
+* `-d` or `-dest_spec`
+* `-p` or `-dest_dir`
+* `-n` or `-dest_file_name`
+
+The destination of each photo is `<dest_spec><desc_dir>/<dest_file_name>`,
+where dest_dir and dest_file_name supports $ syntax. In most cases the
+default values of `dest_dir` and `dest_file_name` works well.
+
+* The default value of dest_file_name is `${f.title}.${f.originalFormat}`.
+* When photos are exported by photoset, the default value of dest_dir is
+`/${s.title}`. 
+* When photos are exported by collection, the default value of dest_dir
+is `/${c.title}/${s.title}`.
+
+With the default settings above command line like below
+
+`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr`
+
+would copy photo `myphoto1.jpg` in Album `my_album` to s3 bucket `mybucket`
+with key `flickr/my_album/myphoto1.jpg` when it exports photo by photoset.
+
+If `my_album` is in collection `my_collection` and command line runs by
+collection, the same photo would be
+`flickr/my_collection/my_album/myphoto1.jpg`.
+
+### More about the expression
+
+In these expressions, `$f` is file, $s is a photoset and `$c` is a collection.
+
+Only `$s` is available for `dest_dir` when photos are exported by photoset.
+`$s` and $c are available for `dest_dir` when photos are exported by
+collection.
+`$f` and whatever is available for `dest_dir` are available for
+`dest_file_format`.
+
+## Execution options
+
+### Max number of files
+Max number of files can be limited by `-m` or `--max_files`, so that users
+can get an idea what's going to happen in the full run. For example:
+
+`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr -m 20`
+
+### Dry run mode
+
+Another way to preview the execution is to dry run the command line tool
+without actually copying any file. With `-r` or `--dry_run` option, Instead
+of copying the file, the tool simply logs a message saying it would copy a
+file from a location to a location. In dry run mode, it becomes particularly
+important to keep the log files. For example:
+
+`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr -r 2>&1 | tee /tmp/flixport.log`
+
+### Multi-thread
+
+By default flixport runs in a single thread, which copies one file after
+another. This is very inefficient if you have large number of files to copy.
+Since the command line is making many TCP calls, the thread is mostly idle
+while waiting for the calls to return. In your final run you almost always
+need to specify number of threads to use with `-t` or `--threads` option to
+keep the run time reasonable. For example:
+
+`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr -t 20`

readme.md

@@ -0,0 +1,145 @@
+# Get started with Flixport
+
+**Get started** | [Advanced](advanced.md)
+
+## Prerequisites
+Can wait to try it? Good. Before start, make sure to go through the checklist
+below. These steps only need to be done once.
+
+### Command line environment
+As of now Flixport is a command line tool. To use it, you should feel
+comfortable with command line environment. This means different setups in
+Windows, Mac or Linux.
+
+### Install Java
+If you haven't, download and install Java. Configure it properly so that it's
+available in command line environment.
+
+Java version 8 or above is required. Verify your Java installation with the
+command below.
+
+```
+me@me:~/code/flixport$ java -version
+openjdk version "1.8.0_171-google-v7"
+OpenJDK Runtime Environment (build 1.8.0_171-google-v7-228093407-228093407)
+OpenJDK 64-Bit Server VM (build 25.171-b01, mixed mode)
+```
+
+### Download Flixport
+Find the latest release of Flixport in Github release page. Download the
+`flixport-#.#.#.jar` file. Go to the directly with the download jar file, you
+can verify the jar file by running it with `-h` for help.
+
+```
+me@me:~/Downloads$ java -jar flixport-0.0.1.jar -h
+[USAGE]
+  flixport <OPTIONS>
+[OPTIONS]
+  -n --aws_region     <value>   AWS region
+  -k --app_key        <value>   Flickr application key
+...
+```
+
+### Create an API key in Flickr
+
+Goto https://www.flickr.com/services/api, click "API keys" and create a key
+if you haven't done so.
+
+Find details in http://docs.cyclopsgroup.org/flixport
+
+## Configure Flixport
+
+Flixport command line is mostly configured in `$HOME/.flixport/cli.properties`
+file. The properties file can define any command line options with long name
+as the property name. In another word if you set an option with the long name
+in cli.properties file, it becomes default so you don't need to specify in
+command line. Therefore typically you would configure:
+
+### Flickr app
+
+Flick API key and secret
+
+```
+app_key=0123456789abcdef0123456789abcdef
+app_secret=0123456789abcdef
+```
+
+### S3 IAM
+
+If you are exporting photos to S3 , you can store AWS IAM key and secret in
+properties file.
+
+```
+aws_key=AKIA0123456789ABCDEF
+aws_secret=0123456789ABCDEF0123456789ABCDEF01234567
+```
+
+## Run it
+
+### Quick examples
+
+```
+java -jar flixport-0.0.1.jar -d file:$HOME/Downloads/flickr -t 20
+java -jar flixport-0.0.1.jar -d s3:mys3bucket/some/path -t 20
+```
+
+`-t 20` runs the command in 20 threads in parallel.
+
+### Options
+
+Before start, take a look at the available command line options. Run it with
+`-h` or `--help` to see them.
+
+```
+me@me:~/Downloads$ java -jar flixport-0.0.1.jar -h
+[USAGE]
+  flixport <OPTIONS>
+[OPTIONS]
+  -n --aws_region     <value>   AWS region
+  -k --app_key        <value>   Flickr application key
+  -r --dry_run                  Log action without actually copying files
+  -t --threads        <integer> Number of threads to use
+  -y --aws_key        <value>   AWS credentials key
+  -z --aws_secret     <value>   AWS credentials secret
+  -x --flixport_dir   <dir>     Directory for application specific local files
+  -d --dest_spec      <value>   Destination spec
+  -e --max_attempts   <integer> Max number of attempts to export a photoset.
+  -w --by_collection            Nested output by collection
+  -m --max_files      <integer> Max number of files to copy
+  -n --dest_file_name <format>  File name of the output file
+  -h --help                     Show help message
+  -c --dest_creds     <file>    Destination storage credentials file or spec
+  -p --dest_dir       <dir>     Directory of destination
+  -a --force_auth               Force to authenticate
+  -s --app_secret     <value>   Flickr application secret
+```
+
+The destination, `-d` or `--dest_spec` is a mandatory option that specifies
+where to export photos to. As of now it supports:
+
+* S3: `-d s3:mys3bucket/some/optional/path`
+* Google storage service: `-d gs:mygcsbucket/some/optional/path`
+* Local disk: `-d file:/tmp/my/flickr`
+
+In case of S3, IAM key and secret is required. They are specified with
+`--aws_key` and `--aws_secret` options in command line or `cli.properties`.
+
+In case of Google storage service, JSON key file is required. You can specify
+it with `-c` or `--dest_creds` with the path to the key file. For example:
+
+`java -jar flixport-0.0.1.jar -d gs:mygcsbucket/some/optional/path -c $HOME/my-gcp-key.json`
+
+To export to local disk, no other options are required.
+
+`java -jar flixport-0.0.1.jar -d file:$HOME/Downloads/flickr`
+
+### Preview with a limit
+
+Before exporting all photos, you almost always want to run it with a limit for
+the number of photos to export. This step gives you and idea how the result
+will look like and where the files end up. File limit is set with `-m` or
+`--max_files` options.
+
+You can run the command over and over again, it exports the given number of
+new photos a time incrementally.
+