diff --git a/README.md b/README.md deleted file mode 100644 index 4f678cc..0000000 --- a/README.md +++ /dev/null @@ -1 +0,0 @@ -Find details in http://docs.cyclopsgroup.org/flixport diff --git a/advanced.md b/advanced.md new file mode 100644 index 0000000..8c4b4fd --- /dev/null +++ b/advanced.md @@ -0,0 +1,80 @@ +# Advanced options + +[Get started](./) | **Advanced** + +## By photoset vs. collection + +Flixport scans all photos in 2 different approaches, by photoset or by +collection. User makes the choice with `-w` or `--by_collection` and while +photoset is the default approach. By collection means Flixport traverses +all collections first, then iterates photoset in each leaf collection. +By photoset simply means it iterates all photosets. Why does it matter to user? + +### Destination path + +The way Flixport figures out the exact destination of a particular photo is +based on 3 options: + +* `-d` or `-dest_spec` +* `-p` or `-dest_dir` +* `-n` or `-dest_file_name` + +The destination of each photo is `/`, +where dest_dir and dest_file_name supports $ syntax. In most cases the +default values of `dest_dir` and `dest_file_name` works well. + +* The default value of dest_file_name is `${f.title}.${f.originalFormat}`. +* When photos are exported by photoset, the default value of dest_dir is +`/${s.title}`. +* When photos are exported by collection, the default value of dest_dir +is `/${c.title}/${s.title}`. + +With the default settings above command line like below + +`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr` + +would copy photo `myphoto1.jpg` in Album `my_album` to s3 bucket `mybucket` +with key `flickr/my_album/myphoto1.jpg` when it exports photo by photoset. + +If `my_album` is in collection `my_collection` and command line runs by +collection, the same photo would be +`flickr/my_collection/my_album/myphoto1.jpg`. + +### More about the expression + +In these expressions, `$f` is file, $s is a photoset and `$c` is a collection. + +Only `$s` is available for `dest_dir` when photos are exported by photoset. +`$s` and $c are available for `dest_dir` when photos are exported by +collection. +`$f` and whatever is available for `dest_dir` are available for +`dest_file_format`. + +## Execution options + +### Max number of files +Max number of files can be limited by `-m` or `--max_files`, so that users +can get an idea what's going to happen in the full run. For example: + +`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr -m 20` + +### Dry run mode + +Another way to preview the execution is to dry run the command line tool +without actually copying any file. With `-r` or `--dry_run` option, Instead +of copying the file, the tool simply logs a message saying it would copy a +file from a location to a location. In dry run mode, it becomes particularly +important to keep the log files. For example: + +`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr -r 2>&1 | tee /tmp/flixport.log` + +### Multi-thread + +By default flixport runs in a single thread, which copies one file after +another. This is very inefficient if you have large number of files to copy. +Since the command line is making many TCP calls, the thread is mostly idle +while waiting for the calls to return. In your final run you almost always +need to specify number of threads to use with `-t` or `--threads` option to +keep the run time reasonable. For example: + +`java -jar flixport-0.0.1.jar -d s3:mybucket/flickr -t 20` diff --git a/readme.md b/readme.md new file mode 100644 index 0000000..a5b0253 --- /dev/null +++ b/readme.md @@ -0,0 +1,145 @@ +# Get started with Flixport + +**Get started** | [Advanced](advanced.md) + +## Prerequisites +Can wait to try it? Good. Before start, make sure to go through the checklist +below. These steps only need to be done once. + +### Command line environment +As of now Flixport is a command line tool. To use it, you should feel +comfortable with command line environment. This means different setups in +Windows, Mac or Linux. + +### Install Java +If you haven't, download and install Java. Configure it properly so that it's +available in command line environment. + +Java version 8 or above is required. Verify your Java installation with the +command below. + +``` +me@me:~/code/flixport$ java -version +openjdk version "1.8.0_171-google-v7" +OpenJDK Runtime Environment (build 1.8.0_171-google-v7-228093407-228093407) +OpenJDK 64-Bit Server VM (build 25.171-b01, mixed mode) +``` + +### Download Flixport +Find the latest release of Flixport in Github release page. Download the +`flixport-#.#.#.jar` file. Go to the directly with the download jar file, you +can verify the jar file by running it with `-h` for help. + +``` +me@me:~/Downloads$ java -jar flixport-0.0.1.jar -h +[USAGE] + flixport +[OPTIONS] + -n --aws_region AWS region + -k --app_key Flickr application key +... +``` + +### Create an API key in Flickr + +Goto https://www.flickr.com/services/api, click "API keys" and create a key +if you haven't done so. + +Find details in http://docs.cyclopsgroup.org/flixport + +## Configure Flixport + +Flixport command line is mostly configured in `$HOME/.flixport/cli.properties` +file. The properties file can define any command line options with long name +as the property name. In another word if you set an option with the long name +in cli.properties file, it becomes default so you don't need to specify in +command line. Therefore typically you would configure: + +### Flickr app + +Flick API key and secret + +``` +app_key=0123456789abcdef0123456789abcdef +app_secret=0123456789abcdef +``` + +### S3 IAM + +If you are exporting photos to S3 , you can store AWS IAM key and secret in +properties file. + +``` +aws_key=AKIA0123456789ABCDEF +aws_secret=0123456789ABCDEF0123456789ABCDEF01234567 +``` + +## Run it + +### Quick examples + +``` +java -jar flixport-0.0.1.jar -d file:$HOME/Downloads/flickr -t 20 +java -jar flixport-0.0.1.jar -d s3:mys3bucket/some/path -t 20 +``` + +`-t 20` runs the command in 20 threads in parallel. + +### Options + +Before start, take a look at the available command line options. Run it with +`-h` or `--help` to see them. + +``` +me@me:~/Downloads$ java -jar flixport-0.0.1.jar -h +[USAGE] + flixport +[OPTIONS] + -n --aws_region AWS region + -k --app_key Flickr application key + -r --dry_run Log action without actually copying files + -t --threads Number of threads to use + -y --aws_key AWS credentials key + -z --aws_secret AWS credentials secret + -x --flixport_dir Directory for application specific local files + -d --dest_spec Destination spec + -e --max_attempts Max number of attempts to export a photoset. + -w --by_collection Nested output by collection + -m --max_files Max number of files to copy + -n --dest_file_name File name of the output file + -h --help Show help message + -c --dest_creds Destination storage credentials file or spec + -p --dest_dir Directory of destination + -a --force_auth Force to authenticate + -s --app_secret Flickr application secret +``` + +The destination, `-d` or `--dest_spec` is a mandatory option that specifies +where to export photos to. As of now it supports: + +* S3: `-d s3:mys3bucket/some/optional/path` +* Google storage service: `-d gs:mygcsbucket/some/optional/path` +* Local disk: `-d file:/tmp/my/flickr` + +In case of S3, IAM key and secret is required. They are specified with +`--aws_key` and `--aws_secret` options in command line or `cli.properties`. + +In case of Google storage service, JSON key file is required. You can specify +it with `-c` or `--dest_creds` with the path to the key file. For example: + +`java -jar flixport-0.0.1.jar -d gs:mygcsbucket/some/optional/path -c $HOME/my-gcp-key.json` + +To export to local disk, no other options are required. + +`java -jar flixport-0.0.1.jar -d file:$HOME/Downloads/flickr` + +### Preview with a limit + +Before exporting all photos, you almost always want to run it with a limit for +the number of photos to export. This step gives you and idea how the result +will look like and where the files end up. File limit is set with `-m` or +`--max_files` options. + +You can run the command over and over again, it exports the given number of +new photos a time incrementally. +