From 118e26f8e226485f0457f684dede58ed940dbb59 Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Sat, 18 Mar 2017 11:16:43 +0000 Subject: [PATCH] Version v1.36 --- MANUAL.html | 969 ++++++++--- MANUAL.md | 1215 +++++++++++--- MANUAL.txt | 1240 +++++++++++--- Makefile | 10 +- RELEASE.md | 13 +- docs/content/changelog.md | 83 + docs/content/commands/rclone.md | 25 +- docs/content/commands/rclone_authorize.md | 23 +- docs/content/commands/rclone_cat.md | 38 +- docs/content/commands/rclone_check.md | 43 +- docs/content/commands/rclone_cleanup.md | 23 +- docs/content/commands/rclone_config.md | 23 +- docs/content/commands/rclone_copy.md | 23 +- docs/content/commands/rclone_copyto.md | 23 +- docs/content/commands/rclone_cryptcheck.md | 6 +- docs/content/commands/rclone_dedupe.md | 23 +- docs/content/commands/rclone_delete.md | 23 +- .../commands/rclone_genautocomplete.md | 23 +- docs/content/commands/rclone_gendocs.md | 23 +- docs/content/commands/rclone_listremotes.md | 23 +- docs/content/commands/rclone_ls.md | 23 +- docs/content/commands/rclone_lsd.md | 23 +- docs/content/commands/rclone_lsl.md | 23 +- docs/content/commands/rclone_md5sum.md | 23 +- docs/content/commands/rclone_mkdir.md | 23 +- docs/content/commands/rclone_mount.md | 54 +- docs/content/commands/rclone_move.md | 23 +- docs/content/commands/rclone_moveto.md | 23 +- docs/content/commands/rclone_obscure.md | 6 +- docs/content/commands/rclone_purge.md | 23 +- docs/content/commands/rclone_rmdir.md | 23 +- docs/content/commands/rclone_rmdirs.md | 23 +- docs/content/commands/rclone_sha1sum.md | 23 +- docs/content/commands/rclone_size.md | 23 +- docs/content/commands/rclone_sync.md | 23 +- docs/content/commands/rclone_version.md | 23 +- docs/content/downloads.md | 46 +- fs/version.go | 2 +- rclone.1 | 1457 ++++++++++++++--- snapcraft.yaml | 4 +- 40 files changed, 4609 insertions(+), 1154 deletions(-) diff --git a/MANUAL.html b/MANUAL.html index 4e58ea6f0..231384eb1 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -12,7 +12,7 @@

Rclone

Logo

@@ -28,6 +28,7 @@
  • Hubic
  • Backblaze B2
  • Yandex Disk
    • +
  • SFTP
  • The local filesystem

    • Features

    @@ -66,9 +67,9 @@ unzip rclone-current-linux-amd64.zip cd rclone-*-linux-amd64

    Copy binary file

    -
    sudo cp rclone /usr/sbin/
-sudo chown root:root /usr/sbin/rclone
-sudo chmod 755 /usr/sbin/rclone
    +
    sudo cp rclone /usr/bin/
+sudo chown root:root /usr/bin/rclone
+sudo chmod 755 /usr/bin/rclone

    Install manpage

    sudo mkdir -p /usr/local/share/man/man1
 sudo cp rclone.1 /usr/local/share/man/man1/
@@ -101,8 +102,39 @@ sudo mandb 
        - hosts: rclone-hosts
       roles:
           - rclone
    +

    Installation with snap

    +

    Quickstart

    + +

    See below for how to install snapd if it isn't already installed

    +

    Arch

    +
    sudo pacman -S snapd
    +

    enable the snapd systemd service:

    +
    sudo systemctl enable --now snapd.socket
    +

    Debian / Ubuntu

    +
    sudo apt install snapd
    +

    Fedora

    +
    sudo dnf copr enable zyga/snapcore
+sudo dnf install snapd
    +

    enable the snapd systemd service:

    +
    sudo systemctl enable --now snapd.service
    +

    SELinux support is in beta, so currently:

    +
    sudo setenforce 0
    +

    to persist, edit /etc/selinux/config to set SELINUX=permissive and reboot.

    +

    Gentoo

    +

    Install the gentoo-snappy overlay.

    +

    OpenEmbedded/Yocto

    +

    Install the snap meta layer.

    +

    openSUSE

    +
    sudo zypper addrepo http://download.opensuse.org/repositories/system:/snappy/openSUSE_Leap_42.2/ snappy
+sudo zypper install snapd
    +

    OpenWrt

    +

    Enable the snap-openwrt feed.

    Configure

    -

    First you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file .rclone.conf in your home directory by default. (You can use the --config option to choose a different config file.)

    +

    First you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file. (See the --config entry for how to find the config file and choose its location.)

    The easiest way to make the config is to run rclone with the config option:

    rclone config

    See the following for detailed instructions for

    @@ -118,6 +150,7 @@ sudo mandb
  • Hubic
  • Microsoft One Drive
  • Yandex Disk
    • +
  • SFTP
  • Crypt - to encrypt other remotes

    • Usage

    @@ -203,9 +236,12 @@ rclone --dry-run --min-size 100M delete remote:path

    rclone check

    Checks the files in the source and destination match.

    Synopsis

    -

    Checks the files in the source and destination match. It compares sizes and MD5SUMs and prints a report of files which don't match. It doesn't alter the source or destination.

    -

    --size-only may be used to only compare the sizes, not the MD5SUMs.

    +

    Checks the files in the source and destination match. It compares sizes and hashes (MD5 or SHA1) and logs a report of files which don't match. It doesn't alter the source or destination.

    +

    If you supply the --size-only flag, it will only compare the sizes not the hashes as well. Use this for a quick check.

    +

    If you supply the --download flag, it will download the data from both remotes and check them against each other on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.

    rclone check source:path dest:path
    +

    Options

    +
          --download   Check by downloading rather than with hash.

    rclone ls

    List all the objects in the path with size and path.

    Synopsis

    @@ -307,7 +343,7 @@ two-3.txt: renamed from: two.txt

    Or

    rclone dedupe rename "drive:Google Photos"
    rclone dedupe [mode] remote:path
    -

    Options

    +

    Options

          --dedupe-mode string   Dedupe mode interactive|skip|first|newest|oldest|rename. (default "interactive")

    rclone authorize

    Remote authorization.

    @@ -324,7 +360,14 @@ two-3.txt: renamed from: two.txt 
    rclone cat remote:path/to/dir

    Or like this to output any .txt files in dir or subdirectories.

    rclone --include "*.txt" cat remote:path/to/dir
    +

    Use the --head flag to print characters only at the start, --tail for the end and --offset and --count to print a section in the middle. Note that if offset is negative it will count from the end, so --offset -1 --count 1 is equivalent to --tail 1.

    rclone cat remote:path
    +

    Options

    +
          --count int    Only print N characters. (default -1)
+      --discard      Discard the output instead of printing.
+      --head int     Only print the first N characters.
+      --offset int   Start printing at offset N (or from end if -ve).
+      --tail int     Only print the last N characters.

    rclone copyto

    Copy files from source to dest, skipping already copied

    Synopsis

    @@ -341,9 +384,21 @@ if src is directory see copy command for full details

    This doesn't transfer unchanged files, testing by size and modification time or MD5SUM. It doesn't delete files from the destination.

    rclone copyto source:path dest:path
    +

    rclone cryptcheck

    +

    Cryptcheck checks the integritity of a crypted remote.

    +

    Synopsis

    +

    rclone cryptcheck checks a remote against a crypted remote. This is the equivalent of running rclone check, but able to check the checksums of the crypted remote.

    +

    For it to work the underlying remote of the cryptedremote must support some kind of checksum.

    +

    It works by reading the nonce from each file on the cryptedremote: and using that to encrypt each file on the remote:. It then checks the checksum of the underlying file on the cryptedremote: against the checksum of the file it has just encrypted.

    +

    Use it like this

    +
    rclone cryptcheck /path/to/files encryptedremote:path
    +

    You can use it like this also, but that will involve downloading all the files in remote:path.

    +
    rclone cryptcheck remote:path encryptedremote:path
    +

    After it has run it will log the status of the encryptedremote:.

    +
    rclone cryptcheck remote:path cryptedremote:path

    rclone genautocomplete

    Output bash completion script for rclone.

    -

    Synopsis

    +

    Synopsis

    Generates a bash shell autocompletion script for rclone.

    This writes to /etc/bash_completion.d/rclone by default so will probably need to be run with sudo or as root, eg

    sudo rclone genautocomplete
    @@ -353,36 +408,39 @@ if src is directory 
    rclone genautocomplete [output_file]

    rclone gendocs

    Output markdown docs for rclone to the directory supplied.

    -

    Synopsis

    +

    Synopsis

    This produces markdown docs for the rclone commands to the directory supplied. These are in a format suitable for hugo to render into the rclone.org website.

    rclone gendocs output_directory

    rclone listremotes

    List all the remotes in the config file.

    -

    Synopsis

    +

    Synopsis

    rclone listremotes lists all the available remotes from the config file.

    When uses with the -l flag it lists the types too.

    rclone listremotes
    -

    Options

    +

    Options

      -l, --long   Show the type as well as names.

    rclone mount

    Mount the remote as a mountpoint. EXPERIMENTAL

    -

    Synopsis

    +

    Synopsis

    rclone mount allows Linux, FreeBSD and macOS to mount any of Rclone's cloud storage systems as a file system with FUSE.

    This is EXPERIMENTAL - use with care.

    First set up your remote using rclone config. Check it works with rclone ls etc.

    -

    Start the mount like this

    +

    Start the mount like this (note the & on the end to put rclone in the background).

    rclone mount remote:path/to/files /path/to/local/mount &

    Stop the mount with

    fusermount -u /path/to/local/mount
    +

    Or if that fails try

    +
    fusermount -z -u /path/to/local/mount

    Or with OS X

    -
    umount -u /path/to/local/mount
    +
    umount /path/to/local/mount

    Limitations

    -

    This can only write files seqentially, it can only seek when reading.

    -

    Rclone mount inherits rclone's directory handling. In rclone's world directories don't really exist. This means that empty directories will have a tendency to disappear once they fall out of the directory cache.

    -

    The bucket based FSes (eg swift, s3, google compute storage, b2) won't work from the root - you will need to specify a bucket, or a path within the bucket. So swift: won't work whereas swift:bucket will as will swift:bucket/path.

    +

    This can only write files seqentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount.

    +

    The bucket based remotes (eg Swift, S3, Google Compute Storage, B2, Hubic) won't work from the root - you will need to specify a bucket, or a path within the bucket. So swift: won't work whereas swift:bucket will as will swift:bucket/path. None of these support the concept of directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.

    Only supported on Linux, FreeBSD and OS X at the moment.

    rclone mount vs rclone sync/copy

    File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone mount can't use retries in the same way without making local copies of the uploads. This might happen in the future, but for the moment rclone mount won't do that, so will be less reliable than the rclone command.

    +

    Filters

    +

    Note that all the rclone filters can be used to select a subset of the files to be visible in the mount.

    Bugs

    rclone mount remote:path /path/to/mountpoint
    -

    Options

    +

    Options

          --allow-non-empty           Allow mounting over a non-empty directory.
       --allow-other               Allow access to other users.
       --allow-root                Allow access to root user.
@@ -416,7 +474,7 @@ if src is directory
       --write-back-cache          Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.

    rclone moveto

    Move file or directory from source to dest.

    -

    Synopsis

    +

    Synopsis

    If source:path is a file or directory then it moves it to a file or directory named dest:path.

    This can be used to rename files or upload single files to other than their existing name. If the source is a directory then it acts exacty like the move command.

    So

    @@ -431,9 +489,14 @@ if src is directory

    This doesn't transfer unchanged files, testing by size and modification time or MD5SUM. src will be deleted on successful transfer.

    Important: Since this can cause data loss, test first with the --dry-run flag.

    rclone moveto source:path dest:path
    +

    rclone obscure

    +

    Obscure password for use in the rclone.conf

    +

    Synopsis

    +

    Obscure password for use in the rclone.conf

    +
    rclone obscure password

    rclone rmdirs

    Remove any empty directoryies under the path.

    -

    Synopsis

    +

    Synopsis

    This removes any empty directories (or directories that only contain empty directories) under the path that it finds, including the path if it has nothing in.

    This is useful for tidying up remotes that rclone has left a lot of empty directories in.

    rclone rmdirs remote:path
    @@ -462,26 +525,42 @@ if src is directory

    If you are using the root directory on its own then don't quote it (see #464 for why), eg

    rclone copy E:\ remote:backup

    Server Side Copy

    -

    Drive, S3, Dropbox, Swift and Google Cloud Storage support server side copy.

    +

    Most remotes (but not all - see the overview) support server side copy.

    This means if you want to copy one folder to another then rclone won't download all the files and re-upload them; it will instruct the server to copy them in place.

    Eg

    rclone copy s3:oldbucket s3:newbucket

    Will copy the contents of oldbucket to newbucket without downloading and re-uploading.

    -

    Remotes which don't support server side copy (eg local) will download and re-upload in this case.

    -

    Server side copies are used with sync and copy and will be identified in the log when using the -v flag.

    +

    Remotes which don't support server side copy will download and re-upload in this case.

    +

    Server side copies are used with sync and copy and will be identified in the log when using the -v flag. The may also be used with move if the remote doesn't support server side move.

    Server side copies will only be attempted if the remote names are the same.

    This can be used when scripting to make aged backups efficiently, eg

    rclone sync remote:current-backup remote:previous-backup
 rclone sync /path/to/files remote:current-backup
    -

    Options

    +

    Options

    Rclone has a number of options to control its behaviour.

    Options which use TIME use the go time parser. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

    Options which use SIZE use kByte by default. However a suffix of b for bytes, k for kBytes, M for MBytes and G for GBytes may be used. These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively.

    -

    --bwlimit=SIZE

    -

    Bandwidth limit in kBytes/s, or use suffix b|k|M|G. The default is 0 which means to not limit bandwidth.

    +

    --backup-dir=DIR

    +

    When using sync, copy or move any files which would have been overwritten or deleted are moved in their original hierarchy into this directory.

    +

    If --suffix is set, then the moved files will have the suffix added to them. If there is a file with the same path (after the suffix has been added) in DIR, then it will be overwritten.

    +

    The remote in use must support server side move or copy and you must use the same remote as the destination of the sync. The backup directory must not overlap the destination directory.

    +

    For example

    +
    rclone sync /path/to/local remote:current --backup-dir remote:old
    +

    will sync /path/to/local to remote:current, but for any files which would have been updated or deleted will be stored in remote:old.

    +

    If running rclone from a script you might want to use today's date as the directory name passed to --backup-dir to store the old files, or you might want to pass --suffix with today's date.

    +

    --bwlimit=BANDWIDTH_SPEC

    +

    This option controls the bandwidth limit. Limits can be specified in two ways: As a single limit, or as a timetable.

    +

    Single limits last for the duration of the session. To use a single limit, specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The default is 0 which means to not limit bandwidth.

    For example to limit bandwidth usage to 10 MBytes/s use --bwlimit 10M

    -

    This only limits the bandwidth of the data transfer, it doesn't limit the bandwith of the directory listings etc.

    +

    It is also possible to specify a "timetable" of limits, which will cause certain limits to be applied at certain times. To specify a timetable, format your entries as "HH:MM,BANDWIDTH HH:MM,BANDWITH...".

    +

    An example of a typical timetable to avoid link saturation during daytime working hours could be:

    +

    --bwlimit "08:00,512 12:00,10M 13:00,512 18:00,30M 23:00,off"

    +

    In this example, the transfer bandwidth will be set to 512kBytes/sec at 8am. At noon, it will raise to 10Mbytes/s, and drop back to 512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it will be completely disabled (full speed). Anything between 11pm and 8am will remain unlimited.

    +

    Bandwidth limits only apply to the data transfer. The don't apply to the bandwith of the directory listings etc.

    Note that the units are Bytes/s not Bits/s. Typically connections are measured in Bits/s - to convert divide by 8. For example let's say you have a 10 Mbit/s connection and you wish rclone to use half of it - 5 Mbit/s. This is 5/8 = 0.625MByte/s so you would use a --bwlimit 0.625M parameter for rclone.

    +

    --buffer-size=SIZE

    +

    Use this sized buffer to speed up file transfers. Each --transfer will use this much memory for buffering.

    +

    Set to 0 to disable the buffering for the minimum memory use.

    --checkers=N

    The number of checkers to run in parallel. Checkers do the equality checking of files during a sync. For some storage systems (eg s3, swift, dropbox) this can take a significant amount of time so they are run in parallel.

    The default is to run 8 checkers in parallel.

    @@ -492,7 +571,10 @@ rclone sync /path/to/files remote:current-backup

    Eg rclone --checksum sync s3:/bucket swift:/bucket would run much quicker than without the --checksum flag.

    When using this flag, rclone won't update mtimes of remote files if they are incorrect as it would normally.

    --config=CONFIG_FILE

    -

    Specify the location of the rclone config file. Normally this is in your home directory as a file called .rclone.conf. If you run rclone -h and look at the help for the --config option you will see where the default location is for you. Use this flag to override the config location, eg rclone --config=".myconfig" .config.

    +

    Specify the location of the rclone config file.

    +

    Normally the config file is in your home directory as a file called .config/rclone/rclone.conf (or .rclone.conf if created with an older version). If $XDG_CONFIG_HOME is set it will be at $XDG_CONFIG_HOME/rclone/rclone.conf

    +

    If you run rclone -h and look at the help for the --config option you will see where the default location is for you.

    +

    Use this flag to override the config location, eg rclone --config=".myconfig" .config.

    --contimeout=TIME

    Set the connection timeout. This should be in go time format which looks like 5s for 5 seconds, 10m for 10 minutes, or 3h30m.

    The connection timeout is the amount of time rclone will wait for a connection to go through to a remote object storage system. It is 1m by default.

    @@ -500,6 +582,9 @@ rclone sync /path/to/files remote:current-backup

    Mode to run dedupe command in. One of interactive, skip, first, newest, oldest, rename. The default is interactive. See the dedupe command for more information as to what these options mean.

    -n, --dry-run

    Do a trial run with no permanent changes. Use this to see what rclone would do without actually doing it. Useful when setting up the sync command which deletes files in the destination.

    +

    --ignore-checksum

    +

    Normally rclone will check that the checksums of transferred files match, and give an error "corrupted on transfer" if they don't.

    +

    You can use this option to skip that check. You should only use it if you have had the "corrupted on transfer" error message and you are sure you might want to transfer potentially corrupted data.

    --ignore-existing

    Using this option will make rclone unconditionally skip all files that exist on the destination, no matter the content of these files.

    While this isn't a generally recommended option, it can be useful in cases where your files change due to encryption. However, it cannot correct partial transfers in case a transfer was interrupted.

    @@ -512,6 +597,12 @@ rclone sync /path/to/files remote:current-backup

    Normally rclone would skip any files that have the same modification time and are the same size (or have the same checksum if using --checksum).

    --log-file=FILE

    Log all of rclone's output to FILE. This is not active by default. This can be useful for tracking down problems with syncs in combination with the -v flag. See the Logging section for more info.

    +

    --log-level LEVEL

    +

    This sets the log level for rclone. The default log level is INFO.

    +

    DEBUG is equivalent to -vv. It outputs lots of debug info - useful for bug reports and really finding out what rclone is doing.

    +

    INFO is equivalent to -v. It outputs information about each transfer and prints stats once a minute by default.

    +

    NOTICE is the default log level if no logging flags are supplied. It outputs very little when things are working normally. It outputs warnings and significant events.

    +

    ERROR is equivalent to -q. It only output error messages.

    --low-level-retries NUMBER

    This controls the number of low level retries rclone does.

    A low level retry is used to retry a failing operation - typically one HTTP request. This might be uploading a chunk of a big file for example. You will see low level retries in the log with the -v flag.

    @@ -553,11 +644,26 @@ rclone sync /path/to/files remote:current-backup

    Data transfer volume will still be reported in bytes.

    The rate is reported as a binary unit, not SI unit. So 1 Mbit/s equals 1,048,576 bits/s and not 1,000,000 bits/s.

    The default is bytes.

    +

    --suffix=SUFFIX

    +

    This is for use with --backup-dir only. If this isn't set then --backup-dir will move files with their original name. If it is set then the files will have SUFFIX added on to them.

    +

    See --backup-dir for more info.

    +

    --syslog

    +

    On capable OSes (not Windows or Plan9) send all log output to syslog.

    +

    This can be useful for running rclone in script or rclone mount.

    +

    --syslog-facility string

    +

    If using --syslog this sets the syslog facility (eg KERN, USER). See man syslog for a list of possible facilities. The default facility is DAEMON.

    +

    --track-renames

    +

    By default rclone doesn't not keep track of renamed files, so if you rename a file locally then sync it to a remote, rclone will delete the old file on the remote and upload a new copy.

    +

    If you use this flag, and the remote supports server side copy or server side move, and the source and destination have a compatible hash, then this will track renames during sync, copy, and move operations and perform renaming server-side.

    +

    Files will be matched by size and hash - if both match then a rename will be considered.

    +

    If the destination does not support server-side copy or move, rclone will fall back to the default behaviour and log an error level message to the console.

    +

    Note that --track-renames is incompatible with --no-traverse and that it uses extra memory to keep track of all the rename candidates.

    +

    Note also that --track-renames is incompatible with --delete-before and will select --delete-after instead of --delete-during.

    --delete-(before,during,after)

    This option allows you to specify when files on your destination are deleted when you sync folders.

    -

    Specifying the value --delete-before will delete all files present on the destination, but not on the source before starting the transfer of any new or updated files. This uses extra memory as it has to store the source listing before proceeding.

    -

    Specifying --delete-during (default value) will delete files while checking and uploading files. This is usually the fastest option. Currently this works the same as --delete-after but it may change in the future.

    -

    Specifying --delete-after will delay deletion of files until all new/updated files have been successfully transfered.

    +

    Specifying the value --delete-before will delete all files present on the destination, but not on the source before starting the transfer of any new or updated files. This uses two passes through the file systems, one for the deletions and one for the copies.

    +

    Specifying --delete-during will delete files while checking and uploading files. This is the fastest option and uses the least memory.

    +

    Specifying --delete-after (the default value) will delay deletion of files until all new/updated files have been successfully transfered. The files to be deleted are collected in the copy pass then deleted after the copy pass has completed sucessfully. The files to be deleted are held in memory so this mode may use more memory. This is the safest mode as it will only delete files if there have been no errors subsequent to that. If there have been errors before the deletions start then you will get the message not deleting files as there were IO errors.

    --timeout=TIME

    This sets the IO idle timeout. If a transfer has started but then becomes idle for this long it is considered broken and disconnected.

    The default is 5m. Set to 0 to disable.

    @@ -569,9 +675,9 @@ rclone sync /path/to/files remote:current-backup

    If an existing destination file has a modification time equal (within the computed modify window precision) to the source file's, it will be updated if the sizes are different.

    On remotes which don't support mod time directly the time checked will be the uploaded time. This means that if uploading to one of these remoes, rclone will skip any files which exist on the destination and have an uploaded time that is newer than the modification time of the source file.

    This can be useful when transferring to a remote which doesn't support mod times directly as it is more accurate than a --size-only check and faster than using --checksum.

    -

    -v, --verbose

    -

    If you set this flag, rclone will become very verbose telling you about every file it considers and transfers.

    -

    Very useful for debugging.

    +

    -v, -vv, --verbose

    +

    With -v rclone will tell you about each file that is transferred and a small number of significant events.

    +

    With -vv rclone will become very verbose telling you about every file it considers and transfers. Please send bug reports with a log with this setting.

    -V, --version

    Prints the version number

    Configuration Encryption

    @@ -636,7 +742,7 @@ export RCLONE_CONFIG_PASS

    This option defaults to false.

    This should be used only for testing.

    --no-traverse

    -

    The --no-traverse flag controls whether the destination file system is traversed when using the copy or move commands.

    +

    The --no-traverse flag controls whether the destination file system is traversed when using the copy or move commands. --no-traverse is not compatible with sync and will be ignored if you supply it with sync.

    If you are only copying a small number of files and/or have a large number of files on the destination then --no-traverse will stop rclone listing the destination and save time.

    However if you are copying a large number of files, escpecially if you are doing a copy where lots of the files haven't changed and won't need copying then you shouldn't use --no-traverse.

    It can also be used to reduce the memory usage of rclone when copying - rclone --no-traverse copy src dst won't load either the source or destination listings into memory so will use the minimum amount of memory.

    @@ -659,16 +765,49 @@ export RCLONE_CONFIG_PASS

    See the filtering section.

    Logging

    -

    rclone has 3 levels of logging, Error, Info and Debug.

    -

    By default rclone logs Error and Info to standard error and Debug to standard output. This means you can redirect standard output and standard error to different places.

    -

    By default rclone will produce Error and Info level messages.

    +

    rclone has 4 levels of logging, Error, Notice, Info and Debug.

    +

    By default rclone logs to standard error. This means you can redirect standard error and still see the normal output of rclone commands (eg rclone ls).

    +

    By default rclone will produce Error and Notice level messages.

    If you use the -q flag, rclone will only produce Error messages.

    -

    If you use the -v flag, rclone will produce Error, Info and Debug messages.

    +

    If you use the -v flag, rclone will produce Error, Notice and Info messages.

    +

    If you use the -vv flag, rclone will produce Error, Notice, Info and Debug messages.

    +

    You can also control the log levels with the --log-level flag.

    If you use the --log-file=FILE option, rclone will redirect Error, Info and Debug messages along with standard error to FILE.

    +

    If you use the --syslog flag then rclone will log to syslog and the --syslog-facility control which facility it uses.

    +

    Rclone prefixes all log messages with their level in capitals, eg INFO which makes it easy to grep the log file for different kinds of information.

    Exit Code

    -

    If any errors occurred during the command, rclone with an exit code of 1. This allows scripts to detect when rclone operations have failed.

    +

    If any errors occurred during the command, rclone will exit with a non-zero exit code. This allows scripts to detect when rclone operations have failed.

    During the startup phase rclone will exit immediately if an error is detected in the configuration. There will always be a log message immediately before exiting.

    -

    When rclone is running it will accumulate errors as it goes along, and only exit with an non-zero exit code if (after retries) there were no transfers with errors remaining. For every error counted there will be a high priority log message (visibile with -q) showing the message and which file caused the problem. A high priority message is also shown when starting a retry so the user can see that any previous error messages may not be valid after the retry. If rclone has done a retry it will log a high priority message if the retry was successful.

    +

    When rclone is running it will accumulate errors as it goes along, and only exit with an non-zero exit code if (after retries) there were still failed transfers. For every error counted there will be a high priority log message (visibile with -q) showing the message and which file caused the problem. A high priority message is also shown when starting a retry so the user can see that any previous error messages may not be valid after the retry. If rclone has done a retry it will log a high priority message if the retry was successful.

    +

    Environment Variables

    +

    Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.

    +

    Options

    +

    Every option in rclone can have its default set by environment variable.

    +

    To find the name of the environment variable, first take the long option name, strip the leading --, change - to _, make upper case and prepend RCLONE_.

    +

    For example to always set --stats 5s, set the environment variable RCLONE_STATS=5s. If you set stats on the command line this will override the environment variable setting.

    +

    Or to always use the trash in drive --drive-use-trash, set RCLONE_DRIVE_USE_TRASH=true.

    +

    The same parser is used for the options and the environment variables so they take exactly the same form.

    +

    Config file

    +

    You can set defaults for values in the config file on an individual remote basis. If you want to use this feature, you will need to discover the name of the config items that you want. The easiest way is to run through rclone config by hand, then look in the config file to see what the values are (the config file can be found by looking at the help for --config in rclone help).

    +

    To find the name of the environment variable, you need to set, take RCLONE_ + name of remote + _ + name of config file option and make it all uppercase.

    +

    For example to configure an S3 remote named mys3: without a config file (using unix ways of setting environment variables):

    +
    $ export RCLONE_CONFIG_MYS3_TYPE=s3
+$ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
+$ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
+$ rclone lsd MYS3:
+          -1 2016-09-21 12:54:21        -1 my-bucket
+$ rclone listremotes | grep mys3
+mys3:
    +

    Note that if you want to create a remote using environment variables you must create the ..._TYPE variable as above.

    +

    Other environment variables

    +

    Configuring rclone on a remote / headless machine

    Some of the configurations (those involving oauth2) require an Internet connected web browser.

    If you are trying to set rclone up on a remote or headless box with no browser available on it (eg a NAS or a server in a datacenter) then you will need to use an alternative means of configuration. There are two ways of doing it, described below.

    @@ -1024,6 +1163,14 @@ user2/stuff R/W +SFTP +- +Yes +Depends +No +- + + The local filesystem All Yes @@ -1043,7 +1190,7 @@ The hashes are used when transferring data as an integrity check and can be spec

    Case Insensitive

    If a cloud storage systems is case sensitive then it is possible to have two files which differ only in case, eg file.txt and FILE.txt. If a cloud storage system is case insensitive then that isn't possible.

    This can cause problems when syncing between a case insensitive system and a case sensitive system. The symptom of this is that no matter how many times you run the sync it never completes fully.

    -

    The local filesystem may or may not be case sensitive depending on OS.

    +

    The local filesystem and SFTP may or may not be case sensitive depending on OS.