Commit Graph

103 Commits (master)

Author SHA1 Message Date
Bernd Schoolmann 158870bcdb
fs: Add --track-renames-strategy for configurable matching criteria for --track-renames
This commit adds the `--track-renames-strategy` flag which allows the
user to choose the strategy for tracking renames when using the
`--track-renames` flag.

This can be "hash" or "modtime" or both currently.

This, when used with `--track-renames-strategy modtime` enables
support for tracking renames in encrypted remotes.

Fixes #3696
Fixes #2721
2020-03-20 13:04:56 +00:00
Nick Craig-Wood 47e2d5c415 config: fsync the config file after writing #3411
This should help with data integrity
2020-03-16 18:20:16 +00:00
Franklyn Tackitt 2b3d13a841 fs: Use --cutoff-mode hard,soft,catious instead of 3 --max-transfer-mode flags
Fixes #2672
2020-03-01 09:49:55 +00:00
Shing Kit Chan 6f1766dd9e fs: Add support for --max-transfer-cutoff modes #2672
This also adds max transfer cut off check for server side copies too
2020-03-01 09:49:55 +00:00
boosh 0d7573dd81 fs: Add --max-duration flag to control the maximum duration of a transfer session
This gives you more control over how long rclone will run for, making
it easier to script backups, e.g. via cron. Once the `--max-duration`
time limit is reached, no new transfers will be initiated, but those
already in-flight will be allowed to complete.

Fixes #985
2020-01-24 13:28:56 +00:00
Nick Craig-Wood 3afb2a4798 config: use SpaceSepList for argument to --password-command
This is to enable arguments with spaces in.
2020-01-23 15:39:15 +00:00
Nick Craig-Wood 62dbdcdbcc config: use the environment variable which goes with --password-command 2020-01-23 15:39:15 +00:00
Damon Permezel 06df133159 config: add --password-command to allow dynamic config password - fixes #3694 2020-01-23 15:39:15 +00:00
Nick Craig-Wood 277d94feac fshttp: add --expect-continue-timeout default 1s - fixes #3835
Before this change the expect/continue timeout was set to
--conntimeout which was 60s by default which is too long to wait.

This was noticed when using s3 with a proxy which apparently didn't
support expect / continue properly.

Set --expect-continue-timeout 0 to disable expect/continue.
2020-01-18 09:49:22 +00:00
Nick Craig-Wood 375d25f158 sync: implement --order-by flag to order transfers - fixes #1205 2020-01-16 15:24:36 +00:00
Nick Craig-Wood 3c620d521d backend: adjust backends to have encoding parameter
Fixes #3761
Fixes #3836
Fixes #3841
2020-01-16 14:40:36 +00:00
Nick Craig-Wood 207474abab sync: add --no-check-dest flag - fixes #3616 2019-12-29 16:47:57 +00:00
Barry Muldrey 2784c3234b fs/config/configflags: fix --compare-dest and --copy-dest help strings
from rsync manual:

--compare-dest=DIR
    This option instructs rsync to use DIR on the destination machine as an
    additional hierarchy to compare destination files against doing transfers
    (if the files are missing in the destination directory). If a file is found
    in DIR that is identical to the sender's file, the file will NOT be
    transferred to the destination directory. This is useful for creating
    a sparse backup of just files that have changed from an earlier backup.

--copy-dest=DIR
    This option behaves like --compare-dest, but rsync will also copy unchanged
     files found in DIR to the destination directory using a local copy.
      This is useful for doing transfers to a new destination while leaving
       existing files intact, and then doing a flash-cutover when all files
        have been successfully transferred.
2019-11-12 13:37:58 +00:00
buengese 5ddfa9f7f6 config: SetValueAndSave ignore error if config section does not exist yet 2019-11-09 16:44:08 +00:00
Nick Craig-Wood 049ff1f269 config: check a remote exists when creating a new one 2019-11-05 12:39:33 +00:00
Nick Craig-Wood 3f7af64316 config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
Xiaoxing Ye 520ddbcceb config: do not open browser on headless if google fs
On google fs (drive, google photos, and google cloud storage), if
headless is selected, do not open browser.

This also supplies a new option "auth-no-open-browser" for authorize
if the user does not want it.

This should fix #3323.
2019-10-30 14:12:42 +00:00
Nick Craig-Wood 1868c77e16 rc: fix formatting of docs 2019-10-27 10:43:40 +00:00
Nick Craig-Wood 1b2ffbeca0 cmd: fix environment variables not setting command line flags
Before this fix quite a lot of the commands were ignoring environment
variables intended to set flags.
2019-10-14 17:02:09 +01:00
Nick Craig-Wood 6e378d7d32 config: fix setting of non top level flags from environment variables
Before this fix, attempting to set a non top level environment
variable would fail with "Couldn't find flag".

This fixes it by passing in the flags that the env var is being set
from.

Fixes #3615
2019-10-14 17:02:09 +01:00
Nick Craig-Wood 455b9280ba config: use alternating Red/Green in config to make more obvious 2019-10-08 19:59:47 +01:00
Nick Craig-Wood 262ba28dec config: add ReadNonEmptyLine utility function 2019-09-30 11:23:21 +01:00
Nick Craig-Wood 6959c997e2 config: remove error: can't use --size-only and --ignore-size together.
As sometimes they are useful together, for example when the remote
changes the sizes of the uploaded files.

See: https://forum.rclone.org/t/files-upload-will-be-auto-compress-how-do-i-sync-a-file-to-remote/10578
2019-09-14 10:13:44 +01:00
Nick Craig-Wood f1347139fa config: check config names more carefully and report errors - fixes #3506
Before this change it was possible to make a remote with an invalid
name in the config file, either manually or with `rclone config
create` (but not with `rclone config`).

When this remote was used, because it was invalid, rclone would
presume this remote name was a local directory for a very suprising
user experience!

This change checks remote names more carefully and returns errors
- when the user tries to use an invalid remote name on the command line
- when an invalid remote name is used in `rclone config create/update/password`
- when the user tries to enter an invalid remote name in `rclone config`

This does not prevent the user entering a remote name with invalid
characters in the config manually, but such a remote will fail
immediately when it is used on the command line.
2019-09-06 12:07:09 +01:00
Nick Craig-Wood 5932acfee3 rc: fix docs for config/create /update /password 2019-09-03 08:34:15 +01:00
Nick Craig-Wood 693112d57e config: Fix generated passwords being stored as empty password - Fixes #3492 2019-08-28 12:11:03 +01:00
Nick Craig-Wood 193c30d570 Review random string/password generation
- factor password generation into lib/random.Password
- call from appropriate places
- choose appropriate use of random.String vs random.Password
2019-08-25 11:09:19 +01:00
Nick Craig-Wood 24161d12ab fs: make sure config is persisted to the config file when using config.Mapper 2019-08-14 20:54:08 +01:00
Nick Craig-Wood eb087a3b04 operations: disable multi thread copy for local to local copies #3419
...unless --multi-thread-streams has been set explicitly
2019-08-14 15:07:38 +01:00
Nick Craig-Wood 23a0332185 config: don't offer hidden values for editing in the config - fixes #3416 2019-08-14 08:40:22 +01:00
Nick Craig-Wood 3ecbd603ab rc: move job expire flags to rc to fix initalization problem
See: https://forum.rclone.org/t/rc-rc-job-expire-interval-bug/11188

rclone was ignoring the --rc-job-expire-duration and --rc-job-interval
flags.  This turned out to be an initialization order problem and was
fixed by moving those flags out of global config into rc config.
2019-08-10 17:12:22 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Aleksandar Jankovic 68e641f6cf accounting: add limits and listing to stats groups 2019-07-28 14:48:19 +01:00
justinalin 520fb03bfd log: add --use-json-log for JSON logging 2019-07-28 12:05:50 +01:00
Nick Craig-Wood 8fdce31700 config: Fix hiding of options from the configurator 2019-07-24 20:34:29 +01:00
yparitcher 8e8b78d7e5 Implement --compare-dest & --copy-dest Fixes #3278 2019-07-22 19:42:29 +01:00
yparitcher de14378734 Implement --suffix without --backup-dir for current dir
Fixes #2801
2019-07-01 10:46:26 +01:00
Nick Craig-Wood 0ae844d1f8 config: reset environment variables in config file test to fix build 2019-06-22 17:49:23 +01:00
Maran ba72e62b41 fs/config: Add method to reload configfile from disk
Fixes #3268
2019-06-19 14:47:54 +01:00
Aleksandar Jankovic f78cd1e043 Add context propagation to rclone
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
2019-06-19 11:59:46 +01:00
Wojciech Smigielski 2793d4b4cc remove duplicate code 2019-06-15 10:02:25 +01:00
Wojciech Smigielski 30ac9d920a enable creating encrypted config through external script invocation - fixes #3127 2019-06-15 10:02:25 +01:00
Aleksandar Jankovic 93207ead9c rc/jobs: make job expiry timeouts configurable 2019-06-15 09:55:32 +01:00
albertony 976a020a2f Use rclone.conf from rclone executable directory if already existing 2019-06-12 10:08:00 +01:00
Nick Craig-Wood 903ede52cd config: make config create/update encrypt passwords where necessary
Before this change when using "rclone config create" it wasn't
possible to add passwords in one go, it was necessary to call "rclone
config password" to add the passwords afterwards as "rclone config
create" didn't obscure passwords.

After this change "rclone config create" and "rclone config update"
will obscure passwords as necessary as will the corresponding API
calls config/create and config/update.

This makes "rclone config password" and its API config/password
obsolete, however they will be left for backwards compatibility.
2019-06-10 18:08:55 +01:00
garry415 1124c423ee fs: add --ignore-case-sync for forced case insensitivity - fixes #2773 2019-06-03 21:12:10 +01:00
Nick Craig-Wood edda6d91cd Use go-homedir to read the home directory more reliably 2019-05-11 23:39:04 +01:00
Nick Craig-Wood 7d70e92664 operations: enable multi threaded downloads - Fixes #2252
This implements the --multi-thread-cutoff and --multi-thread-streams
flags to control multi thread downloading to the local backend.
2019-05-11 23:35:19 +01:00
Peter Berbec 0d511b7878 cmd: implement --stats-one-line-date and --stats-one-line-date-format 2019-05-11 17:39:57 +01:00
Jon bbe28df800 docs: Fix typo: Dump HTTP bodies -> Dump HTTP headers 2019-05-11 16:40:40 +01:00