yves
/
rclone
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

drive: Rework token aquisition into config framework and store token in config file

s3-about
Nick Craig-Wood 2014-03-16 14:01:17 +00:00
parent 1b3a49929b
commit b4dd693d23
3 changed files with 182 additions and 128 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

87
README.md
View File

@ -1,11 +1,13 @@
Rclone
======
[![Logo](http://rclone.org/rclone-120x120.png)](http://rclone.org/)
Sync files and directories to and from
* Openstack Swift / Rackspace cloud files / Memset Memstore
* Amazon S3
* Google Drive
* Amazon S3
* Openstack Swift / Rackspace cloud files / Memset Memstore
* The local filesystem
Features
@ -16,6 +18,11 @@ Features
* Copy mode to just copy new/changed files
* Sync mode to make a directory identical
* Check mode to check all MD5SUMs
* Can sync to and from network, eg two different Drive accounts
Home page
* http://rclone.org/
Install
-------
@ -30,8 +37,9 @@ Or alternatively if you have Go installed use
go get github.com/ncw/rclone
and this will build the binary in `$GOPATH/bin`. You can then modify
the source and submit patches.
and this will build the binary in `$GOPATH/bin`.
You can then modify the source and submit patches.
Configure
---------
@ -75,9 +83,6 @@ Choose a number from below, or type in your own value
* US Region, Northern Virginia only.
* Leave location constraint empty.
2) https://s3-external-1.amazonaws.com
* US West (Oregon) Region
* Needs location constraint us-west-2.
3) https://s3-us-west-2.amazonaws.com
[snip]
* South America (Sao Paulo) Region
* Needs location constraint sa-east-1.
@ -89,8 +94,6 @@ Choose a number from below, or type in your own value
1)
* US West (Oregon) Region.
2) us-west-2
* US West (Northern California) Region.
3) us-west-1
[snip]
* South America (Sao Paulo) Region.
9) sa-east-1
@ -240,11 +243,57 @@ Google drive
Paths are specified as drive://path Drive paths may be as deep as required.
FIXME describe how to set up initially
The initial setup for drive involves getting a token from Google drive
which you need to do in your browser. The `rclone config` walks you
through it.
So to copy a local directory to a drive directory called backup
Here is an example of how to make a remote called `drv`
rclone sync /home/source s3://backup
```
$ ./rclone config
n) New remote
d) Delete remote
q) Quit config
e/n/d/q> n
name> drv
What type of source is it?
Choose a number from below
1) swift
2) s3
3) local
4) drive
type> 4
Google Application Client Id - leave blank to use rclone's.
client_id>
Google Application Client Secret - leave blank to use rclone's.
client_secret>
Remote config
Go to the following link in your browser
https://accounts.google.com/o/oauth2/auth?access_type=&approval_prompt=&client_id=XXXXXXXXXXXX.apps.googleusercontent.com&redirect_uri=urn%3XXXXX%3Awg%3Aoauth%3XX.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=state
Log in, then type paste the token that is returned in the browser here
Enter verification code> X/XXXXXXXXXXXXXXXXXX-XXXXXXXXX.XXXXXXXXX-XXXXX_XXXXXXX_XXXXXXX
--------------------
[drv]
client_id =
client_secret =
token = {"AccessToken":"xxxx.xxxxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
```
You can then use it like this
rclone lsd drv://
rclone ls drv://
To copy a local directory to a drive directory called backup
rclone copy /home/source drv://backup
Google drive stores modification times accurate to 1 ms.
License
-------
@ -255,25 +304,27 @@ COPYING file included in this package).
Bugs
----
Save the google drive auth in this config file too!
Describe how to do the google auth.
* Doesn't sync individual files yet, only directories.
* Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
* quota is 100.0 requests/second/user
* Empty directories left behind with Local and Drive
* eg purging a local directory with subdirectories doesn't work
Contact and support
-------------------
The project website is at:
- https://github.com/ncw/rclone
* https://github.com/ncw/rclone
There you can file bug reports, ask for help or contribute patches.
Authors
-------
- Nick Craig-Wood <nick@craig-wood.com>
* Nick Craig-Wood <nick@craig-wood.com>
Contributors
------------
- Your name goes here!
* Your name goes here!

195
drive/drive.go
View File

@ -9,12 +9,6 @@ package drive
// FIXME list directory should list to channel for concurrency not
// append to array
// FIXME perhaps have a drive setup mode where we ask for all the
// params interactively and store them all in one file
// - don't need to store client* apparently
// NB permissions of token file is too open
// FIXME need to deal with some corner cases
// * multiple files with the same name
// * files can be in multiple directories
@ -22,14 +16,13 @@ package drive
// * files with / in name
import (
"errors"
"encoding/json"
"flag"
"fmt"
"io"
"log"
"mime"
"net/http"
"os"
"path"
"strings"
"sync"
@ -40,36 +33,106 @@ import (
"github.com/ncw/rclone/fs"
)
// Constants
const (
rcloneClientId = "202264815644.apps.googleusercontent.com"
rcloneClientSecret = "X4Z3ca8xfWDb1Voo-F9a7ZxJ"
driveFolderType = "application/vnd.google-apps.folder"
)
// Globals
var (
// Flags
driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
)
// Register with Fs
func init() {
fs.Register(&fs.FsInfo{
Name: "drive",
NewFs: NewFs,
Name: "drive",
NewFs: NewFs,
Config: Config,
Options: []fs.Option{{
Name: "client_id",
Help: "Google Application Client Id.",
Examples: []fs.OptionExample{{
Value: "202264815644.apps.googleusercontent.com",
Help: "rclone's client id - use this or your own if you want",
}},
Help: "Google Application Client Id - leave blank to use rclone's.",
}, {
Name: "client_secret",
Help: "Google Application Client Secret.",
Examples: []fs.OptionExample{{
Value: "X4Z3ca8xfWDb1Voo-F9a7ZxJ",
Help: "rclone's client secret - use this or your own if you want",
}},
}, {
Name: "token_file",
Help: "Path to store token file.",
Examples: []fs.OptionExample{{
Value: path.Join(fs.HomeDir, ".gdrive-token-file"),
Help: "Suggested path for token file",
}},
Help: "Google Application Client Secret - leave blank to use rclone's.",
}},
})
}
// Configuration helper - called after the user has put in the defaults
func Config(name string) {
// See if already have a token
tokenString := fs.ConfigFile.MustValue(name, "token")
if tokenString != "" {
fmt.Printf("Already have a drive token - refresh?\n")
if !fs.Confirm() {
return
}
}
// Get a drive transport
t, err := newDriveTransport(name)
if err != nil {
log.Fatalf("Couldn't make drive transport: %v", err)
}
// Generate a URL for the user to visit for authorization.
authUrl := t.Config.AuthCodeURL("state")
fmt.Printf("Go to the following link in your browser\n")
fmt.Printf("%s\n", authUrl)
fmt.Printf("Log in, then type paste the token that is returned in the browser here\n")
// Read the code, and exchange it for a token.
fmt.Printf("Enter verification code> ")
authCode := fs.ReadLine()
_, err = t.Exchange(authCode)
if err != nil {
log.Fatalf("Failed to get token: %v", err)
}
}
// A token cache to save the token in the config file section named
type tokenCache string
// Get the token from the config file - returns an error if it isn't present
func (name tokenCache) Token() (*oauth.Token, error) {
tokenString, err := fs.ConfigFile.GetValue(string(name), "token")
if err != nil {
return nil, err
}
if tokenString == "" {
return nil, fmt.Errorf("Empty token found - please reconfigure")
}
token := new(oauth.Token)
err = json.Unmarshal([]byte(tokenString), token)
if err != nil {
return nil, err
}
return token, nil
}
// Save the token to the config file
//
// This saves the config file if it changes
func (name tokenCache) PutToken(token *oauth.Token) error {
tokenBytes, err := json.Marshal(token)
if err != nil {
return err
}
tokenString := string(tokenBytes)
old := fs.ConfigFile.MustValue(string(name), "token")
if tokenString != old {
fs.ConfigFile.SetValue(string(name), "token", tokenString)
fs.SaveConfig()
}
return nil
}
// FsDrive represents a remote drive server
type FsDrive struct {
svc *drive.Service // the connection to the drive server
@ -141,19 +204,6 @@ func (m *dirCache) Flush() {
// ------------------------------------------------------------
// Constants
const (
// defaultDriveTokenFile = ".google-drive-token" // FIXME root in home directory somehow
driveFolderType = "application/vnd.google-apps.folder"
)
// Globals
var (
// Flags
driveAuthCode = flag.String("drive-auth-code", "", "Pass in when requested to make the drive token file.")
driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
)
// String converts this FsDrive to a string
func (f *FsDrive) String() string {
return fmt.Sprintf("Google drive root '%s'", f.root)
@ -214,39 +264,15 @@ OUTER:
return
}
// Ask the user for a new auth
func MakeNewToken(t *oauth.Transport) error {
if *driveAuthCode == "" {
// Generate a URL to visit for authorization.
authUrl := t.Config.AuthCodeURL("state")
fmt.Fprintf(os.Stderr, "Go to the following link in your browser\n")
fmt.Fprintf(os.Stderr, "%s\n", authUrl)
fmt.Fprintf(os.Stderr, "Log in, then re-run this program with the -drive-auth-code parameter\n")
fmt.Fprintf(os.Stderr, "You only need this parameter once until the drive token file has been created\n")
return errors.New("Re-run with --drive-auth-code")
}
// Read the code, and exchange it for a token.
//fmt.Printf("Enter verification code: ")
//var code string
//fmt.Scanln(&code)
_, err := t.Exchange(*driveAuthCode)
return err
}
// NewFs contstructs an FsDrive from the path, container:path
func NewFs(name, path string) (fs.Fs, error) {
// Makes a new drive transport from the config
func newDriveTransport(name string) (*oauth.Transport, error) {
clientId := fs.ConfigFile.MustValue(name, "client_id")
if clientId == "" {
return nil, errors.New("client_id not found")
clientId = rcloneClientId
}
clientSecret := fs.ConfigFile.MustValue(name, "client_secret")
if clientSecret == "" {
return nil, errors.New("client_secret not found")
}
tokenFile := fs.ConfigFile.MustValue(name, "token_file")
if tokenFile == "" {
return nil, errors.New("token-file not found")
clientSecret = rcloneClientSecret
}
// Settings for authorization.
@ -257,7 +283,22 @@ func NewFs(name, path string) (fs.Fs, error) {
RedirectURL: "urn:ietf:wg:oauth:2.0:oob",
AuthURL: "https://accounts.google.com/o/oauth2/auth",
TokenURL: "https://accounts.google.com/o/oauth2/token",
TokenCache: oauth.CacheFile(tokenFile),
TokenCache: tokenCache(name),
}
t := &oauth.Transport{
Config: driveConfig,
Transport: http.DefaultTransport,
}
return t, nil
}
// NewFs contstructs an FsDrive from the path, container:path
func NewFs(name, path string) (fs.Fs, error) {
t, err := newDriveTransport(name)
if err != nil {
return nil, err
}
root, err := parseDrivePath(path)
@ -266,22 +307,10 @@ func NewFs(name, path string) (fs.Fs, error) {
}
f := &FsDrive{root: root, dirCache: newDirCache()}
t := &oauth.Transport{
Config: driveConfig,
Transport: http.DefaultTransport,
}
// Try to pull the token from the cache; if this fails, we need to get one.
token, err := driveConfig.TokenCache.Token()
token, err := t.Config.TokenCache.Token()
if err != nil {
err := MakeNewToken(t)
if err != nil {
return nil, fmt.Errorf("Failed to authorise: %s", err)
}
} else {
if *driveAuthCode != "" {
return nil, fmt.Errorf("Only supply -drive-auth-code once")
}
return nil, fmt.Errorf("Failed to get token: %s", err)
}
t.Token = token

28
notes.txt
View File

@ -4,7 +4,7 @@ Todo
* FIXME: ls without an argument for buckets/containers?
* FIXME: More -dry-run checks for object transfer
* Might be quicker to check md5sums first? for swift <-> swift certainly, and maybe for small files
* Ignoring the pseudo directories
* swift: Ignoring the pseudo directories
* if object.PseudoDirectory {
* fmt.Printf("%9s %19s %s\n", "Directory", "-", fs.Remote())
* Make Account wrapper
@ -20,20 +20,6 @@ Todo
* Add max object size to fs metadata - 5GB for swift, infinite for local, ? for s3
* tie into -max-size flag
Drive
* Do we need the secrets or just the code? If just the code then
can make a web service which does the request on the clients
behalf so don't need to expose the client secrets
* Apparently we don't need -drive-client-id or -drive-client-secret once we have a token
* Make a cgi which we send the user to
* It has the client secrets
* It gets google to authenticate
* It receives the token back
* It displays the token to the user to paste in to the code
* Should be https really
* Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
* quota is 100.0 requests/second/user
Ideas
* could do encryption - put IV into metadata?
* optimise remote copy container to another container using remote
@ -45,7 +31,6 @@ Ideas
* Google cloud storage: https://developers.google.com/storage/
* rsync over ssh
* dropbox: https://github.com/nickoneill/go-dropbox (no MD5s)
* grive seems to have its secrets in the source code which would make things easier!
Need to make directory objects otherwise can't upload an empty directory
* Or could upload empty directories only?
@ -57,17 +42,6 @@ s3
* Otherwise can set metadata
* Returns etag and last modified in bucket list
Bugs
local & drive need to delete directories
2013/01/18 16:31:32 Waiting for deletions to finish
2013/01/18 16:31:32 z3: FIXME Skipping directory
2013/01/18 16:31:32 z3/x: Deleted
2013/01/18 16:31:32 Deleting path
2013/01/18 16:31:32 Rmdir failed: remove z3: directory not empty
------------------------------------------------------------
Non verbose - not sure number transferred got counted up? CHECK