go/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-01-24 12:56:36 +02:00
Code Issues Releases Activity

azureblob: revamp authentication to include all methods and docs

Browse Source 
The updates the authentication to include

- Auth from the environment
    1. Environment Variables
    2. Managed Service Identity Credentials
    3. Azure CLI credentials (as used by the az tool)
- Account and Shared Key
- SAS URL
- Service principal with client secret
- Service principal with certificate
- User with username and password
- Managed Service Identity Credentials

And rationalises the auth order.
This commit is contained in:
Nick Craig-Wood 2022-11-30 11:45:34 +00:00
parent f3c8b7a948
commit d7cb17848d
2 changed files with 552 additions and 107 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

267
backend/azureblob/azureblob.go
View File

@ -40,7 +40,6 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"io/ioutil"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@ -65,6 +64,7 @@ import (
"github.com/rclone/rclone/fs/config" "github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/configmap" "github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/configstruct" "github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/config/obscure"
"github.com/rclone/rclone/fs/fserrors" "github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/fshttp" "github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash" "github.com/rclone/rclone/fs/hash"
@ -110,21 +110,20 @@ func init() {
NewFs: NewFs, NewFs: NewFs,
Options: []fs.Option{{ Options: []fs.Option{{
Name: "account", Name: "account",
Help: `Storage Account Name. Help: `Azure Storage Account Name.
Leave blank to use SAS URL or Emulator. Set this to the Azure Storage Account Name in use.
If this is blank then if env_auth is set it will be read from the Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
environment variable AZURE_CLIENT_ID.
If this is blank and if env_auth is set it will be read from the
environment variable ` + "`AZURE_STORAGE_ACCOUNT_NAME`" + ` if possible.
`, `,
}, { }, {
Name: "env_auth", Name: "env_auth",
Help: `Read credentials from runtime (environment variables). Help: `Read credentials from runtime (environment variables, CLI or MSI).
Pull credentials from AZURE_TENANT_ID and AZURE_CLIENT_{ID,SECRET} environment vars. See the [authentication docs](/azureblob#authentication) for full info.`,
See EnvironmentCredential in the Azure docs for more info.
Other authentication methods will, if specified, override this flag.`,
Default: false, Default: false,
}, { }, {
Name: "key", Name: "key",
@ -136,6 +135,78 @@ Leave blank to use SAS URL or Emulator.`,
Help: `SAS URL for container level access only. Help: `SAS URL for container level access only.
Leave blank if using account/key or Emulator.`, Leave blank if using account/key or Emulator.`,
}, {
Name: "tenant",
Help: `ID of the service principal's tenant. Also called its directory ID.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
`,
}, {
Name: "client_id",
Help: `The ID of the client in use.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
`,
}, {
Name: "client_secret",
Help: `One of the service principal's client secrets
Set this if using
- Service principal with client secret
`,
}, {
Name: "client_certificate_path",
Help: `Path to a PEM or PKCS12 certificate file including the private key.
Set this if using
- Service principal with certificate
`,
}, {
Name: "client_certificate_password",
Help: `Password for the certificate file (optional).
Optionally set this if using
- Service principal with certificate
And the certificate has a password.
`,
IsPassword: true,
}, {
Name: "client_send_certificate_chain",
Help: `Send the certificate chain when using certificate auth.
Specifies whether an authentication request will include an x5c header
to support subject name / issuer based authentication. When set to
true, authentication requests include the x5c header.
Optionally set this if using
- Service principal with certificate
`,
Default: false,
Advanced: true,
}, {
Name: "username",
Help: `User name (usually an email address)
Set this if using
- User with username and password
`,
Advanced: true,
}, {
Name: "password",
Help: `The user's password
Set this if using
- User with username and password
`,
IsPassword: true,
Advanced: true,
}, { }, {
Name: "service_principal_file", Name: "service_principal_file",
Help: `Path to file containing credentials for use with a service principal. Help: `Path to file containing credentials for use with a service principal.
@ -148,6 +219,10 @@ Leave blank normally. Needed only if you want to use a service principal instead
> azure-principal.json > azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details. See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
It may be more convenient to put the credentials directly into the
rclone config file under the ` + "`client_id`, `tenant` and `client_secret`" + `
keys instead of setting ` + "`service_principal_file`" + `.
`, `,
Advanced: true, Advanced: true,
}, { }, {
@ -334,29 +409,37 @@ rclone does if you know the container exists already.
// Options defines the configuration for this backend // Options defines the configuration for this backend
type Options struct { type Options struct {
Account string `config:"account"` Account string `config:"account"`
EnvAuth bool `config:"env_auth"` EnvAuth bool `config:"env_auth"`
Key string `config:"key"` Key string `config:"key"`
ServicePrincipalFile string `config:"service_principal_file"` SASURL string `config:"sas_url"`
UseMSI bool `config:"use_msi"` Tenant string `config:"tenant"`
MSIObjectID string `config:"msi_object_id"` ClientID string `config:"client_id"`
MSIClientID string `config:"msi_client_id"` ClientSecret string `config:"client_secret"`
MSIResourceID string `config:"msi_mi_res_id"` ClientCertificatePath string `config:"client_certificate_path"`
Endpoint string `config:"endpoint"` ClientCertificatePassword string `config:"client_certificate_password"`
SASURL string `config:"sas_url"` ClientSendCertificateChain bool `config:"client_send_certificate_chain"`
ChunkSize fs.SizeSuffix `config:"chunk_size"` Username string `config:"username"`
UploadConcurrency int `config:"upload_concurrency"` Password string `config:"password"`
ListChunkSize uint `config:"list_chunk"` ServicePrincipalFile string `config:"service_principal_file"`
AccessTier string `config:"access_tier"` UseMSI bool `config:"use_msi"`
ArchiveTierDelete bool `config:"archive_tier_delete"` MSIObjectID string `config:"msi_object_id"`
UseEmulator bool `config:"use_emulator"` MSIClientID string `config:"msi_client_id"`
DisableCheckSum bool `config:"disable_checksum"` MSIResourceID string `config:"msi_mi_res_id"`
MemoryPoolFlushTime fs.Duration `config:"memory_pool_flush_time"` Endpoint string `config:"endpoint"`
MemoryPoolUseMmap bool `config:"memory_pool_use_mmap"` ChunkSize fs.SizeSuffix `config:"chunk_size"`
Enc encoder.MultiEncoder `config:"encoding"` UploadConcurrency int `config:"upload_concurrency"`
PublicAccess string `config:"public_access"` ListChunkSize uint `config:"list_chunk"`
NoCheckContainer bool `config:"no_check_container"` AccessTier string `config:"access_tier"`
NoHeadObject bool `config:"no_head_object"` ArchiveTierDelete bool `config:"archive_tier_delete"`
UseEmulator bool `config:"use_emulator"`
DisableCheckSum bool `config:"disable_checksum"`
MemoryPoolFlushTime fs.Duration `config:"memory_pool_flush_time"`
MemoryPoolUseMmap bool `config:"memory_pool_use_mmap"`
Enc encoder.MultiEncoder `config:"encoding"`
PublicAccess string `config:"public_access"`
NoCheckContainer bool `config:"no_check_container"`
NoHeadObject bool `config:"no_head_object"`
} }
// Fs represents a remote azure server // Fs represents a remote azure server
@ -625,7 +708,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
case opt.EnvAuth: case opt.EnvAuth:
// Read account from environment if needed // Read account from environment if needed
if opt.Account == "" { if opt.Account == "" {
opt.Account, _ = os.LookupEnv("AZURE_CLIENT_ID") opt.Account, _ = os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME")
} }
// Read credentials from the environment // Read credentials from the environment
options := azidentity.DefaultAzureCredentialOptions{ options := azidentity.DefaultAzureCredentialOptions{
@ -636,41 +719,19 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
return nil, fmt.Errorf("create azure enviroment credential failed: %w", err) return nil, fmt.Errorf("create azure enviroment credential failed: %w", err)
} }
case opt.UseEmulator: case opt.UseEmulator:
if opt.Account != "" { if opt.Account == "" {
opt.Account = emulatorAccount opt.Account = emulatorAccount
} }
if opt.Key == "" { if opt.Key == "" {
opt.Key = emulatorAccountKey opt.Key = emulatorAccountKey
} }
if opt.Endpoint != "" { if opt.Endpoint == "" {
opt.Endpoint = emulatorBlobEndpoint opt.Endpoint = emulatorBlobEndpoint
} }
sharedKeyCred, err = service.NewSharedKeyCredential(opt.Account, opt.Key) sharedKeyCred, err = service.NewSharedKeyCredential(opt.Account, opt.Key)
if err != nil { if err != nil {
return nil, fmt.Errorf("create new shared key credential for emulator failed: %w", err) return nil, fmt.Errorf("create new shared key credential for emulator failed: %w", err)
} }
case opt.UseMSI:
// Specifying a user-assigned identity. Exactly one of the above IDs must be specified.
// Validate and ensure exactly one is set. (To do: better validation.)
var b2i = map[bool]int{false: 0, true: 1}
set := b2i[opt.MSIClientID != ""] + b2i[opt.MSIObjectID != ""] + b2i[opt.MSIResourceID != ""]
if set > 1 {
return nil, errors.New("more than one user-assigned identity ID is set")
}
var options azidentity.ManagedIdentityCredentialOptions
switch {
case opt.MSIClientID != "":
options.ID = azidentity.ClientID(opt.MSIClientID)
case opt.MSIObjectID != "":
// FIXME this doesn't appear to be in the new SDK?
return nil, fmt.Errorf("MSI object ID is currently unsupported")
case opt.MSIResourceID != "":
options.ID = azidentity.ResourceID(opt.MSIResourceID)
}
cred, err = azidentity.NewManagedIdentityCredential(&options)
if err != nil {
return nil, fmt.Errorf("failed to acquire MSI token: %w", err)
}
case opt.Account != "" && opt.Key != "": case opt.Account != "" && opt.Key != "":
sharedKeyCred, err = service.NewSharedKeyCredential(opt.Account, opt.Key) sharedKeyCred, err = service.NewSharedKeyCredential(opt.Account, opt.Key)
if err != nil { if err != nil {
@ -702,9 +763,69 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
_ = f.cntSVC(containerName) _ = f.cntSVC(containerName)
f.isLimited = true f.isLimited = true
} }
case opt.ClientID != "" && opt.Tenant != "" && opt.ClientSecret != "":
// Service principal with client secret
options := azidentity.ClientSecretCredentialOptions{
ClientOptions: policyClientOptions,
}
cred, err = azidentity.NewClientSecretCredential(opt.Tenant, opt.ClientID, opt.ClientSecret, &options)
if err != nil {
return nil, fmt.Errorf("error creating a client secret credential: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.ClientCertificatePath != "":
// Service principal with certificate
//
// Read the certificate
data, err := os.ReadFile(env.ShellExpand(opt.ClientCertificatePath))
if err != nil {
return nil, fmt.Errorf("error reading client certificate file: %w", err)
}
// NewClientCertificateCredential requires at least one *x509.Certificate, and a
// crypto.PrivateKey.
//
// ParseCertificates returns these given certificate data in PEM or PKCS12 format.
// It handles common scenarios but has limitations, for example it doesn't load PEM
// encrypted private keys.
var password []byte
if opt.ClientCertificatePassword != "" {
pw, err := obscure.Reveal(opt.Password)
if err != nil {
return nil, fmt.Errorf("certificate password decode failed - did you obscure it?: %w", err)
}
password = []byte(pw)
}
certs, key, err := azidentity.ParseCertificates(data, password)
if err != nil {
return nil, fmt.Errorf("failed to parse client certificate file: %w", err)
}
options := azidentity.ClientCertificateCredentialOptions{
ClientOptions: policyClientOptions,
SendCertificateChain: opt.ClientSendCertificateChain,
}
cred, err = azidentity.NewClientCertificateCredential(
opt.Tenant, opt.ClientID, certs, key, &options,
)
if err != nil {
return nil, fmt.Errorf("create azure service principal with client certificate credential failed: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.Username != "" && opt.Password != "":
// User with username and password
options := azidentity.UsernamePasswordCredentialOptions{
ClientOptions: policyClientOptions,
}
password, err := obscure.Reveal(opt.Password)
if err != nil {
return nil, fmt.Errorf("user password decode failed - did you obscure it?: %w", err)
}
cred, err = azidentity.NewUsernamePasswordCredential(
opt.Tenant, opt.ClientID, opt.Username, password, &options,
)
if err != nil {
return nil, fmt.Errorf("authenticate user with password failed: %w", err)
}
case opt.ServicePrincipalFile != "": case opt.ServicePrincipalFile != "":
// Try loading service principal credentials from file. // Loading service principal credentials from file.
loadedCreds, err := ioutil.ReadFile(env.ShellExpand(opt.ServicePrincipalFile)) loadedCreds, err := os.ReadFile(env.ShellExpand(opt.ServicePrincipalFile))
if err != nil { if err != nil {
return nil, fmt.Errorf("error opening service principal credentials file: %w", err) return nil, fmt.Errorf("error opening service principal credentials file: %w", err)
} }
@ -719,6 +840,28 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
if err != nil { if err != nil {
return nil, fmt.Errorf("error creating a client secret credential: %w", err) return nil, fmt.Errorf("error creating a client secret credential: %w", err)
} }
case opt.UseMSI:
// Specifying a user-assigned identity. Exactly one of the above IDs must be specified.
// Validate and ensure exactly one is set. (To do: better validation.)
var b2i = map[bool]int{false: 0, true: 1}
set := b2i[opt.MSIClientID != ""] + b2i[opt.MSIObjectID != ""] + b2i[opt.MSIResourceID != ""]
if set > 1 {
return nil, errors.New("more than one user-assigned identity ID is set")
}
var options azidentity.ManagedIdentityCredentialOptions
switch {
case opt.MSIClientID != "":
options.ID = azidentity.ClientID(opt.MSIClientID)
case opt.MSIObjectID != "":
// FIXME this doesn't appear to be in the new SDK?
return nil, fmt.Errorf("MSI object ID is currently unsupported")
case opt.MSIResourceID != "":
options.ID = azidentity.ResourceID(opt.MSIResourceID)
}
cred, err = azidentity.NewManagedIdentityCredential(&options)
if err != nil {
return nil, fmt.Errorf("failed to acquire MSI token: %w", err)
}
default: default:
return nil, errors.New("no authentication method configured") return nil, errors.New("no authentication method configured")
} }
@ -732,7 +875,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
} }
u, err := url.Parse(fmt.Sprintf("https://%s.%s", opt.Account, storageDefaultBaseURL)) u, err := url.Parse(fmt.Sprintf("https://%s.%s", opt.Account, storageDefaultBaseURL))
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to make azure storage URL from account and endpoint: %w", err) return nil, fmt.Errorf("failed to make azure storage URL from account: %w", err)
} }
opt.Endpoint = u.String() opt.Endpoint = u.String()
} }
@ -1776,7 +1919,7 @@ func urlEncode(in string) string {
if noNeedToEncode[c] { if noNeedToEncode[c] {
_ = out.WriteByte(c) _ = out.WriteByte(c)
} else { } else {
_, _ = out.WriteString(fmt.Sprintf("%%%2X", c)) _, _ = out.WriteString(fmt.Sprintf("%%%02X", c))
} }
} }
return out.String() return out.String()

392
docs/content/azureblob.md
View File

@ -116,11 +116,80 @@ MD5 hashes are stored with blobs. However blobs that were uploaded in
chunks only have an MD5 if the source remote was capable of MD5 chunks only have an MD5 if the source remote was capable of MD5
hashes, e.g. the local disk. hashes, e.g. the local disk.
### Authenticating with Azure Blob Storage ### Authentication {#authentication}
Rclone has 3 ways of authenticating with Azure Blob Storage: There are a number of ways of supplying credentials for Azure Blob
Storage. Rclone tries them in the order of the sections below.
#### Account and Key #### Env Auth
If the `env_auth` config parameter is `true` then rclone will pull
credentials from the environment or runtime.
It tries these authentication methods in this order:
1. Environment Variables
2. Managed Service Identity Credentials
3. Azure CLI credentials (as used by the az tool)
These are described in the following sections
##### Env Auth: 1. Environment Variables
If `env_auth` is set and environment variables are present rclone
authenticates a service principal with a secret or certificate, or a
user with a password, depending on which environment variable are set.
It reads configuration from these variables, in the following order:
1. Service principal with client secret
- `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID.
- `AZURE_CLIENT_ID`: the service principal's client ID
- `AZURE_CLIENT_SECRET`: one of the service principal's client secrets
2. Service principal with certificate
- `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID.
- `AZURE_CLIENT_ID`: the service principal's client ID
- `AZURE_CLIENT_CERTIFICATE_PATH`: path to a PEM or PKCS12 certificate file including the private key.
- `AZURE_CLIENT_CERTIFICATE_PASSWORD`: (optional) password for the certificate file.
- `AZURE_CLIENT_SEND_CERTIFICATE_CHAIN`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header.
3. User with username and password
- `AZURE_TENANT_ID`: (optional) tenant to authenticate in. Defaults to "organizations".
- `AZURE_CLIENT_ID`: client ID of the application the user will authenticate to
- `AZURE_USERNAME`: a username (usually an email address)
- `AZURE_PASSWORD`: the user's password
##### Env Auth: 2. Managed Service Identity Credentials
When using Managed Service Identity if the VM(SS) on which this
program is running has a system-assigned identity, it will be used by
default. If the resource has no system-assigned but exactly one
user-assigned identity, the user-assigned identity will be used by
default.
If the resource has multiple user-assigned identities you will need to
unset `env_auth` and set `use_msi` instead. See the [`use_msi`
section](#use_msi).
##### Env Auth: 3. Azure CLI credentials (as used by the az tool)
Credentials created with the `az` tool can be picked up using `env_auth`.
For example if you were to login with a service principal like this:
az login --service-principal -u XXX -p XXX --tenant XXX
Then you could access rclone resources like this:
rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER
Or
rclone lsf --azureblob-env-auth --azureblob-acccount=ACCOUNT :azureblob:CONTAINER
Which is analogous to using the `az` tool:
az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login
#### Account and Shared Key
This is the most straight forward and least flexible way. Just fill This is the most straight forward and least flexible way. Just fill
in the `account` and `key` lines and leave the rest blank. in the `account` and `key` lines and leave the rest blank.
@ -129,7 +198,7 @@ in the `account` and `key` lines and leave the rest blank.
This can be an account level SAS URL or container level SAS URL. This can be an account level SAS URL or container level SAS URL.
To use it leave `account`, `key` blank and fill in `sas_url`. To use it leave `account` and `key` blank and fill in `sas_url`.
An account level SAS URL or container level SAS URL can be obtained An account level SAS URL or container level SAS URL can be obtained
from the Azure portal or the Azure Storage Explorer. To get a from the Azure portal or the Azure Storage Explorer. To get a
@ -156,6 +225,60 @@ Container level SAS URLs are useful for temporarily allowing third
parties access to a single container or putting credentials into an parties access to a single container or putting credentials into an
untrusted environment such as a CI build server. untrusted environment such as a CI build server.
#### Service principal with client secret
If these variables are set, rclone will authenticate with a service principal with a client secret.
- `tenant`: ID of the service principal's tenant. Also called its "directory" ID.
- `client_id`: the service principal's client ID
- `client_secret`: one of the service principal's client secrets
The credentials can also be placed in a file using the
`service_principal_file` configuration option.
#### Service principal with certificate
If these variables are set, rclone will authenticate with a service principal with certificate.
- `tenant`: ID of the service principal's tenant. Also called its "directory" ID.
- `client_id`: the service principal's client ID
- `client_certificate_path`: path to a PEM or PKCS12 certificate file including the private key.
- `client_certificate_password`: (optional) password for the certificate file.
- `client_send_certificate_chain`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header.
**NB** `client_certificate_password` must be obscured - see [rclone obscure](/commands/rclone_obscure/).
#### User with username and password
If these variables are set, rclone will authenticate with username and password.
- `tenant`: (optional) tenant to authenticate in. Defaults to "organizations".
- `client_id`: client ID of the application the user will authenticate to
- `username`: a username (usually an email address)
- `password`: the user's password
Microsoft doesn't recommend this kind of authentication, because it's
less secure than other authentication flows. This method is not
interactive, so it isn't compatible with any form of multi-factor
authentication, and the application must already have user or admin
consent. This credential can only authenticate work and school
accounts; it can't authenticate Microsoft accounts.
**NB** `password` must be obscured - see [rclone obscure](/commands/rclone_obscure/).
#### Managed Service Identity Credentials {#use_msi}
If `use_msi` is set then managed service identity credentials are
used. This authentication only works when running in an Azure service.
`env_auth` needs to be unset to use this.
However if you have multiple user identities to choose from these must
be explicitly specified using exactly one of the `msi_object_id`,
`msi_client_id`, or `msi_mi_res_id` parameters.
If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is
set, this is is equivalent to using `env_auth`.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/azureblob/azureblob.go then run make backenddocs" >}} {{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/azureblob/azureblob.go then run make backenddocs" >}}
### Standard options ### Standard options
@ -163,9 +286,15 @@ Here are the Standard options specific to azureblob (Microsoft Azure Blob Storag
#### --azureblob-account #### --azureblob-account
Storage Account Name. Azure Storage Account Name.
Set this to the Azure Storage Account Name in use.
Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
Leave blank to use SAS URL.
Properties: Properties:
@ -174,32 +303,24 @@ Properties:
- Type: string - Type: string
- Required: false - Required: false
#### --azureblob-service-principal-file #### --azureblob-env-auth
Path to file containing credentials for use with a service principal. Read credentials from runtime (environment variables, CLI or MSI).
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "<name>" \
--role "Storage Blob Data Owner" \
--scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
> azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
See the [authentication docs](/azureblob#authentication) for full info.
Properties: Properties:
- Config: service_principal_file - Config: env_auth
- Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE - Env Var: RCLONE_AZUREBLOB_ENV_AUTH
- Type: string - Type: bool
- Required: false - Default: false
#### --azureblob-key #### --azureblob-key
Storage Account Key. Storage Account Shared Key.
Leave blank to use SAS URL. Leave blank to use SAS URL or Emulator.
Properties: Properties:
@ -221,6 +342,169 @@ Properties:
- Type: string - Type: string
- Required: false - Required: false
#### --azureblob-tenant
ID of the service principal's tenant. Also called its directory ID.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
Properties:
- Config: tenant
- Env Var: RCLONE_AZUREBLOB_TENANT
- Type: string
- Required: false
#### --azureblob-client-id
The ID of the client in use.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
Properties:
- Config: client_id
- Env Var: RCLONE_AZUREBLOB_CLIENT_ID
- Type: string
- Required: false
#### --azureblob-client-secret
One of the service principal's client secrets
Set this if using
- Service principal with client secret
Properties:
- Config: client_secret
- Env Var: RCLONE_AZUREBLOB_CLIENT_SECRET
- Type: string
- Required: false
#### --azureblob-client-certificate-path
Path to a PEM or PKCS12 certificate file including the private key.
Set this if using
- Service principal with certificate
Properties:
- Config: client_certificate_path
- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PATH
- Type: string
- Required: false
#### --azureblob-client-certificate-password
Password for the certificate file (optional).
Optionally set this if using
- Service principal with certificate
And the certificate has a password.
**NB** Input to this must be obscured - see [rclone obscure](/commands/rclone_obscure/).
Properties:
- Config: client_certificate_password
- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PASSWORD
- Type: string
- Required: false
### Advanced options
Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).
#### --azureblob-client-send-certificate-chain
Send the certificate chain when using certificate auth.
Specifies whether an authentication request will include an x5c header
to support subject name / issuer based authentication. When set to
true, authentication requests include the x5c header.
Optionally set this if using
- Service principal with certificate
Properties:
- Config: client_send_certificate_chain
- Env Var: RCLONE_AZUREBLOB_CLIENT_SEND_CERTIFICATE_CHAIN
- Type: bool
- Default: false
#### --azureblob-username
User name (usually an email address)
Set this if using
- User with username and password
Properties:
- Config: username
- Env Var: RCLONE_AZUREBLOB_USERNAME
- Type: string
- Required: false
#### --azureblob-password
The user's password
Set this if using
- User with username and password
**NB** Input to this must be obscured - see [rclone obscure](/commands/rclone_obscure/).
Properties:
- Config: password
- Env Var: RCLONE_AZUREBLOB_PASSWORD
- Type: string
- Required: false
#### --azureblob-service-principal-file
Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "<name>" \
--role "Storage Blob Data Owner" \
--scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
> azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
It may be more convenient to put the credentials directly into the
rclone config file under the `client_id`, `tenant` and `client_secret`
keys instead of setting `service_principal_file`.
Properties:
- Config: service_principal_file
- Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE
- Type: string
- Required: false
#### --azureblob-use-msi #### --azureblob-use-msi
Use a managed service identity to authenticate (only works in Azure). Use a managed service identity to authenticate (only works in Azure).
@ -241,25 +525,6 @@ Properties:
- Type: bool - Type: bool
- Default: false - Default: false
#### --azureblob-use-emulator
Uses local storage emulator if provided as 'true'.
Leave blank if using real azure storage endpoint.
Provide `account`, `key`, and `endpoint` if desired to override their _azurite_ defaults.
Properties:
- Config: use_emulator
- Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
- Type: bool
- Default: false
### Advanced options
Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).
#### --azureblob-msi-object-id #### --azureblob-msi-object-id
Object ID of the user-assigned MSI to use, if any. Object ID of the user-assigned MSI to use, if any.
@ -299,6 +564,19 @@ Properties:
- Type: string - Type: string
- Required: false - Required: false
#### --azureblob-use-emulator
Uses local storage emulator if provided as 'true'.
Leave blank if using real azure storage endpoint.
Properties:
- Config: use_emulator
- Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
- Type: bool
- Default: false
#### --azureblob-endpoint #### --azureblob-endpoint
Endpoint for the service. Endpoint for the service.
@ -502,6 +780,21 @@ Properties:
- "container" - "container"
- Allow full public read access for container and blob data. - Allow full public read access for container and blob data.
#### --azureblob-no-check-container
If set, don't attempt to check the container exists or create it.
This can be useful when trying to minimise the number of transactions
rclone does if you know the container exists already.
Properties:
- Config: no_check_container
- Env Var: RCLONE_AZUREBLOB_NO_CHECK_CONTAINER
- Type: bool
- Default: false
#### --azureblob-no-head-object #### --azureblob-no-head-object
If set, do not do HEAD before GET when getting objects. If set, do not do HEAD before GET when getting objects.
@ -541,8 +834,17 @@ See [List of backends that do not support rclone about](https://rclone.org/overv
## Azure Storage Emulator Support ## Azure Storage Emulator Support
You can run rclone with storage emulator (usually _azurite_). You can run rclone with the storage emulator (usually _azurite_).
To do this, just set up a new remote with `rclone config` following instructions described in introduction and set `use_emulator` config as `true`. You do not need to provide default account name neither an account key. But you can override them in the _account_ and _key_ options. (Prior to v1.61 they were hard coded to _azurite_'s `devstoreaccount1`.) To do this, just set up a new remote with `rclone config` following
the instructions in the introduction and set `use_emulator` in the
advanced settings as `true`. You do not need to provide a default
account name nor an account key. But you can override them in the
`account` and `key` options. (Prior to v1.61 they were hard coded to
_azurite_'s `devstoreaccount1`.)
Also, if you want to access a storage emulator instance running on a different machine, you can override _Endpoint_ parameter in advanced settings, setting it to `http(s)://<host>:<port>/devstoreaccount1` (e.g. `http://10.254.2.5:10000/devstoreaccount1`). Also, if you want to access a storage emulator instance running on a
different machine, you can override the `endpoint` parameter in the
advanced settings, setting it to
`http(s)://<host>:<port>/devstoreaccount1`
(e.g. `http://10.254.2.5:10000/devstoreaccount1`).