rclone/docs/content/hdfs.md

---
title: "HDFS Remote"
description: "Remote for Hadoop Distributed Filesystem"
versionIntroduced: "v1.54"
---

# {{< icon "fa fa-globe" >}} HDFS

[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)
is a distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/)
framework.

Paths are specified as `remote:` or `remote:path/to/dir`.

## Configuration

Here is an example of how to make a remote called `remote`. First run:

```console
rclone config
```

This will guide you through an interactive setup process:

```text
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[skip]
XX / Hadoop distributed file system
   \ "hdfs"
[skip]
Storage> hdfs
** See help for hdfs backend at: https://rclone.org/hdfs/ **

hadoop name node and port
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Connect to host namenode at port 8020
   \ "namenode:8020"
namenode> namenode.hadoop:8020
hadoop user name
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Connect to hdfs as root
   \ "root"
username> root
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
Configuration complete.
Options:
- type: hdfs
- namenode: namenode.hadoop:8020
- username: root
Keep this "remote" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
hadoop               hdfs

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
```

This remote is called `remote` and can now be used like this

See all the top level directories

```console
rclone lsd remote:
```

List the contents of a directory

```console
rclone ls remote:directory
```

Sync the remote `directory` to `/home/local/directory`, deleting any excess files.

```console
rclone sync --interactive remote:directory /home/local/directory
```

### Setting up your own HDFS instance for testing

You may start with a [manual setup](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)
or use the docker image from the tests:

If you want to build the docker image

```console
git clone https://github.com/rclone/rclone.git
cd rclone/fstest/testserver/images/test-hdfs
docker build --rm -t rclone/test-hdfs .
```

Or you can just use the latest one pushed

```console
docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs
```

**NB** it need few seconds to startup.

For this docker image the remote needs to be configured like this:

```ini
[remote]
type = hdfs
namenode = 127.0.0.1:8020
username = root
```

You can stop this image with `docker kill rclone-hdfs` (**NB** it does not use
volumes, so all data uploaded will be lost.)

### Modification times

Time accurate to 1 second is stored.

### Checksum

No checksums are implemented.

### Usage information

You can use the `rclone about remote:` command which will display filesystem
size and current usage.

### Restricted filename characters

In addition to the [default restricted characters set](/overview/#restricted-characters)
the following characters are also replaced:

| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
| :         | 0x3A  | ：           |

Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8).

<!-- autogenerated options start - DO NOT EDIT - instead edit fs.RegInfo in backend/hdfs/hdfs.go and run make backenddocs to verify --> <!-- markdownlint-disable-line line-length -->
### Standard options

Here are the Standard options specific to hdfs (Hadoop distributed file system).

#### --hdfs-namenode

Hadoop name nodes and ports.

E.g. "namenode-1:8020,namenode-2:8020,..." to connect to host namenodes at port 8020.

Properties:

- Config:      namenode
- Env Var:     RCLONE_HDFS_NAMENODE
- Type:        CommaSepList
- Default:     

#### --hdfs-username

Hadoop user name.

Properties:

- Config:      username
- Env Var:     RCLONE_HDFS_USERNAME
- Type:        string
- Required:    false
- Examples:
    - "root"
        - Connect to hdfs as root.

### Advanced options

Here are the Advanced options specific to hdfs (Hadoop distributed file system).

#### --hdfs-service-principal-name

Kerberos service principal name for the namenode.

Enables KERBEROS authentication. Specifies the Service Principal Name
(SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\"
for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.

Properties:

- Config:      service_principal_name
- Env Var:     RCLONE_HDFS_SERVICE_PRINCIPAL_NAME
- Type:        string
- Required:    false

#### --hdfs-data-transfer-protection

Kerberos data transfer protection: authentication|integrity|privacy.

Specifies whether or not authentication, data signature integrity
checks, and wire encryption are required when communicating with
the datanodes. Possible values are 'authentication', 'integrity'
and 'privacy'. Used only with KERBEROS enabled.

Properties:

- Config:      data_transfer_protection
- Env Var:     RCLONE_HDFS_DATA_TRANSFER_PROTECTION
- Type:        string
- Required:    false
- Examples:
    - "privacy"
        - Ensure authentication, integrity and encryption enabled.

#### --hdfs-encoding

The encoding for the backend.

See the [encoding section in the overview](/overview/#encoding) for more info.

Properties:

- Config:      encoding
- Env Var:     RCLONE_HDFS_ENCODING
- Type:        Encoding
- Default:     Slash,Colon,Del,Ctl,InvalidUtf8,Dot

#### --hdfs-description

Description of the remote.

Properties:

- Config:      description
- Env Var:     RCLONE_HDFS_DESCRIPTION
- Type:        string
- Required:    false

<!-- autogenerated options stop -->

## Limitations

- Erasure coding not supported, see [issue #8808](https://github.com/rclone/rclone/issues/8808)
- No server-side `Move` or `DirMove`.
- Checksums not implemented.