A do-it-yourself NAS on kubernetes

As one of the last steps in moving my VM based setup at home to kubernetes, it is now the turn to migrate my ‘NAS’. The term ‘NAS’ is a too much honor for it since it is basically just a single shared file system exposed over SSHFS, Samba, and rsync. The setup discussed below however still supports multiple shares. Basically what I need from my NAS setup is the following:

  • it must be accessible on Windows and linux
  • efficient backups of important files must be possible using rsync (such as my complete home directory on linux)

For accessibility on linux I use sshfs and for other OSes I use samba. For the backup functionality I use Rsync.

A single pod

For the kubernetes setup I will use the simplest setup I can find. This amounts to a single pod with three containers:

  • a samba container
  • an Rsync container
  • an SFTP container used for SSHFS

PlantUML Syntax:<br />
allow_mixing<br />
scale 1.0<br />
hide circle</p>
<p>node “NAS” {<br />
component “samba” as samba<br />
component “rsync” as rsync<br />
component “sftp” as sftp<br />
<p>database “nas-data” as data1</p>
<p>NAS -down-> data1<br />

I am using a Deployment with replica count of 1. In this case, using a Deployment is ok since the underlying storage supports concurrent access. The pod can use 1 or more data volumes.

The Deployment is defined as follows:

apiVersion: apps/v1
kind: Deployment
  name: nas
  namespace: example-com
  replicas: 1
      app: nas-server
        app: nas-server
      terminationGracePeriodSeconds: 0                  # A 
        - name: nas
          # amd64 image
            - name: TZ
              value: Europe/Amsterdam
            - name: USERID
              value: "1016"
            - name: GROUPID
              value: "1016"
            - name: PASSWORD
                  name: nas-password
                  key: password
          image: dperson/samba:amd64@sha256:e1d2a7366690749a7be06f72bdbf6a5a7d15726fc84e4e4f41e967214516edfd
            - -g
            - mangled names = no
            - -g                                      # A 
            - catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6
            - -g
            - create mask = 0600
            - -g
            - force create mode = 0600
            - -g
            - create directory mask = 0700
            - -g
            - directory mask = 0700
            - -g
            - force directory mode = 0700
            - -u
            - erik;$(PASSWORD)
            - -s
            - data;/mount/files;no;no;no;erik         # B
            - containerPort: 137                      # C 
              protocol: UDP
            - containerPort: 138
              protocol: UDP
            - containerPort: 139                      
            - containerPort: 445
            - name: nas-data                          # D 
              mountPath: /mount
        - name: rsync
          image: repo.example.com/rsync:1.0           # E 
            - containerPort: 873                      # F 
            - name: rsyncd-secret                     # G
              mountPath: /etc/rsyncd.conf
              subPath: rsyncd.conf
            - name: rsyncd-secret                     # H 
              mountPath: /etc/rsyncd.secrets
              subPath: rsyncd.secrets
            - name: nas-data                          # I 
              mountPath: /data
        - name: sftp
          image: cat.wamblee.org/sftp:1.0             # J
            - containerPort: 22                       # K 
            - name: sftp-secret                       # L 
              mountPath: /etc/sftpusers.conf
              subPath: sftpusers.conf
            - name: nas-data                          # M 
              mountPath: /data
        - name: rsyncd-secret
            secretName: rsyncd-secret
            defaultMode: 0600
        - name: sftp-secret
            secretName: sftp-secret
        - name: nas-data  
            claimName: nas-data


The samba container

For the samba container, I am using a third-party container:

  • # A: These mappings are used to map non-supported characters by samba (such as ‘:’) to characters that are supported. Without this, filenames containing these type of characters would be shown as mangled names.
  • # B: Configuration of the a single samba share. This option can be repeated to support multiple shares.
  • # C: The UDP ports are not used and can be removed from the Deployment. The UDP ports are required for automatic discovery of the samba shares, but these options do not work without host networking. I decided not to pursue this discovery for security reasons. Now, I simply need to configure the explicit path to the samba share.
  • # D: The container requires the data directory to be mounted at /mount.

The rsync container

  • # E: I am using a custom container built using a Dockerfile:
    FROM rockylinux:9.0.20220720
    RUN yum makecache
    RUN yum install rsync -y
    EXPOSE 873
    CMD ["/usr/bin/rsync", "--no-detach", "--daemon", "--config", "/etc/rsyncd.conf"]

    This container simply runs rsync and exposes port 873 by default when run from docker for testing. The setup assumes that /etc/rsyncd.conf and /etc/rsyncd.secrets are mounted in the container. Using this approach I achieve full configurability of rsync without having to define custom scripts that use environment variables. For the samba container I decided to simply use a third-party container since samba can be a bit more difficult to setup and I did not want to spend to much time on that. Rsync however is usually very simple to setup.

  • # F: Exposing the rsync port
  • # G: Mounting the rsyncd secrets file.
  • # H: Mounting the rsyncd.conf file
  • # I: Mounting the data directory

An example rsyncd.secrets file is as follows and defines the username and password in plain text through a kubernetes Secret:


This is not very secure in general but since a Secret is used and the file is only accessible within the NAS container, it is still ok enough for home use (no, my password is not actually abc123).

An example rsyncd.conf file that refers to the secret is as follows:

pid file = /var/run/rsyncd.pid
log file = /dev/stdout
timeout = 300
max connections = 10
port = 873
    uid = root
    gid = root
    hosts deny = *
    hosts allow =
    read only = false
    path = /data/files
    comment = data directory
    auth users = erik
    strict modes = true
    secrets file = /etc/rsyncd.secrets

The above configuration allows user erik to access the data ‘share’ based on the above mentioned secrets file.

The SFTP container

  • # J:  A custom container is used for SSH based on the following Dockerfile:
    FROM rockylinux:9.0.20220720
    ENV TINI_VERSION v0.19.0
    #ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
    COPY tini /tini
    RUN chmod +x /tini
    ENTRYPOINT ["/tini", "--"]
    RUN yum makecache
    RUN yum install openssh-server passwd -y
    RUN ssh-keygen -A
    RUN rm -f /run/nologin
    COPY entrypoint.sh /
    CMD ["/entrypoint.sh", "/usr/sbin/sshd", "-De"]
    EXPOSE 22

    The above dockerfile is a bit more complex. It used tini for additional robustness when stopping the container. It is required for proper signal handling for this container when stopping the container. In this case I have downloaded a version of tini once before building instead of downloading it dynamically. This approach increases reproducibility of building the container. The ssh-keygen command is there to generate host keys required by sshd. The removal of /run/nologin is there to tell sshd that the ‘system’ has booted up and login is now allowed.  The entrypoint script initializes the container based on the /etc/sftpusers.conf file. The sshd -D option makes sure that sshd runs in the foreground and does not detach. The -e option will make sshd log to stdout.

  • # K: The container port 22 for ssh.
  • # L: The /etc/sftpusers.conf file that is used to configure users. This configures users, groups, authorized keys, and passwords for users in a simple to understand configuration file. The syntax of this file is a follows:
    authorized USERNAME PUBLICKEY1
    authorized USERNAME PUBLICKEY2

    This provides a compact definition of users. The entrypoint script uses this file to create the configured groups and users inside the container, adds authorized keys for the users (optionally) and defines the password (optionally). The entrypoint script is as follows:

    if [[ ! -r "/etc/sftpusers.conf" ]]
      echo "/etc/sftpusers.conf not found, exiting" 1>&2
      exit 1
    function group
      echo group "$@"
      groupadd -g "$2" "$1"  
    function user
      echo user "$@"
      useradd -u "$id" -g "$group" "$user"
      cat << EOF >> /etc/ssh/sshd_config
      Match User $user
          ChrootDirectory $dir
          ForceCommand internal-sftp
    function authorized 
      echo authorized "$@"
      mkdir -p /home/$user/.ssh
      echo "$cert" >> /home/$user/.ssh/authorized_keys
      chmod -R go-rwx /home/$user/.ssh
      chown -R "$user" /home/$user/.ssh
    function password 
      echo password
      echo "$password" | passwd --stdin "$user"
    . /etc/sftpusers.conf
    echo "ARGS: $@"
    exec "$@"

    As can be seen the script defines functions named group, user, authorized, and password, then it sources the sftpusers.conf file which simply invokes these functions. It also uses chroot for the configured users and forces sftp. Obviously, for a more serious setup there should be error handling in the script.

  • # M: The data directory.

The service

The services on the pod are exposed by a single load balancer service which exposes the samba ports (139 and 445), the rsync port (873), and the SFTP port (22).

apiVersion: v1
kind: Service
  name: nas
  namespace: example-com
    app: nas-server
  type: LoadBalancer
    - port: 139
      name: netbios-ssn
    - port: 445
      name: microsoft-ds
    - port: 873
      name: rsync
    - port: 22
      name: sftp

Network policies

There is one additional NetworkPolicy that allows access from only my local network. The network policy also allows egress since some of the services use reverse DNS lookups and these can cause delays when DNS is blocked:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
  name: nas-egress-ingress
  namespace: example-com
      app: nas-server
    - ports:
        - port: 139
        - port: 445
        - port: 873
        - port: 22
        - ipBlock:
    - ports:
        - port: 53
          protocol: TCP
        - port: 53
          protocol: UDP


Usage of samba on windows must be done by providing the full host and share name in the form \\host\share since discovery does not work with this setup.

Usage of rsync without an interactive password prompt can be done using either:

echo PASSWORD | rsync -avz --password-file - . USER@HOST::data/

or by defining the password in the RSYNC_PASSWORD environment variable.

Automounting of directories using sshfs can be done as follows. In auto.master:

/autofs /etc/auto.misc

And in auto.misc:

data          -fstype=fuse,nodev,nonempty,noatime,allow_other,max_read=65536,uid=500,gid=100 :sshfs#USER@HOST:/

The above automount does user id mapping to uid 500 and gid 100 which correspond to the user that will access the share. For more complex setups using custom ports and jump hosts etc, just configure the connection details in /root/.ssh/config and then refer to the host defined in the config file.

Final thoughts

This post provided a simple setup of NAS-like functionality on kubernetes. For optimal configurability I created custom rsync and sftp containers, to avoid the general pattern of having to define a lot of environment variables or command line arguments. I might decide later to do the same for the samba container if I need more flexibility.



This entry was posted in Devops/Linux. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *