A do-it-yourself NAS on kubernetes

Posted on November 10, 2022 by erik

As one of the last steps in moving my VM based setup at home to kubernetes, it is now the turn to migrate my ‘NAS’. The term ‘NAS’ is a too much honor for it since it is basically just a single shared file system exposed over SSHFS, Samba, and rsync. The setup discussed below however still supports multiple shares. Basically what I need from my NAS setup is the following:

  • it must be accessible on Windows and linux
  • efficient backups of important files must be possible using rsync (such as my complete home directory on linux)

For accessibility on linux I use sshfs and for other OSes I use samba. For the backup functionality I use Rsync.

A single pod

For the kubernetes setup I will use the simplest setup I can find. This amounts to a single pod with three containers:

  • a samba container
  • an Rsync container
  • an SFTP container used for SSHFS

PlantUML Syntax:<br /> allow_mixing<br /> scale 1.0<br /> hide circle</p> <p>node “NAS” {<br /> component “samba” as samba<br /> component “rsync” as rsync<br /> component “sftp” as sftp<br /> }</p> <p>database “nas-data” as data1</p> <p>NAS -down-> data1<br />

I am using a Deployment with replica count of 1. In this case, using a Deployment is ok since the underlying storage supports concurrent access. The pod can use 1 or more data volumes.

The Deployment is defined as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nas
  namespace: example-com
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nas-server
  template:
    metadata:
      labels:
        app: nas-server
    spec:
      terminationGracePeriodSeconds: 0                  # A 
      containers:
        - name: nas
          # amd64 image
          env:
            - name: TZ
              value: Europe/Amsterdam
            - name: USERID
              value: "1016"
            - name: GROUPID
              value: "1016"
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nas-password
                  key: password
          image: dperson/samba:amd64@sha256:e1d2a7366690749a7be06f72bdbf6a5a7d15726fc84e4e4f41e967214516edfd
          args:
            - -g
            - mangled names = no
            - -g                                      # A 
            - catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6
            - -g
            - create mask = 0600
            - -g
            - force create mode = 0600
            - -g
            - create directory mask = 0700
            - -g
            - directory mask = 0700
            - -g
            - force directory mode = 0700
            - -u
            - erik;$(PASSWORD)
            - -s
            - data;/mount/files;no;no;no;erik         # B
          ports:
            - containerPort: 137                      # C 
              protocol: UDP
            - containerPort: 138
              protocol: UDP
            - containerPort: 139                      
            - containerPort: 445
          volumeMounts:
            - name: nas-data                          # D 
              mountPath: /mount
        - name: rsync
          image: repo.example.com/rsync:1.0           # E 
          ports:
            - containerPort: 873                      # F 
          volumeMounts:
            - name: rsyncd-secret                     # G
              mountPath: /etc/rsyncd.conf
              subPath: rsyncd.conf
            - name: rsyncd-secret                     # H 
              mountPath: /etc/rsyncd.secrets
              subPath: rsyncd.secrets
            - name: nas-data                          # I 
              mountPath: /data
        - name: sftp
          image: cat.wamblee.org/sftp:1.0             # J
          ports:
            - containerPort: 22                       # K 
          volumeMounts:
            - name: sftp-secret                       # L 
              mountPath: /etc/sftpusers.conf
              subPath: sftpusers.conf
            - name: nas-data                          # M 
              mountPath: /data
      volumes:
        - name: rsyncd-secret
          secret:
            secretName: rsyncd-secret
            defaultMode: 0600
        - name: sftp-secret
          secret:
            secretName: sftp-secret
        - name: nas-data  
          persistentVolumeClaim:
            claimName: nas-data

 

The samba container

For the samba container, I am using a third-party container:

  • # A: These mappings are used to map non-supported characters by samba (such as ‘:’) to characters that are supported. Without this, filenames containing these type of characters would be shown as mangled names.
  • # B: Configuration of the a single samba share. This option can be repeated to support multiple shares.
  • # C: The UDP ports are not used and can be removed from the Deployment. The UDP ports are required for automatic discovery of the samba shares, but these options do not work without host networking. I decided not to pursue this discovery for security reasons. Now, I simply need to configure the explicit path to the samba share.
  • # D: The container requires the data directory to be mounted at /mount.

The rsync container

  • # E: I am using a custom container built using a Dockerfile: 
    FROM rockylinux:9.0.20220720
RUN yum makecache
RUN yum install rsync -y
EXPOSE 873
CMD ["/usr/bin/rsync", "--no-detach", "--daemon", "--config", "/etc/rsyncd.conf"]

    This container simply runs rsync and exposes port 873 by default when run from docker for testing. The setup assumes that /etc/rsyncd.conf and /etc/rsyncd.secrets are mounted in the container. Using this approach I achieve full configurability of rsync without having to define custom scripts that use environment variables. For the samba container I decided to simply use a third-party container since samba can be a bit more difficult to setup and I did not want to spend to much time on that. Rsync however is usually very simple to setup.

  • # F: Exposing the rsync port
  • # G: Mounting the rsyncd secrets file.
  • # H: Mounting the rsyncd.conf file
  • # I: Mounting the data directory

An example rsyncd.secrets file is as follows and defines the username and password in plain text through a kubernetes Secret:

erik:abc123

This is not very secure in general but since a Secret is used and the file is only accessible within the NAS container, it is still ok enough for home use (no, my password is not actually abc123).

An example rsyncd.conf file that refers to the secret is as follows:

pid file = /var/run/rsyncd.pid
log file = /dev/stdout
timeout = 300
max connections = 10
port = 873
[data]
    uid = root
    gid = root
    hosts deny = *
    hosts allow = 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
    read only = false
    path = /data/files
    comment = data directory
    auth users = erik
    strict modes = true
    secrets file = /etc/rsyncd.secrets

The above configuration allows user erik to access the data ‘share’ based on the above mentioned secrets file.

The SFTP container

  • # J:  A custom container is used for SSH based on the following Dockerfile: 
    FROM rockylinux:9.0.20220720

ENV TINI_VERSION v0.19.0
#ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
COPY tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

RUN yum makecache
RUN yum install openssh-server passwd -y
RUN ssh-keygen -A
RUN rm -f /run/nologin
COPY entrypoint.sh /

CMD ["/entrypoint.sh", "/usr/sbin/sshd", "-De"]

EXPOSE 22

    The above dockerfile is a bit more complex. It used tini for additional robustness when stopping the container. It is required for proper signal handling for this container when stopping the container. In this case I have downloaded a version of tini once before building instead of downloading it dynamically. This approach increases reproducibility of building the container. The ssh-keygen command is there to generate host keys required by sshd. The removal of /run/nologin is there to tell sshd that the ‘system’ has booted up and login is now allowed.  The entrypoint script initializes the container based on the /etc/sftpusers.conf file. The sshd -D option makes sure that sshd runs in the foreground and does not detach. The -e option will make sshd log to stdout.

  • # K: The container port 22 for ssh.
  • # L: The /etc/sftpusers.conf file that is used to configure users. This configures users, groups, authorized keys, and passwords for users in a simple to understand configuration file. The syntax of this file is a follows: 
    group GROUPNAME GROUPID
user USERNAME USERID GROUPNAME DIRECTORY
authorized USERNAME PUBLICKEY1
authorized USERNAME PUBLICKEY2
password USERNAME PASSWORD

    This provides a compact definition of users. The entrypoint script uses this file to create the configured groups and users inside the container, adds authorized keys for the users (optionally) and defines the password (optionally). The entrypoint script is as follows:

    #!/bin/bash


if [[ ! -r "/etc/sftpusers.conf" ]]
then
  echo "/etc/sftpusers.conf not found, exiting" 1>&2
  exit 1
fi

function group
{
  echo group "$@"
  groupadd -g "$2" "$1"  
}

function user
{
  user="$1"
  id="$2"
  group="$3"
  dir="$4"

  echo user "$@"
  useradd -u "$id" -g "$group" "$user"
  
  cat << EOF >> /etc/ssh/sshd_config

  Match User $user
      ChrootDirectory $dir
      ForceCommand internal-sftp

EOF
}

function authorized 
{
  echo authorized "$@"
  user="$1"
  shift
  cert="$@"

  mkdir -p /home/$user/.ssh
  echo "$cert" >> /home/$user/.ssh/authorized_keys
  chmod -R go-rwx /home/$user/.ssh
  chown -R "$user" /home/$user/.ssh
}

function password 
{
  echo password
  user="$1"
  password="$2"
  echo "$password" | passwd --stdin "$user"
}


. /etc/sftpusers.conf

echo "ARGS: $@"
exec "$@"

    As can be seen the script defines functions named group, user, authorized, and password, then it sources the sftpusers.conf file which simply invokes these functions. It also uses chroot for the configured users and forces sftp. Obviously, for a more serious setup there should be error handling in the script.

  • # M: The data directory.

The service

The services on the pod are exposed by a single load balancer service which exposes the samba ports (139 and 445), the rsync port (873), and the SFTP port (22).

apiVersion: v1
kind: Service
metadata:
  name: nas
  namespace: example-com
spec:
  selector: 
    app: nas-server
  type: LoadBalancer
  loadBalancerIP: 192.168.1.245
  ports:
    - port: 139
      name: netbios-ssn
    - port: 445
      name: microsoft-ds
    - port: 873
      name: rsync
    - port: 22
      name: sftp

Network policies

There is one additional NetworkPolicy that allows access from only my local network. The network policy also allows egress since some of the services use reverse DNS lookups and these can cause delays when DNS is blocked:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: nas-egress-ingress
  namespace: example-com
spec:
  podSelector:
    matchLabels:
      app: nas-server
  ingress:
    - ports:
        - port: 139
        - port: 445
        - port: 873
        - port: 22
      from:
        - ipBlock:
            cidr: 192.168.1.0/24
  egress:
    - ports:
        - port: 53
          protocol: TCP
        - port: 53
          protocol: UDP

Usage

Usage of samba on windows must be done by providing the full host and share name in the form \\host\share since discovery does not work with this setup.

Usage of rsync without an interactive password prompt can be done using either:

echo PASSWORD | rsync -avz --password-file - . USER@HOST::data/

or by defining the password in the RSYNC_PASSWORD environment variable.

Automounting of directories using sshfs can be done as follows. In auto.master:

/autofs /etc/auto.misc

And in auto.misc:

data          -fstype=fuse,nodev,nonempty,noatime,allow_other,max_read=65536,uid=500,gid=100 :sshfs#USER@HOST:/

The above automount does user id mapping to uid 500 and gid 100 which correspond to the user that will access the share. For more complex setups using custom ports and jump hosts etc, just configure the connection details in /root/.ssh/config and then refer to the host defined in the config file.

Final thoughts

This post provided a simple setup of NAS-like functionality on kubernetes. For optimal configurability I created custom rsync and sftp containers, to avoid the general pattern of having to define a lot of environment variables or command line arguments. I might decide later to do the same for the samba container if I need more flexibility.

 

 

This entry was posted in Devops/Linux. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *