<\/p>\n
For accessibility on linux I use sshfs and for other OSes I use samba. For the backup functionality I use Rsync.<\/p>\n
A single pod<\/h2>\n
For the kubernetes setup I will use the simplest setup I can find. This amounts to a single pod with three containers:<\/p>\n
- \n
- a samba container<\/li>\n
- an Rsync container<\/li>\n
- an SFTP container used for SSHFS<\/li>\n<\/ul>\n
\nallow_mixing
\nscale 1.0
\nhide circle<\/p>\nnode “NAS” {
\ncomponent “samba” as samba
\ncomponent “rsync” as rsync
\ncomponent “sftp” as sftp
\n}<\/p>\ndatabase “nas-data” as data1<\/p>\n
NAS -down-> data1
\n\" usemap=\"#plantuml_map\"><\/p>\nI am using a Deployment<\/em> with replica count of 1. In this case, using a Deployment<\/em> is ok since the underlying storage supports concurrent access. The pod can use 1 or more data volumes.<\/p>\n
The Deployment<\/em> is defined as follows:<\/p>\n
apiVersion: apps\/v1\r\nkind: Deployment\r\nmetadata:\r\n name: nas\r\n namespace: example-com\r\nspec:\r\n replicas: 1\r\n selector:\r\n matchLabels:\r\n app: nas-server\r\n template:\r\n metadata:\r\n labels:\r\n app: nas-server\r\n spec:\r\n terminationGracePeriodSeconds: 0 # A \r\n containers:\r\n - name: nas\r\n # amd64 image\r\n env:\r\n - name: TZ\r\n value: Europe\/Amsterdam\r\n - name: USERID\r\n value: \"1016\"\r\n - name: GROUPID\r\n value: \"1016\"\r\n - name: PASSWORD\r\n valueFrom:\r\n secretKeyRef:\r\n name: nas-password\r\n key: password\r\n image: dperson\/samba:amd64@sha256:e1d2a7366690749a7be06f72bdbf6a5a7d15726fc84e4e4f41e967214516edfd\r\n args:\r\n - -g\r\n - mangled names = no\r\n - -g # A \r\n - catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6\r\n - -g\r\n - create mask = 0600\r\n - -g\r\n - force create mode = 0600\r\n - -g\r\n - create directory mask = 0700\r\n - -g\r\n - directory mask = 0700\r\n - -g\r\n - force directory mode = 0700\r\n - -u\r\n - erik;$(PASSWORD)\r\n - -s\r\n - data;\/mount\/files;no;no;no;erik # B\r\n ports:\r\n - containerPort: 137 # C \r\n protocol: UDP\r\n - containerPort: 138\r\n protocol: UDP\r\n - containerPort: 139 \r\n - containerPort: 445\r\n volumeMounts:\r\n - name: nas-data # D \r\n mountPath: \/mount\r\n - name: rsync\r\n image: repo.example.com\/rsync:1.0 # E \r\n ports:\r\n - containerPort: 873 # F \r\n volumeMounts:\r\n - name: rsyncd-secret # G\r\n mountPath: \/etc\/rsyncd.conf\r\n subPath: rsyncd.conf\r\n - name: rsyncd-secret # H \r\n mountPath: \/etc\/rsyncd.secrets\r\n subPath: rsyncd.secrets\r\n - name: nas-data # I \r\n mountPath: \/data\r\n - name: sftp\r\n image: cat.wamblee.org\/sftp:1.0 # J\r\n ports:\r\n - containerPort: 22 # K \r\n volumeMounts:\r\n - name: sftp-secret # L \r\n mountPath: \/etc\/sftpusers.conf\r\n subPath: sftpusers.conf\r\n - name: nas-data # M \r\n mountPath: \/data\r\n volumes:\r\n - name: rsyncd-secret\r\n secret:\r\n secretName: rsyncd-secret\r\n defaultMode: 0600\r\n - name: sftp-secret\r\n secret:\r\n secretName: sftp-secret\r\n - name: nas-data \r\n persistentVolumeClaim:\r\n claimName: nas-data\r\n<\/pre>\n
<\/p>\n
The samba container<\/h2>\n
For the samba container, I am using a third-party container<\/a>:<\/p>\n
- \n
- # A<\/em>: These mappings are used to map non-supported characters by samba (such as ‘:’) to characters that are supported. Without this, filenames containing these type of characters would be shown as mangled names.<\/li>\n
- # B<\/em>: Configuration of the a single samba share. This option can be repeated to support multiple shares.<\/li>\n
- # C<\/em>: The UDP ports are not used and can be removed from the Deployment. <\/em>The UDP ports are required for automatic discovery of the samba shares, but these options do not work without host networking. I decided not to pursue this discovery for security reasons. Now, I simply need to configure the explicit path to the samba share.<\/li>\n
- # D<\/em>: The container requires the data directory to be mounted at \/mount<\/em>.<\/li>\n<\/ul>\n
The rsync container<\/h2>\n
- \n
- # E<\/em>: I am using a custom container built using a Dockerfile:<\/em>\n
FROM rockylinux:9.0.20220720\r\nRUN yum makecache\r\nRUN yum install rsync -y\r\nEXPOSE 873\r\nCMD [\"\/usr\/bin\/rsync\", \"--no-detach\", \"--daemon\", \"--config\", \"\/etc\/rsyncd.conf\"]\r\n<\/pre>\n
This container simply runs rsync<\/em> and exposes port 873 by default when run from docker for testing. The setup assumes that \/etc\/rsyncd.conf<\/em> and \/etc\/rsyncd.secrets<\/em> are mounted in the container. Using this approach I achieve full configurability of rsync<\/em> without having to define custom scripts that use environment variables. For the samba container I decided to simply use a third-party container since samba can be a bit more difficult to setup and I did not want to spend to much time on that. Rsync however is usually very simple to setup.<\/li>\n
- # F<\/em>: Exposing the rsync port<\/li>\n
- # G<\/em>: Mounting the rsyncd secrets<\/a><\/em> file.<\/li>\n
- # H<\/em>: Mounting the rsyncd.conf<\/em><\/a> file<\/li>\n
- # I<\/em>: Mounting the data directory<\/li>\n<\/ul>\n
An example rsyncd.secrets<\/em> file is as follows and defines the username and password in plain text through a kubernetes Secret<\/em>:<\/p>\n
erik:abc123\r\n<\/pre>\n
This is not very secure in general but since a Secret is used and the file is only accessible within the NAS container, it is still ok enough for home use (no, my password is not actually abc123).<\/p>\n
An example rsyncd.conf<\/em> file that refers to the secret is as follows:<\/p>\n
pid file = \/var\/run\/rsyncd.pid\r\nlog file = \/dev\/stdout\r\ntimeout = 300\r\nmax connections = 10\r\nport = 873\r\n[data]\r\n uid = root\r\n gid = root\r\n hosts deny = *\r\n hosts allow = 192.168.0.0\/16 172.16.0.0\/12 10.0.0.0\/8\r\n read only = false\r\n path = \/data\/files\r\n comment = data directory\r\n auth users = erik\r\n strict modes = true\r\n secrets file = \/etc\/rsyncd.secrets\r\n<\/pre>\n
The above configuration allows user erik<\/em> to access the data ‘share’ based on the above mentioned secrets file.<\/p>\n
The SFTP container<\/h2>\n
- \n
- # J<\/em>:\u00a0 A custom container is used for SSH based on the following Dockerfile:\n
FROM rockylinux:9.0.20220720\r\n\r\nENV TINI_VERSION v0.19.0\r\n#ADD https:\/\/github.com\/krallin\/tini\/releases\/download\/${TINI_VERSION}\/tini \/tini\r\nCOPY tini \/tini\r\nRUN chmod +x \/tini\r\nENTRYPOINT [\"\/tini\", \"--\"]\r\n\r\nRUN yum makecache\r\nRUN yum install openssh-server passwd -y\r\nRUN ssh-keygen -A\r\nRUN rm -f \/run\/nologin\r\nCOPY entrypoint.sh \/\r\n\r\nCMD [\"\/entrypoint.sh\", \"\/usr\/sbin\/sshd\", \"-De\"]\r\n\r\nEXPOSE 22\r\n<\/pre>\nThe above dockerfile is a bit more complex. It used tini<\/a><\/em> for additional robustness when stopping the container. It is required for proper signal handling for this container when stopping the container. In this case I have downloaded a version of tini<\/em> once before building instead of downloading it dynamically. This approach increases reproducibility of building the container. The ssh-keygen<\/em> command is there to generate host keys required by sshd<\/em>. The removal of \/run\/nologin<\/em> is there to tell sshd<\/em> that the ‘system’ has booted up and login is now allowed.\u00a0 The entrypoint script initializes the container based on the \/etc\/sftpusers.conf<\/em> file. The sshd -D<\/em> option makes sure that sshd runs in the foreground and does not detach. The -e<\/em> option will make sshd log to stdout.<\/li>\n
- # K<\/em>: The container port 22 for ssh<\/em>.<\/li>\n
- # L<\/em>: The \/etc\/sftpusers.conf<\/em> file that is used to configure users. This configures users, groups, authorized keys, and passwords for users in a simple to understand configuration file. The syntax of this file is a follows:\n
group GROUPNAME GROUPID\r\nuser USERNAME USERID GROUPNAME DIRECTORY\r\nauthorized USERNAME PUBLICKEY1\r\nauthorized USERNAME PUBLICKEY2\r\npassword USERNAME PASSWORD\r\n<\/pre>\n
This provides a compact definition of users. The entrypoint script uses this file to create the configured groups and users inside the container, adds authorized keys for the users (optionally) and defines the password (optionally). The entrypoint script is as follows:<\/p>\n
#!\/bin\/bash\r\n\r\n\r\nif [[ ! -r \"\/etc\/sftpusers.conf\" ]]\r\nthen\r\n echo \"\/etc\/sftpusers.conf not found, exiting\" 1>&2\r\n exit 1\r\nfi\r\n\r\nfunction group\r\n{\r\n echo group \"$@\"\r\n groupadd -g \"$2\" \"$1\" \r\n}\r\n\r\nfunction user\r\n{\r\n user=\"$1\"\r\n id=\"$2\"\r\n group=\"$3\"\r\n dir=\"$4\"\r\n\r\n echo user \"$@\"\r\n useradd -u \"$id\" -g \"$group\" \"$user\"\r\n \r\n cat << EOF >> \/etc\/ssh\/sshd_config\r\n\r\n Match User $user\r\n ChrootDirectory $dir\r\n ForceCommand internal-sftp\r\n\r\nEOF\r\n}\r\n\r\nfunction authorized \r\n{\r\n echo authorized \"$@\"\r\n user=\"$1\"\r\n shift\r\n cert=\"$@\"\r\n\r\n mkdir -p \/home\/$user\/.ssh\r\n echo \"$cert\" >> \/home\/$user\/.ssh\/authorized_keys\r\n chmod -R go-rwx \/home\/$user\/.ssh\r\n chown -R \"$user\" \/home\/$user\/.ssh\r\n}\r\n\r\nfunction password \r\n{\r\n echo password\r\n user=\"$1\"\r\n password=\"$2\"\r\n echo \"$password\" | passwd --stdin \"$user\"\r\n}\r\n\r\n\r\n. \/etc\/sftpusers.conf\r\n\r\necho \"ARGS: $@\"\r\nexec \"$@\"\r\n<\/pre>\nAs can be seen the script defines functions named group<\/em>, user<\/em>, authorized<\/em>, and password<\/em>, then it sources the sftpusers.conf<\/em> file which simply invokes these functions. It also uses chroot<\/em> for the configured users and forces sftp<\/em>. Obviously, for a more serious setup there should be error handling in the script.<\/li>\n
- # M<\/em>: The data directory.<\/li>\n<\/ul>\n
The service<\/h2>\n
The services on the pod are exposed by a single load balancer service which exposes the samba ports (139 and 445), the rsync port (873), and the SFTP port (22).<\/p>\n
apiVersion: v1\r\nkind: Service\r\nmetadata:\r\n name: nas\r\n namespace: example-com\r\nspec:\r\n selector: \r\n app: nas-server\r\n type: LoadBalancer\r\n loadBalancerIP: 192.168.1.245\r\n ports:\r\n - port: 139\r\n name: netbios-ssn\r\n - port: 445\r\n name: microsoft-ds\r\n - port: 873\r\n name: rsync\r\n - port: 22\r\n name: sftp\r\n<\/pre>\n
Network policies<\/h2>\n
There is one additional NetworkPolicy<\/em> that allows access from only my local network. The network policy also allows egress since some of the services use reverse DNS lookups and these can cause delays when DNS is blocked:<\/p>\n
kind: NetworkPolicy\r\napiVersion: networking.k8s.io\/v1\r\nmetadata:\r\n name: nas-egress-ingress\r\n namespace: example-com\r\nspec:\r\n podSelector:\r\n matchLabels:\r\n app: nas-server\r\n ingress:\r\n - ports:\r\n - port: 139\r\n - port: 445\r\n - port: 873\r\n - port: 22\r\n from:\r\n - ipBlock:\r\n cidr: 192.168.1.0\/24\r\n egress:\r\n - ports:\r\n - port: 53\r\n protocol: TCP\r\n - port: 53\r\n protocol: UDP\r\n\r\n<\/pre>\n
Usage<\/h2>\n
Usage of samba on windows must be done by providing the full host and share name in the form \\\\host\\share<\/em> since discovery does not work with this setup.<\/p>\n
Usage of rsync<\/em> without an interactive password prompt can be done using either:<\/p>\n
echo PASSWORD | rsync -avz --password-file - . USER@HOST::data\/\r\n<\/pre>\n
or by defining the password in the RSYNC_PASSWORD<\/em> environment variable.<\/p>\n
Automounting of directories using sshfs<\/em> can be done as follows. In auto.master:<\/p>\n
\/autofs \/etc\/auto.misc\r\n<\/pre>\n
And in auto.misc:<\/p>\n
data -fstype=fuse,nodev,nonempty,noatime,allow_other,max_read=65536,uid=500,gid=100 :sshfs#USER@HOST:\/\r\n<\/pre>\n
The above automount does user id mapping to uid 500 and gid 100 which correspond to the user that will access the share. For more complex setups using custom ports and jump hosts etc, just configure the connection details in \/root\/.ssh\/config<\/em> and then refer to the host defined in the config file.<\/p>\n
Final thoughts<\/h2>\n
This post provided a simple setup of NAS-like functionality on kubernetes. For optimal configurability I created custom rsync and sftp containers, to avoid the general pattern of having to define a lot of environment variables or command line arguments. I might decide later to do the same for the samba container if I need more flexibility.<\/p>\n
<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
As one of the last steps in moving my VM based setup at home to kubernetes, it is now the turn to migrate my ‘NAS’. The term ‘NAS’ is a too much honor for it since it is basically just … Continue reading
- # K<\/em>: The container port 22 for ssh<\/em>.<\/li>\n
- # F<\/em>: Exposing the rsync port<\/li>\n
- # B<\/em>: Configuration of the a single samba share. This option can be repeated to support multiple shares.<\/li>\n
- # A<\/em>: These mappings are used to map non-supported characters by samba (such as ‘:’) to characters that are supported. Without this, filenames containing these type of characters would be shown as mangled names.<\/li>\n