28 Jan 2019. Hosting static sites with minio and nginx on k8s


Summary

I’ve traditionally hosted this site and other services on hardware set up at home, either in my living room or tucked away in a utility closet. This setup meant I’d simply store the files in /var/www/ and leave it at that. Recently, though, I’ve gotten deeper into homelabbing—I installed Talos OS on several mini-PCs and have been experimenting with them ever since. This inspired me to overhaul the way I host this and other websites.

To set up a MinIO bucket, start by installing the mc client and configuring it for your instance. Once it’s set up, you can create a bucket and enable download access with these commands:


> mc mb myminio/static
Bucket created successfully ‘myminio/static’.
> mc anonymous download myminio/static
Access permission for ‘myminio/static’ is set to ‘download’
        

Next, copy your website files into the static bucket. Keep in mind that this setup allows public downloads from your MinIO bucket, so be cautious about uploading any sensitive files.

Once the initial MinIO setup is complete, we can move on to Kubernetes. For most of my applications, I use app-template, a Helm chart that simplifies deploying any application—in this case, NGINX. To begin, create a HelmRelease with the following configuration:

                
---
# yaml-language-server: $schema=https://raw.githubusercontent.com/bjw-s/helm-charts/main/charts/other/app-template/schemas/helmrelease-helm-v2beta2.schema.json
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: &app nginx-static
    namespace: default
    annotations:
      configmap.reloader.stakater.com/reload: "nginx-configmap"
spec:
    interval: 15m
    chart:
    spec:
        # renovate: registryUrl=https://bjw-s.github.io/helm-charts
        chart: app-template
        version: 3.5.1
        sourceRef:
        kind: HelmRepository
        name: bjw-s
        namespace: flux-system
    values:
    controllers:
        nginx-static:
        strategy: Recreate
        containers:
            main:
            image:
                repository: ghcr.io/nginxinc/nginx-unprivileged
                tag: "1.27.2"
                pullPolicy: Always
            ports:
                - containerPort: 8080

    service:
        app:
        controller: nginx-static
        ports:
            http:
            port: 8080

    ingress:
        app:
        className: external
        annotations:
            external-dns.alpha.kubernetes.io/target: "external.${SECRET_DOMAIN}"
        hosts:
            - host: "www.${SECRET_DOMAIN}"
            paths:
                - path: /
                service:
                    identifier: app
                    port: http               
    persistence:
        config:
        type: configMap
        name: nginx-configmap # referenced ConfigMap name
        globalMounts:
            - path: /etc/nginx/nginx.conf
            subPath: nginx.conf  # Ensure this matches the key name in the ConfigMap
            readOnly: true

        var/log/nginx:
        type: emptyDir
        var/cache/nginx:
        type: emptyDir
        var/run:
        type: emptyDir                
                
            

If you have stakater/Reloader, each time you update this ConfigMap, your Helm chart will automatically reload. This setup allows you to modify your NGINX configuration through a simple Git pull request.

In the next chart, you'll see the nginx-configmap, which is where you'll define your NGINX configuration.

        
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configmap
  namespace: default
data:
  nginx.conf: |
    events {
        multi_accept       on;
        worker_connections 65535;
    }

    http {
        charset                utf-8;
        sendfile               on;
        tcp_nopush             on;
        tcp_nodelay            on;
        server_tokens          off;
        log_not_found          off;
        types_hash_max_size    2048;
        types_hash_bucket_size 64;
        client_max_body_size   16M;

        # MIME
        include                mime.types;
        default_type           application/octet-stream;

        # Logging
        access_log  /var/log/nginx/access.log combined buffer=512k flush=1m;
        error_log   /var/log/nginx/error.log warn;

        # Set the DNS resolver to the Kubernetes DNS service
        resolver kube-dns.kube-system.svc.cluster.local valid=30s;

        server {
            listen	8080;
            listen	[::]:8080;
            server_name www.${SECRET_DOMAIN} ${SECRET_DOMAIN};

            proxy_read_timeout 10;

            location / {
                # If the request is for "/", proxy to index.html in the S3 path
                if ($uri = "/") {
                    rewrite ^ /index.html break;
                }

                # Proxy all requests to S3
                proxy_set_header Host $http_host;
                # set this to your minio host
                proxy_pass http://minio.database.svc.cluster.local:9000/static/${SECRET_DOMAIN}$uri;
            }
      } 
    }
    

Each domain should be specified in its own server block. After applying these Helm releases, your website should be visible!