In this post, we take a look at how to apply custom Nginx configuration directives when you’re using the NGINX Gateway Fabric.

What’s the NGINX Gateway Fabric?⌗

The NGINX Gateway Fabric is an implementation of the Kubernetes Gateway API.

What’s the Gateway API?⌗

The Gateway API is an evolution of the Ingress API; it aims to provide a flexible mechanism for managing north/south network traffic (that is, traffic entering or exiting your Kubernetes cluster), with additional work to support east/west traffic (traffic between pods in your cluster).

What’s this about custom configuration?⌗

I’ve deployed a local development cluster, and I wanted to be able to push images into an image registry hosted on the cluster. This requires (a) running a registry, which is easy, and (b) somehow exposing that registry outside the cluster, which is also easy unless you decide to make it more complex.

In this case, I decided that rather than running an Ingress provider I was going to start familiarizing myself with the Gateway API, so I deployed NGINX Gateway Fabric. My first attempt at pushing an image into the registry looked like this:

$ podman push --tls-verify=false example registry.apps.cluster1.house/example:latest
Getting image source signatures
Copying blob b9fe5313d237 done   |
Copying blob cc2447e1835a done   |
Copying blob cb8b0886acfb done   |
Copying blob c4219a5645ea [===>----------------------------------] 9.3MiB / 80.2MiB | 372.7 MiB/s
Copying blob c6e5c62d1726 done   |
Copying blob 9ee7eb11f876 done   |
Copying blob f064c46326cb done   |
Copying blob 9c45ffa2a02a done   |
Copying blob 9a6c9897f309 done   |
Copying blob 27a0dbb2828e done   |
Error: writing blob: uploading layer chunked: StatusCode: 413, <html>
<head><title>413 Request Entity Too Large<...

Nginx, by default, restricts the maximum size of a request body to 1m, which is to say, 1 megabyte. You can increase (or remove) this limit by setting the client_max_body_size parameter…but how do you do this in the context of a managed deployment like the NGINX Gateway Fabric?

Via the API?⌗

As of this writing, there is no mechanism to apply custom configuration options via the API (although there is ongoing work to provide this, see issue #1258).

What about dropping a config file into conf.d?⌗

My first thought was that I could mount a custom configuration file into /etc/nginx/conf.d, along the lines of:

...
containers:
  - name: nginx
    volumeMounts:
      - name: nginx-extra-conf
        mountPath: /etc/nginx/conf.d/client_max_body_size.conf
        subPath: client_max_body_size
...
volumes:
  - name: nginx-extra-conf
    configMap:
      name: nginx-extra-conf

…but this fails because the Nginx controller explicitly cleans out that directory on startup and is unhappy if it is unable to delete a file.

Replacing nginx.conf⌗

Right now, the solution is to replace /etc/nginx/nginx.conf. This is a relatively simple operation using kustomize to apply a patch to the deployment manifests.

Grab the original configuration⌗

First, we need to retrieve the original nginx.conf:

mkdir configs
podman run --rm --entrypoint cat \
  ghcr.io/nginxinc/nginx-gateway-fabric/nginx:1.0.0 /etc/nginx/nginx.conf > configs/nginx.conf

Modify configs/nginx.conf as necessary; in my case, I added the following line to the http section:

client_max_body_size 0;

Patch the deployment⌗

We can deploy the stock NGINX Gateway Fabric with a kustomization.yaml file like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  nginxGatewayVersion: v1.0.0

resources:
- https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.0.0/crds.yaml
- https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.0.0/nginx-gateway.yaml
- https://raw.githubusercontent.com/nginxinc/nginx-gateway-fabric/v1.0.0/deploy/manifests/service/nodeport.yaml

To patch the Deployment resource, we extend the kustomization.yaml with the following patch:

patches:
  - patch: |
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-gateway
        namespace: nginx-gateway
      spec:
        template:
          spec:
            containers:
              - name: nginx
                volumeMounts:
                  - mountPath: /etc/nginx/nginx.conf
                    name: nginx-conf-override
                    subPath: nginx.conf
            volumes:
              - name: nginx-conf-override
                configMap:
                  name: nginx-conf-override

And then we add a confdigMapGenerator to generate the nginx-conf-override ConfigMap:

configMapGenerator:
  - name: nginx-conf-override
    namespace: nginx-gateway
    options:
      disableNameSuffixHash: true
    files:
      - configs/nginx.conf

Now when we deploy from this directory…

kubectl apply -k . --server-side

…the deployment includes our patched nginx.conf and we are able to successfully push images into the cluster registry.

I’ve included the complete kustomization.yaml alongside this post.