Here I’ll only show you how to set user quotas. You can easily also add group quotas by
adding e.g. grpquota to the fileSystem.options list and then configure the group
quotas.

Setting up user quotas first is easier (baby steps).

First you need to add pkgs.linuxquotas package to your environment.systemPackage
and add options = [ "defaults" "usrquota" "jqfmt=vfsv0" ];
to the fileSystem option.

E.g.

  fileSystems."/" = {
    device = "/dev/sda";
    fsType = "ext4";
    options = [ "defaults" "usrquota" "jqfmt=vfsv0" ];
  };

Rebuild NixOS (nixos-rebuild switch).

Run quotacheck -aumv to verify everything got properly installed and configured. You
should get the following output:

 $ quotacheck -aumv
quotacheck: Scanning /dev/sda [/] done
quotacheck: Cannot stat old user quota file //aquota.user: No such file or directory. Usage will not be subtracted.
quotacheck: Cannot stat old group quota file //aquota.group: No such file or directory. Usage will not be subtracted.
quotacheck: Checked 114196 directories and 380989 files
quotacheck: Old file not found.

With this command we have also created /aquota.user binary file where quotas are defined.

Your filesystem is now set up to use disk quotas. To turn them on, use the quotaon
command:

 $ quotaon -auv 
/dev/sda [/]: user quotas turned on

Next, we need to configure the limits for our users. Example:

$ edquota -u foo_com
...
Disk quotas for user foo_com (uid 1000):
  Filesystem                   blocks       soft       hard     inodes     soft     hard
  /dev/sda                         72       1000       2000         18        0        0

Now let’s log in as foo_com user and test our limits:

$ quota -s
Disk quotas for user foo_com (uid 1000): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
       /dev/sda    196K   1000K   2000K              20       0       0   

$ dd if=/dev/zero of=upload_test bs=1M count=1     
$ quota -s
Disk quotas for user foo_com (uid 1000): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
       /dev/sda   1200K*  1000K   2000K   7days      16       0       0        

With this, we have reached the soft quota but we can still write until we reach the hard
quota or until the end of the grace period (which we can change with edquota -u foo_com -T).

Now let’s try to reach the hard quota and see what happens:

$ dd if=/dev/zero of=upload_test bs=2M count=1
dd: error writing 'upload_test': Disk quota exceeded
1+0 records in
0+0 records out
1867776 bytes (1.9 MB, 1.8 MiB) copied, 0.00201586 s, 927 MB/s

$ ls -lah
total 2.0M
drwxr-x--- 4 foo_com wwwrun 4.0K Feb 11 12:47 .
drwxr-x--- 3 root    wwwrun 4.0K Feb 11 09:35 ..
drwxr-xr-x 3 foo_com wwwrun 4.0K Feb 11 12:14 .cache
drwxr-xr-x 3 foo_com wwwrun 4.0K Feb 11 12:15 .config
-rw-r--r-- 1 foo_com wwwrun 1.8M Feb 11 12:48 upload_test
...

We get Disk quota exceeded error when trying to write to the disk. dd managed to
write only 1.8M of (empty) data to upload_test before the quota throws the error.

Ref:

• Implementing quotas to restrict disk space usage
• nix-quota-module