Keep expanding a file server VMDK or move the data to an iSCSI LUN?

There is no particular size at which a VMDK stops making sense. If expanding it is supported by your setup and the current approach meets your needs, keeping it is probably the simpler choice.

A VMDK can stay within hypervisor-managed workflows, including VM snapshots and storage migration where supported. Moving to guest-connected iSCSI puts that storage connection inside the operating system, so check what you would lose or need to manage separately.

I would start with backup and recovery, not disk size. If your backup relies on VM snapshots, check whether it will protect a LUN connected directly inside the guest. An agent-based backup may handle that storage, but verify the configuration rather than assuming it will.

Also test the restore you actually need. A file restore and a full server recovery can have very different recovery times. If you already back up both the disk and its files, check whether both copies serve a purpose before changing anything.

Guest iSCSI is more appealing if you need the storage to move independently of the VM, perhaps to a physical server or a different hypervisor. That flexibility comes with guest-side setup and management, including multipathing if your SAN design supports it.

It is not automatically faster just because it is iSCSI. Both options are already using the SAN here, so performance depends on the storage, network and configuration.

Shared storage for a cluster is another possible reason to look beyond an ordinary VMDK. That said, shared VMDKs may also be available depending on the hypervisor configuration. A normal file server does not need guest iSCSI just because its disk is getting bigger.