In windows you can disable inheritance on the specific file or folder, assign the proper user ownership and remove the unwanted groups/users.
Another option is to only create a directory specifically for them that they have access to or create them their own share with the proper permissions.
To my knowledge there isn’t any “locking” mechanism. I know that SMB works as intended when user A has excel document open it will notify user B if they are trying to open the same file. I think its up to the program in question if it is locking the file on the SMB share. Not TrueNAS locking the file.