There is an issue with CUBA Studio *.vmoptions file which is used to change memory settings.
To make the Memory Settings dialog work you need to create empty file named cuba-studio64.exe.vmoptions in the CUBA Studio configuration folder (C:\Users\user\.CubaStudio2019.2\config on Windows). Or rename the studio64.exe.vmoptions file if exists.
Then restart CUBA Studio, go to Help -> Change Memory Settings and set desired Maximum Heap Size value.