I found a bug that might be related more to Gazebo rather than Sphnix, but:
starting sphinx through ssh -X always leads to a memory leak. The simulator slowly fills up the RAM till 100%, at which point the machine freezes.
I could get around this issue by using export DISPLAY=:0 which starts the simulator GUI on the remote machine.