Computational chemistry, Locally - by CasualPhysicsEnjoyer

Casual Physics Enjoyer

SubscribeSign in

Computational chemistry, Locally?<br>My rant on ssh<br>CasualPhysicsEnjoyer<br>Jul 21, 2026

Share

One of my projects is predicting the spectra of proteins. To do this, I design molecules, then get a computer to optimise their geometries and find the energy levels. Most do this kind of work on a University high performance computing cluster.<br>But here's something I noticed when doing this many times. When I need to remote into a cluster, the first part is to log in. I need to first login to the University VPN, which means using some outdated software to get the VPN set up from my computer. And then there is the authentication part, where the VPN software makes you jump through multiple authentication and 2FA processes to log in. And then when the 2FA window finally comes, I need to dig out my phone from somewhere in my room to get the code. Probably on average I find that the VPN + ssh + 2FA process takes 30 seconds every time I need to do something, which generally feels like too much.<br>It's not too much work overall, but there are a lot of intermediate steps and riff-raff.<br>I call this clunkiness. Something is clunky when there are many intermediate steps required to do one task. You know those really annoying shower controls in fancy hotels when you need to adjust two separate knobs for the pressure and temperature separately? That's clunky.<br>It’s clunky to do work in University compute clusters.<br>A bit of clunkiness in using something is usually fine. But if the clunkiness is even slightly more than a bit, the effects can be disastrous. If there are too many intermediate steps to get started to work on something, I become less motivated to work on it, forget about it, and in the worst case scenario just drop the endeavour entirely. Which means that an entire stream of work can be lost for small and trivial reasons. This happens with a lot of things! For example, it's really clunky to descale my coffee machine, and so I never end up doing it, and so it always breaks.<br>The friction in using these clusters has been enough to actually stop me from doing work that I would've done otherwise. If I was a full-time PhD student I would probably muscle through each awkward step to login. But I'm not, I work on this independently and am short of time. I am also mentally tired since I do this after work. And these things make it such that access to the compute needs to be instant and no-faff for me to justify getting started. This is something independent scientists try to optimise, which I wrote about in one of my older posts.<br>In terms of background, many individuals did research whilst they were in university, but had to give it up as they started their jobs. There were also individuals who worked as engineers in top tech companies, science adjacent companies, or start-up founders, who already had high level experience in project management and research. Some had already submitted papers! Most tried to do research in their free time out of pure passion, and chose day jobs in industry to support financial and lifestyle goals.

Source: What I Learned from Speaking to 500 Independent Scientists<br>There is also a layer of friction between doing the computations and then analysing the logs that comp chemistry programs spit out. If you want to visualise orbitals or look at the outputs of your simulations, it's really awkward to run Python whilst being in the cluster, and the cluster has no GUI display. The way I used to do this was by doing SCP and copying the log files back and forth. Efficient I know... I used to use MacFUSE with sshfs but it turned out to be really unreliable as well, and quite slow. I wrote about the procedure a few years ago.<br>If you're like me and a beginner on working on remote HPC clusters with Slurm for computational physics / chemistry, one thing that I naively used to do was<br>- write python scripts, input files and logic on my local computer, using IDEs and tools on my laptop, and then<br>- ssh / rsync them to the remote cluster, and then<br>- do the computation, and then<br>- ssh or rsync them back.<br>But this is kind of annoying for small projects when you just start out. Especially when you end up writing a bunch of boiler plate code to sync files back and forth - leading to more and more lines of code. And guess what - I spent too much time on that code!

*So instead, I realised that I can use macFUSE with sshfs to mount the remote directory via to my local, so that I can just treat it like a directory as any other . There are some fiddly bits like changing your security on permissions on mac M1, but I found the tutorial pretty easy to follow.*

Source: MacOS sshfs<br>Most computational chemistry programs output a ridiculous text log file without a proper API. Something that would save the death by a thousand cuts nature of this problem is a python wrapper or parser.

Source: Tools to parse log files easier<br>And then...