Future on R

I recently learned that my simple parallelization code relying on the future and future.lapply packages in RStudio on OpenOnDemand in a Singularity image on our SLURM HPC cluster was not working unfortunately. Since then I’ve spent over a day and a half just trying to debug, as well as see if I could use future.batchtools. At first I thought ChatGPT was quite helpful, but ultimately it was noticing the wrong errors and definitely led me down a wild goose chase.

The problem

I had a rather simple line future_lapply(files, read.nextstrain.json), where read.nextstrain.json came from the treeio package. I thought it would just work, and it does on my own machine, but on the cluster it seemed to be taking interminably long. The first thing I noticed was that I hadn’t set plan(multisession), so I did that. However I still continued running into issues.

When I took a look, it seemed that doing one tree on a single node and a single CPU, it would use up all of the CPU (CPU utilization was 90+%) with a lower amount of memory (CPU memory usage was around 1GB/4), but when I tried to do it on a single node and 2 CPUs for a list of 2 trees under the idea that each tree would go to a different CPU, CPU utilization suddenly became 1% and memory usage also around 1%, which indicated the program wasn’t running at all.

Things I have tried

• Running a simple future_lapply(1:2, function(i) Sys.info()[["nodename"]]) to see if it was an RStudio issue. It worked, so that means in general I can run future_lapply and it’ll allocate to 2 different notes. I did indeed get the simple command to work, so it was not an RStudio issue.
• Using future.batchtools, on suggestion of ChatGPT. This seemed nice in theory but in practice led me down a very long rabbit chase because I was consistently getting the following error no matter what I did:

 > library("future.batchtools")
Loading required package: parallelly
Loading required package: future
> plan(batchtools_slurm)
> x %<-% { Sys.sleep(5); 3.14 }
Error: Failed to submit BatchtoolsSlurmFuture (<none>). The reason was: error in running command
TROUBLESHOOTING INFORMATION:
batchtools::submitJobs() was called with the following 'resources' argument:
 list()
 Error in system2(command = sys.cmd, args = sys.args, stdin = stdin, stdout = TRUE,  : 
  error in running command

Foolishly I kept asking ChatGPT about it without thinking critically about it, and it had me create something like 15 different files and directories in an attempt to troubleshoot including a custom makeClusterFunctionsSLURM.R from batchtools before I realized it was 1) basically making things up to take me in circles (as in the solutions were rehashes of before and led to the same errors) rather than be like no clue and 2) it was actually trying to address the part where nothing had been passed into resources, which after finally going to read the documentation in batchtools and future.batchtools, I realized it wasn’t even the error, you can pass in no arguments to resources and it’ll be fine. Anyway, I’ve made an issue and given up on that path for now. ChatGPT did suggest that I had an I/O issue though, which I think might be true.

• Running the same simple command but as an Rscript command from an sbatch script. If it doesn’t work, would indicate it is an issue with R. Here, it did run but all seemed to be on one core rather than 2…

Loading required package: parallelly
[1] "PID = 1238017, Host = node2333.oscar.ccv.brown.edu, Affinity = pid 1238157's current affinity list: 15,40"
[1] "PID = 1238017, Host = node2333.oscar.ccv.brown.edu, Affinity = pid 1245859's current affinity list: 15,40"
Execution halted
(base) [aguang@login009 scripts]$ jobstats 11138495

================================================================================
                              Slurm Job Statistics
================================================================================
         Job ID: 11138495
  NetID/Account: aguang/default
       Job Name: test2.sh
          State: FAILED
          Nodes: 1
      CPU Cores: 2
     CPU Memory: 16GB (8GB per CPU-core)
  QOS/Partition: normal/batch
        Cluster: oscar
     Start Time: Wed Apr 23, 2025 at 4:34 PM
       Run Time: 00:09:16
     Time Limit: 2-00:00:00

                              Overall Utilization
================================================================================
  CPU utilization  [|||||||||||||||||||||||                        47%]
  CPU memory usage [|                                               2%]

                              Detailed Utilization
================================================================================
  CPU utilization per node (CPU time used/run time)
      node2333: 00:08:47/00:18:32 (efficiency=47.4%)

  CPU memory usage per node - used/allocated
      node2333: 390.3MB/16.0GB (195.1MB/8.0GB per core of 2)

I have to admit that I don’t have the expertise to understand what is going on from here. This does suggest to me though that I have been using future (and perhaps job allocations?) wrong this whole time on Oscar and don’t actually understand how to use it correctly, and it has never really sped up my computations like I thought it would. The next time I run code that is supposed to be parallelized in R but is less I/O based, I’ll see if I can do some tests again to see what the resource allocation used actually looks like.

The workaround

Ultimately I made a slurm array batch script to do what I wanted to do. It basically has #SBATCH --array=0-n where n is the number of trees I am trying to read in. It then has an array of tree filepaths trees, and uses the $SLURM_ARRAY_TASK_ID to access the filepath and run a separate job to read in each tree using a very simple R script that just runs read.nextstrain.json and then write_rds. Full script template:

#!/bin/bash
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --array=0-n
#SBATCH --mem=4G
#SBATCH -t 8:00:00
WORKDIR={insert working directory}
export SINGULARITY_BINDPATH=$WORKDIR
SINGULARITY_IMG={path to singularity image}
trees=(
    tree1
    tree2
    ...
    treen
)
tree=${trees[$SLURM_ARRAY_TASK_ID]}
singularity exec ${SINGULARITY_IMG} Rscript ${WORKDIR}/scripts/readtrees.R $tree

It turns out to be way faster than anything I am doing in RStudio anyway, including even reading a single tree. I guess the overhead for RStudio really slows things down.