R_PARALLELLY_RANDOM_PORTS now
supports multiple, comma-separated port specifications, e.g.
"20001:20999" and
"1068:1099,20001:20999,40530".help("makeClusterPSOCK") on how to use
systemd-run to limit workers’ CPU quota and memory
allowances.availableCores() does a better job detecting
cgroups v2 cpu.max CPU restrictions.Now argument rshcmd of makeNodePSOCK()
can be a function. It must accept at least two arguments named
rshopts and worker. The rshopts
argument is a character vector of length zero or more. The
worker argument is a string hostname. The function must
return a single string.
Now makeNodePSOCK() accepts
rscript_sh = "none", which skips quoting the Rscript
call.
Now makeNodePSOCK() accepts rscript_sh
of length one or two. If length(rscript_sh) == 2, then
rscript_sh[1] is for the inner and
rscript_sh[2] is for the outer shell quoting of the Rscript
call. More precisely, rscript_sh[1] is for Rscript
arguments that need shell quoting
(e.g. Rscript -e "<expr>"), and
rscript_sh[2] is for the whole Rscript ...
call.
Add makeClusterSequential() available for R (>=
4.4.0).
Starting with R 4.5.0 (currently R-devel), one can use
parallel::makeCluster(n, type = parallelly::PSOCK) as an
alternative to parallelly::makeClusterPSOCK(n). Similarly,
type = parallelly::MPI creates a cluster using
parallelly::makeClusterMPI(), and
type = parallelly::SEQ creates a cluster using
parallelly::makeClusterSequential().
Add serializedSize() for calculating the size of an
object by counting the number of bytes required to serialize
it.
R_PARALLELLY_MAXWORKERS_LOCALHOST
was interpreted as integers rather than doubles.makeClusterPSOCK(nworkers) gained protection against
setting up too many localhost workers relative to number of available
CPU cores. If nworkers / availableCores() is greater than
1.0 (100%), then a warning is produced. If greater than 3.0 (300%), an
error is produced. These limits can be configured by R option
parallelly.maxWorkers.localhost. These checks are skipped
if nworkers inherits from AsIs,
e.g. makeClusterPSOCK(I(16)). The current 3.0 (300%) limit
is likely to be decreased in a future release. A few packages fail
R CMD check --as-cran with this validation enabled. For
example, one package uses 8 parallel workers in its examples, while
R CMD check --as-cran only allows for two. To give such
packages time to be fixed, the CRAN-enforced limits are ignored for
now.makeClusterPSOCK() could produce a confusing error
Invalid port: NA if a non-available port was requested.
Now the error message is more informative, e.g.
Argument 'port' specifies non-available port(s): 80.isNodeAlive() and killNode() now
support also worker processes that run on remote machines. They do this
by connecting to the remote machine using the same method used to launch
the worker, which is typically SSH, and do their R calls that
way.
isNodeAlive() and killNode() gained
argument timeout for controlling the maximum time, in
seconds, before giving up and returning NA.
Add cloneNode(), which can be used to “restart”
RichSOCKnode cluster nodes.
Argument worker for makeNodePSOCK() now
takes the optional, logical attribute localhost to manually
specify that the worker is a localhost worker.
Add print() for RichSOCKnode, which
gives more details than print() for
SOCKnode.
print() for RichSOCKnode and
RichSOCKcluster report on nodes with broken
connections.
Add as.cluster() for RichSOCKnode,
which returns a RichSOCKcluster.
Introduce R option
parallelly.supportsMulticore.disableOn to control where
multicore processing is disabled by default.
Calling killNode() on RichSOCKnode node
could theoretically kill a process on the current machine with the same
process ID (PID), although the parallel worker (node) is running on
another machine.
isNodeAlive() on RichSOCKnode node
could theoretically return TRUE because there was a process with the
same process ID (PID) on the current machine, although the parallel
worker (node) is running on another machine.
isLocalHost() for SOCK0node was not
declared an S3 method.
freePort() defaults to
default = NA_integer_, so that NA_integer_ is
returned when no free port could be found. However, in R (< 4.0.0),
which does not support port querying, we use
default = "random".help("makeClusterPSOCK") that
rscript_sh = "cmd" is needed if the remote machines run MS
Windows.makeClusterPSOCK(..., verbose = TRUE) would not show
verbose output. One still had to set option
parallelly.debug to TRUE.
availableWorkers() could produce false sanity-check
warnings on mismatching ‘PE_HOSTFILE’ content and ‘NSLOTS’ for certain
SGE-cluster configurations.
availableWorkers(constraints = "connections"), which limits
the number of workers that can be be used to the current number of free
R connections according to freeConnections(). This is the
maximum number of PSOCK, SOCK, and MPI parallel cluster
nodes we can open without running out of available R connections.availableCores() would produce a warning
In is.na(constraints) : is.na() applied to non-(list or vector) of type 'NULL'
when running with R (< 4.0.0).
availableWorkers() did not acknowledge the
"cgroups2.cpu.max" and "Bioconductor" methods
added to availableCores() in parallelly
1.33.0 (2022-12-13). It also did not acknowledge methods
"cgroups.cpuset" and "cgroups.cpuquota" added
in parallelly 1.31.0 (2022-04-07), and
"nproc" added in parallelly 1.26.1
(2021-06-29).
When makeClusterPSOCK() failed to connect to all
parallel workers within the connectTimeout time limit,
could either produce
Error in sprintf(ngettext(failed, "Cluster setup failed (connectTimeout=%.1f seconds). %d worker of %d failed to connect.", : invalid format '%d'; use format %f, %e, %g or %a for numeric objects
instead of an informative error message, or an error message with the
incorrect information.
Add killNode() to terminate cluster nodes via
process signaling. Currently, this is only supported for parallel
workers on the local machine, and only those created by
makeClusterPSOCK().
makeClusterPSOCK() and likes now assert the running
R session has enough permissions on the operating system to do system
calls such as system2("Rscript --version"). If not, an
informative error message is produced.
On Unix, availableCores() queries also control
groups v2 (cgroups v2) field cpu.max for a possible CPU
quota allocation. If a CPU quota is set, then the number of CPUs is
rounded to the nearest integer, unless its less that 0.5, in case it’s
rounded up to a single CPU. An example, where cgroups CPU quotas can be
set to limit the total CPU load, is with Linux containers,
e.g. docker run --cpus=3.5 ....
Add support for
availableCores(methods = "connections"), which returns the
current number of free R connections per freeConnections().
This is the maximum number of PSOCK, SOCK, and MPI
parallel cluster nodes we can open without running out
of available R connections. A convenient way to use this and all other
methods is
availableCores(constraints = "connections").
Now availableCores() recognizes environment variable
IS_BIOC_BUILD_MACHINE, which is set to true by the
Bioconductor (>= 3.16) check servers. If true, then a maximum of four
(4) cores is returned. This new environment variable replaces legacy
variable BBS_HOME used in Bioconductor (<=
3.15).
availableCores() splits up method
"BiocParallel" into two; "BiocParallel" and
"Bioconductor". The former queries environment variable
BIOCPARALLEL_WORKER_NUMBER and the latter
IS_BIOC_BUILD_MACHINE. This means
availableCores(which = "all") now reports on
both.
isNodeAlive() will now produce a once-per-session
informative warning when it detects that it is not possible to check
whether another process is alive on the current machine.
Add section to
help("makeClusterPSOCK", package = "parallelly") explaining
why R CMD check may produce “checking for detritus in the
temp directory … NOTE” and how to avoid them.
Add section ‘For package developers’ to
help("makeClusterPSOCK", package = "parallelly")
reminding us that we need to stop all clusters we created in package
examples, tests, and vignettes.
isNodeAlive() failed to record which method works for
testing if a process exists or not, which meant it would keep trying all
methods each time. Similarly, if none works, it would still keep trying
each time instead of returning NA immediately. On some systems, failing
to check whether a process exists could result in one or more warnings,
in which case those warnings would be produced for each call to
isNodeAlive().host element of the SOCK0node or
SOCKnode objects created by makeClusterPSOCK()
lost attribute localhost for localhost workers. This made
some error messages from the future package less
informative.revtunnel of
makeNodePSOCK(), and therefore also of
makeClusterPSOCK(), is now NA, which means
it’s agile to whether rshcmd[1] specifies an SSH client, or
not. If SSH is used, then it will resolve to
revtunnel = TRUE, otherwise to
revtunnel = FALSE. This removed the need for setting
revtunnel = FALSE, when non-SSH clients are used.availableCores() and availableWorkers()
gained support for the ‘Fujitsu Technical Computing Suite’ job
scheduler. Specifically, they acknowledges environment variables
PJM_VNODE_CORE, PJM_PROC_BY_NODE, and
PJM_O_NODEINF. See
help("makeClusterPSOCK", package = "parallelly") for an
example.makeClusterPSOCK() would fail with
Error: node$session_info$process$pid == pid is not TRUE
when running R in Simplified Chinese (LANGUAGE=zh_CN),
Traditional Chinese (Taiwan) (LANGUAGE=zh_TW), or Korean
(LANGUAGE=ko) locales.
Some warnings and errors showed the wrong call.
Changes to option parallelly.availableCores.system
would be ignored if done after the first call to
availableCores().
availableCores() with option
parallelly.availableCores.system set to less that
parallel::detectCores() would produce a warning,
e.g. “[INTERNAL]: Will ignore the cgroups CPU set, because it contains
one or more CPU indices that is out of range [0,0]: 0-7”.
freePort()
to "random", which used to be "first". The
main reason for this is to make sure the default behavior is to return a
random port also on R (< 4.0.0) where we cannot test whether or not a
port is available.On Unix, availableCores() now queries also control
groups (cgroups) fields cpu.cfs_quota_us and
cpu.cfs_period_us, for a possible CPU quota allocation. If
a CPU quota is set, then the number of CPUs is rounded to the nearest
integer, unless its less that 0.5, in case it’s rounded up to a single
CPU. An example, where cgroups CPU quotas can be set to limit the total
CPU load, is with Linux containers,
e.g. docker run --cpus=3.5 ....
In addition to cgroups CPU quotas, availableCores()
also queries cgroups for a possible CPU affinity, which is available in
field cpuset.set. This should give the same result as what
the already existing ‘nproc’ method gives. However, not all systems have
the nproc tool installed, in which case this new approach
should work. Some high-performance compute (HPC) environments set the
CPU affinity so that jobs do not overuse the CPUs. It may also be set by
Linux containers,
e.g. docker run --cpuset-cpus=0-2,8 ....
The minimum value returned by availableCores() is
one (1). This can be overridden by new option
parallelly.availableCores.min. This can be used to test
parallelization methods on single-core machines,
e.g. options(parallelly.availableCores.min = 2L).
The ‘nproc’ result for availableCores() was ignored
if nproc > 9.
availableCores() would return the ‘fallback’ value
when only ‘system’ and ‘nproc’ information was available. However, in
this case, we do want it to return ‘nproc’ when ‘nproc’ != ‘system’,
because that is a strong indication that the number of CPU cores is
limited by control groups (cgroups) on Linux. If ‘nproc’ == ‘system’, we
cannot tell whether cgroups is enabled or not, which means we will fall
back to the ‘fallback’ value if there is no other evidence that another
number of cores are available to the current R process.
Technically, canPortBeUsed() could falsely return
FALSE if the port check was interrupted by, say, a user
interrupt.
freePort(ports, default = "random") would always use
return ports[1] if the system does not allow testing if a
port is available or not, or if none of the specified ports are
available.
makeNodePSOCK(), and therefore also
makeClusterPSOCK(), gained argument
rscript_sh, which controls how Rscript
arguments are shell quoted. The default is to make a best guess on what
type of shell is used where each cluster node is launched. If launched
locally, then it whatever platform the current R session is running,
i.e. either a POSIX shell ("sh") or MS Windows
("cmd"). If remotely, then the assumption is that a POSIX
shell ("sh") is used.
makeNodePSOCK(), and therefore also
makeClusterPSOCK(), gained argument
default_packages, which controls the default set of R
packages to be attached on each cluster node at startup. Moreover, if
argument rscript specifies an ‘Rscript’ executable, then
argument default_packages is used to populate Rscript
command-line option --default-packages=.... If
rscript specifies something else, e.g. an ‘R’ or ‘Rterm’
executable, then environment variable
R_DEFAULT_PACKAGES=... is set accordingly when launching
each cluster node.
Argument rscript_args of
makeClusterPSOCK() now supports "*" values.
When used, the corresponding element will be replaced with the
internally added Rscript command-line options. If not specified, such
options are appended at the end.
makeClusterPSOCK() did not support backslashes
(\) in rscript_libs, backslashes that may
originate from, for example, Windows network drives. The result was that
the worker would silently ignore any rscript_libs
components with backslashes.
The package detects when R CMD check runs and adjust
default settings via environment variables in order to play nicer with
the machine where the checks are running. Some of these environment
variables were in this case ignored since parallelly
1.26.0.
makeClusterPSOCK() launches parallel workers with
option socketOptions set to "no-delay" by
default. This decreases the communication latency between workers and
the main R session, significantly so on Unix. This option requires R
(>= 4.1.0) and has no effect in early versions of R.Added argument socketOptions to
makeClusterPSOCK(), which sets the corresponding R option
on each cluster node when they are launched.
Argument rscript_envs of
makeClusterPSOCK() can also be used to unset environment
variables cluster nodes. Any named element with value
NA_character_ will be unset.
Argument rscript of makeClusterPSOCK()
now supports "*" values. When used, the corresponding
element will be replaced with the "Rscript", or if
homogenous = TRUE, then absolute path to current
‘Rscript’.
makeClusterPSOCK() example on how to launch workers
distributed across multiple CPU Groups on MS Windows 10.isForkedChild() would only return TRUE in a forked
child process, if and only if, it had already been called in the parent
R process.
Using argument rscript_startup would cause
makeClusterPSOCK() to fail in R-devel (>=
r80666).
example("isNodeAlive") now uses
\donttest{} to avoid long (> 10
Add isNodeAlive() to check whether a cluster and
cluster nodes are alive or not.
Add isForkedChild() to check whether or not the
current R process is a forked child process.
Environment variable
R_PARALLELLY_SUPPORTSMULTICORE_UNSTABLE was incorrectly
parsed as a logical instead of a character string. If the variables was
set to, say, "quiet", this would cause an error when the
package was loaded.
makeClusterPSOCK() failed to fall back to
setup_strategy = "sequential", when not supported by the
current R version.
availableCores() and availableWorkers()
now respects environment variable
BIOCPARALLEL_WORKER_NUMBER introduced in
BiocParallel (>= 1.27.2). They also respect
BBS_HOME which is set on the Bioconductor check servers to
limit the number of parallel workers while checking Bioconductor
packages.makeClusterPSOCK() and
parallel::makeCluster() failed with error “Cluster setup
failed. setup_strategy = "parallel" and when the
tcltk package is loaded when running R (>= 4.0.0
&& <= 4.1.0) on macOS. Now parallelly forces
setup_strategy = "sequential" when the
tcltk package is loaded on these R versions.makeClusterPSOCK(..., setup_strategy = "parallel")
would forget to close an socket connection used to set up the workers.
This socket connection would be closed by the garbage collector
eventually with a warning.
parallelly::makeClusterPSOCK() would fail with
“Error in freePort(port) : Unknown value on argument ‘port’: ‘auto’” if
environment variable R_PARALLEL_PORT was set to a port
number.
parallelly::availableCores() would produce ‘Error in
if (grepl(“^ [1-9]$”, res)) return(as.integer(res)) : argument is of
length zero’ on Linux systems without nproc
installed.
print() on RichSOCKcluster mentions when
the cluster is registered to be automatically stopped by the garbage
collector.setup_strategy = "parallel" when using
makeClusterPSOCK() or parallel::makeCluster().
The symptom is that they, after a long wait, result in “Error in
makeClusterPSOCK(workers, …) : Cluster setup failed. setup_strategy = "sequential for
parallelly and parallel when running
in the RStudio Console. If you wish to override this behavior, you can
always set option parallelly.makeNodePSOCK.setup_strategy
to "parallel", e.g. in your ~/.Rprofile file.
Alternatively, you can set the environment variable
R_PARALLELLY_MAKENODEPSOCK_SETUP_STRATEGY=parallel, e.g. in
your ~/.Renviron file.nproc installed,
availableCores() would be limited by environment variables
OMP_NUM_THREADS and OMP_THREAD_LIMIT, if set.
For example, on conservative systems that set
OMP_NUM_THREADS=1 as the default,
availableCores() would pick this up via nproc
and return 1. This was not the intended behavior. Now those environment
variables are temporarily unset before querying nproc.R_PARALLELLY_* (and R_FUTURE_*)
environment variables are now only read when the
parallelly package is loaded, where they set the
corresponding parallelly.* option. Previously, some of
these environment variables were queried by different functions as a
fallback to when an option was not set. By only parsing them when the
package is loaded, it decrease the overhead in functions, and it
clarifies that options can be changed at runtime whereas environment
variables should only be set at startup.makeClusterPSOCK() now support setting up cluster
nodes in parallel similarly to how
parallel::makePSOCKcluster() does it. This significantly
reduces the setup turnaround time. This is only supported in R (>=
4.0.0). To revert to the sequential setup strategy, set R option
parallelly.makeNodePSOCK.setup_strategy to
"sequential".
Add freePort() to get a random TCP port that can be
opened.
parallelly.availableCores.fallback and
environment variable R_PARALLELLY_AVAILABLECORES_FALLBACK
was ignored since parallelly 1.22.0, when support for
‘nproc’ was added to availableCores().ssh client. This means that regardless whether you are on
Linux, macOS, or Windows 10, setting up parallel workers on external
machines over SSH finally works out of the box without having to install
PuTTY or other SSH clients. This was possible because a workaround was
found for a Windows 10 bug preventing us from using reverse tunneling
over SSH. It turns out the bug reveals itself when using hostname
‘localhost’ but not ‘127.0.0.1’, so we use the latter.availableCores() gained argument omit to
make it easier to put aside zero or more cores from being used in
parallel processing. For example, on a system with four cores,
availableCores(omit = 1) returns 3. Importantly, since
availableCores() is guaranteed to always return a positive
integer, availableCores(omit = 4) == 1, even on systems
with four or fewer cores. Using availableCores() - 4 on
such systems would return a non-positive value, which would give an
error downstream.makeClusterPSOCK(), or actually
makeNodePSOCK(), did not accept all types of environment
variable names when using rscript_envs, e.g. it would give
an error if we tried to pass
_R_CLASS_MATRIX_ARRAY_.
makeClusterPSOCK() had a “length > 1 in coercion
to logical” bug that could affect especially MS Windows 10
users.
plink of the PuTTY software, (ii)
ssh in the RStudio distribution, and (iii) ssh
of Windows 10. Previously, the latter was considered first but that
still has a bug preventing us from using reverse tunneling.makeClusterPSOCK(), or actually
makeNodePSOCK(), gained argument quiet, which
can be used to silence output produced by
manual = TRUE.
c() for cluster objects now warns about
duplicated cluster nodes.
Add isForkedNode() to test if a cluster node runs in
a forked process.
Add isLocalhostNode() to test if a cluster node runs
on the current machine.
Now availableCores() and
availableWorkers() avoid recursive calls to the custom
function given by options parallelly.availableCores.custom
and parallelly.availableWorkers.custom,
respectively.
availableWorkers() now recognizes the Slurm
environment variable SLURM_JOB_NODELIST,
e.g. "dev1,n[3-4,095-120]". It will use
scontrol show hostnames "$SLURM_JOB_NODELIST" to expand it,
if supported on the current machine, otherwise it will attempt to parse
and expand the nodelist specification using R. If either of environment
variable SLURM_JOB_CPUS_PER_NODE or
SLURM_TASKS_PER_NODE is set, then each node in the nodelist
will be represented that number of times. If in addition, environment
variable SLURM_CPUS_PER_TASK (always a scalar), then that
is also respected.
parallelly. prefix for
options and the R_PARALLELLY_ prefix for environment
variables. Settings that use the corresponding future. and
R_FUTURE_ prefixes are still recognized.availableCores() did not respect environment
variable SLURM_TASKS_PER_NODE when the job was allocated
more than one node.
Above argument quiet was introduced in
future 1.19.1 but was mistakenly dropped from
parallelly 1.20.0 when that was released, and therefore
also from future (>= 1.20.0).
availableCores(), availableWorkers(),
and freeCores() gained argument logical, which
is passed down to parallel::detectCores() as-is. The
default is TRUE but it can be changed by setting the R option
parallelly.availableCores.logical. This option can in turn
be set via environment variable
R_PARALLELLY_AVAILABLECORES_LOGICAL which is applied (only)
when the package is loaded.
Now makeClusterPSOCK() asserts that there are enough
free connections available before attempting to create the parallel
workers. If too many workers are requested, an informative error message
is produced.
Add availableConnections() and
freeConnections() to infer the maximum number of
connections that the current R installation can have open at any time
and how many of those are currently free to be used. This limit is
typically 128 but may be different in custom R installations that are
built from source.
Now availableCores() queries also Unix command
nproc, if available. This will make it respect the number
of CPU/cores limited by ‘cgroups’ and Linux containers.
PSOCK cluster workers are now set up to communicate using little
endian (useXDR = FALSE) instead of big endian
(useXDR = TRUE). Since most modern systems use little
endian, useXDR = FALSE speeds up the communication
noticeably (10-15%) on those systems. The default value of this argument
can be controlled by the R option
parallelly.makeNodePSOCK.useXDR or the corresponding
environment variable
R_PARALLELLY_MAKENODEPSOCK_USEXDR.
Add cpuLoad() for querying the “average” system load
on Unix-like systems.
Add freeCores() for estimating the average number of
unused cores based on the average system load as given by
cpuLoad().
R_FUTURE_AVAILABLECORES_FALLBACK and
R_FUTURE_AVAILABLECORES_SYSTEM, none of the
R_PARALLELLY_* and R_FUTURE_* ones where
recognized.find_rshcmd() which was never meant to be
exported.makeClusterPSOCK() gained argument
validate to control whether or not the nodes should be
tested after they’ve been created. The validation is done by querying
each node for its session information, which is then saved as attribute
session_info on the cluster node object. This information
is also used in error messages, if available. This validation has been
done since version 1.5.0 but now it can be disabled. The default of
argument validate can be controlled via an R options and an
environment variable.
Now makeNodePSOCK(..., rscript_envs = "UNKNOWN")
produces an informative warning on non-existing environment variables
that was skipped.
makeClusterPSOCK() would produce an error on ‘one
node produced an error: could not find function “getOptionOrEnvVar”’ if
parallelly is not available on the node.
makeClusterPSOCK() would attempt to load
parallelly on the worker. If it’s not available on the
worker, it would result in a silent warning on the worker. Now
parallelly is not loaded.
makeClusterPSOCK(..., tries = n) would retry to
setup a cluster node also on errors that were unrelated to node setup or
node connection errors.
The error message on using an invalid rscript_envs
argument for makeClusterPSOCK() reported on the value of
rscript_libs (sic!).
makeNodePSOCK(..., rscript_envs = "UNKNOWN") would
result in an error when trying to launch the cluster node.
find_rshcmd() which was never meant to be
exported.availableCores(), and
availableWorkers(), supportsMulticore(),
as.cluster(), autoStopCluster(),
makeClusterMPI(), makeClusterPSOCK(), and
makeNodePSOCK() from the future
package.isConnectionValid() and connectionId()
adopted from internal code of the future package.Renamed environment variable
R_FUTURE_MAKENODEPSOCK_tries used by
makeClusterPSOCK() to
R_FUTURE_MAKENODEPSOCK_TRIES.
connectionId() did not return -1L on
Solaris for connections with internal ‘nil’ pointers because they were
reported as ‘0’ - not ‘nil’ or ‘0x0’.
Now availableCores() better supports Slurm.
Specifically, if environment variable SLURM_CPUS_PER_TASK
is not set, which requires that option
--slurm-cpus-per-task=n is specified and
SLURM_JOB_NUM_NODES=1, then it falls back to using
SLURM_CPUS_ON_NODE, e.g. when using
--ntasks=n.
Now availableCores() and
availableWorkers() supports LSF/OpenLava. Specifically,
they acknowledge environment variable LSB_DJOB_NUMPROC and
LSB_HOSTS, respectively.
makeClusterPSOCK() will now retry to create a
cluster node up to tries (default: 3) times before giving
up. If argument port species more than one port
(e.g. port = "random") then it will also attempt find a
valid random port up to tries times before giving up. The
pre-validation of the random port is only supported in R (>= 4.0.0)
and skipped otherwise.
makeClusterPSOCK() skips shell quoting of the
elements in rscript if it inherits from
AsIs.
makeClusterPSOCK(), or actually
makeNodePSOCK(), gained argument quiet, which
can be used to silence output produced by
manual = TRUE.
plan(multisession),
plan(cluster, workers = <number>), and
makeClusterPSOCK() which they both use internally, sets up
localhost workers twice as fast compared to versions since
future 1.12.0, which brings it back to par with a
bare-bone
parallel::makeCluster(..., setup_strategy = "sequential")
setup. The slowdown was introduced in future 1.12.0
(2019-03-07) when protection against leaving stray R processes behind
from failed worker startup was implemented. This protection now makes
use of memoization for speedup.print() on RichSOCKcluster gives
information not only on the name of the host but also on the version of
R and the platform of each node (“worker”), e.g. “Socket cluster with 3
nodes where 2 nodes are on host ‘localhost’ (R version 4.0.0
(2020-04-24), platform x86_64-w64-mingw32), 1 node is on host ‘n3’ (R
version 3.6.3 (2020-02-29), platform x86_64-pc-linux-gnu)”.
It is now possible to set environment variables on workers before
they are launched by makeClusterPSOCK() by specify them as
as <name>=<value> as part of the
rscript vector argument,
e.g. rscript=c("ABC=123", "DEF='hello world'", "Rscript").
This works because elements in rscript that match regular
expression "^ [[:alpha:]_][[:alnum:]_]*=.*" are no longer
shell quoted.
makeClusterPSOCK() now returns a cluster that in
addition to inheriting from SOCKcluster it will also
inherit from RichSOCKcluster.
Made makeClusterPSOCK() and
makeNodePSOCK() agile to the name change from
parallel:::.slaveRSOCK() to
parallel:::.workRSOCK() in R (>= 4.1.0).
makeClusterPSOCK(..., rscript) will not try to
locate rscript[1] if argument homogeneous is
FALSE (or inferred to be FALSE).
makeClusterPSOCK(..., rscript_envs) would result in
a syntax error when starting the workers due to non-ASCII quotation
marks if option useFancyQuotes was not set to
FALSE.
makeClusterPSOCK() gained argument
rscript_envs for setting environment variables in workers
on startup,
e.g. rscript_envs = c(FOO = "3.14", "BAR")._R_CHECK_LIMIT_CORES_ set. To
better emulate CRAN submission checks, the future
package will, when loaded, set this environment variable to TRUE if
unset and if R CMD check is running. Note that
future::availableCores() respects
_R_CHECK_LIMIT_CORES_ and returns at most 2L
(two cores) if detected.makeClusterPSOCK()
draws a random port from (when argument port is not
specified) can now be controlled by environment variable
R_FUTURE_RANDOM_PORTS. The default range is still
11000:11999 as with the parallel
package.?makeClusterPSOCK
with instructions on how to troubleshoot when the setup of local and
remote clusters fail.makeClusterPSOCK() could produce warnings like
“cannot open file
‘/tmp/alice/Rtmpi69yYF/future.parent=2622.a3e32bc6af7.pid’: No such
file”, e.g. when launching R workers running in Docker
containers.
makeClusterMPI() did not work for MPI clusters with
‘comm’ other than ‘1’.
Now availableCores() also recognizes PBS environment
variable NCPUS, because the PBSPro scheduler does not set
PBS_NUM_PPN.
If, option future.availableCores.custom is set to a
function, then availableCores() will call that function and
interpret its value as number of cores. Analogously, option
future.availableWorkers.custom can be used to specify a
hostnames of a set of workers that availableWorkers() sees.
These new options provide a mechanism for anyone to customize
availableCores() and availableWorkers() in
case they do not (yet) recognize, say, environment variables that are
specific the user’s compute environment or HPC scheduler.
makeClusterPSOCK() gained support for argument
rscript_startup for evaluating one or more R expressions in
the background R worker prior to the worker event loop launching. This
provides a more convenient approach than having to use, say,
rscript_args = c("-e", sQuote(code)).
makeClusterPSOCK() gained support for argument
rscript_libs to control the R package library search path
on the workers. For example, to prepend the folder
~/R-libs on the workers, use
rscript_libs = c("~/R-libs", "*"), where "*"
will be resolved to the current .libPaths() on the
workers.
makeClusterPSOCK() did not shell quote the Rscript
executable when running its pre-tests checking whether localhost Rscript
processes can be killed by their PIDs or not.makeClusterPSOCK() fails to create one of many
nodes, then it will attempt to stop any nodes that were successfully
created. This lowers the risk for leaving R worker processes
behind.makeClusterPSOCK() in future (>=
1.11.1) produced warnings when argument rscript had
length(rscript) > 1.makeClusterPSOCK() fails to connect to a worker,
it produces an error with detailed information on what could have
happened. In rare cases, another error could be produced when generating
the information on what the workers PID is.makeClusterPSOCK()
and makeNodePSOCK() can now be controlled via environment
variables in addition to R options that was supported in the past. An
advantage of using environment variables is that they will be inherited
by child processes, also nested ones.R CMD check is running or not. If it is, then a
few future-specific environment variables are adjusted such that the
tests play nice with the testing environment. For instance, it sets the
socket connection timeout for PSOCK cluster workers to 120 seconds
(instead of the default 30 days!). This will lower the risk for more and
more zombie worker processes cluttering up the test machine (e.g. CRAN
servers) in case a worker process is left behind despite the main R
processes is terminated. Note that these adjustments are applied
automatically to the checks of any package that depends on, or imports,
the future package.makeClusterPSOCK() would fail to connect to a
worker, for instance due to a port clash, then it would leave the R
worker process running - also after the main R process terminated. When
the worker is running on the same machine,
makeClusterPSOCK() will now attempt to kill such stray R
processes. Note that parallel::makePSOCKcluster() still has
this problem.makeClusterPSOCK() produces more informative error
messages whenever the setup of R workers fails. Also, its verbose
messages are now prefixed with “[local output]” to help distinguish the
output produced by the current R session from that produced by
background workers.
It is now possible to specify what type of SSH clients
makeClusterPSOCK() automatically searches for and in what
order,
e.g. rshcmd = c("<rstudio-ssh>", "<putty-plink>").
Now makeClusterPSOCK() preserves the global RNG
state (.Random.seed) also when it draws a random port
number.
makeClusterPSOCK() gained argument
rshlogfile.
makeClusterPSOCK(..., rscript = "my_r") would in some
cases fail to find the intended my_r executable.Add makeClusterMPI(n) for creating MPI-based
clusters of a similar kind as
parallel::makeCluster(n, type = "MPI") but that also
attempts to workaround issues where parallel::stopCluster()
causes R to stall.
makeClusterPSOCK() and makeClusterMPI()
gained argument autoStop for controlling whether the
cluster should be automatically stopped when garbage collected or
not.
makeClusterPSOCK() produced a warning when environment
variable R_PARALLEL_PORT was set to random
(e.g. as on CRAN).makeClusterPSOCK() now produces a more informative
warning if environment variable R_PARALLEL_PORT specifies a
non-numeric port.makeClusterPSOCK(), and therefore
plan(multisession) and plan(multiprocess),
will use the SSH client distributed with RStudio as a fallback if
neither ssh nor plink is available on the
system PATH.makeClusterPSOCK(..., renice = 19) would launch each
PSOCK worker via nice +19 resulting in the error “nice:
‘+19’: No such file or directory”. This bug was inherited from
parallel::makePSOCKcluster(). Now using
nice --adjustment=19 instead.makeClusterPSOCK() now defaults to use the Windows
PuTTY software’s SSH client plink -ssh, if ssh
is not found.
Argument homogeneous of
makeNodePSOCK(), a helper function of
makeClusterPSOCK(), will default to FALSE also if the
hostname is a fully qualified domain name (FQDN), that is, it “contains
periods”. For instance, c('node1', 'node2.server.org') will
use homogeneous = TRUE for the first worker and
homogeneous = FALSE for the second.
makeClusterPSOCK() now asserts that each cluster
node is functioning by retrieving and recording the node’s session
information including the process ID of the corresponding R
process.
makeClusterPSOCK() gained more detailed
descriptions on arguments and what their defaults are.connectTimeout and
timeout of makeNodePSOCK() can now be
controlled via global options.availableCores(method = "mc.cores") is now defunct in
favor of "mc.cores+1".makeClusterPSOCK() treats workers that refer to a
local machine by its local or canonical hostname as
"localhost". This avoids having to launch such workers over
SSH, which may not be supported on all systems / compute
cluster.
Added availableWorkers(). By default it returns
localhost workers according to availableCores(). In
addition, it detects common HPC allocations given in environment
variables set by the HPC scheduler.
Option future.availableCores.fallback, which
defaults to environment variable
R_FUTURE_AVAILABLECORES_FALLBACK can now be used to specify
the default number of cores / workers returned by
availableCores() and availableWorkers() when
no other settings are available. For instance, if
R_FUTURE_AVAILABLECORES_FALLBACK=1 is set system wide in an
HPC environment, then all R processes that uses
availableCores() to detect how many cores can be used will
run as single-core processes. Without this fallback setting, and without
other core-specifying settings, the default will be to use all cores on
the machine, which does not play well on multi-user systems.
Creation of cluster futures (including multisession ones) would
time out already after 40 seconds if all workers were busy. New default
timeout is 30 days (option future.wait.timeout).
availableCores(methods = "_R_CHECK_LIMIT_CORES_")
would give an error if not running R CMD check.
Added makeClusterPSOCK() - a version of
parallel::makePSOCKcluster() that allows for more flexible
control of how PSOCK cluster workers are set up and how they are
launched and communicated with if running on external machines.
Added generic as.cluster() for coercing objects to
cluster objects to be used as in
plan(cluster, workers = as.cluster(x)). Also added a
c() implementation for cluster objects such that multiple
cluster objects can be combined into a single one.
user to remote() was ignored
(since 1.1.0).workers = "localhost" they (again) use the exact same R
executable as the main / calling R session (in all other cases it uses
whatever Rscript is found on the PATH). This
was already indeed implemented in 1.0.1, but with the added support for
reverse SSH tunnels in 1.1.0 this default behavior was lost.cluster()
and remote() to connect to remote clusters / machines. As
long as you can connect via SSH to those machines, it works also with
these future. The new code completely avoids incoming firewall and
incoming port forwarding issues previously needed. This is done by using
reverse SSH tunneling. There is also no need to worry about internal or
external IP numbers.availableCores() also acknowledges environment
variable NSLOTS set by Sun/Oracle Grid Engine (SGE).availableCores() returns 3L
(=2L+1L) instead of 2L if
_R_CHECK_LIMIT_CORES_ is set.availableCores() also acknowledges the number of
CPUs allotted by Slurm.availableCores("mc.cores") returns
getOption("mc.cores") + 1L, because option
mc.cores specifies “allowed number of additional R
processes” to be used in addition to the main R process.