Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ ci:

repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: c59bba8fb259db0fec2bbb77ad8ba51ea7341b56 # frozen: v0.15.20
rev: 321478e58f4938179c6b86e4ddfa923d1547a49b # frozen: v0.16.6
hooks:
- id: ruff
args:
Expand Down Expand Up @@ -33,12 +33,12 @@ repos:
# isort rules get confused in the temporary files
- --ignore=I
- repo: https://github.com/rbubley/mirrors-prettier
rev: 39e2973981e6d2f9b6c543b0086a2d2393abdc89 # frozen: v3.9.4
rev: 0ee178619d696787ca73d210cc191d720868c631 # frozen: v3.9.6
hooks:
- id: prettier

- repo: https://github.com/zizmorcore/zizmor-pre-commit
rev: e3eebf65325ccc992422292cb7a4baee967cf815 # frozen: v1.26.1
rev: cef8b8350da46d8114c7e6b7272aebdccdf193ce # frozen: v1.30.0
hooks:
- id: zizmor
args:
Expand Down
3 changes: 3 additions & 0 deletions docs/source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,14 @@ You can similarly run MPI code using IPyParallel (requires [mpi4py](https://mpi4
```python
import ipyparallel as ipp


def mpi_example():
from mpi4py import MPI

comm = MPI.COMM_WORLD
return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"


# request an MPI cluster with 4 engines
with ipp.Cluster(engines='mpi', n=4) as rc:
# get a broadcast_view on the cluster which is best
Expand Down
2 changes: 1 addition & 1 deletion docs/source/reference/db.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ or in your {file}`ipcontroller_config.py`:

```python
c.IPController.db_class = "NoDB"
c.IPController.db_class = "DictDB" # default
c.IPController.db_class = "DictDB" # default
c.IPController.db_class = "MongoDB"
c.IPController.db_class = "SQLiteDB"
```
Expand Down
15 changes: 10 additions & 5 deletions docs/source/reference/details.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,14 @@ def f(a):
def inner():
# inner will have a closure
return a

return inner


f1 = f(1)
f2 = f(2)
f1() # returns 1
f2() # returns 2
f1() # returns 1
f2() # returns 2
```

`f1` and `f2` will have closures referring to the scope in which `inner` was defined,
Expand All @@ -126,12 +128,15 @@ def g(b):
def inner():
# this inner will *not* have a closure
return a

return inner


g1 = g(1)
g2 = g(2)
g1() # raises NameError on 'a'
a=5
g2() # returns 5
g1() # raises NameError on 'a'
a = 5
g2() # returns 5
```

`g1` and `g2` _will_ be sendable with IPython, and will treat the engine's namespace as
Expand Down
3 changes: 3 additions & 0 deletions docs/source/reference/launchers.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ and can be selected via short lowercase string naming the kind of launcher, e.g.

```python
import ipyparallel as ipp

c = ipp.Cluster(engines="mpi")
```

Expand Down Expand Up @@ -102,6 +103,8 @@ For example:
```python
from traitlets import Integer
from ipyparallel.cluster.launcher import EngineLauncher


class MyLauncher(EngineLauncher):
pid = Integer(
help="The pid of the process",
Expand Down
1 change: 1 addition & 0 deletions docs/source/reference/messages.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,7 @@ data_pub messages are constructed via the {func}`ipyparallel.datapub.publish_dat

```python
from ipyparallel.datapub import publish_data

ns = dict(x=my_array)
publish_data(ns)
```
Expand Down
5 changes: 2 additions & 3 deletions docs/source/reference/mpi.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,12 +89,11 @@ distributed array. Save the following text in a file called {file}`psum.py`:
from mpi4py import MPI
import numpy as np


def psum(a):
locsum = np.sum(a)
rcvBuf = np.array(0.0, 'd')
MPI.COMM_WORLD.Allreduce([locsum, MPI.DOUBLE],
[rcvBuf, MPI.DOUBLE],
op=MPI.SUM)
MPI.COMM_WORLD.Allreduce([locsum, MPI.DOUBLE], [rcvBuf, MPI.DOUBLE], op=MPI.SUM)
return rcvBuf
```

Expand Down
67 changes: 30 additions & 37 deletions docs/source/reference/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,24 +111,21 @@ circumstances:

```python
{
"ssh": "",
"interface": "tcp://127.0.0.1",
"registration": 54886,
"control": 54888,
"mux": 54890,
"hb_ping": 54891,
"hb_pong": 54892,
"task": 54894,
"iopub": 54896,
"broadcast": [
54900,
54901
],
"key": "7e99e423-c437d4daf7cf23ee84cae803",
"location": "mylaptop",
"pack": "json",
"unpack": "json",
"signature_scheme": "hmac-sha256"
"ssh": "",
"interface": "tcp://127.0.0.1",
"registration": 54886,
"control": 54888,
"mux": 54890,
"hb_ping": 54891,
"hb_pong": 54892,
"task": 54894,
"iopub": 54896,
"broadcast": [54900, 54901],
"key": "7e99e423-c437d4daf7cf23ee84cae803",
"location": "mylaptop",
"pack": "json",
"unpack": "json",
"signature_scheme": "hmac-sha256",
}
```

Expand All @@ -143,25 +140,21 @@ $> ipcontroller --ip=* --ssh=login.mycluster.com

```python
{
"ssh": "login.mycluster.com",
"interface": "tcp://*",
"registration": 55836,
"control": 55837,
"mux": 55839,
"task": 55843,
"task_scheme": "lru",
"iopub": 55845,
"notification": 55852,
"broadcast": [
55847,
55848,
55849
],
"key": "70bc97ac-e66ac5143885ca8b376d4cb7",
"location": "mylaptop",
"pack": "json",
"unpack": "json",
"signature_scheme": "hmac-sha256"
"ssh": "login.mycluster.com",
"interface": "tcp://*",
"registration": 55836,
"control": 55837,
"mux": 55839,
"task": 55843,
"task_scheme": "lru",
"iopub": 55845,
"notification": 55852,
"broadcast": [55847, 55848, 55849],
"key": "70bc97ac-e66ac5143885ca8b376d4cb7",
"location": "mylaptop",
"pack": "json",
"unpack": "json",
"signature_scheme": "hmac-sha256",
}
```

Expand Down
2 changes: 2 additions & 0 deletions docs/source/tutorial/direct.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Option 1. starting a new cluster

```python
import ipyparallel as ipp

cluster = ipp.Cluster(n=4)
cluster.start_cluster_sync()
```
Expand All @@ -32,6 +33,7 @@ or a JupyterLab extension.

```python
import ipyparallel as ipp

cluster = ipp.Cluster.from_file()
```

Expand Down
1 change: 1 addition & 0 deletions docs/source/tutorial/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ For example:

```python
import ipyparallel as ipp

# start cluster, connect client
with ipp.Cluster(n=4) as rc:
e_all = rc[:]
Expand Down
31 changes: 19 additions & 12 deletions docs/source/tutorial/process.md
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,7 @@ $ ipython profile create --parallel --profile=ssh
To use this mode, select the SSH launchers in {file}`ipcluster_config.py`:

```python
c.Cluster.engine_launcher_class = 'ssh' # or 'sshproxy'
c.Cluster.engine_launcher_class = 'ssh' # or 'sshproxy'
# and if the Controller is also to be remote:
c.Cluster.controller_launcher_class = 'ssh'
```
Expand All @@ -389,11 +389,17 @@ Engines are specified in a dictionary, by hostname and the number of engines to
on that host.

```python
c.SSHEngineSetLauncher.engines = { 'host1.example.com' : 2,
'host2.example.com' : 5,
'host3.example.com' : (1, ['--profile-dir=/home/different/location']),
'host4.example.com' : {'n': 3, 'engine_args': ['--profile-dir=/away/location'], 'engine_cmd': ['/home/venv/bin/python', '-m', 'ipyparallel.engine']},
'host5.example.com' : 8 }
c.SSHEngineSetLauncher.engines = {
'host1.example.com': 2,
'host2.example.com': 5,
'host3.example.com': (1, ['--profile-dir=/home/different/location']),
'host4.example.com': {
'n': 3,
'engine_args': ['--profile-dir=/away/location'],
'engine_cmd': ['/home/venv/bin/python', '-m', 'ipyparallel.engine'],
},
'host5.example.com': 8,
}
```

- The `engines` dict, where the keys are the host we want to run engines on and
Expand Down Expand Up @@ -655,6 +661,7 @@ c.IPController.ip = '*'
c.IPController.location = '10.0.1.5'
# or to get an automatic value, try this:
import socket

hostname = socket.gethostname()
# alternate choices for hostname include `socket.getfqdn()`
# or `socket.gethostname() + '.local'`
Expand All @@ -670,10 +677,10 @@ After doing this, your {file}`ipcontroller-client.json` file will look something

```python
{
"url":"tcp:\/\/*:43447",
"exec_key":"9c7779e4-d08a-4c3b-ba8e-db1f80b562c1",
"ssh":"login.mycluster.net",
"location":"10.0.1.5"
"url": "tcp:\/\/*:43447",
"exec_key": "9c7779e4-d08a-4c3b-ba8e-db1f80b562c1",
"ssh": "login.mycluster.net",
"location": "10.0.1.5",
}
```

Expand Down Expand Up @@ -775,7 +782,7 @@ You can also specify a file or python command to be run at startup of the
Engine:

```python
c.IPEngine.startup_script = u'/path/to/my/startup.py'
c.IPEngine.startup_script = '/path/to/my/startup.py'

c.IPEngine.startup_command = 'import numpy, scipy, mpi4py'
```
Expand All @@ -786,7 +793,7 @@ It's also useful on systems with shared filesystems to run the engines
in some scratch directory. This can be set with:

```python
c.IPEngine.work_dir = u'/path/to/scratch/'
c.IPEngine.work_dir = '/path/to/scratch/'
```

[^cite_mpi]: Message Passing Interface (MPI) <https://www.mpi-forum.org>
Expand Down
7 changes: 5 additions & 2 deletions docs/source/tutorial/task.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,14 +137,17 @@ You can also require specific objects, not just module names:
def foo(a):
return a * a


@ipp.require(foo)
def bar(b):
return foo(b)


@ipp.require(bar)
def baz(c, d):
return bar(c) - bar(d)


view.apply_sync(baz, 4, 5)
```

Expand Down Expand Up @@ -373,9 +376,9 @@ assigned to an engine at a given time. This limit is set with the

```python
# the most common choices are:
c.TaskSheduler.hwm = 0 # (minimal latency, default in IPython < 0.13)
c.TaskSheduler.hwm = 0 # (minimal latency, default in IPython < 0.13)
# or
c.TaskScheduler.hwm = 1 # (most-informed balancing, default in ≥ 0.13)
c.TaskScheduler.hwm = 1 # (most-informed balancing, default in ≥ 0.13)
```

In IPython \< 0.13, the default is 0, or no-limit. That is, there is no limit to the number of
Expand Down
Loading