Python Worklets
Since version 2.077009 Netdisco has support for backend worker components written in Python. This document is a guide to implementing such components.
A "worklet" is plug-in code that does some functionality inside a backend worker. Netdisco jobs are made up from many worklets that activate at different phases of a worker (early, main, late, etc.) and for different drivers (snmp, cli, etc.).
# file netdisco/worklet/stats/early.py
import platform
from netdisco.util.worklet import debug, context as c
def main():
c.stash.set('python_ver', platform.python_version())
debug('python version is: ' + c.stash.get('python_ver'))
c.status.info('stashed Python version')
if __name__ == '__main__':
main()Configure the extra_python_worker_plugins setting to tell Netdisco about your
worklets. It’s a list of the module names (with dot separators). The order of entries
is not significant.
You can implement whole backend jobs in Python, or just add some worklets in Python to a backend job already implemented in Perl. Netdisco ensures worklets are called in the right order (by phase, namespace, driver priority, etc.).
The action, phase, and driver of the worklet can be encoded in the module name.
So, the stats example above is configured as:
extra_python_worker_plugins:
- 'stats.early'The more complex arpnip job could in theory be configured as:
extra_python_worker_plugins:
- 'arpnip.check'
- 'arpnip.nodes.early'
- 'arpnip.nodes.main': ['direct', 'cli', 'snmp']
- 'arpnip.subnets.main.snmp'
- 'arpnip.nodes.store'
- 'arpnip.hooks.late'You can avoid repeating the same action and phase for multiple drivers by
providing a list of drivers as in the arpnip example, above. Note that
they are still separate module files.
For additional configuration on a worklet, such as priority or ACLs, instead
provide a dictionary with settings:
extra_python_worker_plugins:
- 'arpnip.nodes.extra.main':
priority: 250
only: ['vendor:cisco', 'os:ios-xr']Any settings provided this way will override what’s in the module name. Only the job and phase are required in the module name, and it must of course be unique (you can add some namespace components at the end to achieve this).
If you add entries with local worklets to extra_python_worker_plugins they’re
merged with the built-in ones from python_worker_plugins.
Use the PYTHONPATH environment variable
to let Netdisco load your Python files. However, they are not automatically
discovered - see above for the extra_python_worker_plugins setting.
Netdisco will load worklets from a namespace package
called netdiscox or whatever you set extra_python_worker_package_namespace
to, followed by the directory worklet.
Metadata, including the job, phase, and driver of the worklet are encoded in the worklet file name.
For example the linter worklet lives in the filesystem at:
.../PYTHONPATH/netdisco/worklet/linter/main.pyThe model for Python worklet naming is:
netdisco/worklet/<job name> (required)
/<optional>/<job-stage>/<components>/...
/<phase> (required)
/<optional driver>
/<optional>/<namespace>/<components>/...
.py (required)Another example, this time all the worklets for the arpnip job:
netdisco/worklet/arpnip/check.py
netdisco/worklet/arpnip/nodes/early.py
netdisco/worklet/arpnip/nodes/main/direct.py
netdisco/worklet/arpnip/nodes/main/cli.py
netdisco/worklet/arpnip/nodes/main/snmp.py
netdisco/worklet/arpnip/subnets/main/snmp.py
netdisco/worklet/arpnip/store.py
netdisco/worklet/arpnip/hooks/late.pyNote that in Perl, several worklets can be bundled into the same Package file. In Python, each worklet is in its own file.
One more example; a worklet added to target a very specific situation could have the following configuration and filename to make it unique:
extra_python_worker_plugins:
- 'discover.nexthopneighbors.main.cli.juniper_junos':
only: 'route-server.ip.att.net' .../PYTHONPATH/netdisco/worklet/discover/nexthopneighbors/main/cli/juniper_junos.pyWe use the if __name__ == '__main__': idiom to enter the worklet, as in
the examples.
Integration with the Netdisco application is via a "context" god-object instance with several convenience functions:
from netdisco.util.worklet import debug, context as cYou can import the Context as context or c - either will work. Context provides:
Read-only access to Job parameters, such as action, device, subaction (or the "extra" alias of subaction), and port.
from netdisco.util.worklet import context as c
device = c.job.device
port = c.job.port
# etc...The stash provides a way to pass data between all the worklets, both Perl and Python. It is a dictionary, and will throw KeyError if an entry isn’t found.
from netdisco.util.worklet import context as c
filename = c.stash.get('file_to_lint')
c.stash.set('new_filename', '/path/to/config.yaml')If you’re also writing Perl worklets, this is analogous to the vars cache,
and the two are synchronised. That is, entries in vars will appear in the
Stash in Python, and keys updated by Python workers will appear back in vars
afterwards.
Read-only access to Netdisco configuration settings at the time the worklet is executed. It is a dictionary, and will throw KeyError if an entry isn’t found. You cannot modify the runtime configuration.
from netdisco.util.worklet import context as c
log_level = c.setting('log')The device_auth setting is always limited before executing a worklet to
only the entries that match the worklet’s device/platform/driver.
Each worklet can record a status and log message in Netdisco. The status can be "done", "defer", "info", or "error".
from netdisco.util.worklet import context as c
if found_issues:
c.status.error('Lint errors, view with --debug')
else:
c.status.done('Linted OK')Worklets in check phase must PASS (status "done") or the job will be aborted.
For any other phase, it doesn’t matter. It’s fine not to set a status (or if
in doubt, use "info" which is a neutral response). Remember that the first
worklet of any job stage which returns "done" causes the rest to be skipped.
Rudimentary access to Netdisco’s database is available. This uses SQLAlchemy
with the psycopg (version 3) driver underneath. As of writing there’s no ORM
support. The db variable aliases the SQLAlchemy engine.
from sqlalchemy import text
from netdisco.util.worklet import context as c
with c.db.begin() as conn:
conn.execute(
text("INSERT INTO some_table (x, y) VALUES (:x, :y)"),
[{"x": 6, "y": 8}, {"x": 9, "y": 10}],
)Submit a log message, which is printed if Netdisco’s log setting is debug.
from netdisco.util.worklet import debug, context as c
debug('python version is: ' + c.stash.get('python_ver'))In the debug log, the Python subprocess and worklets are indicated by lines as in the example below. Debug log messages are prefixed with the process ID of the Python process.
[68569] 2025-01-29 16:31:27 debug ⮕ worker PythonShim netdisco.worklet.stats.early p0
[68569] 2025-01-29 16:31:27 debug 🐍 starting persistent Python worklet subprocess
[68569] 2025-01-29 16:31:27 debug [68572] python version is: 3.12.5
[68569] 2025-01-29 16:31:27 debug ⬅ (info) stashed Python versionNetdisco orders the worklets as it ever did - the Python ones are no different to Perl. Netdisco merges all worklets it finds according to their config.
If you use a driver name they’re prioritised (direct, cli, snmp), and the
first that returns done status wins. It’s OK not to specify a driver, of
course.
Anything put into vars() will appear in the Stash. So, you could use an
early phase worker to gather data from devices or the database to pass
to Python for processing.
# App/NetdiscoX/Worker/Plugin/MyJob.pm
# phase => early
vars->{'file_to_lint'} ||= '/path/to/file.yaml'; # netdisco/worklet/myjob/main.py
c.stash.get('file_to_lint')Anything put into the Stash will appear back in vars() after the worklet
finishes. So, you can gather data from a device then stash it for saving to
the database in a Perl store worklet.
# netdisco/worklet/myjob/main.py
c.stash.set('python_ver', platform.python_version()) # App/NetdiscoX/Worker/Plugin/MyJob.pm
# phase => store
$row->update({ python_ver => vars->{'python_ver'} })