Opacus v1.0.0
We are excited to announce the release of Opacus 1.0. This release packs in lot of new features and bug fixes, and most importantly, brings forth new APIs that are simpler, more modular, and easily extensible.
We have bumped up the major version number from 0 to 1 and have introduced breaking changes; although, the major version bump also indicates a step-function upgrade in the capabilities.
What's new?
With this release we're introducing a slightly different approach to the user-facing library API. While heavily based on the old API, updated API better represents abstractions and algorithms used in DP in ML, enabling private training exactly as it's described in the papers, with no assumptions or simplifications. And in doing so we maintain our focus on high performance training.
Clearer semantics
Previously, PrivacyEngine accepted model as an argument, and then needed to be explicitly attached to optimizer. While simple, it wasn't very clear. The new syntax brings abundant clarity with an explicit make_private() method.
| Opacus 0.x | Opacus 1.0 |
|---|---|
privacy_engine = PrivacyEngine(
model,
sample_rate=0.01,
alphas=[10, 100],
noise_multiplier=1.3,
max_grad_norm=1.0,
)
privacy_engine.attach(optimizer) |
privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
module=model,
optimizer=optimizer,
data_loader=data_loader,
noise_multiplier=1.1,
max_grad_norm=1.0,
) |
To avoid mutually exclusive method parameters, we're now providing separate method to initialize training loop if epsilon is to be provided instead of noise_multiplier
model, optimizer, data_loader = privacy_engine.make_private_with_epsilon(
module=model,
optimizer=optimizer,
data_loader=data_loader,
epochs=EPOCHS,
target_epsilon=EPSILON,
target_delta=DELTA,
max_grad_norm=MAX_GRAD_NORM,
)Increased focus on data handling
You might have noticed that we are now passing data loader to make_private in addition to module and optimizer. This is intentional. Batch sampling is an important component of DP-SGD (e.g. privacy accounting relies on amplification by sampling) and Poisson sampling is quite tricky to get right, so now Opacus takes control of three PyTorch training objects: model, optimizer, and data loader.
More modularised components
This release makes more functionalities modular, allowing for easy extensibility, while embracing cleaner semantics:
- model is wrapped with
GradSampleModule, which computes per sample gradients. - optimizer is wrapped with
DPOptimizer, which does gradient clipping and noise addition. - data loader is transformed to a
DPDataLoader, which performs uniform-with-replacement batch sampling, as required by privacy accountant. - Module validation and fix follows the same pattern as
GradSampleModuleresulting in compartmentalized validation code that is easily extensible and over-rideable.
Privacy analysis
Privacy analysis functions are now promoted into an Accounant class allowing for a more generic API. This has already allowed us to implement two accountants: RDP (default and recommended one) and Gaussian DP accountant; and will enable you to add more without having to worry about messing with the core library.
- eps, alpha = privacy_engine.get_privacy_spent(delta=target_delta)
+ eps = privacy_engine.get_epsilon(delta=target_delta)Working around device memory
Training with Opacus consumes more memory as it needs to keep track of per-sample gradients. Opacus 0.x featured the concept of virtual steps - you could decouple the logical batch size (that defined how often model weights are updated and how much DP noise is added) and physical batch size (that defined the maximum physical batch size processed by the model at any one time). While the concept is extremely useful, it suffers from serious flaws when used with Poisson sampling. Opacus 1.0 introduces a BatchMemoryManager for your dataloader, which takes care of the logical and physical batch sizes internally.
Dynamic privacy parameters
Opacus now supports changes to the privacy parameters during training, and adjusts the privacy accounting accordingly.
Use various schedulers provided in opacus.scheduler module to adjust the amount of noise during training (the implementation mimics the interface of lr_schedulers).
For all the other parameters Opacus supports subsequent calls to make_private method, while maintaining consistent privacy accounting.
Designed to be extensible
Opacus 1.0 is designed to be flexible and extensible.
GradSampleModulesupports user-provided grad samplers for custom modules.DPOptimizercan easily be extended with additional or alternative functionality.- Support for user-provided privacy accountants via optimizer hooks.
- Support for custom model validation rules.
PEP 3102
Almost all functions are now PEP 3102 compliant; meaning they only accept keyword arguments. You no longer have to memorize or be confused by the position of the arguments to be passed to a functions. This also makes the API future proof as adding non-default arguments becomes easier.
Lightning Support
Now you can add DP training to PyTorch Ligthning code. The lightning framework allows you to make the code cleaner and avoid boilerplate; simply add make_private call to configure_optimizers() method of your LightningModel. A Lightning version of MNIST task is available as a guide at examples/mnist_lightning.py.
Tutorials
We have updated all the existing tutorials and also added some new tutorials to aid migration. While the changes to the library has been significant, we expect user facing changes to be minimal and simple. Please feel free to reach out to us on our forum if you need help.
New features and bug fixes
We have also added new features and fixed some bugs along the way. Some of the notable ones are: