Program Ensembles and Workflows

A ProgramEnsemble runs multiple quantum programs in parallel — handling scheduling, circuit batching, progress tracking, and result aggregation. Typical use cases include parameter sweeps, molecular dissociation curves, problem decomposition, and algorithm comparison.

Built-in Ensemble Workflows

Divi provides several ready-made ensemble workflows. Each is covered in detail on its own page — this section gives a quick overview and links.

VQE Hyperparameter Sweeps

VQEHyperparameterSweep runs VQE across multiple molecular configurations (bond lengths, ansätze) in parallel. See Ground-State Energy Estimation with VQE for configuration and examples.

Time Evolution Trajectories

TimeEvolutionTrajectory runs one time-evolution program per time point and collects expectation values into a trajectory. See Hamiltonian Time Evolution for full details.

Problem Decomposition (Graph / QUBO / Matching)

PartitioningProgramEnsemble decomposes a large QAOAProblem into sub-problems, solves each partition with QAOA (or PCE / IterativeQAOA), and stitches the per-partition results into a global solution using a configurable aggregation strategy (see Aggregation Strategies). Graph, QUBO, and matching partitioning are all covered in Combinatorial Optimization with QAOA and PCE.

Aggregation Strategies

After each partition is solved, its quantum program returns several candidate bitstrings ranked by probability. An aggregation strategy stitches these per-partition candidates into a single global solution — the choice of strategy controls the quality/cost trade-off across the full problem. aggregate_results and get_top_solutions accept a strategy — an AggregationStrategy — and default to BeamSearchStrategy.

Hierarchical aggregation

HierarchicalStrategy is a divide-and-conquer alternative: partitions are split into groups of group_size, each group is solved independently with a beam of width max_per_group, and the resulting group pools are combined in a pairwise merge tree. Deferring cross-group commitment lets each group keep prefixes that a single left-to-right beam would prune early. The trade-off is that interactions spanning partitions in different groups are only scored at merge time, not during per-group pruning.

The knobs trade solution quality against cost:

  • group_size — partitions per group (larger groups defer more commitment).

  • k_per_partition — candidates fetched from each partition.

  • max_per_group — solutions kept per group and per merge level; the main quality/cost dial.

  • merge_width — limits how many solutions from each group are paired during a merge step, capping each merge’s cost to merge_width squared. Lower it to tame cost on problems with many partitions; the default (None) uses all max_per_group entries.

Unlike beam search, top_n does not widen the search for the hierarchical strategy — it only sets how many of the final solutions are returned. Calling get_top_solutions() with a larger n never inflates the search cost.

from divi.qprog import HierarchicalStrategy

solution, energy = qaoa_partition.aggregate_results(
    strategy=HierarchicalStrategy(
        group_size=4, k_per_partition=20, max_per_group=200, merge_width=50
    )
)

When to use hierarchical aggregation

  • Many partitions with localized coupling: when strongly-coupled partitions can land in the same group, each group explores more alternatives before committing to a cross-group assignment — recovering combinations a greedy left-to-right beam would prune early.

  • Finer cost control: merge_width caps per-merge cost independently of max_per_group, a knob beam search does not offer.

  • Prefer beam search when coupling is global or unpredictable across partition boundaries: groups are formed in partition order, so the grouping then provides little benefit over a wider beam.

Top-N Solutions

PartitioningProgramEnsemble exposes a get_top_solutions method that returns multiple ranked global solutions. It accepts the same strategy argument.

from divi.qprog import BeamSearchStrategy

top_solutions = qaoa_partition.get_top_solutions(
    n=5, strategy=BeamSearchStrategy(beam_width=5, n_partition_candidates=10)
)

# Return type is problem-dependent:
#   Graph  → list[(node_indices, energy)]
#   QUBO   → list[(solution_array, energy)]
for rank, (solution, energy) in enumerate(top_solutions, 1):
    print(f"{rank}. Energy: {energy:.6f}, Solution: {solution}")

This is useful when you want to inspect alternative solutions or post-process candidates with domain-specific constraints. Beam search widens its beam to at least n when needed; hierarchical returns the n best from its final merged pool (without widening the search, as noted above).

For constrained problems such as maximum-weight matching, partition boundaries can produce globally invalid raw candidates even when each partition candidate is locally valid. aggregate_results keeps the default forgiving behavior and repairs matching conflicts. To inspect only raw candidates that are already valid globally, use get_top_solutions(..., strict=True). The returned list may contain fewer than n entries.

Sampling from Pre-Trained Parameters

sample_solution() mirrors the standalone sample_solution() across every sub-program in one call. Use it when you already have trained parameters for each partition (e.g. from a prior run(), a loaded checkpoint, or an external training routine) and only need to re-sample — no EXPECTATION jobs are dispatched.

Two usage paths:

  • params_per_program=None — each sub-program uses its own _best_params. After a prior run() on the same ensemble, just call ensemble.sample_solution(blocking=True).

  • params_per_program={program_id: params, ...} — pass explicit per-partition parameters. Unknown program IDs raise ValueError; program IDs present in the ensemble but missing from the dict fall back to that program’s own _best_params and emit a single UserWarning listing all fallbacks (silence with suppress_strict_warning=True).

# Re-sample a previously trained partitioning ensemble and aggregate
# the global solution — without re-paying for the optimizer.
ensemble.sample_solution(blocking=True)
solution, energy = ensemble.aggregate_results(
    strategy=BeamSearchStrategy(beam_width=3)
)

# Or: bring trained parameters in from elsewhere (per partition).
ensemble.sample_solution(
    params_per_program={pid: params[pid] for pid in pids},
    blocking=True,
)

The full lifecycle infrastructure (executor pool, merged batching, progress UI, cancellation, blocking / non-blocking semantics) is shared with run(). No sub-program mutates its own optimizer-side state (best_params, losses_history, current_iteration).

Custom Ensemble Workflows

You can create custom program ensemble workflows by inheriting from ProgramEnsemble:

Custom Ensemble Implementation

from divi.qprog import ProgramEnsemble, VQE
from divi.qprog.optimizers import MonteCarloOptimizer
from divi.backends import CircuitRunner, MaestroSimulator
import pennylane as qp
import numpy as np

class CustomParameterSweep(ProgramEnsemble):
    def __init__(self, backend: CircuitRunner, molecules):
        super().__init__(backend)
        self.molecules = molecules

    def create_programs(self):
        """Generate one VQE program per molecule."""
        super().create_programs()
        for i, mol in enumerate(self.molecules):
            vqe = VQE(
                molecule=mol,
                optimizer=MonteCarloOptimizer(),
                backend=self.backend,
                max_iterations=10,
            )
            self._programs[f"sweep_{i}"] = vqe

    def aggregate_results(self):
        """Collect and analyze results from all programs"""
        super().aggregate_results()
        results = {}
        for program_id, program in self._programs.items():
            if program.losses_history:  # Check if program completed
                final_loss = program.best_loss
                results[program_id] = {
                    'energy': final_loss,
                    'params': program.best_params,
                    'circuits': program.total_circuit_count
                }
        return results

# Usage
mol1 = qp.qchem.Molecule(symbols=["H", "H"], coordinates=np.array([[0.0, 0.0, 0.0], [0.0, 0.0, 0.74]]))
mol2 = qp.qchem.Molecule(symbols=["Li", "H"], coordinates=np.array([[0.0, 0.0, 0.0], [0.0, 0.0, 1.6]]))
mol3 = qp.qchem.Molecule(symbols=["H", "F"], coordinates=np.array([[0.0, 0.0, 0.0], [0.0, 0.0, 0.92]]))
molecules = [mol1, mol2, mol3]

# Use a local simulator
local_backend = MaestroSimulator()
sweep = CustomParameterSweep(local_backend, molecules)
sweep.create_programs()
sweep.run(blocking=True)

results = sweep.aggregate_results()
print(results)

Progress Monitoring and Control

Divi provides automatic progress tracking for long-running ensembles. When you execute an ensemble that contains compatible programs (like VQE or QAOA), a rich progress display appears in your console showing the status of each program in real-time.

When circuit batching is active (the default), an additional batch status line appears below the per-program progress bars. It shows the merged job’s polling status — how many circuits were merged, which programs are part of the current flush group, and the backend job ID.

Stopping an Ensemble

You can gracefully stop a running ensemble at any time by pressing Ctrl+C. The KeyboardInterrupt is caught during join(), which cancels any in-flight backend jobs and allows currently running programs to finish their current iteration before shutting down.

Circuit Batching

By default, run() merges the circuit submissions from all programs in the ensemble into single backend calls. This behavior is controlled by BatchConfig.

How it works

Each optimization iteration, every program calls submit_circuits on its backend. With batching enabled, these calls are intercepted by a coordinator that:

  1. Collects circuit submissions from all active programs (barrier-based flush).

  2. Merges them into a single payload with namespaced circuit tags.

  3. Submits the merged payload to the real backend in one call.

  4. Polls for results once (instead of N times).

  5. Demultiplexes the results back to each program by tag prefix.

This happens transparently — programs are unaware they’re sharing a backend call.

When to use batching

  • Cloud backends (QoroService): batching reduces the number of API calls, authentication round-trips, and polling loops. This is the primary use case.

  • Local simulators (QiskitSimulator): batching adds synchronization overhead for no network benefit. The simulator already parallelizes circuits internally.

Limiting batch size

By default the coordinator waits for all active programs to submit before merging circuits. For large ensembles this can produce very large merged jobs. Use max_batch_size to cap the number of circuits per flush:

from divi.qprog import BatchConfig

# Flush as soon as 50 circuits are pending (partial flush)
ensemble.run(blocking=True, batch_config=BatchConfig(max_batch_size=50))

When the pending circuit count reaches max_batch_size the coordinator flushes immediately — even if some programs haven’t submitted yet. Those programs will be included in a later flush. This reduces per-job size on the backend and can improve latency for large ensembles.

max_batch_size controls merging granularity, not individual payload size. A single program that submits more circuits than the limit will still flush normally.

Cloud submission with one merged job

When submitting through QoroService, every submit_circuits call costs an HTTP round trip and a scheduler queue slot. A 512-program ensemble batched in chunks of ~14 produces ~37 round trips; merging all 512 into a single job amortizes that to one. Pass max_concurrent_programs=-1 to size the executor pool to the entire ensemble and bypass the default 256-program barrier cap:

from divi.qprog import BatchConfig

# All programs run concurrently -> single merged backend submission.
ensemble.run(
    blocking=True,
    batch_config=BatchConfig(max_concurrent_programs=-1),
)

The -1 sentinel follows the same convention as scikit-learn’s n_jobs=-1 (“use all available”). An explicit positive integer works too, e.g. max_concurrent_programs=512.

For ensembles where each program emits many circuits per call, combine max_concurrent_programs with max_batch_size to bound the merged payload:

ensemble.run(
    batch_config=BatchConfig(
        max_concurrent_programs=20,   # how many programs run at once
        max_batch_size=1024,          # cap circuits per merged call
    ),
)

Explicit values of max_concurrent_programs above 1024 emit a UserWarning — that’s a soft cap meant to flag the most common mistake (reaching for max_concurrent_programs when the user actually wanted max_batch_size). The -1 form is silent because it’s an intentional opt-in.

Disabling batching

Pass BatchConfig(mode=BatchMode.OFF) to disable batching entirely:

from divi.qprog import BatchConfig, BatchMode

# Each program submits circuits independently
ensemble.run(blocking=True, batch_config=BatchConfig(mode=BatchMode.OFF))

# Merged submissions (default)
ensemble.run(blocking=True)

Next Steps