Scaling Houdini Simulations with PDG on the Cloud

Houdini's PDG (Procedural Dependency Graph) changed how we think about production workflows. Chain together wedges, simulations, renders, and post-processing into a single executable graph. Hit "Cook" and let the system figure out the rest.

There's just one problem: your workstation has 64 cores. Your PDG graph has 50,000 work items.

The math doesn't work. A 1000-frame render across 48 wedge variations generates 48,000 individual tasks. Even on high-end hardware, you're looking at days of processing; with your machine completely locked up the entire time.

This is exactly what render farms are for. But most cloud services don't understand PDG. They expect you to:

• Manually split your graph into separate submissions
• Export work items and manage dependencies yourself
• Run PDG locally and only offload individual renders
• Babysit the process, submitting each stage after the previous completes

We built something different.

Drop & Render's solution

With Drop & Render, you submit your entire PDG graph in a single click. We handle everything else:

wedge1 (48 variations)
     │
ropfetch1 (240 frames per variation = 11,520 renders)
     │
waitforall1
     │
imagemagick1 (composite each variation)
     │
ffmpegencodevideo1 (encode final video)
     │
[Drop & Render Node] (submit to farm)

That 11,520-frame render? It fans out across hundreds of machines simultaneously. What would take a week locally finishes in hours.

Part 1: Automatic Dependency Detection

We Understand Your Graph

PDG graphs aren't just lists of tasks; they're complex webs of dependencies. A render can't start until its wedge values are set. A video can't encode until all frames are rendered. A composite can't run until its inputs exist.

When you connect our node to your graph, we analyze the entire upstream network. We see every connection, every dependency, every sync point. This isn't file-path scanning or string matching; we understand the actual PDG structure.

The result: your graph's dependencies become job dependencies on the farm. Renders wait for wedges. Composites wait for renders. Videos wait for composites. All automatic.

Sync Points Just Work

PDG's waitforall nodes are natural boundaries in your workflow. We recognize these and translate them directly into our job system. Everything upstream of a sync point must complete before anything downstream begins.

No manual job sequencing. No watching dashboards. Your graph's logic is preserved exactly.

Part 2: Intelligent Parallelization

Thousands of Tasks, Hundreds of Machines

A single PDG node might generate 10,000 work items. We don't submit these as 10,000 separate jobs (startup overhead would kill you) or as one giant job (wastes the farm's parallelism).

Instead, we group work items into optimally-sized chunks that balance:

• Machine utilization: Keep every node busy
• Startup efficiency: Amortize scene load time across multiple work items
• Progress granularity: Get meaningful updates, not just 0% → 100%

Your 48,000 work items might become 6,000 parallel subjobs, each processing a small batch. All 6,000 can run simultaneously if the farm has capacity.

Sequential vs. Parallel: We Know the Difference

Not everything can be parallelized:

We detect this automatically based on node type and settings. Simulations run on a single powerful machine from start to finish. Independent renders fan out across the entire farm. You don't configure this; we figure it out.

Part 3: Smart Hardware Routing

GPU Tasks to GPU Machines

Rendering an EXR sequence with Redshift? That needs GPU power. Running a Python script to rename files? That's CPU work.

We analyze your work items and route them to the appropriate hardware:

• GPU rendering: Redshift, Karma XPU, Octane → GPU nodes
• CPU simulation: FLIP, Vellum, RBD → High-core-count CPU nodes
• General processing: File ops, scripts, compositing → Cost-effective CPU pools

This happens automatically. GPU-intensive work gets GPU machines. Everything else runs on cheaper infrastructure. You get optimal performance without overpaying.

Part 3: Preserving Wedge Variations

Your Parameters Travel With Your Work

PDG's power comes from procedural variation. A wedge node might set color_variation, scale_factor, and material_preset across dozens of combinations. Downstream nodes read these values to produce unique outputs.

On a local machine, this state lives in memory. On a cloud farm, each job starts fresh on a different machine.

We solve this transparently. When a wedge job completes, its parameter values are preserved. When the downstream render jobs start; potentially on completely different machines; those values are restored. Your renders see exactly the same wedge attributes they would locally.

This works across any number of chained wedges and any depth of downstream processing. Your graph's procedural logic is fully preserved.

Part 4: Real-Time Progress and Downloads

Watch Your Renders Arrive

Traditional farm workflow: submit, wait hours, download everything at the end. For a 10-hour job, that's 10 hours of zero visibility.

We do it differently:

Frame 1 completes on farm -> Downloads immediately
Frame 2 completes on farm -> Downloads immediately
    ... (continues in parallel with rendering)
Frame 11,520 completes on farm -> Downloads immediately

As each work item finishes, its output streams back to your machine. You can review frames while the job is still running. Spot a problem early? Cancel and iterate, don't wait for 10,000 broken frames.

Per-Node Download Control

Maybe you're rendering 48 wedge variations but only care about the final composites. Or you want the EXR sequences but not the intermediate caches.

You control downloads per-node:

Intermediate data stays on the farm. Final outputs come to you. This saves bandwidth and local storage while keeping everything accessible for dependent jobs.

Part 5: Files Land Where You Expect

No Path Wrangling

Your Houdini scene has paths like:

$HIP/render/beauty.$F4.exr
$HIP/cache/sim.$F4.bgeo.sc

When outputs download, they go to exactly those locations. Not some farm-specific folder structure. Not a downloads directory you have to sort through. The actual paths your project expects.

This means:

• Your comp scripts work immediately
• Your preview viewers find the frames
• Your version control sees the right structure
• Your delivery pipeline continues unchanged

The farm is invisible. Files appear as if you rendered locally; just much, much faster.

Part 6: Skip Downloads, Keep Caches on Farm

The Multi-Terabyte Workflow

Sometimes your PDG graph generates massive intermediate data:

Downloading 3.2 TB just to feed it back into the next job? That's bandwidth torture.

The "Keep on Farm" Option

For any node, you can disable downloads. The outputs stay on our storage:

1. Simulation job completes → 2 TB stays on farm
2. Meshing job starts → Uses simulation data directly (no transfer)
3. Render job starts → Uses meshes directly (no transfer)
4. Final frames download → 50 GB to your machine

You only transfer what you actually need locally. Everything else moves at datacenter speeds between farm machines.

Reconnecting Farm Assets

The next day, you want to render a new camera angle using yesterday's simulation. Instead of re-uploading terabytes:

1. Open our "Add Synced Asset" browser
2. Select the simulation cache from farm storage
3. Submit your new render job

The cache is already there. Your new job starts immediately with full access to the previous data.

Supported Workflows

Our PDG support covers the full spectrum of production tasks:

If PDG can cook it, we can farm it.

A Real Production Example

The Scenario

You're creating a product visualization with 24 color variations, 3 lighting setups, and 4 camera angles. Each combination renders at 4K over 120 frames.

24 colors × 3 lights × 4 angles × 120 frames = 34,560 frames

Local machine

• Render time per frame: 3 minutes
• Total render time: 72 days
• Your workstation: completely unusable for 10 weeks

With Drop & Render

1. Build your PDG graph with wedges for color, lighting, and camera
2. Connect our node at the end
3. Click submit

The job fans out across the farm. All 34,560 frames render in parallel across hundreds of GPUs. Total wall-clock time: 4-6 hours.

Your workstation stays free. You review results as they stream in. By lunch, you have all 288 variations rendered and ready for client review.

Getting Started

It's Just One Node

1. Install our plugin: Adds the Drop & Render TOP node
2. Connect it: Wire it to the end of any chain you want to farm
3. Run scene check: We verify everything will work
4. Adjust settings: Choose which outputs to download
5. Submit: That's it

Your graph cooks on the farm exactly as it would locally, just massively parallelized.

What We Handle

• Dependency ordering across all jobs
• Work item chunking for optimal parallelism
• GPU vs CPU routing
• State preservation between stages
• Real-time output streaming
• Path translation between platforms
• File deduplication via our CAS system

What You Don't Do

• Split your graph manually
• Manage job dependencies
• Configure hardware allocation
• Worry about path differences
• Wait for everything to finish before downloading
• Re-upload unchanged assets

Conclusion

PDG is Houdini's answer to production-scale procedural workflows. But it's only as powerful as the hardware running it.

Drop & Render turns your workstation PDG into datacenter PDG:

• One-click submission: Your entire graph, not just pieces
• Automatic parallelization: Thousands of tasks across hundreds of machines
• Preserved dependencies: Your graph logic runs exactly as designed
• Real-time streaming: Watch outputs arrive, don't wait for completion
• Smart routing: GPU work to GPUs, CPU work to CPUs

The result: workflows that would take weeks locally complete in hours. Your machine stays free. Your deadlines get met.

Ready to scale your PDG workflows?

Get started with Drop & Render

Questions? Our team is here to help; reach out through our support page.