In Salesforce, batch jobs are a cornerstone for processing large datasets efficiently. However, many use cases require multiple batch jobs to run sequentially, with dependencies between them. This can include scenarios like complex data migrations or chained data processing tasks.
The challenge is that Salesforce doesn’t natively support chaining batch jobs in a flexible and maintainable way. To address this, I developed the ChainableBatchJob – a solution for orchestrating batch jobs with ease while adding features like error handling, delayed execution, and notifications.
The Problem with Independent Batch Jobs
Today, chaining batch jobs often involves hard-coding the next job to start at the end of the previous one. For example:
While this approach works, it’s inflexible for testing and troubleshooting. You can’t easily run or debug a single job in isolation without modifying the code. Additionally, handling errors or introducing conditions for the next job in the chain becomes cumbersome. These limitations make it difficult to adapt to new requirements or maintain complex processes.
Introducing ChainableBatchJob
The ChainableBatchJob solves these issues by decoupling job chaining logic from individual batch jobs. This provides a centralized way to manage and configure job chains dynamically, without requiring hard-coded dependencies.
Key Features
- Fluent Interface for Configuration: Easily set up jobs and chain them using a readable, fluent API.
- Error Handling: Decide whether to continue the chain if a job fails.
- Delay Execution: Schedule jobs to run after a specific delay.
- Notifications: Receive emails when a job completes.
- Reusability: Run jobs independently or as part of a chain.
How It Works
Core Components
- ChainableBatchJob: The orchestrator that manages the sequence of jobs.
- ChainableJob: An abstract base class wrapping Database.Batchable for individual batch jobs, extended to define specific tasks.
- YourJob: The link in the chain, where you build what needs to be done. Feels identical to writing your own Batchable.
Execution Flow
Initialize the chain, add the links/jobs, then start processing:
It handles starting each job in sequence, managing failures, and notifying stakeholders as configured.
While building chain elements, I was able to start the chain as Anonymous Apex, choosing which Job(s) I wanted to run and debug.
Configuring Chain Properties
One of the features of the ChainableBatchJob is the ability to configure properties at the chain level. These properties, such as batch size or delay, apply to every job in the chain by default but can be overridden for specific jobs as needed.
Setting Chain Properties
For example, you can configure the chain to use a batch size of 2000 for all jobs:
Overriding Properties for Specific Jobs
If one of the jobs in the chain is particularly complex and requires a smaller batch size, you can override the chain’s batch size for that job:
This flexibility ensures that the chain remains efficient while accommodating the specific requirements of each job.
Real-World Use Case: Ad-Hoc Data Migrations
One practical application of the ChainableJob was in a project involving data migrations. Each migration step involved different data objects and transformations, making a hard-coded approach impractical. Using ChainableBatchJob, we can:
- Configure each migration step as a ChainableJob.
- Set custom filters and attributes for each step.
- Monitored progress and received notifications upon completion.
This approach reduced the time spent on debugging and ensured consistent execution across all migration steps.
One of the properties we created gave us the ability to provide a SOQL addition, allowing us to inject an AND clause into our batch processing queries. This allowed us to adjust after product deployment, and allowed us to create 5 slices of data, and 5 chains of batch jobs that all ran in parallel to greatly reduce the overall time for that particularly large dataset.
While data migrations are a primary use case, this approach can be applied to sequential data processing (e.g., ETL workflows), large-scale data cleanup, or anything else!
Getting Started
- Define Your Jobs: Extend ChainableJob to create custom batch jobs.
- Configure the Chain: Use ChainableBatchJob to set up and manage the sequence.
- Run and Monitor: Start the chain and monitor its execution using logs and notifications.
Challenges and Future Enhancements
While the ChainableBatchJob addresses many limitations of traditional batch job chaining, there are areas for improvement:
- Conditional Branching: Adding support for dynamic job execution based on conditions.
- Visualization: Currently this is built with the developer as the “end user”. Providing a UI to view and manage job chains.
- Job Status: It might be nice storing additional details about the job executions themselves in a Custom Object, making it easier to report and possibly resuming a failed job after corrective action is taken.
Conclusion
The ChainableBatchJob makes chaining Salesforce Apex batch jobs more flexible, maintainable, and powerful. Whether you’re handling data migrations or building complex workflows, it’s a tool that simplifies automation and reduces technical debt.
Learn More About Partnering With Us!
Learn More About Partnering With Us!
