Article: Apache DolphinScheduler 3.x flips the architecture on its head. Teams still on 1.3 must re-wire their cluster layout and rewrite the database schema before they can upgrade. The old single-node master disappears, replaced by a plug-in-driven, decentralized system that schedules, stores and logs jobs differently.

Why the jump matters

Version 1.3 clung to a centralized master that handled every scheduling decision and offered only twelve built-in task types. 3.x swaps that for a microkernel that loads task types and storage adapters as plugins, and it eliminates the single point of failure by letting masters and workers talk through a registry service. Operators gain scalability and resilience; developers extend the scheduler simply by dropping a JAR instead of hacking core code.

The architectural overhaul

  • Microkernel plug-in system – Task definitions, resource handlers and custom health checks now live in separate modules. Add a new task type by placing its plug-in on the classpath and restarting the affected nodes.
  • Decentralized coordination – Masters and workers discover each other via a registry. The registry can be ZooKeeper (the historic default), a JDBC-backed relational database, or an Etcd cluster. Pick the technology that matches your existing stack.
  • Expanded task catalog – Built-in tasks have grown from 12 to over 30, covering cloud-native workloads and machine-learning pipelines.
  • MasterServer vs WorkerServer – The MasterServer now handles DAG partitioning, submission and health monitoring. The WorkerServer acts as a pure execution engine that also streams logs. This split clarifies responsibilities and lets you size each layer independently.
  • Fault tolerance via Watcher – The Watcher watches the registry for node failures. When a Master or Worker drops, the registry triggers an automatic failover.
  • gRPC log transport – Remote log retrieval moved from a Netty-based protocol to gRPC, which the source says delivers better performance.

Database refactoring you can’t ignore

The schema overhaul is the most concrete blocker for any upgrade:

1.3 Table 3.x Table What changed
t_ds_process_definition t_ds_workflow_definition The term “process” was renamed to “workflow” to match the UI and API vocabulary.
t_ds_process_instance t_ds_workflow_instance Same semantic shift for runtime records.

Beyond renaming, 3.x extracts task metadata that previously lived inside JSON blobs into dedicated relational tables, making data management cleaner.

Migration checklist

  1. Back up everything – Export the full 1.3 database dump and copy the conf directory.
  2. Map old tables to new names – Run a script that renames t_ds_process_definition to t_ds_workflow_definition and t_ds_process_instance to t_ds_workflow_instance. Verify foreign-key constraints afterward.
  3. Migrate JSON task fields – Copy JSON-encoded task data into the new relational tables. Test a handful of DAGs to confirm the scheduler reads the new layout.
  4. Choose a registry – Keep ZooKeeper if you already run it; otherwise set up a JDBC-backed registry or Etcd cluster and point all nodes to the new address.
  5. Deploy plug-ins – Package any custom task types you used in 1.3 as 3.x-compatible plug-ins and deploy them to each MasterServer.
  6. Roll out MasterServer first – Start a fresh MasterServer instance pointing at the migrated database. Verify it lists existing workflows and that the health dashboard shows no errors.
  7. Add WorkerServers – Bring up WorkerServers one at a time. Watch the registry for successful registration and confirm that logs flow over gRPC.
  8. Run smoke tests – Trigger a few low-risk DAGs covering the most common task types. Check that logs appear in the UI and that task status updates propagate correctly.
  9. Monitor for failover – Simulate a MasterServer crash and watch the Watcher promote a standby. Confirm that in-flight tasks continue without manual restart.

What could bite you

The new plug-in model is powerful, but it raises the bar for custom development.

Bottom line: Moving from 1.3 to 3.x is not a simple version bump; it requires a coordinated database rename, JSON-to-relational migration, and a re-architecture of your cluster around a plug-in-enabled, registry-driven model. Follow the checklist step-by-step, test early, and you’ll gain a scheduler that scales out, recovers automatically, and speaks the same protocol as modern cloud services.