4 GitHub Actions Patterns for Monorepo ETL
Running three sites from one monorepo creates problems. You face three separate ETL jobs, three content rebuilds, and three deployment pipelines. If you do not coordinate them, they collide.
I spent six weeks testing schedules to stabilize this setup. Here are the four patterns I use.
- Use Time Offsets for Cron Jobs
Running all ETL jobs at the same time causes failures. Jobs compete for API rate limits. When HuggingFace or GitHub returns a 429 error, the run fails.
I use 30-minute offsets to prevent this.
- Job one starts at 02:00
- Job two starts at 02:30
- Job three starts at 03:00
Thirty minutes is enough time to finish a heavy pull before the next job starts. This keeps your logs clean and prevents API collisions.
- Use Skip Flags to Stop Unnecessary Rebuilds
Every ETL job ends with a Vercel deployment. A problem arises when article commits also trigger rebuilds. Without a plan, every article update forces all three sites to rebuild. This wastes build minutes.
I use a [skip publish-articles] tag in ETL commit messages.
I added a step in my workflow to check for this flag. If the tag exists, the workflow skips the build and deploy step. This keeps ETL pipelines separate from article pipelines.
- Use Path Filters to Target Specific Sites
You do not want one site's update to trigger three deployments. I configure each site's workflow to watch only its own folder and the shared packages folder.
Example logic:
- AI tools site only watches apps/ai-tools/
- OSS site only watches apps/oss-alternatives/
If you change code in the shared directory, all sites will rebuild. I accept this trade-off because shared code changes are rare.
- Add Manual Dispatch for Control
Cron handles the daily routine. Manual dispatch handles everything else. I use workflow_dispatch so I can:
- Re-run a failed ETL job
- Force a refresh after adding data
- Run a dry run to test data without writing to the database
The GitHub UI shows a dropdown menu for these choices. This prevents typos and makes debugging easy.
Summary
These patterns are not complex. They are explicit. I keep a simple markdown file in my repository to document these rules. This ensures I do not break the system when I add new workflows.