Introducing Scheduled Scraping Workflows
Automate recurring web scraping with AgentQL's new experimental Scheduled Workflows feature. Download JSON data, streamline workflows, and build your own always-on assistant for regular data collection and research.
AgentQL now supports scheduled scraping jobs! 🎉
You told us scraping the same sites over and over was a hassle—especially when paired with manual scheduling tools. We heard you loud and clear. We created an experimental scheduling feature to give you a simpler, more intuitive way to automate recurring jobs.
We built this experimental feature for developers who regularly scrape the same websites for updated information. Instead of juggling custom scripts and cron jobs, you can now set up and automate these tasks directly from the AgentQL Dev Portal. And here’s the best part: you can download the scraped data as JSON, so it’s ready to drop into your workflows.
If your work involves tracking updates on dynamic sites—like news headlines, product prices, or public datasets—this feature is your new best friend. Think of it as your always-on assistant for regular data collection.
Get started scheduling
Head over to the new scheduling page on our Dev Portal and use the Add New Workflow button.
Set the URL(s) for the pages that you'd like to extract data from and the AgentQL query you'd like to run, and specify when you'd like it to run.
The full Step-by-Step Guide to scheduling scraping jobs is available in our documentation.
Because this feature is still experimental, we're limiting the number of scraping jobs you can schedule to:
- 2 workflows per user
- 5 urls per workflow
- 10 runs per workflow
If you need more scraping jobs, please reach out to us.
We’re excited to see how you use Scheduled Scraping to streamline your workflows. This is just the start—try it out, and let us know how we can make it even better on our community Discord. Your input will shape what comes next!
Happy scheduling! 🐟
—The Tiny Fish Team Building AgentQL