DocsWebhooks
Webhooks
Receive real-time notifications when jobs complete or fail.
Webhooks allow SnowScrape to send HTTP POST requests to your server when specific events occur. This is useful for triggering workflows, updating databases, or sending notifications.
Supported Events
| Event | Description |
|---|---|
| job.completed | Job finished successfully |
| job.failed | Job execution failed |
| job.started | Job started executing |
| job.paused | Job was paused |
Webhook Payload
All webhook requests include a JSON payload with event details:
{
"event": "job.completed",
"timestamp": "2024-01-20T15:30:00Z",
"data": {
"job_id": "job_abc123",
"job_name": "Product Prices",
"status": "success",
"results_count": 150,
"duration_ms": 45000,
"download_url": "https://api.snowscrape.com/jobs/job_abc123/download"
}
}Failure Payload
{
"event": "job.failed",
"timestamp": "2024-01-20T15:30:00Z",
"data": {
"job_id": "job_abc123",
"job_name": "Product Prices",
"status": "failed",
"error": {
"code": "TIMEOUT",
"message": "Request timeout after 30000ms",
"failed_urls": 12,
"successful_urls": 138
}
}
}Setting Up Webhooks
Via Dashboard
- Go to Webhooks in the sidebar
- Click Create Webhook
- Enter your endpoint URL
- Select which events to subscribe to
- Save and test the webhook
Via API
POST /webhooks
{
"url": "https://your-server.com/webhook",
"events": ["job.completed", "job.failed"],
"secret": "your_webhook_secret"
}Verifying Webhooks
Each webhook request includes a signature header for verification:
X-SnowScrape-Signature: sha256=abc123...Verification Code (Node.js)
const crypto = require('crypto');
function verifyWebhook(payload, signature, secret) {
const expected = 'sha256=' + crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expected)
);
}Verification Code (Python)
import hmac
import hashlib
def verify_webhook(payload: bytes, signature: str, secret: str) -> bool:
expected = 'sha256=' + hmac.new(
secret.encode(),
payload,
hashlib.sha256
).hexdigest()
return hmac.compare_digest(signature, expected)Retry Policy
SnowScrape retries failed webhook deliveries with exponential backoff:
- First retry: 1 minute after failure
- Second retry: 5 minutes
- Third retry: 30 minutes
- Fourth retry: 2 hours
- Fifth retry (final): 24 hours
Webhooks are considered successful if your server returns a 2xx status code within 10 seconds.
Best Practices
- Respond quickly - Return 200 immediately, process async
- Handle duplicates - Use the event ID for idempotency
- Verify signatures - Always validate the webhook origin
- Use HTTPS - Webhook URLs must use HTTPS in production
- Monitor failures - Check the webhook logs for delivery issues
Security Note
Never expose your webhook secret. Store it securely in environment variables and regenerate it if compromised.