Dashboard

Live overview of crawl progress, queue health and latest activity.

never

Pending — awaiting claim

Processing — in flight

Completed — rows_inserted > 0

Failed — terminal error

Run queue

Live Trigger.dev queue — EXECUTING at the top, QUEUED below in pickup order (oldest first, since concurrencyLimit = 1).

#	Task	Status	Queued at	Started	Origin
Loading…

Recent runs

Latest Trigger.dev runs across this project — click any to inspect its crawl trace.

Task	Status	Started	Origin
Loading…

System

Connection state for the services this UI talks to.

Trigger.dev—

ClickHouse (flights)—

ClickHouse (otel)—

HyperDX—

Serverv—

Trigger

Fire a Trigger.dev task or kick off the queue worker. All inputs are validated against the DB and config.

Single origin Full scan Seed & run queue Run one pending item Sync Ryanair routes

Crawl one Ryanair origin Trigger.dev

Loading…

Origin IATA * Type a 3-letter code — autocomplete from the routes table.

Date from *

Date to *

Adults

Destinations filter Comma-separated. Leave empty to use every destination in airline_routes for the origin.

Run ID (optional)

Set this to correlate the run's trace with a known crawl_run_id.

Pacing

Request delay (ms)

Jitter (ms)

Cooldown (ms)

// No run yet.

Full Ryanair scan Trigger.dev

Loading…

Fan-out across every origin in airline_routes (or a subset you pick). Uses the FARFND /cheapestPerDay endpoint — 1 HTTP call per (origin, destination).

Date from *

Date to *

Adults

Run ID

Origins override Empty = every origin in airline_routes. Ctrl/Cmd-click to pick specific ones.

Destinations filter

Pacing

Request delay (ms)

Jitter (ms)

Cooldown (ms)

// No run yet.

Seed & run crawl queue Trigger.dev

Inserts one pending row per (origin, destination) into crawl_progress and immediately fires the crawl-queue-worker task, which loops at concurrency 1.

Airline *

Origins * Pick one or more origins from the routes table.

Date from *

Date to *

Adults

Worker max iterations

Pacing

// No run yet.

Resume worker on an existing run Trigger.dev

Triggers the crawl-queue-worker task with an existing crawlRunId. No re-seeding — the worker only drains pending rows already stamped with that run id (plus any legacy empty-id rows as a fallback). Concurrency=1 + the airline Pacer keep the effective rate unchanged.

Airline *

Existing run id (UUID) *

Max iterations

requestDelayMs

requestJitterMs

// No run yet.

Run one pending queue item Trigger.dev

Loading…

Airline *

Origin *

Destination *

Date from *

Date to *

Force (steal from worker)

Origin	Destination	From	To
Click "Refresh pending list" to load.

// No run yet.

Sync Ryanair routes Trigger.dev

Loading…

Source *

Concurrency

Origins override Empty = full list.

// No run yet.

Queue

Crawl progress ledger — what is pending, in flight, completed and failed. Backed by crawl_progress_latest.

Pending Processing Completed Failed

Origin	Destination	From	To	Inserted / Started / Completed	Rows	Trace
Loading…

Traces

Resolve a crawl run / Trigger run into the OTel trace id, then jump straight to HyperDX.

Open HyperDX ↗

Resolve a run → trace

Paste a crawl_run_id (UUID) or a Trigger.dev run id to fetch every log & span recorded for that trace, plus a direct HyperDX link.

crawl_run_id / Trigger run id UUIDs are converted to the 32-char trace id used by ClickHouse.

Time window

Trace summary

Trace id—

crawl_run_id—

Spans0

Logs0

Open in HyperDX ↗

Spans

All otel_traces rows tagged with this trace id, in order.

Span	Kind	Duration	Status
No spans.

Logs timeline

All otel_logs rows for the trace, top-to-bottom in time.

No logs.

Recent traces in ClickHouse

Most recent OTel traces observed in otel.otel_traces. Click any to copy its trace id.

Started	Span count	Root span	Service	Trace id
Click "Show recent traces" to load.

Maintenance

ClickHouse jobs, backfills and one-shot scripts.

Backfill crawl progress

Targets crawl_progress rows with status='completed' AND rows_inserted=0 and writes a corrected row (ReplacingMergeTree collapses it).

Mode *

Dry run (rows mode only)

Error message (mark-failed mode)

// No run yet.

Inspect / requeue

List / inspect / requeue destinations in crawl_progress.

Mode

Airline

Origin *

Date from *

Date to *

Destinations filter

Include completed (requeue only)

// No run yet.

One-shot scripts

Maintenance CLI scripts that don't return a Trigger run id.

// No run yet.