feat(flowlord): Add cross-date ID history

apps/flowlord/README.md

@@ -13,10 +13,26 @@
 - **Alerting** - Slack notifications for failed tasks and incomplete jobs with smart frequency management
 - **RESTful API** - Web UI and API for monitoring workflows, viewing task history, and managing alerts
 
-[![Static Badge](https://img.shields.io/badge/API%20Docs-green)](API.md)
-
 <br clear="all"/>
 
+## Contents
+
+- [Overview](#overview)
+- [Monitoring & Troubleshooting](#monitoring--troubleshooting)
+- [Web Dashboard](#web-dashboard)
+  - [Row actions (Tasks vs Workflow)](#row-actions-tasks-vs-workflow)
+- [Workflow](#workflow)
+  - [Phase](#phase)
+  - [Template](#template)
+- [Rule](#rule)
+  - [cron](#cron)
+  - [files](#files)
+  - [require](#require)
+  - [batch](#batch)
+- [API](#api)
+  - [POST `/backload`](#post-backload)
+  - [POST `/rerun`](#post-rerun)
+
 ## Overview 
 
 ![](flowlord.drawio.svg)
@@ -61,21 +77,20 @@ The tasks and workflow tables both expose a leading action column, but the behav
 
 | View | Row control | What it does |
 |------|-------------|----------------|
-| **Tasks** (`/web/task`) | Rerun (replay icon) | Opens a confirmation modal, then [`POST /rerun`](API.md#post-rerun) to re-queue **that same task** (same ID and info; retry-related meta is stripped and `rerun=manual` is set). Does **not** open the backload form. |
+| **Tasks** (`/web/task`) | Rerun (replay icon) | Opens a confirmation modal, then [`POST /rerun`](#post-rerun) to re-queue **that same task** (same ID and info; retry-related meta is stripped and `rerun=manual` is set). Does **not** open the backload form. |
 | **Workflow** (`/web/workflow`) | Run (play icon) | Navigates to `/web/backload` with query parameters prefilled from the phase row (task type, job, workflow file). Optional `preview=1` triggers a dry-run preview on the backload page. |
 
 Use **backload** when you want to generate tasks from phase rules and templates (single time, date range, or batch). Use **rerun** on the tasks page when you want another attempt of an **existing** task record without leaving the task history view.
 
-**Rerun vs automatic retry:** When a task finishes with `error`, Flowlord may retry it automatically according to the phase `retry` count and `retry_delay` rule—each retry reuses the **same task ID** and info and updates retry metadata (`retry`, `delayed`, etc.). A manual rerun does the same ID/info reuse for operators but clears automatic retry fields, tags `rerun=manual`, and sends immediately without waiting for `retry_delay`. See [POST `/rerun`](API.md#post-rerun) in the API doc.
+**Rerun vs automatic retry:** When a task finishes with `error`, Flowlord may retry it automatically according to the phase `retry` count and `retry_delay` rule—each retry reuses the **same task ID** and info and updates retry metadata (`retry`, `delayed`, etc.). A manual rerun does the same ID/info reuse for operators but clears automatic retry fields, tags `rerun=manual`, and sends immediately without waiting for `retry_delay`. See [POST `/rerun`](#post-rerun).
 
 The shared helper `buildBackloadUrl` in `handler/static/utils.js` is wired on the workflow page only (`enableRowBackloadActions` in `workflow.tmpl`). The tasks page registers `enableRowRerunActions` in `task.js` instead.
 
 | Files View | Tasks View | Alerts View | Workflow View |
 |:----------:|:----------:|:-----------:|:-------------:|
 | [![Files View](../../internal/docs/img/flowlord_files.png)](../../internal/docs/img/flowlord_files.png) | [![Tasks View](../../internal/docs/img/flowlord_tasks.png)](../../internal/docs/img/flowlord_tasks.png) | [![Alerts View](../../internal/docs/img/flowlord_alerts.png)](../../internal/docs/img/flowlord_alerts.png) | [![Workflow View](../../internal/docs/img/flowlord_workflow.png)](../../internal/docs/img/flowlord_workflow.png) |
 
-
-## workflow 
+## Workflow
 A workflow consists of one or more phases as a way to define of how a set of task is to be scheduled and run and the dependencies between them. 
 
 ``` toml 
@@ -137,7 +152,7 @@ The timestamp is derived from the parent task's info string and supports the fol
 | hour  | 2006-01-02T15             | ?hour=2000-01-02T13        | 
 | time  | 2006-01-02T15:04:05Z07:00 | ?time=2000-01-02T13:12:15Z | 
 
-## rule
+## Rule
 defition on how the phase's task/job is to be created 
 
 ### cron 
@@ -208,3 +223,73 @@ rule="cron=0 7 7 * * *&for=-168h&by=day
 template=?day={yyyy}-{mm}-{dd}
 ```
 
+## API
+
+Operations API on `status_port` (default **8080**). Base URL: `http://<host>:<status_port>/`. Set `status_port = 0` to disable HTTP (orchestration only).
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/status` | Liveness check (`ok`). |
+| GET | `/info` | App stats, workflow phase summary, cache timing (JSON). |
+| GET | `/refresh` | Reload workflows from disk, recycle cache retention, sync SQLite backup (JSON). |
+| GET | `/recap` | Today's task recap (JSON if `Accept: application/json`, else plain text). |
+| GET | `/task/{id}` | Task journal for a task ID (all attempts/events). |
+| GET | `/notify` | Slack OK notification with basic app stats (requires Slack config). |
+| POST | `/backload` | Generate tasks from a phase template (dry run or execute). |
+| POST | `/rerun` | Re-queue an existing task attempt from the cache. |
+
+### POST `/backload`
+
+Generate tasks from a workflow phase template. Flowlord looks up the phase by `Task` / `Job` / `Workflow`, takes its `template` (and default meta from the phase `rule` if you omit meta fields), then expands tasks over a single time (`At`) or a range (`From`–`To`) using the `By` iterator.
+
+**Time selection:** use either `At` (one timestamp) or `From` / `To` (range). Accepted formats: `2006-01-02`, `2006-01-02T15`, or RFC3339. If none are set, today is used. `By` defaults to `day` (`hour`, `week`, and `month` are also valid).
+
+**Meta:** provide either `meta` (key → list of values; each combination yields a task) or `meta-file` (JSON lines path), not both. If both are empty, phase rule defaults apply.
+
+**Execute:**
+
+- `Execute` omitted / `false` — dry run; status prefixed with `DRY RUN ONLY:`.
+- `Execute: true` — add generated tasks to cache and send each to the message bus.
+
+Example request (comments for illustration; omit them in real JSON):
+
+```jsonc
+{
+  "Task": "worker",              // phase task / topic (required)
+  "Job": "daily",                // optional job; omit or "" if none
+  "Workflow": "path/to/wf.toml", // optional; disambiguates when the same task exists in multiple files
+  "From": "2024-01-01",          // range start (use with To); formats: date, hour, or RFC3339
+  "To": "2024-01-07",            // range end
+  // "At": "2024-01-15T10",      // alternative to From/To: single timestamp only
+  "By": "day",                   // iterator: day (default), hour, week, month
+  "meta": {                      // optional; each key's values are crossed into tasks
+    "region": ["us", "eu"]
+  },
+  // "meta-file": "gs://bucket/rows.json",  // alternative to meta: one task per JSON line
+  "Execute": false               // false/omit = dry run; true = cache + send to bus
+}
+```
+
+**Response (200):** `{ Status, Count, Tasks }`. Errors: `400` bad input / missing template / meta conflict; `500` send failure in execute mode.
+
+### POST `/rerun`
+
+Re-queue an existing task attempt. Same task ID and `info` as the original (like automatic retry), but operator-driven: does not increment phase `retry`, does not honor `retry_delay`, and does not require a prior failure.
+
+**Body:**
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `type` | yes | Task type (topic). |
+| `job` | no | Job name (empty if none). |
+| `id` | yes | Task ID to reuse. |
+| `created` | yes | `created` timestamp of the cache row to rerun. |
+
+```json
+{ "type": "task1", "job": "", "id": "abc-123", "created": "2024-01-15T10:00:00Z" }
+```
+
+Loads the matching cache row, builds a new outbound task with `task.NewWithID`, strips `retry` / `retried` / `delayed`, sets `rerun=manual`, and sends immediately.
+
+**Response (200):** `{ Status: "Rerun queued", Task }`. Errors: `400` invalid/missing fields; `404` no matching record; `500` bus send failure.
+

apps/flowlord/handler.go

@@ -450,7 +450,8 @@ func (tm *taskMaster) htmlFiles(w http.ResponseWriter, r *http.Request) {
 	w.Write(filesHTML(files, dt, datesWithData))
 }
 
-// htmlTask handles GET /web/task - displays task summary and table for a specific date
+// htmlTask handles GET /web/task - displays task summary and table for a specific date,
+// or cross-date ID history when an id filter is present.
 func (tm *taskMaster) htmlTask(w http.ResponseWriter, r *http.Request) {
 	dt, _ := time.Parse("2006-01-02", r.URL.Query().Get("date"))
 	if dt.IsZero() {
@@ -466,41 +467,65 @@ func (tm *taskMaster) htmlTask(w http.ResponseWriter, r *http.Request) {
 		filter.Page = 1
 	}
 
-	// Get task summary statistics for the date
-	summaryStart := time.Now()
-	taskStats, err := tm.taskCache.GetTaskRecapByDate(dt)
-	summaryTime := time.Since(summaryStart)
-	if err != nil {
-		log.Printf("Error getting task summary: %v", err)
-		taskStats = sqlite.TaskStats{}
-	}
-
-	// Get filtered and paginated tasks
-	queryStart := time.Now()
-	tasks, totalCount, err := tm.taskCache.GetTasksByDate(dt, filter)
-	queryTime := time.Since(queryStart)
-	if err != nil {
-		log.Printf("Error getting tasks: %v", err)
-		tasks = []sqlite.TaskView{}
-		totalCount = 0
+	historyMode := len(filter.ID) > 0
+	if !historyMode {
+		filter.Date = &dt
 	}
 
-	// Get dates with tasks for calendar highlighting
-	datesWithData, _ := tm.taskCache.DatesByType("tasks")
+	var (
+		taskStats   sqlite.TaskStats
+		tasks       []sqlite.TaskView
+		totalCount  int
+		hourlyStats [24]sqlite.TaskCounts
+		queryTime   time.Duration
+	)
 
-	var hourlyStats [24]sqlite.TaskCounts
-	if len(filter.ID) > 0 {
-		if _, stats, err := tm.taskCache.GetHourlyCountsByDate(dt, filter); err != nil {
-			log.Printf("Error getting hourly counts: %v", err)
+	if historyMode {
+		queryStart := time.Now()
+		var err error
+		tasks, totalCount, err = tm.taskCache.GetTasks(filter)
+		queryTime = time.Since(queryStart)
+		if err != nil {
+			log.Printf("Error getting tasks by ID: %v", err)
+			tasks = []sqlite.TaskView{}
+			totalCount = 0
+		}
+		taskStats = sqlite.TaskStats{}
+		// Hourly chart is day-shaped; omit in history mode (zero array + HistoryMode in template).
+		// Still need type/job keys for column filter dropdowns (UniqueTypes / JobsByType).
+		keys, keyErr := tm.taskCache.TypeJobKeys(&sqlite.TaskFilter{ID: filter.ID})
+		if keyErr != nil {
+			log.Printf("Error getting type/job keys for history filters: %v", keyErr)
 		} else {
-			hourlyStats = stats
+			taskStats = keys
 		}
 	} else {
+		summaryStart := time.Now()
+		var err error
+		taskStats, err = tm.taskCache.GetTaskRecapByDate(dt)
+		summaryTime := time.Since(summaryStart)
+		if err != nil {
+			log.Printf("Error getting task summary: %v", err)
+			taskStats = sqlite.TaskStats{}
+		}
+
+		queryStart := time.Now()
+		tasks, totalCount, err = tm.taskCache.GetTasks(filter)
+		queryTime = summaryTime + time.Since(queryStart)
+		if err != nil {
+			log.Printf("Error getting tasks: %v", err)
+			tasks = []sqlite.TaskView{}
+			totalCount = 0
+		}
+
 		_, hourlyStats = taskStats.HourlyCounts(filter)
 	}
 
+	// Get dates with tasks for calendar highlighting
+	datesWithData, _ := tm.taskCache.DatesByType("tasks")
+
 	w.Header().Set("Content-Type", "text/html")
-	htmlBytes := taskHTML(tasks, taskStats, totalCount, dt, filter, datesWithData, summaryTime+queryTime, hourlyStats)
+	htmlBytes := taskHTML(tasks, taskStats, totalCount, dt, filter, datesWithData, queryTime, hourlyStats, historyMode)
 	w.Write(htmlBytes)
 }
 
@@ -568,15 +593,18 @@ func filesHTML(files []sqlite.FileMessage, date time.Time, datesWithData []strin
 }
 
 // taskHTML renders the task summary and table HTML page
-func taskHTML(tasks []sqlite.TaskView, taskStats sqlite.TaskStats, totalCount int, date time.Time, filter *sqlite.TaskFilter, datesWithData []string, queryTime time.Duration, hourlyStats [24]sqlite.TaskCounts) []byte {
+func taskHTML(tasks []sqlite.TaskView, taskStats sqlite.TaskStats, totalCount int, date time.Time, filter *sqlite.TaskFilter, datesWithData []string, queryTime time.Duration, hourlyStats [24]sqlite.TaskCounts, historyMode bool) []byte {
 	renderStart := time.Now()
 
 	// Calculate navigation dates
 	prevDate := date.AddDate(0, 0, -1)
 	nextDate := date.AddDate(0, 0, 1)
 
-	// Get unfiltered counts for summary section (always show full day stats)
+	// Get unfiltered counts for summary section (always show full day stats in day mode)
 	unfilteredCounts := taskStats.TotalCounts()
+	if historyMode {
+		unfilteredCounts = sqlite.TaskCounts{Total: totalCount}
+	}
 
 	// Get unique types and jobs from TaskStats for filter dropdowns
 	types := taskStats.UniqueTypes()
@@ -596,17 +624,25 @@ func taskHTML(tasks []sqlite.TaskView, taskStats sqlite.TaskStats, totalCount in
 		endIdx = 0
 	}
 
+	pageTitle := "Task Dashboard"
+	dateLabel := date.Format("Monday, January 2, 2006")
+	if historyMode {
+		pageTitle = "Task History"
+		dateLabel = "All dates"
+	}
+
 	data := map[string]interface{}{
-		"Date":          date.Format("Monday, January 2, 2006"),
+		"Date":          dateLabel,
 		"DateValue":     date.Format("2006-01-02"),
 		"PrevDate":      prevDate.Format("2006-01-02"),
 		"NextDate":      nextDate.Format("2006-01-02"),
 		"Tasks":         tasks,
 		"Counts":        unfilteredCounts,
 		"HourlyStats":   hourlyStats,
+		"HistoryMode":   historyMode,
 		"Filter":        filter,
 		"CurrentPage":   "task",
-		"PageTitle":     "Task Dashboard",
+		"PageTitle":     pageTitle,
 		"isLocal":       isLocal,
 		"DatesWithData": datesWithData,
 		"UniqueTypes":   types,
@@ -635,8 +671,8 @@ func taskHTML(tasks []sqlite.TaskView, taskStats sqlite.TaskStats, totalCount in
 	renderTime := time.Since(renderStart)
 
 	// Single consolidated log with all metrics
-	log.Printf("Task page: date=%s filters=[id=%q type=%v job=%v result=%v sort=%q/%q] total=%d filtered=%d page=%d/%d query=%v render=%v size=%.2fMB",
-		date.Format("2006-01-02"), filter.ID, filter.Type, filter.Job, filter.Result, filter.Sort, filter.Direction,
+	log.Printf("Task page: date=%s history=%v filters=[id=%q type=%v job=%v result=%v sort=%q/%q] total=%d filtered=%d page=%d/%d query=%v render=%v size=%.2fMB",
+		date.Format("2006-01-02"), historyMode, filter.ID, filter.Type, filter.Job, filter.Result, filter.Sort, filter.Direction,
 		unfilteredCounts.Total, totalCount, filter.Page, totalPages,
 		queryTime, renderTime, float64(htmlSize)/(1024*1024))
 

apps/flowlord/handler/alert.tmpl

@@ -43,7 +43,7 @@
                             data-full-text="{{.TaskID}}"
                             title="{{if and .TaskID (ne .TaskID "unknown")}}Click link to view tasks, or click cell for actions{{else}}Click for actions{{end}}">
                             {{if and .TaskID (ne .TaskID "unknown")}}
-                            <a class="task-id-link" href="{{if $.isLocal}}./task_preview.html{{else}}/web/task{{end}}?date={{if .TaskTime.IsZero}}{{.CreatedAt.Format "2006-01-02"}}{{else}}{{.TaskTime.Format "2006-01-02"}}{{end}}&id={{.TaskID}}">{{.TaskID}}</a>
+                            <a class="task-id-link" href="{{if $.isLocal}}./task_preview.html{{else}}/web/task{{end}}?id={{.TaskID}}">{{.TaskID}}</a>
                             {{else}}
                             {{.TaskID}}
                             {{end}}

apps/flowlord/handler/static/backload.js

@@ -52,19 +52,10 @@
         return out;
     }
 
-    function taskCreatedDate(task) {
-        if (task && task.created) {
-            const d = task.created.slice(0, 10);
-            if (/^\d{4}-\d{2}-\d{2}$/.test(d)) return d;
-        }
-        return new Date().toISOString().split('T')[0];
-    }
-
     function buildViewAllTasksHref(tasks) {
         const ids = (tasks || []).map(t => t.id).filter(Boolean);
         if (ids.length === 0) return '';
-        const date = taskCreatedDate(tasks[0]);
-        return taskPageBase() + '?date=' + encodeURIComponent(date) + '&id=' + ids.map(encodeURIComponent).join(',');
+        return taskPageBase() + '?id=' + ids.map(encodeURIComponent).join(',');
     }
 
     // Initialize the backload form
@@ -683,8 +674,7 @@
                 const row = document.createElement('tr');
                 let idCell = '';
                 if (showIds && task.id) {
-                    const date = taskCreatedDate(task);
-                    const href = taskPageBase() + '?date=' + encodeURIComponent(date) + '&id=' + encodeURIComponent(task.id);
+                    const href = taskPageBase() + '?id=' + encodeURIComponent(task.id);
                     idCell = `<td class="id-cell id-column"><a class="task-id-link" href="${window.FlowlordUtils.escapeAttr(href)}">${window.FlowlordUtils.escapeHtml(task.id)}</a></td>`;
                 } else if (showIds) {
                     idCell = '<td class="id-cell id-column"></td>';

apps/flowlord/handler/static/task.js

@@ -564,14 +564,7 @@
     function rerunTasksHref(task) {
         const id = task && task.id;
         if (!id) return taskPageBase();
-        let date = '';
-        if (task.created) {
-            date = String(task.created).slice(0, 10);
-        }
-        if (!/^\d{4}-\d{2}-\d{2}$/.test(date)) {
-            date = new Date().toISOString().split('T')[0];
-        }
-        return taskPageBase() + '?date=' + encodeURIComponent(date) + '&id=' + encodeURIComponent(id);
+        return taskPageBase() + '?id=' + encodeURIComponent(id);
     }
 
     function messageFromRerunResponse(responseText, status) {