fix: harden bot runtime stability

README.md

@@ -1,6 +1,6 @@
 # JoblessYu
 
-JoblessYu is a high-performance Discord bot built in Go that scrapes IT job listings from Vietnamese and global job boards (ITViec, Indeed, LinkedIn), classifies them with Groq AI (`llama-3.1-8b-instant`), and serves them via an interactive, 100% ephemeral (`"Only you can see this"`) slash command with multi-keyword search, modal page jumps, and dual real-time Discord Hub status cards.
+JoblessYu is a high-performance Discord bot built in Go that scrapes IT job listings from Vietnamese and global job boards (ITViec, Indeed, LinkedIn), classifies them with Groq AI (`openai/gpt-oss-20b`), and serves them via an interactive, 100% ephemeral (`"Only you can see this"`) slash command with multi-keyword search, modal page jumps, and dual real-time Discord Hub status cards.
 
 ## Architecture
 
@@ -12,7 +12,7 @@ DAILY 5:00 AM ICT (automated cron) / Manual Trigger (`make scrape`)
     │      ├── Go Colly        → ITViec (30 jobs, 24h freshness filter)
     │      └── Cross-Site Deduplication (7-day window → merges alternate URLs)
     │
-    ├── 2. AI ENRICHMENT (Groq — llama-3.1-8b-instant)
+    ├── 2. AI ENRICHMENT (Groq — openai/gpt-oss-20b)
     │      ├── Classifies: level, type, expertise, tags, salary, remote, summary
     │      ├── 1-job request loop with 18s throttle (~5,050 TPM safely under 6,000 TPM limit)
     │      ├── Regex fallback only when Groq is unreachable (network errors)
@@ -95,7 +95,7 @@ DISCORD_GUILD_ID=your_guild_id
 DISCORD_CHANNEL_ID=your_hub_channel_id
 DATABASE_URL=your_neon_postgres_connection_string
 GROQ_API_KEY=your_groq_api_key
-AI_MODEL=llama-3.1-8b-instant
+AI_MODEL=openai/gpt-oss-20b
 JOB_RETENTION_DAYS=30
 ```
 
@@ -157,7 +157,7 @@ make clean        # Remove build artifacts
 |---|---|---|
 | Bot Gateway | Go + discordgo | Dual static pinned cards + ephemeral components |
 | Scrapers | Python JobSpy (Indeed, LinkedIn) + Go Colly (ITViec) | 30/30/30 target scrape distribution |
-| AI Enrichment | Groq (`llama-3.1-8b-instant`) | 1-job request loop @ 18s delay (~5,050 TPM) |
+| AI Enrichment | Groq (`openai/gpt-oss-20b`) | 1-job request loop @ 18s delay (~5,050 TPM) |
 | Database | Neon PostgreSQL (Serverless) | GIN Trigram indexes (`pg_trgm`) + LISTEN/NOTIFY |
 | Container | Docker Multi-stage (Go 1.24 static + Python 3.11) | HTTP `/healthz` probe on port 8080 |
 | CI | GitHub Actions | Automated Go test + static analysis |

docs/AI/changelog.md

@@ -3,6 +3,45 @@
 > Emergency backup context. Grand scheme from the foundation.
 > Roll up progress-log entries here at slice boundaries so context survives session resets.
 
+## [Slice T] — 2026-08-20
+
+### Summary
+P0 Runtime Safety & Long-Run Stability Pass. Fixed the Groq empty-response panic, made the parallel Colly scraper safe and cancellable, made the Postgres LISTEN loop reconnect correctly, serialized Discord hub refresh/debounce state, made bot shutdown idempotent, and added hard memory bounds to UI caches and captured Python subprocess output.
+
+### Files touched
+- modified: `internal/job/groq.go` and `internal/job/groq_test.go` (empty `choices` guard and regression test)
+- modified: `internal/job/colly_scraper.go` and `internal/job/colly_scraper_test.go` (mutex-protected shared results and context cancellation test)
+- modified: `internal/job/store.go` (fresh-connection LISTEN retry loop with context-aware backoff)
+- modified: `internal/bot/bot.go` and `internal/bot/handlers.go` (serialized card refreshes, idempotent shutdown, bounded caches)
+- created: `internal/bot/bot_test.go` (cache-cap regression test)
+- modified: `internal/scraper/scraper.go` (64 KiB thread-safe tail capture for Python output)
+- modified: `docs/AI/progress-log.md`, `docs/AI/structure.md`, and `docs/AI/modules.md` (runtime behavior and verification notes)
+
+### Verification
+- Go race-enabled tests: PASS
+- Go vet: PASS
+- Go build: PASS
+- Python syntax check: PASS
+- Ruff: one existing import-order issue remains
+
+## [Slice S] — 2026-08-18 / 2026-08-20
+
+### Summary
+Groq Model Migration to `openai/gpt-oss-20b`, Multi-Key Pool & High-Density Prompt Compression. Migrated primary Groq AI LLM model from deprecated `llama-3.1-8b-instant` to `openai/gpt-oss-20b` (Groq's official direct non-reasoning replacement). Implemented Multi-Key pooling in `GroqExtractor` supporting comma-separated keys (`GROQ_API_KEY=key1,key2`) with round-robin balancing, per-key rate limiting, key-level TPD daily exhaustion isolation, and instant failover on 429 errors. Compressed `skills.md` from 9.8KB to 2.3KB (~400 tokens), cutting per-call token usage by ~75% while retaining 100% of 24 IT categories, Vietnamese seniority signals, tag categories, and schema constraints. Added TPD/RPD quota exhaustion interception in `enricher.go` with compound duration parsing (`16m3.36s`). Tuned throttle to 15s with `lastCall` timestamp-based adaptive rate-limiting in `groq.go`. Implemented `sanitizeJD()` to strip raw HTML tags, `<script>`/JSON-LD blobs, and decode HTML entities.
+
+### Files touched
+- modified: `internal/config/config.go` (updated default `AIModel` to `openai/gpt-oss-20b`)
+- modified: `.env` & `.env.example` (updated `AI_MODEL=openai/gpt-oss-20b`)
+- modified: `internal/job/enricher.go` (added TPD daily quota fallback and compound retry parser)
+- modified: `internal/job/groq.go` (added `sanitizeJD()`, `lastCall` adaptive throttle, tuned 15s delay & 800 max tokens)
+- modified: `internal/job/skills.md` (added rule 5 forbidding `<think>` tags)
+- modified: `README.md` (updated model references in documentation)
+- modified: `docs/AI/modules.md` (updated Groq model specs)
+- modified: `docs/AI/progress-log.md` (recorded Slice S progress)
+- modified: `docs/AI/changelog.md` (this entry)
+
+---
+
 ## [Slice Q & R] — 2026-08-14
 
 ### Summary
@@ -639,4 +678,3 @@ Reverted Groq AI enrichment to single-job requests with an 18s delay to eliminat
 - `make scrape`: PASS (Updated Card 1 & Card 2 live in Discord)
 - Migration 007 active on Neon DB.
 
-

internal/bot/bot.go

@@ -16,6 +16,8 @@ import (
 const (
 	jobsCacheTTL        = 30 * time.Minute
 	jobsCacheSweepEvery = 1 * time.Minute
+	jobsCacheMaxEntries = 128
+	uiStateMaxEntries   = 256
 )
 
 type cachedJobs struct {
@@ -57,6 +59,7 @@ type Bot struct {
 	cacheLock sync.RWMutex
 
 	stopJanitor chan struct{}
+	stopOnce    sync.Once
 }
 
 var commands = []*discordgo.ApplicationCommand{
@@ -173,6 +176,43 @@ func (b *Bot) evictExpired() {
 			delete(b.uiState, id)
 		}
 	}
+	b.trimCachesLocked()
+}
+
+// trimCachesLocked enforces hard entry limits in addition to the TTL. A busy
+// Discord server can create many independent ephemeral searches before the
+// one-minute janitor runs; without a cap, each search retains up to 1,000 full
+// JobEntry values (including descriptions) in memory.
+func (b *Bot) trimCachesLocked() {
+	for len(b.jobsCache) > jobsCacheMaxEntries {
+		var oldestID string
+		var oldest time.Time
+		for id, entry := range b.jobsCache {
+			if oldestID == "" || entry.insertedAt.Before(oldest) {
+				oldestID = id
+				oldest = entry.insertedAt
+			}
+		}
+		if oldestID == "" {
+			break
+		}
+		delete(b.jobsCache, oldestID)
+	}
+
+	for len(b.uiState) > uiStateMaxEntries {
+		var oldestID string
+		var oldest time.Time
+		for id, entry := range b.uiState {
+			if oldestID == "" || entry.insertedAt.Before(oldest) {
+				oldestID = id
+				oldest = entry.insertedAt
+			}
+		}
+		if oldestID == "" {
+			break
+		}
+		delete(b.uiState, oldestID)
+	}
 }
 
 func (b *Bot) Start() error {
@@ -213,8 +253,16 @@ func (b *Bot) Start() error {
 				lastKnownCount := -1
 				ticker := time.NewTicker(30 * time.Second)
 				defer ticker.Stop()
+				updateRequests := make(chan struct{}, 1)
+				requestUpdate := func() {
+					select {
+					case updateRequests <- struct{}{}:
+					default:
+						// A refresh is already queued or being debounced.
+					}
+				}
 
-				updateCardsIfChanged := func() {
+				refreshCardsIfChanged := func() {
 					rCtx, rCancel := context.WithTimeout(context.Background(), 10*time.Second)
 					defer rCancel()
 					details := b.fetchStatusDetails(rCtx)
@@ -230,32 +278,49 @@ func (b *Bot) Start() error {
 				}
 
 				// Initial check to prime lastKnownCount
-				updateCardsIfChanged()
+				refreshCardsIfChanged()
 
 				// 1. Postgres LISTEN event listener (push notifications)
+				listenerCtx, listenerCancel := context.WithCancel(context.Background())
+				defer listenerCancel()
+				go func() {
+					select {
+					case <-b.stopJanitor:
+						listenerCancel()
+					case <-listenerCtx.Done():
+					}
+				}()
 				if statsService, ok := b.jobService.(*job.JobService); ok {
-					ctx, cancel := context.WithCancel(context.Background())
-					go func() {
-						<-b.stopJanitor
-						cancel()
-					}()
-					var debounceTimer *time.Timer
-					go statsService.ListenForJobChanges(ctx, func() {
-						if debounceTimer != nil {
-							debounceTimer.Stop()
-						}
-						debounceTimer = time.AfterFunc(500*time.Millisecond, func() {
-							updateCardsIfChanged()
-						})
-					})
+					go statsService.ListenForJobChanges(listenerCtx, requestUpdate)
 				}
 
 				// 2. 30s fail-safe polling ticker (handles Neon PgBouncer transaction pooler drops)
+				var debounceTimer *time.Timer
+				var debounceC <-chan time.Time
 				for {
 					select {
 					case <-ticker.C:
-						updateCardsIfChanged()
+						refreshCardsIfChanged()
+					case <-updateRequests:
+						if debounceTimer == nil {
+							debounceTimer = time.NewTimer(500 * time.Millisecond)
+						} else {
+							if !debounceTimer.Stop() {
+								select {
+								case <-debounceTimer.C:
+								default:
+								}
+							}
+							debounceTimer.Reset(500 * time.Millisecond)
+						}
+						debounceC = debounceTimer.C
+					case <-debounceC:
+						debounceC = nil
+						refreshCardsIfChanged()
 					case <-b.stopJanitor:
+						if debounceTimer != nil {
+							debounceTimer.Stop()
+						}
 						return
 					}
 				}
@@ -266,13 +331,26 @@ func (b *Bot) Start() error {
 	return nil
 }
 
+func (b *Bot) signalStop() bool {
+	stopped := false
+	b.stopOnce.Do(func() {
+		close(b.stopJanitor)
+		stopped = true
+	})
+	return stopped
+}
+
 func (b *Bot) CloseWithoutOffline() {
-	close(b.stopJanitor)
+	if !b.signalStop() {
+		return
+	}
 	b.session.Close()
 }
 
 func (b *Bot) Stop() {
-	close(b.stopJanitor)
+	if !b.signalStop() {
+		return
+	}
 
 	// Update static availability card to Offline with real DB stats
 	if b.notifier != nil {

internal/bot/bot_test.go

@@ -0,0 +1,38 @@
+package bot
+
+import (
+	"fmt"
+	"testing"
+	"time"
+)
+
+func TestTrimCachesLockedCapsEntries(t *testing.T) {
+	b := &Bot{
+		jobsCache: make(map[string]cachedJobs),
+		uiState:   make(map[string]cachedCriteria),
+	}
+
+	for i := 0; i < jobsCacheMaxEntries+10; i++ {
+		b.jobsCache[fmt.Sprintf("job-%03d", i)] = cachedJobs{insertedAt: time.Unix(int64(i), 0)}
+	}
+	for i := 0; i < uiStateMaxEntries+10; i++ {
+		b.uiState[fmt.Sprintf("state-%03d", i)] = cachedCriteria{insertedAt: time.Unix(int64(i), 0)}
+	}
+
+	b.cacheLock.Lock()
+	b.trimCachesLocked()
+	b.cacheLock.Unlock()
+
+	if got := len(b.jobsCache); got != jobsCacheMaxEntries {
+		t.Fatalf("jobs cache length = %d, want %d", got, jobsCacheMaxEntries)
+	}
+	if got := len(b.uiState); got != uiStateMaxEntries {
+		t.Fatalf("ui state length = %d, want %d", got, uiStateMaxEntries)
+	}
+	if _, ok := b.jobsCache["job-000"]; ok {
+		t.Error("oldest jobs cache entry was not evicted")
+	}
+	if _, ok := b.uiState["state-000"]; ok {
+		t.Error("oldest UI state entry was not evicted")
+	}
+}

internal/bot/handlers.go

@@ -54,6 +54,7 @@
 
 	b.cacheLock.Lock()
 	b.uiState[msg.ID] = cachedCriteria{state: state, insertedAt: time.Now()}
+	b.trimCachesLocked()
 	b.cacheLock.Unlock()
 }
 
@@ -217,7 +218,7 @@
 	sessionKey := i.Interaction.ID
 	components := buildJobResultV2Components(jobs[0], 1, len(jobs), sessionKey)
 
-	msg, err := s.FollowupMessageCreate(i.Interaction, true, &discordgo.WebhookParams{
+	_, err = s.FollowupMessageCreate(i.Interaction, true, &discordgo.WebhookParams{
 		Components: components,
 		Flags:      discordgo.MessageFlagsEphemeral | discordgo.MessageFlagsIsComponentsV2,
 	})
@@ -231,9 +232,7 @@
 	if sessionKey != "" {
 		b.jobsCache[sessionKey] = entry
 	}
-	if msg != nil && msg.ID != "" {
-		b.jobsCache[msg.ID] = entry
-	}
+	b.trimCachesLocked()
 	b.cacheLock.Unlock()
 }
 
@@ -424,7 +423,7 @@
 	return ""
 }
 
 func buildJobPaginationComponents(page, total int, sessionKey string) []discordgo.MessageComponent {
 	return []discordgo.MessageComponent{
 		discordgo.ActionsRow{Components: buildJobNavigationButtons(page, total, sessionKey)},
 	}
@@ -529,6 +528,7 @@
 func (b *Bot) saveCriteriaState(messageID string, state criteriaState) {
 	b.cacheLock.Lock()
 	b.uiState[messageID] = cachedCriteria{state: state, insertedAt: time.Now()}
+	b.trimCachesLocked()
 	b.cacheLock.Unlock()
 }
 
@@ -595,7 +595,7 @@
 	return res
 }
 
 func mapLevelToQuery(v string) string {
 	res := mapLevelsToQuery([]string{v})
 	if len(res) > 0 {
 		return res[0]
@@ -603,7 +603,7 @@
 	return ""
 }
 
 func mapLocationToQuery(v string) string {
 	res := mapLocationsToQuery([]string{v})
 	if len(res) > 0 {
 		return res[0]
@@ -611,14 +611,14 @@
 	return ""
 }
 
 func mapPositionToQuery(v string) string {
 	if v == "all" || v == "" {
 		return ""
 	}
 	return v
 }
 
 func mapJobTypeToQuery(v string) string {
 	res := mapJobTypesToQuery([]string{v})
 	if len(res) > 0 {
 		return res[0]

internal/config/config.go

@@ -42,6 +42,10 @@ func Load() *Config {
 		AIModel:    os.Getenv("AI_MODEL"),
 	}
 
+	if cfg.GroqAPIKey == "" {
+		cfg.GroqAPIKey = os.Getenv("GROQ_API_KEYS")
+	}
+
 	// Fallback check for legacy env variable names if DISCORD_CHANNEL_ID is not set
 	if cfg.DiscordChannelID == "" {
 		if v := os.Getenv("DISCORD_STATUS_CHANNEL_ID"); v != "" {
@@ -66,7 +70,7 @@ func Load() *Config {
 
 	// AI config defaults.
 	if cfg.AIModel == "" {
-		cfg.AIModel = "llama-3.1-8b-instant"
+		cfg.AIModel = "openai/gpt-oss-20b"
 	}
 	if cfg.GroqAPIKey == "" {
 		log.Println("config: GROQ_API_KEY is not set; AI enrichment will be skipped")