From Scattered Scripts to a Full Analysis Platform in One Session

Previous Threads work lived inside ByTheWeiCo — tangled into a larger codebase with no database, no sync pipeline, no standalone interface. This session extracted it into its own project and built the entire stack.

What Got Built

Layer	Details
Postgres	13 tables, GIN text search, materialized views, 49K posts seeded in 10.5s
Sync Worker	Docker container, auto-pulls every 30 min, graceful shutdown via AbortController
Analysis Pipeline	Shannon entropy, Zipf, surprise scores, PMI knowledge graph → writes JSON + Postgres
Dashboard	7 Astro pages, 10 API routes, 9 React islands
Knowledge Graph	1,638 nodes, 11,155 edges, interactive force-directed viz
Interactions	409 users mapped, 1,668 @mention interactions extracted

Commands

# ─── Docker (pick your laziness level) ───
npm run docker:db # just Postgres
npm run docker:data # Postgres + sync worker
npm run docker:web # Postgres + dashboard on :4321
npm run docker:all # everything
npm run docker:down # stop all
npm run docker:rebuild # rebuild + restart

# ─── Manual sync ───
npm run sync # pull from Threads API (incremental)
npm run sync:replies # backfill reply history
npm run sync:metrics # backfill engagement metrics

# ─── Analysis ───
npm run analyze # info theory + knowledge graph → JSON + Postgres
npm run full # sync + analyze

# ─── Database ───
npm run db:seed # backfill posts.json → Postgres (~10s)

# ─── Development ───
npm run dev # Astro dev server on :4321
npm run build # production build
npm test # vitest

The CTO Pattern

The key technique: treat Claude Code as a team, not a single assistant.

1. Infrastructure — one agent built Docker + Postgres + sync worker + seed script
2. /simplify — 3 parallel review agents found 150 lines of duplication → extracted shared module
3. Blind validation — agent with zero context independently validated everything (61 tool calls, all pass)
4. Build sprint — 4 build agents + 1 validator simultaneously (static pages, API routes, React islands, pipeline)
5. Dev team — 3 named devs (Alex, Jordan, Sam) worked from a shared workboard on quick wins

Pages

Route	Type	What
/	Static	Overview dashboard, stat cards, treemap, tag bars, engagement heatmap, surprise scatter
/taxonomy	Static	Squarified treemap, sub-tag distribution, classification audit
/chronology	Static	Monthly stacked bars, calendar heatmap, hour/day charts, mutual information
/discourse	Static	9-category deep dive, surprise distribution, Foucault Index
/network	Static+React	Force-directed knowledge graph (1,638 nodes)
/explore	SSR+React	Full-text search, tag/date/surprise filters, paginated post browser
/interactions	SSR+React	Force-graph of @mention interaction network (409 users)

API Routes

GET /api/posts?tag=philosophy&q=foucault&from=2025-01-01&surprise_min=10&page=1
GET /api/tags
GET /api/tags?parent=philosophy
GET /api/graph?types=tag,sub_tag&min_weight=0.5
GET /api/metrics?top=20&sort=views
GET /api/search?q=power+knowledge&limit=20
GET /api/sync-status
GET /api/interactions?username=sleeping_ox
GET /api/mutual-info
GET /api/engagement-heatmap?metric=views
GET /api/surprise-engagement?limit=500

Key Numbers

• Zipf exponent: 0.999 (essentially perfect natural language)
• Word entropy: 10.376 bits
• Vocabulary: 25,804 unique words across 446,549 total
• Tag entropy: 3.073 bits across 20 categories
• Strongest MI pair: tag × post_length (NMI = 0.54)
• Most interacted user: @feministajones (108 interactions)

Repo

github.com/Storiesbywei/threads-analysis (private)