Forum Mining — Methodology, Context & Content
What we built to mine real workplace problems from Thai + English forums, and what it found. Reusable Lab pipeline. 2026-07-27.
1. Context — why we did this
The goal was cheap
problem discovery for a venture (new-manager development, then rescoped to all workplace-human problems). Instead of guessing what people struggle with, we read what they say
unprompted on public forums.
- The discipline: found-data mining ranks pain, never willingness-to-pay. It's the cheap first pass to rank real problems, not a substitute for talking to buyers.
- Scope evolved: started narrow (new-manager keywords) → Tong pushed for a blind sweep (don't only find what you search for) → rescoped to all workplace-human problems (discovery wide, entry narrow).
2. Methodology — how the pipeline works
Stage A — Get the raw posts (the hard part)
Free web access is blocked from a datacenter IP (Reddit 403, Pantip patchy). So we moved to Apify managed scrapers (residential proxies get past the blocks). The actor-selection lessons:
| Actor / method | Verdict | Lesson |
|---|
| WebSearch + WebFetch agents (free) | partial | Works for search snippets; direct fetch blocked on Reddit/Pantip. Query-biased. |
| trudax/reddit-scraper-lite | ✗ capped | Returns ~10/listing (~36 total). Useless for bulk. |
| gravityzer0/pantip-scraper | ✗ empty | Returned 0 posts. |
| mai_amm/pantip-consumer-intelligence | ✓ Thai | Rooms + tags + keywords; rich fields (body, room, likes). Tags don't filter reliably (noisy); rooms = true blind sweep; keywords = clean but biased. |
| practicaltools/apify-reddit-api | ✓ English | $2/1k, but caps 100/run → fire multiple sorts (top all/year/month, hot, new) and dedup. |
| Reddit official API | ✗ gated | Free + clean, but app creation blocked behind Reddit's Responsible-Builder review (new accounts can't create). |
Stage B — Clean it (the two-stage insight)
Because tags are noisy and keywords are biased, the fix (Tong's call): sweep broad, then filter by title, then read only the relevant ones. You eyeball every title instead of trusting a tag or a search term, dodging both failure modes. The body text comes with the post, so "go into it" is a filter step, not a re-scrape.
Stage C — Code it (bottom-up, at scale)
- Embed the corpus in a generated Workflow file (run by
scriptPath, since inline args are too big).
- Derive a bottom-up taxonomy from a sample (don't impose categories, let the data name themes).
- Classify every post in parallel batches (~70/batch) into the taxonomy + tag each for
human, intensity, fundable. Run on Sonnet (mechanical work, cheap).
- Aggregate deterministically in JS (counts, TH/EN split, fundability share).
- Adversarial verify pass (on the stronger model): vocal-minority check, confounds, "which is most fundable," what's missing.
Stage D — Guard against yourself
- Confound check: when a split looked cultural (Thai vs Western), we pulled a matched general-English sample to test if it was really just the source. It was, so we corrected the claim.
- Budget discipline: Apify FREE = $5/mo cap. Test-small-then-scale; flag overruns (the Reddit run cost $2 vs a $1 estimate, said so).
3. Content — what we actually pulled
| Source | What it is | Posts |
|---|
| Pantip (kw + room sweeps) | Thai general work forum (สีลม / work tags) | 601 |
| r/managers | English, manager-only community | 329 |
| r/jobs + r/antiwork + r/careerguidance | English general work forums | 599 |
| r/humanresources + r/AskHR | English HR practitioners (buyer-adjacent) | 398 |
| (earlier) WebSearch saturation run | Thai+English, query-based | 122 |
| Total | 4 platforms, 8 communities | ~2,000 |
Apify spend ~$9 across two free accounts. Coded on Sonnet; verified on the stronger model.
4. What it found (headline)
- The map: 22 workplace-human problem themes. Loudest pain = toxic bosses + coworkers. Most fundable = new-manager development (upstream + pre-budgeted).
- Self-correction: the "Thais avoid the hard conversation" moat was a sampling artifact (general forum vs manager sub), not culture. The confound check killed our own claim.
- Pitch angle: HR doesn't name "manager development" as their problem, it's invisible as a category. Lead with the fires HR owns (bad terminations, harassment traced to a manager, disengaged hiring managers), sell manager practice as the upstream fix.
5. Reusable SOP (to run this again for any venture)
- Pick the communities (a general forum + a practitioner/buyer forum per language).
- Thai: mai_amm/pantip-consumer-intelligence, sweep by room (blind). English: practicaltools/apify-reddit-api, multi-sort + dedup.
- Filter by title to the relevant set (don't trust tags).
- Generate the coding Workflow (sample→taxonomy→Sonnet classify→aggregate→verify).
- Run a confound check on any split that looks like a "finding" before you believe it.
- Remember what it can't do: this ranks pain. Only a buyer conversation tests demand.
The honest bottom line: this pipeline is fast, cheap (~$9), and self-correcting, and it produced a real problem map, a named buyer, and a pitch angle. It did not, and cannot, tell us whether anyone will pay. That's the next thing, and it's a conversation, not a scrape.