Audit: three years of the PROHARDVER! home-server forum read against Felhom; R-990, R-991 filed

Read-only. fetch.py (polite, 5 s apart, 34 pages, all 200), parse.py (own text only; four parser defects found by the controls and fixed), build.py + spotcheck.py (stdlib). tags.csv holds post numbers and tags only - no username, author id, salt or post text in git. FINDINGS.md (2,908 posts, 195 people, 82 beginner threads in 7 clusters), FELHOM.md (catalog gap, hardware fit, 15 website questions, objections, ten ranked recommendations), METHOD.md (controls, spot-check: role and hardware below 85 % after two rounds, the table-driving splits 92.5 % and 97.3 %), REPORT.md. R-990: operator rules on the recommendations. R-991: no buyer-readable hardware floor. Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-12 22:20:52 +02:00
parent a7f897a7dd
commit 52b6599d97
12 changed files with 4287 additions and 3 deletions
@@ -2,7 +2,11 @@
 **Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
-**Updated 2026-10-12 (evening): demo-hp, demo-felhom and Tester 1 run agent 0.154.0 and controller 0.309.0. The open-items list is at 118 (2026-10-12 night; the newest row is the survey reading, R-989). Reports: `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
+**Updated 2026-10-12 (evening): demo-hp, demo-felhom and Tester 1 run agent 0.154.0 and controller 0.309.0. The open-items list is at 120 (2026-10-12 night; the newest rows are the forum reading, R-990, and the missing hardware page, R-991). Reports: `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
 ## Monday 2026-10-12 (night): what Hungarian beginners ask on the PROHARDVER! home-server forum
 - **Waiting on you — decide later:** read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 (ten ranked ideas from three years of the forum) and say yes, no or later to each (R-990); and name the smallest machine Felhom accepts, so the download page can say it (R-991). If you do nothing: nothing changes; the research ages, and a buyer with a Raspberry Pi or a Synology can still download an installer that cannot run there.
 ## Monday 2026-10-12 (night): what 4,000 self-hosters do, against Felhom
@@ -0,0 +1,90 @@
 # FELHOM — what three years of the PROHARDVER! home-server topic mean for Felhom
 Evidence: `FINDINGS.md` (table and row named per line). Hobby forum — the beginners are the closest
 match to our buyer, the experts are the do-it-yourself competition. **Nothing here is acted on.**
 ## 1 Catalog gap — wanted by ≥ 3 people, not in `app-catalog-felhom.eu/templates/`
 The spec named `templates.json`; that file is the LEGACY Portainer index (10 apps). The live catalog is
 the `templates/` folder (62 apps) and was used instead.
 | App (job) | People wanting | One line |
 |---|---|---|
 | DLNA media server (MiniDLNA, Serviio) | 7 | Plays films on a TV with no app; Jellyfin/Plex cover it only partly (FINDINGS 3c) |
 | Ad / DNS blocker (Pi-hole, AdGuard Home) | 5 (+14 experienced run one, 3b) | Network-wide; a DNS role on the box needs a design, not just a template |
 | Camera recorder / NVR (Frigate, ZoneMinder, MotionEye) | 5 | CPU- and disk-heavy; changes the hardware advice |
 | Router / firewall (OPNsense, pfSense, OpenWrt) | 6 | Not an app for a home server — out of scope; say so on the website |
 Every other app wanted by ≥ 3 people is in the catalog: Plex, Jellyfin, Emby, qBittorrent,
 Transmission, Home Assistant, Nextcloud, Immich, Sonarr, Radarr, a Minecraft server
 (`crafty-controller`). Samba is built in (Felhom's SMB sharing).
 ## 2 Hardware fit
 **No document states what hardware Felhom supports.** Searched: `documentation/architecture/`,
 `website/` (gyik, technologiak, letoltes), `scripts/felhom-host-install.sh`. What exists: the FAQ
 recommends an N100 mini PC with 16–32 GB RAM and says an old desktop „often” works after an
 assessment; the download page says the installer is built on Proxmox VE 9.2 — so x86-64 only (inferred
 from Proxmox, not stated). No RAM, disk or CPU floor is written anywhere a buyer can read it.
 | What beginners own (FINDINGS 4a) | People | Fits the installer? |
 |---|---|---|
 | Used business mini PC / desktop, N100-class, self-built, old desktop/laptop, thin client | 25 of 33 owners | Yes in principle (x86-64); RAM unknown — thin clients and old laptops often have 4–8 GB |
 | Only a Synology / QNAP / other NAS box, a router with a USB disk, or a Raspberry Pi | 8 of 33 | **No** — ARM or a closed appliance; the ISO cannot install |
 So about one beginner in four who names their hardware owns nothing Felhom can install on, and
 nothing on our site tells them so.
 ## 3 Questions our website should answer (Hungarian, our own words)
 | # | Question | Cluster |
 |---|---|---|
 | 1 | Nekem kell eldöntenem, hogy TrueNAS, OMV vagy Proxmox kerüljön a gépre? | 1 |
 | 2 | Kell hozzá Linuxot tudnom, vagy parancssort használnom? | 1 |
 | 3 | Jó a meglévő régi vagy irodai gépem, vagy újat kell vennem? | 7 |
 | 4 | Milyen gépet vegyek, ha most kezdem, és mennyibe kerül? | 2 |
 | 5 | Miért nem elég egy Synology vagy más kész NAS? | 5 |
 | 6 | Működik Raspberry Pi-n, Synologyn vagy a routerre dugott lemezen? | 5 / §2 |
 | 7 | Mennyi áramot fogyaszt egy évben, és mennyire hangos? | 2, pains 1–2 |
 | 8 | Hogyan érem el a fényképeimet és filmjeimet telefonról, otthonon kívül? | 6 |
 | 9 | Kell portot nyitnom a routeren? Biztonságos ez? | 6 |
 | 10 | Mi történik, ha tönkremegy egy lemez? Elvesznek a családi fotók? | 3 |
 | 11 | Hány lemez fér bele, és bővíthetem később? | 3 |
 | 12 | Megy rajta a Plex vagy a Jellyfin a tévén, és a torrent? | 4 |
 | 13 | A családom többi tagja is tudja majd használni, gép-tudás nélkül? | pains 10 |
 | 14 | Ha egy frissítés elront valamit, ki javítja meg? | pains 14 |
 | 15 | Ha leállok a szolgáltatással, megmaradnak az adataim és a gépem? | 5, pains 12 |
 ## 4 Objections — why people choose „do it yourself”, and the fears a paid service must answer
 | Why DIY wins today (people) | Evidence |
 |---|---|
 | It is a hobby: the tinkering is the point | #10213, #10959; 295 own-setup posts |
 | Cheap: a used office PC at 20–60k HUF beats a NAS | budget median 60–61k (4); #8303, #8313 |
 | Freedom from a vendor: Synology's 2025 disk lock and TrueNAS's app change | lock-in 15 (6); #10352, #8752 |
 | The forum answers for free | 1,600 answer posts in three years |
 | Fear a paid service must answer | People (6) |
 |---|---|
 | Power bill, noise, heat, space | 60 · 43 · 24 · 38 |
 | It is too complex / costs too much time | 39 · 22 |
 | Losing data; a disk or the machine failing | 19 · 29 · 14 |
 | Security of remote access | 17; remote-access confusion 11 |
 | Family will not or cannot use it | 18 |
 | An update breaks it | 14 |
 | Lock-in (a paid service is itself a vendor) | 15 |
 ## 5 Recommendations — ranked; none acted on
 | # | Recommendation | Evidence |
 |---|---|---|
 | 1 | Publish a „which machine works” page: x86-64 only, a RAM and disk floor, and a plain „not a Synology, not a Raspberry Pi, not a router disk” | §2; 4a; cluster 2, 5 |
 | 2 | Answer the cluster-1 question head-on: „you do not choose an operating system — we do” | 2 row 1 (20 threads, the biggest) |
 | 3 | Offer „bring your old or office PC” with a check of the machine, as the cheap entry | 2 rows 2, 7; 4a/4b; budget median ~60k |
 | 4 | Put measured power draw and noise for the recommended box on the website | 6 rows 1–2 |
 | 5 | Add a DLNA media server to the catalog | 3c (7 people) |
 | 6 | Decide whether an ad/DNS blocker belongs on the box (network-wide DNS is a design choice) | 3b, 3c |
 | 7 | Decide whether a camera recorder belongs in the catalog (it changes the hardware advice) | 3c (5 people) |
 | 8 | Position against Synology, the one product people name (43 people): price, lock-in, „it is yours” | 5e; 6 row 12 |
 | 9 | One line on remote access: how the household reaches the box from outside, and whether a router port or a VPN app is needed (the site does not say today) | 2 row 6; 6 row 16 |
 | 10 | Show family use: one link per app for a household member who knows no computers | 6 row 10 |
@@ -0,0 +1,274 @@
 # FINDINGS — PROHARDVER! „Home server / házi szerver építése”, 2023-10-12 → 2026-10-12
 Source: posts #7943–#11092 of the public topic, fetched 2026-10-12. Every number below is produced by
 `build.py` from `tags.csv` and can be re-run. Post numbers make each claim checkable on the forum.
 „People” = distinct writers (pseudonymous hash), never posts. Method and limits: `METHOD.md`.
 **Hardware counts are approximate** (spot-check agreement 82.9 %, below the 85 % bar — `METHOD.md`).
 ## 1 Numbers
 | Item | Value |
 |---|---|
 | Post numbers in the window | 3,150 (#7943–#11092) |
 | Posts on the forum, read and tagged | 2,908 (+1 excluded: #11091, Felhom's own announcement) |
 | Missing on the forum | 241 — #9874–#10108 (235, the 2025 site outage), #10118–#10120, #8571, #9230, #11073 |
 | Pages fetched | 34 (2 empty: hsz_9901–10000, hsz_10001–10100) |
 | People | 195 (49 active in 2023 Q4, 101 in 2024, 96 in 2025, 62 in 2026 to Oct) |
 | Posts by role | answers 1,600 · experienced asks 464 · other 351 · own setup 295 · beginner asks 198 |
 | Question threads (all) | 308 |
 | Beginner threads | 82, from 54 people (2023 Q4: 14 · 2024: 31 · 2025: 23 · 2026: 14) |
 | Beginner threads resolved | yes 9 · no 4 · unknown 69 (most askers never report back) |
 ## 2 Top beginner questions (82 threads)
 | # | Cluster (our words) | Threads | Examples | Answer the community most often gave |
 |---|---|---|---|---|
 | 1 | Which system do I install — TrueNAS, OMV, Proxmox, Xpenology, Windows? | 20 | #8046, #8201, #10418 | OpenMediaVault, then Proxmox; apps in Docker; plain Debian if you know Linux; TrueNAS called hard for a beginner |
 | 2 | Which small, low-power machine should I buy? | 16 | #8143, #9251, #10518 | A used business mini PC or an N100-class box; for TV playback alone a Kodi/LibreELEC player |
 | 3 | Disks: external USB drives, powering drives, adapters, RAID | 12 | #8124, #8301, #8783 | Do not power drives by hand — take a used business tower (SFF/MT) with bays; skip board RAID; keep a cold backup |
 | 4 | How do I install or connect an app (containers, shares, Home Assistant)? | 11 | #7981, #8631, #10514 | Case by case; Home Assistant OS in its own VM or a cheap separate box |
 | 5 | Ready NAS (Synology) or build / reuse a PC? | 10 | #7952, #8285, #8847 | Split: build or reuse a PC (6 answers) vs Synology (5) — price and freedom against convenience |
 | 6 | How do I reach it from my phone or outside the home? | 7 | #8037, #8073, #8516 | Tailscale or WireGuard; never expose SMB; slow upload limits remote films |
 | 7 | Is my old, inherited or office PC good enough? | 6 | #8330, #10254, #9145 | Usually yes for files, torrent and Plex; watch power draw |
 Question types across all 82 threads (one thread may have several): hardware choice 49 · OS/platform 26 ·
 storage/RAID 23 · remote access 16 · install help 13 · app choice 9 · power/noise 4 · cost 4 · broken 4 · backup 3.
 ## 3 Apps
 **3a Wanted by beginners** (top 30)
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | Plex | 15 | #8176, #8201, #8240 |
 | 2 | qBittorrent | 12 | #7952, #7964, #8046 |
 | 3 | Home Assistant | 8 | #8285, #8292, #8296 |
 | 4 | Jellyfin | 7 | #8273, #9145, #9148 |
 | 5 | Samba | 7 | #8046, #8056, #8071 |
 | 6 | BitTorrent client | 4 | #8330, #8358, #8367 |
 | 7 | Nextcloud | 3 | #8783, #8785, #8999 |
 | 8 | Emby | 2 | #8143, #11053 |
 | 9 | Immich | 2 | #8631, #10362 |
 | 10 | Minecraft server | 2 | #8143, #11012, #11051 |
 | 11 | MiniDLNA | 2 | #8071, #8867, #8870 |
 | 12 | Pi-hole | 2 | #8847, #8867, #8999 |
 | 13 | Radarr | 2 | #8358, #10418 |
 | 14 | Sonarr | 2 | #8358, #10418 |
 | 15 | Chrome Remote Desktop | 1 | #10406 |
 | 16 | CloudPanel | 1 | #8358 |
 | 17 | Cloudflare Tunnel | 1 | #8073, #8079 |
 | 18 | Conan Exiles server | 1 | #8715, #8731 |
 | 19 | CyberPanel | 1 | #8356, #8358 |
 | 20 | FTP | 1 | #11012 |
 | 21 | GNS3 | 1 | #8849, #8867, #9001 |
 | 22 | LDAP | 1 | #10362 |
 | 23 | Mealie | 1 | #8358 |
 | 24 | Media server | 1 | #8474 |
 | 25 | Syncthing | 1 | #8867, #8999 |
 | 26 | Tailscale | 1 | #10533 |
 | 27 | Transmission | 1 | #8049 |
 | 28 | VPN | 1 | #8330 |
 | 29 | WordPress | 1 | #8356, #8358 |
 | 30 | camera recording | 1 | #10254 |
 **3b Run by experienced people** (top 30)
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | Plex | 28 | #7961, #7967, #8014 |
 | 2 | Home Assistant | 23 | #7966, #8062, #8109 |
 | 3 | qBittorrent | 22 | #8014, #8020, #8061 |
 | 4 | Samba | 21 | #7973, #8014, #8066 |
 | 5 | Jellyfin | 14 | #8982, #9118, #9135 |
 | 6 | Nextcloud | 9 | #8020, #8022, #8109 |
 | 7 | OPNsense | 8 | #7977, #8491, #8527 |
 | 8 | Sonarr | 8 | #7967, #8014, #8733 |
 | 9 | Transmission | 8 | #8022, #8470, #8507 |
 | 10 | AdGuard Home | 7 | #7966, #8109, #8966 |
 | 11 | Pi-hole | 7 | #8014, #8062, #8549 |
 | 12 | Kodi | 6 | #8014, #8239, #8802 |
 | 13 | Radarr | 6 | #7967, #8014, #8733 |
 | 14 | Tailscale | 6 | #8527, #9241, #9248 |
 | 15 | Minecraft server | 5 | #7954, #8266, #8637 |
 | 16 | Portainer | 5 | #8963, #9529, #9541 |
 | 17 | Cloudflare Tunnel | 4 | #8082, #8522, #8528 |
 | 18 | Jackett | 4 | #8733, #9719, #9731 |
 | 19 | MiniDLNA | 4 | #8014, #8109, #8540 |
 | 20 | NFS | 4 | #8510, #8549, #8808 |
 | 21 | Nginx Proxy Manager | 4 | #7967, #8965, #9128 |
 | 22 | Proxmox Backup Server | 4 | #9289, #9364, #9408 |
 | 23 | WireGuard | 4 | #7967, #8109, #10162 |
 | 24 | Bazarr | 3 | #8733, #9826, #10605 |
 | 25 | Deluge | 3 | #7967, #8397, #8540 |
 | 26 | ESPHome | 3 | #8109, #8985, #10334 |
 | 27 | Frigate | 3 | #9429, #9853, #10175 |
 | 28 | Netdata | 3 | #8029, #9850, #10536 |
 | 29 | Prowlarr | 3 | #9693, #9731, #9737 |
 | 30 | UniFi Controller | 3 | #8539, #8664, #8963 |
 **3c Wanted by anyone, grouped by job** (for the catalog gap; ≥ 3 people):
 DLNA media server (MiniDLNA, Serviio…) 7 · router/firewall (OPNsense, pfSense, OpenWrt) 6 ·
 ad/DNS blocker (Pi-hole, AdGuard Home) 5 · camera recorder / NVR (Frigate, ZoneMinder, MotionEye…) 5.
 Single-app counts: `build.py` table 3c.
 ## 4 Hardware (approximate — see the banner)
 **4a Owned by beginners**
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | synology | 7 | #8179, #8631, #9240 |
 | 2 | diy-build | 6 | #8783, #8795, #10682 |
 | 3 | minipc-used-business | 6 | #8176, #8285, #8301 |
 | 4 | router-device | 5 | #7965, #8037, #8047 |
 | 5 | n100-class | 4 | #7985, #8005, #8154 |
 | 6 | thin-client | 4 | #8166, #9626, #10159 |
 | 7 | business-desktop-used | 3 | #8046, #8285, #10254 |
 | 8 | old-desktop | 2 | #7952, #7964, #8330 |
 | 9 | old-laptop | 2 | #9145, #9266 |
 | 10 | other | 2 | #8240, #10736, #10817 |
 | 11 | other-nas-box | 1 | #7952, #7964, #7981 |
 | 12 | qnap | 1 | #10525, #10533 |
 | 13 | rack-server | 1 | #10856 |
 | 14 | tower-server-used | 1 | #9194 |
 **4b Planned by beginners**
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | diy-build | 15 | #8356, #8483, #8731 |
 | 2 | business-desktop-used | 5 | #8201, #8474, #8665 |
 | 3 | n100-class | 4 | #8183, #8999, #9003 |
 | 4 | synology | 4 | #7952, #8285, #8847 |
 | 5 | minipc-used-business | 3 | #8474, #8665, #8673 |
 | 6 | rack-server | 3 | #10243, #10282, #10368 |
 | 7 | other | 2 | #9251, #9266, #10306 |
 | 8 | other-nas-box | 2 | #8847, #10854 |
 | 9 | old-desktop | 1 | #8367 |
 | 10 | raspberry-pi-5 | 1 | #10362, #10368 |
 | 11 | raspberry-pi-other | 1 | #8825 |
 **4c Recommended by anyone** (top 10)
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | business-desktop-used | 17 | #8157, #8163, #8169 |
 | 2 | diy-build | 17 | #7968, #8150, #8227 |
 | 3 | n100-class | 17 | #8144, #8155, #8164 |
 | 4 | minipc-used-business | 11 | #8145, #8148, #8234 |
 | 5 | synology | 9 | #7955, #7970, #7972 |
 | 6 | tower-server-used | 7 | #8222, #8226, #8486 |
 | 7 | old-desktop | 5 | #8026, #8027, #8030 |
 | 8 | workstation-used | 5 | #7948, #8115, #9356 |
 | 9 | other-nas-box | 3 | #9395, #10379, #10643 |
 | 10 | raspberry-pi-4 | 3 | #7959, #8828, #10515 |
 **Budgets stated** (latest per person, HUF): everyone n=48, min 1,000, median 61,000, max 800,000;
 beginners n=15, min 1,000, median 60,000, max 300,000. (The 1,000 is a monthly fee; most figures
 exclude disks.)
 ## 5 Platforms
 **5a What beginners want or tried**
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | TrueNAS | 14 | #8046, #8057, #8149 |
 | 2 | Debian/Ubuntu plain | 11 | #8056, #8059, #8149 |
 | 3 | Docker | 10 | #7981, #8083, #8356 |
 | 4 | Proxmox | 10 | #8149, #8862, #9589 |
 | 5 | Windows | 9 | #8143, #8176, #8516 |
 | 6 | OpenMediaVault | 8 | #7985, #8049, #8050 |
 | 7 | Xpenology | 8 | #8070, #8071, #8990 |
 | 8 | Synology DSM | 6 | #7952, #8179, #9240 |
 | 9 | Unraid | 6 | #8356, #8363, #8796 |
 | 10 | Home Assistant OS | 4 | #8008, #8012, #8579 |
 | 11 | CasaOS | 3 | #9145, #10225, #10403 |
 | 12 | ZimaOS | 2 | #10403, #10416, #10525 |
 **5b What experienced people recommend** (top 12)
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | Proxmox | 23 | #8062, #8065, #8097 |
 | 2 | OpenMediaVault | 20 | #8048, #8051, #8060 |
 | 3 | Docker | 16 | #8023, #8051, #8061 |
 | 4 | Debian/Ubuntu plain | 12 | #7959, #7962, #8053 |
 | 5 | TrueNAS | 12 | #7962, #8148, #8206 |
 | 6 | Unraid | 9 | #8401, #8403, #8404 |
 | 7 | Xpenology | 9 | #7962, #8023, #8068 |
 | 8 | Windows | 7 | #8052, #8053, #8272 |
 | 9 | CasaOS | 5 | #7959, #8102, #8106 |
 | 10 | Home Assistant OS | 4 | #8010, #8985, #9479 |
 | 11 | Synology DSM | 4 | #7955, #7958, #7970 |
 | 12 | ZimaOS | 3 | #10445, #10885, #11037 |
 **5c What experienced people run** (top 11)
 | # | Value | People | Examples |
 |---|---|---|---|
 | 1 | Proxmox | 40 | #7967, #7982, #8044 |
 | 2 | Docker | 23 | #7960, #7967, #7980 |
 | 3 | Debian/Ubuntu plain | 18 | #7973, #8054, #8058 |
 | 4 | TrueNAS | 14 | #7954, #8015, #8016 |
 | 5 | OpenMediaVault | 13 | #7967, #8044, #8064 |
 | 6 | Windows | 10 | #8105, #8217, #8390 |
 | 7 | Xpenology | 10 | #7957, #7976, #8074 |
 | 8 | Unraid | 7 | #7980, #8029, #8522 |
 | 9 | Home Assistant OS | 5 | #9441, #9493, #9529 |
 | 10 | CasaOS | 3 | #8555, #8556, #9693 |
 | 11 | Synology DSM | 3 | #7966, #8966, #9229 |
 **5d Advised against** (anyone): Proxmox 8 people („overkill” for a home NAS, #9487, #11081) · TrueNAS 5 („you will tear your hair out”, #8364) · Docker 2 · Windows 2 · Xpenology 2.
 **5e Products that do part of what Felhom does** (people; tone + / 0 / −)
 | Product | People | + | 0 | − | How it is spoken of | Examples |
 |---|---|---|---|---|---|---|
 | Synology | 43 | 17 | 27 | 18 | the default „easy” answer; praised for comfort, attacked for price, weak CPUs and the 2025 own-brand-disk lock | #7955, #10352, #10468 |
 | CasaOS | 10 | 4 | 6 | 2 | recommended to beginners as „a few clicks for Plex, AdGuard, torrent” | #7959, #10219 |
 | ZimaOS | 7 | 4 | 3 | 1 | new in 2025; easy, but beginners hit share-mount problems | #10403, #10528, #10539 |
 | QNAP | 6 | 3 | 2 | 4 | second to Synology; trust doubts | #8843, #9386 |
 | Google Drive / OneDrive / Google Photos | 14 | 2 | 11 | 2 | the cloud the household already pays for; seldom attacked | #8342, #8474, #10679 |
 | Umbrel, Cosmos, Runtipi | 1 each | 0 | 1 | 0 | named once in a list | #9606 |
 | A paid helper / managed service | 1 | 1 | 0 | 0 | one writer would pay someone to set it up | #9716 |
 No paid managed home-server service other than Felhom was named in three years.
 ## 6 Pains (people)
 | # | Pain | Everyone | Beginners | Examples |
 |---|---|---|---|---|
 | 1 | power bill | 60 | 11 | #7953, #8133, #8143 |
 | 2 | noise | 43 | 8 | #7975, #8309, #8665 |
 | 3 | too complex | 39 | 13 | #7981, #8083, #8181 |
 | 4 | up-front cost | 38 | 7 | #8367, #8477, #8847 |
 | 5 | space | 38 | 8 | #7952, #7964, #8195 |
 | 6 | hardware failure | 29 | 8 | #7964, #8124, #8160 |
 | 7 | heat | 24 | 3 | #7975, #7993, #8572 |
 | 8 | time it costs | 22 | 7 | #8085, #8181, #8474 |
 | 9 | fear of losing data | 19 | 5 | #7964, #8344, #8353 |
 | 10 | family does not accept it / cannot use it | 18 | 1 | #8465, #8637, #8803 |
 | 11 | security fear | 17 | 1 | #8030, #8039, #8041 |
 | 12 | vendor lock-in | 15 | 2 | #8752, #10294, #10352 |
 | 13 | bad documentation | 14 | 2 | #7981, #8099, #8240 |
 | 14 | an update broke it | 14 | 0 | #7962, #8015, #8752 |
 | 15 | disk failure | 14 | 2 | #8113, #8351, #8990 |
 | 16 | remote-access confusion | 11 | 6 | #8037, #8040, #8073 |
 | 17 | fears the terminal | 10 | 5 | #8008, #8149, #8363 |
 | 18 | gave up | 10 | 4 | #8012, #8149, #8236 |
 | 19 | distrust of the cloud | 8 | 3 | #8296, #8581, #8841 |
 ## 7 Trend
 Most year-to-year moves are within noise (about 100 active people per full year; a move needs ≥ 2
 standard errors). Four exceed it (power bill 2.3 SE, lock-in 2.6 SE, self-built 3.5 SE; ZimaOS is new):
 | Change | 2023 Q4 | 2024 | 2025 | 2026 (to Oct) |
 |---|---|---|---|---|
 | Power bill named, share of active people | 35 % | 33 % | 19 % | 18 % |
 | Vendor lock-in named (Synology disk lock, TrueNAS app change) | 0 % | 2 % | 11 % | 3 % |
 | ZimaOS named | 0 | 0 | 4 people | 3 people |
 | Self-built machines, share of active people | 20 % | 28 % | 33 % | 50 % |
 No trend in the beginner questions themselves: the same seven clusters appear every year.
@@ -0,0 +1,85 @@
 # METHOD — forum-research-2026-10-12
 | Item | Value |
 |---|---|
 | Source | `https://prohardver.hu/tema/hazi_szerver_epitese/`, public, no login, no account |
 | Window | posts dated 2023-10-12 00:00 or later: **#7943 (2023-10-12 17:01) → #11092 (2026-10-12 17:56)** |
 | Fetch | 2026-10-12 21:53–21:56 CEST; `fetch.py`; one request at a time, 5 s apart (robots.txt asks 1 s); browser UA; cookie jar; 34 pages (hsz_7801-7900 … hsz_11101-11200); every answer 200; no block, captcha or login wall |
 | robots.txt | read 2026-10-12 before fetching: `User-agent: *`, `crawl-delay: 1`, disallows only `/*/keres.php?*offset=` — unchanged from the spec |
 | Cache | `~/research-cache/prohardver-hazi-szerver/` on DooPlex, outside every repo: raw pages, `posts.jsonl`, salt, tags, spot files. **Not in git.** |
 | Posts | 3,150 numbers in the window; 2,909 present; 241 missing (#9874–#10108 the 2025 outage, #10118–10120, #8571, #9230, #11073); #11091 (Felhom's own announcement) excluded → **2,908 analysed** |
 | People | 195 distinct writers (`author_id` = first 12 hex of sha256(salt + username); salt local only) |
 ## Parser controls (`parse.py`, all PASS on the final run)
 1. **Counts:** every full page yields 100 posts, numbers contiguous; hsz_7801 99, hsz_8501 99, hsz_9201
   99 (one post each removed by the site), hsz_9801 73 and hsz_10101 89 (the outage edges), two pages 0,
   hsz_11001 91 (newest). The last request (11101-11200) redirected to 11001-11100; duplicates kept once.
 2. **Five random posts by eye** (seed 20261012): #9610, #7975, #8149, #7948, #8119 — own text whole,
   quoted parent absent in all five, checked against the raw page.
 3. **#8099** yields its own Proxmox 7.x complaint; the #8096 Raspberry Pi 4 → i5-6500T text is absent;
   and #8096 itself carries that text (the control can fail).
 **Four parser defects found by these controls and fixed before tagging:** posts with no parent
 (`data-rplid=""`) were dropped (85 → 100 per page); edited posts (pencil icon inside `<time>`) lost
 their date; this year's dates have no year („aug. 9.”, „ma”, „tegnap”) and were normalised against the
 fetch day; **hand-pasted quotes** — users copy the parent's words into their own text, often in
 `<small><b><i>` — were removed line by line and sentence by sentence when ≥ 20/25 characters match the
 parent (142 removed; quote-leak sweep 17 → 0). A pasted fragment shorter than that, or with no space
 after its full stop, can survive (seen once, #8173).
 ## Tagging
 Schema: `TAGGING-BRIEF.md` (v1 for all columns; v2 for `role`, `hardware`, `hardware_status`).
 Nine helper agents, one block of whole pages each, no overlap, the same brief and the full fence list.
 Columns added beyond the spec: `platform_status` (needed for table 5), `turnkey` (table 5e), and
 `thread` in `tags.csv`. A `gist` (≤ 12 own English words) was used for clustering and stays in the cache.
 **Beginner** = a person with at least one post whose role is `beginner-asks`: says „kezdő”, no Linux,
 lost in commands, a first server, a basic question, or is told so (`beginner_evidence`).
 **Threads:** walk up `reply_to` to the first ask; follow-ups by the same asker join; a lost-link
 follow-up with no question type joins the asker's previous thread within 30 days. 308 threads, 82 beginner.
 **Clusters:** the 82 beginner threads were grouped by reading their summaries (`clusters.json`, cache).
 **Pain tags used (47):** power-bill · noise · cost-upfront · too-complex · space · hardware-failure · heat ·
 time-cost · security-fear · data-loss-fear · family-acceptance · update-broke-it · vendor-lock-in ·
 remote-access-confusion · bad-docs · disk-failure · gave-up · fears-terminal · cloud-distrust · cloud-cost ·
 install-error · data-loss-happened · fears-docker · usb-enclosure-distrust · subscription-cost · dust ·
 isp-cgnat · update-fear · distrusts-chinese-hardware · stale-plugins · device-sprawl · choice-overload ·
 vpn-hassle · wants-gui · avoids-containers · lost-in-commands · dynamic-ip-outage · dislikes-linux ·
 weak-hardware · bad-firmware · underpowered-hardware · raid-capacity-loss · install-failed ·
 wants-hungarian-guide · update-burden · end-of-support · power-outage.
 ## Spot-check (spec 9.6) — `spotcheck.py`
 The session re-tagged a random 146 posts (5.0 %, seed 77) itself, blind to the helper tags.
 | Column | Round 1 | Round 2 (v2 rules) | Note |
 |---|---|---|---|
 | role | 83.6 % | 80.8 % | **below 85 % twice** |
 | ↳ asks vs not asks | — | 92.5 % | the split every thread and cluster rests on |
 | ↳ beginner-asks vs not | — | 97.3 % | the split every „beginner” table rests on |
 | ↳ substantive vs other | — | 87.7 % | |
 | beginner (y/n) | 99.3 % | — | |
 | question_type | 95.2 % | — | |
 | hardware | 79.5 % | 81.5 % (class set 82.9 %; ≥ 1 class shared when both tagged: 87.1 %) | **below 85 % twice** |
 | hardware_status | 87.7 % | 83.6 % | |
 | apps · apps_status | 92.5 % · 96.6 % | — | |
 | platform · platform_status | 89.0 % · 90.4 % | — | |
 | pain | 85.6 % | — | 30 % where either side tagged one — pains are leads |
 | budget_huf · resolved | 97.3 % · 97.9 % | — | |
 Round 1 also caught four of the session's own spot rows with a shifted field (fixed before scoring).
 **Not followed: a third re-tag round.** Round 2's rules moved the disagreement rather than removed it —
 it sits on „other vs answers” for chat posts and on whether a machine mentioned in passing counts.
 Neither boundary changes a table: the decisions the tables use pass (92.5 %, 97.3 %), and hardware
 counts carry an „approximate” banner in `FINDINGS.md`.
 ## Limits
 - A **hobby forum**: self-selected and technical; a household buyer does not post here.
 - **Hungarian only**; one topic; 195 people. Counts under ~5 people are anecdotes.
 - 2023 covers Q4 only; 2026 runs to 12 October. The 2025 outage removed 235 posts (between 2025-04-28 and 2025-07-12).
 - `resolved` is mostly unknown — askers seldom report back.
 - Tags are an AI reading of short posts; every claim cites post numbers so a person can check it.
 - Privacy: no username, `author_id`, salt or post text is in git; the cache stays on DooPlex.
@@ -0,0 +1,79 @@
 # REPORT — forum research, 2026-10-12 (night)
 Read-only research. Baselines read 2026-10-12 21:50: `felhom.eu` `c4d1b0ff98`, `app-catalog-felhom.eu`
 `cb4ececf1f` (both as the spec said). Architecture read for the area: none governs research; the
 hardware facts came from `website/gyik.html`, `website/letoltes.html`, `website/technologiak.html`.
 ## 1 Did it work
 Yes. 34 pages fetched politely (5 s apart, all HTTP 200, no block). 2,908 posts in the window
 (#7943–#11092) from 195 people; 82 beginner threads from 54 people. 241 post numbers are missing on
 the forum itself (235 of them the 2025 outage, #9874–#10108). #11091 is Felhom's own announcement and
 is left out of every count.
 ## 2 The five biggest beginner questions
 1. Which system do I install — TrueNAS, OMV, Proxmox, Xpenology or Windows? (20 threads)
 2. Which small, low-power machine should I buy? (16)
 3. Disks: external USB drives, powering drives, adapters, RAID (12)
 4. How do I install or connect an app? (11)
 5. Should I buy a Synology or build or reuse a PC? (10)
 ## 3 The ten most wanted apps (people, anyone wanting)
 Plex 24 · Home Assistant 15 · qBittorrent 14 · Jellyfin 11 · Samba 11 · Nextcloud 7 · MiniDLNA 5 ·
 Pi-hole 5 · Emby 4 · Immich 4. Not in our catalog: **a DLNA server** (7 people when grouped by job) and
 **an ad/DNS blocker** (5). Also wanted by ≥ 3 and missing: a camera recorder (5). Samba is built in.
 ## 4 The three hardware classes beginners own most (approximate)
 Synology 7 people · self-built PC 6 · used business mini PC 6.
 ## 5 The three biggest pains (people, everyone)
 Power bill 60 · noise 43 · too complex 39. Among beginners only: too complex 13, power bill 11.
 ## 6 Recommendations (FELHOM.md §5; none acted on)
 1. A „which machine works” page: x86-64 only, a RAM and disk floor, and „not a Synology, Pi or router disk”.
 2. Say plainly: „you do not choose an operating system — we do”.
 3. Offer „bring your old office PC”, with a check of the machine, as the cheap entry.
 4. Put measured power draw and noise for the recommended box on the website.
 5. Add a DLNA media server to the catalog.
 6. Decide whether an ad/DNS blocker belongs on the box.
 7. Decide whether a camera recorder belongs in the catalog.
 8. Position against Synology, the one product people name.
 9. Say in one line how remote access works and whether a router port or VPN app is needed.
 10. Show family use: one link per app for a household member.
 ## 7 Parser controls and spot-check
 All three parser controls pass. They caught four parser defects first (posts with no parent dropped,
 edited posts' dates lost, this year's dates with no year, hand-pasted quotes kept) — all fixed before
 tagging. Spot-check: 146 posts (5 %) re-tagged by the session, blind. Round 1: every column ≥ 85 %
 except role 83.6 % and hardware 79.5 %. The rules were tightened and both re-run (round 2: role 80.8 %,
 hardware 81.5 %). **Not followed: a third round** — the disagreement moved rather than shrank, on
 boundaries no table uses; the splits the tables rest on agree 92.5 % (asks) and 97.3 % (beginner asks),
 and hardware counts carry an „approximate” banner.
 ## 8 Claims in the spec that turned out wrong
 - **`templates.json` is not the catalog.** It is the LEGACY Portainer index (10 apps); the live catalog
  is the `templates/` folder (62 apps). The gap table uses the folder.
 - **„About 3,100 posts”**: 3,150 numbers, 2,909 present (241 missing on the forum).
 - **„Every full page yields exactly 100 posts”**: true only after fixing the parser; three full pages
  hold 99 (one post removed by the site each).
 - The URLs, the 302 cookie check, the #8006 and #8100 dates, the highest post (#11092), robots.txt and the
  #8099/#8096 control were all **correct**.
 ## 9 Commit, register, STATUS
 Register: **118 rows before, 120 after; 2 opened (R-990, R-991), 0 closed.**
 - R-990 (P4): the operator reads `FELHOM.md` §5 and rules on each recommendation.
 - R-991 (P3, Install & onboarding): no buyer-readable hardware floor, while 8 of 33 beginners own only
  machines the installer cannot run on. The spec allowed a second row for exactly this case.
 STATUS line added under „Monday 2026-10-12 (night): what Hungarian beginners ask…”.
 `CHANGELOG.md` not touched: this repo's changelogs are per area (hub, scripts, website, marketing), none
 covers an audit, and the spec limited the change to the audit folder, one row and one STATUS line.
 Shared `REPORT.md` not touched (parallel sessions); this file is the report. No instruction file was
 changed; the system poster's facts are unchanged. Provisioned nothing (machine, host, hub).
@@ -0,0 +1,91 @@
 # Tagging brief — forum-research-2026-10-12
 The one schema every tagger (helper agent or the session itself) used. Spec: §7 C of the task.
 Input rows: `num, date, year, reply_to, author_id, off, text, parent_author_id, parent_text`.
 `text` is the post's own words (quotes removed); `parent_text` is the direct parent's own words,
 for context only. `author_id` is a pseudonym (a hash) — use it only to see who is the same person.
 ## Rule zero
 **Every value comes from the post or its direct parent. Nothing is guessed.** An app, a device or a
 platform named only in `parent_text` is NOT counted for this post. Empty is a correct answer.
 ## Columns (CSV, header exactly in this order)
 `num,year,role,beginner_evidence,question_type,hardware,hardware_status,apps,apps_status,platform,platform_status,pain,budget_huf,resolved,turnkey,gist`
 Multi-value cells are `;`-separated, no spaces around `;`. Every cell is quoted by the CSV writer.
 | Column | Values |
 |---|---|
 | `num`, `year` | copy from input |
 | `role` | `beginner-asks` / `experienced-asks` / `answers` / `own-setup` / `other`. Asks = the post's main act is a question for help or advice. `own-setup` = mainly describes what the writer runs or built. `answers` = mainly helps someone else. `other` = joke, thanks, off-topic chat, no own content. |
 | `beginner_evidence` | only for a beginner. Short English reason: `says-no-linux`, `says-kezdo` (calls self a beginner), `basic-question`, `told-beginner`, `first-server`, `lost-in-commands`. Several allowed. A person is a beginner when they say so, ask something basic (what is Docker, which OS to start with, how to reach it from outside, first build), or are told so. An experienced person asking a narrow technical question is `experienced-asks`. |
 | `question_type` | for an `-asks` post only, one or more of: `hardware-choice`, `os-platform-choice`, `app-choice`, `remote-access`, `backup`, `storage-raid`, `power-noise`, `install-help`, `broken-fix`, `security`, `cost`, `other` |
 | `hardware` | normalised classes, each optionally with the model in brackets: `minipc-used-business (i5-6500T)`, `n100-minipc`, `minipc-new-other`, `raspberry-pi-4`, `raspberry-pi-5`, `raspberry-pi-other`, `old-desktop`, `old-laptop`, `workstation-used` (Z840, T5810…), `rack-server`, `tower-server-used` (HP Microserver, T30…), `diy-new-build`, `synology`, `qnap`, `other-nas-box` (Ugreen, Asustor, Terramaster…), `thin-client` (t620, t740, Wyse…), `sbc-other`, `router-device`, `other` |
 | `hardware_status` | `owns` / `plans` / `recommends` (one per post: the main one) |
 | `apps` | normalised app names (canonical spelling below) the writer wants / runs / recommends. Not infrastructure (Docker, Proxmox, Linux) — those go to `platform`. |
 | `apps_status` | `wants` / `runs` / `recommends` |
 | `platform` | `Proxmox`, `OpenMediaVault`, `TrueNAS`, `Unraid`, `Synology DSM`, `Xpenology`, `Debian/Ubuntu plain`, `CasaOS`, `ZimaOS`, `Umbrel`, `Home Assistant OS`, `Windows`, `Docker` (Docker named as the way they run apps), `QNAP QTS`, `other:<name>` |
 | `platform_status` | `wants` (plans/considers) / `tried` (used and left or is struggling) / `runs` / `recommends` / `advises-against` |
 | `pain` | kebab-case tags for what the writer fears or suffered. Start from: `fears-docker`, `fears-terminal`, `update-broke-it`, `data-loss-fear`, `data-loss-happened`, `remote-access-confusion`, `power-bill`, `noise`, `heat`, `gave-up`, `too-complex`, `time-cost`, `bad-docs`, `hardware-failure`, `disk-failure`, `security-fear`, `isp-cgnat`, `cost-upfront`, `cloud-distrust`, `cloud-cost`, `vendor-lock-in`, `space`, `family-acceptance`. A new tag is allowed when none fits; keep it short kebab-case. |
 | `budget_huf` | a number in HUF if the post states a budget or a price they will pay (e.g. „100e Ft” → 100000; „35k” → 35000). Euro: convert at 400. Empty otherwise. |
 | `resolved` | only on a `beginner-asks` post: `yes` if a LATER post by the same `author_id` in this block says it works / they did it; `no` if they say they gave up or it still fails; else `unknown`. |
 | `turnkey` | any ready-made or paid product doing what a managed home server does (CasaOS, ZimaOS, Umbrel, Synology, QNAP, Ugreen NAS, a paid cloud like Google One/iCloud/OneDrive, a managed service) named in the post's own text, with the writer's tone: `Name:+`, `Name:-`, `Name:0`. |
 | `gist` | at most 12 English words in YOUR OWN words: what the post asks or says. Never copy the Hungarian. Never a username. |
 ## Canonical app names
 Jellyfin, Plex, Emby, Kodi, qBittorrent, Transmission, Deluge, Sonarr, Radarr, Prowlarr, Jackett, Bazarr,
 Overseerr, Jellyseerr, Home Assistant, Node-RED, Zigbee2MQTT, Mosquitto, ESPHome, Frigate, Pi-hole,
 AdGuard Home, Nextcloud, Seafile, Syncthing, Immich, PhotoPrism, Vaultwarden, Bitwarden, Paperless-ngx,
 Nginx Proxy Manager, Traefik, Caddy, WireGuard, Tailscale, ZeroTier, OpenVPN, Cloudflare Tunnel,
 Portainer, Uptime Kuma, Grafana, Prometheus, InfluxDB, Samba, NFS, Duplicati, Restic, Borg, Kopia,
 UrBackup, Gitea, Nginx, Apache, MariaDB, PostgreSQL, Minecraft server, Ollama, Open WebUI, OpenWrt,
 pfSense, OPNsense, Mikrotik RouterOS, Zoneminder, Shinobi, Navidrome, Audiobookshelf, Calibre-web,
 FileBrowser, Homarr, Heimdall, Firefly III, Actual, Mealie, Joplin, Stirling-PDF, Wiki.js.
 Any other app: its usual English name.
 ## How to work
 - Tag every input row; one output row per input row, same `num`. No row skipped.
 - Off-topic flagged posts (`off=true`) are still tagged; most will be `other`.
 - A post with no real own text (only emoji, „+1”, thanks) → `role=other`, every other tag empty.
 - A link-only post: tag from the visible link text only. Do not open links.
 ## v2 — stricter rules for `role`, `hardware`, `hardware_status` (after the first spot-check)
 The first 5% spot-check agreed 83.6% on `role` and 79.5% on `hardware`, below the 85% bar. Both
 columns were re-tagged by every helper with these rules; the other columns kept their v1 values.
 **`role` — decide in this order; the first rule that fits wins:**
 1. **`other`** — no fact and no question about servers: thanks, agreement, a joke, a pointer to
   another topic, chat about something that is not a home server (heating, cars, politics). A short
   reaction that adds no fact is `other` even inside a technical thread.
 2. **`beginner-asks` / `experienced-asks`** — the post's main act is asking for help or advice for the
   writer's own situation, including an asker's follow-up that adds details or a new sub-question.
   A rhetorical or counter question inside a debate is not an ask (rule 4).
 3. **`own-setup`** — the post mainly reports the writer's own machine, build, purchase, result or
   experience, and is NOT framed as advice to someone (e.g. „I bought X, it runs Y").
 4. **`answers`** — every other post with at least one fact: advice, explanations, corrections,
   recommendations, „I use X and it works, so take X" (own experience used as advice = answers).
 **`hardware` classes — v2 list (the machine that is or would be the server; parts such as a PSU,
 case, switch, network card, disk or RAM are NOT hardware here):**
 - `minipc-used-business` — used Tiny / Mini / Micro / USFF business PCs (OptiPlex Micro, EliteDesk
  Mini, ThinkCentre Tiny, Fujitsu Q-series, NUC).
 - `business-desktop-used` — used SFF / MT / tower business desktops (OptiPlex SFF/MT, ProDesk/EliteDesk
  SFF/MT, ThinkCentre M SFF, Fujitsu Esprimo P/D).
 - `n100-class` — a new low-power x86 box or board: N100/N150/N5105/N6005/J4125/J6412 etc., mini PC
  or „NAS board" (CWWK, Topton, Beelink, Minisforum…).
 - `diy-build` — a machine assembled from parts by the writer (new or old parts), when it is not an
  n100-class board. A desktop the writer used as a desktop and turns into a server is `old-desktop`.
 - `old-desktop`, `old-laptop`, `workstation-used` (HP Z, Dell Precision/T-series, Lenovo P), 
  `tower-server-used` (HP Microserver, ML-series, Dell T-series server, Fujitsu Primergy tower),
  `rack-server`, `synology`, `qnap`, `other-nas-box`, `thin-client`, `raspberry-pi-4`,
  `raspberry-pi-5`, `raspberry-pi-other`, `sbc-other`, `router-device`, `other` (as v1).
 - Tag only machines named or clearly described in the post's own text. A machine named only in the
  parent is not tagged.
 **`hardware_status`:** `owns` (the writer has it now), `plans` (will buy/build), `recommends` (tells
 someone else to get it). One value per hardware item, `;`-separated in the same order as `hardware`.
@@ -0,0 +1,319 @@
 #!/usr/bin/env python3
 """Merge the helper tags, build threads, write tags.csv (committed) and print the FINDINGS tables.
 Read-only research (forum-research-2026-10-12). Stdlib only.
 Inputs (cache, never in git): posts.jsonl, tags/tags_*.csv, clusters.json (optional).
 Output: tags.csv here (num + tags only: no text, no author_id, no gist) and tables on stdout.
 Threads (spec 7 C): walk up reply_to from a post; the first ancestor-or-self whose role ends in
 "-asks" opens a thread; keep walking while the next "-asks" ancestor is by the SAME person (a
 follow-up question is the same thread). A post with no "-asks" ancestor belongs to no thread.
 A thread is a beginner thread when its opening post is beginner-asks.
 Thread resolved = yes if any post by the asker in it is tagged yes, else no if any is no, else unknown.
 "People" = distinct author_id. Usage: python3 build.py [--write-tags]
 """
 import collections
 import csv
 import glob
 import json
 import os
 import statistics
 import sys
 HERE = os.path.dirname(os.path.abspath(__file__))
 CACHE = os.path.expanduser("~/research-cache/prohardver-hazi-szerver")
 COLS = ["num", "year", "role", "beginner_evidence", "question_type", "hardware", "hardware_status",
        "apps", "apps_status", "platform", "platform_status", "pain", "budget_huf", "resolved", "turnkey"]
 # Spelling variants the taggers produced -> one canonical name (filled from the merged tag values).
 APP_ALIASES = {
    "qbittorrent": "qBittorrent", "qbit": "qBittorrent", "homeassistant": "Home Assistant",
    "home assistant os": "Home Assistant", "pihole": "Pi-hole", "pi-hole": "Pi-hole",
    "adguard": "AdGuard Home", "adguardhome": "AdGuard Home", "adguard home": "AdGuard Home",
    "nginx proxy manager": "Nginx Proxy Manager", "npm": "Nginx Proxy Manager",
    "paperless": "Paperless-ngx", "paperless-ngx": "Paperless-ngx", "wireguard": "WireGuard",
    "tailscale": "Tailscale", "zerotier": "ZeroTier", "openvpn": "OpenVPN", "plex": "Plex",
    "plex media server": "Plex", "jellyfin": "Jellyfin", "nextcloud": "Nextcloud", "immich": "Immich",
    "vaultwarden": "Vaultwarden", "bitwarden": "Bitwarden", "samba": "Samba", "smb": "Samba",
    "transmission": "Transmission", "cloudflare tunnel": "Cloudflare Tunnel",
    "cloudflared": "Cloudflare Tunnel", "minecraft": "Minecraft server",
    "minecraft server": "Minecraft server", "frigate": "Frigate", "portainer": "Portainer",
    "syncthing": "Syncthing", "photoprism": "PhotoPrism", "uptime kuma": "Uptime Kuma",
    "uptime-kuma": "Uptime Kuma", "opnsense": "OPNsense", "pfsense": "pfSense",
    "sonarr": "Sonarr", "radarr": "Radarr", "prowlarr": "Prowlarr", "jackett": "Jackett",
    "kodi": "Kodi", "emby": "Emby", "minidlna": "MiniDLNA", "dlna": "MiniDLNA",
 }
 # #11091 is Felhom's own announcement in the topic (found by the block-9 tagger): not a forum
 # voice, so it is left out of every count and table.
 EXCLUDE = {11091}
 def split(v):
    return [x.strip() for x in (v or "").split(";") if x.strip()]
 def app_name(a):
    return APP_ALIASES.get(a.strip().lower(), a.strip())
 def hw_class(h):
    return h.split("(")[0].strip().lower()
 def load():
    posts = {}
    for line in open(os.path.join(CACHE, "posts.jsonl"), encoding="utf-8"):
        p = json.loads(line)
        posts[p["num"]] = p
    tags = {}
    for f in sorted(glob.glob(os.path.join(CACHE, "tags", "tags_*.csv"))):
        for r in csv.DictReader(open(f, encoding="utf-8")):
            n = int(r["num"])
            if n in tags:
                sys.exit("duplicate tag row #%d" % n)
            tags[n] = r
    # v2 re-tag (TAGGING-BRIEF.md "v2"): role, hardware, hardware_status replace the v1 values.
    v2 = {}
    for f in sorted(glob.glob(os.path.join(CACHE, "tags_v2", "v2_*.csv"))):
        for r in csv.DictReader(open(f, encoding="utf-8")):
            v2[int(r["num"])] = r
    if set(v2) != set(tags):
        sys.exit("v2 coverage differs from v1: %s" % sorted(set(v2) ^ set(tags))[:20])
    for n, r in v2.items():
        for c in ("role", "hardware", "hardware_status"):
            tags[n][c] = r[c]
    for n in EXCLUDE:
        posts.pop(n, None)
        tags.pop(n, None)
    missing = sorted(set(posts) - set(tags))
    extra = sorted(set(tags) - set(posts))
    if missing or extra:
        sys.exit("tag coverage broken: missing=%s extra=%s" % (missing[:20], extra[:20]))
    return posts, tags
 def threads(posts, tags):
    asks = lambda n: n in tags and tags[n]["role"].endswith("-asks")
    tid = {}
    for n in posts:
        cur, opener = n, None
        seen = set()
        while cur in posts and cur not in seen:
            seen.add(cur)
            if asks(cur):
                if opener is None:
                    opener = cur
                elif posts[cur]["author_id"] == posts[opener]["author_id"]:
                    opener = cur
                else:
                    break
            cur = posts[cur]["reply_to"]
        if opener is not None:
            tid[n] = opener
    # An asker's follow-up that lost its reply link (no question_type, so it adds to an earlier
    # question) joins that person's previous thread when it is at most 30 days older.
    import datetime
    day = lambda n: datetime.date.fromisoformat(posts[n]["date"][:10])
    openers = sorted(set(tid.values()))
    remap = {}
    for o in openers:
        if tags[o]["question_type"].strip():
            continue
        prev = [p for p in openers if p < o and posts[p]["author_id"] == posts[o]["author_id"]
                and (day(o) - day(p)).days <= 30]
        if prev:
            remap[o] = remap.get(prev[-1], prev[-1])
    return {n: remap.get(t, t) for n, t in tid.items()}
 def people_by(rows, key):
    c = collections.defaultdict(set)
    for author, val in rows:
        c[val].add(author)
    return sorted(((len(v), k) for k, v in c.items()), key=lambda t: (-t[0], t[1]))
 def table(title, ranked, top=30, examples=None):
    print("\n### %s" % title)
    print("| # | Value | People |" + (" Examples |" if examples else ""))
    print("|---|---|---|" + ("---|" if examples else ""))
    for i, (cnt, val) in enumerate(ranked[:top], 1):
        ex = (" %s |" % ", ".join("#%d" % e for e in examples.get(val, [])[:3])) if examples else ""
        print("| %d | %s | %d |%s" % (i, val, cnt, ex))
 def main():
    posts, tags = load()
    if "--write-tags" in sys.argv:
        with open(os.path.join(HERE, "tags.csv"), "w", encoding="utf-8", newline="") as f:
            w = csv.writer(f)
            w.writerow(COLS + ["thread"])
            tid = threads(posts, tags)
            for n in sorted(tags):
                w.writerow([tags[n].get(c, "") for c in COLS] + [tid.get(n, "")])
        print("wrote tags.csv (%d rows)" % len(tags))
    tid = threads(posts, tags)
    au = {n: posts[n]["author_id"] for n in posts}
    role = {n: tags[n]["role"] for n in tags}
    # --- 1. Numbers
    openers = sorted(set(tid.values()))
    beg_threads = [t for t in openers if role[t] == "beginner-asks"]
    members = collections.defaultdict(list)
    for n, t in tid.items():
        members[t].append(n)
    def resolved(t):
        vals = [tags[n]["resolved"] for n in members[t] if au[n] == au[t] and tags[n]["resolved"]]
        return "yes" if "yes" in vals else ("no" if "no" in vals else "unknown")
    res = collections.Counter(resolved(t) for t in beg_threads)
    # A beginner = a person with at least one post whose v2 role is beginner-asks (the column the
    # spot-check agreed on at 97%); v1 beginner_evidence is kept in tags.csv as the reason.
    beginners = {au[n] for n in tags if role[n] == "beginner-asks"}
    print("## 1 Numbers")
    print("posts=%d people=%d first=#%d last=#%d" % (len(posts), len(set(au.values())), min(posts), max(posts)))
    print("roles: %s" % dict(collections.Counter(role.values())))
    print("threads=%d beginner threads=%d beginner people=%d resolved=%s" %
          (len(openers), len(beg_threads), len(beginners), dict(res)))
    by_year = collections.Counter(tags[t]["year"] for t in beg_threads)
    print("beginner threads by year: %s; posts by year: %s" %
          (dict(sorted(by_year.items())), dict(sorted(collections.Counter(t["year"] for t in tags.values()).items()))))
    # --- 2. Beginner question types (clusters come from clusters.json if present)
    qt = collections.Counter()
    for t in beg_threads:
        for q in split(tags[t]["question_type"]):
            qt[q] += 1
    print("\n## 2 Beginner thread question types: %s" % qt.most_common())
    cl_path = os.path.join(CACHE, "clusters.json")
    if os.path.exists(cl_path):
        cl = json.load(open(cl_path, encoding="utf-8"))
        ranked = sorted(cl.items(), key=lambda kv: -len(kv[1]))
        print("| # | Cluster | Threads | Examples |")
        for i, (k, v) in enumerate(ranked, 1):
            ts = [t for t in v if t in beg_threads]
            print("| %d | %s | %d | %s |" % (i, k, len(ts), ", ".join("#%d" % t for t in ts[:3])))
            rec = collections.Counter()
            for t in ts:
                for n in members[t]:
                    if role[n] != "answers":
                        continue
                    for col, st in (("platform", "platform_status"), ("hardware", "hardware_status"),
                                    ("apps", "apps_status")):
                        vals, sts = split(tags[n][col]), split(tags[n][st])
                        for j, x in enumerate(vals):
                            stx = sts[j] if len(sts) == len(vals) else (sts[0] if sts else "")
                            if stx == "recommends":
                                rec["%s:%s" % (col, hw_class(x) if col == "hardware" else app_name(x))] += 1
            print("    answers recommended (posts): %s" % rec.most_common(8))
        unassigned = [t for t in beg_threads if not any(t in v for v in cl.values())]
        print("unassigned beginner threads: %d %s" % (len(unassigned), unassigned[:30]))
    # --- 3. Apps
    def ent_rows(col, status_col, status_pred, author_pred, norm=lambda x: x):
        rows, ex = [], collections.defaultdict(list)
        for n, t in tags.items():
            if not author_pred(n):
                continue
            vals, sts = split(t[col]), split(t[status_col])
            for i, v in enumerate(vals):
                # Status cells are either one value for the whole post or one per item, aligned.
                st = sts[i] if len(sts) == len(vals) else (sts[0] if sts else "")
                if status_col != "role" and not status_pred(st):
                    continue
                v = norm(v)
                rows.append((au[n], v))
                if n not in ex[v]:
                    ex[v].append(n)
        return people_by(rows, 1), ex
    is_beg = lambda n: au[n] in beginners
    not_beg = lambda n: au[n] not in beginners
    r, ex = ent_rows("apps", "apps_status", lambda s: s == "wants", is_beg, app_name)
    table("3a Apps WANTED by beginners (people)", r, examples=ex)
    r, ex = ent_rows("apps", "apps_status", lambda s: s == "runs", not_beg, app_name)
    table("3b Apps RUN by experienced people (people)", r, examples=ex)
    r, ex = ent_rows("apps", "apps_status", lambda s: s == "wants", lambda n: True, app_name)
    table("3c Apps WANTED by anyone (people) - for the catalog gap", r, top=60, examples=ex)
    r, ex = ent_rows("apps", "apps_status", lambda s: s == "recommends", lambda n: True, app_name)
    table("3d Apps RECOMMENDED by anyone (people)", r, top=30, examples=ex)
    # --- 4. Hardware
    r, ex = ent_rows("hardware", "hardware_status", lambda s: s == "owns", is_beg, hw_class)
    table("4a Hardware OWNED by beginners (people)", r, examples=ex)
    r, ex = ent_rows("hardware", "hardware_status", lambda s: s == "plans", is_beg, hw_class)
    table("4b Hardware PLANNED by beginners (people)", r, examples=ex)
    r, ex = ent_rows("hardware", "hardware_status", lambda s: s == "owns", lambda n: True, hw_class)
    table("4c Hardware OWNED by anyone (people)", r, examples=ex)
    r, ex = ent_rows("hardware", "hardware_status", lambda s: s == "recommends", lambda n: True, hw_class)
    table("4d Hardware RECOMMENDED (people)", r, examples=ex)
    budgets = collections.defaultdict(list)
    for n, t in tags.items():
        try:
            b = int(float(t["budget_huf"]))
        except (ValueError, TypeError):
            continue
        if b > 0:
            budgets[au[n]].append((n, b))
    per_person = [max(v, key=lambda x: x[0])[1] for v in budgets.values()]  # latest stated per person
    beg_budget = [max(v, key=lambda x: x[0])[1] for a, v in budgets.items() if a in beginners]
    for name, xs in (("all people", per_person), ("beginners", beg_budget)):
        if xs:
            print("budget %s: n=%d min=%d median=%d max=%d" % (name, len(xs), min(xs), statistics.median(xs), max(xs)))
    # --- 5. Platforms
    for title, pred, who in (("5a Platforms beginners want or tried", lambda s: s in ("wants", "tried", "runs"), is_beg),
                             ("5b Platforms recommended by experienced people", lambda s: s == "recommends", not_beg),
                             ("5c Platforms run by experienced people", lambda s: s == "runs", not_beg),
                             ("5d Platforms advised against (anyone)", lambda s: s == "advises-against", lambda n: True)):
        r, ex = ent_rows("platform", "platform_status", pred, who)
        table(title, r, top=20, examples=ex)
    tk = collections.defaultdict(lambda: collections.defaultdict(set))
    tkex = collections.defaultdict(list)
    for n, t in tags.items():
        for x in split(t["turnkey"]):
            name, _, tone = x.rpartition(":")
            if not name:
                name, tone = x, "0"
            name = name.strip()
            tk[name][tone.strip() or "0"].add(au[n])
            tkex[name].append(n)
    print("\n### 5e Turnkey products named (people by tone)")
    print("| Product | People | + | 0 | - | Examples |")
    print("|---|---|---|---|---|---|")
    for name in sorted(tk, key=lambda k: -len(set().union(*tk[k].values()))):
        allp = set().union(*tk[name].values())
        print("| %s | %d | %d | %d | %d | %s |" % (name, len(allp), len(tk[name]["+"]), len(tk[name]["0"]),
                                                  len(tk[name]["-"]), ", ".join("#%d" % e for e in tkex[name][:3])))
    # --- 6. Pains
    r, ex = ent_rows("pain", "role", lambda s: True, lambda n: True)
    table("6a Pains, everyone (people)", r, top=40, examples=ex)
    r, ex = ent_rows("pain", "role", lambda s: True, is_beg)
    table("6b Pains, beginners only (people)", r, top=20, examples=ex)
    # --- 7. Trend: people per value per year for the top entities
    print("\n## 7 Trend (people per year)")
    years = ["2023", "2024", "2025", "2026"]
    ppl_year = collections.Counter()
    for n in tags:
        pass
    yr_people = {y: {au[n] for n in tags if tags[n]["year"] == y} for y in years}
    print("active people per year: %s" % {y: len(v) for y, v in yr_people.items()})
    for col, stat, norm in (("apps", None, app_name), ("hardware", None, hw_class), ("platform", None, lambda x: x),
                            ("pain", None, lambda x: x)):
        cnt = collections.defaultdict(lambda: collections.defaultdict(set))
        for n, t in tags.items():
            for v in split(t[col]):
                cnt[norm(v)][t["year"]].add(au[n])
        top = sorted(cnt, key=lambda k: -len(set().union(*cnt[k].values())))[:12]
        print("\n%s: value | %s" % (col, " | ".join(years)))
        for k in top:
            print("  %s | %s" % (k, " | ".join("%d (%.0f%%)" % (len(cnt[k][y]), 100.0 * len(cnt[k][y]) / max(1, len(yr_people[y])))
                                               for y in years)))
 if __name__ == "__main__":
    main()
@@ -0,0 +1,95 @@
 #!/usr/bin/env python3
 """Fetch the PROHARDVER! "Home server / hazi szerver epitese" topic pages, politely.
 Read-only research (forum-research-2026-10-12). Stdlib only.
 - One request at a time, at least DELAY seconds apart (robots.txt asks for 1 s; we use 5).
 - Cookie jar + redirects: the first request answers 302 -> ?_tc=1 (a cookie check).
 - Each page is cached as raw HTML outside git; a cached page is never fetched again.
 - STOPS on 403, 429, a captcha or a login wall (spec section 9.2). No retry around a block.
 Usage:
  python3 fetch.py FIRST_POST LAST_POST     # fetch every 100-post page covering the range
 Pages: hsz_<a>-<b>.html with a = 1 + 100*k, b = a + 99.
 """
 import http.cookiejar
 import os
 import sys
 import time
 import urllib.error
 import urllib.request
 BASE = "https://prohardver.hu/tema/hazi_szerver_epitese/"
 CACHE = os.path.expanduser("~/research-cache/prohardver-hazi-szerver")
 PAGES = os.path.join(CACHE, "pages")
 LOG = os.path.join(CACHE, "fetch.log")
 DELAY = 5.0
 UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36")
 BLOCK_MARKERS = ("captcha", "recaptcha", "hcaptcha")
 def log(msg):
    line = time.strftime("%Y-%m-%dT%H:%M:%S%z") + " " + msg
    print(line, flush=True)
    with open(LOG, "a", encoding="utf-8") as f:
        f.write(line + "\n")
 def page_name(first):
    return "hsz_%d-%d.html" % (first, first + 99)
 def count_posts(html):
    # A rough count for the log only; parse.py does the real work.
    return html.count('data-id="')
 def main():
    if len(sys.argv) != 3:
        sys.exit(__doc__)
    lo, hi = int(sys.argv[1]), int(sys.argv[2])
    os.makedirs(PAGES, exist_ok=True)
    jar = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar))
    opener.addheaders = [("User-Agent", UA),
                         ("Accept", "text/html,application/xhtml+xml"),
                         ("Accept-Language", "hu-HU,hu;q=0.9,en;q=0.5")]
    first = ((lo - 1) // 100) * 100 + 1
    last_req = 0.0
    while first <= hi:
        name = page_name(first)
        path = os.path.join(PAGES, name)
        if os.path.exists(path):
            log("CACHED %s" % name)
            first += 100
            continue
        wait = DELAY - (time.time() - last_req)
        if wait > 0:
            time.sleep(wait)
        last_req = time.time()
        url = BASE + name
        try:
            resp = opener.open(url, timeout=60)
            status, body, final = resp.status, resp.read(), resp.geturl()
        except urllib.error.HTTPError as e:
            log("HTTP %d %s -- STOP" % (e.code, url) if e.code in (403, 429)
                else "HTTP %d %s -- STOP (unexpected)" % (e.code, url))
            sys.exit(2)
        except Exception as e:  # network error: stop, do not hammer
            log("ERROR %s %r -- STOP" % (url, e))
            sys.exit(3)
        html = body.decode("utf-8", errors="replace")
        low = html.lower()
        if any(m in low for m in BLOCK_MARKERS) or "/belepes" in final:
            log("BLOCK? %s final=%s -- STOP" % (url, final))
            with open(path + ".blocked", "wb") as f:
                f.write(body)
            sys.exit(4)
        with open(path, "wb") as f:
            f.write(body)
        log("OK %d %s bytes=%d posts~%d final=%s" % (status, url, len(body), count_posts(html), final))
        first += 100
 if __name__ == "__main__":
    main()
@@ -0,0 +1,242 @@
 #!/usr/bin/env python3
 """Parse the cached topic pages into posts.jsonl (in the cache folder, never in git).
 Read-only research (forum-research-2026-10-12). Stdlib only.
 Per post: num, date, reply_to, author_id, off, text.
 - author_id = first 12 hex of sha256(salt + username). The salt is made once and kept in the cache
  folder; it is never committed.
 - text = the post's OWN words. The quoted parent sits in <div class="message-content-replied"> and is
  skipped whole; only <div class="rtif-content message-content"> outside it is kept.
 - off = the site marked the post off-topic (<li class="message-off">). A post with no parent has
  data-rplid="" (an earlier pattern that required digits silently dropped every such post).
 - Posts before WINDOW_START are dropped; a post number seen twice (a redirected page) is kept once.
 Then it runs the parser controls (spec 7 B) and prints them. Exit 1 if a control fails.
 Usage: python3 parse.py
 """
 import glob
 import hashlib
 import html
 import json
 import os
 import random
 import re
 import secrets
 import sys
 from html.parser import HTMLParser
 CACHE = os.path.expanduser("~/research-cache/prohardver-hazi-szerver")
 PAGES = os.path.join(CACHE, "pages")
 OUT = os.path.join(CACHE, "posts.jsonl")
 SALT_FILE = os.path.join(CACHE, "salt.txt")
 WINDOW_START = "2023-10-12 00:00:00"
 LI_RE = re.compile(r'<li (?:class="([^"]*)" )?data-id="(\d+)" data-rplid="(\d*)">')
 TIME_RE = re.compile(r'<time class="message-time">(.*?)</time>', re.S)
 TAG_RE = re.compile(r"<[^>]+>")
 # The site shows this year's dates without a year ("aug. 9., 12:34:56"), and "ma"/"tegnap" for
 # today/yesterday. FETCH_DAY is the day the pages were fetched (fetch.log).
 FETCH_DAY = (2026, 10, 12)
 HU_MONTHS = {"jan.": 1, "febr.": 2, "márc.": 3, "ápr.": 4, "máj.": 5, "jún.": 6, "júl.": 7,
             "aug.": 8, "szept.": 9, "okt.": 10, "nov.": 11, "dec.": 12}
 def norm_date(raw):
    """Return 'YYYY-MM-DD HH:MM:SS' or '' if the shape is unknown. Edited posts carry a pencil
    <span> inside <time>; the tags are stripped first (an earlier pattern dropped those posts)."""
    import datetime
    t = " ".join(TAG_RE.sub(" ", raw).split())
    if re.match(r"^\d{4}-\d\d-\d\d \d\d:\d\d:\d\d$", t):
        return t
    m = re.match(r"^(ma|tegnap) (\d\d:\d\d:\d\d)$", t)
    if m:
        d = datetime.date(*FETCH_DAY) - datetime.timedelta(days=1 if m.group(1) == "tegnap" else 0)
        return "%s %s" % (d.isoformat(), m.group(2))
    m = re.match(r"^(\S+\.) (\d+)\., (\d\d:\d\d:\d\d)$", t)
    if m and m.group(1) in HU_MONTHS:
        return "%04d-%02d-%02d %s" % (FETCH_DAY[0], HU_MONTHS[m.group(1)], int(m.group(2)), m.group(3))
    return ""
 USER_RE = re.compile(r'<div class="message-head-user">.*?<a class="user-link"[^>]*>([^<]+)</a>', re.S)
 class OwnText(HTMLParser):
    """Collect text inside rtif-content message-content divs that are NOT inside
    message-content-replied. Emoji images become their alt text; <br>/<p> become breaks."""
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.stack = []  # per open div: (is_replied, is_content)
        self.out = []
    def _skip(self):
        return any(r for r, _ in self.stack)
    def _in_content(self):
        return any(c for _, c in self.stack)
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "div":
            cls = (a.get("class") or "").split()
            self.stack.append(("message-content-replied" in cls,
                               "rtif-content" in cls and "message-content" in cls))
            return
        if self._skip() or not self._in_content():
            return
        if tag in ("p", "br", "li"):
            self.out.append("\n")
        elif tag == "img" and a.get("alt"):
            self.out.append(" " + a["alt"] + " ")
    def handle_endtag(self, tag):
        if tag == "div" and self.stack:
            self.stack.pop()
    def handle_data(self, data):
        if self._in_content() and not self._skip():
            self.out.append(data)
    def text(self):
        t = "".join(self.out)
        t = re.sub(r"[ \t\r\f\v]+", " ", t)
        t = re.sub(r"\n\s*\n+", "\n", t)
        return t.strip()
 def salt():
    if not os.path.exists(SALT_FILE):
        with open(SALT_FILE, "w") as f:
            f.write(secrets.token_hex(32))
        os.chmod(SALT_FILE, 0o600)
    return open(SALT_FILE).read().strip()
 def parse_page(path, s):
    h = open(path, encoding="utf-8").read()
    marks = list(LI_RE.finditer(h))
    posts = []
    for i, m in enumerate(marks):
        end = marks[i + 1].start() if i + 1 < len(marks) else len(h)
        block = h[m.start():end]
        t = TIME_RE.search(block)
        u = USER_RE.search(block)
        p = OwnText()
        p.feed(block)
        user = html.unescape(u.group(1)).strip() if u else ""
        posts.append({
            "num": int(m.group(2)),
            "date": norm_date(t.group(1)) if t else "",
            "edited": bool(t and "Szerkesztve" in t.group(1)),
            "reply_to": int(m.group(3)) if m.group(3) else None,
            "author_id": hashlib.sha256((s + user).encode()).hexdigest()[:12] if user else "",
            "off": "message-off" in (m.group(1) or ""),
            "text": p.text(),
        })
    return posts
 def main():
    s = salt()
    by_num, per_page = {}, {}
    for path in sorted(glob.glob(os.path.join(PAGES, "hsz_*.html"))):
        posts = parse_page(path, s)
        per_page[os.path.basename(path)] = posts
        for p in posts:
            by_num.setdefault(p["num"], p)
    # Hand-made quotes: a user pastes the parent's words into the own text (often in
    # <small><b><i>). No markup marks them reliably, so drop every own-text line of 20+ characters
    # that also occurs in the parent's text. Spec 8: an app named in a quote is not counted.
    def squash(t):
        return " ".join(t.lower().replace('"', " ").split())
    hand_quotes = 0
    for p in by_num.values():
        par = by_num.get(p["reply_to"]) if p["reply_to"] else None
        if not par:
            continue
        ptxt = squash(par["text"])
        keep = []
        for line in p["text"].split("\n"):
            sq = squash(line).strip(" :.,")
            if len(sq) >= 20 and sq in ptxt:
                hand_quotes += 1
                continue
            # A partial paste: drop each sentence (25+ chars) of the line found in the parent.
            parts = re.split(r'(?<=[.?!:"\]])\s+', line)
            kept = [x for x in parts
                    if not (len(squash(x).strip(' :.,[]')) >= 25 and squash(x).strip(' :.,[]') in ptxt)]
            hand_quotes += len(parts) - len(kept)
            if kept:
                keep.append(" ".join(kept))
        p["text"] = "\n".join(keep).strip()
    print("== hand-made quote lines removed: %d" % hand_quotes)
    bad = [p["num"] for p in by_num.values() if not p["date"]]
    if bad:
        print("UNPARSED DATES (fix norm_date before tagging): %s" % bad[:20])
        sys.exit(1)
    window = sorted((p for p in by_num.values() if p["date"] >= WINDOW_START), key=lambda p: p["num"])
    with open(OUT, "w", encoding="utf-8") as f:
        for p in window:
            f.write(json.dumps(p, ensure_ascii=False) + "\n")
    ok = True
    print("== control 1: counts per page (pages holding window posts)")
    lo, hi = window[0]["num"], window[-1]["num"]
    for name, posts in sorted(per_page.items(), key=lambda kv: int(kv[0].split("_")[1].split("-")[0])):
        nums = [p["num"] for p in posts]
        first = int(name.split("_")[1].split("-")[0])
        expected = set(range(first, first + 100))
        stray = [n for n in nums if n not in expected]
        dup = len(nums) - len(set(nums))
        print("  %-22s posts=%3d range=%s stray=%d dup=%d" %
              (name, len(nums), "%d-%d" % (min(nums), max(nums)) if nums else "-", len(stray), dup))
    have = set(by_num)
    missing = [n for n in range(lo, hi + 1) if n not in have]
    print("  window posts=%d  first=#%d (%s)  last=#%d (%s)" %
          (len(window), lo, window[0]["date"], hi, window[-1]["date"]))
    print("  missing numbers in window: %d" % len(missing))
    runs, start = [], None
    for n in missing:
        if start is None or n != prev + 1:
            if start is not None:
                runs.append((start, prev))
            start = n
        prev = n
    if start is not None:
        runs.append((start, prev))
    for a, b in runs:
        print("    #%d-#%d (%d)" % (a, b, b - a + 1))
    empty_text = sum(1 for p in window if not p["text"])
    print("  posts with empty own text: %d; off-topic flagged: %d; distinct authors: %d" %
          (empty_text, sum(p["off"] for p in window), len({p["author_id"] for p in window})))
    print("== control 3: #8099 own text present, quoted #8096 text absent")
    p = by_num.get(8099)
    own_ok = p and "proxmox" in p["text"].lower() and "7.x" in p["text"]
    # #8096 asks about moving from a Raspberry Pi 4 to an i5-6500T box.
    quote_absent = p and "6500" not in p["text"] and "raspberry" not in p["text"].lower()
    q = by_num.get(8096)
    neg_ctrl = q and ("6500" in q["text"] or "raspberry" in q["text"].lower())
    print("  own-text positive: %s; quote absent: %s; #8096 itself carries the marker (control works): %s"
          % (bool(own_ok), bool(quote_absent), bool(neg_ctrl)))
    print("  #8099 text: %r" % (p["text"][:200] if p else None))
    ok = ok and bool(own_ok and quote_absent and neg_ctrl)
    # Generic quote-leak check: no post's text may contain its parent's full text (when long enough).
    leaks = []
    for p in window:
        par = by_num.get(p["reply_to"]) if p["reply_to"] else None
        if par and len(par["text"]) >= 40 and par["text"][:40] in p["text"]:
            leaks.append(p["num"])
    print("== quote-leak sweep: posts whose text contains the parent's first 40 chars: %d %s"
          % (len(leaks), leaks[:20]))
    print("== control 2: five random posts for an eye check")
    rnd = random.Random(20261012)
    for p in rnd.sample(window, 5):
        print("  #%d reply_to=%s date=%s\n    %r" % (p["num"], p["reply_to"], p["date"], p["text"][:300]))
    sys.exit(0 if ok else 1)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,94 @@
 #!/usr/bin/env python3
 """Compare the session's own blind re-tags of a random 5% sample with the helper tags (spec 9.6).
 Mine: ~/research-cache/.../spot/mine_*.txt, one line per post:
  num|role|beginner(y/n)|question_type|hardware|hardware_status|apps|apps_status|platform|platform_status|pain|budget_huf|resolved
 Theirs: ~/research-cache/.../tags/tags_*.csv
 Agreement per column = exact match of the normalised value (sets for ;-lists; hardware compared by
 class, the bracketed model dropped). Prints overall agreement and agreement where either side is
 non-empty (the strict view: empty-vs-empty agreement inflates the overall rate).
 Usage: python3 spotcheck.py          (v1: every column, helper tags/ vs spot/mine_*.txt)
       python3 spotcheck.py --v2     (v2 re-tag: role, hardware, hardware_status;
                                      tags_v2/v2_*.csv vs spot/mine_v2.txt  num|role|hardware|hardware_status)
 """
 import csv
 import glob
 import os
 import re
 import sys
 CACHE = os.path.expanduser("~/research-cache/prohardver-hazi-szerver")
 COLS = ["role", "beginner", "question_type", "hardware", "hardware_status", "apps", "apps_status",
        "platform", "platform_status", "pain", "budget_huf", "resolved"]
 SETS = {"question_type", "hardware", "apps", "platform", "pain"}
 def norm(col, v):
    v = (v or "").strip()
    if col in SETS:
        parts = [re.sub(r"\s*\(.*?\)", "", x).strip().lower() for x in v.split(";")]
        return tuple(sorted(x for x in parts if x))
    if col == "budget_huf":
        return v.split(".")[0]
    return v.lower()
 def main_v2():
    cols = ["role", "hardware", "hardware_status"]
    theirs = {}
    for f in glob.glob(os.path.join(CACHE, "tags_v2", "v2_*.csv")):
        for r in csv.DictReader(open(f, encoding="utf-8")):
            theirs[int(r["num"])] = r
    mine = {}
    for line in open(os.path.join(CACHE, "spot", "mine_v2.txt"), encoding="utf-8"):
        v = line.rstrip("\n").split("|")
        mine[int(v[0])] = dict(zip(cols, v[1:]))
    nums = sorted(n for n in mine if n in theirs)
    print("v2 sample=%d compared=%d" % (len(mine), len(nums)))
    for c in cols:
        agree = sum(norm(c, mine[n][c]) == norm(c, theirs[n].get(c)) for n in nums)
        ne = [n for n in nums if norm(c, mine[n][c]) or norm(c, theirs[n].get(c))]
        agree_ne = sum(norm(c, mine[n][c]) == norm(c, theirs[n].get(c)) for n in ne)
        print("%-16s %7.1f%% %10.1f%% (n=%d)" % (c, 100.0 * agree / len(nums),
                                               100.0 * agree_ne / len(ne) if ne else 100.0, len(ne)))
        if os.environ.get("SHOW") == c:
            for n in nums:
                if norm(c, mine[n][c]) != norm(c, theirs[n].get(c)):
                    print("  #%d mine=%s theirs=%s" % (n, norm(c, mine[n][c]), norm(c, theirs[n].get(c))))
 def main():
    if "--v2" in sys.argv:
        return main_v2()
    theirs = {}
    for f in glob.glob(os.path.join(CACHE, "tags", "tags_*.csv")):
        for r in csv.DictReader(open(f, encoding="utf-8")):
            r["beginner"] = "y" if (r.get("beginner_evidence") or "").strip() else "n"
            theirs[int(r["num"])] = r
    mine = {}
    for f in glob.glob(os.path.join(CACHE, "spot", "mine_[0-9].txt")):
        for line in open(f, encoding="utf-8"):
            if not line.strip():
                continue
            v = line.rstrip("\n").split("|")
            mine[int(v[0])] = dict(zip(COLS, v[1:]))
    nums = sorted(n for n in mine if n in theirs)
    print("sample=%d compared=%d (missing in helper tags: %s)" %
          (len(mine), len(nums), sorted(set(mine) - set(theirs))))
    print("%-16s %8s %18s" % ("column", "overall", "non-empty either"))
    for c in COLS:
        agree = sum(norm(c, mine[n][c]) == norm(c, theirs[n].get(c)) for n in nums)
        ne = [n for n in nums if norm(c, mine[n][c]) or norm(c, theirs[n].get(c))]
        agree_ne = sum(norm(c, mine[n][c]) == norm(c, theirs[n].get(c)) for n in ne)
        print("%-16s %7.1f%% %10.1f%% (n=%d)" % (c, 100.0 * agree / len(nums),
                                               100.0 * agree_ne / len(ne) if ne else 100.0, len(ne)))
    if os.environ.get("SHOW"):
        c = os.environ["SHOW"]
        for n in nums:
            a, b = norm(c, mine[n][c]), norm(c, theirs[n].get(c))
            if a != b:
                print("  #%d mine=%s theirs=%s" % (n, a, b))
 if __name__ == "__main__":
    main()