> For the complete documentation index, see [llms.txt](https://faction-os.gitbook.io/faction-os-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://faction-os.gitbook.io/faction-os-docs/docs/ongoing-projects/synthetic-sound-map.md). # Synthetic Sound Map - Replacement Plan **Date:** 2026-06-01 **Scope:** `apps/web` (the cockpit). The server (`apps/server`) emits no audio. **Goal:** Inventory every synthetically generated sound so each can be swapped for a quality recorded/produced asset. *** ## TL;DR There is exactly **one** synthetic sound engine in the system: the Web Audio **voice barks** in `apps/web/src/lib/voiceSynth.ts`. It procedurally generates **25 distinct cues** (5 factions x 5 "bark kinds") from oscillator recipes - no files involved. Everything else that makes sound already uses **real, file-backed MP3 assets**: | Sound system | Source | Synthetic? | Assets | | --------------------------------------------------------------------- | -------------------------- | ----------------- | -------------------------------- | | Voice barks (hero state transitions, faction-pick flourish, test cue) | `lib/voiceSynth.ts` | **YES - replace** | none (oscillators) | | Faction click voice lines | `lib/factionVoiceLines.ts` | No (real MP3) | `public/speech/*.mp3` (20 files) | | Background music | `lib/backgroundMusic.ts` | No (real MP3) | `public/music/*.mp3` (12 files) | So the replacement work is: **produce 25 short SFX clips** to retire the oscillator synth. The separate Lobster click voice-line content gap is closed for the app runtime, with promotion evidence still pending. > **Status (2026-06-01):** the file-backed engine is built and live; `error` for the four voiced factions now plays recorded speech, and 15 CC0 draft clips are staged for audition. See **§8** for the full implementation state. Sections 1-7 below remain the inventory + design brief. *** ## 1. The synthetic engine - `lib/voiceSynth.ts` WC3-style emotive grunts/chirps/whispers, synthesized live with stacked oscillators + a per-bark gain envelope + a lowpass filter + optional tremolo LFO. The header comment states these are a fallback for licensed MP3 voice lines that can't be redistributed. ### Synthesis primitives (what produces the sound) * `playBark(faction, kind, volume)` - `voiceSynth.ts:190` - the generator. * Per-bark gain envelope: 20 ms attack, exponential decay to silence over the recipe duration (`voiceSynth.ts:219-224`). * Lowpass `BiquadFilter`, Q 0.7, cutoff = `recipe.filterHz` (`:225-229`). * `recipe.stacks` oscillators, detuned by `+/-spreadHz`, overtones at `base x (1 + i * 0.35)` (`:232-238`). * Optional tremolo: an LFO at `recipe.tremolo` Hz modulating each oscillator's frequency (`:239-249`). * Master gain fixed at `0.35` (`voiceSynth.ts:180`). ### The 25 recipes - `STYLE_BASE` (`voiceSynth.ts:28-64`) Each cell is `bark(baseHz, spreadHz, durationMs, stacks, oscType, tremoloHz, filterHz)`. | Faction | idle | active | complete | error | celebration | | ----------- | ------------------------------ | ------------------------------- | ---------------------------- | ---------------------------- | ------------------------------ | | **orc** | 110 Hz saw, 220 ms | 140 Hz square, 320 ms, trem 4 | 180 Hz saw, 460 ms | 90 Hz square, 240 ms, trem 8 | 220 Hz saw, 700 ms, trem 3 | | **human** | 260 Hz tri, 240 ms | 330 Hz tri, 360 ms, trem 3 | 440 Hz tri, 520 ms | 220 Hz saw, 220 ms, trem 6 | 560 Hz tri, 720 ms, trem 2 | | **elf** | 420 Hz sine, 260 ms | 540 Hz sine, 360 ms, trem 2 | 640 Hz sine, 540 ms | 380 Hz tri, 240 ms, trem 4 | 820 Hz sine, 760 ms, trem 1 | | **undead** | 80 Hz saw, 280 ms | 100 Hz saw, 360 ms, trem 5 | 140 Hz square, 460 ms | 70 Hz square, 280 ms, trem 9 | 190 Hz saw, 700 ms, trem 4 | | **lobster** | 1400 Hz square, 90 ms, trem 18 | 1600 Hz square, 140 ms, trem 14 | 1800 Hz tri, 220 ms, trem 10 | 1200 Hz saw, 90 ms, trem 22 | 2200 Hz square, 320 ms, trem 8 | These recipes are the **design brief** for the replacement assets: they encode each faction's intended timbre (orc = low/gravelly, elf = high/pure, undead = sub-bass/dissonant, lobster = fast high-pitched chitter) and the per-event duration/energy. Match the *character*, not the literal waveform. ### `BarkKind` semantics (`voiceSynth.ts:9`) | Kind | Fires when | Feel | | ------------- | ------------------------------------ | -------------------------- | | `idle` | hero goes idle | low-energy acknowledgement | | `active` | hero starts working / awaiting input | alert, "on it" | | `complete` | hero dismissed / task done | satisfied resolution | | `error` | hero errors | harsh, attention-grabbing | | `celebration` | faction picked | triumphant flourish | *** ## 2. Where the synthetic cues are triggered All paths funnel through the runtime boundary (`lib/audioRuntime.ts` -> `store/useAudioRuntimeStore.ts` -> `voiceSynth.playBark`), except the faction-pick flourish which calls `playBark` directly. | Trigger | Site | Cue kind(s) | | ------------------------------------ | ------------------------------------------------------------------------- | -------------------------------------------------- | | **Hero state transitions** (ambient) | `store/useVoiceBarks.ts:60-72`; state-to-kind map at `:84-99` | idle, active, complete, error | | **Faction pick flourish** | `components/FactionPicker.tsx:34` (`playBark(id, "celebration")`, direct) | celebration | | **Settings "test audio" button** | `components/SettingsDrawer.tsx:145-152` | idle | | **Hero "speak" button** | `components/HeroDetailDrawer.tsx:565-575` | idle / active / complete / error (from hero state) | State-to-bark mapping (`useVoiceBarks.ts:84`): `active->active`, `idle->idle`, `awaiting_input->active`, `error->error`, `dismissed->complete`. The contract layer (`lib/audioRuntime.ts`) is asset-agnostic: it normalizes requests, guards duplicates, and maps results to toast copy. **A real-asset playback engine drops in behind this same boundary** with no changes to the trigger sites - the `VoiceSynthAdapter` interface (`audioRuntime.ts:61`), default `DEFAULT_SYNTH_ADAPTER` (`audioRuntime.ts:92`). As of §8 that default is the file-backed adapter (`lib/fileVoiceSynth.ts`), not the raw synth. *** ## 3. Already-real audio (context - NOT synthetic) These are listed so nobody "replaces" them by mistake - they're already quality file assets. ### Faction click voice lines - `lib/factionVoiceLines.ts` Real recorded MP3s, played on hero/faction **click** (distinct from the ambient state-transition barks). Served from `public/speech/`. * 20 files, 4 lines each for **orc / human / elf / undead / lobster**. * `lobster` now has recorded app click lines in `FACTION_CLICK_VOICE_LINES`; public-demo `claw-*` clips remain reference-only unless demo parity is explicitly promoted. ### Background music - `lib/backgroundMusic.ts` 12 real MP3 tracks in `public/music/`, explicit opt-in, no autoplay (`BACKGROUND_MUSIC_TRACKS`, `backgroundMusic.ts:34-71`). Resume point persisted to `localStorage`. **No synthesis here.** *** ## 4. Replacement plan ### A. Retire the oscillator synth (primary work) > **Implemented** (see §8): the file-backed adapter + manifest are live; the remaining work is producing/auditioning the clips themselves. The plan below is retained as the design brief. Produce **25 short SFX clips** - 5 factions x {idle, active, complete, error, celebration} - matching the timbre/duration brief in the recipe table (section 1). Suggested convention (mirrors `public/speech/` + `public/music/`): ``` public/sfx/-.mp3 e.g. public/sfx/orc-active.mp3, public/sfx/elf-celebration.mp3 ``` Then swap the engine behind the existing boundary: 1. Add a file-backed `VoiceSynthAdapter` (load/cache `HTMLAudioElement` or decoded `AudioBuffer` per `faction+kind`; honor `volume`; return the same `VoiceSynthPlayResult` status union). 2. Point `DEFAULT_SYNTH_ADAPTER` (`audioRuntime.ts:92`) at it. 3. Trigger sites, toast copy, mute/blocked/failed handling, and the cue guard need **no changes** - the contract is already in place. 4. Keep `voiceSynth.ts` as the graceful fallback if an asset 404s, or delete it once coverage is confirmed. ### B. Close the lobster voice-line gap (secondary) Implemented for the app runtime: Lobster click lines live in `public/speech/` and are mapped in `FACTION_CLICK_VOICE_LINES`. Remaining work is promotion evidence: rights, credits, loudness, catalog records, and any public-demo parity decision. ### Asset count summary * **25** faction-bark SFX clips (replaces all synthesis). * **4** lobster click voice lines (app runtime content gap closed; promotion evidence still pending). *** ## 5. Files referenced | Purpose | Path | | --------------------------------------- | --------------------------------------------------- | | Synthetic engine (fallback) | `apps/web/src/lib/voiceSynth.ts` | | File-backed adapter (live default) | `apps/web/src/lib/fileVoiceSynth.ts` | | SFX manifest + `pickSfxUrl` | `apps/web/src/lib/sfxAssets.ts` | | CC0 fetch/transcode script | `scripts/fetch-cc0-sfx.mjs` | | Draft clips + provenance | `apps/web/public/sfx/` (+ `CREDITS.md`) | | Asset-agnostic runtime boundary (reuse) | `apps/web/src/lib/audioRuntime.ts` | | Runtime store | `apps/web/src/store/useAudioRuntimeStore.ts` | | Bark trigger (state transitions) | `apps/web/src/store/useVoiceBarks.ts` | | Faction-pick flourish trigger | `apps/web/src/components/FactionPicker.tsx` | | Settings test cue | `apps/web/src/components/SettingsDrawer.tsx` | | Hero speak cue | `apps/web/src/components/HeroDetailDrawer.tsx` | | Real click voice lines | `apps/web/src/lib/factionVoiceLines.ts` | | Real background music (not synthetic) | `apps/web/src/lib/backgroundMusic.ts` | | Existing assets | `apps/web/public/speech/`, `apps/web/public/music/` | *** ## 6. Stopgap evaluation - reuse `public/speech/` files for barks? **Question asked:** can the existing `public/speech/` MP3s cover 4/5 of the faction bark asset needs right now (orc/human/elf/undead have clips; lobster does not)? **Verdict:** technically possible, but the speech files were authored for a *different* job (deliberate click responses) and are a poor fit for most barks. They cleanly cover only **one** of the five cue kinds. ### Why it is not a clean drop-in 1. **No per-kind mapping.** Barks need `faction x 5 kinds` (idle/active/complete/error/celebration). Speech provides `faction x 4 random phrases` with no kind distinction, so every kind would draw from the same pool - an `error` and a `celebration` could both play "zug zug," losing the per-event meaning the synth encodes. 2. **Wrong length/density for ambient cues.** Barks are 90-760 ms ambient texture firing on *every* hero state transition (rate-limited 1 per 1.5 s *per hero*). Speech clips are full \~1-2 s spoken phrases. With several heroes flipping state, idle/active/complete would become near-constant chatter. 3. **Faction-pick double-fire.** `celebration` barks fire from exactly one place * `FactionPicker.tsx:34` - which *already* plays a speech voice line via `playVoiceLine(id)`. Routing celebration to speech would overlap two spoken lines (separate players, they will not cut each other off). The Battlefield "celebrations" are visual only and never trigger a bark. ### Per-kind fit | Bark kind | Frequency | Reuse speech? | Why | | ------------------------------ | ----------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------- | | `idle` / `active` / `complete` | High (every state flip) | **No** | Ambient texture; full phrases here = constant chatter. Keep as short synth grunts. | | `error` | Low | **Yes** | Infrequent + attention-worthy; a faction "speaking up" on error feels better than a synth buzz and will not spam. | | `celebration` | Faction-pick only | **No** | A speech clip already plays there; adding one = two overlapping voices. Keep the synth flourish. | ### Recommendation (implemented - see §8) A **narrowed reuse**: route only the `error` bark for the four voiced factions (orc/human/elf/undead) to their `FACTION_CLICK_VOICE_LINES` pool; lobster falls back to synth on error as well. Leave idle/active/complete on the synth, and leave celebration alone. This delivers real, quality audio at the one moment it clearly helps, with no chatter and no double-fire. The other four cue kinds still want purpose-made SFX, so the §4 plan stands at roughly **\~20** clips remaining (the 25 minus `error` for the four voiced factions). **As implemented (contained, no trigger-site changes):** 1. `lib/sfxAssets.ts` maps `error` for orc/human/elf/undead to their `FACTION_CLICK_VOICE_LINES` pool; `lib/fileVoiceSynth.ts` plays it and delegates every other cue to the synth. 2. Wired as `DEFAULT_SYNTH_ADAPTER` (`audioRuntime.ts:92`). Toast copy, mute/ blocked/failed handling, and the cue guard are unchanged. *** ## 7. Open-source asset sourcing (CC0 candidates) Goal: find openly-licensed clips to replace the 25 synthetic barks. **CC0 (public-domain) is the priority** - the synth exists precisely because the original licensed MP3s could not be redistributed, so anything we ship must be freely redistributable. All sources below are CC0; attribution is optional but courteous. ### Candidate packs | # | Pack | License | Verified? | Format | Best for | | -- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | --------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | P1 | [80 CC0 creature SFX](https://opengameart.org/content/80-cc0-creature-sfx) (rubberduck) | CC0 | **Fetched ✓** | OGG | The workhorse. grunt×5, monster×7, troll×3, roar×3, bug×4, alien×6, burble×2, howl, breath, hurt×5, cute×10, ooh, weird×5, scream×2, spit×3. Mouth-made vocal texture - exactly the bark aesthetic. | | P2 | [orc voice](https://opengameart.org/content/orc-voice) (Tim Rockk) | CC0 | **Fetched ✓** | WAV | Orc grunts, fight sounds, retreat calls. | | P3 | [80 CC0 RPG SFX](https://opengameart.org/content/80-cc0-rpg-sfx) | CC0 | **Fetched ✓** | WAV (zip) | Combat/item supplements: hurt, die, monster, roar, spell, slime. | | P4 | [Kenney - Interface Sounds](https://kenney.nl/assets/interface-sounds) | CC0 | **Fetched ✓** | (Kenney std) | 100 clean UI tones: clicks, confirms, chimes. For human/elf `complete`, generic `error`. | | P5 | [Kenney - UI Audio](https://kenney.nl/assets/ui-audio) | CC0 (claimed) | Needs check | - | 50 more UI sfx (buttons/switches/clicks). | | P6 | [EFFORT SOUNDS (Male)](https://opengameart.org/content/effort-sounds-male-audio-pack) (VoiceBosch / SoundBiter) | CC0 (claimed) | **Needs check** | - | 41 male exertion grunts - the human-voice gap (idle/active). Verify the pack's specific license before download. | | P7 | [Magic Sounds](https://opengameart.org/content/magic-sounds) / [Spell Sounds Starter Pack](https://opengameart.org/content/spell-sounds-starter-pack) / [Magic Spell SFX](https://opengameart.org/content/magic-spell-sfx) | CC0 (claimed) | Needs check | - | Shimmer/chime/sparkle for elf (high/pure). | | P8 | [Victory Fanfare Short](https://opengameart.org/content/victory-fanfare-short) / [Win Jingle](https://opengameart.org/content/win-jingle) / [Win sound effect](https://opengameart.org/content/win-sound-effect) / [Level up sound effects](https://opengameart.org/content/level-up-sound-effects) | CC0 (claimed) | Needs check | varies (some MP3/WAV) | `celebration` flourishes (human/elf especially). | "Verified ✓" = page fetched and CC0 confirmed in this pass. "Needs check" = CC0 reported by search but the individual pack page was not opened; **confirm the license on the page before downloading** (OpenGameArt entries sometimes carry multiple licenses). ### Proposed per-faction x per-kind mapping Timbre brief from §1: orc = low/gravelly, human = mid, elf = high/pure, undead = sub-bass/dissonant, lobster = fast high chitter. | Faction | idle | active | complete | error | celebration | | ----------- | ------------------------- | ------------------- | ----------------------- | ----------------------------------------- | -------------------------- | | **orc** | P1 grunt (low) | P2 grunt / P1 grunt | P2 "victory" / P1 grunt | P1 hurt *(or existing speech, §6)* | P1 roar / P2 fight | | **human** | P6 soft effort | P6 effort grunt | P4 confirm chime | P4 negative *(or existing speech, §6)* | P8 win jingle / fanfare | | **elf** | P7 soft shimmer / P1 cute | P7 shimmer | P7 chime / P4 chime | P4 negative *(or existing speech, §6)* | P7 sparkle / P8 jingle | | **undead** | P1 breath / weird | P1 monster (low) | P1 weird / monster | P1 scream/hurt *(or existing speech, §6)* | P1 roar (deep) / weird | | **lobster** | P1 bug | P1 bug / alien | P1 burble / alien | P1 alien / spit | P1 alien burst / bug trill | Notes: * **P1 alone covers orc, undead, and lobster** across all five kinds - it is the single highest-value download. * **Human** is the only faction with no good vocal match in P1; P6 (effort grunts) fills idle/active, P4/P8 (UI tone + jingle) fill complete/celebration. * **Elf** leans on P7 (magic shimmer) for its pure/high character rather than vocal grunts. * `error` for the four voiced factions can come from the **existing speech files** instead (the §6 stopgap) if we prefer recorded phrases there. ### Practical notes * **Format:** sources are OGG/WAV; the app convention is MP3 (`public/speech/`, `public/music/`). Transcode to MP3 (`ffmpeg`) for consistency - or serve OGG, since both the Web Audio decode path and `HTMLAudioElement` accept it in modern browsers. * **Editing:** clips will need trimming/normalizing to bark-length (\~90-760 ms per §1) and level-matching; raw pack clips are longer and louder than the 0.35 master gain expects. * **Provenance:** even though CC0 needs no attribution, record each clip's source pack + author + license in a small `public/sfx/CREDITS.md` so the redistribution basis is auditable later. * **Coverage:** all 25 cues have at least one CC0 candidate; nothing requires a paid or attribution-only asset. *** ## 8. Implementation status (2026-06-01) The file-backed engine is built and live behind the existing runtime boundary. ### Shipped (live) * `lib/sfxAssets.ts` - manifest mapping `faction + kind` to real audio URLs, plus `pickSfxUrl` / `hasSfxFiles`. Only mapped cues use files; the rest fall through to the synth. **Restructured (2026-06-01)**: each faction now appears exactly once in the object, with per-cue commented stubs inside. The prior shape had one live entry (`orc: { error: ... }`) followed by a separate commented entry for the same key (`// orc: { idle: ..., active: ..., ... }`); uncommenting would have created a duplicate key and silently dropped the `error` speech mapping. Fixed; human/elf stubs with "not yet sourced" notes also added. * `lib/fileVoiceSynth.ts` - `VoiceSynthAdapter` that plays a mapped file and **falls back to the synth per-cue** on any miss/failure/autoplay-block. Support + unlock delegate to the synth. Dependencies are injectable. * `lib/audioRuntime.ts` - `DEFAULT_SYNTH_ADAPTER` now points at the file adapter (the single documented swap point). No trigger-site changes. * **`error` for orc/human/elf/undead now plays recorded `/speech/` lines** (the §6 stopgap, using assets already in the repo - 4 of 25 cues real, today). * Tests: `tests/fileVoiceSynth.test.ts` (7) green; TypeScript clean. ### Drafted (downloaded, gated OFF until auditioned) * `scripts/fetch-cc0-sfx.mjs` - downloads the verified-CC0 P1 pack and transcodes a curated subset (mono, level-matched, MP3) into `public/sfx/`. * 15 draft clips for **orc / undead / lobster** (5 kinds each) in `public/sfx/`, with `public/sfx/CREDITS.md` provenance. Selected by **filename, not by ear** - so they're left **out of the live manifest**; the synth still plays these cues. Activate per-cue by uncommenting in `lib/sfxAssets.ts` after auditioning. ### Not yet sourced * **human** vocal cues (needs P6 effort-grunt pack) and **elf** (needs P7 magic shimmer) - P1's creature texture doesn't fit them. `human`/`elf` keep the synth for idle/active/complete/celebration; their `error` already uses speech. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://faction-os.gitbook.io/faction-os-docs/docs/ongoing-projects/synthetic-sound-map.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.