feat(world): мульти-источник мира (composite) со свежестью, приоритетом и dual-write#3530
Draft
kvirund wants to merge 25 commits into
Draft
feat(world): мульти-источник мира (composite) со свежестью, приоритетом и dual-write#3530kvirund wants to merge 25 commits into
kvirund wants to merge 25 commits into
Conversation
…l-write [WIP]
Add CompositeWorldDataSource: combine several backends with per-source
freshness ("actuality") and priority, configured via <world_loader><sources>.
Read from the freshest source (ties -> priority), write to all (dual-write),
and rewrite any staler source from the loaded world after boot (self-heal).
- IWorldDataSource gains ListZoneVnums/GetZoneFreshness/GetIndexFreshness,
MarkZoneSynced, IsWritable and per-zone Load hooks (defaults preserve
single-source behaviour).
- YAML freshness from file mtimes (zone dir + zones/index.yaml).
- SQLite freshness from a new zone_sync/world_meta bookkeeping, created lazily
so an old world.db keeps working. Open world.db READWRITE|CREATE so the
engine can write it back.
- Runtime source selection in GameLoader::BootWorld from RuntimeConfiguration.
Verified end-to-end that selection/freshness/resync-decision work (first boot
loads YAML and resyncs SQLite; second boot reads from SQLite).
Known issues, not yet fixed:
1. Post-resync ping-pong: a synced source is stamped with now() instead of the
winner's content version, so the roles flip every boot.
2. The engine's SQLite write path (previously "read-only for now") is buggy:
SaveMobRecord's column/value lists are mismatched, leaving mobs enabled=0,
so reloading an engine-written world.db crashes MobMax::init on a null mob.
…ing-pong) MarkZoneSynced/MarkIndexSynced now carry the winner's content version (Freshness), and the composite stamps every source with the same canonical version after a write/resync. A freshly resynced SQLite backend therefore ends up EQUAL in freshness to the YAML it was rebuilt from -- not newer -- so the two stop trading "stale" roles every boot. Verified: after resync, zone_sync holds the YAML zone mtimes; a second boot sees equal freshness, reads the priority-first source, and rewrites nothing. Note: with equal freshness the first-listed <sources> child wins, so list the source you want to read from first (SQLite for fast boots; YAML still wins whenever its files are actually edited).
…unusable) The mobs INSERT named 59 columns but supplied only 58 values (57 '?' + the literal enabled=1), so sqlite3_prepare_v2 failed and no mob was ever written. A world.db rewritten by the engine therefore came back with zero usable mobs, crashing MobMax::init on a null mob_proto when that db was reloaded. Add the missing placeholder (now 58 '?' + 1 literal = 59 values). Audited every other INSERT/REPLACE in the backend -- all column/value counts now match. With this, an engine-written world.db round-trips: a composite first boot loads YAML and resyncs SQLite; the next boot reads that SQLite and completes a full "Boot db -- DONE" in ~5s.
The SQLite read path converts TEXT utf8->koi8 (codepages::utf8_to_koi) and the Python converter stores UTF-8, but every engine-side Save*Record bound runtime KOI8-R strings verbatim. Reloading an engine-written world.db therefore ran utf8->koi8 over already-KOI8 bytes, garbling every name/description and breaking the checksum of essentially every entity. Add a free BindTextKoi() helper (koi8->utf8 via codepages::koi_to_utf8, NULL-safe, ASCII passes through) and route all 51 text binds through it. Cuts the YAML-vs-SQLite load checksum diff from ~84k entities to ~22k; rooms (40975->10) and triggers (11712->6) now match. Remaining diffs are structural row-count bugs in other Save* paths (zone_commands, obj_applies, *_flags) tracked separately.
zone_commands uses a semantic layout (cmd_type + typed arg_* columns) read back per command type by LoadZoneCommands, but SaveZoneCommands wrote an obsolete generic layout (command_order/command/arg1..arg4) -- columns that don't exist. Every INSERT silently failed to prepare, so a resynced/dual-written zone lost ALL its reset commands (zone 1000: 0 rows; total 7345 vs converter 46019). Also the in-memory args are rnums (ResolveZoneCmdVnumArgsToRnums), while the format stores vnums. Rewrite the writer as the exact inverse of LoadZoneCommands: emit cmd_type strings, map each command's args to the right typed columns, and convert entity args rnum->vnum (mirroring YamlWorldDataSource::SaveZone). Skip '*'/unknown like the YAML backend. zone 1000 now round-trips 96 commands, total 45978; ZONE load checksum diffs drop 529 -> 17 (the remainder is the pre-existing YAML<->SQLite format gap, also present when loading converter-built databases).
…option Rooms have two exit arrays: dir_option_proto (the world-file template, what the loader fills and the checksum/YAML writer read) and dir_option (the live runtime copy, populated from proto during zone resets). SaveRoomRecord serialized dir_option, which is empty at boot -- the resync/dual-write runs before any zone reset -- so every non-dungeon room was written with ZERO exits (only 179 of ~38.5k had any, those happening to be reset already). Reloading an engine-written world.db then lost almost all exits. Read dir_option_proto (matching LoadRooms / SerializeRoom / the YAML backend) and DELETE room_exits before re-inserting so a re-save replaces rather than duplicates. Non-dungeon rooms with exits 179 -> 38466; total room_exits matches the converter (240737 vs 240845); 495/500 sampled rooms now match exactly (the few left are a to_room 0-vs-(-1) edge case).
…hem) SaveObjectRecord only deleted the objects row; with foreign_keys OFF nothing cascaded, so a re-save (resync / OLC dual-write) appended a second copy of every apply. LoadObjectApplies fills the next empty slot per row, so duplicates loaded as extra applies and changed the object. Clear obj_applies for the vnum first. Round-trip OBJ load-checksum diffs 8375 -> 2083.
…ity_triggers SaveRoomRecord/SaveMobRecord serialized room->script->script_trig_list / mob.script->script_trig_list -- the RUNTIME attached-trigger list, which is empty until zone resets run (the resync/dual-write happens before any reset), so almost all room and mob trigger bindings were lost. Read proto_script (the world-file trigger vnums) instead, matching the object writer and the YAML backend. Also DELETE entity_triggers for the entity before re-inserting (foreign_keys is OFF, no cascade) so a re-save replaces rather than duplicates. Round-trip OBJ diffs 2083 -> 484 and the *_SCRIPTS aggregate checksums now match; total 16508 -> 14906.
kvirund
force-pushed
the
feat/world-multi-source
branch
from
2ed5c25 to
961eef3
Compare
… stack garbage) SaveMobRecord declared `char special_buf[kMaxStringLength];` uninitialised, then called FlagData::tascii() on it. tascii APPENDS (it strlen()s the buffer and writes after), so it serialised whatever stack garbage was there ahead of the real npc flag tokens -- e.g. a 353-byte blob instead of "z0A0B0b1c1d1". On reload from_string() parsed the garbage into junk npc_flags. The YAML backend already zeroes the buffer first; do the same. Round-trip MOB diffs 13777 -> 11177.
The hand-maintained mob_action_flag_map only listed plane-0 EMobFlag names, so SaveFlagsToTable's reverse lookup returned "" for every plane-1+ bit (kSwimming, kFlying, the breath/appears/season flags, ...) and silently dropped them; reload lost those action flags. Regenerate the map from the full EMobFlag enum (76 entries) and DELETE mob_flags before re-insert so a re-save replaces rather than duplicates. Round-trip MOB diffs 11177 -> 10281.
…ped) Same bug as the action-flag map: mob_affect_flag_map listed only plane-0 EAffect names, so SaveFlagsToTable dropped every plane-1+ affect (kHolyLight, kDrunked, kDisguise, ... 30 of them) on save. Regenerate from the full EAffect enum (90 entries). Round-trip MOB diffs 10281 -> 8667.
SaveMobRecord bound default_pos, start_pos and sex as integers (left as a "TODO: add lookup"), but LoadMobs reads all three as enum NAMES via position_map / gender_map. So every reloaded mob fell back to the default position and kMale. Write the names through ReverseLookupFlag, matching the loader and the converter. Round-trip MOB diffs 8667 -> 6742.
…w presence The per-mob spell data is the memorised-count array SplMem -- that is what the checksum and the YAML backend serialise, and the mob_spells schema already has a `count` column the converter fills. SaveMobSpells instead wrote presence from SplKnw (count defaulting to 1) and LoadMobSpells loaded back into SplKnw, so every mob lost its spell memory on reload. Write/read SplMem with the real count (setting SplKnw too), and DELETE mob_spells before re-insert. MOB 6742 -> 663.
…resync) SaveObjectRecord re-saves get_skills() with INSERT OR REPLACE, which updates matching rows but leaves behind any obj_skills row the object no longer has, so a resave kept stale converter-written skills. Add the missing DELETE FROM obj_skills before the insert loop. Round-trip OBJ 484 -> 127.
…eyword Exit keywords carry two forms -- the nominative `keyword` and the accusative `vkeyword` used by open/close. ExitData::set_keywords splits a "keyword|vkeyword" string on load, but SaveRoomRecord wrote only the keyword, so every door reloaded with vkeyword == keyword and lost the accusative form. Join the two with '|' on write. Round-trip ROOM 622 -> 10.
…_load table The SQLite schema had no table for mob.dl_list, so every mob's death-load (the items dropped into its corpse) was lost on reload. Add a mob_death_load table (created lazily on save, ordered by load_order), write each LoadingItem, delete before re-insert, and load it back into dl_list. MOB round-trip 663 -> 297.
…om_flags Flags the enums have no name for (e.g. the 14-19 gap in the mob act plane, and assorted room bits present in legacy data) were silently dropped on save because ReverseLookupFlag returned "" and SaveFlagsToTable skipped the bit. Emit an UNUSED_<absolute-bit> token for those bits -- the same convention the object loader already understood -- and teach the mob (action/affect) and room loaders to read it back. With the flag writer now lossless, also delete room_flags before re-insert. Round-trip MOB 297 -> 20, ROOM 10 -> 0.
The ObjVal map (potion spells/levels, refilled-potion proto vnum, ...) returned by get_all_values() lived only in value0..3 plus nowhere -- the SQLite schema had no home for it, so it vanished on reload. Store ObjVal::serialize_values() in a lazily-created obj_extra_values table and reload it via init_values_from_file, stripping the "Vals:" header (init_from_file reads bare key/val pairs and a leading non-numeric token would abort the parse). Round-trip OBJ 127 -> 29.
…opped) SaveObjectRecord only wrote extra flags (and with an empty category at that) plus wear flags; the anti, no and weapon-affect flag sets were never written, so they survived a reload only by accident through stale converter rows -- and lost any plane the converter's maps didn't name. Write all four categories through SaveFlagsToTable with the right category and UNUSED_ fallback, give wear the same fallback, and DELETE obj_flags before re-insert now that the writer is lossless. Round-trip OBJ 29 -> 0.
…ill() SaveMobRecord wrote GetSkill(), which layers equipment/affect/room modifiers onto the trained value, so a resave baked in those instance modifiers and the skill came back changed. Iterate GetCharSkills() and store the raw skill_level (matching SerializeMob and the YAML backend), and delete mob_skills before re-insert. MOB round-trip 20 -> 2.
SaveTriggers only re-wrote the triggers still present in memory; a trigger that existed in the db but was dropped from the source (e.g. a legacy trigger absent from YAML) survived. That stale trigger shifted every later trigger rnum, which in turn corrupted the rnum-valued 'T'/'V' zone-reset command args on reload. Delete the zone's whole vnum/100 trigger block up front, then re-insert what remains. Round-trip ZONE 17 -> 2 (and a phantom trigger removed).
SaveTriggerRecord skipped empty command lines (if (!cmd->cmd.empty())), so a blank line inside a DG script vanished and the command count changed on reload. Mirror the YAML backend: write an empty line as a single space (ParseTriggerScript trims it back to "" on load, keeping the node count) and join lines without a trailing newline. Round-trip TRIG 6 -> 0.
…b_resistances Two small mob-fidelity fixes: (1) position and sex are stored as enum names, but an out-of-range value with no symbolic name (e.g. a corrupt default_pos of 15) reverse-looked-up to "" and snapped to the default on reload. Store the raw int when there is no name and parse it back, matching YamlWorldDataSource::ParseEnum. (2) DELETE mob_resistances before re-insert so stale converter-written resistances no longer survive a resave. MOB round-trip 2 -> 1.
…overrode) mob_saves had no DELETE before the re-insert, so converter-written rows (which use a different save indexing) survived and, depending on row order, overrode the engine's apply_saving_throw values on reload -- a mob's saving throws came back scrambled. Add the missing DELETE. MOB round-trip 1 -> 0.
A reset command whose target failed to resolve at boot is marked '*' by ResolveZoneCmdVnumArgsToRnums; SaveZoneCommands' switch hit `default: continue` and dropped it, so the dead line vanished and every later command shifted on reload. Persist it as a DISABLED cmd_type carrying the raw args and rebuild the '*' command on load. Round-trip ZONE 2 -> 0 -- the full world now round-trips YAML-load == SQLite-load with zero diffs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Зачем
Мульти-источник мира: грузиться из самого свежего источника, писать во все, держать их в синхроне. Цель — быстрый старт из SQLite (~5 c против ~30 c YAML на полном мире) при сохранении YAML как удобного для ручной правки источника.
Модель (
<world_loader><sources>)now()), поэтому источники не «пинг-понгуют» ролями между стартами.YAML-свежесть — mtime файлов; SQLite — таблицы
zone_sync/world_meta(создаются лениво, старыйworld.dbработает).Что внутри
CompositeWorldDataSource+ runtime-выбор вGameLoader::BootWorldиз конфига.IWorldDataSource:ListZoneVnums/GetZoneFreshness/GetIndexFreshness/MarkZoneSynced/MarkIndexSynced/IsWritable(дефолты сохраняют поведение одиночных источников).SaveMobRecord— рассинхрон числа колонок/значений (крашMobMax::init).BindTextKoi).SaveZoneCommandsпереписан под реальную семантическую схему (писал в несуществующие колонки → reset-команды терялись).Проверено
Полный мир, round-trip
load YAML → resync SQLite → load SQLite: загрузка из движком-записаннойworld.dbпроходит целиком (Boot db -- DONE, ~5 c). Расхождение контрольных сумм загрузки YAML↔SQLite: 84089 → 22189 сущностей.Известное оставшееся (поэтому Draft)
Движковые writer’ы ряда дочерних таблиц (
obj_flags,room_flags,room_exits,mob_flags/skills/feats/saves/spells,entity_triggers,extra_descriptions) ещё пишут данные, которые загрузчик читает неверно — их надо переписать как точную инверсию соответствующихLoad*(как уже сделано сzone_commands). Отдельно: ~13.7k расхождений по мобам — это предсуществующий разрыв формата YAML↔SQLite (конвертерная БД расходится с YAML так же), а не этот writer.