Skip to content

[Other] setup (Mac): Swift 1.5 IQ3_S, opt-in with --family swift #3537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
oceanio wants to merge 2 commits into
base: main
Choose a base branch
from
+165 −39
Draft

[Other] setup (Mac): Swift 1.5 IQ3_S, opt-in with --family swift #3537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6aba19a
setup (Mac): Swift 1.5 IQ3_S, opt-in with --family swift
Oct 20, 2026
ce64d62
setup (Mac): Swift 1.5 reads images, with its own projector
Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 36 additions & 4 deletions docs/MAC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ API on `http://127.0.0.1:8080/v1`, the Anthropic API on `/v1/messages`, and the
| Option | Meaning |
| --- | --- |
| `--model Q2_0\|IQ3_XXS\|IQ3_S\|UD-Q4_K_XL\|UD-Q5_K_XL\|UD-Q6_K_XL\|Gyro-S\|Gyro-M` | The pack. The `UD-` files are Unsloth's. |
| `--family qwen\|unsloth\|gyro` | The model family. |
| `--family qwen\|unsloth\|gyro\|swift` | The model family. `swift`: Swift 1.5 IQ3_S, opt-in ([below](#swift-15-opt-in)). |
| `--vision yes\|no\|gpu\|cpu` | Picture reader. |
| `--context N`, `--port`, `--host`, `--api-key` | Server settings. |
| `--draft-vocab`, `--no-browser` | Draft vocabulary; do not open the browser. |
Expand Down Expand Up @@ -145,6 +145,37 @@ Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac tested her
checks the decoder against the format definition and the expert kernels against a double-precision reference.
`STRATA_METAL_GYRO_TILES=0` turns off the dense-weight tiles.

### Swift 1.5 (opt-in)

`./setup.sh --family swift` sets up Swift 1.5 IQ3_S. Nothing else changes: without `--family swift` the size menu and every other
model are set up as before.

- **Why setup repacks it:** Swift's two shards cut through a layer (IQ3_S: layer 14's gate is in shard 1, its up and down in shard 2), and the
Metal engine maps one file. Setup writes the experts and dense tensors into one GGUF with `metal/tools/metal_repack.py`, as it
does for Unsloth's files (54.8 GB, 1,223 tensors, every one checked by SHA-256 against its shard). The n-gram table stays in
Swift's shard 1 and is read from the SSD.
- **The pack:** `tools/iq_pack.py` writes Swift's 96 router tensors, stored as F32, as BF16. All 96 are exact (max error 0).
- **The draft layer:** the original model's MTP layer, as on a PC.
- **The engine:** no change. The decode window maps the repacked file (`PLE on`).
- **Images:** on by default, as for the original model, with Swift's own projector (UkisAI's
`mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf`, pinned by SHA-256 as on a PC). Three test pictures were read right:
the count of shapes by color, an invoice's text word for word, and the order of five bars.
- **Sizes:** IQ3_S only. IQ2_XS and IQ3_XXS are not measured on a Mac, so setup does not offer them.

Measured with SC117's abliterated Swift 1.5 IQ3_S (`SC117/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF`, the same file
layout as UkisAI's release), resident, `--spec 2` with MTP, engine 0.1.45 with the Mac build fix (the one 0.2.0 carries). UkisAI's own files are not measured.

| | Swift 1.5 IQ3_S | Qwen3.8-Flash-Next IQ3_S (SC117 abliterated), the same Mac |
|---|---|---|
| Decode, 4 prompts up to 200 tokens (story, Chinese, code, primes), greedy | 93.1 / 97.2 / 110.4 / 104.6 tok/s | 94.3 / 95.1 / 109.2 / 108.5 tok/s |
| 6.8K-token needle, 3 documents | 3 of 3 found | 3 of 3 found |
| Prompt, 6.8K tokens | 1,636-1,731 tok/s | 1,651-1,767 tok/s |
| 3 short reasoning questions, thinking on | 3 of 3 right, 73-136 tokens | 3 of 3 right, 74-76 tokens |

The answers read correctly. Swift's are worded differently from the original's, as a fine-tune's are. Not measured: a
comparison with the CUDA or HIP engine on the same Swift file, the KL harness, contexts over 8K, streaming from the SSD on a
smaller Mac, and a long server run. The three reasoning questions are too easy to show Swift's shorter thinking.

## Measured (M5 Max, 128 GB)

**Measured on:** Apple M5 Max, 128 GB, macOS 27.2, Qwen3.8-Flash-Next packs, greedy, 200 generated tokens, Strata engine version
Expand Down Expand Up @@ -225,9 +256,10 @@ from each other: top-1 agreement 95.4% against 95.8%, mean KL 0.0207 against 0.0
- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.
- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).
- **Streaming** is measured at 32 GB only.
- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured. Setup repacks
Unsloth's shards into one file for the GPU; a Gyro file is used as it is. The Swift and Coder releases are not set up on Mac
yet, and Qwen3.6 does not run on the Mac engine.
- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured. Swift 1.5
IQ3_S is opt-in (`--family swift`) and measured with SC117's abliterated build only (above). Setup repacks Unsloth's and
Swift's shards into one file for the GPU; a Gyro file is used as it is. The Coder release is not set up on Mac yet, and
Qwen3.6 does not run on the Mac engine.

### Quality

Expand Down
60 changes: 38 additions & 22 deletions metal/setup_mac.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,10 @@
import setup as S # noqa: E402
from strata_control import mac_fit as _fit # noqa: E402
from strata_control.mac_fit import ( # noqa: E402,F401 (re-exported: the tests and callers use these names here)
BIG_MAC_GB, CONTEXTS, GIB_GB, GPU_SHARE, MAC_ABOUT, MAC_MODELS, MIN_MACOS, OVERHEAD_GB, RECOMMENDED,
BIG_MAC_GB, CONTEXTS, GIB_GB, GPU_SHARE, MAC_ABOUT, MAC_MODELS, MAC_SWIFT, MIN_MACOS, OVERHEAD_GB, RECOMMENDED,
RESIDENT_LEAVE_SHARE, STREAM_COPY_GAIN, STREAM_MEASURED, STREAM_MTP_FROM_GB, STREAM_SPEC, TESTED_CONTEXT,
_sysctl, gpu_budget_gb, mac_info, mac_problems, macos_tuple, model_need_gb, model_note, recommend_context,
recommend_model, stream_mtp, streams)
_sysctl, gpu_budget_gb, mac_info, mac_model, mac_problems, macos_tuple, model_need_gb, model_note,
recommend_context, recommend_model, stream_mtp, streams)

# Apple's metal-cpp headers for the macOS 26 SDK (https://developer.apple.com/metal/cpp/), pinned by SHA-256
METAL_CPP_URL = "https://developer.apple.com/metal/cpp/files/metal-cpp_26.zip"
Expand All @@ -63,14 +63,21 @@
}


def model_shard_names(model: str) -> list[str]:
def model_shard_names(model: str, family: str | None = None) -> list[str]:
"""The model's files in order (shard 1 first)."""
n = MAC_MODELS[model].get("shards", 2)
if MAC_MODELS[model].get("family") == "gyro": # agentionai's one file (the n-gram table is inside it)
m = mac_model(model, family)
n = m.get("shards", 2)
if m.get("family") == "gyro": # agentionai's one file (the n-gram table is inside it)
return [S.model_file(S.FAMILIES["gyro"], model, 1)]
if MAC_MODELS[model].get("family") == "unsloth":
if m.get("family") == "unsloth":
return [f"Qwen3.8-Flash-Next-{model}-{i:05d}-of-{n:05d}.gguf" for i in range(1, n + 1)]
return [S.model_file(S.FAMILIES["qwen"], model, i) for i in range(1, n + 1)]
return [S.model_file(S.FAMILIES[m.get("family", "qwen")], model, i) for i in range(1, n + 1)]


def repacks(model: str, family: str | None = None) -> bool:
"""Shards that cut through layers (Unsloth's 4-6, Swift's 2) are repacked into one GGUF the GPU maps."""
m = mac_model(model, family)
return m.get("shards", 2) > 2 or m.get("family") == "swift"


def shard_sha(name: str):
Expand Down Expand Up @@ -158,7 +165,9 @@ def engine_args(pack: Path, shards: list, rt: Path, ctx: int, stream: bool = Fal

# the image encoder's projector (setup.py's pinned revision): name -> (bytes, SHA-256)
VISION_SHA256 = {"mmproj-Qwen3.8-Flash-Next-BF16.gguf":
(907543008, "b1a82259702816a5330d7bd7607cd9676b11780e79ff7348c21103ff3ce49bd0")}
(907543008, "b1a82259702816a5330d7bd7607cd9676b11780e79ff7348c21103ff3ce49bd0"),
"mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf":
(907543520, "cd1140f4abba943ce8d9fa92fed8407e389954d26907efde7b72153c75bd8a60")}
VISION_EXE = ROOT / "build-vision" / "bin" / "strata-vision"


Expand Down Expand Up @@ -326,9 +335,9 @@ def start(cfg_path: Path, port: int | None, open_browser: bool = True) -> int:
# ------------------------------------------------------------------------------------------------ main
def main(argv=None) -> int:
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--family", choices=["qwen", "unsloth", "gyro"], default=None,
ap.add_argument("--family", choices=["qwen", "unsloth", "gyro", "swift"], default=None,
help="qwen = Qwen3.8-Flash-Next (GSQ-RCO), unsloth = Unsloth's UD-*_K_XL, gyro = agentionai's Gyro-S / Gyro-M "
"(follows --model)")
"(follows --model), swift = Swift 1.5 (IQ3_S; opt-in)")
ap.add_argument("--model", choices=list(MAC_MODELS))
ap.add_argument("--context", type=int)
ap.add_argument("--port", type=int)
Expand Down Expand Up @@ -389,7 +398,13 @@ def main(argv=None) -> int:
rec, notes = recommend_model(info["ram_gb"])
for n in notes:
S.warn(n)
if a.model:
swift = a.family == "swift"
if swift:
model = a.model or "IQ3_S"
if model not in MAC_SWIFT:
S.fail(f"Swift 1.5 on a Mac: {', '.join(MAC_SWIFT)} only; {model} is not set up here yet",
"run ./setup.sh --family swift --model IQ3_S")
elif a.model:
model = a.model
else:
names = [m for m in MAC_MODELS if not MAC_MODELS[m].get("missing")] # --model offers the others
Expand All @@ -399,7 +414,8 @@ def main(argv=None) -> int:
+ (" (recommended)" if m in RECOMMENDED else "") + (" <- default" if m == rec else ""))
model = names[int(S.ask("Which size?", [str(i) for i in range(1, len(names) + 1)],
str(names.index(rec) + 1), a.yes)) - 1]
for n in model_note(model, info["ram_gb"]):
family = "swift" if swift else MAC_MODELS[model].get("family", "qwen")
for n in model_note(model, info["ram_gb"], family):
S.warn(n)
vision = a.vision or ("yes" if S.ask("Should the model also read images? (the image encoder, ~1 GB more)",
["y", "n"], "y", a.yes) == "y" else "no") # on by default (strata_control/vision_reader.py)
Expand All @@ -412,7 +428,6 @@ def main(argv=None) -> int:
if ctx > TESTED_CONTEXT:
S.say(f" context {ctx}: measured on the Mac up to {TESTED_CONTEXT} (needles found at 8K-32K); longer is kept")
S.ok(f"{model}, context {ctx}")
family = MAC_MODELS[model].get("family", "qwen")
if a.family and a.family != family:
S.warn(f"--family {a.family}: {model} is a {family} file; using {family}")
fam = S.FAMILIES[family]
Expand Down Expand Up @@ -451,7 +466,7 @@ def main(argv=None) -> int:
S.step(5, f"downloading {fam['title']} {model}")
models_dir = Path(a.gguf_dir) if a.gguf_dir else models_root / tag
models_dir.mkdir(parents=True, exist_ok=True)
files = [models_dir / n for n in model_shard_names(model)]
files = [models_dir / n for n in model_shard_names(model, family)]
for s in files:
if s.exists() and S.done(s):
S.ok(f"{s.name} already downloaded")
Expand Down Expand Up @@ -480,9 +495,10 @@ def main(argv=None) -> int:
# The repack and the pack move 100+ GB through the disk and the pack's python holds up to 20 GB: no engine may run beside
# them (an engine of 70-90 GB with a repack froze the 128 GB M5 Max twice). A lock beside the GPU's file tells the engine
# and start() to wait; a running engine stops setup here.
lock_dir = data / "metal" if len(files) > 2 else files[0].parent
repack = repacks(model, family)
lock_dir = data / "metal" if repack else files[0].parent
native0 = data / "metal" / f"{tag.lower()}-metal.gguf"
work = (len(files) > 2 and not (native0.exists() and S.done(native0))) or \
work = (repack and not (native0.exists() and S.done(native0))) or \
not ((pack / "native_experts.txt").exists() and (pack / "tokenizer" / "vocab.json").exists())
prep_lock = lock_dir / PREPARE_LOCK
if work:
Expand All @@ -493,14 +509,14 @@ def main(argv=None) -> int:
lock_dir.mkdir(parents=True, exist_ok=True)
prep_lock.write_text(str(os.getpid()))
atexit.register(lambda: prep_lock.unlink() if prep_lock.exists() else None)
if len(files) > 2:
# many shards: the experts and dense tensors into one GGUF the GPU maps (or streams), the n-gram table read
# from its own shard (every tensor checked byte for byte: --verify)
if repack:
# shards that cut through layers: the experts and dense tensors into one GGUF the GPU maps (or streams), the
# n-gram table read from its own shard (every tensor checked byte for byte: --verify)
native = data / "metal" / f"{tag.lower()}-metal.gguf"
native.parent.mkdir(parents=True, exist_ok=True)
if not (native.exists() and S.done(native)):
S.say(f" Writing the model's experts and weights into one file for the GPU (one time, "
f"{MAC_MODELS[model]['shard1_gb']:.0f} GB) ...")
f"{mac_model(model, family)['shard1_gb']:.0f} GB) ...")
S.run([sys.executable, str(ROOT / "metal" / "tools" / "metal_repack.py"), "--shard1", str(files[0]),
"--out", str(native), "--verify"])
S.mark(native)
Expand All @@ -517,7 +533,7 @@ def main(argv=None) -> int:
prep_lock.unlink(missing_ok=True)
mtp = (S.find_in(roots, "mtp/mtp-q2_0.gguf") or data / "mtp" / "mtp-q2_0.gguf").parent
rt = mtp / "rt"
stream = streams(model, info["ram_gb"])
stream = streams(model, info["ram_gb"], family)
use_mtp = stream_mtp(stream, info["ram_gb"])
if stream and not use_mtp:
S.ok("the experts will stream from the SSD: no MTP draft layer (one token per step is faster there); "
Expand Down
36 changes: 23 additions & 13 deletions strata_control/mac_fit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@
"Gyro-S": {"shard1_gb": 29.7, "tested": True, "family": "gyro", "shards": 1, "file_gb": 58.5},
"Gyro-M": {"shard1_gb": 63.2, "tested": False, "family": "gyro", "shards": 1, "file_gb": 92.0},
}
# Swift 1.5 (UkisAI's fine-tune, --family swift; opt-in, never offered by the size menu): its two shards cut through a
# layer (IQ3_S: layer 14's gate in shard 1, its up and down in shard 2) and shard 1 holds the n-gram table, so setup
# repacks it like Unsloth's files. shard1_gb: the repacked file (IQ3_S: 54.8 GB measured). Only IQ3_S is measured.
MAC_SWIFT = {"IQ3_S": {"shard1_gb": 54.8, "tested": True, "family": "swift"}}

# The recommended sizes (owner's decision): IQ3_S is the default where it stays resident, UD-Q4_K_XL the higher-quality one
RECOMMENDED = ("IQ3_S", "UD-Q4_K_XL") # the owner's recommended sizes (UD-IQ4_XS is not offered on Mac yet)
Expand Down Expand Up @@ -133,13 +137,18 @@ def gpu_budget_gb(ram_gb: float) -> float:
return ws


def model_need_gb(model: str) -> float:
return MAC_MODELS[model]["shard1_gb"] + OVERHEAD_GB
def mac_model(model: str, family: str | None = None) -> dict:
"""The pack's entry: Swift's own sizes for --family swift, MAC_MODELS otherwise."""
return MAC_SWIFT[model] if family == "swift" else MAC_MODELS[model]


def streams(model: str, ram_gb: float) -> bool:
def model_need_gb(model: str, family: str | None = None) -> float:
return mac_model(model, family)["shard1_gb"] + OVERHEAD_GB


def streams(model: str, ram_gb: float, family: str | None = None) -> bool:
"""The pack does not fit Metal's share of the memory: the engine streams its experts from the SSD."""
return model_need_gb(model) > gpu_budget_gb(ram_gb)
return model_need_gb(model, family) > gpu_budget_gb(ram_gb)


def recommend_model(ram_gb: float) -> tuple[str, list[str]]:
Expand All @@ -161,18 +170,19 @@ def recommend_model(ram_gb: float) -> tuple[str, list[str]]:
"draft layer (slower when streaming)"]


def model_note(model: str, ram_gb: float) -> list[str]:
def model_note(model: str, ram_gb: float, family: str | None = None) -> list[str]:
"""What a chosen pack risks on this Mac (said, never refused)."""
notes = []
if streams(model, ram_gb):
notes.append(f"{model} needs ~{model_need_gb(model):.0f} GB of the {gpu_budget_gb(ram_gb):.0f} GB Metal keeps "
"resident on this Mac: its experts stream from the SSD (automatic)"
+ (f"; measured on a 32 GB M4: {STREAM_MEASURED[model]}" if model in STREAM_MEASURED else
"; not measured streaming"))
if MAC_MODELS[model].get("missing"):
notes.append(f"{model} does not run on this Metal engine yet ({MAC_MODELS[model]['missing']} are missing): the "
m = mac_model(model, family)
if streams(model, ram_gb, family):
notes.append(f"{model} needs ~{model_need_gb(model, family):.0f} GB of the {gpu_budget_gb(ram_gb):.0f} GB Metal "
"keeps resident on this Mac: its experts stream from the SSD (automatic)"
+ (f"; measured on a 32 GB M4: {STREAM_MEASURED[model]}"
if model in STREAM_MEASURED and family != "swift" else "; not measured streaming"))
if m.get("missing"):
notes.append(f"{model} does not run on this Metal engine yet ({m['missing']} are missing): the "
"engine will stop when it loads; kept as you chose (for testing a newer engine)")
elif not MAC_MODELS[model]["tested"]:
elif not m["tested"]:
notes.append(f"{model} has not been measured on a Mac yet")
return notes

Expand Down
Loading
Loading