Fork of dreamingwell/apple-llm-performance — credit to its author. Maintained for my own use; a work in progress, so some things may be wrong or break.
Open-weight models × Apple silicon. Pick your Mac, pick a job, see what fits.
Open-weights model from Kimi. Intelligence: 46
Open-weights model from Tencent. Intelligence: 35
Open-weights model from Alibaba. Intelligence: 34
Open-weights model from DeepSeek. Intelligence: 33
Open-weights model from Z AI. Intelligence: 32
Open-weights model from Google. Intelligence: 31
Open-weights model from Thinking Machines. Intelligence: 31
Open-weights model from InclusionAI. Intelligence: 30
Open-weights model from Xiaomi. Intelligence: 30
Open-weights model from MiniMax. Intelligence: 29
Open-weights model from Alibaba. Intelligence: 29
Open-weights model from StepFun. Intelligence: 29
Open-weights model from Z AI. Intelligence: 28
Open-weights model from Xiaomi. Intelligence: 27
Open-weights model from Mistral. Intelligence: 26
Open-weights model from Meta. Intelligence: 25
Open-weights model from Sapiens AI. Intelligence: 24
Open-weights model from DeepSeek. Intelligence: 23
Open-weights model from Nanbeige. Intelligence: 20
Open-weights model from Alibaba. Intelligence: 20
Open-weights model from China Mobile. Intelligence: 19
Open-weights model from MiniMax. Intelligence: 19
Open-weights model from Reflection. Intelligence: 18
Open-weights model from Institute of Foundation Models. Intelligence: 18
Open-weights model from Apodex. Intelligence: 14
Open-weights model from NVIDIA. Intelligence: 14
Open-weights model from Upstage. Intelligence: 14
Open-weights model from Thinking Machines. Intelligence: 13
Open-weights model from Google. Intelligence: 12
Open-weights model from Thinking Machines. Intelligence: 12
Open-weights model from NVIDIA. Intelligence: 12
Open-weights model from LG AI Research. Intelligence: 8
Open-weights model from Cohere. Intelligence: 6
Open-weights model from NVIDIA. Intelligence: 6
Open-weights model from IBM. Intelligence: 5
Open-weights model from IBM. Intelligence: 5
Open-weights model from Mistral. Intelligence: 5
Open-weights model from Google. Intelligence: 4
Open-weights model from Mistral. Intelligence: 4
Open-weights model from Cohere. Intelligence: 3
Open-weights model from NVIDIA. Intelligence: 2
Open-weights model from SK Telecom. Intelligence: None
Open-weights model from Sapiens AI. Intelligence: None
Open-weights model from Swiss AI Initiative. Intelligence: None
Open-weights model from Swiss AI Initiative. Intelligence: None
Open-weights model from ServiceNow. Intelligence: None
Open-weights model from Celeris. Intelligence: None
Open-weights model from Deep Cogito. Intelligence: None
Open-weights model from Cohere. Intelligence: None
Open-weights model from Nous Research. Intelligence: None
Open-weights model from Nous Research. Intelligence: None
Open-weights model from DeepSeek. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from ByteDance Seed. Intelligence: None
Open-weights model from Baidu. Intelligence: None
Open-weights model from Baidu. Intelligence: None
Open-weights model from LG AI Research. Intelligence: None
Open-weights model from LG AI Research. Intelligence: None
Open-weights model from LG AI Research. Intelligence: None
Open-weights model from TII UAE. Intelligence: None
Open-weights model from AI9Stars. Intelligence: None
Open-weights model from AI9Stars. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from Google. Intelligence: None
Open-weights model from IBM. Intelligence: None
Open-weights model from IBM. Intelligence: None
Open-weights model from IBM. Intelligence: None
Open-weights model from IBM. Intelligence: None
Open-weights model from IBM. Intelligence: None
Open-weights model from IBM. Intelligence: None
Open-weights model from Nous Research. Intelligence: None
Open-weights model from Nous Research. Intelligence: None
Open-weights model from Tencent. Intelligence: None
Open-weights model from Naver. Intelligence: None
Open-weights model from Multiverse Computing. Intelligence: None
Open-weights model from Prime Intellect. Intelligence: None
Open-weights model from China Mobile. Intelligence: None
Open-weights model from China Mobile. Intelligence: None
Open-weights model from AI21 Labs. Intelligence: None
Open-weights model from AI21 Labs. Intelligence: None
Open-weights model from AI21 Labs. Intelligence: None
Open-weights model from Institute of Foundation Models. Intelligence: None
Open-weights model from Institute of Foundation Models. Intelligence: None
Open-weights model from Institute of Foundation Models. Intelligence: None
Open-weights model from Institute of Foundation Models. Intelligence: None
Open-weights model from Institute of Foundation Models. Intelligence: None
Open-weights model from Institute of Foundation Models. Intelligence: None
Open-weights model from KwaiKAT. Intelligence: None
Open-weights model from Kimi. Intelligence: None
Open-weights model from Kimi. Intelligence: None
Open-weights model from Liquid AI. Intelligence: None
Open-weights model from Liquid AI. Intelligence: None
Open-weights model from Liquid AI. Intelligence: None
Open-weights model from Liquid AI. Intelligence: None
Open-weights model from Liquid AI. Intelligence: None
Open-weights model from Liquid AI. Intelligence: None
Open-weights model from InclusionAI. Intelligence: None
Open-weights model from InclusionAI. Intelligence: None
Open-weights model from InclusionAI. Intelligence: None
Open-weights model from InclusionAI. Intelligence: None
Open-weights model from Meta. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from Meta. Intelligence: None
Open-weights model from Meta. Intelligence: None
Open-weights model from Meta. Intelligence: None
Open-weights model from Meta. Intelligence: None
Open-weights model from Meta. Intelligence: None
Open-weights model from LongCat. Intelligence: None
Open-weights model from LongCat. Intelligence: None
Open-weights model from Mistral. Intelligence: None
Open-weights model from Mistral. Intelligence: None
Open-weights model from Inception. Intelligence: None
Open-weights model from Korea Telecom. Intelligence: None
Open-weights model from OpenBMB. Intelligence: None
Open-weights model from OpenBMB. Intelligence: None
Open-weights model from OpenBMB. Intelligence: None
Open-weights model from Mistral. Intelligence: None
Open-weights model from Mistral. Intelligence: None
Open-weights model from Mistral. Intelligence: None
Open-weights model from Allen Institute for AI. Intelligence: None
Open-weights model from Allen Institute for AI. Intelligence: None
Open-weights model from Motif Technologies. Intelligence: None
Open-weights model from Motif Technologies. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from Nanbeige. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from NVIDIA. Intelligence: None
Open-weights model from Nex AGI. Intelligence: None
Open-weights model from Amazon. Intelligence: None
Open-weights model from Amazon. Intelligence: None
Open-weights model from Amazon. Intelligence: None
Open-weights model from Amazon. Intelligence: None
Open-weights model from Amazon. Intelligence: None
Open-weights model from Allen Institute for AI. Intelligence: None
Open-weights model from Allen Institute for AI. Intelligence: None
Open-weights model from Allen Institute for AI. Intelligence: None
Open-weights model from Allen Institute for AI. Intelligence: None
Open-weights model from Microsoft. Intelligence: None
Open-weights model from Microsoft. Intelligence: None
Open-weights model from Microsoft. Intelligence: None
Open-weights model from Multiverse Computing. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Alibaba. Intelligence: None
Open-weights model from Perplexity. Intelligence: None
Open-weights model from Reka AI. Intelligence: None
Open-weights model from InclusionAI. Intelligence: None
Open-weights model from InclusionAI. Intelligence: None
Open-weights model from Sarvam. Intelligence: None
Open-weights model from Sarvam. Intelligence: None
Open-weights model from Upstage. Intelligence: None
Open-weights model from Upstage. Intelligence: None
Open-weights model from Upstage. Intelligence: None
Open-weights model from StepFun. Intelligence: None
Open-weights model from Cohere. Intelligence: None
Open-weights model from Trillion Labs. Intelligence: None
Open-weights model from Arcee AI. Intelligence: None
Zyphra's 800M-parameter open TTS (about 100M active per token, latent MoE): 43 languages, 27 built-in voices, voice cloning from a reference clip, 44.1 kHz mono. MLX-Audio lists it in its TTS table and mlx-community ships the quants; llama.cpp merged mtmd support in November 2026 and ggml-org ships the GGUFs.
In MLX-Audio's TTS table with mlx-community/ZONOS3-Mini; the engine's model README ships CLI and Python examples with the 27 built-in voices, voice cloning and streaming. llama.cpp also merged mtmd TTS support in November 2026 (PR 32020) and ggml-org ships BF16 and Q8_0 GGUFs.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
Thinking Machines' first looped transformer, released Nov 20: a 26.4B MoE (3.4B active) whose middle 12 of 24 layers run three times over shared weights, so each token costs 5.4B effective active parameters. Image+text in, 1M context, about 20 GB in the lab's NVFP4/MXFP8 checkpoint - the card targets 32 GB Apple silicon, and at 32 KiB/token of KV a full 1M context is another 32 GB on top of the weights. AA Index 20.1, level with Qwen3.8-27B on the lab's own table. llama.cpp and Ollama ship local support; mlx-lm has no model class yet.
Inkling 2 text+vision merged Nov 14 (PR 32217) with chat-template, attention-mask and mmproj fixes, and the looped arch landed Nov 20 (PR 32694); all of it is in release b12602, the current one. A looped-arch follow-up (PR 32695) is still open and the lab itself says local support for looped models is new, so first runs are unverified here. ggml-org ships the GGUFs.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
In the Ollama library (inkling-2-mini) with the lab's benchmark summary in the model description; the card lists Ollama on Apple silicon as a supported local path. On Mac, Ollama defaults to the MLX engine since v0.30 and mlx-lm has no inkling2 class, so whether the Mac MLX path serves it is unverified - the GGUF path should work.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
lmstudio-community ships the GGUF quants and the app's catalogue lists them, but loading needs the looped-arch code that landed in upstream llama.cpp only on Nov 20 (PR 32694); the app's bundled engine must be current, which is unverified.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
oMLX's LLM coverage is mlx-lm's plus its hand-written kernels for GLM-5.2, MiniMax M3 and Qwen3.5; there is no inkling2 support, so it cannot load the mlx-community quants.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
No inkling2 implementation in the vLLM Metal fork as of Nov 21; upstream vLLM ships a CUDA recipe, which the card links.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
No inkling2 implementation in vllm-mlx as of Nov 21; upstream vLLM ships a CUDA recipe, which the card links.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
mlx-lm has no inkling2 or inkling2_loop model class as of Nov 21; mlx-community already ships 4-bit and oQ4e-mtp MLX quants that nothing can load until the class lands.
0 open of 0 tracked on this engine
Nothing open tracked against this engine for this model.
What shipped, broke and moved on Apple silicon today — from r/LocalLLaMA, Hugging Face, the lab blogs and the engine release feeds.
A new open-weight MoE from the Maxwell Team, trending on Hugging Face: 35B total parameters with only 3B active per token, 256K-token context, 45 transformer layers. Its headline feature is Maxwell Block Sparse Attention, which bounds each KV group's main attention at 4,096 tokens while keeping the full-history KV cache available, and the card claims the MBSA kernel is 6.7x faster than dense GQA at 256K - the economics long-context inference on a Mac has been missing. No MLX or GGUF quant has landed yet, so nothing on Apple silicon can load it today; with 35B total parameters the first 4-bit conversion would be roughly 18 GB resident.
Microsoft's Windows Applied Sciences Group published Aion 1.0 Instruct, the foundational small language model for the Windows ecosystem: 2.3B parameters, 32K context, eight languages, MIT-licensed, and aimed at on-device document comprehension, summarization and chat. This is the unquantized preview - the card says quantized versions follow, and an optimized Aion paired with a content-moderation model will ship as the Windows OS SLM by the end of January 2027. GGUF is already listed as a supported format, and at 2.3B the model is about 4.6 GB in 16-bit, so it fits comfortably on any Mac once a quant lands.
NVIDIA released an official external speculative-decoding draft for Qwen3.8-27B, the flagship this page tracks: a complete 4.21B model rather than a draft head, sharing the target's tokenizer and proposing blocks of candidate tokens that Qwen3.8-27B verifies by rejection sampling, which cuts decode latency without changing the target's output distribution. The Qwen3.5-4B draft base is Apache 2.0 and the checkpoint ships under the NVIDIA Open Model Agreement. The card officially targets only vLLM on NVIDIA Blackwell under Linux; using it on a Mac would mean vLLM on Metal with the 4.2B draft resident alongside the 27B target, a path the card does not cover and that remains unverified.
Nothing new shipped in the last day. The rollup re-checks on the next update.
What each engine is, what its API actually implements, and the defects that follow
you whichever model you load on it. All seven speak OpenAI on /v1/chat/completions with SSE
streaming, and none is desktop-only — but “OpenAI-compatible” covers a wide range, and the
differences land on exactly the features an agent leans on: whether tool-call arguments stream as deltas or
arrive only after the turn, whether constrained decoding exists at all, and whether tool_choice
is implemented. Five of the seven also serve Anthropic /v1/messages, so Claude Code can point
at them directly.
The reference GGUF runtime, with a first-class Metal backend. New architectures land here earlier and more completely than anywhere else on this page, and Ollama, LM Studio, Jan, KoboldCpp and most other local runners are downstream of it. llama-server exposes an OpenAI-compatible endpoint, so nothing here is GUI-only.
usage object plus a timings block that reports cache_n - how many prompt tokens were reused from cache - which is the number you want when tuning an agent loop.tool_choice, and parallel_tool_calls gated on what the jinja template supports. Arguments stream as deltas.response_format accepts both json_object and json_schema, enforced by GBNF grammar sampling at the token level rather than validated afterwards.-np), with per-slot prompt caches you can persist to disk.--jinja flag on both the OpenAI and Anthropic surfaces - without it tools is silently inert. The project's own docs decline to claim spec compliance: “no strong claims of compatibility with OpenAI API spec is being made”.Closed 2026-09-03 as not planned - a decision not to fix rather than a fix, so the defect stands. Constrained decoding is how tool calls are kept well-formed, and past some number of tools the generated grammar fails to parse, which is to say the failure arrives exactly when an agent has enough tools to be useful. Read the closed marker as 'no longer being worked on'.
Filed on Glimmer. 50 KB is an ordinary agent turn once a file or a diff is in the prompt, and the process dies rather than rejecting the request.
GLM-5.2's DSA attention has no V cache to quantise, but the flag pair is validated as if it did, so you cannot set the K type independently. Costs memory on the model that has the least to spare.
Blocks speculative decoding on exactly the builds that carry a sliding-window key. The target model still runs; you lose the draft head's speedup.
One-command pulls from a curated library, running as a launch-agent server. Since v0.30 (May 2026) MLX is the default engine on Apple Silicon rather than llama.cpp, which is the single most important thing to know about it here: on a Mac, Ollama inherits MLX's architecture coverage for the models it serves through that path.
stream_options. The only engine here that publishes an explicit checkbox matrix of what it does and does not implement, which is worth more than most of the feature lists.tool_choice is not implemented - you cannot force a specific call or require that one happen. Also missing: logprobs, n, logit_bias, user.response_format and JSON mode supported./v1/responses is non-stateful only: no previous_response_id, no conversation. And because MLX is now the default engine on Apple Silicon, a tag can resolve to a different backend than you expect.An M5-specific crash in the Metal matmul path. If you are buying an M5 machine, this is the one to check is closed before you plan around Ollama on it.
The call survives and the accompanying text does not. Silent truncation is worse than an error because the agent proceeds on a partial turn.
The server rejects output produced by the model it shipped. From the agent's side it is a hard failure on a turn that was otherwise correct.
The tag says MLX and the manifest says something else, so you may not be running what you think you are. Worth knowing generally: an Ollama tag is a manifest, not a guarantee about the underlying build.
Fixed, closed 2026-09-04. The automatic context sizing could commit more memory than the machine had, and on a Mac that is unified memory, so it took the whole machine rather than just a GPU. Kept listed so the fix is dated.
An index-out-of-range in the MLX runner's dense MLP forward. Importing an ordinary mlx-community checkpoint is a normal thing to want to do.
HTTP 200, plausible usage numbers, and useless vectors. Not an LLM path, but it says something about how failures surface here.
App-first but not app-only. It ships both a llama.cpp engine and its own MLX engine, so for any given model it can take whichever path works, and its catalogue tends to carry a curated quant within a day of a release. The lms CLI and the server run headless.
delta.tool_calls[].function.arguments fragments you accumulate across chunks - the correct OpenAI shape.json_schema only - json_object mode is not supported. Enforcement differs by engine: llama.cpp grammars for GGUF, Outlines for MLX. SDK integrations bind to Zod, Pydantic and msgspec.lms.Filed against a hybrid linear-attention model - the Qwen3.8 shape. A silent 4864-token ceiling makes any agent workload fail in a way that looks like the model being stupid rather than the engine being misconfigured.
The GGUF build of the same model handles them. A vision model that invents what it cannot see is the worst available failure mode, and it is engine-specific.
The capability badge is wrong, so the app steers you away from working tool-calling builds. Cosmetic in code, decision-changing in practice.
Over-allocation relative to the model. On a 256 GB machine running a 200 GB model there is no slack to absorb it.
Tool-call parsing is per-family, so a model whose format has no parser returns its calls as text. Track this if your model is not in the supported list.
A catalogue-side failure rather than an inference one, but the catalogue is how most people get models into this app.
MLX serving built for agent clients: continuous batching, a hot/cold KV cache that spills blocks to SSD and survives a restart, and hand-written Metal kernels for the GLM-5.2, MiniMax M3 and Qwen3.5 families. Its LLM coverage is mlx-lm's, plus those kernels and its own additions - install the custom kernels or the affected families fall back silently to a much slower generic path.
stream_options.include_usage, and SSE keep-alives so a long prefill does not read-timeout the client. It also scales reported token counts so Claude Code's auto-compact triggers at the right moment on a smaller-context model.functions.*.Server-wide, not per model. Discovery racing an in-flight load leaks the whole resident model; on a 256 GB machine that is the entire budget gone until you restart.
Closed as completed in July 2026, with memory also fluctuating between 400 and 520 GB on a 512 GB M3 Ultra. Kept on the list as the reference point for what a healthy GLM-5.2 prefill looks like, and as a reminder that a version bump can cost you a factor of four on this model.
vLLM itself, running on Apple Silicon. This is a plugin in the vllm-project org that keeps vLLM core and swaps the compute layer for MLX, unifying MLX and PyTorch under one lowering path - so you get vLLM's scheduler, paged KV, continuous batching and API surface rather than a lookalike. Despite the name it is an MLX engine underneath: it pins a single exact MLX version, pulls in mlx-lm and mlx-vlm, and builds its Metal kernels as MLX primitives. Two things make it worth attention on new hardware: v0.2.0's unified paged varlen Metal kernel claims 83x TTFT and 3.6x throughput over v0.1.0, and as of August 2026 it uses the M5 Neural Accelerator tensor units to speed up MHA, GQA and MQA prefill - the only engine here that claims M5-specific acceleration. The cost is coverage: its model matrix is a curated list, not everything that exists.
Fixed. PR #648 normalises disabled top_k rows to vocab_size and merged 2026-08-27, closing this. It mattered because continuous batching puts requests with different sampling parameters in one batch, so it was reachable with ordinary mixed traffic. Kept listed so the fix is dated: you want a build from 2026-08-27 or later.
Until this exists, latent-attention models fall back to MLX SDPA with no Metal kernel - which is why the GLM-4.5 row in the support matrix is flagged as slow and untested.
Each request re-ingests the full prompt into the draft model, which costs more than the draft saves. Worth knowing before you reach for spec decode here; the built-in MTP path (#610) is the one to watch.
Design work on how backends get selected per model. Useful for judging how settled the internals are.
A paged KV cache and continuous batching over mlx-lm's model classes. Unaffiliated with vllm-project/vllm despite the name. Because it wraps mlx-lm rather than reimplementing models, an architecture missing upstream is missing here, and mlx-lm's bugs arrive intact.
cached_tokens is not surfaced yet - an open PR - so you cannot see prefix-cache hits from the API.--reasoning-parser.response_format with json_schema.finish_reason, closing a stream mid-flight leaves the generator and request state open, and a strict json_schema decode can wedge. Also bind carefully - non-loopback requests have been reported silently dropped on 0.0.0.0.Over 5-minute hangs on Qwen3-Coder-30B-A3B. Structured output is load-bearing for tool arguments.
Deployment blocker: a server on the LAN is reachable only from localhost. Silent — no log, recv-queue 0.
Filed as an OOM on small Macs, but it cuts the other way here: a hardcoded 32GB ceiling on a 256GB box leaves most of the machine unused.
Fixed, closed 2026-08-30. It mattered because a streaming tool call that ends without a terminator is exactly the shape that reads as a fabricated turn downstream. Kept listed so the fix is dated.
Same class as mlx-lm #1197: mlx-vlm handles a checkpoint the mlx-lm path does not.
Two allocators with different ideas of the budget. Relevant when hot-swapping checkpoints for A/B runs.
Client disconnects are routine in agentic work; leaked request state accumulates.
Observability. Without it there is no way to confirm a hit rate from the API — which is how #730 stayed silent.
Apple's own reference implementation, and the substrate almost everything MLX-shaped depends on. Treat it as the floor: an architecture with no model class here is absent from most of the MLX ecosystem at once. The bundled server is deliberately basic - one model, simple batching - so it is a correctness reference more than a serving layer.
stream_options.include_usage and prompt_tokens_details for cached prompt tokens.tool_calls are parsed and returned. It also implements logprobs and top_logprobs up to 11, which Ollama does not - a genuine inversion of the usual ordering.response_format, no json_schema and no grammar support anywhere in the server. If your agent depends on constrained decoding, this engine cannot give it to you.seed is set - so speculative decoding and concurrency are mutually exclusive here.The json_tools parser never matches, so the call is returned as plain text. Architecture-independent and specific to agent use.
load_model() ends with one mx.eval(model.parameters()), building a single enormous Metal command buffer. At ~390GB it hits kIOGPUCommandBufferCallbackErrorTimeout and the error escapes uncaught, hard-aborting the process. GLM-5.2 at 4-bit is 372-475GB.
Fixed upstream. #1784 makes decode evaluate cache state periodically and #1790 fixes the unevaluated graph in deepseek specifically; both merged, and this closed 2026-08-27. It was the general form of the DeepSeek V4 residency growth - any model class written that way leaked for the life of the process, aborting at around 11k generated tokens rather than failing on the first run.
A single-purpose C and Metal engine for DeepSeek V4 Flash, DeepSeek V4 PRO and GLM-5.2 - deliberately not a general GGUF loader, so only the published checkpoints load. In exchange it gets a disk KV cache that persists sessions across restarts, resident session batching, SSD expert streaming for machines too small to hold the weights, and tensor parallelism across two Macs over Thunderbolt RDMA. Where it applies, it is the fastest option here.
-m.stream_options.include_usage. In thinking mode reasoning streams on its own channel instead of being mixed into the final text, and the Responses surface emits the full Codex event lifecycle - response.output_text.delta, function-call argument events, and terminal response.completed / incomplete / failed.tools and tool_choice on all three surfaces. Schemas are rendered into DeepSeek's DSML format and generated calls mapped back. Tool calls stream incrementally: the header goes out as soon as the DSML invocation is recognised, then argument bytes are forwarded as tool_calls[].function.arguments deltas while generation continues.response_format or grammar support; correctness comes from DSML canonicalisation rather than constrained decoding.--batched-session N preallocates N resident KV sessions with fair scheduling, and idle slots persist to the disk KV cache before reuse. MTP speculative decoding is disabled while native session batching is active.--cors for browser clients; --host 0.0.0.0 is opt-in.Most agent clients are stateless - they resend a longer prompt each turn. If the session is not extended, the disk KV cache and prefix reuse stop paying, which is the main reason to run ds4.
Hits the distributed path specifically. If you are splitting DeepSeek V4 PRO or GLM-5.2 across two Macs, this is the first thing to check when the numbers look impossible.
Tokenization is pure fixed cost before any GPU work. An agent that pastes a file into the prompt pays it on every turn, and it does not show up in the tok/s figures.
Swap quantisations and the cache from the old weights can be restored against the new ones. Silent wrong-context, and it survives a restart because that is what the disk cache is for.
Unconfirmed. On a machine where the model already occupies most of RAM, a slow server-side leak ends as an OOM rather than as swap.
A single report without a resolved cause. Listed because a full-machine crash is a different risk class from a process crash.
You build from a moving main branch. The project describes itself as beta and fast-changing, so pin a commit yourself if you care about reproducibility.
A line-by-line MLX port of a dozen image model families, written from scratch rather than wrapping diffusers - FLUX.2, Z-Image, Qwen Image, Krea 2, Ideogram 4, FIBO and more, plus SeedVR2 for upscaling and Depth Pro for depth. It is deliberately minimal and explicit, and it is the most active image runtime on Apple Silicon by a distance. There is no server: you drive it from the CLI or import it, so putting it behind an API is your job.
Nothing server-wide tracked against this engine.
The centre of gravity for audio on Apple Silicon, and the largest catalogue on this page: twenty-odd TTS families, a similar spread of speech-to-text, speaker diarisation, speech enhancement and music generation. If an audio model has an MLX port at all, this is usually where it lives.
Nothing server-wide tracked against this engine.
Inference and finetuning for image, video and audio generation models. Same author as MLX-Audio. Worth checking the commit history before planning around it - it moves in bursts rather than continuously.
Nothing server-wide tracked against this engine.
Argmax's on-device image generation for Apple Silicon, covering both MLX and Core ML. Solid engineering, but it has not been touched since April 2025, so it predates every model added to this page and should be treated as a reference implementation rather than a live option.
Nothing server-wide tracked against this engine.
Disclaimer. All of this is best effort and provided for entertainment purposes only. No warranty is given as to its accuracy. Benchmark scores are vendor- or aggregator-reported and are not reproduced here; issue states are a twice-daily snapshot; hardware figures are arithmetic, not measurements. Verify anything you intend to spend money on.