Home / Open Source / Inference
All Inference items under Open Source on AIFIRST News, newest first. 9 items.
The ConvRot quantization method, which outperforms standard UDQ8_K_XL in accuracy benchmarks, has been integrated into the llama-cpp-turboquant fork. ConvRot applies a learned rotation to model weights before quantization, reducing the accuracy loss from low-bit formats. The fork is available on GitHub for users who want better quantization quality beyond standard GGUF methods.
A developer ran a 3-day benchmark of Inco AI's DFlash 2 speculative decoding PR in llama.cpp against plain decoding, MTP, n-gram lookup drafters, and the previous DFlash 1 on Qwen 3.8 27B. DFlash 2 introduces a trained drafter specifically for Qwen 3.8 27B and shows meaningful throughput improvements over all comparison methods.
A developer released a llama.cpp fork with HIP optimizations targeting AMD GFX906-class GPUs including the Mi50, Mi60, Radeon VII, and GCN HIP cards. These cards have minimal support in the mainline llama.cpp codebase, leaving many AMD users without an optimized inference path.
AI-Researcher is a self-hosted Python 3.11 dashboard that pulls AI news from roughly 60 sources including RSS, Reddit, Hacker News, arXiv, Hugging Face, GitHub, Google News, and X. It deduplicates and clusters items with optional nomic-embed-text embeddings and ranks them with engagement normalization that weights corroboration across sources. A local Ollama model summarizes the high-priority items. All processing is local with no mandatory third-party API keys. The repository had 4 commits and 0 stars when checked.
llama.cpp is a C/C++ implementation of large language model and vision language model inference, built on the ggml tensor library and MIT licensed. It ships a command line interface, an OpenAI-compatible API server, and a web chat UI, installable as an app, via Docker, from pre-built binaries, or from source. Backends cover Apple Silicon with ARM NEON, Accelerate and Metal, x86 with AVX through AVX512 and AMX, NVIDIA CUDA, AMD HIP, Vulkan and SYCL, plus hybrid CPU and GPU inference. The repository showed 124.9k stars and 22.0k forks with 778 open issues and roughly 1.4k open pull requests, and quantization spans 1.5-bit to 8-bit so output quality depends on the level chosen.
Huzzah is an experimental editor that uses pseudocode files (.hz) instead of natural language prompts to direct AI code generation. When a .hz file is saved, Huzzah generates real code; subsequent edits capture only the diff, reducing token consumption and preserving a persistent record of developer intent. The approach positions coding as design, with specs that are language-agnostic and reusable across multiple implementation targets. The author notes limitations including scalability concerns and challenges with cross-file dependencies.
Vomit is a small open-source tool that intercepts Claude agent mode's intermediate reasoning output and pipes it through a local LLM to produce more readable English summaries. It runs fully locally with no telemetry, supports Llama.app and Ollama backends, and can be used either as an integrated hook replacing Claude's displayed output or as a standalone session monitor via the 'vomit tail' command. The tool targets the verbosity of Claude's token output during agentic tasks.
Shoehorn is a locally run, browser-based tool that quantizes language models with hardware-aware mixed precision, assigning a precision level per tensor rather than applying a preset and targeting about 99.99% of available memory. It scans popular Hugging Face models to work out what fits a given machine and writes standard GGUF v3 files for llama.cpp. It supports macOS on Apple Silicon, Linux on x86-64, and Windows on x86-64. The quantizer core is written from scratch in Rust.
DataMoat auto-captures session files from Claude CLI, Codex, Cursor and OpenClaw, normalizes them into a common schema and encrypts with AES-256-GCM. Thesis: AI work traces are private data assets worth preserving.
AIFIRST 
