r/MachineLearning • • 24d ago

Discussion [D] Self-Promotion Thread

3 Upvotes

Please post your personal projects, startups, product placements, collaboration needs, blogs etc.

Please mention the payment and pricing requirements for products and services.

Please do not post link shorteners, link aggregator websites , or auto-subscribe links.

--

Any abuse of trust will lead to bans.

Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

--

Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.


r/MachineLearning • • 25d ago

Discussion [D] Monthly Who's Hiring and Who wants to be Hired?

10 Upvotes

For Job Postings please use this template

Hiring: [Location], Salary:[], [Remote | Relocation], [Full Time | Contract | Part Time] and [Brief overview, what you're looking for]

For Those looking for jobs please use this template

Want to be Hired: [Location], Salary Expectation:[], [Remote | Relocation], [Full Time | Contract | Part Time] Resume: [Link to resume] and [Brief overview, what you're looking for]

​

Please remember that this community is geared towards those with experience.


r/MachineLearning • • 4h ago

Research Fitting a 4D Mandelbulb with KS3 vs MLP [R]

Thumbnail
youtube.com
3 Upvotes

For the past year I have been studying using a fast K-dim hermite spline manifold algorithm, called K-Splanifolds or KS, for general purpose fast regression. In this video I show the third version, KS3, compared to MLP and older KS versions fitting a 4D mandelbulb via gradient descent on random samples. KS3 converges faster and has nearly identical latency on a GPU as MLP.

KS blocks are suitable for a wide range of regression problems and I have found KS3 ranges from considerably faster convergence as MLP (noise removal, geometric fit, financial prediction) to neck and neck (LLMs). You can view code for all three versions of the algorithm and a paper for KS1 here: https://github.com/curvedinf/k-splanifolds

I am an independent and if anyone would like to explore this with me, let me know. Also if anyone would like to donate access to 8x H100, I would like to attempt a world record nanogpt speedrun with KS3 using configurations I have found beat the WR on proxy hardware.


r/MachineLearning • • 5h ago

Research Scoring IDP [R]

3 Upvotes

This is a "survey of current practices" type of research for a niche area, nothing earth-shattering. I’ve written a 2-part Medium series (free, no paywall) on scoring Intelligent Document Processing (IDP). It focuses on Key Information Extraction (KIE) from structured documents, especially forms. I lean heavy on metrics derived from the confusion matrix: different ways to map items such as blanks or misreads to the four possible values, how to count them, when to use each metric.

Topics I include, but not limited to: recording raw outcomes before mapping them to the confusion matrix (correct, misread, missed, invented, correct blank); confusion matrix scoring for text fields and checkboxes; misread conventions and their effect on scores (FN-only, FP-only; FN & FP); how blanks, NULLs, unreported checkboxes, and unreadable values are counted; F1 vs. Recall vs. Accuracy, including how correctly identified blanks inflate Accuracy; Jaccard Index (IoU) as Accuracy without True Negatives, and its conversion to F1; Precision, Specificity, and False Positive Rate for LLM hallucination control; micro, macro, and weighted F1; choosing a headline score and stating the convention behind it; confidence vs. correctness and calibration (ECE, MCE); and Key Information Extraction (KIE) vs. OCR metrics (CER, WER).

I believe these would fall under "machine learning scoring techniques" applied to Intelligent Document Processing (IDP)--how to transparently and fairly measure (score) rather than how to implement. It may also help folks understand how to read results coming from different benchmarks.

I try not to recommend one opinionated way of doing things and instead cover the variations I’ve run into and the tradeoffs between them. I’m not endorsing any vendor; the series is about scoring itself.


r/MachineLearning • • 2h ago

Project Deepity: a hand-tuned C++/CUDA predictive coding library, beats jpc (JAX) and ngc-learn on MNIST (CPU benchmarks + source inside) [P]

0 Upvotes

I've been building a predictive coding (PC) engine from scratch in C++20, with a thin Python layer on top, and just cut the first public release. Posting here for feedback and because I haven't seen many from-scratch PC implementations with real kernel-level optimization work behind them.

What predictive coding is: instead of backprop's single global error signal flowing backward through a stored computation graph, each layer in a PCN holds a local belief about its own state and compares it against a top-down prediction from the layer above. Training alternates between "settling" (every layer locally reduces its own prediction error over several steps) and a weight update that only looks at that layer's own error. No layer needs the transpose of another layer's weights, and there's no backward graph to keep in memory.

What Deepity actually is:

- A C++20 core with hand-written AVX2/AVX-512 SIMD kernels (via SLEEF for the transcendental activations), OpenBLAS/MKL for GEMM, and a CUDA backend (cuBLAS + CUTLASS + CUDA graph capture for the settling loop) that's still being tuned for large-scale training (not available yet).

- Python bindings via nanobind. pip install pydeepity, import deepity, define a network the same way you would in PyTorch/Keras.

- Several PCN variants: a plain classic-settling network, Direct Kolen-Pollack feedback (far fewer settling steps, some accuracy cost), and a "kitchen sink" network with µPC scaling, residual connections, incremental PC, error-based PC and momentum settling all available as independent toggles, off by default.

The headline benchmark (matched config across all three PC implementations: 784-512-512-10 sigmoid network, T=20 settling steps, batch 250, 100 epochs, AdamW):

Deepity: 98.21% in 487.5s

jpc (JAX): 97.55% in 531.4s

ngc-learn: 98% in 1707.7s

Testing was done on a single machine (12th Gen i7-12700H laptop CPU), reproducible from the repo, with raw logs in the repository. A plain PyTorch backprop run on the same architecture gets 97.51% in 627.8s included as a reference point (not a fair comparison, since it's one pass per batch where PC does 20 settling passes).

Kernel-level numbers, if you care about where the time actually goes: a Google Benchmark harness measuring a full PC train step sustains ~690 GFLOPS/s, and the same FLOP accounting applied to the real end-to-end MNIST run above works out to ~675 GFLOPS/s.

What's not done yet, to be upfront about it: this is alpha software (0.1.0a3), solo-maintained so far. The CPU path is the mature, benchmarked one. The CUDA backend builds and runs, but large-scale ImageNet training on it is still being optimized, and the convolution path is currently im2col+cuBLAS rather than cuDNN (TF32 is on, but it still trails cuDNN's specialized small-kernel conv algorithms). No CUDA-vs-CPU numbers on purpose, as I don't want to publish a number I haven't actually validated.

Install is deliberately light, just numpy and rich, no bundled ML framework:

pip install pydeepity

from deepity import SimplePCN, Linear, TanH
import numpy as np

net = SimplePCN(Linear(2, 8), TanH(), Linear(8, 1), batch_size=1)
net.configure(learning_rate=0.05, inference_rate=0.3, lmbda=0.0001,
  optimizer="SGD")

x = np.array([1.0,-1.0], dtype=np.float32)
y = np.array([1.0], dtype=np.float32)

energy = net.train_step(x, y, steps=50) # returns total pred. error

Repo (MIT licensed): https://github.com/Deepity-HQ/Deepity

PyPI: https://pypi.org/project/pydeepity

Happy to answer questions about the algorithm, the kernel fusion work, or why PC is or isn't worth your time compared to backprop. Looking for contributors too, especially anyone with C++/CUDA performance or build-systems/DevOps background.


r/MachineLearning • • 13h ago

Project PBT doesn’t make grokking faster, but it can escape a hyperparameter range where nothing groks [P]

0 Upvotes

I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch).

Results

  • vs a grid (equal compute, 3 seeds): the best fixed cell reached 95% val acc in 317 ± 29 steps; PBT took 2667 ± 2244. PBT was slower in every seed, but the grid’s corner already contained the answer.
  • Dead-zone start (lr 1e-4 to 1e-3, wd 0.01 to 0.08, where no fixed setting groks): fixed 0/3, slow PBT 0/3, fast PBT (every 100 steps, ×0.5/×2) 3/3 at about 6850 steps.
  • Its path: lr drops to the floor, then wd climbs to the ceiling, then lr rises. Val loss rewards low lr × wd early on, which is the wrong direction for grokking, then flips sign later.
  • Ablations: a random donor still escapes, just later. With no selection, members grok and then the population loses it.
  • 10 seeds: wide random search hits 95% at a median of 300 steps, vs 6650 for PBT from the dead zone. PBT also sometimes collapses into the lr-floor/wd-ceiling corner and never leaves.
  • Replay: the winning lr/wd schedule with no PBT, on fresh inits: 55/60 reach 95%. Constant knobs at the schedule’s geometric mean: 0/60. The schedule carries the result, not the weight copying.

Takeaway: PBT is a rescue tool for a bad range, not a speed-up.

Caveats: one task, one model, one split. 3 to 10 seeds per test, with p-values mostly above 0.05. The replayed schedules are post-hoc winners. I haven’t tested a range with no grokking region at all, which is where PBT should beat random search.

Write-up with interactive figures: https://shauryasharma.tech/pbt-grokking.html
Code and all results: https://github.com/Shaurya-34/grok-pbt


r/MachineLearning • • 1d ago

News ICLR 2027 is still handing out desk rejects [N]

70 Upvotes

Thank you for your efforts in protecting the community from AI slop.


r/MachineLearning • • 1d ago

News Nanbeige releases Nanbeige4.3-33B, an Apache 2.0 teacher model for smaller agents [N]

7 Upvotes

Nanbeige released Nanbeige4.3-33B as the teacher for its planned smaller agents, including an upcoming 3B model. the weights are Apache 2.0, and the model card lists a 512K context window.

the team reports 80.9 on Terminal-Bench 2.1 versus 73.0 for Qwen3.8-27B, using the same harness and inference settings within each benchmark. it does not win everywhere. on SWE-Bench Pro, the reported scores are 60.3 versus 61.7. these are the releasing team's evaluations, not independent replication.

one practical detail for reproducing the agent results: the evaluations retain earlier reasoning with preserve_thinking=true. the card recommends that setting for multi-turn tool use, but not general chat. comparing task success and total generated tokens with that setting held fixed would be more useful than treating the headline scores as a deployment ranking.

source: https://huggingface.co/Nanbeige/Nanbeige4.3-33B


r/MachineLearning • • 1d ago

Discussion memt-1.0 architecture breakdown gated deltanet-2, mixture-of-storage, and 66x smaller storage footprint for agent memory [D]

Post image
13 Upvotes

been digesting the details behind memt-1.0's memory-native architecture recently, and the engineering tradeoffs between full context kv caching and internalized parametric memory are pretty wild.

instead of using append-only prompt stacking or external rag pipelines, memt-1.0 bakes memory directly into the model computation layer using gated deltanet-2 with a stable token-level memory update mechanism within each turn, combined with a mixture-of-storage (mos) layer to scale capacity.

a few quick technical takeaways from the setup:

internalizing procedural experience: standard agents retrieve past execution logs as raw text, which bloats the context window. memt-1.0 continuously internalizes retrieved procedural experience into model weights/states during multi-turn agentic tasks.

efficiency at 128k context: at a history length of 128k tokens, memt-1.0-nano delivers roughly 1.86x end-to-end inference speedup and a 66x reduction in persistent storage compared to a full-context kv cache setup.

general capability preservation: training parametric memory layers usually degrades base llm capabilities, but their two-stage training pipeline (continued pre-training + post-training across 34b tokens) keeps general reasoning competitive while outperforming existing parametric memory baselines.

for those building long-horizon agents, is internalizing state via architecture-level update layers (like gated deltanet / mos) the endgame to escape kv cache memory bottlenecks?


r/MachineLearning • • 2d ago

Discussion Increasing number of UG authors in A* conferences with no supervisor [D]

93 Upvotes

What do you all think about this? It seems increasingly common for undergraduate students to conduct research and publish at A* conferences without a supervisor. As a graduate student, I find it both impressive and intimidating, and it sometimes makes me question my own progress.


r/MachineLearning • • 1d ago

Discussion What's the hardest part of creating edge cases for fine-tuning open-weight models? [D]

0 Upvotes

I'm interested in how people build datasets for fine-tuning open-weight models, especially when it is about making a model handling edge cases than just perform well on common examples.

I'm curious about the data side of the process.

  • Finding edge cases: How do you identify edge cases that are rare or unusual. Also how do you make edge cases on an existing dataset to make it more specialized for what you need.
  • Generating examples: Do you write them by yourself, or use a good model to generate synthetic data, mine real data, etc?
  • Quality control: How do you catch incorrect, repetitive, or unrealistic examples?
  • Coverage: How do you know when your dataset covers enough of the failure cases that matter?
  • Measuring impact: Have you had cases where adding more edge cases actually made the model worse or failed to improve it?

r/MachineLearning • • 22h ago

Discussion I ran a blind test on self-hosted AI agents. Here are the results, guess who's who [D]

0 Upvotes

everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess

The tasks

  1. Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people).
  2. Rename a function across 3 files with messy spacing. All 5 passed.
  3. Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4".
  4. Count the errors in 3 log files. All 5 got 43.
  5. Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want.
Agent 1. Contacts 2. Rename 3. Bug fix 4. Errors 5. Receipt Total
A ✅ 4:54 ✅ 0:28 ✅ 0:43 ✅ 0:26 ✅ 3:58 10:29
B ✅ 6:03 ✅ 1:11 ✅ 0:55 ✅ 0:54 ✅ 15:04 24:07
C ✅ 13:32 ✅ 0:24 ❌ 0:22 ✅ 0:57 ❌ 0:49 16:04
D ✅ 6:16 ✅ 1:12 ✅ 0:56 ✅ 0:56 ✅ 2:50 12:10
E ✅ 5:50 ✅ 1:01 ✅ 0:21 ✅ 0:26 ✅ 3:56 11:34

Setup

One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw

Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two.


r/MachineLearning • • 3d ago

Research [R] The Hive Mind is a Single Reinforcement Learning Agent: Deriving the Exact Equivalence Between Swarm Dynamics and Multi-Armed Bandits

Thumbnail
arxiv.org
68 Upvotes

TL;DR

We prove a mathematical equivalence between collective decision-making via local imitation (such as honey bees choosing a nest site via the waggle dance) and single-agent Reinforcement Learning (RL) in a multi-armed bandit setting. Even when individual members have no learning capability and merely blindly imitate peers, the population distribution pi updates according to a macroscopic bandit RL algorithm that we call Maynard-Cross Learning (MCL), where N individuals act as N parallel environment rollouts with an effective per-sample step size of alpha = 1/N.

Main intuition

In nature, optimal decision-making emerges through two seemingly distinct paradigms:

  1. Individual trial-and-error (RL): A single agent interacts with its environment and updates its policy based on reward signals.
  2. Collective imitation (Swarm Intelligence / Evolutionary Game Theory): Simple individuals copy neighbors' strategies (e.g., best-of-n collective decision-making).

Biologists have long described insect colonies as "superorganisms" or "hive minds" with parallels to vertebrate brains. We wanted to see if this metaphor holds up mathematically under the lens of RL: Does a swarm of non-learning imitators literally execute a single-agent RL update at the macro level?

Core development

The paper's core theoretical development establishes a mathematical equivalence between decentralized social imitation in large populations and single-agent reinforcement learning (RL) in multi-armed bandits. By treating the proportion of individuals committed to each option as the action-probability policy of a single macro-agent, we use evolutionary replicator dynamics to bridge local individual interactions and global policy updates.

First, we show that a population following the classic "imitation of success" protocol (where individuals copy neighbors with a probability proportional to those neighbors' payoffs) evolves identically to the expected update of the Cross Learning bandit algorithm. Extending this approach to the "weighted voter" model of the honey bee waggle dance (where scouts advertise nesting sites for a duration proportional to site quality and peers blindly copy the first advertisement they encounter without comparing payoffs) we derive a new macro-level RL algorithm that we call Maynard-Cross Learning. Under this duality, each non-learning individual acts as a single parallel environment sample whose fractional influence on the group mirrors an RL learning rate equal to the inverse of the swarm size.

In practice, this development demonstrates how blind local imitation aggregates into a coherent, parallelized reinforcement learner at the collective level.

Impact

By proving a mathematical equivalence between local social imitation and single-agent reinforcement learning, this work provides a formal mechanism explaining how "hive mind" intelligence and trial-and-error learning emerge at the group level from simple, non-learning individuals. Beyond substantiating evolutionary theories of collective cognition in biology, this framework offers new tools for analyzing imitative dynamics in social and economic systems and for designing scalable, decentralized swarm robotics algorithms.

Links

Paper: https://arxiv.org/abs/2410.17517
Project page: https://hivemind.yannbouteiller.com/
Code/reproducibility: https://github.com/MISTLab/HiveMindRL


r/MachineLearning • • 2d ago

Project Got a pretrained music transformer to jam with me live, no fine-tuning, just steering the sampling [P]

2 Upvotes

I wanted to see how far you can get making a pretrained symbolic music model "listen" to a live player without training it. Turns out pretty far.

Setup: I play a MIDI keyboard, the Anticipatory Music Transformer (Thickstun et al., music-medium-800k) plays a second piano. While I play I track my key, the chord I'm holding and my tempo, and add biases to the logits before sampling. Out-of-key notes get pushed down, chord tones and tensions get pushed up, the keys right around my hands get pushed down so it doesn't step on me, and when I stop it gets pulled up to where a melody would be. There's also a small boost for continuing the intervals of my last phrase, which makes it sound like it's answering.

It samples about a second ahead, and when I change chord I throw away what it had planned and resample. Runs on a Jetson Orin Nano.

Video + code: https://github.com/RiccardoCataldi/duetto

The offsets are all hand-tuned by ear, which feels a bit dumb. Has anyone tried something more principled for this, like a small learned conditioning head or guidance-style tricks on a symbolic model?


r/MachineLearning • • 3d ago

Discussion Is symbolic ai dead? [D]

71 Upvotes

New research keeps finding things that keep reintroducing symbolic ai structures to machine learning and I have wondered why more research hasnt been done with it. There was a heyday in the 70s for it but after neural nets came out it seems to be completely out of the discussions. I think it could make a comeback if we can resolve the primary issues of self-instantiation and feedback loops. Even using several random reservoir nets and a little bit of math creates strong signals for the component parts that contribute to what something is made of and how it works.


r/MachineLearning • • 3d ago

Discussion Are we solving LLM memory by remembering more, when the real problem is knowing what to forget?[D]

21 Upvotes

Long-context models can now handle hundreds of thousands, even millions of tokens. But I've been wondering if we're focusing too much on recall and not enough on what happens when information changes.

Say a coding agent reads that an API timeout is 30 seconds. Later, the documentation changes it to 5 seconds. Both versions are still in context, but the agent keeps using 30.

It hasn't forgotten anything. It just hasn't updated what it knows.

I've been reading about different memory architectures, and this distinction seems pretty important.

With Transformers and RAG, old and new information can coexist, and the model has to resolve the conflict when reading it. In simple linear attention, associations accumulate in a shared state, so writing something new doesn't necessarily remove what's already there. DeltaNet tries to address this through prediction-error-based updates. BDH (Dragon Hatchling) takes another route with Hebbian-style synaptic plasticity.

But even with these approaches, there's still the question of when to update something, when to keep it, and how to avoid messing up unrelated memories.

I also wonder if we're testing this properly. Most long-context evaluations focus on whether a model can find information. What if we instead tested whether it changes its answer when given newer, contradictory evidence, while still remembering what was previously true?

Curious what people here think. Is this mostly a problem with how memory is stored and updated, or with how models reason over conflicting information?

Also, are there any good benchmarks specifically targeting this? Would be interested in reading them.


r/MachineLearning • • 3d ago

Research AAAI 2027 phase 2 reviews discussion [D]

45 Upvotes

AAAI phase 2 reviews are expected to be visible in the next 24 hours (24/10). Hope for the best outcomes for those who stay in this phase, and let's share your reviews!


r/MachineLearning • • 3d ago

Research I just came across Next-Concept Modeling and the idea is fascinating[R]

83 Upvotes

I've been learning how language models work, especially next-token prediction, and came across this paper on Next-Concept Modeling (NCM).

The interesting part is how it generates text.

Instead of predicting tokens one by one, NCM learns a hierarchy of discrete codes. It generates the coarse level first, then predicts the codes at each finer level in parallel.

A decoder finally converts those representations into text.

The researchers report up to 31x faster generation compared with diffusion language model baselines on specific benchmarks.

I'm still learning about the architecture, especially the VQ-VAE part.

Curious if anyone here has explored this approach.

Paper: https://arxiv.org/abs/2610.14162


r/MachineLearning • • 3d ago

Project Study Notes on Transformers and Hamiltonian Monte Carlo [P]

23 Upvotes

I’m sharing my study notes in case they might be useful to someone else.

The Transformer notes are mostly based on the original architecture described in Attention Is All You Need.

https://doi.org/10.5281/zenodo.22031925

I’ve also written notes on Hamiltonian Monte Carlo (HMC).

https://doi.org/10.5281/zenodo.21841086

I’d appreciate any feedback on the notes, including corrections, suggestions for improvement, or topics you think I should add.


r/MachineLearning • • 3d ago

Discussion How does arXiv prepublishing work with ICLR's double-blind review? [R]

26 Upvotes

Hey folks,

maybe a really dumb question from an inexperienced PhD student. I recently submitted to ICLR and spotted a bunch of papers that I was already familiar with because I had seen them on arXiv before.

Doesn't this break the double-blind review process? I mean, on the one hand, people pre-publishing their work want to establish priority as quickly as possible, but on the other hand, they still want their papers to be treated as valid submissions in a double-blind review system.

I generally avoid pre-publishing my work for this reason, although this can cause some trouble when it comes to novelty claims, because I'm worried about getting desk-rejected.

I mean, I can just Google the paper title or even parts of the abstract and immediately find the preprint. It seems quite frustrating...


r/MachineLearning • • 3d ago

Project Stenosis detection in coronary angiograms, need recall over precision [D]

3 Upvotes

Starting a project to detect candidate stenoses in x-ray coronary angiograms. The goal is high recall, false positives are fine as long as it doesn't flag half the image.

I already tried YOLO on a couple of in-house datasets earlier. It looked fine in-domain and fell apart on the other dataset, and a big chunk of that turned out to be annotation format. One set had fixed-size boxes around a point, the other had boxes fitted to the lesion. Same disease, basically two different tasks.

I also have vessel segmentation and centerlines, so my first instinct is a radius profile along the vessel to flag narrowing, then maybe a network to filter the candidates. Has anyone actually done something like that, or is there a smarter way in?

The part that bugs me most is evaluation. The labels come from doctors and they obviously don't mark every narrowing, so any recall I compute against them is probably too optimistic. How do you get an honest number when the ground truth is incomplete?

Also open to papers, repos, or public datasets beyond ARCADE. And if you think I'm approaching this wrong, say so.


r/MachineLearning • • 4d ago

Research Carbon-X is an open 14.8B model trained jointly on DNA and text [R]

47 Upvotes

Carbon-X is a 14.8B parameter decoder-only model that takes raw DNA in a normal chat and answers in English. It shares one token stream for text and DNA, using 6-mer tokens, and was mid-trained on 302B tokens.

Hugging Face reports a 66.7% average across five DNA benchmarks with thinking disabled, versus 67.5% for GPT-6.1 Sol. The weights fit on one 80 GB GPU in bfloat16 and are Apache 2.0. It has not been clinically validated, and the model card says exact sequence operations should still be handled by code.

the useful part is making sequence analysis local enough for sensitive datasets. has anyone tested whether the benchmark gains hold on private variant datasets?

source: https://huggingface.co/HuggingFaceBio/Carbon-X


r/MachineLearning • • 4d ago

Project We shrank our steerable small TTS to 11.84 MB. WER is unchanged, UTMOS drops ~0.1 [p]

5 Upvotes

A while back I posted SteerTTS v1, a small English TTS you steer with numbers (https://huggingface.co/WhissleAI/steer-tts). Today I am releasing the edge variant:

https://huggingface.co/WhissleAI/steer-tts-edge

Size: everything an app has to ship (weights, voices, text front end and Python runtime) totals 11.84 MB.

How:

  • Acoustic model (9.25M params, attention + bidirectional Mamba-3) is quantization-aware trained to int4
  • 6.2M-param Vocos-style vocoder is distilled from charactr/vocos-mel-24khz, also int4 QAT
  • Text front end is pure numpy, with no NLTK, g2p_en, torchaudio or num2words at runtime

Quality (Open TTS Leaderboard protocol, run locally, not an official submission):

Seed-TTS WER / UTMOS CV3-Eval WER / UTMOS
v1 fp32 (no refiner) 0.96% / 4.20
edge 0.96% / 4.11

Intelligibility is essentially unchanged. UTMOS drops about 0.1, almost all of it from the small int4 vocoder.

Steering survives quantization. Every one of the 12 controls (rate, loudness, pitch, pitch range, final rise, pauses, tilt, breathiness, and more) keeps at least 95% of its fp32 effect. You also get 50 style tags and inline markup:

Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow> <whisper>Don't tell anyone.</whisper>

Speed (CPU only, a 2018 Threadripper under load): RTF 0.053 with 4 threads (~19x realtime), ~120 ms to first audio, with sentence-level streaming.

Limitations:

  • English only, and 200 voices (a subset of the full model’s 2,825)
  • No voice cloning, free-text prompts or flow refiner on device. Those stay in the full model.
  • Weights are dequantized to fp32 at load, so the win is size, not speed. Unless we write new kernels for edge devices.
  • Emotion tags are mostly style switches, and fast/anger are weak

I need feedback, especially on the vocoder quality gap and any edge targets you’d like to see (ONNX, mobile, WASM).

Some examples here:

https://whissleai.github.io/steer-tts/


r/MachineLearning • • 4d ago

Project I built Open-XTTS-Trainer: Now you can add new language scripts and train all 3 XTTS v2 components in 1 click [P]

3 Upvotes

TL;DR: When Coqui AI shut down, they left XTTS v2 without official training code for the DVAE audio tokenizer and Latent HiFi-GAN vocoder, and fine-tuning new scripts (Arabic, Indic, etc.) broke on fixed vocabularies. I built Open-XTTS-Trainer — an open-source, modular 3-component training suite under MPL-2.0 with verified GPU loss convergence on real human speech and a 1-click Colab demo.

Why I Built This

XTTS v2 is one of the best local zero-shot TTS models we have, but training/adapting it has been painful since Coqui closed:

  1. Missing Component Recipes: Only GPT fine-tuning was commonly available. Stage 1 (DVAE) and Stage 3 (HiFi-GAN vocoder) training pipelines were either unreleased or fragmented.
  2. The Tokenizer Wall: XTTS was hardcoded for 17 languages. Feeding text with new Unicode scripts (Hindi, Arabic, Thai, Dhivehi, etc.) caused token out-of-vocab or matrix dimension crashes.
  3. Dependency Rot: Upgrading to modern PyTorch (2.4+) broke on missing torchcodec and licensing prompts.

What’s in the Suite

  • Stage 1 (Audio Tokenizer — DVAE): Full training & fine-tuning loop with Gumbel-Softmax codebook annealing and mel-spectrogram reconstruction loss.
  • Stage 2 (Text & Vocab Extender — GPT): Scans a new language corpus, discovers unique Unicode glyphs, appends them to the tokenizer, and dynamically resizes the GPT embedding matrix without corrupting existing weights. Includes the autoregressive GPT training loop.
  • Stage 3 (Neural Vocoder — Latent HiFi-GAN): Latent-conditioned vocoder training with Multi-Period (MPD) and Multi-Scale (MSD) discriminators and feature-matching loss.
  • Dependency-Rot Free: Runs out-of-the-box on modern PyTorch (Python 3.10–3.13) without torchcodec blocking.

Verification & GPU Benchmarks

Tested on an NVIDIA T4 GPU using real human speech (LJSpeech + JFK address):

  • HiFi-GAN Mel Reconstruction Loss: Plunged 67% ($190.83 \rightarrow 63.49$) over 10 epochs.
  • Discriminator Equilibrium: MPD + MSD stabilized cleanly at 3.62 (no mode collapse).
  • DVAE Mel Error: Dropped 67% ($8.14 \rightarrow 2.71$).

(Note: These benchmarks verify component-level training convergence and gradient stability. Full end-to-end text-to-speech inference connects the trained modules into the standard XTTS generation pipeline).

Links & Code

Feedback, PRs, and experiments with new languages are very welcome!


r/MachineLearning • • 3d ago

Project I made Transformer Architecture from scratch [P]

Thumbnail
github.com
0 Upvotes

I implemented the 2017 transformer architecture using pytorch, the model right now doesn't predict very good due to the fact that I am using on 2 layers of encoder and decoder with just 1000 sentences to train it , due to the fact that it takes a lot of time to train my implementation. Looking at the code is there a way that I can reduce the time it takes for training or maybe improve its results?