I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch).

Results

• vs a grid (equal compute, 3 seeds): the best fixed cell reached 95% val acc in 317 ± 29 steps; PBT took 2667 ± 2244. PBT was slower in every seed, but the grid’s corner already contained the answer.
• Dead-zone start (lr 1e-4 to 1e-3, wd 0.01 to 0.08, where no fixed setting groks): fixed 0/3, slow PBT 0/3, fast PBT (every 100 steps, ×0.5/×2) 3/3 at about 6850 steps.
• Its path: lr drops to the floor, then wd climbs to the ceiling, then lr rises. Val loss rewards low lr × wd early on, which is the wrong direction for grokking, then flips sign later.
• Ablations: a random donor still escapes, just later. With no selection, members grok and then the population loses it.
• 10 seeds: wide random search hits 95% at a median of 300 steps, vs 6650 for PBT from the dead zone. PBT also sometimes collapses into the lr-floor/wd-ceiling corner and never leaves.
• Replay: the winning lr/wd schedule with no PBT, on fresh inits: 55/60 reach 95%. Constant knobs at the schedule’s geometric mean: 0/60. The schedule carries the result, not the weight copying.

Takeaway: PBT is a rescue tool for a bad range, not a speed-up.

Caveats: one task, one model, one split. 3 to 10 seeds per test, with p-values mostly above 0.05. The replayed schedules are post-hoc winners. I haven’t tested a range with no grokking region at all, which is where PBT should beat random search.

Write-up with interactive figures: https://shauryasharma.tech/pbt-grokking.html
Code and all results: https://github.com/Shaurya-34/grok-pbt

I've been building a predictive coding (PC) engine from scratch in C++20, with a thin Python layer on top, and just cut the first public release. Posting here for feedback and because I haven't seen many from-scratch PC implementations with real kernel-level optimization work behind them.

What predictive coding is: instead of backprop's single global error signal flowing backward through a stored computation graph, each layer in a PCN holds a local belief about its own state and compares it against a top-down prediction from the layer above. Training alternates between "settling" (every layer locally reduces its own prediction error over several steps) and a weight update that only looks at that layer's own error. No layer needs the transpose of another layer's weights, and there's no backward graph to keep in memory.

What Deepity actually is:

- A C++20 core with hand-written AVX2/AVX-512 SIMD kernels (via SLEEF for the transcendental activations), OpenBLAS/MKL for GEMM, and a CUDA backend (cuBLAS + CUTLASS + CUDA graph capture for the settling loop) that's still being tuned for large-scale training (not available yet).

- Python bindings via nanobind. pip install pydeepity, import deepity, define a network the same way you would in PyTorch/Keras.

- Several PCN variants: a plain classic-settling network, Direct Kolen-Pollack feedback (far fewer settling steps, some accuracy cost), and a "kitchen sink" network with µPC scaling, residual connections, incremental PC, error-based PC and momentum settling all available as independent toggles, off by default.

The headline benchmark (matched config across all three PC implementations: 784-512-512-10 sigmoid network, T=20 settling steps, batch 250, 100 epochs, AdamW):

Deepity: 98.21% in 487.5s

jpc (JAX): 97.55% in 531.4s

ngc-learn: 98% in 1707.7s

Testing was done on a single machine (12th Gen i7-12700H laptop CPU), reproducible from the repo, with raw logs in the repository. A plain PyTorch backprop run on the same architecture gets 97.51% in 627.8s included as a reference point (not a fair comparison, since it's one pass per batch where PC does 20 settling passes).

Kernel-level numbers, if you care about where the time actually goes: a Google Benchmark harness measuring a full PC train step sustains ~690 GFLOPS/s, and the same FLOP accounting applied to the real end-to-end MNIST run above works out to ~675 GFLOPS/s.

What's not done yet, to be upfront about it: this is alpha software (0.1.0a3), solo-maintained so far. The CPU path is the mature, benchmarked one. The CUDA backend builds and runs, but large-scale ImageNet training on it is still being optimized, and the convolution path is currently im2col+cuBLAS rather than cuDNN (TF32 is on, but it still trails cuDNN's specialized small-kernel conv algorithms). No CUDA-vs-CPU numbers on purpose, as I don't want to publish a number I haven't actually validated.

Install is deliberately light, just numpy and rich, no bundled ML framework:

pip install pydeepity

from deepity import SimplePCN, Linear, TanH
import numpy as np

net = SimplePCN(Linear(2, 8), TanH(), Linear(8, 1), batch_size=1)
net.configure(learning_rate=0.05, inference_rate=0.3, lmbda=0.0001,
  optimizer="SGD")

x = np.array([1.0,-1.0], dtype=np.float32)
y = np.array([1.0], dtype=np.float32)

energy = net.train_step(x, y, steps=50) # returns total pred. error

Repo (MIT licensed): https://github.com/Deepity-HQ/Deepity

PyPI: https://pypi.org/project/pydeepity

Happy to answer questions about the algorithm, the kernel fusion work, or why PC is or isn't worth your time compared to backprop. Looking for contributors too, especially anyone with C++/CUDA performance or build-systems/DevOps background.

I am a 3rd year PhD student in a small lab working on ML in healthcare. My PI is new (we have no PhD graduates yet - lab is 4 years old).

I want to publish in a top ML conference (I know its necessary for an ML researcher job).

And my PI is saying you need to do method research and I don't know how to do it?

We don't have the amount of data to do foundational models. Our data is very small.

I feel like my PI had me jumping around topics he was excited about for the 1st two years of my PhD and nothing paned out or I had no idea what I was doing.

I started thinking of a certain idea but I needed guidance on how to implement it... My PI was no help and tried pushing me to a new idea. He is usually very "attacky" to my ideas, I always get the feeling I need to prepare to really really defend any idea I have to him.

It's either too incremental, too applied, or not impactfull enough. I tried asking what exactly is a method research? His example was the transformers paper, and his own paper (which was incremental according to his current criteria).

I now have no idea what I am doing, where I am going or what I am going to do...

I still have zero idea what my PhD research question is...

Do you guys have any advice?

* this is my 1st time posting so please be gentle 😅

For the past year I have been studying using a fast K-dim hermite spline manifold algorithm, called K-Splanifolds or KS, for general purpose fast regression. In this video I show the third version, KS3, compared to MLP and older KS versions fitting a 4D mandelbulb via gradient descent on random samples. KS3 converges faster and has nearly identical latency on a GPU as MLP.

KS blocks are suitable for a wide range of regression problems and I have found KS3 ranges from considerably faster convergence as MLP (noise removal, geometric fit, financial prediction) to neck and neck (LLMs). You can view code for all three versions of the algorithm and a paper for KS1 here: https://github.com/curvedinf/k-splanifolds

I am an independent and if anyone would like to explore this with me, let me know. Also if anyone would like to donate access to 8x H100, I would like to attempt a world record nanogpt speedrun with KS3 using configurations I have found beat the WR on proxy hardware.

This is a "survey of current practices" type of research for a niche area, nothing earth-shattering. I’ve written a 2-part Medium series (free, no paywall) on scoring Intelligent Document Processing (IDP). It focuses on Key Information Extraction (KIE) from structured documents, especially forms. I lean heavy on metrics derived from the confusion matrix: different ways to map items such as blanks or misreads to the four possible values, how to count them, when to use each metric.

• Scoring IDP, Part 1: Confidence Is Not Correctness
• Scoring IDP, Part 2: Approaches to Evaluating Form Extraction

Topics I include, but not limited to: recording raw outcomes before mapping them to the confusion matrix (correct, misread, missed, invented, correct blank); confusion matrix scoring for text fields and checkboxes; misread conventions and their effect on scores (FN-only, FP-only; FN & FP); how blanks, NULLs, unreported checkboxes, and unreadable values are counted; F1 vs. Recall vs. Accuracy, including how correctly identified blanks inflate Accuracy; Jaccard Index (IoU) as Accuracy without True Negatives, and its conversion to F1; Precision, Specificity, and False Positive Rate for LLM hallucination control; micro, macro, and weighted F1; choosing a headline score and stating the convention behind it; confidence vs. correctness and calibration (ECE, MCE); and Key Information Extraction (KIE) vs. OCR metrics (CER, WER).

I believe these would fall under "machine learning scoring techniques" applied to Intelligent Document Processing (IDP)--how to transparently and fairly measure (score) rather than how to implement. It may also help folks understand how to read results coming from different benchmarks.

I try not to recommend one opinionated way of doing things and instead cover the variations I’ve run into and the tradeoffs between them. I’m not endorsing any vendor; the series is about scoring itself.