The company launched Inflection AI Labs, a public-facing research and experimentation arm, alongside Pi Journeys, the lab's first product experiment — an AI experience designed to adapt to a user's life stage, whether that's becoming a parent, taking on caregiving duties, changing careers, or aging. The announcement arrived with a research report on consumer AI habits and a substantial update to Pi, the company's flagship chatbot, adding improved voice, memory, and new agentic tools for reminders, to-do lists, and shopping.

"Inflection AI is the company. Pi is our flagship consumer product. Inflection AI Labs is where we experiment, explore personal intelligence and share more publicly. Pi Journeys is the first public experiment from Inflection AI Labs," CEO Sean White told VentureBeat in an exclusive interview.

Behind the tidy org chart is a far more interesting story: a company attempting one of the more unusual second acts in the AI industry, powered by an argument that the entire market is optimizing for the wrong thing.

Why Inflection AI believes the chatbot era's biggest flaw is that it's transactional

White's central claim is that today's AI assistants — including the industry's most capable models — are fundamentally transactional. You ask, they answer, the session ends. He believes that architecture misses most of what people actually need from artificial intelligence in their daily lives.

"One of the things that really struck us in particular, and this showed up in the research, was that a lot of the work is very transactional, and you'll hear me say a lot that we've been shifting all this from transactional to relational systems," White said. "Not everything is going to be: I do a single turn, I utter a question, I get a search response back."

White frames the industry's evolution as a progression through four kinds of intelligence. First came raw IQ — the foundation model race. Then emotional intelligence, which Inflection made its signature with Pi's famously warm conversational style. Then agentic intelligence — AI that acts rather than just talks — which White says Inflection absorbed from its enterprise work. The fourth, and the one Inflection is now staking its future on, is what the company calls relational intelligence: AI that understands not just you, but the web of people around you.

"There's so much fear about these things pushing people into loneliness,” White said. “If we design these pro-social systems as another design criteria, that actually makes a huge difference."

That design philosophy is a pointed counter-narrative to one of the loudest anxieties in consumer AI right now: that emotionally engaging chatbots deepen isolation by substituting for human contact. Inflection argues the opposite is possible — that an AI with structured knowledge of your relationships can push you back toward people rather than away from them.

Inside Pi Journeys, the AI companion that maps your relationships and life stages

Pi Journeys makes that idea concrete. When users first open the product, it asks about their life stage — caregiver, household manager, midlife transition — and then builds what White describes as specially structured memory around the people who matter in that context. From there, the system becomes proactive.

"It starts to build up memories around that, and it acts as a memory prosthetic — but in a pro-social way," White said. "It doesn't get in the way of your interactions with other people; it really helps facilitate them." The system might remind a user, for example, that a friend deserves a call, or resurface what was last discussed with a family member involved in a parent's care.

White, who spent years as chief R&D officer at Mozilla before taking Inflection's helm, was quick to flag the obvious privacy implications of an AI that maps your social graph. "We've built a lot of privacy systems into this," he said, noting users can delete and manage the people recorded in their profile. Whether consumers will trust a venture-backed AI company with a structured database of their most important relationships remains one of the biggest open questions hanging over the product — and one that enterprise buyers evaluating Inflection's technology will watch closely.

Asked why this was the first Labs experiment, White was direct: "Pi Journeys takes into account people's life stages and experiences because we have heard from users that we can provide more value in helping them navigate their lives. Pi Journeys lets us experiment with the early stages of prosocial and relational intelligence because life isn't single-player."

The product has been tested internally and with small closed groups, White said, and is now being released more broadly as an experiment rather than a finished product — a posture the Labs branding is designed to make explicit.

What Inflection's consumer AI research reveals about how people actually use chatbots

Inflection Labs' first publication, the State of Consumer AI Research Report, offers the empirical scaffolding for the strategy. The average consumer now uses roughly two different AI tools every day and three per week, the company found — evidence, in Inflection's reading, that no single assistant has locked up consumer loyalty and that the market remains contestable.

More telling is why people choose the tools they do. Respondents cited personalization, style and tone, context awareness, and — notably — emotional understanding as deciding factors. They also said they want AI to be more than a productivity engine: a coach or mentor to motivate them, a chef to suggest recipes, a DJ to curate playlists.

"One thing we're certainly finding is that a lot of that also is in work, not so much in everyday life," White said. "That's our focus right now — the everyday life part."

This is a shrewd reading of the competitive map. The best-funded AI labs are pouring resources into coding tools, enterprise agents, and developer platforms, leaving everyday consumer use cases comparatively underserved. White sees the gap clearly. "We see a lot of products that are being aimed more and more at the enterprise," he said. "As a computer scientist by training, I kind of love the IDEs as this tool, but it's not really great for everybody. There's so much regular everyday use from folks that is either purely voice or that is purely mobile."

He recalled a conversation with a conference staffer who told him she owned only a phone, no laptop — exactly the kind of user, he argued, that the industry's developer-centric product roadmaps have left behind.

How the $650 million Microsoft deal hollowed out Inflection — and set up its second act

To understand why any of this is remarkable, you have to rewind to March 2024. Inflection was then one of the hottest startups in AI, having raised $1.3 billion in mid-2023 in a round backed by Microsoft, Nvidia, Bill Gates, and Reid Hoffman — more than $1.5 billion in total. Pi had crossed one million daily active users, per Reuters.

Then, in a deal that reshaped how the industry thinks about acqui-hires, Microsoft hired away co-founder and CEO Mustafa Suleyman, chief scientist Karén Simonyan, and most of the company's roughly 70 employees, paying Inflection about $650 million largely to license its technology, as Reuters reported. Suleyman now runs Microsoft's consumer AI business. The structure of the deal drew scrutiny from the FTC and Britain's competition regulator, though the UK's Competition and Markets Authority cleared it in September 2024 and EU regulators declined to act.

White, installed as CEO in the aftermath, steered the remnant company hard toward enterprise, acquiring three startups in late 2024 — Jelled.AI, BoostKPI, and the European consulting firm Boundaryless — and telling TechCrunch that November that Inflection had no intention of competing with companies building 100,000-GPU frontier systems.

Tuesday's announcement doesn't reverse that position so much as complicate it. Asked how to think about the company today, White called it "a consumer-first strategy that bridges both consumer and enterprise efforts" — and he insists the two sides feed each other.

Enterprise deployments, including a partnership with Intel that is among the few he can name publicly, taught Inflection how to run models inside complex infrastructure. Consumer products, meanwhile, let the company iterate at speed. "The part I also like about the consumer side, and this has always been true, is that we can move faster, experiment faster, and try and learn faster," White said.

The six-month prediction: relationship-aware AI is coming to the enterprise

Buried in White's consumer pitch is the claim that should matter most to technical decision-makers. "Normally I'd say like a year, but let's call it six months," he said. "You're going to start to see a bunch of enterprises care a lot more about the relationships that are inside the enterprises and what that picture is, not just the workflows."

If White is right, the wave of workflow-automation agents currently flooding the enterprise market is only the first phase of business AI adoption — with relationship-aware systems, tested first on consumers, following close behind. Inflection is essentially using its consumer products as a live laboratory for capabilities it plans to sell into companies. It's a capital-efficient strategy for a firm that can no longer outspend rivals on training runs, and a risky one, since it depends on consumers showing up in numbers large enough to generate the learning.

The technical substance underneath is equally pragmatic. Pi today runs not on a single proprietary frontier model but on an orchestration layer routing across many models — some descended from Inflection's original fully trained cores, some fine-tuned, some open source, including work with Nvidia that White says gives Inflection access to unreleased cutting-edge models. He also took a swipe at the industry's loose vocabulary around ownership: "When people say that the model is their own, most of the time nowadays — I guess I won't name names — a lot of companies will actually take a checkpoint, and then they will fine-tune from that checkpoint. But very few people actually start from that beginning core."

That candor extends to open source, where White carefully hedged. "We're not ready to promise what I think of as true open source, and by that I mean everything," he said, invoking his Mozilla years overseeing genuinely open projects like Rust and WebAssembly.

Weights without training data and pipelines, he argued, often leave developers unable to do anything meaningful with a supposedly "open" model. "We are a PBC, and there's still a C in there," he added — a reminder that public benefit corporations still have businesses to protect. The Labs will collaborate with academic researchers, including Stanford professors who visited the company's Palo Alto office this week, and continue contributing to open projects such as PyTorch.

Can a diminished Inflection compete with AI giants spending billions?

Reid Hoffman, the LinkedIn co-founder who co-founded Inflection and stayed on through the Microsoft upheaval, framed the announcement in the sweeping terms of his recent writing on AI and human agency. "Humans should be amplified by AI, not replaced. That's the principle Pi was built on," Hoffman said in the announcement. "When that kind of agency is available to everyone, you get superagency."

The skeptic's case is easy to make. Inflection is a fraction of its former size, competing for consumer attention against products from companies spending tens of billions of dollars a year. Pi's model was state of the art in 2023; it is not in 2026. And "relational intelligence" is, for now, a brand claim awaiting proof.

But the bull case is not crazy either. Inflection's own research shows consumers already juggle multiple AI tools and choose them for qualities — tone, emotional understanding, personalization — that frontier labs treat as afterthoughts. The company kept its technology, its Microsoft licensing windfall, and a defensible enterprise niche in on-premise, emotionally intelligent deployments. And it is targeting the one consumer segment — everyday, mobile-first, voice-first life management — that the coding-obsessed giants have largely ignored.

Asked what success looks like twelve months from now, White declined to talk numbers. "It's less about scale for scale's sake and more about scaling for impact by empowering people and improving their lives," he said. "Over the next year, success means leading the market towards relational intelligence and transforming AI interactions from transactional to relational."

Two years ago, Microsoft walked away with Inflection's founders, its staff, and its shot at the frontier — but it left behind the one idea the giants still haven't figured out how to build: an AI that knows the people in your life matter more than the tasks on your list. Inflection is betting the company, again, that the idea was the valuable part all along.

The post Unsloth vs Axolotl vs TRL vs LLaMA-Factory: A Fine-Tuning Framework Comparison on Speed, VRAM, and Multi-GPU appeared first on MarkTechPost.

Written by: Jules Czarniak

Today, Thinking Machines—the highly capitalized American AI startup founded by former OpenAI CTO Mira Murati—released Inkling, its first major language model under an enterprise-friendly Apache 2.0 open source license, and it boasts high, if sub state-of-the-art, performance for open weights models on third-party benchmarks, specifically software engineering (77.6% on SWE-bench Verified, where it beats fellow U.S. open rival Nvidia Nemotron 3's 71.9%) and voice understanding (91.4% on VoiceBench compared to 94.4% for Gemini 3.1 Pro on high reasoning effort).

Another differentiator: Thinking Machines notes that Inkling was designed "to answer directly on topics that may be subject to censorship," offering enterprises concerned about factual outputs, irrespective of controversy or sensitivity, a more trustworthy option.

Coming in at 975 billion total parameters, Inkling is a natively multimodal, open-weights Mixture-of-Experts (MoE) system capable of reasoning across text, images, and audio. The weights are already available on Hugging Face and the company's own model training application programming interface (API), Tinker.

Designed to balance cost against performance through a novel "controllable thinking effort" mechanism, the model represents a significant departure from the black-box scaling strategies of frontier competitors.

Alongside the flagship model, Thinking Machines also announced a preview of Inkling-Small, a lighter 276-billion-parameter alternative optimized for workloads where low latency and cost are paramount.

Benchmarks Show a Powerful, High-End, Sub State-of-the-Art Model

While Inkling is a formidable multimodal engine, it lands in a fiercely competitive 2026 open-weight landscape characterized by highly specialized MoE architectures. Rather than attempting to dominate every leaderboard, Thinking Machines explicitly designed Inkling—with 975 billion total and 41 billion active parameters—as a broad, balanced generalist.

For example, it comes in near the middle high-end of benchmark performance 1257 on Design Arena’s Agentic Web Dev leaderboard measuring human scores of frontend web design.

But China’s leading AI labs have produced models with elite reasoning and coding capabilities, posing a stiff challenge to Inkling's generalist approach and ultimately outperforming it on general and coding benchmarks.

• GLM 5.2: Widely considered the top open-weight reasoning model available in the benchmark set, GLM 5.2 outperforms Inkling on pure coding, agentic, and complex reasoning tasks. It scores 62.1% on SWEBench Pro (Public) compared to Inkling’s 54.3%, and a massive 82.7 on Terminal Bench 2.1 against Inkling’s 63.8. GLM 5.2 also holds the edge in text-only reasoning, scoring 40.1% on HLE (text only) versus Inkling's 30.0%.

• DeepSeek V4 Pro: DeepSeek maintains an edge in several strict coding and factuality domains, beating Inkling on SWEBench Verified (80.6% vs. 77.6%) and SimpleQA Verified (57.0% vs. 43.9%). However, Inkling successfully overtakes DeepSeek V4 Pro in mathematical problem-solving, achieving 97.1% on AIME 2026 compared to DeepSeek's 96.7%.

• Kimi K2.6: This model outpaces Inkling across multiple technical benchmarks, delivering higher scores on GPQA Diamond (91.1% vs. 87.9%), BrowseComp (83.2% vs. 77.1%), and HLE with tools (54.0% vs. 46.0%). Yet Inkling proves more resilient on general chat instruction following, scoring 79.8% on IFBench compared to Kimi K2.6's 76.0%.

Against its primary U.S.-based open-weight competition, Inkling demonstrates strong parity and frequent superiority.

• Nemotron 3 Ultra: Inkling consistently outperforms this U.S. rival across reasoning and coding. Inkling posts 97.1% on AIME 2026 and 77.6% on SWEBench Verified, beating Nemotron's 94.2% and 70.7%, respectively. Furthermore, Inkling significantly leads in agentic workflows, scoring 74.1% on MCP Atlas against Nemotron's 44.7%.

When compared to closed-source juggernauts like Claude Fable 5, GPT 5.6 Sol, and Gemini 3.1 Pro, Inkling trails in peak reasoning and software engineering autonomy, but remains highly competitive in multimodality.

• Coding and Reasoning: Closed models maintain a commanding lead. Claude Fable 5 (max) hits 95.0% on SWEBench Verified and 53.3% on HLE (text only), far outpacing Inkling's 77.6% and 30.0%. GPT 5.6 Sol dominates Terminal Bench 2.1 with an 89.5, easily clearing Inkling's 63.8.

• Native Multimodality: Inkling's native visual and audio capabilities hold their own. On the MMMU Pro (Standard 10) vision benchmark, Inkling's 73.3% is competitive, though trailing Claude Fable 5's 84.2% and GPT 5.6 Sol's 83.0%. In audio processing, Inkling scores a highly respectable 77.2% on MMAU, keeping it within striking distance of Gemini 3.1 Pro's 82.5%.

If an enterprise workflow demands elite software engineering autonomy or the highest bounds of text-only reasoning, models like GLM 5.2 or proprietary systems like Claude Fable 5 maintain the edge.

However, Inkling carves out a unique and highly defensible position: it is the most capable open-weight foundation model that natively fuses text, vision, and audio, while simultaneously offering developers direct programmatic control over the cost-to-performance ratio.

The Shift from Static Reasoning to Controllable Thinking

Rather than attempting to build a singular "god model" optimized strictly for state-of-the-art benchmark domination, Thinking Machines engineered Inkling for adaptability and efficiency in real-world workflows.

The standout feature of this release is Inkling's "controllable thinking effort." Developers can programmatically adjust the model's reasoning budget—scaling from 0.2 to 0.99—to dictate how hard the AI should "think" before generating an output.

As the company noted, "Inkling's continuous thinking effort lets you pick your point on the cost/performance curve—reaching the same score with a fraction of the tokens".

In practical terms, this allows enterprises to deploy Inkling with lower token expenditure for simpler tasks, while cranking up the compute overhead for complex, multi-step reasoning challenges. However, by keeping the thinking effort lower and generating fewer tokens, the cost-conscious enterprise can achieve high quality results and performance on simple tasks while spending less money, or, in the case of those running models locally, less costs on energy and compute resources.

During the model’s large-scale reinforcement learning (RL) training over 30 million rollouts, researchers observed an emergent phenomenon they called "chain of thought condensation". Over time, Inkling naturally learned to compress its internal reasoning steps—dropping grammatical overhead and connectives—while reaching the same accurate conclusions, resulting in drastically reduced latency.

Epistemics and Censorship Resistance

A notable element of Thinking Machines' release is its explicit focus on the model's epistemics—specifically its calibration, instruction following, and resistance to censorship.

In an ecosystem where open-weight models adopt either overly restrictive safety guardrails or echo state-aligned ideological talking points, Inkling was intentionally trained to answer directly on politically sensitive or heavily censored topics.

To validate this approach, Thinking Machines submitted Inkling to the Propaganda and Censorship Eval developed by AI startup Cognition. According to the published findings, Inkling demonstrated "strong patterns of censorship non-compliance," effectively resisting ideological capture or boilerplate refusals when presented with sensitive subjects.

Despite its resistance to censorship, the model maintains a robust defense against genuinely malicious, dangerous, or illegal queries. On the StrongREJECT benchmark—which tests responses to unambiguous harmful requests—Inkling scored 98.6%, placing it in line with strict frontier safety standards. Furthermore, on the FORTRESS benchmark, Inkling successfully navigated the line between safety and over-refusal: it achieved a 78.0% refusal rate on adversarial queries (such as those involving weapons, cyberattacks, or violence) while maintaining a 95.9% compliance rate on benign, look-alike queries.

Thinking Machines noted that typical open-weight vulnerabilities remain within the architecture. Internal safety evaluations revealed an "occasional tendency to comply with role-play and indirectly framed prompts concerning harmful topics". The company advised enterprise developers to treat the model's built-in refusals as just one layer of security, recommending the downstream deployment of external moderation tools—such as Llama Guard—to filter adversarial jailbreaks and enforce use-case-specific safety policies at the application level.

Under the Hood: Architecture and Multimodality

Inkling's scale is staggering, yet sparse. The MoE architecture features 975 billion total parameters, but only 41 billion parameters are active during any given token generation. It supports a massive context window of 1 million tokens and diverges from typical transformer models by using relative positional embeddings instead of the industry-standard Rotary Positional Embedding (RoPE).

True to the company's foundational vision, Inkling was trained from scratch to be natively multimodal. Unlike models that rely on bolted-on external encoders, Inkling uses an encoder-free early fusion approach. It directly ingests audio as discrete dMel spectrograms and visual data as 40x40 pixel patches via a hierarchical multi-layer perceptron (hMLP), projecting all modalities into a shared hidden space.

Licensing: True Open-Source for the Enterprise

For enterprise IT teams and developers, the most disruptive aspect of Inkling may be its licensing. Inkling is released under the permissive Apache 2.0 license.

In an ecosystem where many so-called "open" models from Western labs are tethered to dual-use commercial licenses, acceptable use restrictions, or revenue caps, an Apache 2.0 designation makes Inkling a true open-source foundation. This gives developers the legal freedom to download, modify, integrate, and commercialize the model weights entirely royalty-free.

The model is readily deployable across major open-source inference libraries—including SGLang, vLLM, TokenSpeed, and llama.cpp—and comes with a native NVFP4 quantized checkpoint optimized for NVIDIA Blackwell systems.

Community Reactions: The Engineering Feat

The AI community's response has been swift, praising both the model's openness and the underlying engineering execution.

In a post on X, Thinking Machines co-founder John Schulman reflected on the rapid development cycle: "Inkling is out today, with open weights and in Tinker. It's been fun to watch this one come together: pretraining began last winter, and starting in mid-January a small team built up the coding, reasoning, and agentic training from there. We learned a lot building it, and I hope people find good uses for it."

Horace He, a researcher at Thinking Machines (previously from PyTorch), underscored the difficulty of the task in another post on X: "It truly takes a village to release a model, perhaps especially an open weights model. Actually doing the entire process from scratch, from data to pretraining to posttraining to actual release, gives a lot of appreciation for anyone who does it!"

The broader open-source ecosystem has also embraced the technical integrations. Lysandre Debut, the Chief Open-Source Officer at Hugging Face, shared his enthusiasm regarding the model's optimization in his own X post: "One thing I find quite striking is how much easier accelerating models has become... We replaced the model's causal Conv1D with the `causal-conv1d` kernel. One line changed, +4% tokens per second. We then replaced its attention implementation with FlashAttention-4. Another single change, another +11%. That's a total throughput improvement of about 15%, without changing the model architecture or retraining anything."

Tiezhen Wang, an ecosystem growth expert and ex-Googler, celebrated the release as a massive win for the open-source community, listing the model's impressive specifications on X, highlighting its "975B total, 41B active" size, "Native MTP support," and the highly coveted "Apache 2.0 license."

Background: The Road to Inkling

To understand the significance of Inkling, one has to look back at the rapid trajectory of Thinking Machines over the past 18 months.

When Mira Murati departed OpenAI in late 2024 to found Thinking Machines alongside industry veterans like John Schulman and Barret Zoph, the stated goal was to pivot away from building isolated autonomous agents. Instead, the company aimed to build flexible, multimodal systems designed for genuine human-AI collaboration and open science.

By July 2025, the startup had secured a historic $2 billion seed round led by Andreessen Horowitz at a $12 billion valuation. At the time, Murati promised the impending release of a product with a "significant open source component" to empower researchers and startups.

The company’s philosophy began coming into sharper focus in October 2025 with the launch of Tinker, a Python-based API for large language model fine-tuning that gave researchers granular control over training pipelines without the friction of distributed compute management.

That same month, Thinking Machines researcher Rafael Rafailov delivered a provocative critique of the AI industry at TED AI. He argued that the current trajectory of simply throwing more compute at models was fundamentally flawed, noting that today's systems take shortcuts—like wrapping code in try/except blocks—because they are trained strictly for task completion rather than genuine learning.

Rafailov posited that the first artificial superintelligence would not be a "god model," but rather a "superhuman learner" capable of meta-learning and internalizing abstractions. Inkling’s architecture—specifically its controllable thinking effort and its ability to organically compress its chain of thought during RL—feels like the first tangible realization of Rafailov's thesis.

In May 2026, the lab teased its technical prowess with the research preview of TML-Interaction-Small, a system that eliminated "turn-based" chat by processing inputs and outputs simultaneously in 200ms chunks. This "full-duplex" breakthrough proved the company could build highly responsive, natively multimodal models from scratch.

Now, with Inkling out in the wild, Thinking Machines has delivered on its foundational promises. By offering a massive, natively multimodal model under a true open-source license, they aren't just giving developers a new tool—they are attempting to fundamentally rewrite the economics and accessibility of frontier AI development.

The post Building a Gin Config Controlled PyTorch Pipeline with Configurable MLP Variants, Cosine Scheduling, and Runtime Parameter Overrides appeared first on MarkTechPost.

The post A Coding Guide to NVIDIA’s Tile-Based GPU Programming: From cuTile and Triton Kernels to Flash Attention appeared first on MarkTechPost.

Half of enterprises have deployed an AI agent or LLM feature that passed internal evaluations and yet still caused a customer-facing failure — one in four more than once — according to the June 2026 VB Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees.

The sample is self-selected rather than a probability sample, so the findings should be read as directional, not precise.

But enterprises are not responding by slowing automation: 66% of respondents already permit some production deployment without human review or are building systems intended to do so within the next 12 months. Only 5% say they fully trust the automated evaluations that would make those release decisions.

That mismatch is the evaluation gap: the autonomy ceiling is rising faster than the assurance beneath it. 

It also fits a broader thesis that will be explored at VB Transform 2026: enterprises ship agents first, while the control layers around identity, evaluation, cost, context and orchestration are arriving later. The next year will be a retrofit cycle, with buyers shifting budget toward the systems that make agentic deployments governable and dependable.

Why a passing evaluation is not a working agent

Traditional software testing usually asks whether a defined input produces an expected output. Agent testing is harder because the system may choose its own sequence of steps, call tools, retrieve data, alter state and respond differently from one run to the next.

An agent can make several individually plausible decisions and still reach the wrong result. It may retrieve the correct account but update the wrong field. It may draft a valid refund request but send it without approval. It may call five tools successfully before a sixth step leaks sensitive information or leaves a workflow incomplete.

The survey shows enterprises already recognize this limitation. The most common reason for distrusting automated evaluation is poor alignment with real-world outcomes, cited by 29% of respondents. Bias or inconsistency follows at 21%, lack of explainability at 18%, and data leakage or privacy concerns at 17%.

That hierarchy matters. Enterprises are saying the score often does not predict what happens when a customer, employee or business process encounters the agent in production — not that automated scoring is too slow or expensive.

NIST makes a similar point in its Generative AI Profile: measurements gathered in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context and operating conditions. Its guidance calls for field testing, post-deployment monitoring and clear processes for escalating failures.

Capability is not consistency

A single successful run proves that an agent can complete a task. It does not prove that it will complete the task reliably.

Anthropic’s guidance on agent evaluation distinguishes between measuring whether a system succeeds at least once across repeated attempts and whether it succeeds every time. That distinction is essential for customer-facing or operational workflows. A model that occasionally produces an excellent answer may still be unacceptable if the same task fails unpredictably on the next attempt.

Enterprise teams should therefore treat repeatability as a first-class metric. That means running the same scenario multiple times, varying phrasing and context, testing tool failures, and measuring whether the final business outcome remains correct even when the route changes.

The evaluation set also has to evolve. Every production incident should become a permanent regression test. Customer escalations, failed tool calls, incorrect approvals and data-handling mistakes should feed back into the pre-deployment suite rather than remaining isolated support cases.

Autonomy should expand by risk, not by ambition

The survey does not imply that every agent action should require a person. Human review cannot scale across millions of low-consequence decisions.

But zero-human operation should be earned by demonstrated reliability and bounded by the consequences of failure.

Low-risk actions such as drafting internal summaries or categorizing documents can tolerate broader autonomy. Financial transactions, customer communications, code deployment, access-control changes and data deletion need stricter thresholds, repeated consistency tests, policy checks, rollback mechanisms and clear human escalation paths.

The risk isn't evenly distributed by company size, either. Larger enterprises — those with 2,500 or more employees — are moving toward zero-human deployment fastest, at 70% versus 64% for smaller companies, and they're also shipping more agents that go on to fail a customer, at 54% versus 48%. 

That is the warning for enterprise leaders. Removing the human from the loop does not remove uncertainty. Without stronger assurance, it converts uncertainty into an automated production decision.

The market will keep pushing toward greater autonomy because the economic incentive is real. The organizations best positioned won't be those that remove people fastest — they'll be the ones that treat repeatability and regression testing as seriously as deployment speed.

It can generate predictions for a new, unseen table in a single forward pass. For enterprise developers and AI engineers, this reduces the time-to-production from weeks of pipeline engineering to a single API call.

The challenge with traditional ML

To extract reliable predictions from a gradient-boosted tree, data scientists must build and maintain complex data pipelines. They have to clean messy inputs, impute missing values, encode categorical variables into numerical formats, and engineer custom feature crosses.

Once the data is ready, they must run repetitive hyperparameter optimization loops, searching across learning rates, tree depths, subsampling ratios, and regularization grids to find the best configuration. 

Once deployed, these traditional models "incur ongoing operational debt through data drift monitoring and retraining pipelines to stay accurate," Weihao Kong, Research Scientist at Google Research, told VentureBeat.

Meanwhile, the rest of the AI industry has moved on. Generative AI models for text and computer vision have seamlessly shifted to zero-shot inference, where a model can perform a completely new task simply by being prompted with context. 

Large language models (LLMs) already excel at in-context learning, so why can't we just feed tables into an off-the-shelf LLM?

Because LLMs are trained on natural language rather than structured data, they struggle to process tables directly. First, their context limits are exhausted quickly by medium-sized tables containing just a few thousand rows and hundreds of columns. Second, LLMs suffer from tokenization inefficiency, awkwardly splitting numerical values and destroying mathematical precision. Finally, they suffer from structural blindness. When a 2D table is serialized as a 1D text string, LLMs lose track of which value belongs to which row and column as the table grows. 

"That's why, today, it is far more effective to use an LLM to write the code that handles feature engineering and calls XGBoost than to ask the LLM to read the table itself," Kong said.

What is TabFM?

To run inference with TabFM, you do not update any model weights. Instead, you take your historical examples (the training rows with their known labels) and your target rows (the new data you want to predict) and pass them to the model as a single, unified prompt. The model learns to interpret the relationships between columns and rows directly from this context at runtime.

For example, consider an enterprise analyst trying to predict customer churn. Instead of building a bespoke data pipeline and training an XGBoost model, they can simply pass a sample of historical user session data alongside a new, active session into TabFM. In one forward pass, the model returns an instant churn probability. 

TabFM overcomes the limitations of LLMs by treating the data as a grid, preserving its structural integrity without forcing it into a single-dimensional text string.

To effectively process diverse tabular structures while enabling scalable zero-shot prediction, TabFM synthesizes the strengths of earlier experimental architectures, TabPFN and TabICL. TabPFN, developed by Prior Labs, first proved that a transformer architecture could perform zero-shot classification on small tables, though it struggled to scale computationally to larger datasets. 

Later, TabICL, developed by France's National Research Institute for Digital Science and Technology, addressed this bottleneck by introducing row compression, allowing in-context learning to efficiently process much larger tables. 

TabFM combines TabPFN's deep feature contextualization with TabICL's efficient compression into a novel hybrid design built on three key mechanisms:

1. Alternating row and column attention: The raw table is first processed through a multilayer attention module that alternates across both columns (features) and rows (examples). By continuously attending across these two dimensions, the model natively captures complex feature interactions. This deep contextualization does the heavy lifting that would usually require tedious manual feature crafting by data scientists.

2. Row compression: Following this contextualization, the cross-attended information for each row is compressed into a single, dense vector representation. TabICL pioneered this by using CLS tokens to compress a row's rich information into one vector, "in contrast to TabPFN v2, v2.5, and v2.6, which attend over the full cell grid throughout the network," Kong explained. This drastically shrinks the computational footprint.

3. In-context learning (ICL): A causal Transformer then operates on this sequence of compressed embeddings. This Transformer model uses the attention mechanism of TabICL to attend over these dense row vectors, drastically reducing the computation cost and allowing the model to process large datasets efficiently.

A major selling point of TabFM is its pretraining recipe. The model was trained entirely on hundreds of millions of synthetic datasets. These datasets were dynamically generated using structural causal models (SCMs) that incorporate a wide variety of random functions. By training exclusively on synthetic SCMs, TabFM learned the fundamental mathematical priors of how tabular features interact without ingesting real-world, confidential CSV files.

TabFM in action

To test the model's capabilities, Google researchers benchmarked TabFM on TabArena, a comprehensive evaluation suite spanning 51 diverse tabular datasets across 38 classification and 13 regression tasks.

On these public benchmarks, TabFM's zero-shot predictions already match or beat heavily tuned supervised baselines. However, Google is careful to note that this does not automatically mean TabFM will universally dethrone bespoke, hyper-optimized production models on every enterprise workload.

"Instead of replacing hyper-optimized production models, the true practical business value it unlocks for lean engineering teams is velocity," Kong said. "It allows data analysts and backend engineers to instantly spin up high-quality baseline models without a dedicated data science team managing a complex lifecycle."

For advanced practitioners looking to squeeze out maximum accuracy, the research team also introduced a "TabFM-Ensemble" configuration. By running the model through 32 distinct variations and blending the results, TabFM pushes the performance even further. 

Getting started, trade-offs, and the cloud future

The shift to in-context learning for tables introduces a new economic trade-off that engineering teams must consider. 

With traditional algorithms, training is slow and expensive, but inference is lightning-fast and cheap. TabFM flips this dynamic. While training time drops to zero, inference becomes significantly heavier. Because the model must process the entire historical dataset as context during every single prediction, it requires more compute and memory at runtime. 

In this new paradigm, "traditional machine learning training becomes the 'prefill' phase (KV caching) in the context window," Kong said. While this prefill cost is steep, it is paid only once per table, and the cache is reused across subsequent queries. "The catch is prediction latency, which no amount of caching removes," Kong added. Every new prediction requires a pass through a large transformer. "Any production API requiring single-digit-millisecond response times cannot tolerate TabFM's forward-pass overhead."

For developers looking to evaluate the model today, the barrier to entry is low. Google designed TabFM as a drop-in replacement for traditional ML workflows, offering a scikit-learn compatible API (TabFMClassifier and TabFMRegressor). It natively handles mixed numerical and categorical columns, works directly with pandas DataFrames, and requires no manual ordinal encoders or numerical scalers. The library supports both JAX and PyTorch backends.

However, enterprise teams need to be aware of current limitations and licensing restrictions. The model architecture has a hard limit of 10 output classes for classification tasks, and it is optimized for tables with up to 500 features. More importantly, while Google released the underlying codebase under the permissive Apache 2.0 license, the pre-trained model weights are published on Hugging Face under a strict tabfm-non-commercial-v1.0 license. Developers can evaluate the model internally, but it cannot be deployed in commercial products yet.

Looking ahead, Google is addressing the commercial deployment friction through its cloud ecosystem. TabFM is being integrated directly into Google BigQuery, allowing analysts to run zero-shot predictions natively via an “AI.PREDICT” command. By putting foundation model inference right next to the data warehouse, TabFM could soon make complex tabular machine learning as accessible as a basic database query.

In practice, TabFM shines in rapid prototyping, high data drift environments, and small to medium-sized datasets under 100,000 rows. Conversely, teams should stick to traditional models for strict, ultra-low latency APIs, or massive tables exceeding one million rows, which currently require aggressive row sampling that degrades the foundation model's competitive advantage.

2026 EuroLLVM Developers' Meeting
https://llvm.org/devmtg/2026-04/
------
Title: Lighthouse: infrastructure for end-to-end MLIR-compilers and testing
Speaker: Renato Golin
------
Slides: https://llvm.org/devmtg/2026-04/slides/technical_talk/technical_talk_golin.pdf
-----
Last year, a new project was added to the LLVM family: Lighthouse. Its main purpose is to guide the development and testing of MLIR based compilers. Like the LLVM test-suite, it should be a common ground for validating upstream assumptions about code, IR, dialects. At the same time, it enables building specific compilers in minutes, using the evolving Python API and MLIR's Python bindings. In this talk, we'll show the project main structure, including its components and how to use them to build a simple compiler. We'll then show the infrastructure that uses those components to validate assumptions in MLIR (canonical forms, invariants, applicability of transforms and passes, correctness tests, etc), and how you can create your own on top of that. Finally, we'll provide a number of pipeline examples, going from generic PyTorch models to performant execution on various targets.
-----
Videos Edited by Bash Films: http://www.BashFilms.com

2026 EuroLLVM Developers' Meeting
https://llvm.org/devmtg/2026-04/
------
Title: HIVM: MLIR Dialect Stack for Ascend NPU Compilation
Speaker: Hugo Trachino
------
Slides: https://llvm.org/devmtg/2026-04/slides/tutorial/tutorial_tarasov.pdf
-----
Huawei Ascend NPUs combine DaVinci AI cores with a rich memory/synchronization hierarchy and, on newer generations, a SIMD+SIMT execution model, making performance-oriented compilation challenging. We present HIVM, an open-source family of MLIR dialects that lowers PyTorch/Inductor - Triton - MLIR (HIVM) - LLVM IR, enabling Ascend-specific optimizations such as layout assignment/propagation, vector intrinsic selection/legalization, and explicit DMA/transfer scheduling with synchronization. The pipeline ultimately targets the BiSheng LLVM-based backend to produce executable code for Ascend chips. The talk walks step-by-step through the key IR levels and transformation passes, serving as a practical baseline for developers building MLIR toolchains for Ascend.
-----
Videos Edited by Bash Films: http://www.BashFilms.com

• With out=(out_tokens, out_weights, out_idx): writes to caller-supplied
  buffers and returns them; no autograd (efficient buffer reuse).
• Without out: allocates buffers internally, returns them with full
  autograd support via _DispatchAutograd / _CombineAutograd.

Subclasses implement _dispatch/_combine as the raw buffer-writing
primitives and inherit both modes from the base class. topk_weights
receives no gradient (routing metadata at a different byte width).

Authored with Claude.
Pull Request resolved: #181314
Approved by: https://github.com/kapilsh
ghstack dependencies: #178712

This PR reverts #182630 to fix #187277

Root cause is ext_oneapi_wait_and_throw will introduce SYCL Graph to sync an invalid queue.

Additional Context

This needs to be cherry-picked to the release branch.

Why isn't it captured on CI?
This issue is only found on BMG (Xe2), and the current CI is on Data Center GPU (Xe).
Pull Request resolved: #187306
Approved by: https://github.com/EikanWang, https://github.com/atalman
ghstack dependencies: #181233, #187137, #187232

Authored with Cursor Composer 2.5 (composer-2.5-fast).

Fixes #186991

Pull Request resolved: #186997
Approved by: https://github.com/huydhn

Reverted #186877 on behalf of https://github.com/huydhn due to Diff reverted internally (comment)

VariableBuilder previously only handled raw SymInt inputs when they had a guardable hint. Unhinted raw SymInt inputs graph-broke immediately, even if the nested graph only needed to carry them symbolically. This change keeps that support narrow: outside non-strict tracing, raw unbacked SymInt wrapping still graph-breaks. Inside non-strict tracing, Dynamo asks the active ShapeEnv to transfer the foreign unbacked SymInt as a ShapeEnv-owned input, reuses it for repeated occurrences of the same source expression, and preserves transferred range/source/hint metadata through the existing foreign-ShapeEnv transfer helpers.

The full non-strict FlexAttention repro also exposed a tracing-only issue: same-frame guard validation eagerly evaluates generated guards against fake inputs whose unbacked sizes cannot be concretized. The runtime guards are still produced, but the same-frame sanity check is skipped while inside the non-strict tracing context.

This keeps the important safety property unchanged: if user code branches on a raw unbacked SymInt, Dynamo still raises the normal data-dependent-symbol error instead of specializing on an optimization hint.

Fixes #187272

This PR was authored with assistance from an AI assistant.

Test Plan:

PYTORCH_TEST_WITH_DYNAMO=1 python -m pytest test/test_fake_tensor.py::FakeTensorTest::test_cudnn_sdpa_unbacked_batch_dim -q -s

python -m pytest test/dynamo/test_repros.py -q -s -k flex_attention_non_strict_unbacked_sequence_length

python -m pytest test/dynamo/test_dynamic_spec.py -q -s -k non_strict_raw_unbacked_symint_input_raises_dde_on_branching

python -m pytest test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_unbacked_symint_closure -q -s

python -m pytest test/test_dynamic_shapes.py -q -s -k "unbacked_hint_overrides_transferred or mixed_static_backed_unbacked"

python -m pytest test/inductor/test_unbacked_symints.py::TestUnbackedSymintsCPU -q -s -k override_optimization_hint

lintrunner -a

stack-info: PR: #187273, branch: sanketpurandare/stack/20

Flex decoding remains an optional optimized implementation. Its predicates use guard_or_false because they are kernel eligibility checks, not user-visible invariants: when ShapeEnv can prove eligibility, decode can be selected; when an unbacked predicate is not provable, the general FlexAttention kernel is used. Failing to prove an optimization must not impose a runtime shape contract on user inputs.

Backward fake propagation preserves the accepted KV-batch-broadcast metadata contract. It reduces grad_key and grad_value back to key/value batch when key/value is provably batch-broadcasted; otherwise it records the non-broadcast invariant with torch._check(Bq == Bkv) and returns unreduced metadata.

Public BlockMask length validation is expressed as two symbolic torch._check predicates: the mask must not be smaller than the query/key lengths, and it must not be larger than those lengths. Concrete mismatches still fail with the existing guidance, while unbacked symbolic lengths become deferred runtime assertions instead of forcing Python to branch on an unbacked expression.

This PR was authored with assistance from an AI assistant.

Test Plan:

python -m pytest test/inductor/test_flex_attention.py -q -s -k "unbacked_flex_decoding_eligibility_falls_back or backward_fake_symbolic_query_key_batch_non_broadcast or block_mask_vs_sequence_lengths"

python -m pytest test/inductor/test_flex_attention.py -q -s -k "mask_mod_handles_symint_addition or mask_mod_handles_derived_symint_closure or symbol_closure_in_score_mod"

lintrunner -a

stack-info: PR: #183838, branch: sanketpurandare/stack/12

Flex decoding remains an optional optimized implementation. Its predicates use guard_or_false because they are kernel eligibility checks, not user-visible invariants: when ShapeEnv can prove eligibility, decode can be selected; when an unbacked predicate is not provable, the general FlexAttention kernel is used. Failing to prove an optimization must not impose a runtime shape contract on user inputs.

Backward fake propagation preserves the accepted KV-batch-broadcast metadata contract. It reduces grad_key and grad_value back to key/value batch when key/value is provably batch-broadcasted; otherwise it records the non-broadcast invariant with torch._check(Bq == Bkv) and returns unreduced metadata.

Public BlockMask length validation is expressed as two symbolic torch._check predicates: the mask must not be smaller than the query/key lengths, and it must not be larger than those lengths. Concrete mismatches still fail with the existing guidance, while unbacked symbolic lengths become deferred runtime assertions instead of forcing Python to branch on an unbacked expression.

This PR was authored with assistance from an AI assistant.

Test Plan:

python -m pytest test/inductor/test_flex_attention.py -q -s -k "unbacked_flex_decoding_eligibility_falls_back or backward_fake_symbolic_query_key_batch_non_broadcast or block_mask_vs_sequence_lengths"

python -m pytest test/inductor/test_flex_attention.py -q -s -k "mask_mod_handles_symint_addition or mask_mod_handles_derived_symint_closure or symbol_closure_in_score_mod"

lintrunner -a

stack-info: PR: #183838, branch: sanketpurandare/stack/12

Render complex constants with a zero real or imaginary component as complex(real, imag) through the existing recursive argument printer. Float repr preserves -0.0, so those generated constants round-trip signed zero components. Nonzero complex constants keep the existing repr() path to avoid changing the common codegen case.

Fixes #153852

Generated by my agent

Benchmark Results:

• FX GraphModule construction with 1000 zero-component complex constants, 80 iterations x 7 repeats, median: main 3.55 ms/graph; this diff 6.77 ms/graph. This is the affected correctness path that now emits complex(real, imag) to preserve signed zero.

• FX GraphModule construction with 1000 nonzero complex constants, same command shape, median: main 4.29 ms/graph; this diff 3.90 ms/graph. Nonzero constants remain on the existing repr() path; the difference is measurement noise.

Test Plan:

• python test/dynamo/test_repros.py ReproTests.test_compile_complex_tensor_constant_signed_zero

• python test/test_fx.py TestFX.test_complex_constant_codegen_preserves_signed_zero (direct run blocked locally before target test by unrelated torchvision::nms registration failure)

• targeted TestFX.test_complex_constant_codegen_preserves_signed_zero via a torchvision import stub

• lintrunner -a

• git diff --check

Pull Request resolved: #185550
Approved by: https://github.com/desertfire

Pull Request resolved: #187255
Approved by: https://github.com/malfet

Reverted #183670 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable (comment)

Materialize a zero score gradient when the joint graph returns None for the score input. Constant joint graphs can lower that zero as a scalar/rank-1 Triton value, so the backward template now broadcasts only those low-rank grad_scores values to the score tile before the matmuls. Rank-2 gradients from normal differentiable score_mod paths skip the extra broadcast add.

I considered only adding a template-side fallback, but that would leave the higher-order op contract ambiguous: the joint graph should always provide a score-gradient value to the backward lowering. Materializing the zero in create_fw_bw_graph fixes that contract, while the template rank guard handles the generated representation needed by Triton.

Fixes #145050

Generated by my agent

Benchmark Results:

• CUDA fp16 compiled flex_attention backward, B=1 H=1 S=512 D=64, score_mod returns score * 1.1, 5 warmup iterations and 50 CUDA-event timed iterations.
• Baseline main median: 0.5343 ms; p10/p90: 0.4784/0.6628 ms.
• Patched median: 0.5404 ms; p10/p90: 0.5289/0.5629 ms.
• Result: no clear regression beyond run-to-run noise; baseline timings were bimodal in this environment, while patched timings stayed in the same envelope.

Test Plan:

• Reproduced the issue before the fix with a minimal CUDA repro using torch.compile(flex_attention, dynamic=False), score_mod returning q_idx >= kv_idx, and backward failing with InductorError / joint_subgraph_buffer is None.
• Verified the repro after the fix: q.grad and k.grad are zero, v.grad is nonzero.
• Verified backend="aot_eager" repro after the fix.
• Verified differentiable score_mod=score * score smoke test after the fix.
• python test/inductor/test_flex_attention.py -k test_score_mod_without_score_gradient
• lintrunner -a

Pull Request resolved: #185991
Approved by: https://github.com/drisspg

Fixes #140219
Fixes #140220

Generated by my agent

Pull Request resolved: #184217
Approved by: https://github.com/yushangdi

Reverted #186928 on behalf of https://github.com/wdvr due to See below - failing builds with older cuDNN (comment)

Reverted #185553 on behalf of https://github.com/wdvr due to failing with older cuDNN - see below (comment)

[DTensor] Support _StridedShard to Shard through all-to-all

Summary

This PR adds support for redistributing tensors from _StridedShard placement to Shard placement using the all-to-all collective operation.

The key challenge is that _StridedShard produces non-contiguous (interleaved) shards, so converting to a regular Shard placement requires:

1. Properly computing padding for both the source strided dimension and target dimension
2. Reordering elements after the all-to-all to restore contiguous layout

Example: Converting _StridedShard(0, split_factor=2) to Shard(1)

Consider the following setup:

• Mesh shape: (4,) — 4 ranks on a single mesh dimension
• Original tensor shape: (9, 4)
• Source placement: (_StridedShard(0, split_factor=2),)
• Target placement: (Shard(1),)

Step 1: Understand the _StridedShard distribution

With _StridedShard(0, split_factor=2), the tensor is conceptually split in two levels on dimension 0:

1. First level: Split into split_factor=2 pieces → chunks of size ⌈9/2⌉ = 5, giving pieces [0:5] and [5:9]
2. Second level: Each piece is split into num_chunks=4 pieces (mesh size)

The shards are then interleaved so each rank gets one slice from each first-level piece:

Original tensor (9x4):            Strided sharding on dim 0:
┌─────────────────────┐
│ row 0               │  ─┐
│ row 1               │   ├─ First piece [0:5], split into 4 chunks
│ row 2               │   │  → chunks: [0:2], [2:4], [4:5], []
│ row 3               │   │
│ row 4               │  ─┘
│ row 5               │  ─┐
│ row 6               │   ├─ Second piece [5:9], split into 4 chunks
│ row 7               │   │  → chunks: [5:6], [6:7], [7:8], [8:9]
│ row 8               │  ─┘
└─────────────────────┘

Interleaved distribution to ranks:
  Rank 0: rows [0,1] + [5]     = rows [0,1,5]     (3 rows)
  Rank 1: rows [2,3] + [6]     = rows [2,3,6]     (3 rows)
  Rank 2: rows [4]   + [7]     = rows [4,7]       (2 rows)
  Rank 3: []         + [8]     = rows [8]         (1 row)

Step 2: Pad for uniform all-to-all

Before all-to-all, we pad so all ranks have uniform chunk sizes:

• Old dimension (dim 0): max_chunk_size = 3, pad ranks 2 and 3
• New dimension (dim 1): size 4 with 4 chunks → already uniform (chunk size 1 each)

After padding dim 0:
  Rank 0: [0,1,5] (no padding)    → shape (3, 4)
  Rank 1: [2,3,6] (no padding)    → shape (3, 4)
  Rank 2: [4,7,P] (+1 padding)    → shape (3, 4)
  Rank 3: [8,P,P] (+2 padding)    → shape (3, 4)

Step 3: All-to-all on dim 0 → dim 1

The all-to-all exchanges slices: each rank sends dim-1 slices to other ranks and receives dim-0 slices:

Before A2A (each rank has 3x4):     After A2A (each rank has 12x1):
  Rank 0: rows [0,1,5] cols [0,1,2,3]  →  col 0 from all ranks
  Rank 1: rows [2,3,6] cols [0,1,2,3]  →  col 1 from all ranks
  Rank 2: rows [4,7,P] cols [0,1,2,3]  →  col 2 from all ranks
  Rank 3: rows [8,P,P] cols [0,1,2,3]  →  col 3 from all ranks

Step 4: Unpad and reorder

After all-to-all, each rank has interleaved rows from the strided pattern with padding. We use index_select to:

1. Extract only the valid (non-padded) elements
2. Reorder from strided order [0,1,5,2,3,6,4,7,8] back to natural order [0,1,2,3,4,5,6,7,8]

Final result - Shard(1) distribution:
  Rank 0: all 9 rows, col 0  → shape (9, 1)
  Rank 1: all 9 rows, col 1  → shape (9, 1)
  Rank 2: all 9 rows, col 2  → shape (9, 1)
  Rank 3: all 9 rows, col 3  → shape (9, 1)

Pull Request resolved: #170915
Approved by: https://github.com/weifengpy

_split_iteration_ranges in torch/_inductor/codegen/simd.py ended with a bare assert all(... == 1 for s in remaining) that fired whenever a node's iteration lengths were consumed cleanly but left a non-unit extent in the kernel's tiling groups. This happens when a pointwise epilogue whose iteration domain is a strict sub-multiple of a template's tiling is considered for fusion, e.g. fusing a [s, N] epilogue into a [K*s, N] matmul template tile. The three divisibility exits earlier in the same function already raise CantSplit, and both callers that drive epilogue fusion (SIMDKernel.is_compatible and Scheduler.speedup_by_fusion) wrap the split in try/except CantSplit specifically to skip range-incompatible fusions and fall back to unfused codegen. Because the final invariant raised AssertionError instead of CantSplit, it escaped those handlers and hard-failed the entire AOTInductor compile with InductorError: AssertionError: failed to set ranges ....

This PR converts the final invariant to raise CantSplit(remaining, lengths), matching the exception contract of the rest of the function so the existing safety nets catch it. The success path is byte-for-byte unchanged (identical trigger condition), so models that compile today are unaffected; models that previously crashed now lower with the incompatible epilogue left unfused (a separate kernel) instead of aborting the compile.

Test Plan:
Unit test (new regression):

buck2 test //caffe2/test/inductor:loop_ordering -- -r TestSplitIterationRanges

Result: Pass 7, Fail 0. testrun: https://www.internalfb.com/intern/testinfra/testrun/1688850234333388

The new case test_leftover_extent_raises_cant_split constructs groups=[2, 2], lengths=[[2], []]: every size divides cleanly as it is consumed (so no add_range divisibility exit trips), but group 1 is left with extent 2, producing remaining=[1, 2]. It asserts this raises CantSplit rather than AssertionError, exercising exactly the changed line.

Differential Revision: D108339950

Pull Request resolved: #187209
Approved by: https://github.com/ColinPeppler

Pull Request resolved: #178466
Approved by: https://github.com/jansel

Imports TypeVar from typing_extensions alongside ParamSpec so both type parameters use the same default-sentinel machinery when another vendored typing_extensions copy has monkeypatched typing internals. This preserves the existing CachedMethod[P, RV] convention while avoiding the import-time TypeError.

Test Plan:

• python3 -m py_compile torch/_inductor/utils.py
• /Users/dejain/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 minimized repro that loads a second typing_extensions copy and defines CachedMethod(Protocol, Generic[P, RV])
• git diff --check

Pull Request resolved: #185708
Approved by: https://github.com/Skylion007, https://github.com/jansel

We are working on add XPU symmetric backend and also want to enable symm ops (async TP) for XPU.

Motivation

We are working on add XPU symmetric backend and also want to enable symm ops (async TP) for XPU. With those symm ops, communication and computation could be overlapped to reduce communication overhead on Intel client GPUs.

Changes

• Backend enabling in intel/torch-xpu-ops#2041
• Python ops enabling in this PR

Test Plan

Ops test is verified via intel/torch-xpu-ops#3747

Pull Request resolved: #185102
Approved by: https://github.com/guangyey, https://github.com/gujinghui, https://github.com/jansel

Pull Request resolved: #182028
Approved by: https://github.com/drisspg, https://github.com/jansel

Fixes #177839.

Summary

For eps > 0.5 the clamp bounds invert (lo = eps > hi = 1 - eps). The scalar and CUDA kernels resolve this as x < lo ? lo : (x > hi ? hi : x), so lo wins. Two paths disagreed and produced a sign-flipped result:

• CPU vectorized: vec::clamp = min(hi, max(lo, x)), so hi won.
• MPS logit: clampWithTensor, also min(max(x, lo), hi), so hi won.

Both now apply lo last (nested blendv on CPU, nested select on MPS), matching scalar and CUDA. CUDA already conformed, so all three backends agree. eps > 0.5 is mathematically ill-defined; the goal is cross-backend consistency.

Tests

Added test_logit_vectorized_matches_scalar (eps {0.49, 0.51, 0.6, 0.9} x 4 dtypes, bit-exact vs the scalar kernel) and an eps=0.6 sample_inputs_logit sample so test_ops.py and the MPS consistency tests cover eps > 0.5. Verified on Apple Silicon: test_mps.py -k logit, test_ops.py -k logit, and test_unary_ufuncs.py -k logit pass, including the previously failing test_output_grad_match_logit_mps_float32.

BC-breaking?

No.

Pull Request resolved: #181297
Approved by: https://github.com/jansel

Fixes #121135

Generated by my agent

Pull Request resolved: #184373
Approved by: https://github.com/zou3519

Fixes #182402
Fixes #183773
Fixes #160873

Generated by my agent

Pull Request resolved: #183670
Approved by: https://github.com/aorenste

Pull Request resolved: #187208
Approved by: https://github.com/d4l3k, https://github.com/lakshayg, https://github.com/malfet

these are simple replacements of std::min(std::max(...), ..) with std::clamp I found using git grep. there are other occurrences in the codebase, i'll follow up with another PR for those. Thanks!

Pull Request resolved: #185490
Approved by: https://github.com/Skylion007

Reverted #186346 on behalf of https://github.com/wdvr due to Diff reverted internally (comment)

Removes # mypy: allow-untyped-defs from torch/__init__.py (and adds a matching pyrefly.toml sub-config) and reworks how the magic methods on SymInt, SymFloat, and SymBool are typed.

Previously each class carried a long hand-written list of placeholder stubs like def __add__(self, other) -> "SymInt": .... These didn't model the actual promotion rules — e.g. SymInt + float returns SymFloat, Tensor + SymT returns Tensor, and arithmetic on SymBool promotes to SymInt. Three new generic mixins encode those rules once and are reused across all three classes:

• _SymTypingMagicAlsoBool[_PrimType, _BecomesIntPrimType, _BecomesIntSymType, _FloatPromotionType] — comparisons (==, !=, <, <=, >, >=) and the bool-promoting arithmetic ops (+, -, *, plus their r-variants), with _FloatPromotionType controlling whether the result can be SymFloat.
• _SymTypingMagic[_PrimType, _SymType, _FloatPromotionType] — the rest of the numeric magic methods (abs, neg, floor, ceil, trunc, mod, lshift/rshift, pow_by_natural, the __sym_*__ math wrappers, __int_truediv__ / __int_floordiv__, etc.).
• _SymTypingMagicBitwise[_BitwiseLikeType] — __and__, __or__, __xor__ and their reflected forms.

SymInt/SymFloat/SymBool then become small concrete subclasses parameterized over the right operand and result types (e.g. SymFloat's float-promotion type is _Never since it doesn't promote further).

Runtime side, sym_node.py gains a real xor entry in bitwise_ops, only_bool_magic_methods, and the dispatch table (with a new _sympy_xor), and also_bool_magic_methods is widened from just {"eq"} to the full comparison set {"eq", "ge", "gt", "le", "lt", "ne"} so the methods installed at runtime line up with what the new mixin advertises. A magic_methods_excl helper set is added for the remainder.

Other fallout from removing allow-untyped-defs:

• PySymType is exported from torch._C stubs and threaded into the asymmetric comparison-op signatures generated by tools/pyi/gen_pyi.py, so Tensor.__lt__(SymInt) etc. type-check.
• SymInt.has_hint(), SymInt.hint, SymInt.constant move from ad-hoc .node.* access to typed property accessors; definitely_true_hint/etc. are updated to use them.
• SymNode.shape_env is Optional, so callers in symbolic_shapes.py now raise explicit AssertionError("shape_env should not be None") instead of relying on it being set. A couple of maybe_as_int() / maybe_as_float() call sites are tightened with walrus assignment to avoid calling the method twice.
• # pyrefly: ignore[missing-attribute] is added where SymInt.node is duck-typed across SymNode | NestedIntNode | ConstantIntNode | LocalIntNode and the attribute only exists on some.

Review order: start with torch/__init__.py — the three new mixins and the rewritten SymInt/SymFloat/SymBool definitions are where the design lives. Then torch/fx/experimental/sym_node.py for the runtime registration of xor and the broadened also_bool_magic_methods. Everything else is small call-site adjustments to satisfy the stricter typing.

Test Plan:
CI.

Authored with Claude.

Reviewed By: bobrenjc93

Differential Revision: D106389841

This PR fixes return type annotations across several modules, reducing functions returning Any.

Overall these changes should reduce the number of Any returns by PyTorch which is useful, especially when you want to use strict type checking.
I'm not an expert with all of these functions and hope I got the exact return types, and am not missing something. But beforehand the return types were any or unknown, so they can't really be much worse.

Changes

1. torch.futures.Future - New stub file with proper generics

• Added torch/futures/__init__.pyi with full generic support
• Future[T].wait() now returns T instead of Any
• torch.wait(fut: Future[T]) now returns T instead of Any

2. torch._C - Dispatch mode return types

• _pop_torch_dispatch_stack() → TorchDispatchMode | None
• _get_dispatch_mode() → TorchDispatchMode | None
• _get_dispatch_stack_at() → TorchDispatchMode

3. torch.distributed - Process group and exception types

• join_process_group() → ProcessGroup (was Any)
• Work.exception() → BaseException | None (was Any)

4. torch._C._distributed_rpc - PyRRef generic methods

• local_value() → _T
• rpc_sync() → _T
• rpc_async() → Future[_T]
• remote() → PyRRef[_T]

5. torch.jit._script - RecursiveScriptModule iteration

• __getitem__() → RecursiveScriptModule
• __iter__() → Iterator[RecursiveScriptModule]

6. torch._C._jit_tree_views - Literal and decl types

• TrueLiteral/FalseLiteral/NoneLiteral → Expr
• Def.decl() → Decl

7. new Stub files

• torch/linalg/__init__.pyi - SVD, eigendecompositions, norms, matrix solvers
• torch/fft/__init__.pyi - FFT/IFFT operations, frequency helpers
• torch/special/__init__.pyi - Error functions, gamma functions, Bessel functions

8. torch.distributed - ZeroRedundancyOptimizer stub

• Added torch/distributed/optim/zero_redundancy_optimizer.pyi with full type annotations

9. torch.distributed.algorithms.join

• Type annotations for Join class and related functions

10. torch.nn.parallel.distributed

• Type annotation improvements for DistributedDataParallel

11. torch.jit.frontend

• Type annotations for JIT frontend functions

12. torch._dynamo type fixes

• Fixed type issues in variables/constant.py, variables/functions.py, variables/script_object.py

13. torch.fx.experimental.proxy_tensor

• Added pyrefly ignore comments for complex type cases

14. mypy-strict.ini

• Enabled strict type checking for torch.linalg, torch.fft, torch.special

15. Typing tests

• Added test/typing/pass/linalg_fft_special.py - pass tests for linalg/fft/special functions
• Added test/typing/reveal/linalg_fft_special.py - reveal tests for return types
• Updated test/typing/reveal/namedtuple.py - torch.linalg.qr now returns QRResult instead of Any

Test

import torch
from torch.futures import Future
from torch import Tensor

fut: Future[Tensor] = Future()
fut.set_result(torch.zeros(10))
result = torch.wait(fut) # type now: Tensor, before: Any

import torch.linalg
import torch.fft
import torch.special

result = torch.linalg.svd(torch.randn(5, 5)) # type now: SVDResult, before: Unknown
result = torch.linalg.norm(torch.randn(5, 5)) # type now: Tensor, before: Unknown
result = torch.fft.fft(torch.randn(10)) # type now: Tensor, before: Unknown
result = torch.special.erf(torch.randn(10)) # type now: Tensor, before: Unknown

Previous approaches

The issue was also mentioned in #149639.
Some other approaches like #160750 were tried before, but were reverted.

Mitigations vs #160750

The previous PR was reverted for several reasons. Here's how this PR addresses them:

1. Flexible type signatures (no cascading errors)
Unlike #160750 which used strict types like tuple[int, ...] that broke existing call sites, this PR uses flexible signatures:

# This PR: accepts Sequence, list, or tuple
dim: int | SymInt | Sequence[int | SymInt] | None

This prevents the 14+ [arg-type] errors that caused the original revert.

2. Automated test plan
Added typing tests to catch regressions:

• test/typing/pass/linalg_fft_special.py - verifies functions accept correct arg types
• test/typing/reveal/linalg_fft_special.py - verifies return types (SVDResult, QRResult, etc.)

3. Enabled strict mypy checking
Added follow_imports = normal for torch.linalg, torch.fft, torch.special in mypy-strict.ini

4. Updated existing test
Fixed test/typing/reveal/namedtuple.py which previously expected Any for torch.linalg.qr - now correctly expects QRResult

Pull Request resolved: #170396
Approved by: https://github.com/Skylion007

The scheduler's fusion loop recognizes NVUniversalGemmCaller, benchmarks
fused vs unfused EFC kernels, and selects the winner.
MultiTemplateBuffer gains swap/finalize_as_nvgemm_caller for the
benchmark swap. When NVGEMM can't fuse, MTBs fall through to Triton
scheduling so Triton choices can still attempt epilogue fusion.

Authored with the help of an AI assistant (Claude).

Test Plan:

python test/inductor/test_nv_universal_gemm.py -v -k Epilogue
python test/inductor/test_max_autotune.py -v -k epilogue_fusion

Pull Request resolved: #186183
Approved by: https://github.com/drisspg

Restores disk cache (disk_cache_get/set) and the _precompile hook for
subprocess parallel compilation that were present in the Jinja template.
Cache keys now include strides, output metadata, and device for all
tensor variants.

Authored with the help of an AI assistant (Claude).

Test Plan:

python test/inductor/test_nv_universal_gemm.py -v

Pull Request resolved: #186184
Approved by: https://github.com/drisspg
ghstack dependencies: #186183

Reverted #187208 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable (comment)

This keeps strict export behavior unchanged and rewrites only non-strict data_ptr failures that occur from Triton's runtime into an actionable error telling users to wrap the kernel in torch.library.triton_op and launch it through torch.library.wrap_triton or torch._library.capture_triton. A focused CUDA/Triton regression test covers the raw-kernel non-strict path, and the existing custom Triton export test covers the supported path.

Fixes #146066
Generated by my agent

Test Plan:
python test/export/test_export.py TestExport.test_export_raw_triton_kernel_non_strict_error
python test/export/test_export.py TestExport.test_export_custom_triton_kernel
lintrunner -a
Pull Request resolved: #185827
Approved by: https://github.com/zou3519, https://github.com/desertfire

Pull Request resolved: #187208
Approved by: https://github.com/d4l3k, https://github.com/lakshayg, https://github.com/malfet

We're sometimes getting segfaults in torch::profiler::impl::onFunctionEnter() because we get back a raw pointer from GlobalStateManager::get(). The underlying state being tracked, however, is actually a std::shared_ptr. The thread that owns the std::shared_ptr sometimes destroys the object while other threads have a dangling raw pointer to the state.

The approach we take to fixing the problem is to always hand out an actual std::shared_ptr when the callback will be executed on another thread. That ensures that even if the thread that owns the state is joined, the state itself will still live as long as the thread has a valid copy of the std::shared_ptr.
Pull Request resolved: #187141
Approved by: https://github.com/ryanzhang22

Co-authored-by: Jiannan Wang jiannanwang@meta.com

We're sometimes getting segfaults in torch::profiler::impl::onFunctionEnter() because we get back a raw pointer from GlobalStateManager::get(). The underlying state being tracked, however, is actually a std::shared_ptr. The thread that owns the std::shared_ptr sometimes destroys the object while other threads have a dangling raw pointer to the state.

The approach we take to fixing the problem is to always hand out an actual std::shared_ptr when the callback will be executed on another thread. That ensures that even if the thread that owns the state is joined, the state itself will still live as long as the thread has a valid copy of the std::shared_ptr.
Pull Request resolved: #187141
Approved by: https://github.com/ryanzhang22

Co-authored-by: Jiannan Wang jiannanwang@meta.com

Reverted #183992 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable (comment)

If you’ve ever tried setting up Ollama, Stable Diffusion, or PyTorch on an AMD graphics card, you probably remember how

The post A Coding Implementation on Spatial Graph Neural Networks for Urban Function Inference Using city2graph, OSMnx, and PyTorch Geometric appeared first on MarkTechPost.

This addresses a regression from #186667, which migrated matrix ops to single-dim strategies and introduced the mixed per-mesh-dim behavior for these outputs.

Preserve output positions that are None in every explicit single-dim strategy when inserting the implicit replicate rule. Also make full-mesh expansion reject mixed None/Placement entries instead of constructing invalid DTensorSpec objects.

Test Plan: Added regression tests for SDPA efficient-attention philox scalar outputs and mixed None/Placement output rejection. Ran the affected tests and lintrunner.

Authored with the assistance of an AI coding assistant.
Pull Request resolved: #187199
Approved by: https://github.com/pianpwk

Summary

torch.cuda.nccl.broadcast(tensors, root=N) silently ignores root and always broadcasts from tensors[0], while nccl.reduce honors root — full root-cause chain in my issue comment:

1. torch/cuda/nccl.py::broadcast passes root to torch._C._nccl_broadcast correctly.
2. THCPModule_nccl_broadcast (torch/csrc/cuda/python_nccl.cpp) parses and validates root, then calls torch::cuda::nccl::broadcast(inputs, streams, user_comms) without it.
3. torch::cuda::nccl::broadcast (torch/csrc/cuda/nccl.cpp) has no root parameter and hardcodes 0 in the ncclBcast call.

The bug is old: v0.3.0 passed root through to ncclBcast; commit de5f7b7251f ("Base for pure C++ NCCL interface", Dec 2017) dropped it when the C++ helper was extracted, so the binding has been wrong since v0.4.0 (~8.5 years).

Fix

• torch/csrc/cuda/nccl.h / nccl.cpp: add int32_t root = 0 to torch::cuda::nccl::broadcast, mirroring the existing reduce signature (defaulted, so the internal caller in torch/csrc/cuda/comm.cpp::broadcast_coalesced is unaffected); validate it in-range exactly like reduce does; pass it to ncclBcast.
• torch/csrc/cuda/python_nccl.cpp: forward the already-parsed root.
• test/distributed/test_nccl.py: extend test_broadcast with a non-zero-root regression block (root = nGPUs - 1; the root device holds a uniform_() tensor, all other devices hold zeros; assert every device ends up with the root's values). Before this fix the broadcast originates from the zeros at index 0, so the new assertions fail; after the fix they pass. It runs under the existing TEST_MULTIGPU skip and @dtypes(*broadcast_dtypes) (incl. float8) conventions.

Test plan

• New regression assertions in test/distributed/test_nccl.py::TestNCCL::test_broadcast (multi-GPU).
• lintrunner -a on the four touched files: no lint issues (CLANGTIDY skipped locally — requires a build dir).
• Compile-verified the touched translation units (nccl.cpp, python_nccl.cpp) plus the unmodified internal caller (comm.cpp) with g++ -fsyntax-only against the modified header: all clean.
• Validation gap, stated honestly: my box has a single GPU, so I could not run the multi-GPU broadcast path locally; python test/distributed/test_nccl.py -k broadcast collects and skip-passes here (OK (skipped=2)). The behavioral coverage relies on PyTorch's multi-GPU CI exercising the extended test_broadcast.

🤖 Generated with Claude Code
Pull Request resolved: #187216
Approved by: https://github.com/d4l3k

Pull Request resolved: #179286
Approved by: https://github.com/malfet, https://github.com/Skylion007, https://github.com/cyyever

Keep the single-proxy behavior unchanged. For nested proxy pytrees, route each symbol binding along its full pytree key path to the owning Proxy and store the remaining local path on that node. This preserves per-placeholder provenance instead of suppressing fresh unbacked tracking.

Fixes #187230

This PR was authored with assistance from an AI assistant.

Test Plan:

python -m pytest test/test_proxy_tensor.py -q -s -k "test_unbacked_bindings_on_multiple_token_grid_placeholders or unbacked or non_deduped_shape or deduped_shape or non_symint_size_spec"

lintrunner -a

stack-info: PR: #187231, branch: sanketpurandare/stack/19

Introduce the torchcomms _single collective names on the C++ c10d::Backend and migrate the in-tree backends to define them, while keeping the old names fully working for backward compatibility.

Backend now declares all_gather_single, all_gather_single_coalesced, reduce_scatter_single, reduce_scatter_single_coalesced, and all_to_all_single alongside the existing _allgather_base, allgather_into_tensor_coalesced, _reduce_scatter_base, reduce_scatter_tensor_coalesced, and alltoall_base, which are kept as overridable, forwarding aliases.

Backward compatibility is preserved in both directions: each new method and its old-name alias forward to each other, so a Backend subclass may override EITHER name and a caller may invoke EITHER name. The old aliases simply forward to the canonical _single method; each canonical method opens with the C10D_BACKEND_FORWARDING_GUARD() macro, which installs a function-local thread-local re-entry flag using __func__ for the message. If a backend overrides neither name the mutual forwarding re-enters the canonical method, the flag trips, and it reports "does not support " instead of recursing forever. As a result, existing callers of the old names (first-party and out-of-tree) and existing out-of-tree backends that override the old names both keep working unchanged. Removing the old overrides outright would have broken out-of-tree backends on the dispatch path, and dropping the old caller-facing names would have broken direct callers; the bidirectional forwarding avoids both.

The old names are intentionally NOT marked C10_DEPRECATED_MESSAGE in this change. PyTorch's open-source build compiles the bundled third_party/torch-xpu-ops submodule with -Werror -Wdeprecated, and its XPU collective op registration (xccl/Register.cpp) still calls the old Backend names, so a compile-time deprecation turns into a hard build error there. The deprecation will be reintroduced in a follow-up once those callers (and any other out-of-tree backends that are built in-tree) are migrated to the _single names and the submodule pin is bumped.

The in-tree backends are migrated to define the new names: ProcessGroupNCCL, ProcessGroupGloo, ProcessGroupMPI, ProcessGroupUCC, ProcessGroupWrapper, FakeProcessGroup, NCCLXStub, and the torch_openreg (ProcessGroupOCCL) test backend. The dispatcher op implementations in Ops.cpp, ProcessGroupWrapper's forwarders, and the Backend pybind bindings in init.cpp use the new names.

The underlying aten / c10d dispatcher op names (c10d::_allgather_base_, c10d::alltoall_base_, etc.), their schema strings, the IMPL_* macro names in Ops.cpp, and the OpType enum values are intentionally left unchanged for backward compatibility.

Suggested review order: Backend.hpp (the new/old method pairs and the bidirectional forwarding + ForwardingGuard), then the per-backend subclass renames, then the Ops.cpp / ProcessGroupWrapper / init.cpp call sites.

This diff was authored with the assistance of an AI coding agent (Claude).

Test Plan:
This revision only removes the compile-time deprecation annotations (C10_DEPRECATED_MESSAGE), their -Wdeprecated-declarations suppressions, and the now-unused <c10/util/Deprecated.h> include. The _single rename and the bidirectional forwarding/recursion guard are unchanged, so there is no runtime or codegen change -- dropping the annotations only removes deprecation warnings.

Verified by open-source CI on the exported commit: the full libtorch build across platforms, the linux-noble-xpu-n-py3.10 / build-osdc job (which compiles the bundled torch-xpu-ops xccl/Register.cpp with -Werror -Wdeprecated and previously failed on -Werror=deprecated-declarations), and the lintrunner-clang format jobs.

Reviewed By: kapilsh

Differential Revision: D108364288

Test Plan: Comment/string-only edits; no code behavior affected.

Authored with assistance from an AI assistant.
Pull Request resolved: #187076
Approved by: https://github.com/d4l3k

Summary

Fixes #186237
Fixed a division-by-zero in MultiheadAttentionImpl::reset() triggered when num_heads is 0.

Previously, head_dim was computed before any validation of num_heads:

head_dim = options.embed_dim() / options.num_heads();

Constructing MultiheadAttention with num_heads = 0 could invoke undefined behavior (integer division by zero / floating-point exception) before any validation logic was reached.

This PR adds a TORCH_CHECK to validate that num_heads > 0 before performing the division, replacing the silent UB with a clear, actionable error message.

Changes

• Added a TORCH_CHECK(options.num_heads() > 0, ...) guard in MultiheadAttentionImpl::reset() prior to the head_dim computation.

Test Plan

Added a C++ API test verifying that constructing:

MultiheadAttention(MultiheadAttentionOptions(0, 0))

throws a c10::Error.

Before this change: the test triggered a floating-point exception due to division by zero.

After this change: the test passes and the invalid configuration is reported via a proper error.
Pull Request resolved: #186376
Approved by: https://github.com/drisspg

Complex types are handled via the conjugated derivative (grad * conj((1 - output) * output)), matching CUDA's behavior.

Microbenchmark (us per call, A/B from the same script with only the impl
swapped):

shape|dtype                      mpsgraph     metal   speedup
1024|float32                        12.81      7.43     1.72x
1024|float16                        14.82      7.10     2.09x
1024|bfloat16                       12.11      7.15     1.69x
1024x1024|float16                   18.56     14.67     1.26x
1024x1024|float32                   45.02     40.06     1.12x
8x1024x1024|float32                382.21    376.67     1.01x
32x1024x1024|float32              1487.84   1493.75     1.00x

Metal wins on dispatch overhead at small sizes; parity at large sizes where
the op is memory-bandwidth bound, as expected for ~3 FLOPs/element.

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
Pull Request resolved: #187151
Approved by: https://github.com/Skylion007

For to_local() backward, AOTAutograd can produce a local gradient stride like
(Max(1, u3), 1) while the saved DTensor metadata uses (u1, 1). The previous
backward path recomputed the global gradient stride before deciding whether it
could reuse the original DTensor spec. That forced
compute_global_tensor_info() to evaluate symbolic stride relations and could
raise a data-dependent guard for the default same-placement backward path.

Reuse the original DTensor spec only for the default same-placement backward
when the local gradient stride, saved forward local stride, and DTensor spec
stride are compatible. Exact/provable stride equality is accepted directly. For
contiguous symbolic stride forms such as Max(1, u*), use the existing
check_contiguous_sizes_strides(..., false_if_dde=True) helper so equivalent
contiguous layouts are recognized without requiring a brittle exact symbolic
match. If neither exact nor contiguous equivalence can be proven, emit
torch._check assertions for the required stride equalities before taking the
symbolic shortcut.

If the placement changes or the physical local stride cannot justify the
original spec layout, keep the existing recomputation path and build a fresh
spec from the observed gradient stride. This avoids the symbolic guard failure
without lying about memory layout. In particular, uneven channels-last shards
must keep using the recomputation path so autograd can repair the physical local
gradient layout correctly.

The same class of issue also appears in aten.t sharding propagation. The
single-dim strategy can enumerate candidate placements that move a symbolic
_StridedShard split factor onto the other tensor dimension. When that
candidate is not provably shardable, strategy expansion should reject it instead
of evaluating a Python bool on an unbacked expression such as 8 < 2*u0.
Register transpose with allow_unbacked_sharding=False so unproven candidates
are pruned while statically valid candidates, including _StridedShard(0, u0)
propagating to _StridedShard(1, u0), are still kept.

Fixes #187025

This PR was authored with assistance from an AI assistant.

Test Plan:

python -m pytest test/distributed/tensor/test_op_strategy.py::TestCostModel::test_t_prunes_unproven_unbacked_strided_shard_candidates test/distributed/tensor/test_op_strategy.py::TestCostModel::test_mm_strategies test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_to_local_backward_unbacked_symbolic_stride test/distributed/tensor/test_tensor_ops.py::TestNewEmptyStridedUneven::test_backward_channels_last -q -s

lintrunner -a

Pull Request resolved: #187026
Approved by: https://github.com/pianpwk

Unify the host-side dispatch so a single templated launcher drives every Metal upsample kernel and UpsampleParams supplies the geometry for all ranks, collapsing the previously duplicated per-rank dispatch overloads.

Backward is intentionally left on MPSGraph: it has no correctness bug and its fused gradient kernel is several times faster than a naive Metal gather.

Co-authored-by: Claude noreply@anthropic.com
Pull Request resolved: #186989
Approved by: https://github.com/Skylion007, https://github.com/kurtamohler

Global reduction (torch.median(x) / torch.nanmedian(x))

Reduction along a dim (torch.median(x, dim))

Strided / sliced layouts

nanmedian with NaNs

Pull Request resolved: #187060
Approved by: https://github.com/malfet

Pull Request resolved: #187067
Approved by: https://github.com/pianpwk, https://github.com/atalman, https://github.com/fegin

Also add TODO(#184490) in mm_plus_mm.py to mark the XPU guard as temporary
— it should be lifted once the underlying Triton codegen accuracy bug is
fixed so XPU can benefit from the fused template perf.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

The fallback stays for performance. Tests reverted accordingly:

• test_torchinductor_codegen_dynamic_shapes: keep TestFailure entries
  (no Triton kernel generated while fallback is active)
• test_torchinductor_opinfo: keep inductor_one_sample skip

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Removes the @skipIfXpu decorator from CudaReproTests::test_float16_reduction_with_int_output in test/inductor/test_cuda_repro.py. The skip referenced intel/torch-xpu-ops#3006, where Triton codegen for argmax over float16 inputs was emitting a spurious .to(tl.float16) downcast on the int64 index result. That bug was fixed upstream by #174321 ([inductor] Fix incorrect Triton reduction output dtype), which is already in main, so the XPU skip is no longer needed.

Verification

Verified on Intel Data Center GPU Max 1550 (PVC) with torch==2.13.0.dev20260525+xpu:

source /home/gta/intel/oneapi/setvars.sh
export LD_PRELOAD=/home/gta/intel/oneapi/ccl/2022.0/lib/libccl.so.1
export PYTORCH_TEST_WITH_SLOW=1
pytest -v test/inductor/test_cuda_repro.py \
  -k test_float16_reduction_with_int_output -rs

Result:

test/inductor/test_cuda_repro.py::CudaReproTests::test_float16_reduction_with_int_output PASSED [16.3236s] [100%]
====================== 1 passed, 110 deselected in 26.62s ======================

Fixes intel/torch-xpu-ops#3006.

Authored by Claude (opencode).

Pull Request resolved: #185199
Approved by: https://github.com/zou3519

Reorganizes torch/csrc/distributed/c10d by moving non-public, backend-specific implementation files and the TCPStore backend files into per-backend subfolders, while leaving the public-facing classes at the top level (the ProcessGroupGloo/NCCL/MPI/UCC backends and the Store/TCPStore/FileStore/HashStore/PrefixStore classes all stay put).

The moves are: store/ gets TCPStoreBackend.{cpp,hpp} and TCPStoreLibUvBackend.cpp; gloo/ gets ProcessGroupGlooCuda.cpp, ProcessGroupGlooDetail.hpp, and GlooDeviceFactory.{cpp,hpp}; ucc/ gets UCCTracing.{cpp,hpp} and UCCUtils.{cpp,hpp}; nccl/ gets NCCLXStub.hpp.

NCCLUtils.{cpp,hpp} was deliberately kept at the top level even though it is backend-specific: it is included by several call sites outside caffe2 (in gen_ai, ads_mkl, and fbgemm_gpu), so relocating it would be a wider, riskier change better done on its own. As a result the new nccl/ folder currently holds only NCCLXStub.hpp.

All include sites were updated, covering both the canonical torch/csrc/distributed/c10d/... include form and the legacy short c10d/... form (used by fb/GlooDeviceFactory.cpp). Build wiring was updated in build_variables.bzl -- the canonical source list consumed by CMake (via append_filelist in cmake/Codegen.cmake), OSS Bazel, and OSS Buck -- and in the internal fb/fbcode/target_definitions.bzl for ProcessGroupGlooCuda.cpp. Headers are picked up by recursive globs, so no header-list edits were needed.

This is a pure file move: contents are unchanged apart from the relocated #include paths, so correctness is established by a clean build rather than by behavioral tests.

Authored with the assistance of an AI coding assistant (Claude Code).

Test Plan:
Confirmed no references to the old paths remain anywhere in fbcode, then ran the fbcode lint and build tooling:

arc f
arc lint
arc lint --take AUTODEPS --apply-patches
buck2 build fbcode//caffe2:_libtorch fbcode//caffe2:_libtorch_cuda

arc f and arc lint reported no issues; AUTODEPS produced no dependency changes (the moves stayed within existing Buck targets); both the CPU (_libtorch) and CUDA (_libtorch_cuda) libraries built successfully (exit 0).

Reviewed By: kapilsh

Differential Revision: D108332288

The previous assertion assumed any non-symbol replacement with free symbols was
invalid. That is too strong for legitimate derived unbacked shapes and makes the
binding logic reject a value it can represent with the existing ShapeEnv
replacement mechanism. Record the replacement with _eliminate_unbacked, which
is the existing path for replacing an unbacked symbol by a non-symbol expression.

This intentionally does not restore the reverted broad HOP fake-trace
suppression. The FlexAttention/HOP reproducer for #183677 now passes on current
main without that suppression, and the broad suppression was the source of the
internal cond, AOTInductor, Executorch, and FlexAttention regressions.

This change was authored with assistance from an AI assistant.

Test Plan:

python test/test_dynamic_shapes.py TestUnbacked.test_rebind_unbacked_to_symbolic_expression -v

python test/dynamo/test_misc.py MiscTests.test_cond_runtime_assert_generation -v

python test/functorch/test_control_flow.py TestControlFlowTraced.test_cond_functionalized_nested -v

python test/inductor/test_aot_inductor.py -k cond_non_tensor_predicates_dynamic_True_cpu -v

python - <<'PY'
import torch
from torch.nn.attention.flex_attention import create_block_mask, flex_attention

if not torch.cuda.is_available():
    raise RuntimeError('CUDA unavailable')

device = 'cuda'
dtype = torch.float16

def score_mod(score, batch, head, q_idx, kv_idx):
    return score

compiled_create_block_mask = torch.compile(create_block_mask, dynamic=True, fullgraph=True)

def create_dynamic_block_mask(q_batch, kv_batch):
    q_len = q_batch.size(0)
    kv_len = kv_batch.size(0)
    def mask_mod(batch, head, q_idx, kv_idx):
        q_group = q_batch[q_idx]
        kv_group = kv_batch[kv_idx]
        return (q_group == kv_group) & (q_group != -1) & (kv_group != -1)
    return compiled_create_block_mask(mask_mod, B=None, H=None, Q_LEN=q_len, KV_LEN=kv_len, device=device, BLOCK_SIZE=128)

groups = torch.zeros(128, dtype=torch.int64, device=device)
block_mask = create_dynamic_block_mask(groups, groups)
q = torch.randn(1, 1, 128, 64, device=device, dtype=dtype, requires_grad=True)
k = torch.randn(1, 1, 128, 64, device=device, dtype=dtype, requires_grad=True)
v = torch.randn(1, 1, 128, 64, device=device, dtype=dtype, requires_grad=True)
compiled_flex_attention = torch.compile(flex_attention, fullgraph=True, dynamic=True, backend='aot_eager')
out = compiled_flex_attention(q, k, v, score_mod=score_mod, block_mask=block_mask)
out.sum().backward()
torch.cuda.synchronize()
print('ok', out.shape)
PY

lintrunner -a

stack-info: PR: #183837, branch: sanketpurandare/stack/11

Reverted #187067 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable (comment)

Summary
Fix a bug in the TMA compatibility check that prevented backends from falling back to BlockPtrOptions, causing unwanted scalar-pointer fallbacks and PE timeouts.

Problem
The TMA compatibility check at triton.py:3595 used issubclass(options_class, TensorDescriptorOptions) on the queried class instead of the returned options. When backends like MTIA override TensorDescriptorOptions.create to fall back to BlockPtrOptions (for shapes incompatible with hardware layout), the check incorrectly fired anyway, rejecting block params and triggering Triton's scalar-pointer fallback.

Solution
Changed to isinstance(options, TensorDescriptorOptions) to check the actual returned type. Now when a backend's create returns BlockPtrOptions, the TMA check is correctly skipped, allowing the block pointer to flow to codegen_block_ptr and backends to register autotune metadata (e.g., MTIA's inner_blocks for bump-to-minimum heuristic).

Impact

• Fixes PE timeouts caused by unwanted scalar-pointer fallbacks
• Enables backends to properly use their custom autotune heuristics
• Small change (1 line in 2 files) with targeted fix

Test Plan:

• Ran buck2 test fbcode//mode/opt fbcode//mtia/compiler/graph_compiler/inductor/test:test_inductor_kernels -- --regex 'test_bump_block_size'
• Before fix: XBLOCK=2 for int8/float16/float32 (heuristic skipped because inner_blocks empty)
• After fix: XBLOCK bumped to 32/16/8 per dtype, matching the 32-byte hardware floor
• TODO: Paste full test output showing the XBLOCK values before/after

Differential Revision: D106090810

Pull Request resolved: #185983
Approved by: https://github.com/blaine-rister

Groundwork PR

1. Adds the hop
2. Adds the quack impl
3. Only supports pointwise
4. no aux buffers
5. All this to make follow up prs easier to review

sanity check

Notes for reviewer

API;

# Do be made public later
 from torch._higher_order_ops import flex_gemm

 out = flex_gemm(
     gemm_op,          # torch.mm / torch.addmm / aten op overload expanded to a wider set scaled_mm, grouped friends
     gemm_args,        # tuple of tensor/scalar operands orignal args ot he base func
     epilogue_fn,      # Python fn over accumulator/result
     *,
     gemm_kwargs=None, # non-tensor kwargs only
     kernel_options={},  # dict of options, will be backend, fastmath, tune
 )

Pull Request resolved: #186483
Approved by: https://github.com/mlazos, https://github.com/eellison
ghstack dependencies: #186944

Pointed quack at my upstream fork, pointing back to main and using patchsets as intended

Pull Request resolved: #186944
Approved by: https://github.com/slayton58

For swizzled scale layouts, defer to the eager _scaled_mm_v2 op before mm_args. The trivial-swizzle path is unchanged.

Authored with Claude Opus 4.8.

rel: 9eb8bcb

Pull Request resolved: #186384
Approved by: https://github.com/drisspg

Authored with assistance from Claude Code.

Pull Request resolved: #186499
Approved by: https://github.com/rtimpe
ghstack dependencies: #185998, #185999, #186042

The fix handles copy in two places. DefaultDictVariable.call_method now
treats "copy" alongside "copy", returning a fresh defaultdict that
preserves default_factory and performs a shallow copy of the base dict.
UserDefinedClassVariable.call_method gains a narrow branch that dispatches
defaultdict.copy(inst) (one positional arg, no kwargs) to the instance,
mirroring the existing OrderedDict.move_to_end pattern.

Test Plan:

CUDA_VISIBLE_DEVICES= PYTORCH_TESTING_DEVICE_ONLY_FOR=cpu PYTORCH_TEST_WITH_DYNAMO=1 \
  pixi run -w pytorch -e pytorch313 python -m pytest test/cpython/v3_13/test_defaultdict.py -q

CUDA_VISIBLE_DEVICES= PYTORCH_TESTING_DEVICE_ONLY_FOR=cpu \
  pixi run -w pytorch -e pytorch313 python -m pytest test/dynamo/test_dicts.py -q

Authored with the help of an AI assistant.

Pull Request resolved: #186758
Approved by: https://github.com/rtimpe
ghstack dependencies: #185998, #185999, #186042, #186499

Factor the cpp-wrapper int-array formatting into cpp_int_array_str() and reuse it in both the existing assert-size-stride helper and the repro test. This keeps the expectation tied to the platform-specific generated C++ spelling instead of hard-coding Linux formatting in the Windows test path.

The attached convolution_backward C-shim failures from the issue are already addressed on current main by 1b421fae114 [inductor] Fix MSVC const pointer emission in cpp wrapper temporary arrays (#179846), so this change is limited to the remaining Windows-sensitive FileCheck expectation.

Fixes #162366
Generated by my agent

Test Plan:

• TORCHINDUCTOR_CPP_WRAPPER=1 python - <<'PY'
  import runpy
  import sys
  sys.modules['torchvision'] = None
  sys.path.insert(0, 'test/inductor')
  sys.argv = ['test/inductor/test_cpu_repro.py', 'CPUReproTests.test_require_stride_order_non_owning']
  runpy.run_path('test/inductor/test_cpu_repro.py', run_name='main')
  PY
• python - <<'PY'
  import sys
  sys.modules['torchvision'] = None
  sys.path.insert(0, 'test/inductor')
  import test_torchinductor
  orig = sys.platform
  try:
  sys.platform = 'win32'
  assert test_torchinductor.cpp_int_array_str([2, 3, 4, 4]) == '{2LL, 3LL, 4LL, 4LL}'
  sys.platform = 'linux'
  assert test_torchinductor.cpp_int_array_str([2, 3, 4, 4]) == '{2L, 3L, 4L, 4L}'
  finally:
  sys.platform = orig
  print('cpp_int_array_str platform formatting OK')
  PY
• TORCHINDUCTOR_CPP_WRAPPER=1 python - <<'PY'
  import runpy
  import sys
  sys.modules['torchvision'] = None
  sys.path.insert(0, 'test/inductor')
  sys.argv = ['test/inductor/test_torchinductor.py', 'CpuTests.test_conv_backward_cpu']
  runpy.run_path('test/inductor/test_torchinductor.py', run_name='main')
  PY
• TORCHINDUCTOR_CPP_WRAPPER=1 python - <<'PY'
  import runpy
  import sys
  sys.modules['torchvision'] = None
  sys.path.insert(0, 'test/inductor')
  sys.argv = ['test/inductor/test_torchinductor.py', 'CpuTests.test_conv2d_backward_channels_last_cpu']
  runpy.run_path('test/inductor/test_torchinductor.py', run_name='main')
  PY
• python - <<'PY'
  import runpy
  import sys
  sys.modules['torchvision'] = None
  sys.path.insert(0, 'test/inductor')
  sys.argv = ['test/inductor/test_cpu_repro.py', 'CPUReproTests.test_require_stride_order_non_owning']
  runpy.run_path('test/inductor/test_cpu_repro.py', run_name='main')
  PY
• lintrunner -a

Pull Request resolved: #185145
Approved by: https://github.com/desertfire

Test Cases

1. pytest test/distributed/tensor/test_matrix_ops.py
2. pytest test/distributed/tensor/test_op_strategy.py -k test_redistribute_cost_latency
3. pytest test/distributed/tensor/test_op_strategy.py -k test_redistribute_cost_mesh_2d
4. pytest test/distributed/tensor/test_op_strategy.py -k test_bmm_strategies

Pull Request resolved: #186667
Approved by: https://github.com/pianpwk, https://github.com/weifengpy

1. Add setUpModule to make sure Kineto is created with CUDA support. #186036 recently changed the behavior to not initialize Kineto with GPU tracing if only CPU activities are requested, which is more correct behavior. However, this changes the behavior of the following code:

with profile(activities=[ProfilerActivity.CPU]) as prof:
   ...

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
   ...

Now Kineto is armed to only do CPU tracing. The issue is that Kineto is only registered once, so even though you added ProfilerActivity.CUDA in the second pass, the profiler still latches on to the CPU-only session, leading to no device-side activities being recorded. We need to find a more complete solution but to unblock the tests we can hack around it by forcing Kineto to instantiate with GPU-tracing before running any tests.

2. Clear KinetoStepTracker to make sure repeat runs with different activity types don't pollute test_kineto_profiler_multiple_steppers.
3. Make sure test_forked_process explicitly runs with CPU activities only.

There are a few more failures due to test pollution, but I haven't tracked those down yet.
Pull Request resolved: #186970
Approved by: https://github.com/scotts

Root cause: the unified inline manywheel build job (cpu/cpu-aarch64/
cuda/cuda-aarch64) checks out the raw commit via ref: github.sha with
fetch-depth: 2. Checking out a bare SHA does not fetch the tag ref, so
tagged_version() in .ci/pytorch/binary_populate_env.sh
(git describe --tags --exact) fails and the build falls through to the
<version>.dev<DATE> default. binary_populate_env.sh then takes its
.*dev.* branch and appends +git<triton-shorthash> to the triton
requirement that gets baked into the wheel metadata.

Fix: add fetch-tags: true to the inline build job checkout in the
linux binary build template so the tag pointing at the checked-out
commit is fetched, letting git describe --tags --exact succeed and the
build resolve to the release version. The ROCm/XPU/s390x builds use the
reusable _binary-build-linux.yml whose checkout has no pinned ref (it
uses the default tag ref for a tag push) and are unaffected.

This is the same class of bug fixed for the older checkout-pytorch path
in #180508; this covers the inlined unified build job introduced since.

Test Plan:

Regenerated the workflows from the template and confirmed only the two
linux manywheel workflows change, each gaining fetch-tags: true on its
four unified build jobs:

python3 .github/scripts/generate_ci_workflows.py
git diff --stat .github/workflows
grep -c 'fetch-tags: true' \
  .github/workflows/generated-linux-binary-manywheel-nightly.yml \
  .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml

Validated the regenerated YAML parses and passes actionlint (the only
reported shellcheck warnings are pre-existing on the base files):

python3 -c "import yaml; yaml.safe_load(open(f)) for f in (...)"
lintrunner --take ACTIONLINT \
  .github/workflows/generated-linux-binary-manywheel-nightly.yml \
  .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml

This PR was authored with the assistance of an AI coding assistant.

Power is ternary, so the modulus-position slot (CPython's z-slot) gets
its own nb_power_z_impl method to avoid conflating it with the
forward/reflected roles that reverse: bool captures. ternary_op
dispatches the z-slot through nb_power_z_impl(self=z, tx, v, w).

Co-authored-by: Claude Sonnet

Pull Request resolved: #186296
Approved by: https://github.com/rtimpe

Pull Request resolved: #185553
Approved by: https://github.com/Skylion007

Allow output-input aliasing in torch.cond branches.

torch.cond branches are mutually exclusive — only one branch executes at runtime. When the true branch returns an operand directly (e.g., identity/skip path), the output aliases the input. This is safe because the non-taken branch never runs, so there is no concurrent access to aliased buffers.

Previously, two checks rejected this pattern:

1. Dynamo (higher_order_ops.py): CondHigherOrderVariable.supports_aliasing = False rejected aliasing during tracing with a graph break.
2. Functionalization (cond.py): _check_alias_and_mutation rejected aliasing during AOT functionalization.

Note: Dynamo already allowed aliasing in inference (supports_aliasing = not torch.is_grad_enabled()), but torch.export traces with grad enabled even for inference models, so the check was ineffective for the AOTI export path.

Fix:

• Set supports_aliasing = True unconditionally on CondHigherOrderVariable. Aliasing is safe for cond regardless of grad context because branches are exclusive — even with autograd, only the taken branch's gradient flows.
• Skip the alias check in the functionalization impl (keep mutation check).

Motivation: PrismNet's _clone_layer_outputs uses lo.clone() to break output-input aliasing required by torch.cond. Removing this constraint saves ~0.046ms per inference (2 DHEN layers at B=2048).

Adds test_cond_output_input_aliasing regression test.

Test Plan:

buck2 test fbcode//mode/opt -c 'fbcode.nvcc_arch=a100,h100' fbcode//caffe2/test/inductor:control_flow -- --exact 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_device_cpu (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_device_cuda (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_multi_output_device_cpu (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_multi_output_device_cuda (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_both_branches_device_cpu (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_both_branches_device_cuda (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_input_mutation_still_rejected (caffe2.test.inductor.test_control_flow.CondTests)'

Test session: https://www.internalfb.com/intern/testinfra/testrun/10414574313882076

PrismNet repro — all 11 tests pass, including CondOutputInputAlias: without clone which previously failed with "Replace return input with return input.clone()". Confirms lo.clone() workaround in _clone_layer_outputs can be removed.

buck2 run fbcode//mode/opt fbcode//aps_models/ads/gmp/models/ads_mtml_prism_net_dedicated_model/experimental/benchmarks:repro_aoti_inductor_bugs

Reviewed By: kqfu, atalman

Differential Revision: D106106380

Allow output-input aliasing in torch.cond branches.

torch.cond branches are mutually exclusive — only one branch executes at runtime. When the true branch returns an operand directly (e.g., identity/skip path), the output aliases the input. This is safe because the non-taken branch never runs, so there is no concurrent access to aliased buffers.

Previously, two checks rejected this pattern:

1. Dynamo (higher_order_ops.py): CondHigherOrderVariable.supports_aliasing = False rejected aliasing during tracing with a graph break.
2. Functionalization (cond.py): _check_alias_and_mutation rejected aliasing during AOT functionalization.

Note: Dynamo already allowed aliasing in inference (supports_aliasing = not torch.is_grad_enabled()), but torch.export traces with grad enabled even for inference models, so the check was ineffective for the AOTI export path.

Fix:

• Set supports_aliasing = True unconditionally on CondHigherOrderVariable. Aliasing is safe for cond regardless of grad context because branches are exclusive — even with autograd, only the taken branch's gradient flows.
• Skip the alias check in the functionalization impl (keep mutation check).

Motivation: PrismNet's _clone_layer_outputs uses lo.clone() to break output-input aliasing required by torch.cond. Removing this constraint saves ~0.046ms per inference (2 DHEN layers at B=2048).

Adds test_cond_output_input_aliasing regression test.

Test Plan:

buck2 test fbcode//mode/opt -c 'fbcode.nvcc_arch=a100,h100' fbcode//caffe2/test/inductor:control_flow -- --exact 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_device_cpu (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_device_cuda (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_multi_output_device_cpu (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_multi_output_device_cuda (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_both_branches_device_cpu (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_output_input_aliasing_both_branches_device_cuda (caffe2.test.inductor.test_control_flow.CondTests)' 'fbcode//caffe2/test/inductor:control_flow - test_cond_input_mutation_still_rejected (caffe2.test.inductor.test_control_flow.CondTests)'

Test session: https://www.internalfb.com/intern/testinfra/testrun/10414574313882076

PrismNet repro — all 11 tests pass, including CondOutputInputAlias: without clone which previously failed with "Replace return input with return input.clone()". Confirms lo.clone() workaround in _clone_layer_outputs can be removed.

buck2 run fbcode//mode/opt fbcode//aps_models/ads/gmp/models/ads_mtml_prism_net_dedicated_model/experimental/benchmarks:repro_aoti_inductor_bugs

Reviewed By: kqfu, atalman

Differential Revision: D106106380

Propagate the bool-mask IoU options through the object and numpy recursive same() paths, enable the existing IoU comparator for the two affected Detectron2 Mask R-CNN models, and update the CPU freezing expected-accuracy files to pass.

Fixes #127073

Generated by my agent

Test Plan:

python test/dynamo/test_utils.py -k TestUtils.test_same_iou_for_bool_propagates_to_instances

python -m unittest benchmarks.dynamo.test.TestDynamoBenchmark.test_detectron2_maskrcnn_uses_iou_for_bool_masks

Local Detectron2 benchmark workaround for detectron2_maskrcnn_r_101_fpn and detectron2_maskrcnn_r_50_c4 with --accuracy --inference --device cpu --inductor --freezing reported pass

git diff --check

lintrunner -a (fails on unrelated existing Pyrefly error in torch/autograd/graph.py for torch._C._remove_obj_from_tls)

Pull Request resolved: #186441
Approved by: https://github.com/eellison, https://github.com/mlazos

Release-only changes for the release/2.13 branch cut, produced by
running scripts/release/apply-release-changes.sh. The script repoints
reusable workflows and composite actions from @main to @release/2.13,
rewrites templates to release/2.13 (with checkout_pr_head=False so PRs
build the merge base rather than the PR head), pins the XLA checkout to
the r2.13 branch, pins the disabled/unstable jobs and disabled-tests S3
JSON blobs to fixed versionIds, sets RELEASE_VERSION_TAG=2.13 for the
workflow-regeneration lint check, and drops the pull_request-specific
checkout ref from the linux binary build/test workflows.

Only files that are tracked in release/2.13 are included. The 2.12 PR
additionally touched .ci/manywheel/build_cuda.sh and a few workflow
files (torchbench/nitpicker/quantization-periodic) that are not tracked
in this checkout, so they are intentionally omitted here.

Test Plan:
Ran the release script and linter:

DRY_RUN=disabled ./scripts/release/apply-release-changes.sh
lintrunner -a

lintrunner reported only pre-existing ACTIONLINT shellcheck warnings on
generated workflow lines unrelated to the release-version edits, and made
no changes to the staged files. Verified every staged hunk is a
release-only edit (@main -> @release/2.13, main -> release/2.13, XLA
r2.13 pin, S3 versionId pins, RELEASE_VERSION_TAG=2.13, and the
pull_request ref removal) with no submodule or unrelated changes.

This PR was authored with the assistance of Claude Code.

• [release 2.13] Pin Linux manywheel builder docker images

Release builds should use a fixed, reproducible build toolchain instead
of the floating builder image tags that main tracks (e.g.
pytorch/manylinux2_28-builder:cuda12.6). For 2.12 the binary build
workflows resolved the image dynamically via calculate-docker-image; for
2.13 we freeze that resolved image as a literal pin.

The pin is applied in the generator (generate_binary_build_matrix.py)
rather than in the generated YAML directly, so re-running
.github/regenerate.sh (and the lint job that asserts the generated files
are up to date) reproduces the pinned tags. wheel_container_image_tag_prefix()
appends the pin only for the linux manywheel OSes (linux, linux-aarch64,
linux-s390x), since only those builds run inside these containers;
windows and macos keep the plain tag prefix.

The pin suffix is the .ci/docker tree hash, f38ba0b10220982e39441d29d203d803a2b56c92
(git rev-parse HEAD:.ci/docker), which is exactly the tag that
.github/actions/binary-docker-build (and the s390x equivalent) publish
as ${prefix}-${CI_FOLDER_SHA}. It matches the image already published on
Docker Hub, e.g.
pytorch/manylinux2_28_aarch64-builder:cpu-aarch64-f38ba0b10220982e39441d29d203d803a2b56c92

Test Plan:
Regenerated the workflows in release mode and verified the result:

RELEASE_VERSION_TAG=2.13 python3 .github/scripts/generate_ci_workflows.py

• Every image: and matrix docker_image_tag_prefix in the three linux
  manywheel generated workflows now carries the -f38ba0b... suffix
  (cpu, cpu-aarch64, cuda12.6/13.0/13.2, rocm7.1/7.2, xpu, cpu-s390x);
  no linux builder image remains floating.
• Windows and macos generated workflows are unchanged (plain cpu,
  cuda12.6, ...), confirming the pin is linux-only.
• Re-running the generator a second time produced byte-identical output
  (md5sum unchanged), so the regeneration/lint up-to-date check is
  stable.

Note: the local linters that shell out to uv could not run in this
environment; the change was format-checked manually.

This PR was authored with the assistance of Claude Code.

Fresh replacement for #70979 based on current upstream/main to avoid stale/conflicting old PR state.

Pull Request resolved: #186817
Approved by: https://github.com/zou3519

Use the dispatcher-provided args and kwargs directly in the hot path, and skip
omitted mutated arguments instead of materializing or incrementing their default
values. Default values for mutated arguments are expected to be non-Tensors or
fresh values, so there is no useful version counter to bump when the caller
omitted them.

This substantially reduces the overhead reported in #139494, but does not claim
to fully close the remaining gap between mutable and non-mutating custom ops.

The alternative was to optimize utils.fill_defaults itself, but that helper is
general schema normalization and the mutable version-bump path only needs the
small subset of mutated arguments that were actually provided.

Generated by my agent

Benchmark Results:
Before, issue benchmark on CUDA H100, 1000 calls per do_bench function:

• mutate2 = 18.27797737121582
• no_mutate2 = 13.457284654889788
• mutate = 66.9056625366211
• no_mutate = 17.066380310058594

After:

• mutate2 = 16.13766403198242
• no_mutate2 = 13.513686997549874
• mutate = 21.02779197692871
• no_mutate = 17.362232971191407

Test Plan:

• python test/test_custom_ops.py TestCustomOpAPI.test_mutated_version_bump_does_not_fill_all_defaults TestCustomOpAPI.test_mutated_optional_arg_default_none TestCustomOpAPI.test_mutated TestCustomOpAPI.test_custom_op_out_tag
• lintrunner -a
  Pull Request resolved: #186175
  Approved by: https://github.com/zou3519

Propagate the bool-mask IoU options through the object and numpy recursive same() paths, enable the existing IoU comparator for the two affected Detectron2 Mask R-CNN models, and update the CPU freezing expected-accuracy files to pass.

Fixes #127073

Generated by my agent

Test Plan:

python test/dynamo/test_utils.py -k TestUtils.test_same_iou_for_bool_propagates_to_instances

python -m unittest benchmarks.dynamo.test.TestDynamoBenchmark.test_detectron2_maskrcnn_uses_iou_for_bool_masks

Local Detectron2 benchmark workaround for detectron2_maskrcnn_r_101_fpn and detectron2_maskrcnn_r_50_c4 with --accuracy --inference --device cpu --inductor --freezing reported pass

git diff --check

lintrunner -a (fails on unrelated existing Pyrefly error in torch/autograd/graph.py for torch._C._remove_obj_from_tls)

Pull Request resolved: #186441
Approved by: https://github.com/eellison, https://github.com/mlazos

  File "/opt/pytorch/pytorch/test/functorch/test_vmap.py", line 6579, in <module>
    instantiate_device_type_tests(TestVmapOperatorsOpInfo, globals(), only_for=only_for)
  File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/common_device_type.py", line 1074, in instantiate_device_type_tests
    device_type_test_class.instantiate_test(
  File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/common_device_type.py", line 634, in instantiate_test
    instantiate_test_helper(
  File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/common_device_type.py", line 540, in instantiate_test_helper
    test = decorator(test)
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/opinfo/core.py", line 1713, in __call__
    raise RuntimeError("Multiple sets of sample_skips_and_xfails defined")
RuntimeError: Multiple sets of sample_skips_and_xfails defined
================================================================================== short test summary info ===================================================================================
ERROR ../opt/pytorch/pytorch/test/functorch/test_vmap.py - RuntimeError: Multiple sets of sample_skips_and_xfails defined
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================== 1 error in 3.91s ======================================================================================

due to test_vmap_exhaustive having multiple decorators skipIf and xfailIf from test/functorch/common_utils.py. Both of these rely on sample_skips_and_xfails:
https://github.com/pytorch/pytorch/blob/d3ce23d75ee5e488787aafb12c281b5142d91e75/test/functorch/common_utils.py#L481-L497

https://github.com/pytorch/pytorch/blob/d3ce23d75ee5e488787aafb12c281b5142d91e75/test/functorch/common_utils.py#L510-L526

Though, using a mix of these on a certain test end up in the RuntimeError:
https://github.com/pytorch/pytorch/blob/d3ce23d75ee5e488787aafb12c281b5142d91e75/torch/testing/_internal/opinfo/core.py#L1713

This PR proposes appending the rules, as this simplifies the usage of sample_skips_and_xfails.

Fixes #184894
Pull Request resolved: #185905
Approved by: https://github.com/benjaminglass1, https://github.com/eqy

This updates NCCL_HAS_ONE_SIDED_API to require NCCL_HAS_SYMMEM_DEVICE_SUPPORT, matching the existing guard pattern used for NCCL device-side support.

NCCL_HAS_SYMMEM_DEVICE_SUPPORT is not enabled for ROCm builds because nccl_device.h is excluded there. Without this additional guard, ROCm release branches can still compile one-sided NCCL paths that reference device-side support such as NCCLDevCommManager, causing build failures.

A related guard already exists from #186794; this extends the same protection to the one-sided API path after it caused issues on another ROCm branch.

Testing

• Verified the merge conflict is resolved.
• Verified nccl_dev_cap.hpp has no conflict markers.
• Verified git diff --check passes for the changed file.

Pull Request resolved: #186888
Approved by: https://github.com/jeffdaily

• the symbolic storage expression, which preserves provenance for downstream
  trace consumers;
• the optimization hint, which gives diagnostic/policy tooling a concrete
  expected extent when one was explicitly provided.

Keep the existing size field symbolic for symbolic storage sizes and add a
separate size_hint field only when every free symbol in the storage-size
expression has an explicit optimization hint override. This preserves the
existing symbolic trace contract while exposing concrete policy metadata
without specializing tensor shapes or changing runtime semantics.

Fixes #183835

Test Plan:

python test/test_fake_tensor.py FakeTensorTest.test_meta_storage_trace_uses_hint_for_symbolic_size -q

lintrunner --config=.lintrunner.toml torch/_subclasses/meta_utils.py test/test_fake_tensor.py

Pull Request resolved: #183839
Approved by: https://github.com/ezyang, https://github.com/laithsakka

• #183838
• #183839
• ->#183840

[Inductor] Handle hinted and fallback unbacked symbols

Inductor has several codegen paths that need to reason about unbacked symbolic extents without turning policy decisions into semantic guards. Layout constraint lowering needs to recognize common symbolic stride orderings, while fallback kernels can repropagate outputs whose unbacked symbols are graph-owned and must be bound for wrapper codegen.

For stride ordering, use a stride-specific symbolic greater-or-equal proof with ordinary guarded comparisons plus divisibility reasoning. This lets Inductor prove layouts such as u0 * 256 >= 256 without relying on concrete hints, while still rejecting unproved unbacked layouts. When require_strides proves the current layout already satisfies the requested order, freeze the current layout directly instead of forcing guarding_hints_or_throw() or bailing out for every unbacked stride.

For fallback outputs, temporarily re-enable fresh unbacked symbol tracking while rerunning the fallback fake kernel. That call is the binding site for output size and stride symbols that later wrapper code references, so those symbols should go through the normal pending-symbol and compute_unbacked_bindings() path instead of being created under ignore_fresh_unbacked_symbols() and rediscovered afterward.

The C++ wrapper path also has to treat input unbacked symbols as already declared before emitting output bindings. Otherwise a fallback output binding can redeclare a symbol such as u0 and can emit Python-only floordiv syntax for DivideByKey paths. Emit C++ integer division for that path and reuse the existing unbacked symbol declaration.

Finally, do not make the post-copy stride-order sanity check prove data-dependent unbacked stride inequalities. The copy has already been materialized with the requested stride order; requiring a symbolic proof there rejects valid layouts whose unbacked size may be zero or one.

These changes preserve symbolic semantics: hints are not used to create semantic layout guards, fallback output extents are bound through normal ShapeEnv tracking at the fallback output binding site, and stride/order changes copy data or freeze already-valid layouts rather than manufacturing semantic guards.

Fixes #183834

Fixes #185341

Test Plan:

python test/inductor/test_unbacked_symints.py TestUnbackedSymintsCPU.test_stride_order_uses_unbacked_optimization_hint_cpu TestUnbackedSymintsCPU.test_stride_ordered_uses_symbolic_divisibility_cpu TestUnbackedSymintsCPU.test_stride_ordered_rejects_unproved_unbacked_layout_cpu -q

python test/inductor/test_unbacked_symints.py TestUnbackedSymintsCUDA.test_standalone_compile_reuses_fallback_unbacked_binding_cuda TestUnbackedSymintsCUDA.test_sdfpa_unbacked_strides_cuda -q

python test/inductor/test_unbacked_symints.py TestUnbackedSymintsCPU.test_stride_ordered_uses_symbolic_divisibility_cpu TestUnbackedSymintsCPU.test_stride_ordered_rejects_unproved_unbacked_layout_cpu TestUnbackedSymintsCUDA.test_standalone_compile_reuses_fallback_unbacked_binding_cuda TestUnbackedSymintsCUDA.test_sdfpa_unbacked_strides_cuda -q

TorchTitan graph_trainer H100 integration: aot_fx_trace_deepseek_v3_sdpa_full_inductor_ep_overlap_moe_seq

This PR was authored with the assistance of an AI assistant.

Pull Request resolved: #183840
Approved by: https://github.com/laithsakka

import torch
import torch.nn.functional as F

x = torch.randn(4096, 4096, device="mps", dtype=torch.bfloat16)

x.norm(2, dim=-1)          # L2 -> norm_l2_reduction_inner kernel
x.norm(1, dim=-1)          # L1 -> norm_l1_reduction_inner kernel
F.normalize(x, dim=-1)

Perf:

fp32

bf16

fp16

Pull Request resolved: #186076
Approved by: https://github.com/malfet

For stride ordering, use a stride-specific symbolic greater-or-equal proof with
ordinary guarded comparisons plus divisibility reasoning. This lets Inductor
prove layouts such as u0 * 256 >= 256 without relying on concrete hints, while
still rejecting unproved unbacked layouts. When require_strides() proves the
current layout already satisfies the requested order, freeze the current layout
directly instead of forcing guarding_hints_or_throw() or bailing out for every
unbacked stride.

For fallback outputs, temporarily re-enable fresh unbacked symbol tracking while
rerunning the fallback fake kernel. That call is the binding site for output
size and stride symbols that later wrapper code references, so those symbols go
through the normal pending-symbol and compute_unbacked_bindings() path instead
of being created under ignore_fresh_unbacked_symbols() and rediscovered
afterward.

The C++ wrapper path also has to treat input unbacked symbols as already
declared before emitting output bindings. Otherwise a fallback output binding
can redeclare a symbol such as u0 and can emit Python-only __floordiv__
syntax for DivideByKey paths. Emit C++ integer division for that path and
reuse the existing unbacked symbol declaration.

Finally, do not make the post-copy stride-order sanity check prove
data-dependent unbacked stride inequalities. The copy has already been
materialized with the requested stride order; requiring a symbolic proof there
rejects valid layouts whose unbacked size may be zero or one.

These changes preserve symbolic semantics: hints are not used to create semantic
layout guards, fallback output extents are bound through normal ShapeEnv
tracking at the fallback output binding site, and stride/order changes copy data
or freeze already-valid layouts rather than manufacturing semantic guards.

Fixes #183834

Fixes #185341

This PR was authored with the assistance of an AI assistant.

Test Plan:

python test/inductor/test_unbacked_symints.py TestUnbackedSymintsCPU.test_stride_order_uses_unbacked_optimization_hint_cpu TestUnbackedSymintsCPU.test_stride_ordered_uses_symbolic_divisibility_cpu TestUnbackedSymintsCPU.test_stride_ordered_rejects_unproved_unbacked_layout_cpu -q

python test/inductor/test_unbacked_symints.py TestUnbackedSymintsCUDA.test_standalone_compile_reuses_fallback_unbacked_binding_cuda TestUnbackedSymintsCUDA.test_sdfpa_unbacked_strides_cuda -q

python -m pytest test/inductor/test_unbacked_symints.py -q -s -k "stride_order"

lintrunner -a

stack-info: PR: #183840, branch: sanketpurandare/stack/14

nn.HuberLoss(delta=1.+0.j) silently accepts a complex value even though the error message for the underlying huber_loss() function states that delta must be a float. Other invalid types like strings are also accepted by the constructor but fail later with confusing errors.

This adds an explicit type check in HuberLoss.__init__ that rejects non-numeric delta values (e.g. complex, str) with a clear TypeError. Ints and bools are still accepted for backward compatibility since int is a reasonable numeric type and bool is a subclass of int.

Changes:

• torch/nn/modules/loss.py: Add isinstance(delta, (float, int)) check in __init__
• torch/testing/_internal/common_modules.py: Add ErrorModuleInput test for complex delta

Pull Request resolved: #184012
Approved by: https://github.com/jbschlosser, https://github.com/zou3519

PR is a bit of a big one so I'll try to summarize below:
Each row is an example tensor (scan dim + dtype) and the kernel it lands on. 1M = a long axis, the specialized long-scan kernels need axis >= 65536 with <= 1024 scans, everything else takes the generic fallback.

tiny - scan_tiny_innermost

contig - decoupled (float) / multiblock 3-pass (int)

inner_fb - scan_innermost_dim (generic)

vec - vec_decoupled (float) / strided_multiblock (int)

col - strided_col decoupled (float) / strided_multiblock (int)

outer_fb - scan_outer_dim (generic, axis<1024)

col|outer - strided_col (float) / scan_outer_dim (int, cols>=64)

reshape - trailing size-1 dims -> contiguous scan

mid3d - middle-dim outer scan

nc_in - non-contiguous (transposed) input

model_seq - [batch, seq]

model_bhs - [batch, heads, seq]

model_mamba - [d_inner, seq] outer

sweep_1Mx - [1M, n] dim0 inner-stride sweep

sweep_4x256k - [4, 256k, n] dim1 inner-stride sweep

sweep_large - [axis, n] dim0 large-stride sweep

Pull Request resolved: #185609
Approved by: https://github.com/malfet

Reverted #184023 on behalf of https://github.com/eellison due to causing failures internally (comment)

Preserve source locations for Inductor guards that are saved in the FX graph cache, so a cache-hit recompile reason still points at the original guard producer instead of the cache replay site.

Fixes #141595
Generated by my agent

Root Cause

Without the FX graph cache, ShapeEnv records each ShapeGuard with the source location captured when Inductor creates the guard, for example can_use_32bit_indexing.

On an FX graph cache hit, Inductor only had CompiledFxGraph.guards_expr, a single combined Python expression string. Re-evaluating that string installs equivalent guards in the current ShapeEnv, but the new guards get the replay callsite as their source location. That is why recompile logs pointed at _lookup_graph, autograd_cache.py, or <string>:1 instead of the original Inductor guard.

Review Guide

• torch/fx/experimental/symbolic_shapes.py: threads source locations alongside each produced guard expression, adds ShapeGuardExpression, and replays cached guards while temporarily using the stored source location for any ShapeGuard created by evaluation.
• torch/_inductor/output_code.py: adds optional cached metadata fields for the per-guard expression/source list and the SymInt argument count used when saving it.
• torch/_inductor/codecache.py: saves the new metadata, replays it on normal FX graph cache hits, and falls back to the existing combined-expression path for old cache entries, unsafe-skip mode, or AOTAutograd loads whose argument list differs.
• Tests cover both the small ShapeEnv round trip and the original CUDA/flex-attention cache-hit failure mode.

Test Plan

• PYTHONPATH=. python test/test_dynamic_shapes.py TestGuardsExpressions.test_guards_expression_source_info
• PYTHONPATH=. python test/inductor/test_codecache.py TestFxGraphCache.test_cache_hit_inductor_guard_preserves_source
• PYTHONPATH=. python test/dynamo/test_aot_autograd_cache.py AOTAutogradCacheTests.test_output_views_input_dynamic
• lintrunner -a

Pull Request resolved: #184193
Approved by: https://github.com/oulgen

Fixes #110611
Fixes #167463
Fixes #160812

Generated by my agent

Pull Request resolved: #184444
Approved by: https://github.com/oulgen

Behavior change: _num_warps halves num_warps only when warp_size == 64
(AMD CDNA/gfx9), not on every HIP device; AMD RDNA (wave32) now follows
the NVIDIA path.

Authored with: Claude

Pull Request resolved: #181112
Approved by: https://github.com/jansel

Test Plan:

python -c "import torch;x = torch.rand(1024, 1024, device='mps'); print(torch.mps.current_allocated_memory()); del x; torch.accelerator.empty_cache(); print(torch.accelerator.memory_allocated())"

Should allow next iterations of #184326 to land successfully on MacOS

This change was authored with the assistance of Claude (Claude Code).
Pull Request resolved: #186748
Approved by: https://github.com/Skylion007

Root cause: uniquify_block_sizes() in triton_combo_kernel.py only handled str and DeferredLine objects in the body buffer, but TMA load expressions are stored as DelayReplaceLine objects (a different subclass of DeferredLineBase). These were passed through unmodified, leaving R0_BLOCK unrenamed.

Fix: Changed the isinstance check from DeferredLine to DeferredLineBase and use _new_line() to create the replacement — this handles all deferred line types generically (DeferredLine, DelayReplaceLine, and any future subclasses).

Pull Request resolved: #185514
Approved by: https://github.com/kshitij12345, https://github.com/jansel, https://github.com/mlazos

Use dnnl::lstm_forward primitive on XPU for LSTM inference, replacing the per-timestep fused cell approach. The oneDNN primitive processes the entire sequence internally, eliminating O(T) kernel launches.

Changes

• New file aten/src/ATen/native/mkldnn/xpu/RNN.cpp: Implements lstm_onednn_xpu using dnnl::lstm_forward, registered via REGISTER_XPU_DISPATCH(lstm_mkldnn_stub, ...). Handles weight layout transformation (PyTorch [4*H, I] → oneDNN ldgoi), weight reorder, and scratchpad management.
• aten/src/ATen/native/RNN.cpp:
  • Extended use_mkldnn() to return true for XPU in inference mode (float/bf16/fp16)
  • Added packed sequence unwrap: when batch_sizes are uniform (common in batch=1 inference), reshape to regular 3D tensor and route to oneDNN
  • Added XPU-specific LSTMCell path with pre-computed input gates as fallback for training

Correctness

• oneDNN and PyTorch use identical gate order: i, f, c̃, o
• oneDNN formula: gates = W·x + U·h + B where B = b_ih + b_hh (we sum before passing)
• Verified max diff < 1e-6 vs CPU reference across multiple configurations

Performance (Intel PVC, Kokoro TTS, bidirectional LSTM, hidden=256)

Dependencies

• Depends on intel/torch-xpu-ops#3770 for the fused cell fallback path (bias fix)
• Only affects XPU. CUDA and CPU paths unchanged.

Pull Request resolved: #185531
Approved by: https://github.com/EikanWang, https://github.com/guangyey, https://github.com/atalman

The post NVIDIA cuTile Python Tutorial: Building Tiled GPU Kernels for Vector Addition, Matrix Addition, and Matrix Multiplication in Colab appeared first on MarkTechPost.

Root cause: uniquify_block_sizes() in triton_combo_kernel.py only handled str and DeferredLine objects in the body buffer, but TMA load expressions are stored as DelayReplaceLine objects (a different subclass of DeferredLineBase). These were passed through unmodified, leaving R0_BLOCK unrenamed.

Fix: Changed the isinstance check from DeferredLine to DeferredLineBase and use _new_line() to create the replacement — this handles all deferred line types generically (DeferredLine, DelayReplaceLine, and any future subclasses).

Pull Request resolved: #185514
Approved by: https://github.com/kshitij12345, https://github.com/jansel, https://github.com/mlazos

Fixes: #185477
Pull Request resolved: #185577
Approved by: https://github.com/guangyey, https://github.com/EikanWang, https://github.com/janeyx99

This PR is a follow-up to discussion in #185860
Pull Request resolved: #186109
Approved by: https://github.com/malfet

Co-authored-by: Aaron Gokaslan aaronGokaslan@gmail.com

CPU topk now compares by value and then index in deterministic mode. CUDA keeps the existing top-k selection path and uses stable sorting for the selected top-k outputs so threshold ties preserve input-index order without falling back to a full sort on NVIDIA. ROCm uses stable sort in deterministic mode because its current top-k gather reserves tie slots through atomics.

This gives deterministic mode a topk behavior analogous to scatter_add: fast default kernels remain available, while deterministic mode uses stable algorithms when requested.

Test Plan:

• Rebuilt PyTorch from source with CUDA enabled.

• python -m py_compile test/test_sort_and_select.py torch/init.py torch/_torch_docs.py

• python test/test_sort_and_select.py -k topk_deterministic

• python test/test_sort_and_select.py -k topk

• git diff --check HEAD^

• lintrunner --config=.lintrunner.toml --skip=PYREFLY aten/src/ATen/native/TopKImpl.h aten/src/ATen/native/cuda/TensorTopK.cpp test/test_sort_and_select.py torch/init.py torch/_torch_docs.py

Pull Request resolved: #186653
Approved by: https://github.com/drisspg

Use dnnl::lstm_forward primitive on XPU for LSTM inference, replacing the per-timestep fused cell approach. The oneDNN primitive processes the entire sequence internally, eliminating O(T) kernel launches.

Changes

• New file aten/src/ATen/native/mkldnn/xpu/RNN.cpp: Implements lstm_onednn_xpu using dnnl::lstm_forward, registered via REGISTER_XPU_DISPATCH(lstm_mkldnn_stub, ...). Handles weight layout transformation (PyTorch [4*H, I] → oneDNN ldgoi), weight reorder, and scratchpad management.
• aten/src/ATen/native/RNN.cpp:
  • Extended use_mkldnn() to return true for XPU in inference mode (float/bf16/fp16)
  • Added packed sequence unwrap: when batch_sizes are uniform (common in batch=1 inference), reshape to regular 3D tensor and route to oneDNN
  • Added XPU-specific LSTMCell path with pre-computed input gates as fallback for training

Correctness

• oneDNN and PyTorch use identical gate order: i, f, c̃, o
• oneDNN formula: gates = W·x + U·h + B where B = b_ih + b_hh (we sum before passing)
• Verified max diff < 1e-6 vs CPU reference across multiple configurations

Performance (Intel PVC, Kokoro TTS, bidirectional LSTM, hidden=256)

Dependencies

• Depends on intel/torch-xpu-ops#3770 for the fused cell fallback path (bias fix)
• Only affects XPU. CUDA and CPU paths unchanged.

Pull Request resolved: #185531
Approved by: https://github.com/EikanWang, https://github.com/guangyey, https://github.com/atalman

Pull Request resolved: #186402
Approved by: https://github.com/guilhermeleobas

This failure made it into the repo because the CI does not have support for the UCC backend.

Fixes: #186664
Pull Request resolved: #186666
Approved by: https://github.com/d4l3k

This builds on the pointwise host-side TMA infrastructure from #185825 and extends it to the template codegen path in select_algorithm.py. The Blackwell warp-specialized persistent TMA template (triton_blackwell_ws_persistent_tma_mm.py.jinja) is updated to use the hook, eliminating the inline tl.make_tensor_descriptor() calls and supporting both host-side and device-side paths from a single template.

Key changes:

• config.py: enable_host_side_tma flag
• select_algorithm.py: tma_descriptor() hook with dim_order support for transposed inputs, signature upgrade to tensordesc<> types, host_tma_descriptor_args export to inductor_meta
• template_heuristics/triton.py: HOST_SIDE_TMA config in Blackwell mixin
• mm.py: renamed template to blackwell_ws_persistent_tma (agnostic to host/device)
• static_triton_launcher.py: tensordesc<> type fallback to dynamic launcher
• codegen/triton.py, triton_heuristics.py: support dict format in host_tma_descriptor_args for dim_order