Recent Summaries

1. This MIT Technology Review Explains article discusses the growing trend of peptide use for wellness purposes, highlighting both the potential benefits and significant risks associated with unregulated and often untested compounds. It emphasizes the lack of human trials for many popular peptides and raises concerns about quality control and the possibility of harmful side effects.

2. Key themes:

   • Peptide popularity: Peptides are increasingly popular among wellness influencers and biohackers, promising various benefits like weight loss, muscle gain, and cognitive enhancement.
   • Regulatory void: Most peptides are sold for "research purposes only," bypassing regulations and creating a market for untested and potentially unsafe substances.
   • Quality concerns: Testing reveals significant variability in purity and potency, with some products containing no active ingredient or harmful contaminants.
   • Potential risks: Side effects of experimental peptides are largely unknown, with some researchers raising concerns about cancer risks and other health issues.
   • FDA Scrutiny: There is impending FDA action and increased scrutiny, but there is also a potential for relaxed FDA rules for alternative medicine.

3. Notable insights:

   • The line between legitimate research and illegal marketing of peptides for human consumption is blurry.
   • Even if some peptides have potential benefits, the lack of proper dosage guidelines and administration protocols makes their use risky.
   • The rapid growth of the peptide market is driven by profit motives, with companies able to make millions without investing in research or safety testing.
   • The article highlights the critical need for more rigorous clinical trials and regulatory oversight to ensure consumer safety.
   • There have already been hospitalizations linked to peptide injections, underscoring the real-world risks associated with these unregulated products.

This Latent Space newsletter announces the discontinuation of SWE-Bench Verified as a reliable benchmark for coding AI models due to saturation and contamination. It features a discussion with Mia Glaese and Olivia Watkins from OpenAI's Frontier Evals team, who explain the reasons behind this decision and endorse SWE-Bench Pro as a more suitable alternative. The discussion also explores the future of coding evaluations, emphasizing the need for more complex, real-world tasks and human-intensive evaluation methods.

• Benchmark Contamination: Frontier models have been exposed to SWE-Bench problems during training, leading to models regurgitating solutions verbatim.
• Flawed Tests: Over 60% of the remaining problems in SWE-Bench Verified are deemed unsolvable due to overly narrow or overly broad test specifications.
• Endorsement of SWE-Bench Pro: OpenAI is officially moving away from SWE-Bench Verified and recommending SWE-Bench Pro as a more challenging and less contaminated benchmark.
• Future of Coding Evals: The focus is shifting toward longer-term tasks, open-ended design decisions, code quality, real-world product building, and human-intensive evaluations.
• Preparedness Framework: OpenAI's work on coding evals is tied to their Preparedness Framework, which aims to track and mitigate potential risks associated with advanced AI capabilities.

This article argues that the future of enterprise AI relies on interoperability between AI agents, moving away from siloed deployments to a collaborative ecosystem. It introduces Agent2Agent (A2A) as an open standard to facilitate cross-vendor communication and highlights the need for robust governance and trust to ensure responsible scaling.

• Interoperability is Key: The central theme is that AI agents must be able to communicate and coordinate actions across systems to unlock their full potential and avoid fragmented gains.

• Open Standards (A2A): The article promotes open protocols like A2A as essential for enabling seamless collaboration between agents from different vendors and technologies.

• Governance & Trust: The piece emphasizes the importance of transparency, auditability, and governance frameworks to ensure responsible and sustainable interoperability at scale.

• From Pilots to Operating Systems: Interoperability enables the transition from isolated AI pilots to AI-powered operating models that transform entire enterprises.

• Siloed AI agents lead to duplicated work, miscommunication, and bottlenecks, hindering enterprise-wide transformation.

• Interoperability requires open protocols, unified data fabrics, and centralized orchestration layers.

• Eaton's implementation demonstrates how interoperable AI agents can improve resolution times, reduce tickets, and enhance employee experience.

• A2A supports enterprise-grade authentication and auditability for robust governance.

• Prioritizing interoperability today is crucial for enterprises aiming to lead in AI-powered collaboration.

1. High-Level Overview: The newsletter focuses on the potential of custom ASICs (Application-Specific Integrated Circuits) for AI models, highlighting Taalas' impressive Llama 3.1 8B inference speed using custom silicon and discussing the economic viability of ASICs per model. It also covers recent developments in frontier model evaluations, particularly Gemini 3.1 Pro, and raises questions about the validity and consistency of AI benchmarks.

2. Key Themes/Trends:

   • Custom ASICs for AI: Exploring the idea of "baking" LLMs into silicon for faster and cheaper inference.
   • Frontier Model Evaluations: Examining the performance of Gemini 3.1 Pro and other models on various benchmarks.
   • Benchmark Reliability: Questioning the consistency and relevance of current AI benchmarks like SWE-bench and ARC-AGI.
   • Token Efficiency and Cost: Highlighting the importance of token efficiency and cost-effectiveness in frontier models.

3. Notable Insights/Takeaways:

   • Taalas' 16,960 tokens per second inference speed with Llama 3.1 8B using custom silicon demonstrates the potential of ASICs.
   • The economic argument for custom ASICs is strengthening, particularly for models with billion-dollar training runs.
   • While Gemini 3.1 Pro shows strong retrieval capabilities and token efficiency, it faces tooling and consistency issues.
   • SWE-bench Verified evaluation methodologies need standardization to ensure fair comparisons across labs.
   • Current benchmarks may not fully capture real-world performance, prompting a debate on what metrics truly matter.

This MIT Technology Review newsletter promotes an exclusive subscriber-only eBook titled "The Great AI Hype Correction of 2025," which reflects on the overblown promises of AI companies and the need to readjust expectations. The eBook is part of a larger "Hype Correction" series. It features articles and analysis that suggest a critical look at the current state and future of AI.

• AI Hype Correction: The overarching theme is a necessary correction of the excessive hype surrounding AI, particularly after a year of reckoning in 2025.

• LLM Limitations: The eBook challenges the notion that Large Language Models (LLMs) are a panacea and highlights the limitations of AI as a quick fix for all problems.

• Bubble Concerns: It raises questions about a potential AI bubble and explores its possible nature.

• Beyond ChatGPT: The content positions ChatGPT as just one point in AI's evolution, not the ultimate end point.

• The eBook argues that the AI industry needs to move beyond unrealistic promises and address the fundamental limitations of current AI technologies.

• It suggests a potential market correction in the AI sector.

• The analysis encourages a more grounded and realistic perspective on the capabilities and impact of AI, moving past the initial excitement surrounding tools like ChatGPT.

• The featured articles imply a growing backlash against AI, potentially fueled by concerns about its applications and connections to controversial figures.

This newsletter focuses on the release of Google's Gemini 3.1 Pro, positioning it as a competitive advance that surpasses previous models in certain areas. It summarizes the key aspects of the release, including performance benchmarks, practical applications, and the general sentiment surrounding its launch.

• Frontier Model Race: The newsletter highlights the continuous cycle of incremental updates among leading AI models, with Gemini 3.1 Pro being Google's latest offering to stay competitive.

• Benchmark Performance: Gemini 3.1 Pro demonstrates strong performance on benchmarks like ARC-AGI-2 (77.1%) and SWE-Bench Verified (80.6%), indicating improved reasoning, coding, and agentic capabilities.

• Practical Applications: The newsletter showcases Gemini 3.1 Pro's capabilities in SVG design and translating textual descriptions into visual aesthetics, demonstrating real-world improvements.

• Market Reaction: The release has generated mixed reactions, including excitement about practical improvements, skepticism about benchmark-targeting, and concerns about real-world agentic task performance.

• The release of Gemini 3.1 Pro appears to be driven by a need for Google to catch up with and potentially surpass competing AI models.

• While benchmark scores are impressive, the newsletter raises concerns about whether these translate into equivalent gains in real-world agentic tasks.

• The initial rollout has faced inconsistencies and availability issues, potentially impacting user experience.