与「C_real」相关的搜索结果

In 2019, OpenAI announced GPT-2 with this post: Today (~5 years later) you can train your own for ~$672, running on one 8XH100 GPU node for 24 hours. Our latest llm.c post gives the walkthrough in some detail: Incredibly, the costs have come down dramatically over the last 5 years due to improvements in compute hardware (H100 GPUs), software (CUDA, cuBLAS, cuDNN, FlashAttention) and data quality (e.g. the FineWeb-Edu dataset). For this exercise, the algorithm was kept fixed and follows the GPT-2/3 papers. Because llm.c is a direct implementation of GPT training in C/CUDA, the requirements are minimal - there is no need for conda environments, Python interpreters, pip installs, etc. You spin up a cloud GPU node (e.g. on Lambda), optionally install NVIDIA cuDNN, NCCL/MPI, download the .bin data shards, compile and run, and you're stepping in minutes. You then wait 24 hours and enjoy samples about English-speaking Unicorns in the Andes. For me, this is a very nice checkpoint to get to because the entire llm.c project started with me thinking about reproducing GPT-2 for an educational video, getting stuck with some PyTorch things, then rage quitting to just write the whole thing from scratch in C/CUDA. That set me on a longer journey than I anticipated, but it was quite fun, I learned more CUDA, I made friends along the way, and llm.c is really nice now. It's ~5,000 lines of code, it compiles and steps very fast so there is very little waiting around, it has constant memory footprint, it trains in mixed precision, distributed across multi-node with NNCL, it is bitwise deterministic, and hovers around ~50% MFU. So it's quite cute. llm.c couldn't have gotten here without a great group of devs who assembled from the internet, and helped get things to this point, especially ademeure, ngc92, @gordic_aleksa, and rosslwheeler. And thank you to @LambdaAPI for the GPU cycles support. There's still a lot of work left to do. I'm still not 100% happy with the current runs - the evals should be better, the training should be more stable especially at larger model sizes for longer runs. There's a lot of interesting new directions too: fp8 (imminent!), inference, finetuning, multimodal (VQVAE etc.), more modern architectures (Llama/Gemma). The goal of llm.c remains to have a simple, minimal, clean training stack for a full-featured LLM agent, in direct C/CUDA, and companion educational materials to bring many people up to speed in this awesome field. Eye candy: my much longer 400B token GPT-2 run (up from 33B tokens), which went great until 330B (reaching 61% HellaSwag, way above GPT-2 and GPT-3 of this size) and then exploded shortly after this plot, which I am looking into now :)

@cayetanaAT AQUÍ el testimonio transcrito de un venezolano torturado por el régimen en 2017. Fue hecho público por @TAMARA_SUJU: Página 1 «Mi nombre es Darwin Antonio Solís Benítez C.I. 15.050.872, Sargento Primero (Reserva Activa), fui privado de libertad el 6 de agosto del 2017 por una comisión del CONAS y DGCIM en Naguanagua Estado Carabobo, en horas de la mañana, al momento de la detención fui golpeado, pateado, apostado y azotado con objetos contundentes (Maderos y culatas de armas) incluso después que ya me habían amarrado de las manos y los pies; luego aparecieron unas personas de la fiscalía que ordenaron mi traslado al hospital Catia debido al estado en que me encontraba, allá me suturaron y limpiaron la sangre y me trasladó un personal del DGCIM al hospitalito en Fuerte Tiuna, Caracas, allí estuve durante un día, realmente no recuerdo cuando era de noche o de día porque los custodios del DGCIM que me acompañaban, me hostigaban y me maldecían diciéndome traidor y dándome puñetazos por las costillas a cada momento mientras no se encontraba el personal de médicos y enfermeros,» Página 2 «en horas de la tarde me sacaron aun sin cumplir el tratamiento para la sede de la DGCIM en Boleíta donde me recibieron con una golpiza y me obligaron a permanecer la noche en cuclillas, durante todo el tiempo estuve con una capucha, el día martes 8 de agosto me dieron varias golpizas y me obligaron a firmar un montón de documentos que no pude leer, luego me llevaron a la base aérea (creo que a la Carlota) todo era muy confuso aun me encontraba aturdido y desorientado, me montaron en un helicóptero y me llevaron al puente paramacay, allá estaba el Director de la DGCIM con el Coronel Franco Quintero y le dijeron a los funcionarios que me dieran trato especial. El trato especial que me dieron fue el de torturarme toda la mañana y la tarde (me golpeaban con una tabla por la planta de los pies, las rodillas, los codos, los glúteos, cortaron por debajo del tabique de mi nariz con un plástico y la estiraban hasta la frente para infringirme dolor, me asfixiaron» Página 3 «hasta el punto de perder la conciencia, creo que más de diez veces, luego en la tarde, ya antes de oscurecer, me tiraron en un montón de basura y mi cuerpo se cubrió de moscas, yo ni siquiera podía moverme, ya estaba oscureciendo y me sacaron de la basura, martillaron los dedos de mis manos con los cañones de los fusiles que cargaban, me tiraron en el cajón de una camioneta y me llevaron donde sus jefes, quienes les ordenaron llevarme a Caracas, me trasladaron a Boleíta por la mañana y el día 10 agosto 2017 me llevaron junto a otros detenidos al Tribunal Militar 3ro de Control (descalso, en interiores y ensangrentado con el cuerpo cubierto de hematomas y apenas podía caminar) cuestión que presenció el Juez Capitán Manuel Anezquita, el Fiscal Teniente Ever Montero y la secretaria Brenda Manzanilla, donde el juez ordenó permanecer recluido en CENAPROMIL Ramo Verde. Desde el momento de mi reclusión en este centro hemos recibido» Página 4 «en varias oportunidades la visita de las comisiones de la DGCIM que vienen a saquearnos, a torturar y a destruir entre las que recuerdo el 12 de octubre 2017 a solicitud de Capitán de Navío José Ramón Bastón Silva, para ese entonces director de CENAPROMIL en lo que los de mi celda fuimos golpeados con tonfas y se llevaron y subieron a una patrulla a Alberto José Polo, a quien agarraron de patadas y puñetazos y le pusieron una capucha con gas pimienta (lo tuvieron en la patrulla un par de horas). La del 14 de enero 2018 donde nos golpearon con una tabla, nos lanzaron al piso dándonos de patadas y agarraron al capitán Jorge Pace y a Yhony Espinoza y les dieron patadas y metieron corriente en los testículos; y el 17 de mayo 2018 cuando tumbaron la puerta de la celda; nos gasearon con gas pimienta y luego con los golpes y los descargas nos hicieron morder por un perro de ataque». [Firma y h. dact.] X. P.

Elon Musk’s goal for The Boring Company is to solve one of the most miserable daily experiences on Earth: traffic Cities are three-dimensional But transportation is still mostly trapped in a two-dimensional surface network Roads, intersections, bottlenecks, traffic lights, accidents, construction, weather - everything gets stacked on the same flat layer until the entire system chokes The Boring Company’s answer is simple but radical: Go underground Build fast, low-cost tunnel networks under major cities and turn transportation into true 3D infrastructure Right now, the focus is on making tunneling dramatically faster and cheaper with machines like Prufrock, which is designed to mine continuously while installing tunnel liner at the same time But the long-term vision goes much further Local Loop tunnels could move people across cities without surface traffic, while future Hyperloop-style systems could connect entire cities at ultra-high speed Imagine going from Los Angeles to San Francisco, New York to Washington D.C., or Dubai to Abu Dhabi in a fraction of today’s travel time - underground, electric, direct, and protected from surface congestion That is the real mission: Building the missing third dimension of transportation This is how you actually attack soul-destroying traffic at civilization scale

Gilles, je vais démonter ta prémisse de départ, parce que tout le reste de ton argument s'effondre avec elle. Tu pars du principe qu'il faut une « sensibilité de gauche » pour ne pas laisser créver les gens de faim. C'est l'inverse total de ce que dit l'histoire économique des 50 dernières années. Les chiffres bruts. 1990 : 2,3 milliards de personnes en pauvreté extrême. 38% de l'humanité. 2025 : 831 millions. Environ 10%. 1,5 milliard d'êtres humains sortis de la misère absolue en 35 ans. La plus grande réduction de souffrance humaine de toute l'histoire de l'espèce. Qui a fait ça ? Pas l'aide internationale. Pas les ONG. Pas les programmes de redistribution. Pas la « sensibilité de gauche ». Le marché. L'ouverture commerciale. La Chine de Deng en 1978 qui abandonne le maoisme. L'Inde en 1991 qui libéralise. Le Vietnam, l'Indonésie, le Bangladesh qui s'ouvrent au capitalisme. Les seuls endroits où l'extrême pauvreté a EXPLOSÉ sur la même période ? Le Vénézuela socialiste : de 27% de pauvres en 2008 à plus de 80% en 2018, avec une inflation de 130 000% et un Vénézuélien moyen qui a perdu 11 kilos par dénutrition. La Corée du Nord. Cuba. Le Zimbabwe de Mugabe. La gauche ne nourrit pas les pauvres. Elle les fabrique. Le capitalisme produit tellement de richesse que même ses « perdants » américains vivent mieux que la classe moyenne soviétique. Un pauvre US a un frigo, une voiture, un téléphone, l'air conditionné, internet. Un pauvre cubain attend du riz. Ton argument selon lequel « le social aux USA est un désastre » repète une légende française. La réalité : le PIB par habitant américain est de 80 000$. Français : 45 000$. Un Mississippien — l'État US le plus pauvre — a un revenu médian supérieur au Français moyen. La vérité que la gauche française refuse de regarder : dans un système libéral, il y a plus de richesse créée, plus largement distribuée, et beaucoup moins de pauvres. Partout. Sans exception. Sur toutes les périodes mesurées. ÊTRE de gauche en 2026 face à ces données, ce n'est pas avoir de la « sensibilité ». C'est ignorer 35 ans de preuves accablantes. C'est préférer la posture morale au résultat. La compassion sans résultats, ça s'appelle de la vanité.

**Summary: Discussion between Jeff Liang and Quant Alex Wu on Optimizing Option Order Execution and Slippage Capture** The core topic of their conversation is: **The current option limit order execution is poor (high slippage, low fill rate), essentially due to the lack of professional high-frequency / algorithmic market-making capabilities. They need to upgrade from “cutting meat with a blunt knife” to a sophisticated Delta-hedging + options market-making system.** ### 1. Problem Diagnosis - Current order placement feels like **“cutting meat with a blunt knife”** — poor queue position, low fill probability, and severe slippage. - Jeff provided concrete data: **Average loss of approximately $5.2 per executed option contract** (slightly less than 1 bp), including fees and rebates — still unacceptable. - Even with perpetual futures maker fee rebates helping a bit, the situation “cannot be ignored.” - **Price checking and adjustment frequency is NOT the root cause.** The real drivers are **fill probability** and **queue position**. ### 2. Fundamental Solution Direction (Alex’s View) - A robust **Delta-hedging system** shares significant technical overlap with high-frequency market-making systems for spot, futures, and perpetual contracts. Without this foundation, one is essentially powerless against adverse selection. - Using **maker orders for Delta hedging** is conceptually the same as **Delta-1 market making for inventory risk management** — the analogy made everything “suddenly clear.” - Options market making and Delta-1 market making are **tightly coupled**: - The Delta-1 system handles the Delta exposure of options. - Options themselves can provide protection for Delta-1 positions. ### 3. Technical Difficulty and Implementation Path - This requires entering the realm of **algo trading / HFT**, involving substantial research and engineering resources. - **Language requirement**: Python is **not sufficient**. Must use **C++ and Rust**. - **Target clients**: Institutional clients and high-net-worth individuals engaging in on-exchange block trading. - **Detailed step-by-step roadmap from scratch (Alex’s plan)**: 1. Collect large volumes of **order book data** (snapshots, incremental updates, tick-by-tick trades) for perpetuals + futures + options. 2. Build **fill probability models + queue models**, including: - Limit order arrival intensity - Fill probability - Queue position - Latency modeling 3. First implement and validate on **Delta-1 products**, then extend the backtesting system to support these HFT primitives. 4. Expand from Delta-1 / single option contracts to **all option contracts** (requires major redesign and validation due to performance demands). 5. Develop specialized algorithms for **limit order posting + aggressive crossing** to reduce overall slippage. 6. Finally, conduct small-capital live trading validation. Alex repeatedly emphasized: **“This project is genuine heavy industry.”** ### 4. Consensus - Delta-One research is the foundation for studying option fill probabilities. - Options market making must be deeply integrated with the Delta-hedging system — they cannot be treated separately. - The current phase is **infrastructure building**, requiring patient and significant investment. **Overall Assessment**: Alex provided a highly professional and systematic optimization roadmap, covering data infrastructure, modeling, and execution layers. Jeff focused on the business pain point (real slippage costs). Both fully agree that a fundamental rebuild of the underlying high-frequency system is necessary. This is a classic **quantitative execution optimization** discussion — starting from a clear business problem and pointing directly toward building institutional-grade HFT-level capabilities.

Just realized I never shared this photo set >"< Will update later with the shots together with Lelouch 🌟 Recently I caught the flu and had to rest for a week. After only three days back at work I suddenly came down with acute gastroenteritis QQ Even after a few more days of recovery I still don’t feel fully back to normal, but the busy schedule is starting again 💦 Fingers crossed my body can keep up rrr... Don’t worry—I’ve been following up with the doctor and taking my meds >""< Everyone, please stay healthy too! ☺️ 剛剛才突然發現居然還沒有分享過這組照片>"< 之後再來更新跟魯路的合照🌟 前陣子因為A流休息了一個禮拜 緊接著工作3天後突然又急性腸胃炎QQ 休養個幾天後感覺還沒全好 又緊接著要開始忙碌了💦 希望我的身體可以撐住rrr... 放心我有乖乖回診吃藥的>""< 大家也要健健康康的！☺️ #反叛的魯路修# #CodeGeass# #C_C# #シーツー# #Cos# #CodeGeass# #C_C# #シーツー# #Cosplay# #Lelouch#

Ok, a few reflections on the book: 1. qntm defines antimemes as self-erasing information, but this book has a different (but related) definition of the concept: antimemes are (a) high-impact and (b) low transmissibility. Roughly, they are "important secrets". 2. The low transmissibility can be because the ideas are dense/difficult to understand (e.g. Moldbug's blog posts), or taboo/socially forbidden, or transient in some way (e.g. daylight savings time annoys people once a year, then we all forget about it example). Thus, these ideas tend to thrive in group chats and small networks of dedicated/passionate people. Eventually, the burst onto the public consciousness, sometimes in disruptive ways. 3. Original ideas are inherently antimemetic: they're very hard to transmit at first because you don't have the right language to talk about them, and they're easy to forget. This is why so few people have them at all. The most important ideas start as antimemes. 4. Generative small groups are the optimal environment for new ideas to arise and be developed. 5. Nadia defines "supermemes" as high impact / *high* transmissibility. Supermemes often present as apocalyptic in some way -- if you don't listen to this, you might literally die. Hence climate change, AI risk, war/nationalism as all cited as example supermemes. The book is very suspicious of supermemes: they suck up everyone's attention and time and result in very little constructive action; they are parasites. 5(a). "Memes" are low impact / high transmissibility. Think cat videos or brief flash-in-the-pan cultural moments that get forgotten quickly. 6. The book points out that there's often a clear "patient zero" for important ideas in the discourse: e.g. Nick C. with the jhanas, Venkatesh Rao popularizing Scott's "Seeing Like a State", and various other 'patient zeros' for now-important ideas are discussed. Ideas that survive often have Champions who talk about them persistently for many years. 7. There's a fun discussion of memetic/information warfare, and how preference cascades can be best understood. A great way to spread antimemes is to form private groups around them but not make the existence of those groups public, and have individual members of the group sometimes promote the ideas in a way that looks uncorrelated. Makes it seem that the support for the idea is more widespread than it might be initially. Eventually you reach a tipping point where it becomes socially ok to express that idea. 8. The Hayekian case for capitalism is an antimeme. Whereas communism is a supermeme: the ideas are very intuitive to everyone, unlike with capitalism where you need a lot of logic to understand how it works and why things end up being better over time. This explains why economists are so resigned to being perennially misunderstood: economic ideas are just quite hard to understand! Luckily, capitalism (a) works (b) can hook into people's greed, and so it survives, even though comparatively few people understand why. 9. It's a fun exercise to identify ideas that are 'on the cusp' / in the dark forest right now, but aren't quite fully acceptable to say out loud yet. I can think of quite a few. 10. Part of the implicit challenge of the book is "what good ideas will you be the champion of?" and "stop thinking so much about dumb culture war stuff and form more small groups to develop and propagandize the actual good ideas!". So even though the book laments the death of the open web, I also read it as pro group chat. 11. Some meta points: it's refreshing to read something that's written like a blog post, but in book form; almost all non-fiction is written in the same journalistic voice nowadays, but this one just gets to the point and packs an impressive number of insights per page. It's also great to read a book that cites the current intellectual scene, more or less as it's happening (most of the citations are URLs to blog posts). 12. Going beyond the book: many words and stories function as containers for ideas that are too complex to be put into legible language. The story of Abraham's sacrifice of Isaac in the Bible (Genesis 22) for example, is not merely a carrier for the idea "you should obey God unquestioningly". The story is more than that, but it's hard to say how except by meditating on it a lot and making it a part of yourself. Kierkegaard's Fear and Trembling, one of my favorite books, is about this: the narrator, a non-believer, tries to explain the Abraham story several different ways and through various conceptual lenses (universal ethics, etc.), and concludes that it's fundamentally inexplicable in plain language. Thus the need for 'faith'. I view this as saying that the idea underlying this is real, but too high-dimensional to be flattened into any kind of explanation; it must instead be felt viscerally. Many of the most important ideas are like this.

Seeding my Bear ʕ•ᴥ•ʔ blog with more random posts, e.g. here's something I had on backlog for a while: # The append-and-review note An approach to note taking that I stumbled on and has worked for me quite well for many years. I find that it strikes a good balance of being super simple and easy to use but it also captures the majority of day-to-day note taking use cases. Data structure. I maintain one single text note in the Apple Notes app just called "notes". Maintaining more than one note and managing and sorting them into folders and recursive substructures costs way too much cognitive bloat. A single note means CTRL+F is simple and trivial. Apple does a good job of optional offline editing, syncing between devices, and backup. Append. Any time any idea or any todo or anything else comes to mind, I append it to the note on top, simply as text. Either when I'm on my computer when working, or my iPhone when on the go. I don't find that tagging these notes with any other structured metadata (dates, links, concepts, tags) is that useful and I don't do it by default. The only exception is that I use tags like "watch:", "listen:", or "read:", so they are easy to CTRL+F for when I'm looking for something to watch late at night, listen to during a run/walk, or read during a flight, etc. Review. As things get added to the top, everything else starts to sink towards the bottom, almost as if under gravity. Every now and then, I fish through the notes by scrolling downwards and skimming. If I find anything that deserves to not leave my attention, I rescue it towards the top by simply copy pasting. Sometimes I merge, process, group or modify notes when they seem related. I delete a note only rarely. Notes that repeatedly don't deserve attention will naturally continue to sink. They are never lost, they just don't deserve the top of mind. Example usage: - Totally random idea springs to mind but I'm on the go and can't think about it, so I add it to the note, to get back around to later. - Someone at a party mentions a movie I should watch. - I see a glowing review of a book while doom scrolling through X. - I sit down in the morning and write a small TODO list for what I'd like to achieve that day. - I just need some writing surface for something I'm thinking about. - I was going to post a tweet but I think it needs a bit more thought. Copy paste into notes to think through a bit more later. - I find an interesting quote and I want to be reminded of it now and then. - My future self should really think about this thing more. - I'm reading a paper and I want to note some interesting numbers down. - I'm working on something random and I just need a temporary surface to CTRL+C and CTRL+V a few things around. - I keep forgetting that shell command that lists all Python files recursively so now I keep it in the note. - I'm running a hyperparameter sweep of my neural network and I record the commands I ran and the eventual outcome of the experiment. - I feel stressed that there are too many things on my mind and I worry that I'll lose them, so I just sit down and quickly dump them into a bullet point list. - I realize while I'm re-ordering some of my notes that I've actually thought about the same thing a lot but from different perspectives. I process it a bit more, merge some of the notes into one. I feel additional insight. When I note something down, I feel that I can immediately move on, wipe my working memory, and focus fully on something else at that time. I have confidence that I'll be able to revisit that idea later during review and process it when I have more time. My note has grown quite giant over the last few years. It feels nice to scroll through some of the old things/thoughts that occupied me a long time ago. Sometimes ideas don't stand the repeated scrutiny of a review and they just sink deeper down. Sometimes I'm surprised that I've thought about something for so long. And sometimes an idea from a while ago is suddenly relevant in a new light. One text note ftw.

I gave a talk at GPU MODE workshop last week on llm.c - the origin story of llm.c - being naked in the world without PyTorch and having to re-invent Array, Autograd, Device, Dtype, Compile, Distributed - how to port a PyTorch layer to 1) explicit PyTorch - and then to 2) write the backward pass - 3) port forward & backward pass to C - 4) string all the layers together - achieving one file of C with no dependencies that compiles and runs ~instantly, where all memory is pre-planned and allocated a single time, fully deterministic, portable code that can run on a potato or a von Neumann probe - how most of llm.c was built at 1am-7am in a water villa porch in Maldives and why this is the recommended way to develop software - convert all of it to run in CUDA on GPU in fp32 - port matmul to cuBLAS - port attention to cuDNN flash-attention - introduce bfloat16 mixed precision - introduce many more optimizations and features like kernel fusions, Packed128, stochastic rounding, full determinism - add multi-GPU training, NCCL, sharded optimizer - add multi-node with MPI or file system or socket - reproduce GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672 in llm.c, achieving (at the time) 29% less memory, 19% faster training that PyTorch nightly, and much faster compile & run - how open source development attracts Avengers from the internet - port to training Llama 3 imminent (branch exists) - many other notable forks - last thought: how software abstractions like Python/PyTorch and everything else really exist only because humans are finite in knowledge, IQ and attention, and how with increasing AI capability LLMs may export custom binaries like llm.c for any application directly, tearing apart and refactoring all abstractions as needed. <|endoftext|> More links in reply