Skip to main content

The $100M Telemetry Bug: What OpenAI's Outage Teaches Us About System Design

· 3 min read

On December 11, 2024, OpenAI experienced a catastrophic outage that took down ChatGPT, their API, and Sora for over four hours. While outages happen to every company, this one is particularly fascinating because it reveals a critical lesson about modern system design: sometimes the tools we add to prevent failures become the source of failures themselves.

The Billion-Dollar Irony

Here's the fascinating part: The outage wasn't caused by a hack, a failed deployment, or even a bug in their AI models. Instead, it was caused by a tool meant to improve reliability. OpenAI was adding better monitoring to prevent outages when they accidentally created one of their biggest outages ever.

It's like hiring a security guard who accidentally locks everyone out of the building.

The Cascade of Failures

The incident unfolded like this:

  1. OpenAI deployed a new telemetry service to better monitor their systems
  2. This service overwhelmed their Kubernetes control plane with API requests
  3. When the control plane failed, DNS resolution broke
  4. Without DNS, services couldn't find each other
  5. Engineers couldn't fix the problem because they needed the control plane to remove the problematic service

But the most interesting part isn't the failure itself – it's how multiple safety systems failed simultaneously:

  1. Testing didn't catch the issue because it only appeared at scale
  2. DNS caching masked the problem long enough for it to spread everywhere
  3. The very systems needed to fix the problem were the ones that broke

Three Critical Lessons

1. Scale Changes Everything

The telemetry service worked perfectly in testing. The problem only emerged when deployed to clusters with thousands of nodes. This highlights a fundamental challenge in modern system design: some problems only emerge at scale.

2. Safety Systems Can Become Risk Factors

OpenAI's DNS caching, meant to improve reliability, actually made the problem worse by masking the issue until it was too late. Their Kubernetes control plane, designed to manage cluster health, became a single point of failure.

3. Recovery Plans Need Recovery Plans

The most damning part? Engineers couldn't fix the problem because they needed working systems to fix the broken systems. It's like needing a ladder to reach the ladder you need.

The Future of System Design

OpenAI's response plan reveals where system design is headed:

  1. Decoupling Critical Systems: They're separating their data plane from their control plane, reducing interdependencies
  2. Improved Testing: They're adding fault injection testing to simulate failures at scale
  3. Break-Glass Procedures: They're building emergency access systems that work even when everything else fails

What This Means for Your Company

Even if you're not operating at OpenAI's scale, the lessons apply:

  1. Test at scale, not just functionality
  2. Build emergency access systems before you need them
  3. Question your safety systems – they might be hiding risks

The future of reliable systems isn't about preventing all failures – it's about ensuring we can recover from them quickly and gracefully.

Remember: The most dangerous problems aren't the ones we can see coming. They're the ones that emerge from the very systems we build to keep us safe.

The Inadequate Equilibrium: How Systems Fail and Where Opportunity Hides

· 4 min read

In 2018, the FDA finally approved fish oil-based nutrition for infants with short bowel syndrome—a treatment that had been saving lives in Europe for decades. The delay wasn’t the result of inept regulators; it was a textbook case of what Eliezer Yudkowsky calls an “inadequate equilibrium”: a stable but suboptimal state where obvious improvements remain unmade. While American infants faced a harrowing 30% mortality rate using soybean-based formulas, European infants, treated with fish oil-based alternatives, saw mortality rates drop to 9%. This stark disparity reveals how even advanced systems can become trapped by inertia.

An inadequate equilibrium arises when no single actor—be it a company, regulator, or individual—has both the incentive and the means to improve the system. Markets, when efficient, tend to eliminate such inefficiencies. But in some domains, systemic constraints entrench failure, creating opportunities for those willing to challenge the status quo.

The Hidden Cost of Systemic Inertia

Take the U.S. healthcare system, where medical errors remain the third leading cause of death, contributing to over 250,000 fatalities annually. Unlike aviation, which reduced accidents by 65% through rigorous error tracking, hospitals rarely track error rates or publish performance metrics. This failure isn’t due to incompetence among healthcare professionals; it’s the product of structural barriers. Hospitals fear litigation, regulatory penalties, and reputational damage, creating a culture of concealment rather than transparency. Inadequate equilibrium: preserved.

Similarly, in cybersecurity, despite rising threats, many organizations continue to rely on outdated practices. Procurement processes, compliance mandates, and sheer organizational inertia create a system where even superior solutions struggle to gain traction. These systemic blind spots—embedded in policy, habit, and culture—lock organizations into suboptimal outcomes.

Lessons from Tech: Breaking Equilibrium

The tech industry, often lauded for its dynamism, isn’t immune to these traps. For decades, programmers endured clunky version control systems. Tools like CVS and Subversion were incremental improvements at best, leaving fundamental inefficiencies unchallenged. Enter Linus Torvalds, an outsider to the version control tooling space, who created Git—not an incremental improvement but a paradigm shift. Git’s distributed model and performance advantages shattered the inadequate equilibrium, demonstrating how bold, outsider-driven innovation can unstick stagnant systems.

A Framework for Spotting Opportunities

Yudkowsky’s concept of inadequate equilibrium offers a lens to identify when systems are ripe for disruption. It hinges on three questions:

  1. Market Efficiency: Does the domain quickly eliminate inefficiencies?
    • Efficient markets, like high-frequency trading, leave little room for obvious opportunities.
    • Inefficient ones, like healthcare, are plagued by opaque pricing and misaligned incentives.
  2. Systemic Constraints: Are there structural barriers preventing improvement?
    • FDA regulations demand costly large-scale studies, deterring solutions like fish oil-based nutrition, even when benefits are already clear.
    • Academic research prioritizes novelty over replication, leaving critical findings unvalidated.
  3. Information Asymmetry: Do you possess insights others lack?
    • Patients often out-research general practitioners on niche conditions.
    • Startups, unburdened by bureaucracy, can outpace incumbents.

Opportunity Beckons

For entrepreneurs and tech leaders, this framework points to actionable strategies:

  • Target domains where systemic constraints limit incumbents.
  • Focus on “good enough” markets that are far from optimal.
  • Seek high-friction problems with low technical barriers.

For example, consider climate tech. Carbon capture is riddled with inadequate equilibria: funding gaps, policy inertia, and entrenched energy interests slow adoption. Yet, those who can bypass these systemic barriers—via modular solutions or unconventional funding models—can transform the landscape.

Breaking the Myth of Impossibility

“Inadequate equilibria” remind us that the reason no one has solved a problem isn’t always technical impossibility—it’s often systemic misalignment. Asking “Why hasn’t someone done this already?” is the wrong question. The right questions are:

  • What incentives sustain the current state?
  • Which barriers can I bypass that others cannot?
  • How can I deliver value without waiting for the system to change?

Consider OpenAI. While academic AI research languished under the weight of grant cycles and publish-or-perish incentives, OpenAI built a moonshot-focused organization that prioritized deployment over papers. By sidestepping traditional academic constraints, they accelerated progress and captured the frontier.

For the Optimist in You

For optimists, inadequate equilibria are more than problems—they’re maps to hidden opportunities. History shows that systems don’t fix themselves; they are fixed by those who see what others overlook and act when others won’t. Whether it’s transforming infant care, rewriting the rules of cybersecurity, or pioneering new tech, the greatest breakthroughs come from understanding not just what’s broken, but why—and daring to fix it.

So the next time you encounter a broken system, don’t dismiss it as an unsolvable mess. Look closer. Somewhere in its constraints lies an opportunity waiting to be seized.

How to Build and Sell Software Effectively

· 4 min read

Building successful software requires both exceptional product development and strategic distribution. Here's a framework for achieving both.

Strategic Foundation

Vision & Mission

  • Vision: Define the future state you aim to create
  • Mission: Outline core actions driving toward that vision
  • Strategic Master Plan: Map key milestones from small wins to major goals

Build Process

Development Approach

  1. Start with PRFAQ (Press Release/FAQ) for customer alignment
  2. Leverage community feedback to identify pain points
  3. Set aggressive timeframes:
    • Features: 2 weeks max
    • Projects: 1 quarter max

Product Evaluation Framework

Track what we build and how they serve customers using a product map:

SolutionUse CaseJobs to Be DoneScore (Quality × Distribution)
[Feature grouping by audience or purpose][Specific feature][Scenario-driven task or goal][Impact assessment]

For example,

SolutionUse CaseJobs to Be DoneScore (Quality × Distribution)
Asset Mobility for Blockchain UsersBridgeFacilitate seamless transfer of assets across blockchain networks...
Transparency for Network ParticipantsCuckoo Scan (Mainnet)Provide users and developers with detailed mainnet transaction and block data...
Cuckoo Sepolia Scan (Testnet)Help developers explore and test in a sandbox environment...

Quality Assessment (Insanely Great Product Framework)

DimensionCore Question1 (Inadequate)3 (Good)5 (Insanely Great)
Magical ExperienceCreates delight?FrustratingPleasantUsers become evangelists
Aesthetic AppealThoughtful design?ClutteredCleanEssential, elegant
Technical ExcellenceSolves complex problems?BasicSolidMakes impossible effortless
Ecosystem FitSeamless integration?High frictionWorks wellOpens new possibilities
Market ImpactCategory transformation?Me-too productIncrementalCategory-defining

Distribution Assessment (Go-to-Market Framework)

DimensionCore Question1 (Inadequate)3 (Good)5 (Insanely Great)
Customer EngagementAttracts/retains?Poor retentionModerate loyaltyBrand evangelists
Brand PerceptionBrand strength?UnrecognizedTrustedIconic
Channel EffectivenessDistribution performance?Limited reachKey segments coveredWide, seamless reach
Marketing InnovationStrategy uniqueness?GenericSome uniquenessTrendsetter
Revenue GrowthSustainable growth?Minimal growthSteady growthMarket leader

Distribution Strategy

Early-Stage Tactics

  • Personalized outreach (cold DMs) - use strategically due to platform risks
  • Content-driven SEO (blogs + tools)
  • Carefully managed affiliate programs
  • Targeted lifecycle emails
  • Supplementary paid advertising

Best Practices

  1. Retention and Referral: Prioritize how to make the product sticky and easy to be recommended
  2. Continuous Feedback: Actively gather and incorporate user input
  3. Platform Selection: Use appropriate tools for each function

Measurement & Iteration

Continuously evaluate and adjust using tools above:

  1. Evaluate and score initiatives in the product master map
  2. Identify misalignments with vision/mission and user feedback
  3. Prioritize adjustments
  4. Update strategic plan as needed

With a centralized map detailing what, where, and how we serve customers—combined with metrics and market feedback—we can navigate iterations more confidently, ensuring every solution is well-managed and improvements are driven by clear, fact-backed insights.

Principles for Second 10 Years at Work

· 6 min read

Early career focuses on continuous learning, gaining specialized expertise, and building foundational leadership skills, while late career shifts toward creating strategic impact, mentoring the next generation, and leaving a meaningful legacy that shapes industries, communities, or personal networks.

1. Start with the customer

  • 1.1 Surface yourself to internal and external buyers and sellers through online and offline platforms
  • 1.2 Conduct customer interviews and gather feedback regularly
  • 1.3 Empathize with customer pain points and act as a proxy for customers
  • 1.4 Predicting and producing what the customer loves is hard, but we can always invent and simplify our product to a better state
  • 1.5 Prioritize features and initiatives that provide the most customer value
  • 1.6 Measure success with customer satisfaction metrics (e.g., NPS, CSAT)

2. Sharpen your communication skills

  • 2.1 Be genuinely interested in people and actively wonder what they are doing/needing/feeling
  • 2.2 Practice and rehearse clear, concise writing and speaking until you're fully prepared for interviews
  • 2.3 Adapt your communication style to suit your audience
  • 2.4 Continuously seek feedback to refine your communication skills
  • 2.5 Leverage storytelling to make complex ideas more relatable and engaging
  • 2.6 Use examples, metaphors, and narratives to simplify and humanize technical or abstract concepts
  • 2.7 Apply communication frameworks like Thank/Reflect/Wish, PREP (Point/Reason/Example/Point), AIDA (Attention/Interest/Desire/Action), and STAR (Situation/Task/Action/Result)

3. Focus on high-leverage, prioritized activities

  • 3.1 Score and identify tasks with the most significant impact on key objectives
  • 3.2 Use frameworks like Eisenhower Matrix or OKRs to prioritize tasks
  • 3.3 Automate or delegate low-impact tasks
  • 3.4 Review priorities regularly to ensure alignment with goals
  • 3.5 Build flexibility into your schedule by leaving space for unplanned activities, and compensate for intense periods with planned downtime

4. Accelerate learning through high-value channels, execution, and accumulation

  • 4.1 Learn by doing, sharing, and documenting: Take on new projects, challenges, and record key lessons
  • 4.2 Seek feedback from peers and mentors after each task, and incorporate it into a personal knowledge base
  • 4.3 Engage in inter-person communication to uncover hidden, tacit knowledge that is often only discovered through direct interaction
  • 4.4 Embrace failure as part of the learning process, fail fast, and capture insights to avoid repeating mistakes
  • 4.5 Iterate and adapt: Build on past successes and failures while questioning assumptions, as what worked before may not hold true today

5. Ask high-value questions

  • 5.1 Challenge assumptions and conventional wisdom with first principles
  • 5.2 Guide strategic decisions by focusing on questions that reveal hidden insights and drive long-term impact
  • 5.3 Search for who is asking what questions globally to identify the most important questions
  • 5.4 Answers are cheap in the age of LLMs. Prioritize asking the right questions

6. Strategic thinking: from small wins to big wins

  • 6.1 Understand the history, status quo, and predict the future with long-term vision
  • 6.2 Break down large goals into smaller, actionable tasks
  • 6.3 Do not shoot for a moving target – maintain focus on clear, stable goals
  • 6.4 Collect data and establish benchmarks to evaluate success
  • 6.5 Identify areas where short-term wins can unlock long-term success
  • 6.6 Celebrate quick wins while keeping an eye on long-term goals
  • 6.7 Regularly review progress and adjust strategies as needed

7. Build allies

  • 7.1 Network widely to broaden your influence
  • 7.2 Help others succeed to build trust and reciprocity
  • 7.3 Collaborate across teams to expand your impact
  • 7.4 Stay in touch and offer support consistently
  • 7.5 Build trust through personal, in-person connections
  • 7.6 Exchange value in every interaction
  • 7.7 Stay active in key professional circles
  • 7.8 Nurture relationships for long-term mutual benefit

8. Stay resilient, optimistic, and pragmatic

  • 8.1 Focus on identifying the right problem and finding effective solutions when challenges arise
  • 8.2 Maintain a positive mindset, especially during setbacks (Reframing, expressing gratitude, etc.)
  • 8.3 Balance optimism with a realistic view of the situation by synthesizing

9. Influence without authority

  • 9.1 Build credibility by demonstrating expertise and delivering results
  • 9.2 Persuade through data and clear reasoning
  • 9.3 Leverage relationships to gain support for your ideas
  • 9.4 Be collaborative, listen to others, and align their goals with yours

10. Be assertive and open-minded

  • 10.1 Speak up confidently in meetings and discussions
  • 10.2 Seek feedback and adjust your approach if needed
  • 10.3 Advocate for your ideas but be willing to change if presented with better information
  • 10.4 Actively listen to others by mirroring, mentalizing, and showing genuine care
  • 10.5 Recognize the role of emotions in interactions and continuously improve your emotional intelligence (EQ)

11. Compete with focus and advantages

  • 11.1 Build a strategy based on specific market needs or niches you can dominate
  • 11.2 Identify and leverage your unique advantages (skills, location, resources)
  • 11.3 Focus on areas where you can outperform others with specialized knowledge or capabilities
  • 11.4 Anticipate future trends and invest in building advantages ahead of time
  • 11.5 Stay focused on your strengths and avoid unnecessary distractions
  • 11.6 Realize people can be both allies and competitors at the same time

12. Mentor and grow others

  • 12.1 Share your knowledge and expertise to help others succeed
  • 12.2 Offer guidance and feedback to team members regularly
  • 12.3 Create opportunities for others to take on new challenges
  • 12.4 Foster a culture of learning and development in your teams
  • 12.5 Leverage media and public platforms to inspire and educate others at scale

13. Stay approachable and flexible

  • 13.1 Remain accessible and responsive to buyers and customers at all times
  • 13.2 Be open to feedback and discussions with colleagues, buyers, and customers at all levels
  • 13.3 Maintain a friendly, open demeanor that encourages collaboration
  • 13.4 Adapt to changing situations without becoming rigid in your approach
  • 13.5 Stay calm and open-minded when unexpected challenges arise
  • 13.6 Be proactive so that the schedule is more favorable to the initiator

14. Lean in but avoid burnout

  • 14.1 Take initiative, but know your limits and learn to manage expectations
  • 14.2 Set boundaries to protect your personal time
  • 14.3 Regularly evaluate your workload, delegate where possible, and schedule regular downtime

15. Physical and mental well-being fuel everything else

  • 15.1 Make a checklist to monitor personal status regularly
  • 15.2 Maintain a regular exercise routine for MIIT, HIIT, Strength, and Flexibility exercises
  • 15.3 Prioritize sleep and proper nutrition
  • 15.4 Practice mindfulness or meditation to manage stress
  • 15.5 Take regular breaks to recharge, both mentally and physically
  • 15.6 Foster an environment where people love to live and work
  • 15.7 Learn to navigate your brain willfully through Relaxed, Focused, and Overheated states, and avoid Overheated states

16. Be creative and stand out

  • 16.1 Seek inspiration from different fields and industries
  • 16.2 Challenge assumptions and ask "what if" questions
  • 16.3 Experiment with different methods and technologies

BlockEden.xyz High-Availability Delegated Blockchain Node Infrastructure

· 5 min read

Node operation made easy with a reliable infrastructure

BlockEden.xyz introduces an advanced blockchain node infrastructure, enhancing the reliability and performance of blockchain operations. This development empowers customers to manage blockchain nodes easily and efficiently, ensuring the high availability and robustness of their blockchain applications.

Blockchain node operation often encounters issues like network instability, uptime monitoring, complex setup processes, and high maintenance costs. These challenges pose significant hurdles for businesses and individuals seeking to leverage blockchain technology.

BlockEden.xyz's delegated staking infrastructure offers a state-of-the-art solution to these challenges. This high-availability blockchain node infrastructure simplifies node setup, reduces operational costs, and ensures stable and consistent network performance. This makes blockchain technology more accessible and practical for a broader range of users.

BlockEden.xyz has an impressive track record, starting with Aptos and maintaining 99.9% uptime since its mainnet launch. Our services have expanded to include Sui, Solana, and 12 EVM blockchains, demonstrating our adaptability and commitment to staying at the forefront of the industry. With over $45 million worth of tokens staked with us, our clients trust us to provide reliable and secure solutions for their web3 and blockchain needs.

What is BlockEden.xyz’s Offering of the Delegated Staking Infra?

  1. Integrating with new networks.
  2. Managed/delegated services to run blockchain nodes, including maintaining a service-level agreement, status monitoring, active on-call, and failover.
  3. Develop tools to improve efficiency, e.g., distributing rewards.

Internal FAQs

What are the OKRs for Delegated Staking Infra Trying to Achieve?

Goals:

  • Build a reliable, safe, extensible, and cost-efficient delegated staking infrastructure to sustainably generate revenue streams for BlockEden.xyz.
    • Reliable: the delegate node should have high availability.
    • Safe: the funds delegated to us should be safe.
    • Extensible: the infra should be reusable to expand to another blockchain network.
    • Cost-efficient: the infra machine cost should be covered by the delegate revenue.
ItemsKey Results
SLO server uptime99.9%
New network onboarding time<= 2 weeks for EVM chains

Why is Delegated Staking Infra Important to BlockEden.xyz?

  • It’s the revenue-generating program, bringing resources to do more strategic infra work.
  • It’s an essential infra component to serve chain RPCs, particularly for latency-sensitive use cases.

How Can Delegated Staking Infra Fail? Would it Cause Customer Dislikes?

  • Failures in delegated staking infrastructure primarily occur due to service disruptions and server downtime. These issues can prevent customers from staking tokens or lead to reduced rewards due to slashing penalties. Such disruptions often result in customer dissatisfaction due to unmet expectations of consistent earnings and reliable service.

What are the Dependencies?

  • Network Selection: The chosen blockchain network plays a critical role. Different networks may have varying protocols, reward structures, and security requirements.
  • Initial Funding: To participate as a validator in the staking process, an initial investment or fund is required. This amount varies depending on the network's criteria and serves as your stake in the network.

What are the Risks?

  • Fund Safety: Ideally, we should make a product that doesn’t bear the liability for the customer's fund. However, it largely depends on the blockchain's features, e.g., isolation of token owner and node operator permission.
  • Crypto Market Volatility: In a sudden crypto price crash or bearish market, the token rewards might be short to cover the machine cost. A liquidation plan should be in place to ensure funds are secured to operate the infra.

How to Use this Staking Infra?

  • BlockEden.xyz offers comprehensive documentation for getting started with our staking infrastructure, including step-by-step guides for both manual staking processes

Why does staking infra matter to me?

  • Ease of Access: You can easily participate in staking and enjoy its benefits without needing deep technical knowledge or extensive resources.
  • Flexibility for Future Growth: We provide the capability for future technical integrations with your needs, ensuring that as your staking needs evolve, our infrastructure can accommodate them.

How to use this staking infra?

  • To get started with our staking infrastructure, BlockEden.xyz offers comprehensive documentation. This includes step-by-step guides for both manual staking processes and technical integrations.
  • We're committed to supporting your journey, addressing concerns and fulfilling requirements as they arise, ensuring a smooth and effective staking experience.

How do I know if this staking infra is the right solution to my problem?

  • To assess if our staking infrastructure suits your requirements, we recommend a phased approach. Start by trialing our service with selected networks. Monitor performance and stability over a set period. Based on this experience and the proven results obtained, you can then evaluate the effectiveness of our solution. This data-driven approach allows for informed decision-making and helps forecast potential future revenues, ensuring alignment with your business objectives.

Exploring CES 2024: Software Innovations, Trends, and Insights

· 6 min read

CES 2024 was bustling with excitement, drawing a massive crowd and featuring a wide range of exhibitors covering everything from consumer electronics products and services that cater to both the general public and professionals, permeating all aspects of work and life.

For those hoping to discover groundbreaking new products, you might be left wanting. The real buzz in recent years seems to center around Open AI's announcements, with their innovations having the potential to disrupt entire industries. The multitude of products on display here leverage less glamorous technology in new combinations to address existing or potential problems more efficiently, more powerfully, lighter, and sleeker. If you have something new that can help consumers, this is the place to showcase it.

I'm not an avid electronics enthusiast, but rather an early majority pragmatist on the Technology Adopt Lifecycle who cares about solutions that genuinely solve problems. I casually observed categories like cars, TVs/monitors, speakers and headphones, fitness bands, and medical devices, enjoying the fair-like atmosphere—your car can fly and dig, your robotic arm can make latte art, but what does that have to do with me? However, appreciating the design of electronic products, the sheer excitement and fun of sound and light, and understanding what electronic machinery can achieve at certain costs and price points is enlightening.

I found the startup and AI areas most intriguing and paid them the most attention. Software, FinTech, Web & Mobile Apps, and productivity tools for personal and corporate use were interspersed among the various hardware solutions.

AI

Rabbit R1 appears to be the only notable portable LLM hardware, lacking a booth but generating significant interest. It's puzzling why one would need a dedicated hardware for AI apps when a smartphone can perform the same functions, and possibly even better. Insights from experts are welcome.

booth.ai employs generative AI to assist e-commerce in photo editing: upload product photos and input text descriptions to generate images, such as adding backgrounds, changing people, or altering poses.

AiD / Cream creates detailed comic images and colors based on artists' sketches. A demo I tried showed impressive results, although the comics weren't overly complex.

Plaud Note, a card attached to the back of a phone for real-time recording, addresses the iPhone's inability to record and uses AI to quickly generate summaries. The product is well-made but not cheap.

vcat.ai generates marketing videos from the details of your product webpage. Although it sounds fancy, the URL merely imports materials from the web page, requiring manual template selection and adjustments in their system afterward.

Keeneat.com offers background music generation for creative professionals, along with a marketplace. This seems marketable, given the increasing number of video creators and the limited, expensive stock music available.

Blovo ChatGPT for animals allows you to inquire about anything pet-related.

FinTech & Enterprise

FinTech's visibility at CES was surprisingly low, with only five companies present, one of which was absent. The largest booth belonged to South Korea's Shinhan Bank, proud of their many years at CES. Their interest in CES stems from showcasing cashier machines as human teller replacements, aligning with the trend towards banking digitization and automation. The biggest challenge mentioned was adapting the UI/UX for older users, as high-net-worth VIP customers are not their primary target.

Woongjin is Korea's leading rental and subscription management system, integrating finance, payment, and insurance solutions, serving many large Korean conglomerates.

Blockchain

veintree (press release) offers identity verification through vein patterns on hands, claiming advantages over iris recognition in terms of anonymity and not requiring specialized equipment. It's being explored as a factor in multi-factor authentication, similar to Windows Hello and Face ID, and is seeking blockchain partnerships, like with Solana, to create a hand-based Worldcoin.

Fog Hashing sells Bitcoin mining machines for home use and professional data center solutions. Due to high electricity costs in California, they recommend mining in other states.

Flux token, an L1 PoW blockchain, supports distributed data centers running data storage, CMS, docker droplets, etc., serving as a platform for censorship-resistant DApps.

Pay.cool is an open-source decentralized crypto payment network from China, enabling consumers to pay merchants with cryptocurrencies. The app's design and their unclear offerings were noted during a conversation, though they are seeking mining partners.

DeluPay from France offers crypto payment solutions. Only their business card was seen, with their website in French, indicating incomplete internationalization.

Health

There were numerous Sleep Tech products, such as motionsleep's snore-control pillows, bokuk's temperature-regulating duvets, frenz's brainwave-monitoring and bone-conduction music headbands, and Lumos tech's sleep masks.

eclypia introduced a non-invasive glucose-monitoring watch.

aqara uses AI and spatial technology to improve elderly care.

Maintaining good physical health and sleep habits is an effortless way to save money. Good health is invaluable.

Tips for Exhibitors

Exhibit location matters: Avoid corners, small meeting rooms, and dead-ends with low foot traffic. Being in the main hall with popular exhibitors is more effective.

Pre-scheduling meetings and using hotel suites for product displays or business discussions can be a cost-effective strategy.

Hardware is easier to exhibit, especially with performance aspects, unlike software, which relies on engaging taglines, flyers, and demos to attract attention. Descriptive scenes and explanations are essential for less obvious hardware.

Exhibitors engrossed in their phones or in private conversations deter visitor engagement.

Sharing a booth among similar businesses, though unofficial, can help manage budgets.

Booking CES space early is crucial; arrangements can usually be made if plans change.

Branding should be visible to assist visitors with note-taking and photos.

If your demo is impressive, include your booth location on materials to attract more visitors through secondary sharing.

Tips for Attendees

Keep an open mind: Exhibitors are seeking industry connections. Criticism should be reserved as many are there to find or offer solutions within the industry.

The Innovation Awards can help narrow down interests amid the overwhelming variety of products.

Evening events and after-parties offer leisure and networking opportunities distinct from the more formal daytime exhibitions.

Protect your CES badge; replacements are costly.

General Observations

South Korea's presence was notably strong, driven by their need for international markets. Their broad category representation and the dedication of individual exhibitors, including non-English speakers, were impressive.

Apple's absence makes sense, given their ecosystem could overshadow many smaller exhibits.

SQPV glass presents solar panels as glass, though the appearance is somewhat grey, potentially fitting for skyscrapers with lower light transmission requirements.

ProtoHologram impressively projects holograms into a display box.

Elon Musk's Boring Company has a loop under LVCC, offering a unique but ordinary tunnel ride experience, only made notable upon realization of its significance.

Conclusion

For consumers, CES 2024 is a haven for electronic enthusiasts to experience and purchase products at discounted rates. It may not cater well to those already familiar with the market and looking for niche advancements.

For businesses, it's an ideal venue for hardware manufacturers and e-commerce to build supply chains and find collaborations, less so for pure software companies.

How to be an illuminator?

· 3 min read

An Illuminator is someone who has a significant and positive impact on the people around them through their interactions and communication style. Key characteristics of an illuminator include:

  1. Persistent Curiosity about Others: Illuminators are genuinely interested in other people. They continuously seek to understand others more deeply.
  2. Skilled in Understanding People: They have either innate talent or have developed the skill to understand others. This involves knowing what to look for in conversations and how to ask the right questions at the right time.
  3. Making Others Feel Valued and Seen: Illuminators have the ability to shine their attention and care on people, making them feel respected, acknowledged, and important.
  4. Bringing Out the Best in Others: Through their interactions, illuminators help people become better versions of themselves. They encourage others to be more honest, sharp, and to realize aspects of themselves they might not have articulated before.
  5. Creating a Sense of Empowerment: Islluminators have the ability to make others feel clever and important. This contrasts with diminishers, who make people feel small and unseen.
  6. Enhancing Productivity and Creativity: Illuminators can significantly boost the productivity and creativity of those around them, simply by being good listeners and engaging thoughtfully in conversations.

A Diminisher, in contrast to an Illuminator, is a person who negatively impacts those around them through their interactions and behaviors. They tend to make others feel small, unimportant, or unseen, often focusing self-centeredly on their own needs and interests.

How do illuminators have great conversations?

To be an effective conversationalist or an "illuminator," one must focus on deeply engaging and understanding others. Here are key takeaways:

  1. Mutual Engagement
    • Engagement is Key: Deep engagement in understanding others' thoughts and perspectives.
    • The Art of Conversation: Fostering two-way exchanges for mutual exploration and understanding. Being present and actively participating in conversations.
  2. Active Listening and Seeking Depth
    • Developing listening skills like full attention and physical engagement (nodding, eye contact).
    • Encouraging free expression and supporting the sharing of stories and experiences.
    • Specificity and Depth: Good conversations involve asking specific questions that encourage others to share detailed stories and personal experiences, helping them articulate their feelings and thoughts more vividly.
  3. Empathy and Understanding
    • Handling Pauses: Embracing pauses for thoughtful reflection and deeper understanding.
    • Looping for Clarity: Repeating or paraphrasing for clarity and confirmation of understanding.
    • Finding common ground in disagreements and prioritizing empathy over being right.
    • Keep the gem statement at the center, even though there are disagreements.
    • Don't be a Topper: Avoiding overshadowing others' experiences with your own.
  4. Facilitating Self-Discovery and Expression
    • Midwife Model: Assisting in others' process of self-discovery and expression.
    • The Importance of Being Listened To: Ensuring interactions make individuals feel heard, understood, and valued.

Being a good conversationalist is about creating a mutual journey of exploration, understanding, and deep listening, where both parties feel heard and valued.

Conducting User Interview

· 3 min read

User interview is a comprehensive process to find out and understand users, their problems, and potential areas for product improvement.

  • Objectives of User Interviews:
    • Refine hypotheses about the user: Understand their profile, motivations, and relationship with the product.
    • Clarify the user's problem: Grasp the problem in the user's words, its severity, and their current solutions or workarounds.
    • Identify potentially more valuable problems: Explore if other issues might be more critical to address.
  • Focus of Interviews:
    • Interviews should center on understanding problems from the user's perspective.
    • Avoid exploring potential solutions to prevent bias and leading the user.
  • Common Pitfalls:
    • Insufficient warm-up, leading to missed insights.
    • Leading questions that bias towards preconceived hypotheses.
    • Strict adherence to a script, missing new and valuable information.
  • Interview Trail Guide Framework:
    • Warm-up Phase: Build rapport and foundational understanding of the user. Small talks to introduce the team and the research. Ask for approval for recording. Emphasize that there are no right or wrong answers.
    • Build Phase: Gradually approach the problem hypothesis, allowing users to express their views, from general experiences to specific experiences.
    • Peak Phase (2/3 of the time): Deep dive into specific problem-related questions. Are we disucssing the right problem? Are users approaching the problem with any alternative ways? How painful is the problem?
  • Conducting the Interview:
    • Start with rapport-building questions.
    • Gradually narrow down to the specific problem.
    • Allow space for unexpected insights and follow up on them.
    • Spend substantial interview time in the peak phase, focusing on the hypothesized problem.
  • Final Steps:
    • Prioritize emerging problems during the interview.
    • Structure interviews to adapt to new insights and user feedback.

Examples

  1. Doordash:

    • Warm-up Phase: Questions might start broadly, touching on general lifestyle and habits, such as weekend routines and eating habits.
    • Build Phase: Narrowing down to food delivery experiences, with questions like why and how users order food delivery, their typical order times, and their general experience with food delivery.
    • Peak Phase: Specific questions about Doordash’s service, such as frequency of use, user experience with Doordash’s delivery time, and alternatives they resort to when delivery times are long. The goal is to assess whether their hypothesis about delivery time being a pain point is accurate.
  2. Rippling Employee Onboarding:

    • Build Phase: Questions might start with general feelings about onboarding new employees, tools or systems currently used, and issues faced with these tools.
    • Peak Phase: Focus on specific problems identified, like steps taken to onboard a new employee, tracking various requirements and deadlines, and the impact of current practices on the onboarding experience.

The Art of Empathy

· 11 min read

Recognition is Humanity's Primary Pursuit

The "still-face experiment" is a psychological experiment in which a caregiver (usually a mother) and her infant engage in normal face-to-face interactions, such as smiling, talking, and making eye contact. Then, the caregiver suddenly changes her behavior, maintaining a still, expressionless face (the "still-face"), and stops responding to the infant's actions. This change typically leads to noticeable stress responses in the infant, such as anxiety, agitation, or crying. After a period, when the caregiver resumes normal interaction, the infant usually gradually returns to the behavior exhibited at the beginning of the experiment.

Infants who are long neglected by their caregivers may develop a crisis of existence, which can cause lasting emotional and psychological harm.

Cold Family Relationships Build Emotional Walls

The quality of relationships determines the quality of life, and childhood relationships can have a lasting impact on one's quality of life.

Children coping with a harsh upbringing may unconsciously develop these four defense mechanisms:

  1. Avoidance: This is a defense mechanism born out of fear. Individuals choose to minimize emotions and relationships due to the harm caused by emotional and interpersonal connections. Such individuals feel most comfortable in superficial exchanges, tend to over-rationalize life, escape into work, strive for self-sufficiency, and pretend they have no needs. They often lack close relationships in childhood and hold low expectations for future interpersonal connections. These individuals may constantly be on the move, unwilling to settle down or be tied down; sometimes they may be overly proactive to avoid showing vulnerability; they manage to make themselves the strong ones others depend on but never seek help from others.
  2. Deprivation: Some children grow up around self-centered adults whose needs are ignored. Such children naturally learn the lesson that "my needs will not be met," which can easily transform into "I do not deserve to have." Those troubled by a deprivation model may feel worthless even after achieving remarkable success. They often carry the belief that there is some defect deep within them that, if known by others, would cause them to be abandoned. When treated poorly, they tend to blame themselves.
  3. Overreactivity: Children who grow up in dangerous environments often have an overactive threat detection system deep within their nervous systems. Such individuals interpret ambiguous situations as threats and perceive neutral faces as angry ones. They are trapped in an overactive mental theater, feeling that the world is full of danger. They overreact to situations without understanding why they do so.
  4. Passive Aggression: Passive aggression is an indirect expression of anger. It is a way for someone who fears conflict and struggles to handle negative emotions to avoid direct communication. Such individuals may grow up in a family where anger is frightening, emotions are unresolved, or love is conditional, learning that direct communication leads to withdrawal of love. Thus, passive aggression becomes a form of emotional manipulation, a subtle power game to extract guilt and love. For example, a husband with passive-aggressive tendencies might encourage his wife to go out with friends for the weekend, seeing himself as a selfless martyr, but becomes angry with her days before the outing and throughout the weekend. He will use various withdrawal and self-pitying behaviors to make her feel like a selfish person while portraying himself as the innocent victim.

The Dual Nature of Defense Mechanisms

These defense mechanisms do not always have negative impacts; they can be a form of overcompensation that leads individuals to extremes, and those who go to extremes may find it easier to achieve worldly success—many successful politicians, for instance, learn from childhood that life is a battle against injustice. Darkness gives them status, power, self-esteem, and resilience.

However, these benefits do not mask the problems caused by these defense mechanisms:

  1. Irrational hostility. They may believe that "all criticism and opponents are not only wrong but also evil."
  2. Individuals can be ensnared by their mechanisms. They may find themselves unable to control irrational actions.
  3. Old mechanisms become outdated. Old habits cannot adapt to the new era (conceptual blindness), such as fighting a modern war with the mindset of cold weapon warfare from World War I, leading to heavy casualties.

Repairing Issues? Communication is More Effective than Introspection

For various reasons, trying to repair the defense mechanisms stemming from a dark childhood through self-reflection often yields poor results. Communication with an external perspective is a more effective choice.

This is where empathy shines. Empathy is crucial at every stage of "knowing a person," and it is especially necessary when accompanying someone through trauma.

Empathy Sounds Easy but is Hard to Practice

If empathy is merely "I feel for you," it indeed sounds easy. However, empathy is a combination of a series of social and emotional skills. Some people are naturally good at these skills, but everyone can improve through practice.

Empathy involves at least three related skills:

  1. Mirroring
  2. Mentalizing
  3. Caring

Mirroring Emotions

People experience emotions in every moment of wakefulness through interactions with the external world. These emotions can be pronounced or subtle. The generation of emotions begins with sensations from every part of the body, transmitted through nerves to the brain, where they are monitored and recognized.

Historically, emotions were once considered a bad thing. For thousands of years, philosophers believed that reason was separate from emotion—reason was the cold, prudent driver, while emotion was the uncontrollable wild horse. This understanding is flawed.

In reality, emotions carry information. When not out of control, emotions are flexible mental abilities that help you navigate life. Emotions assign value to things: they tell you what you want and what you do not want. You pursue out of love and distance yourself out of disdain. Emotions help you adapt to different situations: when you find yourself in a threatening situation, you feel anxious, prompting you to quickly seek danger. Emotions also inform you whether you are moving toward your goals or away from them.

Thus, to understand a person, we should not only understand what they are thinking but also how they feel. These feelings are reflected in the other person's face, eyes, demeanor, and other parts of their body.

Masters of emotional mirroring can quickly experience the emotions of the person in front of them and can rapidly reproduce those emotions in their own bodies. Those skilled in emotional mirroring respond to smiles with smiles, yawns with yawns, and frowns with frowns. They unconsciously adjust their breathing patterns, heart rates, speaking speeds, postures, gestures, and even vocabulary levels to align with the other person. They do this because a good way to understand what another person feels in their body is to experience that emotion in your own body to some extent. Those who have received Botox injections and cannot frown may find it more difficult to perceive others' concerns because they cannot physically reproduce that emotion.

Masters of emotional mirroring have higher emotional granularity, allowing them to finely distinguish different emotional states and experience the world more precisely. They can accurately classify similar emotions: for example, anger, frustration, stress, anxiety, worry, and agitation.

Masters of emotional mirroring build a broad emotional vocabulary through reading literature, listening to music, and reflecting on relationships, enabling them to draw upon it skillfully in life, much like a painter having a wider palette of colors.

Mentalizing Emotions

Most primates can more or less mirror each other's emotions, but only humans can explain why the other person is experiencing their current emotions. This is also known as "projective empathy." When we connect our own memories with another person's current situation, we see more than just "this woman is crying"; we see "a woman who has suffered professional setbacks and public humiliation."

More advanced mentalizing helps us recognize the complexity of emotional states—people can experience multiple emotions simultaneously, and this complexity allows us to detach from empathy and make judgments.

Caring for Emotions

Many con artists are skilled at interpreting people's emotions, but we wouldn't say they are empathetic because they do not genuinely care for others. A child might see you crying and hand you a Band-Aid, but they cannot mentalize that you are crying because you had a tough day, nor can they know what you truly need at that moment.

Effective caring involves stepping outside of one's own experience and realizing that what you need in the same situation may be completely different from what I need. This is challenging; the world is full of "good people," but there are far fewer "effectively kind people." For instance, while some may need alcohol to cope with anxiety, others may need a hug.

Using the skill of caring for emotions, when you receive a gift from someone, write a thank-you note that focuses not on how you will use the gift but on the giver's intentions—what drove you to think this gift was suitable for me, and what you were thinking.

Similarly, cancer patients prefer "those who hug you, praise you, but do not make you feel like you are attending a funeral. Those who give you gifts unrelated to cancer. Those who just want to make you happy, rather than trying to fix you, reminding you that this is just another beautiful day with many interesting things to do."

Levels of Empathy

People with low empathy may think:

  • I find it difficult to know what to do in social situations.
  • If I am late to meet a friend, it usually does not bother me too much.
  • People often tell me that I overdo it when I am making a point in discussions.

People with high empathy may think:

  • Even if it does not involve me, interpersonal conflict is a physical pain for me.
  • I often unconsciously mimic the gestures, accents, and body language of others.
  • When I make a social mistake, I feel extremely uncomfortable.

In any field, truly creative thinking is simply this: a naturally exceptionally sensitive human being. For them, a touch is a blow, a sound is noise, a misfortune is a tragedy, a hint of joy is ecstasy, a friend is a lover, a lover is a god, and failure is death. Add to this fragile being a strong necessity for creation, constantly creating, creating, creating... Due to some strange, unknown inner urgency, they only truly live when they create.

-- Pearl S. Buck

High empathy sounds exhausting, but isn't it also moving? :)

How to Train Yourself to Increase Empathy?

  1. Contact Theory: Organize a group of people to do things together to build bonds and promote mutual understanding. A community is a group of people with shared projects.

  2. Observation and Performance: When people closely observe those around them, they become more empathetic. Actors are particularly good at observing and mimicking people; if you want your child to be more empathetic, encourage them to take drama classes at school.

  3. Literature: Plot-driven books like thrillers and detective stories are less effective; what works are complex, character-driven novels like "Beloved" or "Macbeth."

  4. Discovering and Labeling Emotions: Occasionally pause to use Marc Brackett's mood map and the RULER (Recognize, Understand, Label, Express, and Regulate their emotions) method to identify, understand, label, express, and regulate emotions. Teams led by emotionally intelligent bosses report feeling inspired 75% of the time, while teams with lower emotional intelligence report only 25%.

  5. Experiencing Suffering. Many truly empathetic people have experienced suffering but have not been crushed by it; they do not develop excessive defense mechanisms but instead expose their vulnerabilities to life and speak openly like heroes.

Conclusion

In summary, emotions are embodied, and empathy is not an intellectual activity but training your body to respond in an open and interactive way. The "rational brain" cannot persuade the "emotional body" to escape its own reality; thus, the body must personally experience different realities. Those with empathy can provide this physical presence.

Perception influences emotion, and emotion also affects perception. For example, when feeling afraid, our ears focus on high and low frequencies—the frequencies of screams or roars—rather than the medium frequencies of normal human speech. Anxiety narrows our attention and reduces our peripheral vision; happiness expands our peripheral vision.

Those who feel safe due to the reliability and empathy of others see the world as a broader, more open, and happier place.

And suffering is the badge of honor for practitioners of empathy. Playwright Thornton Wilder once described such a person's remarkable presence in the world: "Without your wounds, where would your power be? It is your regrets that make your low voice tremble into people's hearts. Even angels cannot persuade those suffering and clumsy children on earth, but those crushed by the wheels of life can. Only wounded soldiers can serve love."

Money20/20 Takeaways

· 3 min read

1. Technology Trends and Observations:

  • Shift from "blockchain-powered" to "AI-powered" company claims.

  • Emphasis on AI, machine learning, and blockchain as tools, not business models. Their value depends on their application in products and economic models.

    • Ramp & Finix's AI application on analytics, reporting, and expense automation.
  • Generative AI's potential in reducing labor costs and the challenge to differentiate genuine AI usage from PR hype.

    • Publicly available large models cannot satisfy the specific and refined needs of fintech institutions.
    • The financial sector resists the "black box" nature of previous AI generations and hasn't fully embraced them with the arrival of Gen AI.
    • Institutions should build their own mid-sized models using their data.
    • They should adopt Gen AI's interactive patterns to enhance the experience of existing products.
    • The current goal is to improve products rather than inventing new Gen AI products from nothing.

2. Industry Insights:

  • Insights from Cannabis Banking Summit: Challenges and opportunities in cannabis banking. Engagement with cannabis leading banks and credit unions to enhance risk management and compliance.

    • "Too much cash in our community is a problem", which is inconvenient, expensive, and often dangerous.
    • Working in the challenging realm of cannabis banking fosters the development of robust compliance, governance, and risk management systems, a point underscored by several bankers who have leveraged this expertise to branch into similar high-risk sectors like online gaming and crypto.
  • Pay-by-bank has emerged as a popular payment method that offers a convenient and secure way for customers to make online purchases directly from their bank accounts.

3. Political and Regulatory Influence:

  • Regulatory processes in the U.S. are slow, with a prediction of a 3-5 year adjustment period for banks heavily involved in "banking as a service."
  • The regulatory focus is mainly on the Bank Secrecy Act (BSA) and anti-money laundering (AML) concerns.
  • The increasing cost of compliance in the "banking as a service" space and the challenged promise of middleware platforms to reduce these costs.
  • Banks are signing up for FedNow, but they're mostly only signing up to receive payments, not to send them.

4. Banking as a Service (BaaS) and Open Banking:

  • Rise in "banking as a service" discussions with a perception of it being problematic rather than positive.
  • Open banking's proactive approach in educating policymakers contrasts with the lack of momentum for "banking as a service."
  • Skepticism around the extent and impact of open banking, especially its ability to facilitate account switching.
  • Debate around standard-setting bodies and the belief that Fintech companies haven't been involved enough in shaping these standards.

5. Compliance, Compliance, Compliance. Dodd-Frank Act Section 1033 – Consumer Access to Financial Records

  • Financial Data Exchange (FDX) will become the standard-setting organization for open banking.
  • Big tech firms like Apple face challenges with rules requiring data sharing for companies with over $10 billion in annual revenue.
  • Anticipation of bi-directional data sharing between banks and fintechs leading to further debates.