What are the main considerations for analyzing a company's competitive environment?​

New players, old players, what I buy, who buys from me, and other dimensions that can replace me.

What is the essence of competitiveness?​

Irreplaceability, which is the uniqueness in high demand.

Traditional engineering rubrics focus on specific areas, such as IC (software engineers, product managers, designers) or managers. Can we propose a unified framework to measure the business impact of technical workers? There are several requirements:

1. Comprehensive
2. Actionable
3. Guiding for team building

Answers from Me and My Friends​

• Decisiveness: Strategic, tactical, and decision-making abilities. Making the right decisions with the greatest chance of success for oneself or others, whether personal or organizational. Gathering information, diagnosing to gain insights, proposing effective strategies and execution plans. Innovation. Making quick decisions within opportunity windows/limited time/without all information/unable to accurately predict the future, and improving through iteration.

• Execution Engine: Building an execution engine rather than just focusing on execution itself, to effectively influence one's own and others' output, and to precisely balance speed/quality/scope based on demand, craftsmanship.

• Domain Expertise: Special know-how, experience, uniqueness.

• People and Culture: Working with others: shaping the world together with people. Leading by example. Team collaboration, people management, performance and incentives, culture, values, competence + warmth, mindset, passion, self-motivation, listening and sharing, hiring and coaching teams, humility, low ego. Best idea wins. Growth mindset.

• Product Sense: Good intuition, understanding of the market and data, product management (including operations), project management, collaboration/synthesis, invention and simplification, planning and adaptation, obsession with customers.

• Synergy & Resourcefulness: Resource integration ability, optimizing the network of customers, channels, products, people, technology, and capital on a macro level, forming alliances within the company, and effectively coordinating alignment across different organizations.

Requirements​

• design an auth solution that starts simple but could scale with the business
• consider both security and user experiences
• talk about the future trends in this area

Big Picture: AuthN, AuthZ, and Identity Management​

First-things-first, let's get back to basics

• Authentication: figure out who you are
• Authorization: figure out what you can do

In the beginning... Let there be a simple service...

• Layered Architecture

• Client stores a cookie or token as the proof of login status. (valet key pattern)

• Server persists a corresponding session

• Token is usually in the format of JWT, signed by keys fetched from somewhere secure (environment variables, AWS KMS, HarshiCorp Vault, etc.)

• Popular web frameworks often prepare out-of-box auth solutions

Then, as the business grows, we scale the system with AKF scale cube:

• X-axis: Horizontal clone
• Y-axis: Functional decomposition
• Z-axis: Sharding

Plus Conway's law: organization designs the systems mirroring its communication structure. We usually evolve the architecture to micro-services (see why microservices? for more)

• Btw, "microservices vs. monolith" and "multi-repo vs. mono-repo" are different things.
• For the enterprise, there are employee auth and customer auth. We focus more on the customer auth.

In the microservice world, let's take a functional slice of the authn and authz services, and there is an Identity and Access Management (IAM) team working on it.

• Identity-aware proxy is a reverse proxy that allows either public endpoints or checks credentials for protected endpoints. If the credential is not presented but required, redirect the user to an identity provider. e.g. k8s ingress controller, nginx, envoy, Pomerium, ory.sh/oathkeeper, etc.
• Identity provider and manager is one or a few services that manage the user identity through certain workflows like sign in, forgot password, etc. e.g. ory.sh/kratos, keycloak
• OAuth2 and OpenID Connect provider enables 3rd-party developers to integrate with your service.
• Authorization service controls who can do what.

Authentication​

Identity Provider

• The simplest solution is to submit the user's proof of identity and issue service credentials.
  • bcrypt, scrypt for password hash
• However, modern apps often deal with complex workflows like conditional sign up, multi-step login, forgot password, etc. Those workflows are essentially state transition graphs in the state machine.

Workflow: User Settings and Profile Updates

Ory.sh/Kratos as an Example Architecture

2. Third-party OAuth2​

OAuth2 let the user or client go through four major workflows (not sure which one to use? see this) like

1. Authorization Code Grant for web
2. Implicit Grant for mobile
3. Resource Owner Password Credentials Grant for legacy app
4. Client Credentials Grant for backend application flow

And then finally get the access token and refresh token

1. access token is short-lived, and hence the attacking window is short if it is compromised
2. refresh token works only when combined with client id and secret

The assumption is that there are so many entities involved in this workflow - client, resource owner, authorization server, resource server, network, etc. More entities introduce more exposure to attack. A comprehensive protocol should consider all kinds of edge cases. For example, what if the network is not HTTPs / cannot be fully trusted?

OpenID connect is the identity protocol based on OAuth2, and it defines customizable RESTful API for products to implement Single Sign-On (SSO).

There are a lot of tricky details in those workflows and token handling processes. Don't reinvent the wheel.

3. Multi-factor authentication​

Problem: Credential stuffing attack

Users tend to reuse the same username and password across multiple sites. When one of those sites suffers from a data breach, hackers brute-force attack other sites with those leaked credentials.

• Multi-factor authentication: SMS, Email, Phone Voice OTP, Authenticator TOTP
• Rate limiter, fail to ban, and anomaly detection

Challenge: Bad deliverability of Email or SMS

• Do not share marketing email channels with transactional ones.
• Voice OTP usually has better deliverability.

5. Passwordless​

1. biometric: Fingerprints, facial ID, voice ID

• still having trade-offs

2. QR code

• SQRL standard
• Another way to implement:

3. Push Notification

How could clients subscribe to the server's state? Short polling, long polling, web socket, or server-sent events.

4. Vendors on the market​

Don't reinvent the wheel.

• Managed solutions. Two years ago, I compared Auth0, Okta, Amazon Cognito, onelogin, Firebase Authentication. And quite recently, Okta acquired Auth0. In summary, I would recommend Okta and Auth0 if I am running a new startup and do not want to build my own auth systems.
• On-premise / self-host solutions. ory.sh.

6. Optimization​

Challenge 1: Web login is super slow or cannot submit login form at all.

• JS bundle is too large for mobile web
  • Build a lite PWA version of your SPA (single-page web app). whatever makes the bundle small - e.g. preact or inferno
  • Or do not use SPA at all. Simple MPA (multi-page web app) works well with a raw HTML form submission
• Browser compatibility
  • Use BrowserStack or other tools to test on different browsers
• Data centers are too far away
  • Put static resources to the edge / CDN and relay API requests through Google backbone network
  • Build a local DC 😄

See Web App Delivery Optimization for more info

Challenge 2: Account taking-over

• Fraud Detection with Semi-supervised Learning
• Unsupervised learning by datavisor

Challenge 3: Account creation takes too long

When the backend system gets too large, a user creation may fan out to many services and create a lot of entries in different data sources. It feels bad to wait for 15 seconds at the end of sign up, right?

1. collect and sign up incrementally
2. async

Authorization​

isAuthorized(subject, action, resource)

1. Role-based Access Control (RBAC)​

2. Policy-base Access Control (PBAC)​

{
  "subjects": ["alice"],
  "resources": ["blog_posts:my-first-blog-post"],
  "actions": ["delete"],
  "effect": "allow"
}

Challenge: single point of failure and cascading failures

• preprocess and cache permissions
• leverage request contexts
  • assumptions: requests inside of a datacenter are trusted vs. not trusted
• fail open vs. fail closed

Privacy​

1. PII, PHI, PCI​

Western culture has a tradition to respect privacy, especially after the Nazis murdered millions of people. Here are some typical sensitive data types: Personally Identifiable Information (PII), Protected Health Information (PHI, regulated by HIPAA), and Credit Card or Payment Card Industry (PCI) Information.

2. Differential Privacy​

Redacting sensitive information alone may not be good enough to prevent data associated with other datasets.

Differential privacy helps analysts extract data from the databases containing personal information but still protects individuals' privacy.

3. Decentralized Identity​

To decouple id from a centralized identity provider and its associated sensitive data, we can use decentralized id (DID) instead.

• it is essentially in the format of URN: did:example:123456789abcdefghijk
• it could be derived from asymmetric keys and its target business domain.
  • it does not involve your personal info, unlike the traditional way
  • See DID method for how it is working with blockchains.
• it preserves privacy by
  • use different DIDs for different purposes
  • selective disclosure / verifiable claims

Imagine that Alice has a state-issued DID and wants to buy some alcohol without disclosing her real name and precise age.

A DID solution:

• Alice has an identity profile having did:ebfeb1f712ebc6f1c276e12ec21, name, avatar url, birthday and other sensitive data.
• Create a claim that did:ebfeb1f712ebc6f1c276e12ec21 is over the age 21
• A trusted third-party signs the claim and make it a verifiable claim
• Use the verifiable claim as the proof of age

Summary​

This article is an overview of authn and authz in microservices, and you don't have to memorize everything to be an expert. Here are some takeaways:

1. follow standard protocols and don't reinvent the wheel
2. do not under-estimate the power of the security researchers/hackers
3. it is hard to be perfect, and it does not have to be perfect. Prioritize your development comprehensively

Architects model the world in system thinking to optimize everything. As an engineer and businessman, I am continually working on the orchestration of work and life and maintain a high personal growth rate. Lessons learned are generalized to the BOZ growth loops.

BOZ is the acronym for a big loop that engages three small loops:

• Build and Sell Loop. Build products and sell them. Solve problems and get paid.
• Outlook and Invest Loop. Be radically curious to seek truth from facts. And optimize the web of customers, channels, people, tech, and capital.
• Zen MSEP Loop. Being present and do everything with mindfulness. Fuel the mind and body by moving, sleeping, eating, and playing well with people.

Two sources of distress​

Stress is a good thing for people while the distress is not.

1. Stagnation: You are stuck in one step of the loops.
2. Frictions: Too many frictions prevent small wins.

When the organization grows too big, it becomes unclear that who should do what and who should decide what to do. RACI and DACI are here to clarify those responsibilities.

• RACI: figuring out who do what kinds of work
• DACI: figuring out who decides what to do

The assumption here is that too many projects happen in parallel, and it is easy to lose focus. We should rely more on the team member mutually driving each other than one person as the single point of failure.

RACI​

RACI is an acronym for the model:

• Responsible: who are the ones responsible for certain kinds of tasks?
• Accountable: which single one is the owner of the work and is accountable for the success and failure.
• Consulted: who are the stakeholders that should be kept in the loop before completion.
• Informed: Who should be informed of the progress though they are not formally consulted or directed contributing to the project.

DACI​

RACI is an acronym for the model:

• Driver: who is driving the decision-making process to a conclusion?
• Approver: people who are approving the decision.
• Contributor: who should contribute to the decision.
• Informed: who should be informed of the final decision?

Why are they helpful?​

• they make responsibilities and accountabilities clear
• the earlier those models introduced, the longer they will help the project
• they balance resource allocations to avoid the single point of failures or burnouts

Dropbox's design research team grew from 4 members to 30+ today. How do they scale the efforts healthily, even when the headcount for the team is limited? More researches usually mean more harm if they are done improperly.

And the major challenges for the scaling process are

1. Untrained researchers will ask bad questions, misinterpret data, and make bad decisions.
2. User researches happen anyway, with or without the research operation team.
3. What kind of user research should be done by non-professionals, and what should be done by professionals?

Dropbox's Solution​

1. Provide the right tool for unmoderated research. e.g. UserTesting, UserZoom, Lookback.
2. Real World Wednesday. Like research speed dating, 5 researchers meet 5 users for 15 mins for each.
3. Provide internal consultancy.

How to provide internal consultancy?​

1. Documentation: logistics, templates, privacy and security, tips and best practices (a lot of how-tos).
2. On-on-one consulting

Result​

• For professional research operation team
  • they get more time and energy to focus on bigger strategic projects
• For teams and products
  • they conduct more researches
  • they build better products
  • they are less blocked by the research operation team
  • they respect the research operation team more
• For users
  • they get more empathy

Risks​

• Still, bad researches result in bad decisions.
• It's hard to ensure the quality.

Dropbox's user research team has grown from 4 members to over 30 today. How can they scale their efforts healthily when the team size is limited? If research is done improperly, more user research often leads to more harm.

The main challenges faced during scaling are:

1. Untrained researchers may ask the wrong questions, misinterpret data, and make incorrect decisions.
2. User research will still occur, regardless of whether there is a research team.
3. What type of user research should be conducted by non-professionals, and what type should be conducted by professionals?

Dropbox's Solutions​

1. Provide appropriate tools for unregulated research, such as UserTesting, UserZoom, Lookback, etc.
2. "Real World Wednesdays." Like speed dating, 5 researchers meet with 5 users for 15 minutes each.
3. Offer internal consulting.

How to Provide Internal Consulting?​

1. Documentation: logistics, templates, privacy and security, tips and best practices (a wealth of how-tos).
2. One-on-one consulting.

Results​

• Professional user research team
  • They have more time and energy to focus on larger strategic projects.
• For the team and product
  • They can conduct more research.
  • They create better products.
  • They are less blocked by the research team.
  • They have greater respect for the research team.
• For users
  • They receive more empathy.

Risks​

• Poor research can still lead to incorrect decisions.
• It is difficult to ensure quality.

To enjoy a life of building software, media, and community as a ==hobby (all things here are NOT related to my job) / for pure pleasure== - why build personal infrastructure? And what are the strategies and executions to grow hobby projects? What is my current progress?

Everything starts from playing​

Primary school​

Playing Chinese copy of Tamiya mini 4WD

And play computer games on DOS.

Introduction to programming with Macromedia Authorware

Middle School​

• Built a website with Microsoft FrontPage to track Iraq War 2003
• Built a text-based game with QBasic on Digital Dictionary

And then you can play in the classroom behind piles of textbooks :)

High School​

• Built with Lego Mindstorm RCX for FIRST Lego League Challenge

Lego Robots	Challenge

College & Grad School​

• used SQL injection to add 20 CNY credits to our school's meal card for my roommates. However, two days later, they were called by the cashier... LOL.

• sniffed an audiobook app and then, based on its API, built my own Android App to get free audios.

Why is programming fun?​

… the fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning.

— The Mythical Man-month

OK, that's fun. How is the hobby related to Personal Infra?​

• Builders build for themselves anyway. Why not share them with others?
• Builders build things anyway. Why not make them big?

Meanwhile, I came across some mind-blowing articles.

• Ben Thompson on Amazon's Business Model
• Stephen Wolfram: Seeking the Productive Life: Some Details of My Personal Infrastructure

I conclude:

• Amazon's flywheel is built for business with high fixed costs and optimizing returns to scale.
• My life is with high fixed costs (limited time and energy), and scale economy is worth pursuing in this case.
• I can achieve more than I think with a powerful personal infrastructure that gains progressive advantages over time.

Plus, some take-away from my previous pre-PMF startup experience

1. Seek retention-first, data-driven growth. Sean Ellis: "Focusing on customer acquisition over 'awareness' takes discipline… At a certain scale, awareness/brand building makes sense. However, for the first year or two it's a total waste of money."
2. Table stakes are table stakes. You don't need VC to build initial product. Ideas are just ideas. Build your team, build your product, and collaborate with people.

Developing Personal Infra Strategy​

Execution​

Here is the architecture of my hobby projects.

Tech Stack​

Technologies: React, React Native Expo, GraphQL, KOA, TypeScript, AVA, Webpack, Airflow, MongoDB, Python Pandas and Flask, svelte, Metabase, Golang, etc.

Servers and APIs: Heroku, DigitalOcean, Azure, AWS, Github Pages, BunnyCDN.

Being an early majority to adopt proven new tech

System Architecture​

Focus on building, not wasting time on SRE

Example 1

Example 2

"This architecture is not future-proof! / does not scale!"

Well...

• Services are mostly stateless and horizontal scalable
• Service collocation is a problem but you have to segregate them anyway when collaborating with various people and achieving Personal IaaS (individually-sellable).
• Can always evolve to kubernetes.

Personal Root Metrics​

Living a balanced life and keep everything on track, measured by data

The key metrics for a "retention-first growth" is cohort analysis.

Benchmarks for reference:

• Values in unit of %

Results​

Products

• Beancount.io: Double-entry bookkeeping made easy for living your best financial life
• touchbase.ai: Personal CRM: Smartly engage more in meaningful relationships
• coderoma.com: One coding challenge per day

Framework

• onefx.js.org: Building web & mobile apps with speed & quality

Helped my friends' projects to start from scratch

• CocuSocial: Discover a different food and drink experience
• helped my day job at IoTeX to build staking portal, blockchain explorer, desktop wallet, etc.
• not to mention some other failed projects...

Media:

• TianPan.co: Startup Engineering | 硅谷.io: 创业工程学

Community:

• github.com/puncsky/system-design-and-architecture

Can I use your projects or join your community?​

👍 Definitely and welcome! They are mostly open sourced or open for registration. Thank you for becoming our valued customer or community member!

👏 Feedback is highly appreciated!

❤️ Like it? Check this article at https://tianpan.co and follow me on https://twitter.com/intent/follow?original_referer=https%3A%2F%2Ftianpan.co%2F&region=follow_link&screen_name=tianpan10x :)

Why do some people want to make money by becoming entrepreneurs? Because they want to beat the market—achieving returns that exceed the market at a cost lower than the market—meaning they want to obtain a profit margin higher than the market. The price exceeding the market comes from the scarcity/uniqueness of a product or service; to achieve uniqueness, one must innovate. Therefore, to become an entrepreneur, one must at least be an innovator.

Most companies succeed because they know how to continuously draw inspiration from the right things and consistently generate new ideas. How can one identify the most suitable sources of innovation to outperform competitors and stand out in the industry?

Seven Sources of Innovation​

• Internal

  • Unexpected occurrences: For example, when there was a sudden surge in the purchase of home appliances, Macy's limited sales while Bloomingdale's seized the opportunity to expand its appliance department, thereby increasing profits.

  • Changes in the market and industry: For instance, when the automotive market globalized, Volvo also followed suit, performing better than Citroën, which did not globalize quickly.

  • Weak links in processes: Pharmaceutical sales representative William Connor noticed a troublesome aspect of eye surgery: hemorrhage of the eye ligament. He suggested using enzymes to dissolve the ligament instead of cutting it, significantly reducing surgical risks, and this innovation was widely accepted in the field of ophthalmology. This innovation addressing a shortcoming brought his company substantial profits.

  • The gap between reality and perception (Is TK also a disciple of Drucker?): For example, early on, ferry freight mistakenly believed that the key to reducing time was to increase sailing speed, but in reality, this would lead to skyrocketing costs; the key issue was actually to reduce the time the ship was idle in port.

• External: For example, politics, academia, science

  • Changes in social concepts: The growing enthusiasm for environmental protection and high technology has made the electric vehicle market thrive.
  • Changes in demographic structure: For instance, the increase in digital natives in China and the demand for online communities gave rise to Bilibili.
  • Hybridization of new knowledge: For example, computers are a hybrid product of mathematics, electronics, and programming technology developed over hundreds of years.

Both Small and Large Companies Need Innovation​

A newly established company needs specific goals and plans, as detailed in The Five Stages of Company Building.

In the early stages of entrepreneurship, entrepreneurs should try different fields to find the right market. It is very likely that you will ultimately succeed in a field you never considered. The second step is to establish the correct financial focus. Ensuring that the company has sufficient funds to address issues when they arise is extremely important. The final step is to build a trustworthy management team for the company. This team should be established before the company’s team grows.

Not only small businesses need reform and innovation, but large industries also need fresh blood. In the initial stages, they should standardize the rules for innovating and phasing out the old within the company. Secondly, the newly innovated projects should be managed by new leaders. Lastly, companies should establish reward mechanisms to help improve employee performance and effectively review the impact of innovations.

Four Innovation Strategies​

All In (Fustest with the mostest)​

A wise entrepreneur should aim to become a pioneer in their industry, putting everything on the line to lead the way. Hoffmann-La Roche had a small chemical company, but he cleverly identified the business opportunity in the vitamin industry. Therefore, to produce and sell vitamins, he invested a large sum of money and hired many experts. Although it sounded very risky, this "gamble" ultimately paid off, and he remained a leader in the vitamin industry for 60 years.

Hit Them Where They Ain’t​

Identifying vulnerabilities that competitors overlook is not easy, but there are two ways to achieve this. The first is to imitate competitors' ideas using newer and more appealing methods. For example, IBM imitated the ideas of competitor ENIAC and added more innovative concepts, ultimately profiting from it. Additionally, some companies can win by targeting their opponents' weaknesses, which is especially effective against complacent large companies.

Ecological Niches​

This originally is a biological concept: Ecological niche refers to the environment a species inhabits and its lifestyle habits. Each species has its unique ecological niche, distinguishing it from other species.

A company that specializes in an irreplaceable field is more likely to succeed. A good example is the enzymes developed by William Connors. These enzymes later became a crucial step in cataract surgery. However, it is worth noting that this company could also lose its absolute advantage in the industry if competitors develop substitute drugs.

Changing Values and Characteristics​

To increase demand for your product, you do not necessarily need to change the product itself. Instead, finding a method that better aligns with consumer interests may be more important. Entrepreneurs should understand what makes consumers willing to pay. For example, Gillette's strategy of offering razors for free while charging for blades was based on the company's realization that consumers were unwilling to pay more for blades than the razor itself.

The intercompany payment process is complex and requires integration with various systems, especially accounting software. Routable is dedicated to solving this problem for the middle market (companies with $100 million to$500 million in revenue). What sets Routable apart is that it helps you manage both workflows and the actual flow of money. Recently, startups focused on bill management have been gaining traction. Should we consider buying some stock in bill.com?