Why data partition and routing?​

large dataset ⟶ scale out ⟶ data shard / partition ⟶ 1) routing for data access 2) replica for availability

• Pros
  • availability
  • read(parallelization, single read efficiency)
• Cons
  • consistency

How to do data partition and routing?​

The routing abstract model is essentially just two maps: 1) key-partition map 2) partition-machine map

Hash partition​

1. hash and mod

   • (+) simple
   • (-) flexibility (tight coupling two maps: adding and removing nodes (partition-machine map) disrupt existing key-partition map)

2. Virtual buckets: key--(hash)-->vBucket, vBucket--(table lookup)-->servers

   • Usercase: Membase a.k.a Couchbase, Riak
   • (+) flexibility, decoupling two maps
   • (-) centralized lookup table

3. Consistent hashing and DHT

   • [Chord] implementation
   • virtual nodes: for load balance in heterogeneous data center
   • Usercase: Dynamo, Cassandra
   • (+) flexibility, hashing space decouples two maps. two maps use the same hash, but adding and removing nodes ==only impact succeeding nodes==.
   • (-) network complexity, hard to maintain

Range partition​

sort by primary key, shard by range of primary key

range-server lookup table (e.g. HBase .META. table) + local tree-based index (e.g. LSM, B+)

(+) search for a range (-) log(n)

Usercase: Yahoo PNUTS, Azure, Bigtable

The key to an effective strategy lies in channeling limited energy into the points that can generate the most impact. Good resources should be used where they matter most.

So we must ask, where does this energy come from? Here are some common sources:

• leverage
• proximate objectives
• chain-link systems. Systems without weaknesses are extremely difficult to replicate, such as IKEA.
• design
• focus strategy, providing specific services for particular groups
• growth. The increase in team size should not be deliberate but rather a natural result of the company's product growth.
• advantage. Comparative advantage is domain-specific; a champion runner may not excel in high jumping.
• dynamics. Exploit a wave of exogenous change.
• inertia and entropy.

• "Coherent and complementary actions" mean that these actions directly support each other to create synergy. For example, as a manager, the principle behind introducing processes imposed on others is, ==“I will never ask you to do anything that does not help you in your core job.”==
• Strategic collaboration is not arranged on the spot; it is deliberately designed and centrally imposed on the system.
• Centralization is a bad thing, but a completely fragmented approach is also ineffective, as the interests of various sub-organizations differ. For instance, in manufacturing, sales may prefer to please customers with expedited large orders, while the production department prefers to output steadily and undisturbed over the long term. It is impossible to handle both urgent large orders and maintain steady production simultaneously.
• Organizational collaboration is time-consuming and labor-intensive; if there are not enough benefits, it should not be pursued. Smart organizations do not aim for 100% communication among everyone but rather strive for just the right amount of coordination.

Three Fundamental Elements

1. Diagnosis: Simplifying the problem and identifying challenges.
2. Guiding Policies: How to respond to challenges?
3. Coherent Actions: A series of actions that mutually reinforce each other under the guidance of principles.

Examples

In business, the challenge is usually dealing with change and competition.

1. Diagnosing the specific structure of the challenge rather than simply naming performance goals.
2. Choosing an overall guiding policy for dealing with the situation that builds on or creates some type of leverage or advantage.
3. Designing a configuration of actions and resource allocations that implement the chosen guiding policy.

In many large organizations, the challenge is often diagnosed as internal.

1. The organization’s competitive problems may be much lighter than the obstacles imposed by its own outdated routines, bureaucracy, pools of entrenched interests, lack of cooperation across units, and plain-old bad management.
2. Reorganization and renewal.
3. Changes in people, power, and procedures.

Amazon

There are mainly three reasons:

1. Making choices is a painful process
   • When DEC made strategic choices, there were three factions: "Boxes," "Chips," and "Solution." The voting led to Condorcet’s paradox. The voting paradox: democratic voting can yield irrational results like A > B > C > A. CEO Ken Olsen sought consensus, but fundamentally, you cannot make a sub-organization automatically give up its own enthusiasm. So in the end, everyone chose a compromise solution: "DEC is committed to providing high-quality products and services and being a leader in data processing." This is just empty rhetoric.
   • When Eisenhower campaigned for president in 1952, he promised to withdraw the Soviet Union from Eastern Europe, but after winning the election and conducting extensive research, he made the difficult decision to abandon that promise.
   • Intel CEO Andy Grove shifted the company from producing dynamic random access memory (DRAM) to focusing on microprocessors.
2. People prefer to use templates without thinking
   • Max Weber, the father of sociology, believed in distinguishing formal leaders from those who lead by personal charisma.
   • Peter Drucker, one of the foremost thinkers about management, said, "Effective leadership doesn’t depend on charisma. Dwight Eisenhower, George Marshall, and Harry Truman were singularly effective leaders, yet none possessed any more charisma than a dead mackerel.… Charisma does not by itself guarantee effectiveness as a leader."
   • The academic requirements for leadership generally are:
     1. Develops or has a vision
     2. Inspires people to sacrifice (change) for the good of the organization
     3. Empowers people to accomplish the vision
   • Leadership and strategy have commonalities, but they must not be confused. For example:
     • Leadership encourages self-sacrifice and personal transformation while feeling good about it.
     • Strategy is about articulating what kind of transformation is worth pursuing.
   • A similar confusion arises when having a strategy is mistaken for having a good strategy. Countless books and tutorials provide templates for people to fill in mindlessly, resulting in numerous bad strategies.
3. New Thought: People often believe that human will can overcome all, thinking that attitude determines everything
   • Zhang Defen's "Law of Attraction," the New Thought movement, and the fantasy of "positive energy" regarding outcomes lead people to neglect the real efforts that contribute to results, serving as a form of spiritual opium. We should focus more on imagining the process of doing things, which is akin to simulation training.

A key role of product management (PM), whether as the product-focused founder (CEO, CTO) or the PM leader, is making sure product development efforts are ==focused==.

• finite resources to market needs in the march towards product-market fit
• working furiously to maintain a competitive lead

Is process a bad thing? My guiding principle or golden rule of process is to never ask for something from someone that does not directly help them to get their own work done.

Product Roadmap in Telemetry​

Telemetry Framework​

When the tool overwhelms it isn’t used and so it doesn’t matter - e.g. a giant Gantt chart. The telemetry spreadsheet is more helpful.

This tool is the right level of complexity for projects from 10 to 5000 people in my experience.

benefits: it is a resources allocation guideline

• Any given person can look at their own work and know where they are and where it fits in
• At any time, one can see how much of the time (resource) overall is going to support a given project or goal
• Know which tasks are too far out, too big
• Know which resources are over-constrained

Very Large JS App = a lot of developers + large codebase

How to deal with a lot of developers?​

empathy​

What is a ==senior engineer==? A team of senior engineers without junior engineers is a team of engineers

1. being senior means is that I’d be able to solve almost every problem that somebody might throw at me.
2. make the junior engineers eventually be senior engineers.

what’s the next step of a senior engineer?

1. senior: “I know how I would solve the problem” and because I know how I would solve it I could also teach someone else to do it.
2. next level: “I know how others would solve the problem “

good programming model​

how people write software, e.g. react/redux, npm. Here comes a model that affects all large JS apps - code splitting.

1. People have to think what to bundle and when to load
2. ==route based code splitting==
3. But, what if it is not enough?
   1. lazy loaded every single component of our website
   2. How does Google do? split them by rendering logic, and by application logic. ==simply server side render a page, and then whatever was actually rendered, triggers downloading the associated application bundles.== Google does not do isomorphic rendering - no double rendering

How to deal with a large codebase?​

==Code Removability/Delete-ability==​

e.g. CSS is bad in code removability

1. one big css file. There is this selector in there. Who really knows whether that still matches anything in your app? So, you end up just keeping it there.
2. People thus created CSS-in-JS

==avoid central configuration of your application at all cost==

1. Bad example
   1. central routes configuration
   2. central webpack.config.js
2. Good example
   1. decentralized package.json

avoid central import problem: router imports component A, B, and C

1. to solve this problem, do ==“enhance” instead of “import”==
2. However, developers still have to decide when to enhance and when to import. Since this might lead to very bad situations, we make “enhance” illegal, nobody gets to use it–with one exception: generated code.

avoid base bundle pile of trash

1. e.g. base bundle should never contain UI code
2. Solve this problem with forbidden dependency tests
3. ==most straight forward way must be the right way; otherwise add a test that ensures the right way.==

Be careful with abstractions​

We have to become good at finding the right abstractions: Empathy and experience -> Right abstractions

A bad strategy is formalism, characterized by four fundamental traits

1. Empty rhetoric ==obfuscation== ==fluff== Strategy should not be a mere accumulation of grandiose terms.

   • e.g. a major bank “Our fundamental strategy is one of customer-centric intermediation.” = “Our bank’s fundamental strategy is being a bank.”

2. Failure to confront challenges

   • If you fail to identify and analyze the obstacles, you don’t have a strategy. Instead, you have either a stretch goal, a budget, or a list of things you wish would happen.
   • This reminds me that the choice of root level metrics must be very careful, ==If you can't measure it, you can't improve it==
   • A positive example is DARPA.

3. Mistaking goals for strategy

   • e.g. a 20/20 goal - We will grow revenue by at least 20% each year. We will maintain a profit margin of at least 20%.
   • Metrics are not strategy.

4. Sub-goals that are irrelevant or unrealistic

   • Goal is the overall objective, Objective is the sub-goal.
   • Good strategy works by focusing energy and resources on one, or a very few, pivotal objectives whose accomplishment will lead to a cascade of favorable outcomes.
   • What kind of sub-goals are bad sub-goals?
     1. Dog’s Dinner Objectives - putting everything together without distinction.
     2. Blue-Sky Objectives - unrealistic aspirations.

• CORS allows a site A (data provider) to give permission to site B to read (potentially private) data from site A (using the visitor's browser and credentials).
• CSP allows a site to prevent itself (data consumer) from loading (potentially malicious) content from unexpected sources (e.g. as a defence against XSS).

There is nothing either good or bad, but thinking makes it so. by Shakespeare in Hamlet

The legend of David and Goliath. From a secular perspective, David is weak and Goliath is strong; David is a novice, while Goliath is an experienced giant warrior. Yet, David defeats Goliath with a sling.

The story of Walmart defeating Kmart. The traditional wisdom in American retail is that a supermarket with a wide variety of products needs to be located in an area with at least 100,000 people. However, Walmart can operate in areas with smaller populations. Why? Because Walmart has more efficient supply chain management, creating an organic network between its stores; whereas Kmart's stores are not as closely connected, leading to higher inventory management costs and less bargaining power in procurement.

During the Cold War arms race, Andy Marshall's strategy against the Soviet Union was to develop comparative advantages by leveraging America's economic and technological strengths. This involved developing technologies that would be costly for the opponent to counter but would not pose a threat to the U.S., such as improving missile accuracy and developing silent submarines. Ultimately, this strategy led to the downfall of the Soviet Union.