Skip to main content

Introduction to Beancount.io

· 7 min read

Why bookkeeping?

Everyone has advice about how to manage money. Search Google for "manage money," and you'll get back over 1,690,000,000 links. You'll find tons of life-hacking or self-help articles and books. You'll find professional coaches or courses witch will coach you for a fee. You'll find financial and investment services. Feel free to try what appeals to you and grow your assets through trials and errors.

I think the most important thing to remember is that asking the question of money comes from fear and self-doubt. We all fear change. We all doubt our ability to make more money.

Instead of spending time worrying and doubting, focusing the opposite — your confidence. If you are playing poker and with few chips, you can only make small bets and only win a small amount of money. When you have a lot of chips, you can make big bets and win big. You have more room for taking risks. You can try things which you cannot try when you have fewer chips.

Here is the magic - by understanding more of your financial status, you gain confidence! With more confidence, we can make better judgment and would like to bet the best amount for more significant success, and then win more.

Expenses

Know your expenses and plan for next spending

Where is the end of the wining-more concern? People often talk about the buzzword of financial freedom. However, talk is cheap, but bookkeeping precisely answers the question.

Four main financial statements for the overview of financial status

Unfortunately, bookkeeping is not easy in our modern life. We are in a new age of abundance. We have a lot of accounts - cash, bank accounts, payment apps, credit cards, stock or crypto broker accounts, discount cards, … We have assets like houses, cars, gold, jewelry, … To make things even worse, some of us may live across countries and have to deal with different currencies. How could we draw an accurate map of our financial life and navigate through the future uncertainties?

By "accurate map of our financial life," I mean these four primary financial statements:

  1. Income Statement: It shows how much revenue we earned over a specific period. This statement is usually considered the most important of the financial statements because it reflects the operating results.
  2. Balance Sheets: It answers how much assets, liabilities, and equity of the entity we have. This statement is the second most important because it reports the liquidity and capitalization of our assets.
  3. Cash Flow Statements: It reports our inflows and outflows of cash and answers whether we generated cash. We need enough cash on hand to pay expenses and purchase assets.
  4. Equity Statement: This is not helpful for your personal accounting. However, for a company, this statement reports how it distributes equity among stakeholders.

Income Statement

Balance Sheet

With beancount.io, you can quickly generate statements like the above. But wait… How to prepare data for these statements?

Double-entry Bookkeeping for Correctness

To ensure the accuracy and internalize the error detection into the system, double-entry bookkeeping requires every entry to an account has at-least a corresponding entry to a different account. One transaction involves at least two accounts with two operations - debit (+) and credit (-).

1970-01-01 open Income:BeancountCorp
1970-01-01 open Assets:Cash
1970-01-01 open Expenses:Food
1970-01-01 open Assets:Receivables:Alice
1970-01-01 open Assets:Receivables:Bob
1970-01-01 open Assets:Receivables:Charlie
1970-01-01 open Liabilities:CreditCard

2019-05-31 * "BeancountCorp" "Salary of May 15th to May 31st"
Income:BeancountCorp -888 USD
Assets:Cash 888 USD

2019-07-12 * "Popeyes chicken sandwiches" "dinner with Alice, Bob, and Charlie"
Expenses:Food 20 USD
Assets:Receivables:Alice 20 USD
Assets:Receivables:Bob 20 USD
Assets:Receivables:Charlie 20 USD
Liabilities:CreditCard -80 USD

As you can see in the two examples above, every transaction must fulfill the accounting equation.

Assets = Liabilities + Equity(aka Net Assets)

We used the Beancount syntax by Martin Blais and the web project Fava by Jakob Schnitzer to build this website. And it will alert you if any transaction has any legs not summing to zero.

Error Alert

Now you understand how we enforce the correctness of the ledger. But you may ask what are those "accounts"?

Accounts for money as buckets for water

Thinking your assets as water running in and out of different buckets and "accounts" are those buckets holding your money. With double-entry bookkeeping, it becomes obvious how money is flowing across different accounts, just like how water is flowing across different buckets.

Beancount.io introduces five kinds of accounts.

  1. Income — Its amount is always negative or in debit. This is because you are making money, and then the money is debiting from "Income" account and crediting to your "Assets."
  2. Expenses — Its amount is always positive or in credit. This is because you are spending money, and the money is flowing from the "Assets" or "Liabilities" to the "Expenses."
  3. Liabilities — Its amount is positive or zero. Your credit card liabilities are a good example, which rises and falls in cycles.
  4. Assets — Its amount is positive or zero. Your cash or houses are always worthing some prices.
  5. Equity — Your net assets. The system will calculate automatically for you. Equity = Assets - Liabilities and it reflects how wealthy you are.

Now you can open your customized accounts with those keywords above:

1970-01-01 open Assets:Cash
1970-01-01 open Assets:Stock:Robinhood
1970-01-01 open Assets:Crypto:Coinbase
1970-01-01 open Expenses:Transportation:Taxi
1970-01-01 open Equity:OpeningBalance

Commodities: Tracking your investment

Yes, you can track your investment with beancount.io. For example, we buy 10 Bitcoins at the price of $100 in 2014:

2014-08-08 * "Buy 10 Bitcoin"
Assets:Trade:Cash -1000.00 USD
Assets:Trade:Positions 10 BTC {100.00 USD}

And then three years later, you sell them (originally with costs of 100perunitannotatedwith100.00USD)atthepriceof100 per unit annotated with `{100.00 USD}`) at the price of **10,000 per unit** annotated with @ 10,000.00 USD.

2017-12-12 * "Sell 2 Bitcoin"
Assets:Trade:Positions -2 BTC {100.00 USD} @ 10,000.00 USD
Assets:Trade:Cash 20,000.00 USD
Income:Trade:PnL -19,800.00 USD

Or the same transaction with @@ 20,000.00 USD means that at the price of $20,000 in total.

2017-12-12 * "Sell 2 Bitcoin"
Assets:Trade:Positions -2 BTC {100.00 USD} @@ 20,000.00 USD
Assets:Trade:Cash 20,000.00 USD
Income:Trade:PnL -19,800.00 USD

The sum of all legs of the transaction, including -2 BTC {100.00 USD}, are still, as always, zero.

The costs {100.00 USD} tag is important because you might have bought the same commodity at different costs.

100 BTC {10.00 USD, 2012-08-08}
10 BTC {100.00 USD, 2014-08-08}

If you want to simplify the process, you can set up the account at the beginning with FIFO or LIFO. FIFO stands for first in, first out, while LIFO stands for last in, first out. In the US, IRS uses FIFO to calculate your PnL and tax accordingly.

1970-01-01 open Assets:Trade:Positions "FIFO"

And then when you sell it in shorthand like -2 BTC {}, beancount will apply FIFO strategy automatically and sell the oldest commodity.

Beancount.io

Beancount.io is such a cloud service for recording your financial transactions in text files, visualize them into financial statements (income statement, balance sheet, trial balance, etc.), and helps you live a better financial life. Sign up now - It's in Promotional Period and Free!

The Company's Technology and Market Quadrant Diagram and Gravitational Directions

· 4 min read

Technology

Technology

Market

Market

Leaders

Leaders

The Innovator's Dilemma

The Innovator's Dilemma

Challengers

Challengers

Dragonslayers

Dragonslayers

  • The gravitational direction of technology is towards mediocrity: as technology inflates, excellent technologies tend to become mediocre, and mediocre companies adopt excellent technologies.
  • The gravitational direction of the market is the Matthew effect: markets with a presence will grow larger, while those without will shrink.

Designing a metric system

· 13 min read

Requirements

Log v.s Metric: A log is an event that happened, and a metric is a measurement of the health of a system.

We are assuming that this system’s purpose is to serve metrics - namely, counters, conversion rate, timers, etc. for monitoring the system performance and health. If the conversion rate drops drastically, the system should alert the on-call.

  1. Monitoring business metrics like signup funnel’s conversion rate
  2. Supporting various queries, like on different platforms (IE/Chrome/Safari, iOS/Android/Desktop, etc.)
  3. data visualization
  4. Scalability and Availability

Architecture

Two ways to build the system:

  1. Push Model: Influx/Telegraf/Grafana
  2. Pull Model: Prometheus/Grafana

The pull model is more scalable because it decreases the number of requests going into the metrics databases - there is no hot path and concurrency issue.

Server Farm

Server Farm

write

write

telegraf

telegraf

InfluxDB

InfluxDB

REST API

REST API

Grafana

Grafana

InfluxDB Push Model

InfluxDB Push Model

Prometheus Pull Model

Prometheus Pull Model

Application

Application

Exporter

Exporter

client library

client library

3rd Party


Application

3rd Party<br>Application

pull

pull

Prometheus

Prometheus

Retrieval

Retrieval

Service Discovery

Service Discovery

Storage

Storage

PromQL

PromQL

Alertmanager

Alertmanager

Web UI / Grafana / API Clients

Web UI / Grafana / API Clients

PagerDuty

PagerDuty

Email

Email

Features and Components

Measuring Sign-up Funnel

Take a four-step sign up on the mobile app for example

INPUT_PHONE_NUMBER -> VERIFY_SMS_CODE -> INPUT_NAME -> INPUT_PASSWORD

Every step has IMPRESSION and POST_VERIFICATION phases. And emit metrics like this:

{
"sign_up_session_id": "uuid",
"step": "VERIFY_SMS_CODE",
"os": "iOS",
"phase": "POST_VERIFICATION",
"status": "SUCCESS",
// ... ts, contexts, ...
}

Consequently, we can query the overall conversion rate of VERIFY_SMS_CODE step on iOS like

(counts of step=VERIFY_SMS_CODE, os=iOS, status: SUCCESS, phase: POST_VERIFICATION) / (counts of step=VERIFY_SMS_CODE, os=iOS, phase: IMPRESSION)

Data Visualization

Graphana is mature enough for the data visualization work. If you do not want to expose the whole site, you can use Embed Panel with iframe.

Designing Square Cash or PayPal Money Transfer System

· 19 min read

Clarifying Requirements

Designing a service money transfer backend system like Square Cash (we will call this system Cash App below) or PayPal to

  1. Deposit from and payout to bank
  2. Transfer between accounts
  3. High scalability and availability
  4. i18n: language, timezone, currency exchange
  5. Deduplication for non-idempotent APIs and for at-least-once delivery.
  6. Consistency across multiple data sources.

Architecture

AWS CloudHSM

AWS CloudHSM

Presentation Layer

Presentation Layer

SDK/Docs

SDK/Docs

mobile-dashboard

mobile-dashboard

web-dashboard

web-dashboard

dashboard-client

dashboard-client

mobile-wallet

mobile-wallet

web-wallet

web-wallet

wallet-client

wallet-client

Merchant 


User

Merchant <br>User

End User

End User

web-chrome-extension

web-chrome-extension

Operators

Operators

payment

payment

task-queue

task-queue

financial-reporter

financial-reporter

payment-gateway

payment-gateway

banks / 


vendors

[Not supported by viewer]

side-effect maker

side-effect maker

help service portal

help service portal

User


Profiles


AuthDB


[Not supported by viewer]

api-gateway


monolithic


api-gateway<br>monolithic<br>

Payment


DB


Payment<br>DB<br>

Aurora

Aurora

risk control

risk control

risk control

risk control

Event
Queue

[Not supported by viewer]

Features and Components

Payment Service

The payment data model is essentially “double-entry bookkeeping”. Every entry to an account requires a corresponding and opposite entry to a different account. Sum of all debit and credit equals to zero.

Deposit and Payout

Transaction: new user Jane Doe deposits $100 from bank to Cash App. This one transaction involves those DB entries:

bookkeeping table (for history)

+ debit, USD, 100, CashAppAccountNumber, txId
- credit, USD, 100, RoutingNumber:AccountNumber, txId

transaction table

txId, timestamp, status(pending/confirmed), [bookkeeping entries], narration

Once the bank confirmed the transaction, update the pending status above and the following balance sheet in one transaction.

balance sheet

CashAppAccountNumber, USD, 100

Transfer between accounts within Cash App

Similar to the case above, but there is no pending state because we do not need the slow external system to change their state. All changes in bookkeeping table, transaction table, and balance sheet table happen in one transaction.

i18n

We solve the i18n problems in 3 dimensions.

  1. Language: All texts like copywriting, push notifications, emails are picked up according to the accept-language header.
  2. Timezones: All server timezones are in UTC. We transform timestamps to the local timezone in the client-side.
  3. Currency: All user transferring transactions must be in the same currency. If they want to move across currencies, they have to exchange the currency first, in a rate that is favorable to the Cash App.

For example, Jane Doe wants to exchange 1 USD with 6.8 CNY with 0.2

bookkeeping table

- credit, USD, 1, CashAppAccountNumber, txId
+ debit, CNY, 6.8, CashAppAccountNumber, txId, @7.55 CNY/USD
+ debit, USD, 0.1, ExpensesOfExchangeAccountNumber, txId

Transaction table, balance sheet, etc. are similar to the transaction discussed in Deposit and Payout. The major difference is that the bank or the vendor provides the exchange service.

How to sync across the transaction table and external banks and vendors?

  • retry with idempotency to improve the success rate of the external calls and ensure no duplicate orders.
  • two ways to check if the PENDING orders are filled or failed.
    1. poll: cronjobs (SWF, Airflow, Cadence, etc.) to poll the status for PENDING orders.
    2. callback: provide a callback API for the external vendors.
  • Graceful shutdown. The bank gateway calls may take tens of seconds to finish, and restarting the servers may resume unfinished transactions from the database. The process may create too many connections. To reduce connections, before the shutdown, stop accepting new requests and wait for the existing outgoing ones to wrap up.

Deduplication

Why is Deduplication a concern?

  1. not all endpoints are idempotent
  2. Event queue may be at-least-once.

not all endpoints are idempotent: what if the external system is not idempotent?

For the poll case above, if the external gateway does not support idempotent APIs, in order not to flood with duplicate entries, we must keep record of the order ID or the reference ID the external system gives us with 200, and query GET by the order ID instead of POST all the time.

For the callback case, we can ensure we implement with idempotent APIs, and we mutate pending to confirmed anyway.

Event queue may be at-least-once

  • For the even queue, we can use an exactly-once Kafka with the producer throughput declines only by 3%.
  • In the database layer, we can use idempotency key or deduplication key.
  • In the service layer, we can use Redis key-value store.

Availability and Scalability

Designing payment webhook

· 4 min read

1. Clarifying Requirements

  1. Webhook will call the merchant back once the payment succeeds.
    1. Merchant developer registers webhook information with us.
    2. Make a POST HTTP request to the webhooks reliably and securely.
  2. High availability, error-handling, and failure-resilience.
    1. Async design. Assuming that the servers of merchants are located across the world, and may have a very high latency like 15s.
    2. At-least-once delivery. Idempotent key.
    3. Order does not matter.
    4. Robust & predictable retry and short-circuit.
  3. Security, observability & scalability
    1. Anti-spoofing.
    2. Notify the merchant when their receivers are broken.
    3. easy to extend and scale.

2. Sketch out the high-level design

async design + retry + queuing + observability + security

3. Features and Components

Core Features

  1. Users go to dashboard frontend to register webhook information with us - like the URL to call, the scope of events they want to subscribe, and then get an API key from us.
  2. When there is a new event, publish it into the queue and then get consumed by callers. Callers get the registration and make the HTTP call to external services.

Webhook callers

  1. Subscribe to the event queue for payment success events published by a payment state machine or other services.

  2. Once callers accept an event, fetch webhook URI, secret, and settings from the user settings service. Prepare the request based on those settings. For security...

  • All webhooks from user settings must be in HTTPs

  • If the payload is huge, the prospect latency is high, and we wants to make sure the target reciever is alive, we can verify its existance with a ping carrying a challenge. e.g. Dropbox verifies webhook endpoints by sending a GET request with a “challenge” param (a random string) encoded in the URL, which your endpoint is required to echo back as a response.

  • All callback requests are with header x-webhook-signature. So that the receiver can authenticate the request.

    • For symetric signature, we can use HMAC/SHA256 signature. Its value is HMAC(webhook secret, raw request payload);. Telegram takes this.
    • For asymmetric signature, we can use RSA/SHA256 signature. Its value is RSA(webhook private key, raw request payload); Stripe takes this.
    • If it's sensitive information, we can also consider encryption for the payload instead of just signing.
  1. Make an HTTP POST request to the external merchant's endpoints with event payload and security headers.

API Definition

// POST https://example.com/webhook/
{
"id": 1,
"scheduled_for": "2017-01-31T20:50:02Z",
"event": {
"id": "24934862-d980-46cb-9402-43c81b0cdba6",
"resource": "event",
"type": "charge:created",
"api_version": "2018-03-22",
"created_at": "2017-01-31T20:49:02Z",
"data": {
"code": "66BEOV2A", // or order ID the user need to fulfill
"name": "The Sovereign Individual",
"description": "Mastering the Transition to the Information Age",
"hosted_url": "https://commerce.coinbase.com/charges/66BEOV2A",
"created_at": "2017-01-31T20:49:02Z",
"expires_at": "2017-01-31T21:49:02Z",
"metadata": {},
"pricing_type": "CNY",
"payments": [
// ...
],
"addresses": {
// ...
}
}
}
}

The merchant server should respond with a 200 HTTP status code to acknowledge receipt of a webhook.

Error-handling

If there is no acknowledgment of receipt, we will retry with idempotency key and exponential backoff for up to three days. The maximum retry interval is 1 hour. If it's reaching a certain limit, short-circuit / mark it as broken. Sending out an Email to the merchant.

Metrics

The Webhook callers service emits statuses into the time-series DB for metrics.

Using Statsd + Influx DB vs. Prometheus?

  • InfluxDB: Application pushes data to InfluxDB. It has a monolithic DB for metrics and indices.
  • Prometheus: Prometheus server pulls the metrics values from the running application periodically. It uses LevelDB for indices, but each metric is stored in its own file.

Or use the expensive DataDog or other APM services if you have a generous budget.

The Fine Art of Small Talk

· 3 min read
  • After public speaking, the biggest social fear in the Western world is initiating a conversation with strangers. To conquer the fear of rejection, it helps to realize that in most cases, people appreciate it when you make an effort to speak with them.
  • In some cases, not talking can make you come off as arrogant or aloof. Initiating a conversation can be simple: first, smiling at someone; second, establishing eye contact; third, being the first to introduce yourself.
  • To approach a group of people: 1, demonstrate your interest to the group from a distance, paying attention to the speaker. 2, the group will notice and make room to include you. 3, let the group warm to you before you offer any strong opinions.
  • Guiding a conversation evokes the positive feelings that make people want to work or socialize with them. One easy way of assuming this responsibility is to act like you're a host and ask, "What's your name?". Emphasize "your" to make them feel valued.
  • The best way to improve your conversations is to ask open-ended questions, which demonstrates that you genuinely care about what they have to say.
  • A conversation will inevitably dip into an awkward silence sometimes. You can get it back to a comfortable flow by asking open-ended questions with the current contexts or the acronym FORM: family, occupation, recreation and miscellaneous.
  • To be an active listener, body language is important: avoid crossing arms, hunching shoulders, or fiddling with clothes, hair or jewelry. Instead, lean forward, nod, smile and maintain eye contact.
  • To be an active listener, vertal cues are important: engage by asking follow-up questions about the details, respond enthusiastically to express your intrest, or paraphrasing what the speaker said to clarify.
  • To end a conversation gracefully, the first thing you can do is to circle back to the highlight of your discussion. If you genuinely want to continue the discussion later, exchange contact info, state what you will do next, (if acquaintance) shake hand, and say goodbye.
  • It's important to follow through with whatever it is you say you're doing next. Otherwise, the other side may think you simply were not enjoying your time with them, which hurts feelings.
  • It is a courteous way to end a conversation with introducing your conversation partner to a new person, which will whiden his network and ensure he does not feel that you're abandoning him. Or reversely, you can ask them to introduce you to someone else.

Mobile Analytics Metrics

· 3 min read
NameDefinitionComment
DAU (Daily active users)# of Unique users per dayDownloads are misleading because 80 to 90 percent of those who download an app never return
MAU (Monthly active users)# of Unique users per monthDownloads are misleading
Stickiness(DAU / MAU ) x 100Higher stickiness = higher ROI, mobile stickiness is 20x more than mobile/desktop
Retention rate((# of customers at end of period – # of customers acquired during period) / # of customers at start of period ) x 100High retention is almost always a good thing.
Churn rate1- Customer Retention Rate
CPA (Cost per acquisition)Total Marketing Cost / Total User Acquisitionsthe lower the better
Average daily sessions per DAUhow frequently your users log into your app each day. Not always a good thing.
LTV (Lifetime value)Average value of conversion x Average # of conversions in a timeframe x Average customer valueis losing money = Boolean(LTV < CPA)
ARPU (Average revenue per user)Lifetime revenue of app/ Lifetime # of usersARPU answers when you should be earning more revenue per user
ARPPU (average revenue per paying user)Lifetime revenue of app / Lifetime # of paying users
ROI (Return on Investment)Return / Investmentstay consistent to measure relative progress year-to-year
App load timeshould <= 2 sec
User satisfactionmeasured in CSAT and NPSbetter user satisfaction = more user retention + more LTV
CSAT (customer satisfaction score)(# of satisfied customers / # of survey respondents) x 100ask customers to rate their satisfaction on a scale from 1 to 5. 4 or 5 means satisfied
NPS (Net Promoter Score)((# of promoters – # of detractors ) / # of survey respondents) x 100ask customers to rate their satisfaction on a scale from 1 to 10. Users reply 0 to 6 are detractors. Users reply 8 to 10 are promoters.
Goal achievement% Users that achieve their goals each sessionGoals can be a purchase, a signup, a share, etc.
Marketing Acquisition%, $, and dollar value % of visitors from a top marketing channel

Stratechery: Why Did Amazon Acquire Whole Foods?

· 4 min read

The answer is: Amazon wanted to buy customers for its grocery service.

Background

  • Amazon's acquisition of Whole Foods = Apple's iPhone defeating Palm

    • Do not confuse goals, strategies, and tactics — Apple's strategy:
      • It was not about making phones but about producing personal computers
      • It was not about adding features to phones but about compressing traditional phone functions into one app
      • It was not about replicating the work of carriers but about leveraging its connection with customers
    • The iPhone is the most successful product in history = Amazon is the most dominant company in history
  • Amazon's Goals

    1. Initially, Amazon.com aimed to become a leading retailer based on information products and services, starting with selling books.
    2. Then, Amazon declared, "Our vision is to be Earth's most customer-centric company, where customers can find anything they want to buy online."
    3. ==Amazon's goal is to gain a share of all economic activities==.
  • Amazon's Strategy

    • For businesses: AWS. Assume that all commercial transactions will soon be completed online.
    • For customers: Prime. Assume that high costs and diverse choices are unsustainable. With Prime, customers will not consider other alternatives.
      • However
        • The grocery industry is the largest retail category
        • The grocery industry can continuously remind consumers that there are alternatives to Amazon
  • Amazon's Tactics: Develop grocery services

Why Did Amazon Not Arrive at the Right Tactics?

BooksGroceries
High inventory units = wide selectionLow inventory units (30k - 50k)
StandardizedVaried
Non-perishablePerishable

Amazon's cost disadvantages in fresh produce

  1. Once scale is insufficient, product spoilage will incur high costs.
  2. Scale depends on the specific circumstances of each city.

Why does acquiring Whole Foods (rather than others) solve the business scale issue?

==Business fundamental component model + two basic points 1) High fixed costs 2) High returns==

  • Deconstruct the infrastructure into Minimum Sellable Units (MSUs)
  • These businesses themselves are the first and best customers of these minimum sellable units
  • Resell the minimum sellable units

AWS's three-tier architecture

ServiceFundamental ComponentS3, EC2, RDS, SNS, ...
PlatformAWSHigh fixed costs + scale returns
InfrastructureModular ComponentsData centers, servers, storage, switches, bandwidth
  • MSUs belong to S3, EC2, RDS, SNS, etc.
  • The first and best customer is amazon.com
  • Resell MSUs to non-Amazon developers

Amazon.com's three-tier architecture

ServicePackageFDA, Amazon Pay, ...
PlatformLogistics CenterHigh fixed costs + scale returns
InfrastructureModular SuppliersManufacturers, third-party suppliers, etc.
  • MSUs belong to FDA, Amazon Pay, etc.
  • The first and best customer is Amazon's first-party e-commerce
  • Resell MSUs to third-party suppliers

The insight here is that Amazon's existing grocery does not have a first and best supplier.

The Perfect Customer

Placing Whole Foods into this framework, we can see that ==what Amazon did was not just buy a retailer, but also acquire a customer for its existing business==.

Amazon.com's three-tier architecture + Customers

CustomerAll categories of food, delivery, restaurants
ServiceGroceriesMeat, fruits, vegetables, dry goods, etc.
PlatformLogistics CenterHigh fixed costs + scale returns
InfrastructureModular SuppliersStore brands, name brands, local suppliers, regional suppliers, etc.

Now, Amazon groceries can serve both Amazon Fresh and Whole Foods, and in the future, this foundational platform can also provide services to restaurants and other food-related entities.

Designing Smart Notification of Stock Price Changes

· 15 min read

Requirements

  • 3 million users
  • 5000 stocks + 250 global stocks
  • a user gets notified about the price change when
    1. subscribing the stock
    2. the stock has 5% or 10% changes
    3. since a) the last week or b) the last day
  • extensibility. may support other kinds of notifications like breaking news, earnings call, etc.

Sketching out the Architecture

Contexts:

  • What is clearing? Clearing is the procedure by which financial trades settle – that is, the correct and timely transfer of funds to the seller and securities to the buyer. Often with clearing, a specialized organization acts as an intermediary known as a clearinghouse.
  • What is a stock exchange? A facility where stock brokers and traders can buy and sell securities.

Apple Push Notification service


(APNs)

Apple Push Notification service<br>(APNs)

Google Firebase Cloud Messaging


(FCM)

Google Firebase Cloud Messaging<br>(FCM)

Email Services


AWS SES /sendgrid/etc

Email Services<br>AWS SES /sendgrid/etc

notifier

notifier

External Vendors



Market Prices

[Not supported by viewer]

Robinhood App

Robinhood App

API Gateway

API Gateway

Reverse Proxy

Reverse Proxy

batch write

batch write

price


ticker

[Not supported by viewer]

Time-series DB


influx or prometheus

Time-series DB<br>influx or prometheus

Tick every 5 mins

[Not supported by viewer]

periorical read

periorical read

price


watcher

price<br>watcher

User Settings

User Settings

Notification Queue

Notification Queue

throttler cache

throttler cache

cronjob

cronjob

What are those components and how do they interact with each other?

  • Price ticker
    • data fetching policies
      • option 1 preliminary: fetches data every 5 mins and flush into the time-series database in batches.
      • option 2 advanced: nowadays external systems usually push data directly so that we do not have to pull all the time.
    • ~6000 points per request or per price change.
    • data retention of 1 week, because this is just the speeding layer of the lambda architecture.
  • Price watcher
    • read the data ranging from last week or last 24 hours for each stock.
    • calculate if the fluctuation exceeds 5% or 10% in those two time spans. we get tuples like (stock, up 5%, 1 week).
      • corner case: should we normalize the price data? for example, some abnormal price like someone sold UBER mistakenly for $1 USD.
    • ratelimit (because 5% or 10% delta may occur many times within one day), and then emit an event PRICE_CHANGE(STOCK_CODE, timeSpan, percentage) to the notification queue.
  • Periodical triggers are cron jobs, e.g. Airflow, Cadence.
  • notification queue
    • may not necessarily be introduced in the first place when users and stocks are small.
    • may accept generic messaging event, like PRICE_CHANGE, EARNINGS_CALL, BREAKING_NEWS, etc.
  • Notifier
    • subscribe the notification queue to get the event
    • and then fetch who to notify from the user settings service
    • finally based on user settings, send out messages through APNs, FCM or AWS SES.

Designing Stock Exchange

· 16 min read

Requirements

  • order-matching system for buy and sell orders. Types of orders:
    • Market Orders
    • Limit Orders
    • Stop-Loss Orders
    • Fill-or-Kill Orders
    • Duration of Orders
  • high availability and low latency for millions of users
    • async design - use messaging queue extensively (btw. side-effect: engineers work on one service pub to a queue and does not even know where exactly is the downstream service and hence cannot do evil.)

Architecture

Reverse Proxy

Reverse Proxy

API Gateway

API Gateway

Order Matching

Order Matching

User Store

User Store

settle

settle

Orders

Orders

Stock Meta

Stock Meta

auth

auth

Cache

Cache

Balances & Bookkeeping

Balances & Bookkeeping

external pricing

external pricing

clearing


house

clearing<br>house

Bank, ACH, Visa, etc

Bank, ACH, Visa, etc

Payment

Payment

Audit & Report

Audit & Report

Components and How do they interact with each other.

order matching system

  • shard by stock code
  • order's basic data model (other metadata are omitted): Order(id, stock, side, time, qty, price)
  • the core abstraction of the order book is the matching algorithm. there are a bunch of matching algorithms(ref to stackoverflow, ref to medium)
  • example 1: price-time FIFO - a kind of 2D vector cast or flatten into 1D vector
    • x-axis is price
    • y-axis is orders. Price/time priority queue, FIFO.
      • Buy-side: ascending in price, descending in time.
      • Sell-side: ascending in price, ascending in time.
    • in other words
      • Buy-side: the higher the price and the earlier the order, the nearer we should put it to the center of the matching.
      • Sell-side: the lower the price and the earlier the order, the nearer we should put it to the center of the matching.

x-axis

line of prices

with y-axis cast into x-axis

Id   Side    Time   Qty   Price   Qty    Time   Side  
---+------+-------+-----+-------+-----+-------+------
#3 20.30 200 09:05 SELL
#1 20.30 100 09:01 SELL
#2 20.25 100 09:03 SELL
#5 BUY 09:08 200 20.20
#4 BUY 09:06 100 20.15
#6 BUY 09:09 200 20.15

Order book from Coinbase Pro

The Single Stock-Exchange Simulator

  • example 2: pro-rata

pure pro-rata

How to implement the price-time FIFO matching algorithm?

  • shard by stock, CP over AP: one stock one partition
  • stateful in-memory tree-map
    • periodically iterate the treemap to match orders
  • data persistence with cassandra
  • in/out requests of the order matching services are made through messaging queues
  • failover
    • the in-memory tree-maps are snapshotting into database
    • in an error case, recover from the snapshot and de-duplicate with cache

How to transmit data of the order book to the client-side in realtime?

  • websocket

How to support different kinds of orders?

  • same SELL or BUY: qty @ price in the treemap with different creation setup and matching conditions
    • Market Orders: place the order at the last market price.
    • Limit Orders: place the order with at a specific price.
    • Stop-Loss Orders: place the order with at a specific price, and match it in certain conditions.
    • Fill-or-Kill Orders: place the order with at a specific price, but match it only once.
    • Duration of Orders: place the order with at a specific price, but match it only in the given time span.

Orders Service

  • Preserves all active orders and order history.
  • Writes to order matching when receives a new order.
  • Receives matched orders and settle with external clearing house (async external gateway call + cronjob to sync DB)

References