Why bookkeeping?

Everyone has advice about how to manage money. Search Google for “manage money,” and you’ll get back over 1,690,000,000 links. You’ll find tons of life-hacking or self-help articles and books. You’ll find professional coaches or courses witch will coach you for a fee. You’ll find financial and investment services. Feel free to try what appeals to you and grow your assets through trials and errors.

I think the most important thing to remember is that asking the question of money comes from fear and self-doubt. We all fear change. We all doubt our ability to make more money.

Instead of spending time worrying and doubting, focusing the opposite — your confidence. If you are playing poker and with few chips, you can only make small bets and only win a small amount of money. When you have a lot of chips, you can make big bets and win big. You have more room for taking risks. You can try things which you cannot try when you have fewer chips.

Here is the magic - by understanding more of your financial status, you gain confidence! With more confidence, we can make better judgment and would like to bet the best amount for more significant success, and then win more.

Know your expenses and plan for next spending

Where is the end of the wining-more concern? People often talk about the buzzword of financial freedom. However, talk is cheap, but bookkeeping precisely answers the question.

Four main financial statements for the overview of financial status

Unfortunately, bookkeeping is not easy in our modern life. We are in a new age of abundance. We have a lot of accounts - cash, bank accounts, payment apps, credit cards, stock or crypto broker accounts, discount cards, … We have assets like houses, cars, gold, jewelry, … To make things even worse, some of us may live across countries and have to deal with different currencies. How could we draw an accurate map of our financial life and navigate through the future uncertainties?

By “accurate map of our financial life,” I mean these four primary financial statements:

1. Income Statement: It shows how much revenue we earned over a specific period. This statement is usually considered the most important of the financial statements because it reflects the operating results.
2. Balance Sheets: It answers how much assets, liabilities, and equity of the entity we have. This statement is the second most important because it reports the liquidity and capitalization of our assets.
3. Cash Flow Statements: It reports our inflows and outflows of cash and answers whether we generated cash. We need enough cash on hand to pay expenses and purchase assets.
4. Equity Statement: This is not helpful for your personal accounting. However, for a company, this statement reports how it distributes equity among stakeholders.

With beancount.io, you can quickly generate statements like the above. But wait… How to prepare data for these statements?

Double-entry Bookkeeping for Correctness

To ensure the accuracy and internalize the error detection into the system, double-entry bookkeeping requires every entry to an account has at-least a corresponding entry to a different account. One transaction involves at least two accounts with two operations - debit (+) and credit (-).

1970-01-01 open Income:BeancountCorp
1970-01-01 open Assets:Cash
1970-01-01 open Expenses:Food
1970-01-01 open Assets:Receivables:Alice
1970-01-01 open Assets:Receivables:Bob
1970-01-01 open Assets:Receivables:Charlie
1970-01-01 open Liabilities:CreditCard

2019-05-31 * "BeancountCorp" "Salary of May 15th to May 31st"
  Income:BeancountCorp               -888 USD
  Assets:Cash                         888 USD

2019-07-12 * "Popeyes chicken sandwiches" "dinner with Alice, Bob, and Charlie"
  Expenses:Food 20 USD
  Assets:Receivables:Alice 20 USD
  Assets:Receivables:Bob 20 USD
  Assets:Receivables:Charlie 20 USD
  Liabilities:CreditCard -80 USD

As you can see in the two examples above, every transaction must fulfill the accounting equation.

Assets = Liabilities + Equity(aka Net Assets)

We used the Beancount syntax by Martin Blais and the web project Fava by Jakob Schnitzer to build this website. And it will alert you if any transaction has any legs not summing to zero.

Now you understand how we enforce the correctness of the ledger. But you may ask what are those “accounts”?

Accounts for money as buckets for water

Thinking your assets as water running in and out of different buckets and “accounts” are those buckets holding your money. With double-entry bookkeeping, it becomes obvious how money is flowing across different accounts, just like how water is flowing across different buckets.

Beancount.io introduces five kinds of accounts.

1. Income — Its amount is always negative or in debit. This is because you are making money, and then the money is debiting from “Income” account and crediting to your “Assets.”
2. Expenses — Its amount is always positive or in credit. This is because you are spending money, and the money is flowing from the “Assets” or “Liabilities” to the “Expenses.”
3. Liabilities — Its amount is positive or zero. Your credit card liabilities are a good example, which rises and falls in cycles.
4. Assets — Its amount is positive or zero. Your cash or houses are always worthing some prices.
5. Equity — Your net assets. The system will calculate automatically for you. Equity = Assets - Liabilities and it reflects how wealthy you are.

Now you can open your customized accounts with those keywords above:

1970-01-01 open Assets:Cash
1970-01-01 open Assets:Stock:Robinhood
1970-01-01 open Assets:Crypto:Coinbase
1970-01-01 open Expenses:Transportation:Taxi
1970-01-01 open Equity:OpeningBalance

Commodities: Tracking your investment

Yes, you can track your investment with beancount.io. For example, we buy 10 Bitcoins at the price of $100 in 2014:

2014-08-08 * "Buy 10 Bitcoin"
  Assets:Trade:Cash -1000.00 USD
  Assets:Trade:Positions  10 BTC {100.00 USD}

And then three years later, you sell them (originally with costs of $100 per unit annotated with {100.00 USD}) at the price of $10,000 per unit annotated with @ 10,000.00 USD.

2017-12-12 * "Sell 2 Bitcoin"
  Assets:Trade:Positions                   -2 BTC {100.00 USD} @ 10,000.00 USD
  Assets:Trade:Cash                          20,000.00 USD
  Income:Trade:PnL                          -19,800.00 USD

Or the same transaction with @@ 20,000.00 USD means that at the price of $20,000 in total.

2017-12-12 * "Sell 2 Bitcoin"
  Assets:Trade:Positions                   -2 BTC {100.00 USD} @@ 20,000.00 USD
  Assets:Trade:Cash                          20,000.00 USD
  Income:Trade:PnL                          -19,800.00 USD

The sum of all legs of the transaction, including -2 BTC {100.00 USD}, are still, as always, zero.

The costs {100.00 USD} tag is important because you might have bought the same commodity at different costs.

100 BTC {10.00 USD, 2012-08-08}
10 BTC {100.00 USD, 2014-08-08}

If you want to simplify the process, you can set up the account at the beginning with FIFO or LIFO. FIFO stands for first in, first out, while LIFO stands for last in, first out. In the US, IRS uses FIFO to calculate your PnL and tax accordingly.

1970-01-01 open Assets:Trade:Positions "FIFO"

And then when you sell it in shorthand like -2 BTC {}, beancount will apply FIFO strategy automatically and sell the oldest commodity.

Beancount.io

Beancount.io is such a cloud service for recording your financial transactions in text files, visualize them into financial statements (income statement, balance sheet, trial balance, etc.), and helps you live a better financial life. Sign up now - It’s in Promotional Period and Free!

Requirements

Log v.s Metric: A log is an event that happened, and a metric is a measurement of the health of a system.

We are assuming that this system’s purpose is to serve metrics - namely, counters, conversion rate, timers, etc. for monitoring the system performance and health. If the conversion rate drops drastically, the system should alert the on-call.

1. Monitoring business metrics like signup funnel’s conversion rate
2. Supporting various queries, like on different platforms (IE/Chrome/Safari, iOS/Android/Desktop, etc.)
3. data visualization
4. Scalability and Availability

Architecture

Two ways to build the system:

1. Push Model: Influx/Telegraf/Grafana
2. Pull Model: Prometheus/Grafana

The pull model is more scalable because it decreases the number of requests going into the metrics databases - there is no hot path and concurrency issue.

Features and Components

Measuring Sign-up Funnel

Take a four-step sign up on the mobile app for example

INPUT_PHONE_NUMBER -> VERIFY_SMS_CODE -> INPUT_NAME -> INPUT_PASSWORD

Every step has IMPRESSION and POST_VERIFICATION phases. And emit metrics like this:

{
  "sign_up_session_id": "uuid",
  "step": "VERIFY_SMS_CODE",
  "os": "iOS",
  "phase": "POST_VERIFICATION",
  "status": "SUCCESS",
  // ... ts, contexts, ...
}

Consequently, we can query the overall conversion rate of VERIFY_SMS_CODE step on iOS like

(counts of step=VERIFY_SMS_CODE, os=iOS, status: SUCCESS, phase: POST_VERIFICATION) / (counts of step=VERIFY_SMS_CODE, os=iOS, phase: IMPRESSION)

Data Visualization

Graphana is mature enough for the data visualization work. If you do not want to expose the whole site, you can use Embed Panel with iframe.

Clarifying Requirements

Designing a service money transfer backend system like Square Cash (we will call this system Cash App below) or PayPal to

1. Deposit from and payout to bank
2. Transfer between accounts
3. High scalability and availability
4. i18n: language, timezone, currency exchange
5. Deduplication for non-idempotent APIs and for at-least-once delivery.
6. Consistency across multiple data sources.

Architecture

Features and Components

Payment Service

The payment data model is essentially “double-entry bookkeeping”. Every entry to an account requires a corresponding and opposite entry to a different account. Sum of all debit and credit equals to zero.

Deposit and Payout

Transaction: new user Jane Doe deposits $100 from bank to Cash App. This one transaction involves those DB entries:

bookkeeping table (for history)

+ debit, USD, 100, CashAppAccountNumber, txId
- credit, USD, 100, RoutingNumber:AccountNumber, txId

transaction table

txId, timestamp, status(pending/confirmed), [bookkeeping entries], narration

Once the bank confirmed the transaction, update the pending status above and the following balance sheet in one transaction.

balance sheet

CashAppAccountNumber, USD, 100

Transfer between accounts within Cash App

Similar to the case above, but there is no pending state because we do not need the slow external system to change their state. All changes in bookkeeping table, transaction table, and balance sheet table happen in one transaction.

i18n

We solve the i18n problems in 3 dimensions.

1. Language: All texts like copywriting, push notifications, emails are picked up according to the accept-language header.
2. Timezones: All server timezones are in UTC. We transform timestamps to the local timezone in the client-side.
3. Currency: All user transferring transactions must be in the same currency. If they want to move across currencies, they have to exchange the currency first, in a rate that is favorable to the Cash App.

For example, Jane Doe wants to exchange 1 USD with 6.8 CNY with 0.2

bookkeeping table

- credit, USD, 1, CashAppAccountNumber, txId
+ debit, CNY, 6.8, CashAppAccountNumber, txId, @7.55 CNY/USD
+ debit, USD, 0.1, ExpensesOfExchangeAccountNumber, txId

Transaction table, balance sheet, etc. are similar to the transaction discussed in Deposit and Payout. The major difference is that the bank or the vendor provides the exchange service.

How to sync across the transaction table and external banks and vendors?

• retry with idempotency to improve the success rate of the external calls and ensure no duplicate orders.
• two ways to check if the PENDING orders are filled or failed.
  1. poll: cronjobs (SWF, Airflow, Cadence, etc.) to poll the status for PENDING orders.
  2. callback: provide a callback API for the external vendors.

Deduplication

Why is Deduplication a concern?

1. not all endpoints are idempotent
2. Event queue may be at-least-once.

not all endpoints are idempotent: what if the external system is not idempotent?

For the poll case above, if the external gateway does not support idempotent APIs, in order not to flood with duplicate entries, we must keep record of the order ID or the reference ID the external system gives us with 200, and query GET by the order ID instead of POST all the time.

For the callback case, we can ensure we implement with idempotent APIs, and we mutate pending to confirmed anyway.

Event queue may be at-least-once

• For the even queue, we can use an exactly-once Kafka with the producer throughput declines only by 3%.
• In the database layer, we can use idempotency key or deduplication key.
• In the service layer, we can use Redis key-value store.

Availability and Scalability

• Overall failover strategies: Improving availability with failover: Cold Standby, Hot Standby, Warm Standby, Active-active.
• Service layer scaling: AKF Scale Cube
• Data layer scaling: CQRS Pattern
• Needing a speed layer? Lambda Architecture

1. Clarifying Requirements

1. Webhook to callback the merchant once the payment succeeds.
2. Analytics & metrics.
3. High availability & Failure-resilience.
   1. Async design. Assuming that the servers of merchants are located across the world, and may have a very high latency like 15s.
   2. At-least-once delivery.
   3. Robust & predicable retry.
4. Security: informing the merchants whether a payment succeeds involves real money real transactions, and thus, security is always a concern.

2. Sketch out the high-level design

async design + retry + queuing + time-series DB + security

3. Features and Components

Webhook Gateway

1. Subscribe to the event queue for payment success events published by a payment state machine or other services.
2. Once accept an event, fetch webhook URI, secret, and settings from the user settings service. Prepare the request based on those settings.
3. Make an HTTP POST request to the external merchant’s endpoints with event payload and security headers.

API Definition

// POST https://example.com/webhook/
{
    "id": 1,
    "scheduled_for": "2017-01-31T20:50:02Z",
    "event": {
        "id": "24934862-d980-46cb-9402-43c81b0cdba6",
        "resource": "event",
        "type": "charge:created",
        "api_version": "2018-03-22",
        "created_at": "2017-01-31T20:49:02Z",
        "data": {
          "code": "66BEOV2A", // or order ID the user need to fulfill
          "name": "The Sovereign Individual",
          "description": "Mastering the Transition to the Information Age",
          "hosted_url": "https://commerce.coinbase.com/charges/66BEOV2A",
          "created_at": "2017-01-31T20:49:02Z",
          "expires_at": "2017-01-31T21:49:02Z",
          "metadata": {},
          "pricing_type": "CNY",
          "payments": [
            // ...
          ],
          "addresses": {
            // ...
          }
        }
    }
}

The merchant server should respond with a 200 HTTP status code to acknowledge receipt of a webhook.

If there is no acknowledgment of receipt, we will retry with exponential backoff for up to three days. The maximum retry interval is 1 hour.

Security

• All webhooks from user settings must be in https
• All callback requests are with header x-webhook-signature SHA256 HMAC signature. Its value is HMAC(webhook secret, raw request payload);. We generate the secret for the developer to use.

Background Knowledge: HMAC (message authentication code). A short piece of information used to authenticate a message — In other words, to confirm that the message came from the stated sender (its authenticity) and has not been changed in transit (its integrity). The integrity can be verified by the shared secret between trusted parties against the digest of the original message.

Metrics

The webhook gateway service emits statuses into the time-series DB for metrics.

Using Influx DB vs. Prometheus?

• InfluxDB: Application pushes data to InfluxDB. It has a monolithic DB for metrics and indices.
• Prometheus: Prometheus server pulls the metrics values from the running application periodically. It uses LevelDB for indices, but each metric is stored in its own file.

I will probably choose InfluxDB for easier maintenance of the monolithic data store.

Depending on how much further data aggregation we need, we can build more advanced data pipeline. However, for just counting success/ failures, a simple time-series DB solves the problem.

• After public speaking, the biggest social fear in the Western world is initiating a conversation with strangers. To conquer the fear of rejection, it helps to realize that in most cases, people appreciate it when you make an effort to speak with them.
• In some cases, not talking can make you come off as arrogant or aloof. Initiating a conversation can be simple: first, smiling at someone; second, establishing eye contact; third, being the first to introduce yourself.
• To approach a group of people: 1, demonstrate your interest to the group from a distance, paying attention to the speaker. 2, the group will notice and make room to include you. 3, let the group warm to you before you offer any strong opinions.
• Guiding a conversation evokes the positive feelings that make people want to work or socialize with them. One easy way of assuming this responsibility is to act like you’re a host and ask, “What’s your name?”. Emphasize “your” to make them feel valued.
• The best way to improve your conversations is to ask open-ended questions, which demonstrates that you genuinely care about what they have to say.
• A conversation will inevitably dip into an awkward silence sometimes. You can get it back to a comfortable flow by asking open-ended questions with the current contexts or the acronym FORM: family, occupation, recreation and miscellaneous.
• To be an active listener, body language is important: avoid crossing arms, hunching shoulders, or fiddling with clothes, hair or jewelry. Instead, lean forward, nod, smile and maintain eye contact.
• To be an active listener, vertal cues are important: engage by asking follow-up questions about the details, respond enthusiastically to express your intrest, or paraphrasing what the speaker said to clarify.
• To end a conversation gracefully, the first thing you can do is to circle back to the highlight of your discussion. If you genuinely want to continue the discussion later, exchange contact info, state what you will do next, (if acquaintance) shake hand, and say goodbye.
• It’s important to follow through with whatever it is you say you’re doing next. Otherwise, the other side may think you simply were not enjoying your time with them, which hurts feelings.
• It is a courteous way to end a conversation with introducing your conversation partner to a new person, which will whiden his network and ensure he does not feel that you’re abandoning him. Or reversely, you can ask them to introduce you to someone else.