1. Clarifying Requirements​

1. Webhook will call the merchant back once the payment succeeds.
   1. Merchant developer registers webhook information with us.
   2. Make a POST HTTP request to the webhooks reliably and securely.
2. High availability, error-handling, and failure-resilience.
   1. Async design. Assuming that the servers of merchants are located across the world, and may have a very high latency like 15s.
   2. At-least-once delivery. Idempotent key.
   3. Order does not matter.
   4. Robust & predictable retry and short-circuit.
3. Security, observability & scalability
   1. Anti-spoofing.
   2. Notify the merchant when their receivers are broken.
   3. easy to extend and scale.

2. Sketch out the high-level design​

async design + retry + queuing + observability + security

3. Features and Components​

Core Features​

1. Users go to dashboard frontend to register webhook information with us - like the URL to call, the scope of events they want to subscribe, and then get an API key from us.
2. When there is a new event, publish it into the queue and then get consumed by callers. Callers get the registration and make the HTTP call to external services.

Webhook callers​

1. Subscribe to the event queue for payment success events published by a payment state machine or other services.

2. Once callers accept an event, fetch webhook URI, secret, and settings from the user settings service. Prepare the request based on those settings. For security...

• All webhooks from user settings must be in HTTPs

• If the payload is huge, the prospect latency is high, and we wants to make sure the target reciever is alive, we can verify its existance with a ping carrying a challenge. e.g. Dropbox verifies webhook endpoints by sending a GET request with a “challenge” param (a random string) encoded in the URL, which your endpoint is required to echo back as a response.

• All callback requests are with header x-webhook-signature. So that the receiver can authenticate the request.

  • For symetric signature, we can use HMAC/SHA256 signature. Its value is HMAC(webhook secret, raw request payload);. Telegram takes this.
  • For asymmetric signature, we can use RSA/SHA256 signature. Its value is RSA(webhook private key, raw request payload); Stripe takes this.
  • If it's sensitive information, we can also consider encryption for the payload instead of just signing.

3. Make an HTTP POST request to the external merchant's endpoints with event payload and security headers.

API Definition​

// POST https://example.com/webhook/
{
  "id": 1,
  "scheduled_for": "2017-01-31T20:50:02Z",
  "event": {
    "id": "24934862-d980-46cb-9402-43c81b0cdba6",
    "resource": "event",
    "type": "charge:created",
    "api_version": "2018-03-22",
    "created_at": "2017-01-31T20:49:02Z",
    "data": {
      "code": "66BEOV2A", // or order ID the user need to fulfill
      "name": "The Sovereign Individual",
      "description": "Mastering the Transition to the Information Age",
      "hosted_url": "https://commerce.coinbase.com/charges/66BEOV2A",
      "created_at": "2017-01-31T20:49:02Z",
      "expires_at": "2017-01-31T21:49:02Z",
      "metadata": {},
      "pricing_type": "CNY",
      "payments": [
        // ...
      ],
      "addresses": {
        // ...
      }
    }
  }
}

The merchant server should respond with a 200 HTTP status code to acknowledge receipt of a webhook.

Error-handling​

If there is no acknowledgment of receipt, we will retry with idempotency key and exponential backoff for up to three days. The maximum retry interval is 1 hour. If it's reaching a certain limit, short-circuit / mark it as broken. Sending out an Email to the merchant.

Metrics​

The Webhook callers service emits statuses into the time-series DB for metrics.

Using Statsd + Influx DB vs. Prometheus?

• InfluxDB: Application pushes data to InfluxDB. It has a monolithic DB for metrics and indices.
• Prometheus: Prometheus server pulls the metrics values from the running application periodically. It uses LevelDB for indices, but each metric is stored in its own file.

Or use the expensive DataDog or other APM services if you have a generous budget.

• After public speaking, the biggest social fear in the Western world is initiating a conversation with strangers. To conquer the fear of rejection, it helps to realize that in most cases, people appreciate it when you make an effort to speak with them.
• In some cases, not talking can make you come off as arrogant or aloof. Initiating a conversation can be simple: first, smiling at someone; second, establishing eye contact; third, being the first to introduce yourself.
• To approach a group of people: 1, demonstrate your interest to the group from a distance, paying attention to the speaker. 2, the group will notice and make room to include you. 3, let the group warm to you before you offer any strong opinions.
• Guiding a conversation evokes the positive feelings that make people want to work or socialize with them. One easy way of assuming this responsibility is to act like you're a host and ask, "What's your name?". Emphasize "your" to make them feel valued.
• The best way to improve your conversations is to ask open-ended questions, which demonstrates that you genuinely care about what they have to say.
• A conversation will inevitably dip into an awkward silence sometimes. You can get it back to a comfortable flow by asking open-ended questions with the current contexts or the acronym FORM: family, occupation, recreation and miscellaneous.
• To be an active listener, body language is important: avoid crossing arms, hunching shoulders, or fiddling with clothes, hair or jewelry. Instead, lean forward, nod, smile and maintain eye contact.
• To be an active listener, vertal cues are important: engage by asking follow-up questions about the details, respond enthusiastically to express your intrest, or paraphrasing what the speaker said to clarify.
• To end a conversation gracefully, the first thing you can do is to circle back to the highlight of your discussion. If you genuinely want to continue the discussion later, exchange contact info, state what you will do next, (if acquaintance) shake hand, and say goodbye.
• It's important to follow through with whatever it is you say you're doing next. Otherwise, the other side may think you simply were not enjoying your time with them, which hurts feelings.
• It is a courteous way to end a conversation with introducing your conversation partner to a new person, which will whiden his network and ensure he does not feel that you're abandoning him. Or reversely, you can ask them to introduce you to someone else.

The answer is: Amazon wanted to buy customers for its grocery service.

Background​

• Amazon's acquisition of Whole Foods = Apple's iPhone defeating Palm

  • Do not confuse goals, strategies, and tactics — Apple's strategy:
    • It was not about making phones but about producing personal computers
    • It was not about adding features to phones but about compressing traditional phone functions into one app
    • It was not about replicating the work of carriers but about leveraging its connection with customers
  • The iPhone is the most successful product in history = Amazon is the most dominant company in history

• Amazon's Goals

  1. Initially, Amazon.com aimed to become a leading retailer based on information products and services, starting with selling books.
  2. Then, Amazon declared, "Our vision is to be Earth's most customer-centric company, where customers can find anything they want to buy online."
  3. ==Amazon's goal is to gain a share of all economic activities==.

• Amazon's Strategy

  • For businesses: AWS. Assume that all commercial transactions will soon be completed online.
  • For customers: Prime. Assume that high costs and diverse choices are unsustainable. With Prime, customers will not consider other alternatives.
    • However
      • The grocery industry is the largest retail category
      • The grocery industry can continuously remind consumers that there are alternatives to Amazon

• Amazon's Tactics: Develop grocery services

Why Did Amazon Not Arrive at the Right Tactics?​

Amazon's cost disadvantages in fresh produce

1. Once scale is insufficient, product spoilage will incur high costs.
2. Scale depends on the specific circumstances of each city.

Why does acquiring Whole Foods (rather than others) solve the business scale issue?

==Business fundamental component model + two basic points 1) High fixed costs 2) High returns==

• Deconstruct the infrastructure into Minimum Sellable Units (MSUs)
• These businesses themselves are the first and best customers of these minimum sellable units
• Resell the minimum sellable units

AWS's three-tier architecture

• MSUs belong to S3, EC2, RDS, SNS, etc.
• The first and best customer is amazon.com
• Resell MSUs to non-Amazon developers

Amazon.com's three-tier architecture

• MSUs belong to FDA, Amazon Pay, etc.
• The first and best customer is Amazon's first-party e-commerce
• Resell MSUs to third-party suppliers

The insight here is that Amazon's existing grocery does not have a first and best supplier.

The Perfect Customer​

Placing Whole Foods into this framework, we can see that ==what Amazon did was not just buy a retailer, but also acquire a customer for its existing business==.

Amazon.com's three-tier architecture + Customers

Now, Amazon groceries can serve both Amazon Fresh and Whole Foods, and in the future, this foundational platform can also provide services to restaurants and other food-related entities.

Requirements​

• 3 million users
• 5000 stocks + 250 global stocks
• a user gets notified about the price change when
  1. subscribing the stock
  2. the stock has 5% or 10% changes
  3. since a) the last week or b) the last day
• extensibility. may support other kinds of notifications like breaking news, earnings call, etc.

Sketching out the Architecture​

Contexts:

• What is clearing? Clearing is the procedure by which financial trades settle – that is, the correct and timely transfer of funds to the seller and securities to the buyer. Often with clearing, a specialized organization acts as an intermediary known as a clearinghouse.
• What is a stock exchange? A facility where stock brokers and traders can buy and sell securities.

Apple Push Notification service<br>(APNs)

What are those components and how do they interact with each other?​

• Price ticker
  • data fetching policies
    • option 1 preliminary: fetches data every 5 mins and flush into the time-series database in batches.
    • option 2 advanced: nowadays external systems usually push data directly so that we do not have to pull all the time.
  • ~6000 points per request or per price change.
  • data retention of 1 week, because this is just the speeding layer of the lambda architecture.
• Price watcher
  • read the data ranging from last week or last 24 hours for each stock.
  • calculate if the fluctuation exceeds 5% or 10% in those two time spans. we get tuples like (stock, up 5%, 1 week).
    • corner case: should we normalize the price data? for example, some abnormal price like someone sold UBER mistakenly for $1 USD.
  • ratelimit (because 5% or 10% delta may occur many times within one day), and then emit an event PRICE_CHANGE(STOCK_CODE, timeSpan, percentage) to the notification queue.
• Periodical triggers are cron jobs, e.g. Airflow, Cadence.
• notification queue
  • may not necessarily be introduced in the first place when users and stocks are small.
  • may accept generic messaging event, like PRICE_CHANGE, EARNINGS_CALL, BREAKING_NEWS, etc.
• Notifier
  • subscribe the notification queue to get the event
  • and then fetch who to notify from the user settings service
  • finally based on user settings, send out messages through APNs, FCM or AWS SES.

Requirements​

• order-matching system for buy and sell orders. Types of orders:
  • Market Orders
  • Limit Orders
  • Stop-Loss Orders
  • Fill-or-Kill Orders
  • Duration of Orders
• high availability and low latency for millions of users
  • async design - use messaging queue extensively (btw. side-effect: engineers work on one service pub to a queue and does not even know where exactly is the downstream service and hence cannot do evil.)

Architecture​

Reverse Proxy

Components and How do they interact with each other.​

order matching system​

• shard by stock code
• order's basic data model (other metadata are omitted): Order(id, stock, side, time, qty, price)
• the core abstraction of the order book is the matching algorithm. there are a bunch of matching algorithms(ref to stackoverflow, ref to medium)
• example 1: price-time FIFO - a kind of 2D vector cast or flatten into 1D vector
  • x-axis is price
  • y-axis is orders. Price/time priority queue, FIFO.
    • Buy-side: ascending in price, descending in time.
    • Sell-side: ascending in price, ascending in time.
  • in other words
    • Buy-side: the higher the price and the earlier the order, the nearer we should put it to the center of the matching.
    • Sell-side: the lower the price and the earlier the order, the nearer we should put it to the center of the matching.

x-axis

with y-axis cast into x-axis

Id   Side    Time   Qty   Price   Qty    Time   Side  
---+------+-------+-----+-------+-----+-------+------
#3                        20.30   200   09:05   SELL  
#1                        20.30   100   09:01   SELL  
#2                        20.25   100   09:03   SELL  
#5   BUY    09:08   200   20.20                       
#4   BUY    09:06   100   20.15                       
#6   BUY    09:09   200   20.15                       

Order book from Coinbase Pro

The Single Stock-Exchange Simulator

• example 2: pro-rata

How to implement the price-time FIFO matching algorithm?

• shard by stock, CP over AP: one stock one partition
• stateful in-memory tree-map
  • periodically iterate the treemap to match orders
• data persistence with cassandra
• in/out requests of the order matching services are made through messaging queues
• failover
  • the in-memory tree-maps are snapshotting into database
  • in an error case, recover from the snapshot and de-duplicate with cache

How to transmit data of the order book to the client-side in realtime?

• websocket

How to support different kinds of orders?

• same SELL or BUY: qty @ price in the treemap with different creation setup and matching conditions
  • Market Orders: place the order at the last market price.
  • Limit Orders: place the order with at a specific price.
  • Stop-Loss Orders: place the order with at a specific price, and match it in certain conditions.
  • Fill-or-Kill Orders: place the order with at a specific price, but match it only once.
  • Duration of Orders: place the order with at a specific price, but match it only in the given time span.

Orders Service​

• Preserves all active orders and order history.
• Writes to order matching when receives a new order.
• Receives matched orders and settle with external clearing house (async external gateway call + cronjob to sync DB)

References​

• How Exchange Happen? Simulating a financial exchange in Scala
• Open Sources
  • GitHub - objectcomputing/liquibook: Modern C++ order matching engine
  • GitHub - cyanly/gotrade: A proof of concept of an electronic trading system written in Golang
• The Financial Information eXchange ("FIX") Protocol
• Types of orders: Technical Analysis Course. Training,Coaching & Mentoring for Traders / Investors - Types of orders used when buying or selling a stock
• GitHub - fmstephe/matching_engine: A simple financial trading matching engine. Built to learn more about how they work.
• Automated Algorithmic Trading: Learning Agents for Limit Order Book Trading

I came across "bozo management" on the Blind App, and find it's a very interesting concept. Steve Jobs coined this phrase.

If you cannot watch the video, here are the words from him.

We went through that stage at Apple where we thought, 'Oh, we're going to be a big company, let's go out and hire professional management.' We went out and hired a bunch of professional management; it didn't work at all. Most of them were Bozos. They knew how to manage, but they didn't know how to DO anything.

If you are a great person, why do you want to work for somebody you cannot learn anything from? And you know what's interesting - you know what the best managers are? They are the great individual contributors who never ever wanted to be a manager, but decide they have to be a manager because no one else is able to do as good job as them.

What is DIF?​

Decentralized Identity Foundation builds ecosystem for decentralized identity and ensures interop between all participants.

Why? … What is the problem?​

ID Problem​

The problem of decoupling ID from Personally identifiable information (PII). Identity is composed of a deeply personal collection of data that defines us, and your identity should answer to no one but you.

Specifically, challenges are

1. Decoupling ID from identity providers. DIDs should be self-sovereign and not owned or controlled by central authorities.
2. Decoupling ID lookup from centralized systems. DIDs and data should be able to be found across decentralized systems, so that no central owner can do the evil.
3. Decoupling ID data from indiscreet or unknown storages. DIDs should be able to control precisely what to or what not to share with others.

Uniting the fragmented landscapes​

DIF is the organization uniting the fragmented to solve the DID problem together and build an ecosystem as an industry standard.

How do they organize the efforts?​

• Designing Specs
• Implementations
• Aligning industry participants

People and Organizations​

Working Groups

• ID, Names, Discovery
• Storage and Compute
• Claims and Credentials

Members

• Blockstack
• Microsoft / IBM
• HyperLedger
• RSA
• Ontology
• Civic
• iota
• …

Ecosystem​

DID​

ID that is

1. globally unique
2. resolveable with high availability, and
3. cryptographically verifiable.

DID Format: URN

DID methods (further explained below) define how DIDs work with a specific blockchain.

DID Document​

• DID infrastructure = a global key-value database of <DID, DID Document>
• DID document = public keys, authentication protocols, and service endpoints for verifying the entity and explaining how to use it. It may contain three things:
  • proof purposes
  • verification methods
  • service endpoints

How DID preserves privacy?​

1. Pairwise-pseudonymous DIDs
2. Off-chain private data
3. Selective disclosure

DID method specification (DID <> Blockchain)​

Defining how a DID and DID document are created, resolved, and managed (CRUD) on a specific blockchain.

How to join the DID method registration?​

https://w3c-ccg.github.io/did-method-registry/#the-registry

Examples

• Blockstack
• Arcblock

Learning by Example: Blockstack DID method​

• How Blockstack leverages DID?
• How to create a blockstack DID? How Blockstack acts as an Identity Provider?
• How to resolve a DID?

Blockstack Naming Service (BNS)​

• Naming Layer = username <> pub key & pointer to storage
• BNS is blockchain-agnostic. migrated from namecoint to bitcoin.
• ==a Blockstack DID is defined as a pointer to the nth name registered by an address==.

two categories​

on-chain DIDs

• two tx to register a name on-chain - owner address <> name
• one owner address may have multiple on-chain names and corresponding DIDs
  • e.g. did:stack:v0:15gxXgJyT5tM5A4Cbx99nwccynHYsBouzr-3 means the fourth on-chain name was created and initially assigned to the address 15gxXgJyT5tM5A4Cbx99nwccynHYsBouzr.

off-chain DIDs. a.k.a. subdomains

• encoded in batches, hashed, and written to a blockchain later.

• Off-chain names are instantiated by an on-chain name, indicated by the off-chain name's suffix. cicero.res_publica.id is processed by the owner of res_publica.id but are not owned by it.

Blockstack as an Identity Provider​

Demo: https://bitpatron.co/ login with Blockstack

// authRequestJwt
{
  "typ": "JWT",
  "alg": "ES256K"
}
{
  "jti": "4d06f08b-67a7-4f7c-89fc-b8164b81f67a",
  "iat": 1563432343,
  "exp": 1566110743,
  "iss": "did:btc-addr:19sxvnAxPXZYAEdpF7Tti6MSVhxA8PSdCT",
  "public_keys": [
    "03994ec7b23a8e11e40684c9b2d29febf103bd92c4bbd295f1e2537042c93ac977"
  ],
  "domain_name": "http://localhost:4104",
  "manifest_uri": "http://localhost:4104/manifest.json",
  "redirect_uri": "http://localhost:4104/",
  "version": "1.3.1",
  "do_not_include_profile": true,
  "supports_hub_url": true,
  "scopes": [
    "store_write"
  ]
}

// authResponse
{
  "typ": "JWT",
  "alg": "ES256K"
}
{
  "jti": "30773b78-3595-499f-bbb3-d1e649470c70",
  "iat": 1563432894,
  "exp": 1566111294,
  "iss": "did:btc-addr:1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
  "private_key": "redacted - encrypted with transitKey",
  "public_keys": [
    "027c8547681cc27e27b73ee0f3c0534bdd38993dcb4c1934bf424f0b3a04dcad63"
  ],
  "profile": null,
  "username": "kirbystar.id.blockstack",
  "core_token": null,
  "email": null,
  "profile_url": "https://gaia.blockstack.org/hub/1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH/profile.json",
  "hubUrl": "https://hub.blockstack.org",
  "blockstackAPIUrl": "https://core.blockstack.org",
  "associationToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJjaGlsZFRvQXNzb2NpYXRlIjoiMDNhMTU5YzY4YWQ1ZjFkNzcxMWY2NjJmNThkNjdmMzZlNzY3ZTBjMDBhOTU4ZWY0NzljNzU3MzU0MGFkMzExZjk2IiwiaXNzIjoiMDI3Yzg1NDc2ODFjYzI3ZTI3YjczZWUwZjNjMDUzNGJkZDM4OTkzZGNiNGMxOTM0YmY0MjRmMGIzYTA0ZGNhZDYzIiwiZXhwIjoxNTk0OTY4ODk0LjcwNSwiaWF0IjoxNTYzNDMyODk0LjcwNSwic2FsdCI6IjE4NGVhMWQyMzM3MWQ1MmYyYzhmNTAyOGUwMWYxYmZiIn0.ZceaVcIK2Z8wu6KBYOHQaK7y6BI7NfxrixphOCPs1B4hZcGYDKsuf0anbm4CdAAJbKRifCm-MYHE6fjKD9E7GQ",
  "version": "1.3.1"
}

// acctName response
{
  "blockchain": "bitcoin",
  "status": "submitted_subdomain",
  "last_txid": "851ca5e6c06723e61037aa397966aafa1a6dd7159e9e31e53116106b87101886",
  "zonefile": "$ORIGIN kirbystar.id.blockstack\n$TTL 3600\n_http._tcp\tIN\tURI\t10\t1\t\"https://gaia.blockstack.org/hub/1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH/profile.json\"\n\n",
  "address": "1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
  "zonefile_hash": "a9c016921a9a60e04776251db53a8881e6d128ce"
}

// session
{
  "version": "1.0.0",
  "userData": {
    "username": "kirbystar.id.blockstack",
    "profile": {
      "@type": "Person",
      "@context": "http://schema.org",
      "api": {
        "gaiaHubConfig": {
          "url_prefix": "https://gaia.blockstack.org/hub/"
        },
        "gaiaHubUrl": "https://hub.blockstack.org"
      }
    },
    "email": null,
    "decentralizedID": "did:btc-addr:1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
    "identityAddress": "1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
    "appPrivateKey": "redacted",
    "coreSessionToken": null,
    "authResponseToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJqdGkiOiIyNmM4ZGM2Ny1lNzEwLTRlZDUtYmIxYi0yY2I1ODY5YTRkMTEiLCJpYXQiOjE1NjM0MzU2NjEsImV4cCI6MTU2NjExNDA2MSwiaXNzIjoiZGlkOmJ0Yy1hZGRyOjFEcEtNcXhCbnVTU1FNTnVuMW9iY2lQU2ZEOXJEOEtOVUgiLCJwcml2YXRlX2tleSI6IjdiMjI2OTc2MjIzYTIyNjQzMDMzMzMzODMyNjYzNTYxNjIzNjM0Mzg2MTM1MzYzNTM2MzI2MzMzNjEzMTYyMzY2NTYzNjEzNzMzNjU2NjIyMmMyMjY1NzA2ODY1NmQ2NTcyNjE2YzUwNGIyMjNhMjIzMDMyNjEzMzY1NjEzMzY0NjMzNjMyMzgzODY2MzgzMTY2MzAzNDMwMzU2MzY1MzI2MzMzNjUzNjY0MzMzMzMxMzQzNDM0NjY2MTM4MzA2NTY1MzIzMzMyNjEzMjM2NjQzMzMwMzczMzM2MzY2NTMzMzkzNzM3MzczMjMxMzMzNzM5MzcyMjJjMjI2MzY5NzA2ODY1NzI1NDY1Nzg3NDIyM2EyMjM2MzMzOTYzNjMzMDM5MzQzOTYyMzMzNjY0NjI2NTM1MzIzMTM0MzIzMjM3MzA2MzY1NjIzMTM0MzEzNjY2MzA2MzM0NjYzNDM4MzgzMjY0MzUzOTM0MzMzNDM2NjM2NjYyNjEzNzMxNjM2MjM1MzYzMTM2MzczMDY0MzUzMzY0MzQzNTY1MzMzODYyNjIzNzM1NjMzMjMzMzYzNjMyMzUzODMxMzgzOTM3NjYzMjMwNjMzNTM4MzA2NTMyMzEzODM0MzMzNTMwMzMzNjM1NjE2NDM1NjEzMTM4NjY2MjMyNjY2NDM3MzQ2MzY1NjMzNDM0MzI2NTY0MzY2NTY2NjYzMjM5NjIzNjY2Mzk2MTMyMzgzMTM4MzEzMzYyMzMzMDYyMzkzMTYzMzE2MzM1Mzg2MTM4MzgzMTM5MzQ2MjYzMjIyYzIyNmQ2MTYzMjIzYTIyMzkzNzY0MzM2MzM4NjEzOTYxMzQ2NjMyMzkzNjM1MzM2MzM1MzQzNzY1MzIzNTYzMzMzMzM4MzIzMDYxMzczMjYyMzU2NjY0MzQ2NjM1NjE2NDYxNjY2NjMwMzQzNzM1Mzk2NDM0MzI2MTYxMzgzNDM1MzkzNzY2MzY2MjM5NjEyMjJjMjI3NzYxNzM1Mzc0NzI2OTZlNjcyMjNhNzQ3Mjc1NjU3ZCIsInB1YmxpY19rZXlzIjpbIjAyN2M4NTQ3NjgxY2MyN2UyN2I3M2VlMGYzYzA1MzRiZGQzODk5M2RjYjRjMTkzNGJmNDI0ZjBiM2EwNGRjYWQ2MyJdLCJwcm9maWxlIjpudWxsLCJ1c2VybmFtZSI6ImtpcmJ5c3Rhci5pZC5ibG9ja3N0YWNrIiwiY29yZV90b2tlbiI6bnVsbCwiZW1haWwiOm51bGwsInByb2ZpbGVfdXJsIjoiaHR0cHM6Ly9nYWlhLmJsb2Nrc3RhY2sub3JnL2h1Yi8xRHBLTXF4Qm51U1NRTU51bjFvYmNpUFNmRDlyRDhLTlVIL3Byb2ZpbGUuanNvbiIsImh1YlVybCI6Imh0dHBzOi8vaHViLmJsb2Nrc3RhY2sub3JnIiwiYmxvY2tzdGFja0FQSVVybCI6Imh0dHBzOi8vY29yZS5ibG9ja3N0YWNrLm9yZyIsImFzc29jaWF0aW9uVG9rZW4iOiJleUowZVhBaU9pSktWMVFpTENKaGJHY2lPaUpGVXpJMU5rc2lmUS5leUpqYUdsc1pGUnZRWE56YjJOcFlYUmxJam9pTUROaE1UVTVZelk0WVdRMVpqRmtOemN4TVdZMk5qSm1OVGhrTmpkbU16WmxOelkzWlRCak1EQmhPVFU0WldZME56bGpOelUzTXpVME1HRmtNekV4WmprMklpd2lhWE56SWpvaU1ESTNZemcxTkRjMk9ERmpZekkzWlRJM1lqY3paV1V3WmpOak1EVXpOR0prWkRNNE9Ua3paR05pTkdNeE9UTTBZbVkwTWpSbU1HSXpZVEEwWkdOaFpEWXpJaXdpWlhod0lqb3hOVGswT1RjeE5qWXhMak0yT1N3aWFXRjBJam94TlRZek5ETTFOall4TGpNMk9Td2ljMkZzZENJNklqVXhOMkZsWkdVd1ltVmpOMkpqTlRnek5qY3lOREkwT1RsaE1EVm1OVEEwSW4wLjlkY2VHX3I4OVdDSUVsTklseFNtUGxPblhiSVNsZEZDejJxOTJRMnpJSk9XXzhnTjVYT0xsZnNkREJVamlQZlU3eTNyRGFXSUxfTUJicVVnVnBFanhRIiwidmVyc2lvbiI6IjEuMy4xIn0.0Xqtw-71TJ9ybWx4Uxre0Gxkisay20xn1vqwr0WaKvVeCzwv_NO6YZnVOmGPM4cF4wex06yLYWasqQWgCi-m_g",
    "hubUrl": "https://hub.blockstack.org",
    "gaiaAssociationToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJjaGlsZFRvQXNzb2NpYXRlIjoiMDNhMTU5YzY4YWQ1ZjFkNzcxMWY2NjJmNThkNjdmMzZlNzY3ZTBjMDBhOTU4ZWY0NzljNzU3MzU0MGFkMzExZjk2IiwiaXNzIjoiMDI3Yzg1NDc2ODFjYzI3ZTI3YjczZWUwZjNjMDUzNGJkZDM4OTkzZGNiNGMxOTM0YmY0MjRmMGIzYTA0ZGNhZDYzIiwiZXhwIjoxNTk0OTcxNjYxLjM2OSwiaWF0IjoxNTYzNDM1NjYxLjM2OSwic2FsdCI6IjUxN2FlZGUwYmVjN2JjNTgzNjcyNDI0OTlhMDVmNTA0In0.9dceG_r89WCIElNIlxSmPlOnXbISldFCz2q92Q2zIJOW_8gN5XOLlfsdDBUjiPfU7y3rDaWIL_MBbqUgVpEjxQ"
  },
  "transitKey": "redacted"
}

Getting Entity from DID - Universal Resolver​

Domain Name <--DNS-->  IP
Repensented Entity <--Universal Resolver--> Self-sovereign Identifiers
DID <--Universal Resolver--> DID Document

Drivers for Example:

• did:stack: DID registered from BlockStack, like did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0
• did:btcr: DID registered from BTC
• etc.

Run a resolver​

git clone https://github.com/decentralized-identity/universal-resolver.git
cd universal-resolver/
docker-compose -f docker-compose.yml pull
docker-compose -f docker-compose.yml up

curl -X GET http://localhost:8080/1.0/identifiers/did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0 | jq .

{
  "redirect": null,
  "didDocument": {
    "id": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
    "service": [
      {
        "type": "blockstack",
        "serviceEndpoint": "https://core.blockstack.org"
      }
    ],
    "publicKey": [
      {
        "id": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
        "type": "Secp256k1VerificationKey2018",
        "publicKeyHex": "0232131c807c4b184582280bca141f2583f6a1de2e0d3e6984cdb4724527f581fa"
      }
    ],
    "@context": "https://w3id.org/did/v0.11"
  },
  "resolverMetadata": {
    "duration": 96,
    "driverId": "did-stack",
    "driver": "HttpDriver",
    "didUrl": {
      "didUrlString": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
      "did": {
        "didString": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
        "method": "stack",
        "methodSpecificId": "v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
        "parseTree": null,
        "parseRuleCount": null
      },
      "parameters": null,
      "parametersMap": {},
      "path": "",
      "query": null,
      "fragment": null,
      "parseTree": null,
      "parseRuleCount": null
    }
  },
  "methodMetadata": {}
}

Or use Blockstack's resolver​

https://github.com/blockstack/blockstack-core/blob/master/docs/blockstack-did-spec.md#33--resolving-a-blockstack-did

DID <> Real-world: Verifiable Claims​

Now we know how to recognize and resolve "who's who" without inherently carrying personally-identifiable information. However, what if we want DID to associate with real-world entities?

Imagine that Alice has a state-issued DID and wants to buy some alcohol without disclosing her real name and precise age.

The answer is to use “verifiable claims” (aka: credentials, attestations).

1. claim = properties we know about the entity in subject-property-value relationships, e.g. name, email, age, membership, etc.
2. verifiable = proofs (signatures) attached

Verifiable Claims Data Model​

Identity Profile

A simple identity profile​

{
  "id": "did:ebfeb1f712ebc6f1c276e12ec21",
  "type": ["Identity", "Person"],
  "name": "Alice Bobman",
  "email": "[email protected]",
  "birthDate": "1985-12-14",
  "telephone": "12345678910"
}

A simple claim​

{
  "id": "http://example.gov/credentials/3732",
  "type": ["Credential", "ProofOfAgeCredential"],
  "issuer": "https://dmv.example.gov",
  "issued": "2010-01-01",
  "claim": {
    "id": "did:ebfeb1f712ebc6f1c276e12ec21",
    "ageOver": 21
  }
}

A simple verifiable claim​

{
  "@context": "https://w3id.org/security/v1",
  "id": "http://example.gov/credentials/3732",
  "type": ["Credential", "ProofOfAgeCredential"],
  "issuer": "https://dmv.example.gov",
  "issued": "2010-01-01",
  "claim": {
    "id": "did:ebfeb1f712ebc6f1c276e12ec21",
    "ageOver": 21
  },
  "revocation": {
    "id": "http://example.gov/revocations/738",
    "type": "SimpleRevocationList2017"
  },
  "signature": {
    "type": "LinkedDataSignature2015",
    "created": "2016-06-18T21:19:10Z",
    "creator": "https://example.com/jdoe/keys/1",
    "domain": "json-ld.org",
    "nonce": "598c63d6",
    "signatureValue": "BavEll0/I1zpYw8XNi1bgVg/sCneO4Jugez8RwDg/+
    MCRVpjOboDoe4SxxKjkCOvKiCHGDvc4krqi6Z1n0UfqzxGfmatCuFibcC1wps
    PRdW+gGsutPTLzvueMWmFhwYmfIFpbBu95t501+rSLHIEuujM/+PXr9Cky6Ed
    +W3JT24="
  }
}

It equals to a JOSE JWT verifiable claim

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2Rtdi
5leGFtcGxlLmdvdiIsImlhdCI6MTI2MjMwNDAwMCwiZXhwIjoxNDgzMjI4ODAwL
CJhdWQiOiJ3d3cuZXhhbXBsZS5jb20iLCJzdWIiOiJkaWQ6ZWJmZWIxZjcxMmVi
YzZmMWMyNzZlMTJlYzIxIiwiZW50aXR5Q3JlZGVudGlhbCI6eyJAY29udGV4dCI
6Imh0dHBzOi8vdzNpZC5vcmcvc2VjdXJpdHkvdjEiLCJpZCI6Imh0dHA6Ly9leG
FtcGxlLmdvdi9jcmVkZW50aWFscy8zNzMyIiwidHlwZSI6WyJDcmVkZW50aWFsI
iwiUHJvb2ZPZkFnZUNyZWRlbnRpYWwiXSwiaXNzdWVyIjoiaHR0cHM6Ly9kbXYu
ZXhhbXBsZS5nb3YiLCJpc3N1ZWQiOiIyMDEwLTAxLTAxIiwiY2xhaW0iOnsiaWQ
iOiJkaWQ6ZWJmZWIxZjcxMmViYzZmMWMyNzZlMTJlYzIxIiwiYWdlT3ZlciI6Mj
F9fX0.LwqH58NasGPeqtTxT632YznKDuxEeC59gMAe9uueb4pX_lDQd2_UyUcc6
NW1E3qxvYlps4hH_YzzTuXB_R1A9UHXq4zyiz2sMtZWyJkUL1FERclT2CypX5e1
fO4zVES_8uaNoinim6VtS76x_2VmOMQ_GcqXG3iaLGVJHCNlCu4

DID and IOT​

slides

Takeaways​

DID

• is probably a Differentiator for identity-oriented vendors but a Neutralizer (out of MMRs, neutralizers, and differentiators features) for others.
• preserves privacy in a large group of products, so that no central company could know all your stuff from all your accounts from various products.
• combined with verifiable claims, it leverage crypto but still requires trust in the physical world.

What blockchain developers can do?

• Join DIF.
• Proposing a DID method on how to operate DIDs on their blockchain.
• Building our own decentralized identity provider.

References​

• https://w3c-ccg.github.io/did-primer/
• https://w3c-ccg.github.io/did-spec/
• https://blockstack.github.io/blockstack.js/
• https://medium.com/decentralized-identity/the-rising-tide-of-decentralized-identity-2e163e4ec663?source=collection_home---6------6-----------------------
• https://medium.com/decentralized-identity/a-universal-resolver-for-self-sovereign-identifiers-48e6b4a5cc3c?source=collection_home---6------5-----------------------
• https://github.com/blockstack/blockstack-core/blob/master/docs/blockstack-did-spec.md
• https://www.w3.org/TR/verifiable-claims-use-cases/
• https://www.w3.org/2017/vc/WG/
• https://www.w3.org/TR/vc-data-model/

Answer: Amazon wanted to buy a customer for its grocery services.

Background​

• Amazon acquiring Whole Foods = Apple’s iPhone beating Palm

  • Don’t misunderstand goals vs strategies vs tactics - Apple’s strategy
    • is not to build a phone but to build personal computer
    • is not to add functionalities to a phone but to reduce the phone to an app
    • is not to duplicate the carriers but to leverage their customer connections
  • iPhone is the most successful product of all time = Amazon is the most dominant company of all time

• Amazon's Goal

  1. Initially, Amazon.com’s objective is to be the leading online retailer of information-based products and services, with an initial focus on books.
  2. Then, it says "our vision is to be earth’s most customer centric company; to build a place where people can come to find and discover anything they might want to buy online."
  3. ==Amazon’s goal is to take a cut of all economic activity.==

• Amazon's Strategy

  • to enterprise: AWS. Assuming that all businesses will soon be Internet-enabled businesses.
  • to customer: Prime. Assuming that superior cost/and/superior selection are not sustainable. With prime, alternatives won’t be even considered by customers.
    • However
      • grocery is the largest retail category.
      • grocery is the most persistent opportunity for reminding users there are other alternatives.

• Tactics: develop grocery services

Why hadn’t Amazon figured out the right tactics?​

AmazonFresh’s cost disadvantage

1. High costs of perishable items if not scaled
2. Scale needs to be based on cities

Why can acquiring Whole Food (not doing other things) solve the scale problem?

==Primitives model for business with 1) hight fixed costs 2) high returns to scale==

• decouple infrastructure into Minimal Sellable Units (MSUs)
• business itself is ==The First-And-Best Customer== of those MSUs
• resell MSUs to the outside

AWS Three Layers

• MSUs are S3, EC2, RDS, SNS, etc
• The First-And-Best Customer is amazon.com
• resell MSUs to non-Amazon developers

Amazon.com Three Layers

• MSUs are FDA, Amazon Pay, etc.
• The First-And-Best Customer is Amazon first-party e-commerse
• resell MSUs to 3rd party merchants

The insight here is that grocery business has no first-and-best customer.

Perfect Customer​

After fitting in Whole Foods to the big picture, we can see that ==Amazon is buying more than a retailer - it’s buying a customer==.

Amazon.com Three Layers + Customers

Now Amazon Grocery Services can serve AmazonFresh and WholeFoods, and then in the future restaurants or whatever can consume it.

When faced with uncertainty, people often rely on heuristics to make judgments. Heuristics can be useful, but they can also lead to biases: for example, the judgment that not being able to see clearly = being far away is correct in most cases, but if you intentionally obscure something, people may mistakenly believe it is farther away.

When judging probabilities and valuations, people often use three heuristics: representativeness, availability, and adjustment and anchoring.

Representativeness Heuristic​

1. ==People often confuse representativeness with probability==. A neighbor describes Steve as shy, taciturn, helpful, but uninterested in people and the world. When asked to judge which profession is more likely for him: farmer, salesperson, pilot, librarian, or doctor, people will mistakenly believe Steve is a librarian because it fits their stereotype of that profession.
2. So what factors can help improve probability judgments?
   1. Prior probability, also known as ==base rates==. When considering representativeness, especially when presented with meaningless evidence, people tend to subconsciously ignore base rates and incorrectly use representativeness to make judgments. In the case of Steve, applying prior probability and Bayesian reasoning is necessary to arrive at the correct answer.
   2. Sample size. Sometimes, sample size can better explain issues than the patterns revealed by the sample.
   3. Correct understanding of odds. The gambler's fallacy, for instance, is the incorrect belief that probabilities will self-correct to the mean during a game of chance, while in reality, it is merely self-dilution; the probability of rolling ten "highs" in a row is the same as rolling a "high" and a "low" fifty-fifty.
   4. Predictability. The higher the predictability, the broader the range of estimated values. For example, a brief assessment of a teacher's teaching level should not be sufficient to judge that teacher's performance five years later.
   5. Caution against overconfidence leading to an illusion of validity in one's predictions, caused by confirmation bias, where people tend to seek evidence that supports their views, pay more attention to information that backs their opinions, or interpret information in ways that support their beliefs. Judgments based on several independent events tend to be more accurate than those based on redundant, correlated events.
   6. Regression to the mean. People often overlook this trend, and even when they are aware of it, they can easily misinterpret its causes. For example, in pilot training, instructors find that praising a pilot after a smooth landing leads to relatively worse performance next time; conversely, criticizing after a hard landing leads to relatively better performance next time. Instructors may mistakenly believe that praise and criticism caused the subsequent performance changes, while in reality, it is merely regression to the mean. Thus, people often overestimate the effects of criticism and underestimate the effects of praise.