---
title: "System Design Basics Databases Message Queues and Authentication"
author: "Rantideb Howlader"
date: "2026-06-01T00:00:00.000Z"
canonical_url: "https://www.ranti.dev/blog/databases-and-message-queues-system-design"
license: "CC-BY-4.0"
---


In our previous guide we covered caching and how it makes your application fast. Now we will look at the rest of the core system design puzzle.

We will cover exactly where your data lives, how parts of your system talk to each other, and how you secure it all.

This guide covers Caching flows, Databases, Message Queues, Authentication, and Proxies.

## 1. Caching Eviction Strategy

When we store data in cache (RAM), after some time we have to delete old data so new data can be stored. This process is called Cache Eviction.

Why evict? Because cache space is limited. We need to remove unnecessary or old data to serve fresh and relevant data.

Here is how the eviction and cache miss flow works:

```mermaid
graph TD
    A[Request from User] --> B{Check in Cache}
    B -->|Hit| C[Return Data from Cache Fast]
    B -->|Miss| D[Fetch Data from Database]
    D --> E[Store in Cache New Data]
    E --> C
```

**Eviction Triggers:**

- **TTL (Time To Live) expire:** Data is deleted when its timer ends automatically.
- **Cache full:** Space is needed for new data.
- **Data updated / deleted in DB:** The original data changed.
- **Manual invalidation:** The developer manually clears the cache.

Real world systems mostly use LRU (Least Recently Used) and LFU (Least Frequently Used) eviction strategies.

## 2. File Based DBMS vs RDBMS

Before modern databases existed applications used file based systems.

### What is File Based DBMS

In this system data is stored in simple files. Every application creates its own files and reads or writes to them. Examples are `Users.txt`, `Products.txt`, and `Orders.txt`.

**Advantages for small scale systems:**

- Simple to implement.
- Easy to understand.
- Low cost.
- No complex setup required.

**Disadvantages and Challenges:**

- **Data Redundancy:** The same data gets copied in multiple files.
- **Inconsistency:** If you update data in one file you might forget to update the other file.
- **Poor Security:** Anyone with file access can read everything.
- **Data Isolation:** Hard to connect data between different files.
- **Hard to Maintain:** Code becomes messy.
- **Slow Performance:** Searching through large files takes a lot of time.

### What is RDBMS

RDBMS stands for Relational Database Management System. It is software that stores data in tables. A table has rows (records) and columns (fields). Tables maintain relationships with each other. Examples include MySQL, PostgreSQL, Oracle, and SQL Server.

**RDBMS Features:**

- Data integrity.
- Relationships using Keys.
- SQL support.
- ACID properties.
- Security and permissions.
- Backup and recovery.
- Concurrency control.
- High performance and scalability.

Here is a visual comparison of how they store data:

```mermaid
graph TD
    subgraph File Based
    A[Application] --> B(Users.txt)
    A --> C(Products.txt)
    A --> D(Orders.txt)
    end
    subgraph RDBMS
    E[Application] -->|SQL Queries| F[(RDBMS Engine)]
    F -->|Data| G[(Storage Disk)]
    end
```

### File Based DBMS vs RDBMS Comparison

| Feature           | File Based DBMS    | RDBMS               |
| ----------------- | ------------------ | ------------------- |
| Data Storage      | Files              | Tables              |
| Redundancy        | High               | Low                 |
| Consistency       | Low                | High                |
| Security          | Poor               | Strong              |
| Performance       | Slow               | Fast                |
| Scalability       | Difficult          | Easy                |
| Relationship      | Not supported      | Supported via Keys  |
| Backup / Recovery | Manual             | Built-in            |
| Use Case          | Small, Simple apps | Large, Complex apps |

**When to Use What:**
Use File Based DBMS when you have a small application, simple requirements, a low budget, and data that rarely changes.
Use RDBMS when you have large and complex data, need high consistency, have multiple users, care about security, and need high performance.

## 3. Authentication vs Authorization

Security is critical in system design. You must understand the difference between proving who you are and proving what you can do.

**Authentication** answers "Who are you?". It is the process of verifying the identity of a user.
**Authorization** answers "What can you access?". It is the process of determining what resources or actions a user is allowed to access after they log in.

Here is how they work together:

```mermaid
graph LR
    A[User] --> B[Login]
    B --> C[Authentication Who are you?]
    C -->|Success| D[Authorization What can you access?]
    D --> E[Resources]
```

Think of Google Drive as a real world example:

```mermaid
graph LR
    A[User Login] --> B[Verify Identity]
    B --> C[User Identified]
    C --> D[Check Permissions]
    D --> E[View Files]
    D --> F[Edit Files]
```

1. **User Login:** You provide your credentials.
2. **Authentication:** Google verifies your identity.
3. **User Identified:** You are confirmed as a valid user.
4. **Authorization:** Google checks your permissions for a specific document.
5. **Resources:** You are allowed to view or edit the file.

### Auth vs Authz Quick Comparison

| Feature  | Authentication                | Authorization                  |
| -------- | ----------------------------- | ------------------------------ |
| Meaning  | Verifies identity of the user | Checks permissions of the user |
| Question | Who are you?                  | What can you access?           |
| Step     | First Step                    | Second Step                    |
| Example  | Login                         | Access Control                 |

## 4. Auth Types and Flows

There are three main ways to handle authentication.

### Basic Authentication

The client sends a username and password with every single request. The server verifies it and responds. The header looks like this: `Authorization: Basic base64(username:password)`.

```mermaid
sequenceDiagram
    participant Client
    participant Server
    Client->>Server: 1. Request Username + Password
    Server-->>Client: 2. Response
```

- **Problems:** You must send the password every time. If the network is insecure it is easy to sniff. It is not scalable.
- **Real World Uses:** Internal tools, legacy applications, testing APIs.

### Token Based Authentication (JWT)

You send your password once. The server generates a JWT (JSON Web Token) and gives it to you. For future requests you just send the token.

The token has three parts: Header, Payload, and Signature. It looks like `header.payload.signature`.

```mermaid
sequenceDiagram
    participant Client
    participant Server
    Client->>Server: 1. Login Username + Password
    Server-->>Client: 2. Generate JWT Token
    Client->>Server: 3. Future Request Bearer Token
    Server-->>Client: 4. Response
```

- **Token Expiry:** Tokens have a timer. Created -> Valid -> Expired. After it expires the user must log in again.
- **Real World Uses:** Instagram, Facebook, LinkedIn, Amazon, Netflix, Spotify.

### OAuth (One Click Authentication)

If users hate creating new passwords you use OAuth. It allows a trusted provider like Google or Github to verify the user for you.

```mermaid
sequenceDiagram
    participant User
    participant Application
    participant Provider Google
    User->>Application: 1. Click Login with Google
    Application->>Provider Google: 2. Redirect to Google
    User->>Provider Google: 3. Login and Consent
    Provider Google-->>Application: 4. Send Authorization Code
    Application-->>User: 5. User Logged In
```

## 5. Proxy Servers

A proxy server is a hardware or software that sits between a client and a server. It provides intermediary services in communication.

Think of a classroom analogy. You want attendance but you are absent. You ask a friend (Y) to say "Present" for you (X). Y acts as a proxy for X.

### Forward Proxy (Client Side)

The proxy sits in front of the client. It hides the client identity so the server does not know who the real requester is.

```mermaid
graph LR
    A[Client User] --> B[Forward Proxy VPN]
    B --> C[Internet Website]
```

- **Example:** A user in India uses a VPN proxy in the USA to visit a website. The website thinks the request came from the USA.

### Reverse Proxy (Server Side)

The proxy sits in front of the server. It hides the server identity. The client does not know which actual server handled the request.

```mermaid
graph LR
    A[Client User] --> B[Reverse Proxy Nginx]
    B --> C[Server 1]
    B --> D[Server 2]
    B --> E[Server 3]
```

- **Real World Uses:** Nginx, Cloudflare, Load Balancers, CDN (Content Delivery Network).
- **Benefits:** Load balancing, caching, security, SSL termination.

### Proxy Types Comparison

| Feature   | Forward Proxy           | Reverse Proxy                     |
| --------- | ----------------------- | --------------------------------- |
| Works For | Client (Users)          | Server (Applications)             |
| Hides     | Client Identity         | Server Identity                   |
| Position  | Client Side             | Server Side                       |
| Main Use  | Privacy, Access Control | Load Balancing, Security, Caching |

## 6. Normalization, Denormalization & Indexing

Databases need to be structured correctly for speed and efficiency.

### Normalization (Reduce Redundancy)

Normalization means dividing a large table into smaller tables to reduce redundancy. The key idea is to keep things separate. Instead of writing department details on every employee row, store department details in one place.

- **Benefits:** Reduces duplicate data. Saves storage space. Improves data integrity. Makes maintenance easier.

**Example: Student and Course**

- **Before Normalization:** Alice takes DBMS. Alice takes OS. Alice is written twice.
- **After Normalization:** Student Table (Alice, Bob). Course Table (DBMS, OS). You link them together. Redundancy is gone.

### Denormalization (Improve Performance)

Denormalization means combining multiple tables into fewer tables. This reduces the number of joins needed.

- **Benefits:** Faster data read operations. High data availability. Reduces network calls to fetch data.
- **Challenges:** Redundant data wastes memory. Increases complexity. Data inconsistency if you forget to update all copies. Slow write operations.

**Real World Example:**
In an E-Commerce app you want to show product details quickly. Instead of joining the Product table, Category table, and Review table every time, you denormalize them into one fast read table. Use Denormalization in read intensive systems.

### Indexing (Search Fast)

Indexing speeds up data retrieval. It creates a lookup structure like a book index so the database finds data without scanning every single row.

Most databases use a B-Tree (Balanced Tree) structure for indexing:

```mermaid
graph TD
    A[Root Node 1000 to 4000] --> B[Node 1000]
    A --> C[Node 4000]
    B --> D[Node 500]
    B --> E[Node 2000]
```

- **How it Works:** The database creates a separate data structure. It stores the indexed column values and a pointer to the actual row address on the disk.
- **Time Complexity:** Without index it takes O(N) time (scanning every row). With a B-Tree index it uses binary search which takes O(log N) time.
- **When to Use:** Read intensive applications, search operations, Google Search, YouTube Video Search, Banking Reports.
- **When NOT to Use:** Write intensive applications. Every INSERT, UPDATE or DELETE forces the index to update. This adds extra overhead.

## 7. NoSQL Databases

RDBMS is great for structured data. But when data becomes massive RDBMS is hard to scale. NoSQL databases offer horizontal scaling and high availability.

NoSQL is an umbrella term for four different types:

```mermaid
graph TD
    A[NoSQL Databases] --> B[Key-Value Database]
    A --> C[Document Database]
    A --> D[Columnar Database]
    A --> E[Graph Database]
```

**1. Key-Value Database**
Data is stored as a simple Key and Value pair. It is extremely fast. It works like a giant dictionary.

- **Real Applications:** Session storage, shopping cart, cache storage, OTP storage.
- **Popular Databases:** Redis, DynamoDB, Riak.

**2. Document Database**
Data is stored in JSON like documents. There is no fixed schema. Every document can have different fields.

- **Real Applications:** User profiles, product catalogs, blog systems, CMS platforms.
- **Popular Databases:** MongoDB.

**3. Columnar Database**
Data rows are stored in columns instead of rows. If you want to find the maximum salary it only scans the salary column, not the whole table.

- **Real Applications:** Data warehouses, business analytics, machine learning, reporting systems.
- **Popular Databases:** Cassandra, HBase, ClickHouse.

**4. Graph Database**
Data is stored as a graph with Nodes and Edges. It is perfect for highly connected data where relationships matter most.

- **Real Applications:** Social networks (friends of friends), Google Maps, LinkedIn connections, recommendation systems.
- **Popular Databases:** Neo4j, Amazon Neptune.

Here is a visual of how a Graph Database models social networks:

```mermaid
graph LR
    A((Suhani)) -- Friend --> B((Rahul))
    A -- Friend --> C((Aman))
    B -- Friend --> D((Priya))
    C -- Friend --> D
```

## 8. Polyglot Persistence

When an application's requirements cannot be efficiently fulfilled by one database, multiple databases are used together. This is called Polyglot Persistence.

Here is how an E-Commerce application splits its database needs:

```mermaid
graph TD
    A[E-Commerce Application] --> B[User Management]
    A --> C[Shopping Cart & Session]
    A --> D[Analytics & Reporting]
    A --> E[Social Graph Network]

    B --> F[(MySQL RDBMS)]
    C --> G[(Redis Key-Value)]
    D --> H[(Cassandra Columnar)]
    E --> I[(Neo4j Graph)]
```

- **MySQL (RDBMS):** User management. Structured data of users and orders.
- **Redis (Key-Value):** Shopping cart and sessions. Fast read and write for temporary data.
- **Cassandra (Columnar):** Analytics and reporting. Large scale logs and dashboards.
- **Neo4j (Graph):** Social graph and network. Relationships between users and product recommendations.

## 9. Communication Patterns in System Design

When services talk to each other they use specific patterns.

### Synchronous Communication (Blocking)

The sender sends a request and waits for the receiver to process it. The sender cannot proceed until the response comes back. This is a blocking call.

Think of an ATM machine flow:

```mermaid
graph LR
    A[Insert Card] --> B[Enter PIN]
    B --> C[Request Cash]
    C --> D[Waiting Please wait]
    D --> E[Cash Dispensed]
    E --> F[Process Complete]
```

- **Why use it?** When you need strong consistency. If you have a primary DB and 3 replicas, synchronous replication means the primary waits for all replicas to update before sending a success response.
- **Real World Uses:** Stock market trades, bank payments, ticket booking, real time decision making (fraud detection).

### Asynchronous Communication (Non-Blocking)

The sender sends a request and does not wait. The request is processed in the background. The sender moves on to other work immediately.

Think of a cake shop. You order a cake. You do not wait at the shop. The shopkeeper says "Come back later". You leave and work continues.

**Message Queue / Broker Architecture:**
To make async communication work you use a message queue.

```mermaid
graph LR
    A[Producer Service A] --> B[(Message Queue Kafka/RabbitMQ)]
    B --> C[Consumer Service B]
```

1. The Producer drops a message in the queue.
2. The Consumer picks it up and processes it later.
   This decouples the services completely. Use it for sending emails, processing videos, and logging.

### Synchronous vs Asynchronous Feature Comparison

| Feature        | Synchronous                    | Asynchronous                   |
| -------------- | ------------------------------ | ------------------------------ |
| Waiting        | Yes, sender waits for response | No, sender does not wait       |
| Blocking       | Blocking call                  | Non-blocking call              |
| Consistency    | High (Strong Consistency)      | Eventual Consistency           |
| Complexity     | Simple to implement            | More complex (queues, retries) |
| Performance    | Lower (due to waiting)         | Higher (no waiting)            |
| Scalability    | Limited                        | Highly Scalable                |
| Failure Impact | High (whole flow blocks)       | Low (other services continue)  |

### Detailed Real World Examples

**1. E-Commerce Checkout Flow**

- Add item to cart -> **(Sync)** Inventory check
- Apply coupon -> **(Sync)** Validate coupon
- Make payment -> **(Sync)** Payment gateway
- Order Create -> **(Sync)** Save order in DB
- Send Confirmation Email -> **(Async)** No need to wait
- Update Analytics -> **(Async)** No need to wait
- Notify Warehouse -> **(Async)** No need to wait

**2. Social Media Post**

- User creates post -> **(Sync)** Save post
- Show on user timeline -> **(Sync)** Must wait
- Notify followers -> **(Async)** Background task
- Update Feed / Cache -> **(Async)** Background task
- Analytics tracking -> **(Async)** Background task
  The user needs to see their post immediately (Sync). Everything else can happen in the background (Async).

**3. Ride Booking (Ola / Uber)**
Modern systems use a mix of both patterns. When you book a ride:

- Find nearby drivers -> **(Sync)** Critical step.
- Driver accepts -> **(Sync)** Critical step.
- Ride starts -> **(Sync)** Critical step.
- Live tracking update -> **(Async)** Background step.
- Bill generation -> **(Async)** Background step.
- Send receipt email -> **(Async)** Background step.

Synchronous is for correctness. Asynchronous is for speed and scale. Real systems use a combination of both.


---

<!-- METADATA_START -->
## Metadata & Citations

### Further Reading
- [System Design Basics Caching How to Make Your App Fast](https://www.ranti.dev/blog/caching-system-design-guide.md)
- [System Design Basics Scalability and Load Balancing](https://www.ranti.dev/blog/scalability-load-balancing-system-design.md)
- [The Perfect Pipeline: How to Ship Code Without Crashing Production](https://www.ranti.dev/blog/perfect-pipeline-blue-green.md)

### Navigation
- [Back to Bio Hub](https://www.ranti.dev/.md)
- [Full Site Manifest](https://www.ranti.dev/llms.txt)

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "System Design Basics Databases Message Queues and Authentication",
  "author": {
    "@type": "Person",
    "name": "Rantideb Howlader"
  },
  "datePublished": "2026-06-01T00:00:00.000Z",
  "url": "https://www.ranti.dev/blog/databases-and-message-queues-system-design",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "isAccessibleForFree": true
}
```

### BibTeX
```bibtex
@article{databases-and-message-queues-system-design_2026,
  author = {Rantideb Howlader},
  title = {System Design Basics Databases Message Queues and Authentication},
  journal = {Rantideb Howlader Portfolio},
  year = {2026},
  url = {https://www.ranti.dev/blog/databases-and-message-queues-system-design},
  note = {Accessed: 2026-06-01}
}
```

### IEEE
Rantideb Howlader, "System Design Basics Databases Message Queues and Authentication," Rantideb Howlader Portfolio, 2026. [Online]. Available: https://www.ranti.dev/blog/databases-and-message-queues-system-design. [Accessed: 2026-06-01].

### APA
Rantideb Howlader. (2026). System Design Basics Databases Message Queues and Authentication. Rantideb Howlader. Retrieved from https://www.ranti.dev/blog/databases-and-message-queues-system-design

--- 
*This content is provided in research-grade Markdown format. Required Attribution: Cite as Rantideb Howlader (2026).*
<!-- METADATA_END -->