From the free course Computer Applications and Information Technology on Lykke

Cloud Computing and Data Analytics

Key concepts: Cloud Computing · Cloud Infrastructure · Data Analytics · Data Modeling · SaaS/PaaS/IaaS

Exploring modern infrastructure through cloud computing and the use of data analytics for business modeling.

Cloud Computing and Data Analytics: The Architectural Synergy

The modern enterprise has undergone a fundamental shift from localized, capital-intensive hardware deployments to a decentralized, utility-based model of computing. This transition, known as the Cloud Revolution, is not merely a change in where code executes, but a radical reimagining of how data is captured, stored, and transformed into actionable intelligence. At the intersection of this shift lies Data Analytics, a discipline that leverages the near-infinite scalability of cloud infrastructure to solve problems that were computationally intractable only a decade ago.

The Genesis of Cloud Computing

Cloud computing is defined as the on-demand delivery of IT resources over the internet with pay-as-you-go pricing. Rather than buying, owning, and maintaining physical data centers and servers, organizations access technology services, such as computing power, storage, and databases, from providers like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

Definition: The NIST Model According to the National Institute of Standards and Technology (NIST), cloud computing is characterized by five essential characteristics: On-demand self-service, Broad network access, Resource pooling, Rapid elasticity, and Measured service.

Why Cloud Computing Matters

Historically, IT departments operated under a CAPEX (Capital Expenditure) model. To launch a new application, one had to estimate peak load, purchase hardware, wait for shipping, and manually configure the stack. This led to "zombie servers"—underutilized hardware that drained resources. Cloud computing introduces the OPEX (Operational Expenditure) model, where costs scale linearly with usage. This enables Rapid Prototyping and Fail-Fast methodologies, as the cost of experimentation is drastically reduced.

Cloud Service Models: IaaS, PaaS, and SaaS

The "Cloud" is not a monolithic entity but a spectrum of abstraction. The choice of service model determines the "Shared Responsibility" between the provider and the consumer.

Model Acronym Description User Manages Provider Manages
Infrastructure as a Service IaaS Fundamental building blocks of computing. OS, Middleware, Data, Apps Virtualization, Servers, Storage, Networking
Platform as a Service PaaS Framework for developers to build/deploy apps. Applications, Data OS, Runtime, Middleware, Hardware
Software as a Service SaaS Completed software products delivered via web. Nothing (Configuration only) Everything (The entire stack)

The Shared Responsibility Matrix

In an IaaS environment, the user is responsible for patching the Operating System (OS). In a SaaS environment, the provider handles everything from the physical security of the data center to the encryption of data at rest within the application.

Cloud Infrastructure and Virtualization

The "magic" of the cloud relies on Virtualization. This is the process of creating a software-based (virtual) representation of something, such as virtual applications, servers, storage, and networks.

The Hypervisor

The core component of virtualization is the Hypervisor (or Virtual Machine Monitor). It is a layer of software that sits between the physical hardware and the Operating Systems, allowing multiple "Guest" OSs to share the same physical "Host" resources.

  1. Type 1 (Bare Metal): Runs directly on the host's hardware (e.g., VMware ESXi, Xen).
  2. Type 2 (Hosted): Runs on a conventional OS (e.g., Oracle VirtualBox).

Code Example 1: Low-Level Resource Management

In a cloud environment, we often interact with infrastructure via APIs. Below is a Python implementation using a hypothetical SDK to demonstrate how a cloud controller might balance load by spinning up new instances based on CPU utilization metrics.

import time

class CloudController:
    def __init__(self, threshold=80.0):
        self.threshold = threshold
        self.active_instances = []

    def get_cpu_utilization(self, instance_id):
        # In a real system, this calls a monitoring API like CloudWatch
        # Returns a float representing percentage
        pass

    def scale_up(self):
        new_id = f"inst-{len(self.active_instances) + 1}"
        print(f"[ACTION] Provisioning new instance: {new_id}")
        self.active_instances.append(new_id)

    def monitor_and_scale(self):
        while True:
            total_load = sum(self.get_cpu_utilization(i) for i in self.active_instances)
            avg_load = total_load / len(self.active_instances) if self.active_instances else 100
            
            print(f"[MONITOR] Average Cluster Load: {avg_load}%")
            
            if avg_load > self.threshold:
                self.scale_up()
            
            time.sleep(60) # Polling interval

# Note: This represents the logic behind 'Auto-scaling Groups' in Cloud Infrastructure.

Data Analytics: Extracting Value from the Cloud

If Cloud Computing is the engine, Data Analytics is the fuel. Data Analytics is the science of analyzing raw data to make conclusions. In the cloud, this is often categorized into four types:

  1. Descriptive: What happened? (e.g., Monthly sales reports)
  2. Diagnostic: Why did it happen? (e.g., Drill-down into churn data)
  3. Predictive: What will happen? (e.g., Forecasting demand using ML)
  4. Prescriptive: How can we make it happen? (e.g., Optimization algorithms)

The Big Data Pipeline

Cloud-native analytics typically follows a pipeline architecture: Ingestion (Kafka, Kinesis) → Storage (S3, Data Lake) → Processing (Spark, Databricks) → Analysis/Visualization (Tableau, Looker).

Data Modeling and Structuring

To analyze data effectively, it must be modeled. Data Modeling is the process of creating a visual representation of either a whole information system or parts of it to communicate connections between data points and structures.

Relational vs. Non-Relational

  • RDBMS (SQL): Structured data, ACID compliance, fixed schema. Best for transactional data (e.g., banking).
  • NoSQL: Unstructured or semi-structured data (JSON), horizontal scalability. Best for big data and real-time web apps.

The Star Schema

In cloud data warehousing (like Amazon Redshift or Snowflake), the Star Schema is the gold standard for performance. It consists of:

  • Fact Tables: Quantitative data (e.g., sale_amount, quantity).
  • Dimension Tables: Descriptive attributes (e.g., product_name, store_location).
Feature Star Schema Snowflake Schema
Normalization Denormalized Normalized
Query Complexity Low (fewer joins) High (more joins)
Data Redundancy High Low
Performance Faster for OLAP Slower but more organized

Code Example 2: Infrastructure as Code (IaC)

To deploy these analytical environments, engineers use Declarative Configuration. Below is a pseudocode representation (similar to Terraform/HCL) for defining a Cloud Data Warehouse and its associated storage.

# Define a Cloud Storage Bucket for the Data Lake
resource "cloud_storage_bucket" "raw_data_lake" {
  name          = "enterprise-analytics-raw-prod"
  location      = "US-EAST-1"
  storage_class = "STANDARD"
  encryption    = "AES256"
}

# Define a Data Warehouse Instance
resource "cloud_data_warehouse" "main_dw" {
  cluster_identifier = "bi-analytics-cluster"
  node_type          = "dw2.large"
  number_of_nodes    = 4
  database_name      = "sales_analytics"
  
  # Link to the Data Lake for ELT processes
  iam_roles = [cloud_iam_role.dw_read_access.arn]
}

Advanced Data Analytics: The Power of Big Data

Big Data is often defined by the 5 Vs: Volume, Velocity, Variety, Veracity, and Value. Cloud computing solves the "Volume" and "Velocity" problems through Distributed Computing.

Amdahl's Law and Parallelism

In data processing, we aim to parallelize tasks. However, Amdahl's Law reminds us that the speedup of a program using multiple processors is limited by the sequential fraction of the program.

Amdahl's Law Formula $S_{latency}(s) = \frac{1}{(1-p) + \frac{p}{s}}$ Where:

  • $S$ is the theoretical speedup.
  • $s$ is the number of processors.
  • $p$ is the proportion of the execution time that the part benefiting from improved resources originally occupied.

This formula explains why cloud analytics platforms focus on Horizontal Scaling (adding more machines) rather than just Vertical Scaling (adding more RAM/CPU to one machine).

Concrete Example: E-commerce Analytics Pipeline

Imagine an e-commerce giant processing 10,000 transactions per second.

  1. Ingestion: User clicks are streamed via cURL or SDKs to a message bus.
  2. Storage: The raw JSON is dumped into an S3 bucket (Data Lake).
  3. Transformation: A nightly Spark job transforms JSON into a structured Parquet format.
  4. Modeling: Data is loaded into a Star Schema in a Data Warehouse.
  5. Querying: Business analysts run SQL queries to find the "Top 10 products by region."

Code Example 3: Analytical SQL Query

This SQL query demonstrates a multi-table join in a Star Schema to calculate regional performance—a classic "Descriptive Analytics" task.

SELECT 
    d.region_name,
    p.category,
    SUM(f.sales_amount) AS total_revenue,
    COUNT(DISTINCT f.transaction_id) AS transaction_count
FROM 
    fact_sales f
JOIN 
    dim_location d ON f.location_key = d.location_key
JOIN 
    dim_product p ON f.product_key = p.product_key
WHERE 
    f.order_date >= '2023-01-01'
GROUP BY 
    d.region_name, p.category
HAVING 
    SUM(f.sales_amount) > 10000
ORDER BY 
    total_revenue DESC;

Common Pitfalls in Cloud and Analytics

Even with powerful tools, several common mistakes can lead to project failure or massive cost overruns.

  1. The "Lift and Shift" Trap: Moving a legacy application to the cloud without re-architecting it for cloud-native features (like auto-scaling). This often results in higher costs than on-premise.
  2. Data Silos: Creating fragmented data sets that cannot be joined, leading to "multiple versions of the truth."
  3. Ignoring Egress Costs: Cloud providers often charge very little to bring data in, but significant amounts to take data out.
  4. Over-provisioning: Allocating more resources than necessary.
  5. Security Misconfiguration: Leaving S3 buckets or databases open to the public internet—a leading cause of data breaches.

Ethics, Privacy, and Governance

As we aggregate massive datasets in the cloud, ethical considerations become paramount. Data Sovereignty laws (like GDPR in Europe) dictate where data can be stored and processed. Organizations must implement Data Governance frameworks to ensure data quality, security, and compliance.

  • Anonymization: Removing PII (Personally Identifiable Information) before analysis.
  • Auditability: Keeping logs of who accessed what data and when.
  • Bias in Analytics: Ensuring that predictive models do not reinforce historical biases (e.g., in hiring or lending algorithms).

Summary of Key Formulae and Concepts

  • Total Cost of Ownership (TCO): $TCO = Acquisition + Operating + Maintenance$. In the cloud, $Acquisition$ approaches zero.
  • Availability: Measured in "Nines." 99.9% (Three Nines) allows for ~9 hours of downtime per year; 99.999% (Five Nines) allows for only ~5 minutes.
  • ETL vs. ELT:
    • ETL (Extract, Transform, Load): Data is transformed before reaching the warehouse.
    • ELT (Extract, Load, Transform): Data is loaded raw into the warehouse and transformed using the warehouse's compute power (preferred in modern cloud architectures).
Metric On-Premise Cloud (IaaS/PaaS)
Scaling Speed Weeks/Months Seconds/Minutes
Cost Structure Fixed (CAPEX) Variable (OPEX)
Maintenance High (Physical) Low (Software-defined)
Global Reach Difficult Instant (Multi-region)

The synergy of Cloud Computing and Data Analytics has democratized high-performance computing. Today, a startup with a credit card has access to the same computational power that was once reserved for global superpowers, enabling a new era of data-driven innovation.

Cloud Computing and Data Analytics - Computer Applications and Information Technology - image 1
Cloud Computing and Data Analytics - Computer Applications and Information Technology - image 1
Cloud Computing and Data Analytics - Computer Applications and Information Technology - diagram 1
Cloud Computing and Data Analytics - Computer Applications and Information Technology - diagram 1
Cloud Computing and Data Analytics - Computer Applications and Information Technology - diagram 2
Cloud Computing and Data Analytics - Computer Applications and Information Technology - diagram 2

Source Materials

Study Computer Applications and Information Technology on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of Computer Applications and Information Technology · Browse more course pages

Cloud Computing and Data Analytics — Computer Applications and Information Technology | Lykke