Big Data Architecture For enterprise

  • Published on

  • View

  • Download


  • Big Data Architecture for Enterprise

    Wei Zhang Big Data Architect

    Up up consultant, LLC

  • Design Principles

    Future-proof, scalable and auto recoverable, compatible with existing technologies, loose coupled and layered architecture

  • Centralized Data Governance service

    Build Schema catalog service to track all data entities and attributes for both structured and unstructured data sets

    Establish and enforce proper practices including solution patterns/design, coding, testing automation and release procedues

  • Logical ArchitectureData Transformation and



    Text files Image files XML files EDI files


    Data Distribution

    BI Reports Text files

    Image files XML files EDI files


    Data Processing Pipeline

    Hadoop HDFS MapReduce

    Hive Pig

    Flume Spark


    NoSql MongoDB Cassandra

    Relational Database

    MS Sql Oracle MySql

  • Logical Architecture Data lifecycle control, access audit, replication

    and DR

    On-desk and in-memory data processing technology stack - sql or nosql database, hadoop map reduce, Spark or ETL tool etc

    Central data inventory services for discovery, tracking and optimization

  • Technology Stack

    HDFS, MapReduce, Yarn

    Oozie, Hive, Spark, Kafka, Cassandra, MongoDB

    BI & Reporting, Data acquisition and distribution, Data inventory and data model

  • Schema Catalog

    MongoDB schema store

    Schemas, Entities, attributes defined using Arvo format

    Define all Data Sources, destinations including format, transfer protocol, file system, schedule etc

  • Data Ledger

    Ledger inventory of all business data set across enterprise

    data set producer and consumer registration

    Data set are tagged and can be queried for traceability and usages

  • Data Process and Persistent Relational database for OLTP, data warehouse

    and BI which need to access SQL database and existing systems

    HDFS for source, destination, staging, no structured document, large to huge data processing, data saved in either Arvo or Parquet format for better exchange and performance

    Cassanadra for high frequency, high write transaction systems and MongoDB for document

  • Automated and Regression Testing

    Maven, SBT, Junit, Scalatest

  • Physical Deployment

    Low End: 7.2 RPM / 75 IOPS, 16 core, 128G (data acquisition and distribution)

    Medium: 15k RPM / 175 IOPS, 24 core, 512G (batch processing)

    High End: 6K - 500K IOPS, 80 core, 1.5T (realtime processing/analytics)


View more >