Satyanarayana Annepogu，加拿大安大略省多伦多的开发者
Satyanarayana is available for hire
Hire Satyanarayana

Satyanarayana Annepogu

Verified Expert  in Engineering

Database Developer

Location
Toronto, ON, Canada
Toptal Member Since
October 25, 2022

Satya是一名高级数据工程师，拥有超过15年的IT经验，为银行和保险客户设计和开发数据仓库. 他擅长使用AWS和Azure数据工程堆栈设计和构建现代数据管道和流. Satya是使用AWS和Azure云数据技术交付企业数据解决方案现代化的专家.

InformaticaData EngineeringData WarehousingData Warehouse DesignData AnalysisData AnalyticsMigrationSQLETLData PipelinesRelational DatabasesDatabasesOracleBusiness Intelligence (BI)Data Integration

Portfolio

Heimstaden Services AB
Azure数据工厂，数据工程，数据管道，SQL...
IBM
亚马逊CloudWatch、亚马逊RDS、亚马逊S3 (AWS S3)、亚马逊EC2...
IBM
Autosys, Azure数据工厂，Azure数据库，Azure SQL, Azure SQL数据库...

Experience

ETL Tools - 14 yearsPython - 4 yearsAzure Databricks - 4 yearsAzure Synapse - 4 yearsApache Airflow - 4 yearsRedshift - 4 yearsAWS Glue - 4 years亚马逊网络服务(AWS)——4年

Availability

Part-time

Preferred Environment

Azure Data Factory, Azure Databricks, Informatica ETL, Amazon Web Services (AWS), Apache Airflow, Redshift, AWS Glue, Python, PostgreSQL 10, Azure Synapse

The most amazing...

...project I've done is designing, developing, 并支持基于云的和传统的数据仓库应用程序.

Work Experience

Data Analyst

2022 - 2023
Heimstaden Services AB
  • 担任具有分析师技能的高级数据工程师，并从事ETL体系结构解决方案的工作.
  • 执行需求评估并设计合适的数据流或数据批.
  • 处理具有数据完整性的解决方案优化和端到端数据管道.
  • 在AWS Glue中设计和开发ETL流程，以各种文件类型(JSON)迁移活动和API数据, ORC, 和Parquet)合并为Amazon RedShift.
  • 设计和开发ETL流程，提取Salesforce数据并将其加载到Amazon Redshift.
Technologies: Azure数据工厂，数据工程，数据管道，SQL, Business Intelligence (BI), ETL Tools, Scripting Languages, APIs, Data Wrangling, Amazon S3 (AWS S3), AWS Lambda, Spark, AWS Glue, Amazon EC2, Amazon Elastic MapReduce (EMR), Amazon RDS, Redshift, SQL Stored Procedures, Amazon Aurora, Apache Airflow, Data Analysis, Data Analytics, Amazon CloudWatch, Amazon QuickSight, AWS Data Pipeline Service, PostgreSQL 10, Azure SQL数据仓库(SQL DW), PostgreSQL, Database Optimization, Database Architecture, XML, CI/CD Pipelines, GitHub, Excel 2016, Tableau, Data Build Tool (dbt), NoSQL, Webhooks, BI Reporting, Database Migration, CDC, Data-driven Dashboards, DAX, Microsoft Power BI, Business Services, Apache Spark, Database Design, Database Structure, Database Transactions, Transactions, MySQL, Microsoft Excel, Real Estate, Geospatial Data, OLTP, OLAP, DevOps, Data

AWS Data Engineer

2020 - 2022
IBM
  • 使用AWS服务(如S3、Glue和RedShift)设计和实现数据管道.
  • 使用Python和SQL开发和维护数据处理和转换脚本. 使用AWS数据库服务(如RDS和DynamoDB)优化数据存储和检索.
  • 使用AWS Redshift和Athena构建和维护数据仓库和数据湖.
  • 使用AWS IAM和KMS实现数据安全和访问控制. 使用AWS CloudWatch和其他监控工具对数据管道和系统进行监控和故障排除.
  • 与数据科学家和分析师合作，提供数据见解并支持他们的数据需求.
  • 使用AWS Lambda和其他无服务器技术进行自动化数据处理和部署.
  • 使用AWS Step Functions和其他工作流工具开发和维护ETL工作流. 了解最新的AWS数据服务和技术，并推荐新的解决方案来改进数据工程流程.
Technologies: 亚马逊CloudWatch、亚马逊RDS、亚马逊S3 (AWS S3)、亚马逊EC2, Amazon Web Services (AWS), AWS Glue, AWS IAM, Redshift, Amazon DynamoDB, Python, SQL, PostgreSQL 10, PostgreSQL, Database Optimization, Lambda Functions, Database Architecture, Elasticsearch, AWS Cloud Architecture, XML, CI/CD Pipelines, GitHub, Excel 2016, Tableau, NoSQL, Webhooks, BI Reporting, CDC, Business Services, Apache Spark, Database Design, Database Structure, Database Transactions, Transactions, Microsoft Excel, OLTP, OLAP, DevOps, Identity & Access Management (IAM), Data

Azure数据工程师和数据仓库顾问

2018 - 2020
IBM
  • 使用ADF设计和开发数据摄取管道，使用Databricks和notebook使用PySpark设计和开发处理层. 领导数据管道的规划、开发、测试、实现、文档和支持.
  • 实施项目的各个方面, 包括使用ADF暂停和恢复Azure SQL数据仓库, ADF管道将业务规则用例作为CSV的可重用资产, fixed width, and excel files.
  • 与客户和IBM ETL团队合作, 分析了本地基于informatica的ETL解决方案, 并使用Azure数据工厂管道和Azure Databricks PySpark和Spark SQL设计了ETL解决方案.
  • 与技术和产品利益相关者合作，了解面向数据的项目需求，并帮助实现解决方案的Azure基础架构组件，作为解决方案的一部分，以创建CPD应用程序的第一个可用迭代.
  • 使用PySpark和Spark SQL为各种复杂的数据转换需求编排和自动化管道poc.
  • 使用PowerShell脚本实现管道自动化，使用Azure Data Factory和Azure Databricks对管道进行性能调优.
Technologies: Autosys, Azure数据工厂，Azure数据库，Azure SQL, Azure SQL数据库, Azure Synapse, Data Engineering, SQL, Data Pipelines, JSON, ETL, T-SQL (Transact-SQL), Python, Pipelines, Data Management, Azure, Dimensional Modeling, Data Lakes, Data Architecture, Microsoft SQL Server, Migration, Query Composition, Performance Tuning, Data Warehouse Design, Data Warehousing, Databricks, Relational Databases, Databases, Analytics, Azure Data Explorer, Consulting, Python 3, CSV File Processing, XLSX File Processing, CSV, Postman, Business Intelligence (BI), ETL Tools, Data Migration, Scripting Languages, Orchestration, Machine Learning, APIs, Technical Project Management, Kanban, ETL Development, Data Wrangling, Amazon S3 (AWS S3), Big Data, AWS Lambda, Spark, AWS Glue, Data Transformation, Amazon EC2, Amazon Elastic MapReduce (EMR), Amazon RDS, Redshift, SQL Stored Procedures, Normalization, Scala, Shell Scripting, Architecture, Data Integration, Google Cloud Platform (GCP), Amazon Aurora, Apache Airflow, Data Analysis, Data Analytics, Pandas, Amazon Web Services (AWS), AWS IAM, Amazon CloudWatch, Amazon DynamoDB, PostgreSQL 10, Azure SQL数据仓库(SQL DW), PostgreSQL, Database Optimization, Database Architecture, XML, CI/CD Pipelines, GitHub, Excel 2016, Tableau, Data Build Tool (dbt), NoSQL, Database Migration, Data-driven Dashboards, DAX, Microsoft Power BI, Business Services, Apache Spark, Database Design, Database Structure, Database Transactions, Transactions, MySQL, Microsoft Excel, OLTP, OLAP, Data

高级ETL顾问和团队领导

2009 - 2018
IBM
  • 在高要求的环境中开发解决方案，并为其他团队成员提供实际指导. 负责复杂的ETL需求，设计和评估需求的完整性和准确性.
  • 实现了基于informatica的ETL解决方案，满足严格的性能要求. 与产品开发团队和高级设计师合作开发架构需求，以确保客户对产品满意.
  • 确定需求对ETL团队是否可行，并进行影响评估，以根据需求确定工作量的大小.
  • 开发了整个软件开发生命周期(SDLC)项目计划，以实现ETL解决方案并确定资源需求.
  • 协助并验证所有设计阶段可交付成果的解决方案设计和生产. 管理构建阶段和质量保证代码，以满足需求并遵守ETL体系结构. 解决困难的设计和开发问题.
  • 为团队提供项目目标的远景, 确保讨论和决定导致结束, 保持健康的团队动力.
  • 使团队熟悉客户需求, specifications, design targets, development process, design standards, techniques, 以及支持任务执行的工具.
  • 在塑造和完善ETL Informatica整体架构方面发挥了积极的主导作用. 识别、推荐和实现ETL过程和体系结构改进.
Technologies: Informatica ETL, Netezza, Autosys, Unix Shell Scripting, IBM Db2, Data Engineering, SQL, Data Pipelines, JSON, ETL, Pipelines, Data Management, Informatica, Informatica Cloud, Data Modeling, Dimensional Modeling, PL/SQL, Data Architecture, Query Optimization, Query Composition, Performance Tuning, Data Warehousing, Relational Databases, Databases, Analytics, Consulting, XLSX File Processing, CSV, Business Intelligence (BI), ETL Tools, Scripting Languages, Orchestration, Technical Project Management, Kanban, ETL Development, Data Wrangling, SQL Stored Procedures, Normalization, Shell Scripting, Architecture, Data Analysis, Data Analytics, Excel Macros, Pandas, Amazon Web Services (AWS), AWS IAM, Amazon CloudWatch, Amazon QuickSight, AWS Data Pipeline Service, Database Optimization, Database Architecture, Oracle PL/SQL, PL/SQL Tuning, CI/CD Pipelines, Excel 2016, Database Administration (DBA), Database Structure, Database Transactions, Transactions, MySQL, Microsoft Excel, OLTP, OLAP, Data

Senior ETL Developer

2008 - 2009
Genesys
  • 为类型二维开发映射，用于更新已有的行和在目标中插入新行. 负责执行与不同流程相关的格式化报告.
  • 创建和开发动态报告，如向上钻取和向下钻取、串联和并行. 分析生成、失败、等待和计划的报告数量.
  • Built dashboards for generated, failed, waiting, 还有关于一刻钟的报告, hour, day, month, and year.
Technologies: Informatica ETL, Unix Shell Scripting, Control-M, Data Engineering, SQL, Data Pipelines, JSON, ETL, Pipelines, Data Management, Informatica, PL/SQL, Data Architecture, Query Optimization, Query Composition, Performance Tuning, Data Warehouse Design, Data Warehousing, Relational Databases, Databases, CSV, ETL Tools, Orchestration, Kanban, ETL Development, Data Wrangling, SQL Stored Procedures, Shell Scripting, Data Integration, Excel Macros, Database Optimization, Oracle PL/SQL, PL/SQL Tuning, Excel 2016, Database Transactions, Microsoft Excel, OLTP, OLAP, Data

Senior ETL Developer

2007 - 2008
Magna Infotech Ltd
  • 管理ETL开发和数据仓库应用程序支持活动.
  • 获得了从维度建模到ETL设计的实践经验.
  • 开发了用于更新现有行和在目标中插入新行的类型二维映射.
Technologies: Informatica ETL, Unix Shell Scripting, Oracle, Data Engineering, SQL, Data Pipelines, ETL, Pipelines, Data Management, Informatica, Dimensional Modeling, PL/SQL, Data Architecture, Query Composition, Performance Tuning, Data Warehouse Design, Data Warehousing, Relational Databases, Databases, ETL Tools, ETL Development, SQL Stored Procedures, Excel Macros, Oracle PL/SQL, PL/SQL Tuning

Tool Client Rate (TCR) Desk

TCR Desk是一个基于网络的工具，为大中型企业客户提供权威的现金管理定价安排和联系信息. The business contact center, relationship managers, 以及现金管理销售人员使用该应用程序.

TCR Desk应用程序迁移解决方案利用Azure良好架构框架的最佳实践，并遵守客户端的Azure服务治理规则，以确保解决方案的安全性, resilient, highly available, and scalable. 这些设计原则适用于在客户端的Azure生产环境中实现. 同样的设计将在灾难恢复和没有高可用性和灾难恢复的低级环境中实现.

Contribution
•使用ADF设计和开发数据摄取管道，并使用PySpark使用Databricks和notebook设计和开发处理层.
• Led the planning, design, development, testing, implementation, documentation, and support of data pipelines.
•与ETL团队合作，包括客户和IBM.
•分析本地基于informatica的ETL解决方案，并使用Azure Data Factory管道设计ETL解决方案, Azure Databricks, PySpark, and Spark SQL.

客户盈利能力洞察(CPI)

商业银行客户盈利能力(BBCP)项目旨在为商业银行开发一个新的盈利能力分析平台，并将其应用范围从500万美元以上的信贷部门扩展到所有客户信贷部门.

Contribution
•在高要求的环境中开发解决方案，并为其他团队成员提供实际指导.
负责复杂的ETL需求和设计.
•实施基于informatica的ETL解决方案，满足严格的性能要求.
•与产品开发团队和高级设计师合作，制定架构要求，以确保客户对产品满意.
•评估要求的完整性和准确性.
确定ETL团队的需求是否可行.
•进行影响评估，并根据需求确定工作量的大小.
制定完整的SDLC项目计划，实施ETL解决方案并确定资源需求.
•在塑造和完善ETL Informatica整体架构方面发挥了积极的主导作用.

Achmea Solvency II

该项目旨在建立一套经修订的欧盟范围内的资本要求和风险管理标准，以取代目前的偿付能力要求. It consists of four releases.

《欧博体育app下载》规定，保险公司的所有重大风险都需要更加透明，以便能够计算出在不可预见的情况下需要保留多少资本作为保险. 在这些要求和法规的推动下，Achmea启动了价值管理计划.

一个重要的程序结果是通过集成的精算数据仓库实现了自动报告功能.
• Release-1: Life 400 insurance
• Release-2: Non-life insurance
• Release-3: ALI/AMIS
• Release-4: VITALIS

Contribution
•与建模师进行实际知识转移会议.
•领导设计各层的技术设计会议.
•分析功能设计文档，并为各个层准备分析表.
•广泛从事技术设计生成文档集的工作，并根据当前版本进行修改.

数据分析师- Azure数据工厂专业知识

我是一名高级数据工程师，拥有分析技能，从事ETL架构解决方案的工作, Requirements assessments, 设计合适的数据流或数据批. 此外，我还执行了解决方案优化和端到端数据管道的数据完整性.

Languages

SQL, Python, T-SQL (Transact-SQL)， Python 3, Snowflake, XML, C, c++， Pascal, R, Scala

Frameworks

Apache Spark, Spark

Tools

Informatica ETL, Autosys, Tableau, Postman, AWS Glue, Amazon Elastic MapReduce (EMR), Apache Airflow, AWS IAM, Amazon CloudWatch, Amazon QuickSight, GitHub, Excel 2016, Microsoft Power BI, Microsoft Excel, Control-M, Google Analytics, Power Query

Paradigms

ETL, Dimensional Modeling, Business Intelligence (BI), OLAP, Kanban, Database Design, DevOps, Data Science

Platforms

Oracle, Azure, Amazon Web Services (AWS), Databricks, Amazon EC2, AWS Lambda, Google Cloud Platform (GCP)

Storage

Netezza, IBM Db2, 数据库管理系统(DBMS), Data Pipelines, Relational Databases, Databases, PostgreSQL, SQL Stored Procedures, Data Integration, Database Architecture, Oracle PL/SQL, NoSQL, Database Transactions, MySQL, Azure SQL Databases, Azure SQL, JSON, Data Lakes, PL/SQL, Microsoft SQL Server, Redshift, Amazon Aurora, AWS Data Pipeline Service, PostgreSQL 10, Amazon DynamoDB, Database Administration (DBA), Database Migration, Database Structure, OLTP, Amazon S3 (AWS S3), Datadog, Elasticsearch

Other

Unix Shell Scripting, Informatica, Data Engineering, Pipelines, Data Management, Data Architecture, Migration, Query Composition, Data Warehouse Design, Data Warehousing, CSV File Processing, CSV, ETL Tools, Scripting Languages, Orchestration, Technical Project Management, ETL Development, Data Transformation, Normalization, Shell Scripting, Architecture, Data Analysis, Data Analytics, Database Optimization, PL/SQL Tuning, Data Build Tool (dbt), DAX, Transactions, Data, Azure Data Factory, Azure Databricks, Azure Data Lake, Azure Synapse, Azure SQL数据仓库(SQL DW), Informatica Cloud, Data Modeling, Query Optimization, Performance Tuning, Analytics, XLSX File Processing, Data Migration, APIs, Data Wrangling, Amazon RDS, Excel Macros, Lambda Functions, Big Data Architecture, AWS Cloud Architecture, CI/CD Pipelines, Webhooks, BI Reporting, CDC, Data-driven Dashboards, Business Services, Identity & Access Management (IAM), Azure Data Explorer, Consulting, Machine Learning, Google Analytics 4, Big Data, Data Visualization, Microsoft Power Automate, Real Estate, Geospatial Data, AWS Certified Cloud Practitioner, Microsoft Azure

Libraries/APIs

Pandas

1998 - 2002

技术或电气工程学士学位

贾瓦哈拉尔尼赫鲁理工大学-海德拉巴，印度

JUNE 2023 - JUNE 2026

AWS Certified Cloud Practitioner

AWS

DECEMBER 2021 - DECEMBER 2022

Azure Data Engineer

Microsoft

AUGUST 2021 - PRESENT

Microsoft Azure Fundamentals

Azure