Jagdish is available for hire

Jagdish Wagh

Verified Expert in Engineering

Database Developer

Location

Bayreuth, Bavaria, Germany

Toptal Member Since

July 11, 2022

Jagdish是一位在不同领域工作过的数据工程师, including retail, insurance, and manufacturing. 他在实现ETL管道和使用数据库和Kimball原理创建数据模型方面拥有丰富的经验. Jagdish曾在AWS和Azure云中工作，并使用AWS Redshift和Azure Synapse Analytics实现了一个云数据仓库.

Portfolio

medi GmbH & Co. KG

数据库，Azure分析服务，Azure Synapse, SQL管理工作室...

TietoEVRY

数据分析，数据库，数据仓库设计，Azure Synapse，雪花...

TietoEVRY

Informatica ETL, Teradata, Control-M, Amazon EC2, matilion Redshift ETL...

Experience

Data Modeling - 9 years Data Analytics - 9 years ETL Tools - 9 years SQL - 9 years Data Warehousing - 9 years Azure Data Factory - 8 years Python - 3 years Databricks - 3 years

Availability

Part-time

Preferred Environment

Python, Microsoft Power BI, Data Modeling, Azure Data Factory, SQL, MSBI, Terraform, Databricks, ETL Tools, Data Warehouse Design

The most amazing...

...我用Kimball和data Vault 2开发了一个数据仓库和数据集市应用程序.0 principles using on-prem and cloud environments.

Work Experience

Data Engineer

2020 - PRESENT

medi GmbH & Co. KG

使用本地和云技术迁移和开发数据分析和数据仓库，并实现混合架构. 负责不同的用例、库存控制、生产订单和销售管理座舱.
创建了用于监视数据管道执行时间和错误消息的作业审计框架. 由于这个过程，我们能够尽快解决问题，节省了很多时间.
在自动化中使用PowerShell脚本实现了一个Azure分析多维数据集备份和恢复过程. 由于这个过程，我们可以在一分钟内恢复旧的多维数据集状态.
创建了一个人力资源管理数据集市，用于维护整个组织的工作流. Because of this dashboard, the whole management can see sickness percentage, headcounts, and paternity leave, and they were able to make decisions.

Technologies: 数据库，Azure分析服务，Azure Synapse, SQL管理工作室, Terraform, Azure DevOps Services, Azure Data Lake, Azure Data Factory, Microsoft Power BI, Python, Databricks, Azure Automation, ARM, SQL, ETL, Data Governance, Visual Studio Code (VS Code), Data Warehouse Design, Database Architecture, Unix Shell Scripting, Microsoft SQL Server, Windows PowerShell, Jira, Azure Logic Apps, Cloud, Microsoft Report Builder, Cloud Architecture, PySpark, Delta Lake, Azure Data Lake Analytics, Dedicated SQL Pool (formerly SQL DW), Azure SQL Data Warehouse, Azure SQL Databases, Azure Functions, Azure Stream Analytics, Data Analytics, Data Warehousing, Microsoft, T-SQL (Transact-SQL), Azure DevOps, Database Migration, SSAS Tabular, MSBI, Data Engineering, SSRS Reports, Microsoft PowerPoint, Database Security, Database Design, HTTP REST, Pandas, SQL DDL, Spark SQL, REST APIs, Azure Administrator, SQL Server Administration, Visual Studio 2017, PyCharm, MySQL, Linux, SQL Server 2017, JSON, Data Architecture, Analytics, Microsoft Excel, Data Analysis, Data Lakes, ADF, Data Integration, Large Data Sets, Cloud Infrastructure, Microsoft Azure Cloud Server, Microsoft 365, Microsoft Power Apps, SQL Server Analysis Services (SSAS)

Senior Software Engineer

2019 - 2020

TietoEVRY

Created a data model and enterprise data warehouse from scratch using the Kimball approach and handed it over to the team; built the ETL data pipeline using Databricks, Azure Data Factory.
使用PowerShell和Unix shell脚本创建自动化，使用AzCopy实用程序将数据文件从本地数据中心复制到Azure存储.
参与使用matilion和Informatica云ETL工具构建云数据仓库.
在Informatica和Azure Data Factory的帮助下，作为从本地Teradata迁移到Snowflake数据库和Azure Synapse Analytics的项目的一部分，在多个poc上工作.
使用Terraform和ARM模板在现有平台中集成了新的Azure网络服务.

Technologies: 数据分析，数据库，数据仓库设计，Azure Synapse，雪花, Informatica ETL, Informatica PowerCenter, Azure Data Factory, Teradata, Databricks, Azure DevOps Services, Matillion ETL for Redshift, SQL Server 2017, Azure Stream Analytics, SQL, Redshift, Python 3, Data Warehousing, Flask, REST APIs, Azure Administrator, SQL Server Administration, Visual Studio 2017, MySQL, Linux, JSON, Data Engineering, Apache Airflow, Data Architecture, Analytics, Microsoft Excel, Informatica, Data Analysis, Data Lakes, ADF, Data Integration, Large Data Sets, Cloud Infrastructure, Microsoft Azure Cloud Server, Microsoft 365, SQL Server Reporting Services (SSRS), SQL Stored Procedures, APIs

Software Engineer

2016 - 2019

TietoEVRY

与客户一起了解业务需求，并在QlikView和Power BI中将这些业务需求转化为可操作的报告, saving 17 hours of manual work each week.
通过使用事件Kafka和PySpark集成来自8个数据源的1亿条原始记录，设计并实现了实时数据管道，处理半结构化和非结构化数据，并将处理后的数据存储到Teradata中.
通过在数据库上创建索引和更改业务逻辑，分析了ETL管道所花费的时间和改进的性能.
使用Azure数据工厂完成一个从AWS云到Azure的迁移项目, data migration, Databricks, Azure storage, Container, and Event Hub.

Technologies: Informatica ETL, Teradata, Control-M, Amazon EC2, matilion Redshift ETL, Azure Data Factory, Azure Synapse, Snowflake, ServiceNow, Databricks, Python, Visual Studio Code (VS Code), erwin Data Modeler, Data Warehouse Design, Azure Analysis Services, Data Governance, Unix Shell Scripting, Microsoft SQL Server, Windows PowerShell, Oracle, Redshift, SQL Server Integration Services (SSIS), Jira, Azure Logic Apps, Cloud, Informatica PowerCenter, Informatica Master Data Management (MDM), Kibana, Elasticsearch, Cloud Architecture, PySpark, Azure Data Lake Analytics, Dedicated SQL Pool (formerly SQL DW), Azure SQL Data Warehouse, Azure SQL Databases, Azure Functions, Azure Event Hubs, Azure Stream Analytics, Delta Lake, Data Analytics, Amazon Web Services (AWS), Data Warehousing, Microsoft, T-SQL (Transact-SQL), Azure DevOps, Database Migration, SSAS Tabular, SQL Management Studio, Microsoft Report Builder, MSBI, Data Engineering, SSRS Reports, Microsoft PowerPoint, Database Design, Azure IoT Hub, HTTP REST, Pandas, Toad, Workbench, Azure DevOps Services, Azure Automation, ARM, SQL DDL, Spark SQL, Flask, MySQL, Linux, SQL Server 2017, JSON, Apache Airflow, Data Architecture, Analytics, Microsoft Excel, Informatica, Data Analysis, Data Lakes, Data Integration, Large Data Sets, Cloud Infrastructure, Microsoft 365, SQL Server Reporting Services (SSRS), SQL Stored Procedures, APIs

Software Engineer

2014 - 2016

Trinus

使用shell脚本将SSIS ETL代码迁移到Informatica PowerCenter中.
创建作业监控进程，每日查看所有作业状态, fixed the process, and delivered the data to the end user.
Used Informatica Power Center for ETL extraction, transformation, 并将数据从异构源系统加载到目标数据库中. 使用Informatica web服务转换角色从web服务中提取数据.

Technologies: Informatica ETL, Informatica Data Quality, Unix Shell Scripting, Amazon EC2, Redshift, SQL Server Integration Services (SSIS), SSAS Tabular, SQL, erwin Data Modeler, Data Warehouse Design, Data Governance, Control-M, Microsoft SQL Server, Windows PowerShell, Jira, Informatica PowerCenter, Informatica Master Data Management (MDM), Data Analytics, Amazon Web Services (AWS), Data Warehousing, Microsoft, T-SQL (Transact-SQL), SQL Management Studio, Cloud, MSBI, Data Engineering, Microsoft PowerPoint, Database Design, Pandas, Toad, Workbench, SQL DDL, Linux, Data Architecture, Analytics, Microsoft Excel, Informatica, Data Analysis, Data Integration, APIs

Software Engineer

2013 - 2014

The Digital Group

在从遗留应用程序迁移项目的法律和金融领域工作. 从Oracle Pro*C和SQL加载器到数据仓库，使用Informatica ETL工具和Unix shell脚本.
使用Informatica ETL和Unix shell脚本将应用程序Oracle Pro*C代码转换为管道. 在staging层上创建自动数据质量配置文件规则，以清理数据并在仓库中捕获.
使用表创建报告，数据传递速度很快. 我还捕获了客户的历史信息，以便客户可以看到客户如何更改其信息.

Technologies: Informatica ETL, SQL, Databases, ETL, Unix Shell Scripting, Control-M, Data Warehouse Design, Data Governance, Microsoft SQL Server, Jira, Informatica PowerCenter, Informatica Master Data Management (MDM), Data Analytics, Data Warehousing, Microsoft, T-SQL (Transact-SQL), SQL Management Studio, MSBI, Microsoft PowerPoint, SQL DDL, Informatica, Data Integration

Experience

eCommerce and Manufacturing Data Warehouse

使用Azure云堆栈实现了一个端到端数据仓库项目. The visualization was created in PowerBi.

Outcomes:
• Extracted data from different source systems, including ERP, relational, real-time events, and files, and loaded Datalake, a database staging layer.
•根据客户需求实现各种业务规则，并在报表层将数据交付给客户.
•创建端到端的数据流，直到报表层.
• Built Delta warehouse in Databricks.

Tools and databases:
Datalake, staging, ods, and EDW.

Data Migration to Public Cloud

我一直在从事一个使用Azure服务从本地数据中心迁移到Azure云的项目, database migration, data factory, Datalake, SQL Server database, Azure DevOps, automation, logic app, and PowerBI. 创建端到端迁移路线图并创建云迁移架构.

Education

2008 - 2012

Bachelor's Degree in Information Technology

Mumbai University - Mumbai, India

2007 - 2008

High School Diploma in Science

Amravati University - Amaravati, India

Certifications

APRIL 2021 - PRESENT

Hands On Essentials | Data Warehouse

Snowflake

JULY 2020 - PRESENT

Microsoft Certified | Azure Fundamentals

Microsoft

Skills

Libraries/APIs

PySpark, Pandas, REST APIs

Tools

Jira, SQL Management Studio, Informatica PowerCenter, Microsoft Excel, Terraform, Apache Airflow, Microsoft Power BI, Matillion ETL for Redshift, Informatica ETL, Control-M, Azure DevOps Services, Azure Logic Apps, Azure Automation, Microsoft Report Builder, Informatica Master Data Management (MDM), Kibana, Microsoft PowerPoint, Azure IoT Hub, Toad, Spark SQL, PyCharm, Microsoft Power Apps

Frameworks

ADF, Windows PowerShell, Flask

Paradigms

商业智能，数据库设计，Azure DevOps, ETL, DevOps

Languages

SQL, T-SQL (Transact-SQL)， SQL DDL, Python, Snowflake, Python 3

Platforms

Azure, Microsoft, Databricks, Unix, Azure Synapse, Oracle, Amazon EC2, Azure Event Hubs, Azure Functions, Azure SQL Data Warehouse, Visual Studio 2017, Linux, Dedicated SQL Pool (formerly SQL DW), Amazon Web Services (AWS), Visual Studio Code (VS Code)

Storage

Azure Cloud Services, Microsoft SQL Server, SQL Server Integration Services (SSIS), SSAS Tabular, Teradata, Data Integration, SQL Server Reporting Services (SSRS), SQL Stored Procedures, Database Architecture, Database Migration, Redshift, Azure SQL Databases, Database Security, SQL Server 2017, MySQL, JSON, Data Lakes, SQL Server Analysis Services (SSAS), Databases, Elasticsearch

Other

Data Warehousing, Data Analytics, ETL Tools, Data Modeling, Azure Data Lake, Azure Data Factory, Data Architecture, Data Vaults, Data Warehouse Design, Azure Analysis Services, Analytics, Informatica, Data Analysis, Large Data Sets, Microsoft Azure Cloud Server, APIs, Data Governance, Unix Shell Scripting, ServiceNow, ARM, MSBI, Cloud, Cloud Architecture, Delta Lake, Azure Stream Analytics, Azure Data Lake Analytics, Data Engineering, SSRS Reports, HTTP REST, Workbench, Azure Administrator, SQL Server Administration, Cloud Infrastructure, Microsoft 365, erwin Data Modeler, Informatica Data Quality

Collaboration That Works

How to Work with Toptal

在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.

Share your needs

在与Toptal领域专家的电话中讨论您的需求并细化您的范围.

Choose your talent

在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.

Start your risk-free talent trial

与你选择的人才一起工作，试用最多两周. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring