Data Engineer

Job Duties:

• Create multiple linked services for multiple source systems like SAP HANA, Blob Storage, Sharepoint, ADLS and Flat file like excel, CSV
• Implement the data migration framework with appropriate data load process and sequence (one-time and incremental loads) using Azure Data Factory, Azure blob storage and orchestrate data from on-premises systems using Synapse Azure data factory by creating Azure Data Pipelines and Data flows
• Implement robust connectivity to an OData source using an API. Create a metadata framework for project to execute the pipelines dynamically using Azure Data Factory and load date in to Snowflake
• Create Synapse serverless views on Synapse on demand server and stored procedures in Synapse dedicated server to bring data to datawarehouse
• Prepare data mapping sheets to create a dataset for Power BI dashboards. Implement data transformations such as Synapse mapping dataflows and Synapse notebooks on databricks, workspace (Pyspark+SQL)
• Work on Databricks notebooks to run Spark-Python notebooks through ADF pipelines. Develop PySpark code for data cleansing like trimming of columns and duplicate checks, key duplicates, etc.
• Implement email mechanism and workloads that can be automated using Azure logic Apps
• Impement data extraction, transformation, and aggregation from multiple file formats for analyzing and transforming the data to uncover insights into the customer usage patterns using PySpark and Spark SQL
• Design, develop and optimizing data pipelines and solutions using SQL Server. Create tables dynamically and adding columns in delta tables using PySpark
• Configure secret scope in databricks cluster using Azure Key Vault. Work with stakeholders to recommend and design Azure SQL database and Azure Data Lake storage solutions
• Migrate Tableau reports to Power BI. Use Python modules like Pandas and Numpy and date time to perform extensive data analysis. Prepare lineage and catalog documents for all the application in Azure environment
• Tools and technologies: Azure Data Factory, Azure Blob Storage, Azure SQL DB, Snowflake, Informatica, Azure Synapse, Azure Data Bricks, ADLS GEN2, PySpark, Delta Lake, Blob Storage, Power BI

Location: Anywhere in USA

Please submit your resume and cover letter to: hr@praisetechsol.com