About Lishan
Unlocking the power of data with expertise in machine learning, deep learning, natural language processing, and business intelligence. Agile and design thinking methodologies are applied to deliver results for diverse industries including insurance, retail, banking, and tourism. With specialized knowledge in these industries, I offer valuable insights and solutions tailored to their specific needs. Strong communication skills and a passion for travel and learning bring a unique perspective to every project
German
Native or bilingual
English
Fluent
Tamil
Native or bilingual
French
Fluent
Can work on-site
Düsseldorf (up to 50km), Frankfurt am Main (up to 50km), Köln (up to 50km)
Experience
- ConsultingData & AI EngineerCONSULTING AND AUDITSSeptember 2024 - Today (1 year and 10 months)München, Germany• Engineered an enterprise AI chatbot system leveraging Azure OpenAI, PromptFlow, and Azure Cognitive Search to process internal documentation and ticketing data, enhancing support capabilities.• Developed an automated Confluence & ServiceNow content scraping pipeline to extract, process, and index internal documentation, facilitating AI-powered knowledge retrieval.• Implemented secure vector search functionality using Azure AI Search and custom embedding models, optimizing document retrieval processes while ensuring data security compliance.• Built and optimized PromptFlow pipelines for query intent extraction and contextual response generation, establishing a modular flow architecture with real-time monitoring.• Created a responsive and interactive chat interface with Streamlit, applying custom styling to improve user experience and support real-time interactions.• Developed a PII detection system leveraging Azure Language Service and Azure Text Analytics to identify and mask personally identifiable information (PII) before persisting data into the database, ensuring compliance and preventing chatbot or indexer access to sensitive data.• Architected a robust Azure AI Search integration system with configurable datasources (here a connection to MongoDB), indexes, and indexers, implementing high-watermark change detection for efficient document synchronization.• Designed a comprehensive search index schema with optimized field mappings for title, content, labels, and metadata, enabling advanced search capabilities including semantic search and faceted navigation.• Developed & led the end-to-end architecture design for the chatbot system and its components, ensuring integration, scalability, and maintainability, and thoroughly documented the architecture and functionality of each component for knowledge sharing and future development.
- Mercedes-Benz Tech Innovation GmbHData EngineerAUTOMOBILEOctober 2023 - August 2024 (11 months)Ulm, Germany• Developed and extended customized Python Packages for processing vehicle test data (Gigabyte scale) using PySpark and Delta tables to run as a Kubernetes job.• Developed and extended a customized Python Package for scheduling Kubernetes Pyspark Jobs in Azure Kubernetes Services (AKS)• Solved the existing issue with out-of-memory errors in Pyspark Kubernetes jobs with flexible resource allocation in the Kubernetes cluster• Implemented and expanded Azure DevOps CI/CD pipeline for internal Python packages including automated tests and security checks.• Expanded and optimized existing Data Warehouse architecture using Data Vault 2.0• Build and maintained Docker images using Azure CI/CD pipeline in Azure Container Registry, creating and extended Kubernetes Helm Charts• Implemented granularized Trino access policies for an established reporting mart• Deployed, maintained and extended Production and Non-Production environment in Azure using Bicep (IaaC).• Created and maintained Databricks Pipelines and engaging customer support to resolve emerging issues related to this platform.
- COMMERZBANK AGData EngineerBANKING AND INSURANCEMarch 2023 - September 2023 (7 months)Frankfurt, Germany• Developed an IT-Audit dashboard using Qlik Sense, setting up the ETL pipeline to extract the necessary data from internal audit tools and creating the needed Hive tables.• Conceptualized and developed a dashboard for monitoring data quality of different processes and outlier detection and creating the required ETL and the needed datasets.• Developed a Google Looker dashboard for visualizing logging data, extended the existing data ingestion point using FastAPI to write into Google BigQuery and maintained it’s CI/CD pipeline in OpenShift.• Actively communicated and engaged with stakeholders to meet requirements.• Created documentation for onboarding the new team members.
Recommendations
Be the first to recommend Lishan
Help this freelancer shine by sharing your experience working together.
These freelancer profiles also match your criteria
Agatha Frydrych
Backend Java Software Engineer
4.7
(3)
2
Baptiste Duhen
Fullstack developer
4.6
(4)
5
Amed Hamou
Senior Lead Developer
4
(2)
7
Audrey Champion
Web developer
4.3
(3)
4
Education
- Master of Science - MS, Information Systems, Major : Business Analytics, Econometrics & Data ScienceUniversity of Cologne2021Master of Science - MS, Information Systems, Major : Business Analytics, Econometrics & Data Science
- Bachelor of ScienceUniversity of Cologne2021Bachelor of Science - BS, Information Systems
Certifications
- Databricks Certified Data Engineer AssociateDatabricks2024