#data-engineering

[ follow ]
#generative-ai
Data science
fromInfoWorld
1 day ago

Snowflake launches Openflow to tackle AI-era data ingestion challenges

Openflow simplifies data ingestion, transformation, and observability for enterprises engaging with AI use cases.
more#generative-ai
#analytics
Business intelligence
fromMedium
4 months ago

Serving Data in the Data Engineering Lifecycle: A Comprehensive Guide

Data engineering culminates in serving data for analytics, ML, and operations.
Data quality and trust are critical in serving data effectively.
Business intelligence
fromMedium
4 months ago

Serving Data in the Data Engineering Lifecycle: A Comprehensive Guide

Data engineering culminates in serving data for analytics, ML, and operations.
Data quality and trust are critical in serving data effectively.
more#analytics
#e-commerce
fromHackernoon
1 month ago
Data science

Rajesh Sura: Revolutionizing Global Selection Strategy with Data, AI, and Automation | HackerNoon

fromMedium
2 weeks ago
Data science

Day 3-Revenue Aggregation per Region and Category | Spark Interview Problem.

fromHackernoon
1 month ago
Data science

Rajesh Sura: Revolutionizing Global Selection Strategy with Data, AI, and Automation | HackerNoon

fromMedium
2 weeks ago
Data science

Day 3-Revenue Aggregation per Region and Category | Spark Interview Problem.

more#e-commerce
#apache-spark
Scala
fromMedium
3 months ago

Scala Vs. Python-What Data Engineers Need To Know

Scala improves upon Java while remaining JVM-compatible, making it attractive for organizations.
fromawstip.com
1 month ago
Data science

Spark Scala Exercise 5: Column Operations with DataFramesA Complete Guide for Data Engineers

fromMedium
4 weeks ago
Data science

Understanding the load() Function in Apache Spark: Syntax, Examples, and Best Practices

fromMedium
6 days ago
Data science

Day 6-Sessionization of Web Logs using Time Difference | Apache Spark Interview Problem.

Scala
fromMedium
3 months ago

Scala Vs. Python-What Data Engineers Need To Know

Scala improves upon Java while remaining JVM-compatible, making it attractive for organizations.
fromawstip.com
1 month ago
Data science

Spark Scala Exercise 5: Column Operations with DataFramesA Complete Guide for Data Engineers

fromMedium
4 weeks ago
Data science

Understanding the load() Function in Apache Spark: Syntax, Examples, and Best Practices

fromMedium
6 days ago
Data science

Day 6-Sessionization of Web Logs using Time Difference | Apache Spark Interview Problem.

more#apache-spark
#scalability
Data science
fromMedium
4 months ago

Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting Impact

Good data architecture is essential for effective data engineering and organizational competitiveness.
Data science
fromMedium
4 months ago

Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting Impact

Good data architecture is essential for effective data engineering and organizational competitiveness.
more#scalability
#data-quality
Data science
fromMedium
4 months ago

Understanding Data Generation in Source Systems: How It Works and Real-Time Applications

Data generation is crucial in data engineering lifecycle for reliable processing and transformation.
Data science
fromMedium
4 months ago

Understanding Data Generation in Source Systems: How It Works and Real-Time Applications

Data generation is crucial in data engineering lifecycle for reliable processing and transformation.
more#data-quality
fromHackernoon
2 years ago

Building a Real-Time Change Data Capture Pipeline with Debezium, Kafka, and PostgreSQL | HackerNoon

The article provides a step-by-step guide to setting up a Change Data Capture (CDC) pipeline using PostgreSQL, Debezium, Apache Kafka, and Python.
Data science
Data science
fromHackernoon
5 months ago

LLMs in Data Engineering: Not Just Hype, Here's What's Real | HackerNoon

Large Language Models are transforming data engineering by enhancing performance and operational efficiencies.
fromMedium
1 month ago

Evolvability-It's Mostly About Data Contracts

In analytic systems, tight coupling has been the norm, leading to complex structures that are difficult to adapt or change without significant risk.
DevOps
fromHackernoon
1 month ago

Tired of Copy-Pasting Hive Output? This PySpark Hack Fixes It | HackerNoon

One of the many usecases is in my unit test cases, I have to use this csv file with meaningful data from any database or spark job log.
Data science
Women in technology
fromBusiness Insider
1 month ago

I became a director at Ford after pivoting careers in the last recession. Here are 3 ways to recession-proof your job.

Continuous learning through online courses is key to job security in recessionary times.
Data science
fromChannelPro
1 month ago

Datatonic expands global services with Syntio acquisition

Datatonic expands its services by acquiring data engineering firm Syntio, enhancing global reach and expertise in AI solutions.
#spark
Data science
fromMedium
2 months ago

100 Days of Data Engineering on Databricks Day 44: PySpark vs. Scala:

The choice between PySpark and Scala significantly affects performance and maintainability in Spark development.
fromawstip.com
1 month ago
Data science

Spark Scala Exercise 23: Working with Delta Lake in Spark ScalaACID, Time Travel, and Upserts

frommedium.com
1 month ago
Data science

Spark Scala Exercise 10: Handling Nulls and Data CleaningFrom Raw Data to Analytics-Ready

Data science
fromMedium
2 months ago

100 Days of Data Engineering on Databricks Day 44: PySpark vs. Scala:

The choice between PySpark and Scala significantly affects performance and maintainability in Spark development.
fromawstip.com
1 month ago
Data science

Spark Scala Exercise 23: Working with Delta Lake in Spark ScalaACID, Time Travel, and Upserts

frommedium.com
1 month ago
Data science

Spark Scala Exercise 10: Handling Nulls and Data CleaningFrom Raw Data to Analytics-Ready

more#spark
frommedium.com
1 month ago

Spark Scala Exercise 9: Joining Two Datasets in SparkMastering Inner, Left, Right, and Outer

Join operations are foundational for data analysis, helping to connect disparate datasets in Spark Scala, which is critical for enhancing business logic and data insights.
Data science
Artificial intelligence
fromMedium
2 months ago

These AI & Data Engineering Sessions Are a Must-Attend at ODSC East 2025

Organizations are focusing on efficiently and securely integrating advanced AI models at scale.
Practical strategies and real-world insights are essential for navigating AI and data engineering challenges.
#data-management
Information security
fromMedium
4 months ago

The Future of Data Engineering: Security, Privacy, and the Path Ahead

Security and privacy are essential to data engineering, integral to ethics and resilience amid evolving challenges.
fromHackernoon
4 years ago
Data science

The Two Types of Data Engineers You Meet at Work | HackerNoon

Data engineers are categorized into two archetypes: business-oriented and tech-oriented, each with distinct roles and responsibilities.
Information security
fromMedium
4 months ago

The Future of Data Engineering: Security, Privacy, and the Path Ahead

Security and privacy are essential to data engineering, integral to ethics and resilience amid evolving challenges.
Data science
fromHackernoon
4 years ago

The Two Types of Data Engineers You Meet at Work | HackerNoon

Data engineers are categorized into two archetypes: business-oriented and tech-oriented, each with distinct roles and responsibilities.
more#data-management
#ai
fromMedium
7 months ago
Artificial intelligence

Networking, Hackathons, Meetups, and Other Extra Events Coming to ODSC West 2024

fromMedium
7 months ago
Artificial intelligence

Networking, Hackathons, Meetups, and Other Extra Events Coming to ODSC West 2024

more#ai
Data science
fromMedium
4 months ago

Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting Impact

Good data architecture is vital for effective data engineering and organizational competitiveness.
Data science
fromTechzine Global
7 months ago

With Databricks Apps, business users get more out of data

Databricks Apps enhance data accessibility for business users, enabling quicker insights without extensive engineering work.
fromMedium
9 months ago

The Importance of Data Structures and Algorithms in the Life of a Data Engineer

In the ever-evolving world of data engineering, mastering Data Structures and Algorithms is essential for optimizing data processing tasks and enhancing system design.
Data science
[ Load more ]