Data science
fromTNW | Data-Security
23 hours agoWhy data quality matters when working with data at scale
Data quality should be prioritized from the start to prevent costly issues later in data engineering projects.
#apache-nifi
#apache-nifi
[ follow ]
fromInfoWorld
4 weeks agoMigrating from Apache Airflow v2 to v3
Airflow 3 represents a clear architectural direction for the project: API-driven execution, better isolation, data-aware scheduling and a platform designed for modern scale. While Airflow 2.x is still widely used, it is clearly moving toward long-term maintenance (end-of-life April 2026) with most innovation and architectural investment happening in the 3.x line.
Software development
Data science
fromMedium
3 weeks agoBuilding Consistent Data Foundations at Scale
Building consistent data foundations through intentional architecture, engineering, and governance is essential to prevent fragmentation, support AI adoption, ensure regulatory compliance, and enable reliable organizational decisions at scale.
Business intelligence
fromInfoWorld
3 weeks agoSnowflake's new 'autonomous' AI layer aims to do the work, not just answer questions
Project SnowWork is Snowflake's autonomous AI layer that automates data analysis tasks like forecasting, churn analysis, and report generation without requiring data team intervention.
#ai-automation
Artificial intelligence
fromTechzine Global
3 weeks agoSnowflake's Project SnowWork targets autonomous enterprise AI
Snowflake launches Project SnowWork, an autonomous AI interface that performs enterprise tasks like forecasts and reports without data team involvement, expanding from backend infrastructure to front-office productivity tool.
Artificial intelligence
fromTechzine Global
3 weeks agoSnowflake's Project SnowWork targets autonomous enterprise AI
Snowflake launches Project SnowWork, an autonomous AI interface that performs enterprise tasks like forecasts and reports without data team involvement, expanding from backend infrastructure to front-office productivity tool.
Startup companies
fromInfoQ
2 months agoEtleap Launches Iceberg Pipeline Platform to Simplify Enterprise Adoption of Apache Iceberg
Managed Iceberg pipeline platform unifies ingestion, transformation, orchestration, and table operations inside customers' VPCs, enabling enterprise Iceberg adoption without building custom stacks.
fromTechzine Global
2 months agoSumo Logic launches data pipeline apps for Snowflake and Databricks
Snowflake offers a fully managed data platform, but Sumo Logic users often lack insight into performance, login activity, and operational health. The Sumo Logic Snowflake Logs App analyzes login and access activity to identify anomalies or suspicious behavior. It also optimizes data pipelines with insights into long-running or failing queries. Teams can centralize log data to facilitate correlation across applications, cloud services, and data platforms.
Information security
fromMedium
2 months agoAgentic Workflows in Scala (Without the Buzzwords)
High-level view of the travel search workflow, highlighting parallel searches, explicit decision points, and iterative refinement. In Scala, we define this workflow using Workflows4s, encoding both state and transitions explicitly in the type system. Instead of opaque state blobs or untyped contexts, the state of the process is represented using algebraic data types - types like Started, Found, Sent, and Booked - each corresponding to a distinct point in the workflow's lifecycle.
Scala
fromInfoWorld
2 months agoSnowflake updates developer tools, adds observability features
Snowflake adds observability capabilities via Trail The company also added new observability features in the form of Snowflake Trail, which provides visibility into data quality, pipelines, and applications, enabling developers to monitor, troubleshoot, and optimize their workflows. It is built with OpenTelemetry standards so developers can integrate with popular observability and alert platforms including Datadog, Grafana, Metaplane, PagerDuty, and Slack, among others.
DevOps
fromInfoWorld
2 months agoAI-augmented data quality engineering
SHAP for feature attribution SHAP quantifies each feature's contribution to a model prediction, enabling: LIME for local interpretability LIME builds simple local models around a prediction to show how small changes influence outcomes. It answers questions like: "Would correcting age change the anomaly score?" "Would adjusting the ZIP code affect classification?" Explainability makes AI-based data remediation acceptable in regulated industries.
Artificial intelligence
Artificial intelligence
fromInfoQ
2 months agoAutonomous Big Data Optimization: Multi-Agent Reinforcement Learning to Achieve Self-Tuning Apache Spark
A Q-learning agent autonomously learns and generalizes optimal Spark configurations by discretizing dataset features and combining with Adaptive Query Execution for superior performance.
[ Load more ]