[ui] When viewing logs for a run, the date for a single log row is now shown in the tooltip on the timestamp. This helps when viewing a run that takes place over more than one date.
Added suggestions to the error message when selecting asset keys that do not exist as an upstream asset or in an AssetSelection.
Improved error messages when trying to materialize a subset of a multi-asset which cannot be subset.
[dagster-snowflake] dagster-snowflake now requires snowflake-connector-python>=3.4.0
[embedded-elt] @sling_assets accepts an optional name parameter for the underlying op
[dagster-openai] dagster-openai library is now available.
[dagster-dbt] Added a new setting on DagsterDbtTranslatorSettings called enable_duplicate_source_asset_keys that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!
Log messages in the Dagster daemon for unloadable sensors and schedules have been removed.
[ui] Search now uses a cache that persists across pageloads which should greatly improve search performance for very large orgs.
[ui] groups/code locations in the asset graph’s sidebar are now sorted alphabetically.
Fixed issue where the input/output schemas of configurable IOManagers could be ignored when providing explicit input / output run config.
Fixed an issue where enum values could not properly have a default value set in a ConfigurableResource.
Fixed an issue where graph-backed assets would sometimes lose user-provided descriptions due to a bug in internal copying.
[auto-materialize] Fixed an issue introduced in 1.6.7 where updates to ExternalAssets would be ignored when using AutoMaterializePolicies which depended on parent updates.
[asset checks] Fixed a bug with asset checks in step launchers.
[embedded-elt] Fix a bug when creating a SlingConnectionResource where a blank keyword argument would be emitted as an environment variable
[dagster-dbt] Fixed a bug where emitting events from dbt source freshness would cause an error.
[ui] Fixed a bug where using the “Terminate all runs” button with filters selected would not apply the filters to the action.
[ui] Fixed an issue where typing a search query into the search box before the search data was fetched would yield “No results” even after the data was fetched.
[asset checks] UI performance of asset checks related pages has been improved.
[dagster-dbt] The class DbtArtifacts has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.
Microsoft Teams is now supported for alerts. Documentation
A send sample alert button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.
Added a new run_retries.retry_on_op_or_asset_failures setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.
dagster dev now sets the environment variable DAGSTER_IS_DEV_CLI allowing subprocesses to know that they were launched in a development context.
[ui] The Asset Checks page has been updated to show more information on the page itself rather than in a dialog.
[ui] Fixed an issue where the UI disallowed creating a dynamic partition if its name contained the “|” pipe character.
AssetSpec previously dropped the metadata and code_version fields, resulting in them not being attached to the corresponding asset. This has been fixed.
The new @multi_observable_source_asset decorator enables defining a set of assets that can be observed together with the same function.
[dagster-embedded-elt] New Asset Decorator @sling_assets and Resource SlingConnectionResource have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling) package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.
Added support for op-concurrency aware run dequeuing for the QueuedRunCoordinator.
dagster-polars has been added as an integration. Thanks @danielgafni!
[dagster-dbt] @dbt_assets now supports loading projects with semantic models.
[dagster-dbt] @dbt_assets now supports loading projects with model versions.
[dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
[dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
[UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.
Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
[ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
[ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
[dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!
Observable source assets can now yield ObserveResults with no data_version.
You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
[ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.
[kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.
Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
[dagster-k8s] Include k8s pod debug info in run worker failure messages.
[dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.
A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
[dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
[instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
[asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
[dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.
@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
[auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
[Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.
[ui] Enabled collapsing asset groups in the global asset view when the new experimental asset graph is turned on in User Settings
[ui] The experimental asset graph sidebar now supports keyboard navigation via arrow keys
[ui] You can now right click nodes in the asset graph to materialize them or filter the graph
[ui] Jobs can now be searched by run ID
[ui] You can now launch runs from the job actions menu
[auto-materialize] A new AutoMaterializeRule.materialize_on_cron() rule makes it possible to create policies which materialize assets on a regular cadence.
[auto-materialize] If a partition-mapping-related error occurs within the Asset Daemon, a more informative error message will be provided.
[dagster-databricks] Extended the set of available config options to the Databricks step launcher - thanks @zyd14!
Fixed an issue where some schedules incorrectly skipped ticks during Daylight Savings Times transitions.
Returning a SensorResult from a sensor no longer overwrites a cursor if it was set via the context.
Fixed issue which could cause incorrect execution order when executing multi-assets with can_subset=True alongside assets which were upstream of some assets in the multi-asset, and downstream of others.
Previously, when creating an HourlyPartitionsDefinition with a non-UTC timezone and the default format string (or any format string not including a UTC-offset), there was no way to disambiguate between the first and second instance of the repeated hour during a daylight saving time transition. Now, for the one hour per year in which this ambiguity exists, the partition key of the second instance of the hour will have the UTC offset automatically appended to it.
[asset checks] Fixed a bug that caused an error when passing check_specs to AssetsDefinition.from_graph
[dagster-dbt] Fixed a bug in dagster-dbt that caused some dbt tests to not be selected as asset checks.
[dagster-dbt] Fixed an issue where multiple copies of the dbt manifest were held in memory when loading a dbt project as software-defined assets.
The email_on_failure sensor called deprecated methods on the context. This has been fixed
DagsterInstance.report_runless_asset_event is now public.
AutoMaterializeRule.materialize_on_parent_updated now accepts an updated_parents_filter of type AutoMaterializeAssetPartitionsFilter, which allows only materializing based on updates from runs with a required set of tags.
Fixed an error when trying to directly invoke a run status sensor when passing resources.
[dagster-airbyte][dagster-fivetran] Fixed an issue where EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.
Fixed an issue where some schedules skipped a tick on the day after a fall Daylight Savings Time transition.
Fixed a bug that caused backfill policies that execute multiple partitions in a single run not to work with dynamic partitions.
Fixed a bug that caused an error when build_schedule_from_partitioned_job was used with a job with multi-partitioned assets and the partitions_def argument wasn’t provided to define_asset_job.
We now raise an error early if the empty string is provided as an asset’s group name (Thanks Sierrra!)
Fixed an issue where custom setup and teardown methods were not properly called on nested Pythonic resources.
Added a warning message when op or asset config is passed as an argument not named config.
[dagster-cloud] Fixed an issue where overriding the default I/O manager could break the Snowflake-dbt insights job.
[auto-materialize] Fixed an issue where materializing an unpartitioned parent of a dynamic-partitioned asset would only result in the latest dynamic partition of that asset being requested. Now, all partitions will be requested.
[dagster-embedded-elt] Fixed an issue in dagster-embedded-elt where sling’s updated_at parameter was set to the incorrect type
[dagster-ui] Fixed an issue in the launchpad where selecting a partition wouldn’t correctly overwrite fields using the partition’s specific configuration
Improved failure recovery and retry behavior when the daemon that launches auto-materialization runs fails or crashes in the middle of a tick.
[asset-checks] UI performance for displaying check results is improved
[asset-checks] Removed noisy experimental warning about AssetCheckKey
[op-concurrency] Fixed a bug where concurrency slots were not getting assigned if a run that was assigned a slot was deleted before the slot had actually been claimed during execution.
[dagster-pipes] The PipesSubprocessClient now inherits the environment variables of the parent process in the launched subprocess.
[dagster-pipes] Exceptions are now reported in the event log for framework components and from the external process.
[dagster-k8s] The PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.
[ui] The launch asset backfill modal now offers a preview dialog showing the targeted asset partitions and their backfill policies when partition mapping or varying backfill policies are present.
[asset-checks] New load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules.
Previously, the daemon process would terminate with an error if it believed that a thread might be hanging, which sometimes caused undesirable terminations when doing large backfills or auto-materializing many assets. Now, the daemon process will emit a warning instead of terminate.
[dagster-dbt] dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory.
[dagster-dbt] @dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping.
[dagster-dbt] Self dependencies can now be enabled for dbt models that are represented by @dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:
Fixed an issue where Dagster imports would throw errors when using pydantic<2.0.0 but having pydantic-core installed.
Previously, asset backfills that targeted partitioned assets with a LastPartitionMapping dependency would raise an error. This has been fixed.
Fixed a multipartitions partition mapping bug where a instance is not available to load partitions error was raised.
[asset-checks] Fixed an issue with conflicting op names when using build_asset_with_blocking_check
[ui] Viewing run logs containing very large messages no longer causes the UI to crash in Safari on macOS
[ui] Drilling into the graph of a graph-backed asset with 4+ inputs or outputs no longer causes the asset graph to crash with a rendering error.
[ui] On the backfill details page, clicking to a specific asset correctly shows the partitions that were materialized for that asset when partition mapping is in use.
[ui] The Asset > Partition page now loads partition details more quickly in cases where calculating the staleness of the partition took a significant amount of time.
Fixed a bug introduced in 1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion.
[auto-materialize] Previously, if an asset partition was skipped on a tick for one reason, and then processed and skipped on a later tick for an additional reason, only the most recent skip reason would be tracked. Now, all skip reasons are tracked.
[dagster-dbt] Fixed an issue where if an exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled.
[dagster-dbt] When invoking DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.
Dagster now supports using Pydantic 2 models for Config and Resources. Pydantic 1.10 continues to be supported.
Added a report_asset_observation REST API endpoint for runless external asset observation events
Dramatically improved the performance of partition-mapping, for basic hourly and daily partitions definitions
[ui] When viewing a list of runs, you can quickly add the tag in the “Launched by” column as a filter on the list view. Hover over the tag to see the “Add to filter” button.
[helm] The env vars DAGSTER_K8S_PG_PASSWORD_SECRET and DAGSTER_K8S_INSTANCE_CONFIG_MAP will no longer be set in all pods.
[dagster-pyspark] build_pyspark_zip now takes an exclude parameter that can be used to customize the set of excluded files.
[ui] Links beginning with http://, https:// in unstructured run logs (via context.log) are automatically converted to clickable links