Changelog#

1.6.9 (core) / 0.22.8 (libraries)#

New#

• [ui] When viewing logs for a run, the date for a single log row is now shown in the tooltip on the timestamp. This helps when viewing a run that takes place over more than one date.
• Added suggestions to the error message when selecting asset keys that do not exist as an upstream asset or in an AssetSelection.
• Improved error messages when trying to materialize a subset of a multi-asset which cannot be subset.
• [dagster-snowflake] dagster-snowflake now requires snowflake-connector-python>=3.4.0
• [embedded-elt] @sling_assets accepts an optional name parameter for the underlying op
• [dagster-openai] dagster-openai library is now available.
• [dagster-dbt] Added a new setting on DagsterDbtTranslatorSettings called enable_duplicate_source_asset_keys that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!
• Log messages in the Dagster daemon for unloadable sensors and schedules have been removed.
• [ui] Search now uses a cache that persists across pageloads which should greatly improve search performance for very large orgs.
• [ui] groups/code locations in the asset graph’s sidebar are now sorted alphabetically.

Bugfixes#

• Fixed issue where the input/output schemas of configurable IOManagers could be ignored when providing explicit input / output run config.
• Fixed an issue where enum values could not properly have a default value set in a ConfigurableResource.
• Fixed an issue where graph-backed assets would sometimes lose user-provided descriptions due to a bug in internal copying.
• [auto-materialize] Fixed an issue introduced in 1.6.7 where updates to ExternalAssets would be ignored when using AutoMaterializePolicies which depended on parent updates.
• [asset checks] Fixed a bug with asset checks in step launchers.
• [embedded-elt] Fix a bug when creating a SlingConnectionResource where a blank keyword argument would be emitted as an environment variable
• [dagster-dbt] Fixed a bug where emitting events from dbt source freshness would cause an error.
• [ui] Fixed a bug where using the “Terminate all runs” button with filters selected would not apply the filters to the action.
• [ui] Fixed an issue where typing a search query into the search box before the search data was fetched would yield “No results” even after the data was fetched.

Community Contributions#

• [docs] fixed typo in embedded-elt.mdx (thanks @cameronmartin)!
• [dagster-databricks] log the url for the run of a databricks job (thanks @smats0n)!
• Fix missing partition property (thanks christeefy)!
• Add op_tags to @observable_source_asset decorator (thanks @maxfirman)!
• [docs] typo in MultiPartitionMapping docs (thanks @dschafer)
• Allow github actions to checkout branch from forked repo for docs changes (ci fix) (thanks hainenber)!

Experimental#

• [asset checks] UI performance of asset checks related pages has been improved.
• [dagster-dbt] The class DbtArtifacts has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.

Documentation#

• Added example of writing compute logs to AWS S3 when customizing agent configuration.
• "Hello, Dagster" is now "Dagster Quickstart" with the option to use a Github Codespace to explore Dagster.
• Improved guides and reference to better running multiple isolated agents with separate queues on ECS.

Dagster Cloud#

• Microsoft Teams is now supported for alerts. Documentation
• A send sample alert button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.

1.6.8 (core) / 0.22.8 (libraries)#

Bugfixes#

• [dagster-embedded-elt] Fixed a bug in the SlingConnectionResource that raised an error when connecting to a database.

Experimental#

• [asset checks] graph_multi_assets with check_specs now support subsetting.

1.6.7 (core) / 0.22.7 (libraries)#

New#

• Added a new run_retries.retry_on_op_or_asset_failures setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.
• dagster dev now sets the environment variable DAGSTER_IS_DEV_CLI allowing subprocesses to know that they were launched in a development context.
• [ui] The Asset Checks page has been updated to show more information on the page itself rather than in a dialog.

Bugfixes#

• [ui] Fixed an issue where the UI disallowed creating a dynamic partition if its name contained the “|” pipe character.
• AssetSpec previously dropped the metadata and code_version fields, resulting in them not being attached to the corresponding asset. This has been fixed.

Experimental#

• The new @multi_observable_source_asset decorator enables defining a set of assets that can be observed together with the same function.
• [dagster-embedded-elt] New Asset Decorator @sling_assets and Resource SlingConnectionResource have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling) package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.
• Added support for op-concurrency aware run dequeuing for the QueuedRunCoordinator.

Documentation#

• Fixed reference documentation for isolated agents in ECS.
• Corrected an example in the Airbyte Cloud documentation.
• Added API links to OSS Helm deployment guide.
• Fixed in-line pragmas showing up in the documentation.

Dagster Cloud#

• Alerts now support Microsoft Teams.
• [ECS] Fixed an issue where code locations could be left undeleted.
• [ECS] ECS agents now support setting multiple replicas per code server.
• [Insights] You can now toggle the visibility of a row in the chart by clicking on the dot for the row in the table.
• [Users] Added a new column “Licensed role” that shows the user's most permissive role.

1.6.6 (core) / 0.22.6 (libraries)#

New#

• Dagster officially supports Python 3.12.
• dagster-polars has been added as an integration. Thanks @danielgafni!
• [dagster-dbt] @dbt_assets now supports loading projects with semantic models.
• [dagster-dbt] @dbt_assets now supports loading projects with model versions.
• [dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
• [dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
• [UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.

Bugfixes#

• Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
• Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
• [ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
• [ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
• [dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!

Experimental#

• Observable source assets can now yield ObserveResults with no data_version.
• You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
• [ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.

Documentation#

• Updated docs to reflect newly-added support for Python 3.12.

Dagster Cloud#

• [kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.

1.6.5 (core) / 0.22.5 (libraries)#

New#

• Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
• [dagster-k8s] Include k8s pod debug info in run worker failure messages.
• [dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.

Bugfixes#

• A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
• [dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
• [instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
• [asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
• [dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.

Experimental#

• @observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
• [auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
• [Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.

Documentation#

• Fixed an error in our asset checks docs. Thanks @vaharoni!
• Fixed an error in our Dagster Pipes Kubernetes docs. Thanks @cameronmartin!
• Fixed an issue on the Hello Dagster! guide that prevented it from loading.
• Add specific capabilities of the Airflow integration to the Airflow integration page.
• Re-arranged sections in the I/O manager concept page to make info about using I/O versus resources more prominent.

0.7.5#

New

• Added the IntSource type, which lets integers be set from environment variables in config.

• You may now set tags on pipeline definitions. These will resolve in the following cases:

  1. Loading in the playground view in Dagit will pre-populate the tag container.
  2. Loading partition sets from the preset/config picker will pre-populate the tag container with the union of pipeline tags and partition tags, with partition tags taking precedence.
  3. Executing from the CLI will generate runs with the pipeline tags.
  4. Executing programmatically using the execute_pipeline api will create a run with the union of pipeline tags and RunConfig tags, with RunConfig tags taking precedence.
  5. Scheduled runs (both launched and executed) will have the union of pipeline tags and the schedule tags function, with the schedule tags taking precedence.

• Output materialization configs may now yield multiple Materializations, and the tutorial has been updated to reflect this.

• We now export the SolidExecutionContext in the public API so that users can correctly type hint solid compute functions.

Dagit

• Pipeline run tags are now preserved when resuming/retrying from Dagit.
• Scheduled run stats are now grouped by partition.
• A "preparing" section has been added to the execution viewer. This shows steps that are in progress of starting execution.
• Markers emitted by the underlying execution engines are now visualized in the Dagit execution timeline.

Bugfix

• Resume/retry now works as expected in the presence of solids that yield optional outputs.
• Fixed an issue where dagster-celery workers were failing to start in the presence of config values that were None.
• Fixed an issue with attempting to set threads_per_worker on Dask distributed clusters.

dagster-postgres

• All postgres config may now be set using environment variables in config.

dagster-aws

• The s3_resource now exposes a list_objects_v2 method corresponding to the underlying boto3 API. (Thanks, @basilvetas!)
• Added the redshift_resource to access Redshift databases.

dagster-k8s

• The K8sRunLauncher config now includes the load_kubeconfig and kubeconfig_file options.

Documentation

• Fixes and improvements.

Dependencies

• dagster-airflow no longer pins its werkzeug dependency.

Community

• We've added opt-in telemetry to Dagster so we can collect usage statistics in order to inform development priorities. Telemetry data will motivate projects such as adding features in frequently-used parts of the CLI and adding more examples in the docs in areas where users encounter more errors.

  We will not see or store solid definitions (including generated context) or pipeline definitions (including modes and resources). We will not see or store any data that is processed within solids and pipelines.

  If you'd like to opt in to telemetry, please add the following to $DAGSTER_HOME/dagster.yaml:

  telemetry:
    enabled: true
  

• Thanks to @basilvetas and @hspak for their contributions!

0.7.4#

New

• It is now possible to use Postgres to back schedule storage by configuring dagster_postgres.PostgresScheduleStorage on the instance.
• Added the execute_pipeline_with_mode API to allow executing a pipeline in test with a specific mode without having to specify RunConfig.
• Experimental support for retries in the Celery executor.
• It is now possible to set run-level priorities for backfills run using the Celery executor by passing --celery-base-priority to dagster pipeline backfill.
• Added the @weekly schedule decorator.

Deprecations

• The dagster-ge library has been removed from this release due to drift from the underlying Great Expectations implementation.

dagster-pandas

• PandasColumn now includes an is_optional flag, replacing the previous ColumnExistsConstraint.
• You can now pass the ignore_missing_values flag to PandasColumn in order to apply column constraints only to the non-missing rows in a column.

dagster-k8s

• The Helm chart now includes provision for an Ingress and for multiple Celery queues.

Documentation

• Improvements and fixes.

0.7.3#

New

• It is now possible to configure a Dagit instance to disable executing pipeline runs in a local subprocess.
• Resource initialization, teardown, and associated failure states now emit structured events visible in Dagit. Structured events for pipeline errors and multiprocess execution have been consolidated and rationalized.
• Support Redis queue provider in dagster-k8s Helm chart.
• Support external postgresql in dagster-k8s Helm chart.

Bugfix

• Fixed an issue with inaccurate timings on some resource initializations.
• Fixed an issue that could cause the multiprocess engine to spin forever.
• Fixed an issue with default value resolution when a config value was set using SourceString.
• Fixed an issue when loading logs from a pipeline belonging to a different repository in Dagit.
• Fixed an issue with where the CLI command dagster schedule up would fail in certain scenarios with the SystemCronScheduler.

Pandas

• Column constraints can now be configured to permit NaN values.

Dagstermill

• Removed a spurious dependency on sklearn.

Docs

• Improvements and fixes to docs.
• Restored dagster.readthedocs.io.

Experimental

• An initial implementation of solid retries, throwing a RetryRequested exception, was added. This API is experimental and likely to change.

Other

• Renamed property runtime_type to dagster_type in definitions. The following are deprecated and will be removed in a future version.
  • InputDefinition.runtime_type is deprecated. Use InputDefinition.dagster_type instead.
  • OutputDefinition.runtime_type is deprecated. Use OutputDefinition.dagster_type instead.
  • CompositeSolidDefinition.all_runtime_types is deprecated. Use CompositeSolidDefinition.all_dagster_types instead.
  • SolidDefinition.all_runtime_types is deprecated. Use SolidDefinition.all_dagster_types instead.
  • PipelineDefinition.has_runtime_type is deprecated. Use PipelineDefinition.has_dagster_type instead.
  • PipelineDefinition.runtime_type_named is deprecated. Use PipelineDefinition.dagster_type_named instead.
  • PipelineDefinition.all_runtime_types is deprecated. Use PipelineDefinition.all_dagster_types instead.

0.7.2#

Docs

• New docs site at docs.dagster.io.
• dagster.readthedocs.io is currently stale due to availability issues.

New

• Improvements to S3 Resource. (Thanks @dwallace0723!)
• Better error messages in Dagit.
• Better font/styling support in Dagit.
• Changed OutputDefinition to take is_required rather than is_optional argument. This is to remain consistent with changes to Field in 0.7.1 and to avoid confusion with python's typing and dagster's definition of Optional, which indicates None-ability, rather than existence. is_optional is deprecated and will be removed in a future version.
• Added support for Flower in dagster-k8s.
• Added support for environment variable config in dagster-snowflake.

Bugfixes

• Improved performance in Dagit waterfall view.
• Fixed bug when executing solids downstream of a skipped solid.
• Improved navigation experience for pipelines in Dagit.
• Fixed for the dagster-aws CLI tool.
• Fixed issue starting Dagit without DAGSTER_HOME set on windows.
• Fixed pipeline subset execution in partition-based schedules.

0.7.1#

Dagit

• Dagit now looks up an available port on which to run when the default port is not available. (Thanks @rparrapy!)

dagster_pandas

• Hydration and materialization are now configurable on dagster_pandas dataframes.

dagster_aws

• The s3_resource no longer uses an unsigned session by default.

Bugfixes

• Type check messages are now displayed in Dagit.
• Failure metadata is now surfaced in Dagit.
• Dagit now correctly displays the execution time of steps that error.
• Error messages now appear correctly in console logging.
• GCS storage is now more robust to transient failures.
• Fixed an issue where some event logs could be duplicated in Dagit.
• Fixed an issue when reading config from an environment variable that wasn't set.
• Fixed an issue when loading a repository or pipeline from a file target on Windows.
• Fixed an issue where deleted runs could cause the scheduler page to crash in Dagit.

Documentation

• Expanded and improved docs and error messages.