apache/airflow

AI Summary: The user is experiencing production outages because there's a lack of clear documentation on what and when to clean on their Redis server, which is used with Airflow. This operational ambiguity leads to instability and erodes confidence in Airflow as a stable solution. They propose adding a dedicated 'maintenance' section to the Airflow documentation to detail all necessary Redis cleaning tasks.

AI Summary: This issue requests the addition of an Azure Service Bus Trigger to Airflow, enabling event-driven scheduling via the `MessageQueueTrigger` for `AssetWatcher` components. This new trigger would expand the existing support for message queue-based scheduling, which currently includes SQS and Kafka.

AI Summary: The `GCSToBigQueryOperator` and `BigQueryCreateEmptyTableOperator` in Airflow's Google provider currently only support time-based partitioning for BigQuery tables. This issue requests extending these operators to also support integer range-based partitioning, a feature already available in BigQuery, to provide more flexible table creation options.

AI Summary: The `BigQueryInsertJobOperator` in Airflow 3.0.5 fails when a DAG is triggered by an asset event because `logical_date` is no longer present in the Airflow context for such runs. The operator's `execute` method attempts to access `context["logical_date"]` to auto-generate a `job_id`, leading to an error. The expected behavior is for the operator to generate a `job_id` regardless of the DAG's trigger type.

AI Summary: The issue describes poor error messaging when using Hashicorp Vault as a secrets backend for Airflow. Currently, if Vault authentication fails or a secret/variable is malformed (e.g., missing expected keys like `conn_uri` or `value`), users receive a generic "Variable <> does not exist" message, making debugging difficult. The goal is to improve these error messages to clearly indicate whether a secret is malformed or if there's an authentication problem.

AI Summary: This issue proposes adding support for Podman as an alternative container management option within the Breeze tooling. The motivation stems from Podman's open-source nature, OCI compliance, and perceived benefits in security and performance. The submitter has indicated willingness to provide a pull request for this feature.

AI Summary: This GitHub issue proposes enhancing the 'Search - Task Instances' view by adding comprehensive filtering capabilities. It specifies 11 new filters, including DAG ID, Run ID, date/duration ranges, and operator type, to allow users to more precisely narrow down and investigate task instances. The goal is to improve the usability and analytical power of the task instance search.

AI Summary: The task involves resolving a TODO comment in the `JenkinsJobTriggerOperator` within the Apache Airflow project. This requires understanding the existing code, determining the appropriate implementation for `get_queue_info`, and testing the changes to ensure functionality and maintainability.

AI Summary: The task is to add an `encoding` parameter to the Apache Airflow FTP connection's `extra` field. This parameter should be passed to the `ftplib.FTP` constructor to allow users to specify the encoding used when downloading files from FTP servers, addressing issues with encoding mismatches and `UnicodeDecodeError` exceptions.

AI Summary: Improve the Apache Airflow GitHub README by incorporating 10 suggested changes: creating a more engaging tagline, adding a quick start guide for local setup, including a visual showcase of the UI, providing a primer on key concepts, listing real-world use cases, highlighting key features with benefits, creating a section on where to find help, emphasizing Python-centricity and extensibility, adding a clear versioning and compatibility statement, and including a "Powered By" section with case studies (if available).

AI Summary: The task is to debug why the 'Config' tab is missing from the Apache Airflow 3.x UI when using the FabAuthManager authentication method. This involves investigating potential permission issues or misconfigurations within FabAuthManager's role settings that prevent Admin users from accessing the airflow.config view or menu item. The solution might involve adding necessary permissions or modifying the FabAuthManager configuration.

AI Summary: Update Apache Airflow 3.0 documentation to reflect the use of an AI server instead of a web server in diagrams and explanations related to DAG processing architecture. This involves identifying outdated diagrams and text in the specified documentation link and replacing them with accurate representations of the AI server's role.

AI Summary: The task is to debug a vanishing graph view in the Apache Airflow UI running on the main development branch. The issue occurs on both Firefox and Chromium browsers, with no apparent errors in browser consoles or server logs. The goal is to identify and fix the underlying cause of the graph view disappearing after a short time, ensuring it refreshes correctly instead.

AI Summary: Implement a configurable option in Apache Airflow's React plugin template generator to optionally include AI-powered linting rules. This would simplify project setup for users unfamiliar with React best practices.

AI Summary: The task involves correcting a bug in the `airflow-api.service` systemd script for Apache Airflow 3.0.3. The bug is an incorrect `ExecStart` command, using `airflow api` instead of `airflow api-server`. Additionally, the task requires improving consistency by using `EnvironmentFile` instead of directly specifying environment variables and restoring the `After` clause to ensure proper dependency ordering with services like Redis, RabbitMQ, and PostgreSQL.

AI Summary: The task is to add deprecation warnings to the `airflow.security.permissions` module in Apache Airflow 3.x, guiding users to migrate to the newer `ResourceMethod` and `airflow.api_fastapi.auth.managers.models.resource_details` standards. This involves updating the code to include warnings indicating the new locations for the relevant constants and methods.

AI Summary: The task involves investigating why the `--log-config` argument appears to be missing from the Airflow API server command-line interface, as documented. This requires comparing the documentation against the actual codebase to determine if the documentation is outdated or if there's a bug in the API server implementation. A potential solution would involve updating either the documentation or the code to reflect the correct behavior.

AI Summary: The task is to debug and fix a bug in the Apache Airflow Amazon provider's `eks_get_token` function. The function fails to correctly parse the expiration timestamp from the output of the EKS get-token command, resulting in an empty timestamp and subsequent errors. The solution involves modifying the parsing logic within the `eks_get_token` function to correctly extract the expiration timestamp.

AI Summary: Update the Apache Airflow documentation's TaskFlow API example to use valid attributes, correcting errors related to 'duration' and 'queued_at' attributes no longer existing on the relevant objects. This involves finding suitable replacement attributes to demonstrate the `context` functionality and updating the documentation accordingly.

AI Summary: The task is to enhance Apache Airflow's configuration to allow the use of self-signed SSL certificates without requiring manual certificate verification within the Docker container environment. This involves adding a configuration parameter to disable certificate verification, ideally with a prominent warning, and ensuring the system functions correctly with self-signed certificates without needing to import them into the container.

AI Summary: The task is to fix a bug in Apache Airflow 3.0.3 where sensitive data from extra fields in connections is visible in the UI's edit connection window. The solution should mask sensitive data with ***, respecting the `hide_sensitive_var_conn_fields` and `sensitive_var_conn_names` configurations. This likely involves modifying the Airflow UI code to properly handle and mask sensitive data within the extra fields.

AI Summary: Debug and fix a bug in Apache Airflow 3.0.3 where fetching logs for previous task tries fails when remote logging (S3) is enabled. The issue is reproducible with Amazon provider 9.10.0 and works correctly in 3.0.2. The solution likely involves identifying and correcting a regression introduced between Airflow 3.0.2 and 3.0.3 related to log retrieval from S3.

AI Summary: The issue describes a bug in Apache Airflow 3.0.3 where the BranchSQLOperator fails to skip downstream BranchSQLOperators as expected. The problem seems to stem from a change introduced in 3.0.3 that affects the task skipping logic. A potential fix involves modifying the `_ensure_tasks` function to include `BaseSQLOperator` in the list of node types considered for skipping. The task requires investigating the root cause of the skipping failure in Airflow 3.0.3 and either providing a fix or confirming the proposed solution's effectiveness.

AI Summary: The task is to investigate and fix a blocking issue in the Apache Airflow Amazon provider's `AwsBaseWaiterTrigger`. The trigger blocks the event loop during initialization due to `aiobotocore` loading credentials. The solution likely involves initializing and caching the `aiobotocore` session outside the event loop to prevent blocking.

AI Summary: The task is to update the Apache Airflow documentation to reflect the currently supported Kubernetes versions, ensuring consistency between the documentation and the README file. This involves identifying the discrepancy between the listed Kubernetes versions in the documentation and the README, and then modifying the documentation to match the accurate, up-to-date information found in the README.

AI Summary: The task is to fix a UI bug in Apache Airflow where dropdown menus in trigger forms are truncated when options are long and there's insufficient space below the menu. The solution involves adjusting the UI layout to accommodate longer dropdown menu options, preventing them from being cut off.

AI Summary: The task is to fix a `KeyError` in the Apache Airflow Google BigQuery provider. The error occurs when a BigQuery query returns an empty schema, causing a null key reference. The solution involves adding a check to ensure the 'schema' key exists and has a value before attempting to access its 'fields' key. A pull request is expected.

AI Summary: Add a `labels` section to the dagProcessor deployment in the Apache Airflow Helm chart to allow users to apply labels for better identification and management of the dagProcessor workload.

AI Summary: The issue describes a version incompatibility between the `apache-airflow-providers-sftp` and `apache-airflow-providers-ssh` packages. The `sftp` provider requires a newer version of the `ssh` provider than currently specified in its dependencies. The task is to update the dependency specification in `apache-airflow-providers-sftp`'s `pyproject.toml` file to correctly reflect the required `ssh` provider version (>=4.0.0).

AI Summary: The task involves modifying the Apache Airflow KubernetesPodOperator to provide an option for customizing or disabling the container name prefix in log lines. This requires understanding the existing logging mechanism within the operator's code and implementing a configurable option to control the log output format.

AI Summary: The task involves fixing a bug in Apache Airflow's user interface where the task instance note doesn't refresh correctly after clearing a task instance with a note. The solution requires updating the `TaskInstance/Header.tsx` component to ensure the 'note' state reflects the updated cache value, preventing display of stale data while maintaining editability.

AI Summary: The task requires modifying Apache Airflow's `schedule_tis` and `expand_start_from_trigger` methods to correctly handle templated fields within the `start_from_trigger` feature. This involves rendering templated fields before invoking the `expand_start_from_trigger` method and updating the method to re-assign rendered templated fields to `trigger_kwargs` for proper operator instantiation.

AI Summary: The issue describes a bug in the Apache Airflow Amazon ECS provider where tasks are incorrectly marked as successful even when the underlying ECS task fails to start. The problem stems from the `_check_success_task` method resetting the task ARN to `None` upon encountering a `EcsTaskFailToStart` exception, preventing subsequent retries from launching a new task. A fix requires modifying this method to handle the exception appropriately, ensuring the task is marked as failed after all retries are exhausted.

AI Summary: The task is to fix a bug in Apache Airflow's KubernetesPodOperator where the retry mechanism fails to handle pod evictions. The operator should correctly identify evicted pods and launch new ones instead of attempting to reuse the non-existent evicted pod, preventing 400 Bad Request errors when fetching logs. This involves modifying the KubernetesPodOperator to check the pod's status before attempting to reuse it and ensuring that a new pod is launched upon eviction.

AI Summary: The task involves fixing an invalid interpolation format error in the Apache Airflow 3.0.x `docker-compose.yaml` file. The error occurs because the command line uses `$(id -u)` to export `AIRFLOW_UID`, which Docker Compose interprets incorrectly. The solution is to escape the dollar sign in the command line or use an .env file to predefine `AIRFLOW_UID`. A pull request is expected.

AI Summary: The issue describes a bug in Apache Airflow's S3KeySensor when used in deferrable mode. The `bucket_key` parameter, when a single string, is incorrectly split into individual characters instead of being treated as a single key. The task is to fix this bug by modifying the `get_files_async` function to correctly handle both string and list inputs for `bucket_keys`, ensuring that the sensor functions as expected in both deferrable and non-deferrable modes. This involves debugging the existing code, adding appropriate checks, and writing unit tests to verify the fix.

AI Summary: The task is to debug why Airflow's UI fails to correctly parse and utilize SSH connection extra fields containing a single-line private key with newline characters, while the CLI works correctly. The solution likely involves investigating how the UI handles JSON parsing and the interaction with the SSHOperator, potentially correcting data sanitization or formatting issues within the UI's connection creation process.

AI Summary: The task is to enhance the Apache Airflow Papermill provider to support connecting to remote JupyterHub kernels via HTTPS, incorporating authentication using tokens. This involves modifying the provider's code to handle HTTPS connections and token-based authentication securely.

AI Summary: The task involves modifying Apache Airflow's logging behavior. Specifically, it requires changing the log level for 'Variable not found' errors to 'debug' when a default value is provided in the `Variable.get` function. This aims to reduce log spam, particularly beneficial for environments with numerous DAGs using variables, improving the clarity and usability of Airflow's logging system.

AI Summary: The task requires fixing broken documentation links for the `EmptyOperator` in Apache Airflow versions 2.7 and later. This involves identifying the correct location of the updated documentation and updating the links to point to the correct location. A pull request is expected.

AI Summary: The task is to debug and fix a bug in Apache Airflow 3.0.2 where parameters entered in the UI during a DAG backfill are not passed to the tasks within the DAG runs. The solution involves investigating why the parameters aren't being propagated correctly from the UI to the task context during backfilling, and modifying the Airflow code to ensure parameters are correctly passed to all DAG runs created during a backfill operation, matching the behavior of single DAG run triggers.

AI Summary: Enhance the Apache Airflow documentation to clearly explain how users can access and utilize the `triggering_asset_event` from the `Context` object within their DAGs, covering various methods including Jinja templating. This involves improving existing documentation on asset-driven scheduling to make it more user-friendly and comprehensive.