datafusion-comet

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion Comet

[![Apache licensed][license-badge]][license-url]
[![Discord chat][discord-badge]][discord-url]
[![Pending PRs][pending-pr-badge]][pending-pr-url]
[![Maven Central][maven-badge]][maven-url]

[license-badge]: https://img.shields.io/badge/license-Apache%20v2-blue.svg
[license-url]: https://github.com/apache/datafusion-comet/blob/main/LICENSE.txt
[discord-badge]: https://img.shields.io/discord/885562378132000778.svg?logo=discord&style=flat-square
[discord-url]: https://discord.gg/3EAr4ZX6JK
[pending-pr-badge]: https://img.shields.io/github/issues-search/apache/datafusion-comet?query=is%3Apr+is%3Aopen+draft%3Afalse+review%3Arequired+-status%3Afailure&label=Pending%20PRs&logo=github
[pending-pr-url]: https://github.com/apache/datafusion-comet/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse+review%3Arequired+-status%3Afailure+sort%3Aupdated-desc
[maven-badge]: https://img.shields.io/maven-central/v/org.apache.datafusion/comet-spark-spark4.0_2.13
[maven-url]: https://search.maven.org/search?q=g:org.apache.datafusion%20AND%20comet-spark

<img src="docs/source/_static/images/DataFusionComet-Logo-Light.png" width="512" alt="logo"/>

Apache DataFusion Comet is a high-performance accelerator for Apache Spark. Comet keeps Spark queries
**Arrow-native end-to-end**: operators, expressions, shuffle, and broadcast all stay in Apache Arrow
columnar format, avoiding the per-row overhead of Spark's row-based engine. Within the Arrow-native
pipeline, operators and expressions execute as Rust code (via the [Apache DataFusion] query engine)
or as JVM code that operates directly on Arrow batches. Comet integrates with the Spark ecosystem
without requiring any code changes.

**Comet provides a ~2x speedup over Spark 4.2 for TPC-DS @ SF 1000 (1TB), resulting in ~50% cost savings.**

That speedup gives you a choice: finish the same Spark workload in half the time on the cluster you already
have, or match your current Spark performance on half the resources. Either way, the gain translates directly into
lower cloud bills, reduced on-prem capacity, and lower energy usage, with no changes to your existing Spark SQL,
DataFrame, or PySpark code. Comet runs on commodity hardware: no GPUs, FPGAs, or other specia