# ETL and high performance computing

Midio has an **experimental** native package for using the [**polars data frame library**](https://pola.rs/) for handling large amounts of data.

## How to use it

Start by adding the **polars** package using the [package manager](/midio-docs/package-manager.md).

Then, use one of the functions in the `Polars.Source` module, to import your data into the polars format. Supported formats include CSV, JSON, newline delimited JSON and querying a Postgres database.

These functions return a dataframe object, which can be operated on using either `Polars.Execute Sql` or `Polars.Execute Dynamic Sql`.

### Execute Sql

`Execute Sql` lets you provide a list of inputs using, and then executing an SQL query over those inputs. By default its accepts a single input.

<figure><img src="/files/LwkxD6odbGRq4Rmyn61c" alt=""><figcaption></figcaption></figure>

### Execute Dynamic SqlAbout the output

Works in a simliar way, but expects an object where the key is the name of the source, and the value is a data frame.

<figure><img src="/files/LyAyDlDi5mxtSY4uQgTQ" alt=""><figcaption></figcaption></figure>

## Getting the results

After executing one or more queries, the results can be collected using the `Polars.Collect` function.

<figure><img src="/files/nWSt2UNKM5PLtF1c2FMp" alt=""><figcaption></figcaption></figure>

### About the output

The output is by default in a column-oriented format, meaning you get an object where each field represents a column, and its value is a list containing the values for each row.

<figure><img src="/files/eU8PV6X2HeZQQbdXjd7T" alt=""><figcaption></figcaption></figure>

To get this data converted back to a list of objects, which is often a more useful format, you can use the `Transpose` function.

<figure><img src="/files/riUKXVwFXdeCpBA2bHBs" alt=""><figcaption></figcaption></figure>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.midio.com/midio-docs/etl-and-high-performance-computing.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.