Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
276 changes: 97 additions & 179 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,95 +19,61 @@

## What is this?

ODBC is the standard way desktop tools talk to a database. Excel, Tableau,
Power BI and Python's pyodbc all speak it. Each database needs its own *driver*,
a shared library the tool loads that translates those standard calls into
whatever the database actually speaks.

Writing one is a large job, and most of it has nothing to do with your database.
The driver has to hand out and validate handles, convert every string to and
from UTF-16, report errors in the exact format the standard demands, copy values
into buffers the application supplied, and not crash when the application lies
about how big those buffers are.

`stackable-odbc-core` is that shared part, written once. What you supply is the
part that really is about your database: how to connect and authenticate, how
to run a query and read rows back, how your database's types map onto ODBC's,
and how to answer the catalog questions. For a networked database that is a
client library in its own right, and writing one is not trivial. It is just
not ODBC work. One macro then generates the C entry points the standard
requires.

This is a library rather than a driver you can load on its own. A working driver
is this crate plus a backend, and
[stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite)
is the smallest complete example of one.
This is a reusable crate to help develop ODBC drivers in Rust.

If you don't know what ODBC is, then this repo/crate will probably not be of interest to you.

There are other libraries for ODBC in Rust like [odbc-sys](https://crates.io/crates/odbc-sys) but they all focus on ODBC clients.
This library is exclusively meant to fill the gap to write actual _ODBC drivers_.
Writing a driver is obviously very specific to your target datastore but there is a lot of common stuff that this library handles:

- Hands out and validates handles
- Converts every string to and from UTF-16
- Reports errors in the exact format the standard demands
- Copies values into buffers the application supplied (and handles errors gracefully)
- ...and more

`stackable-odbc-core` is that shared part.
You just need to supply the part that is really about your database:

- How to connect and authenticate
- How to run a query, read rows back, map types onto ODBC's and so on...

A macro then generates all the necessary C plumbing the standard requires.

This is a library rather than a driver you can load on its own.
A working driver is this crate plus a backend, and [stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite) is the smallest complete example of one.

## What you get

- **A database backend is two traits and one macro.** Implement `Backend` and
`StatementBackend`, then call `forward_ffi!`. The compiler names everything
still missing, so there is no list to work through by hand. Core holds no
database-specific code, so a driver never forks or patches it.

- **A handle is a ticket number, not a memory address.** ODBC hands the
application a `SQLHANDLE` that refers to a connection or a running query. The
obvious implementation is a raw pointer, and then an application that frees a
handle twice, or uses one after freeing it, corrupts the driver's memory.
That is undefined behaviour, so the program may crash, or may quietly return a
wrong answer.

Here a handle is a slot number plus a counter. The driver looks it up in its
own table and never follows the pointer the application passed. Freeing bumps
that slot's counter, so every ticket still referring to it stops matching.
Use-after-free and double-free become a clean "invalid handle" error rather
than memory corruption.

- **Two threads can share one connection safely.** The standard requires it,
because "drivers must therefore support safe, multithread access to this
information", and many drivers leave it to the Driver Manager instead. Each
connection here has one lock, shared with every query started on it, so a call
touching both a query and its connection takes a single lock. That leaves no
lock ordering to get wrong, which is the usual way a driver deadlocks.
`SQLCancel` takes no lock at all, because cancelling a slow query must not
wait for the query it is cancelling.

- **The query timeout covers waiting for rows, not just sending the query.** An
application sets `SQL_ATTR_QUERY_TIMEOUT` to say "give up after N seconds".
Most drivers run that clock only while the query is being submitted, but a
database can answer with the column names immediately and then take much
longer to produce the first row. A timer covering only submission bounds
nothing, so this one runs during `SQLFetch` as well.

- **Core builds the catalog answers.** For "what tables exist?" and its
relatives, the standard dictates the exact columns, their order, and how the
rows are sorted. A backend returns ordinary Rust structs with named fields,
and core puts the columns in order, sorts the rows and normalises identifier
case. You cannot get the column order or count wrong because you never write
them, and a column added to one of those result sets is a change in core
alone.

- **Value conversion is already done.** When an application supplies a parameter
as text and asks for it to be treated as a number, the standard has three
large tables saying exactly what each conversion does, down to which warning
to raise when precision is lost. All three are implemented: character, binary
and numeric, including the interval rows and the optional `01S07` warning for
fractional seconds that were rounded away.

- **Windows is a first-class target.** Its Driver Manager is stricter than
unixODBC and it fails quietly, so missing one requirement stops a feature
working with no error to explain why. The known traps are handled: answering
the version query it makes before connecting, reporting the complete function
list it uses to build its dispatch table, and not exporting the deprecated
ODBC 2.x functions, because exporting one replaces the Driver Manager's own
better implementation with yours.

- **Checked by more than unit tests.** Three tools cover what ordinary tests
cannot. Miri runs the code in an interpreter that detects undefined behaviour
and leaked handles, loom re-runs the locking code under every thread
interleaving rather than the one that happened to occur, and cargo-fuzz throws
random input at the buffer-copying code under AddressSanitizer. All three run
on every pull request, alongside the unit tests on Linux and Windows.
- **We handle (pun intended) the Handles.**
If you don't know what a Handle is in ODBC-land, you're lucky.
A driver gets handed various Handles (e.g. `SQLHANDLE`) which are basically just pointers to memory holding its state.
Ours is a slot number plus a counter looked up in our own table, so the pointer the application passed is never followed and a double free is a clean error instead of memory corruption.

- **Two threads can share one connection safely.**
The standard requires it and many drivers leave it to the Driver Manager instead.
This crate handles it correctly by using one lock per connection, shared with its queries, so there is no lock ordering left to get wrong.
`SQLCancel` takes no lock at all, because cancelling a slow query must not wait for the query it is cancelling.

- **The query timeout covers waiting for rows, not just sending the query.**
A database can answer with the column names immediately and take much longer to produce the first row, so `SQL_ATTR_QUERY_TIMEOUT` runs during `SQLFetch` too.

- **Core builds the catalog answers.**
Return ordinary Rust structs with named fields and core puts the columns in the order the standard dictates, sorts the rows and normalises identifier case.
You cannot get the column order or count wrong.

- **Value conversion is already done.**
All three of the standard's conversion tables are implemented: character, binary and numeric, down to the interval rows and the optional `01S07` warning for rounded-away fractional seconds.

- **Windows is a first-class target.**
Its Driver Manager is stricter than unixODBC and it fails quietly.
Getting this correct is annoying.
The known traps are handled and [AGENTS.md](https://github.com/stackabletech/stackable-odbc-core/blob/main/AGENTS.md) has the checklist.

- **Checked by more than unit tests.**
Miri catches undefined behaviour and leaked handles, loom re-runs the locking code under every thread interleaving, and cargo-fuzz throws random input at the buffer copies under AddressSanitizer.
All three run on every pull request, alongside the unit tests on Linux and Windows.

## Writing a driver

Expand All @@ -116,124 +82,76 @@ cargo new --lib stackable-odbc-xyz
cargo add stackable-odbc-core
```

Implement `Backend` and `StatementBackend` for your database, then generate the
C ABI in `lib.rs`:
Implement `Backend` and `StatementBackend` for your database, then generate the C ABI in `lib.rs`:

```rust,ignore
stackable_odbc_core::forward_ffi!(crate::backend::XyzBackend);
```

That one line expands to every exported `SQL*` entry point, plus `ConfigDSNW` on
Windows, each forwarding to the generic implementation in this crate.

`Backend` has four associated types and a body of required methods, but most of
them are one-line capability declarations such as `supports_catalogs`,
`identifier_case` and `sql_conformance`, each answering a single question about
your database. They are required rather than defaulted on purpose: any default
core supplied would be a claim about your database that nobody ever checked, and
a wrong one is invisible, because the driver would confidently tell applications
something untrue and nothing would complain. `StatementBackend` is the opposite,
with one associated type and no required methods, so you override only what your
backend supports.

In practice you do not look the list up. Write the four associated types, run
`cargo check`, and the compiler names what is still missing.

Two traits and one macro bound the surface, not the effort. A backend for a
real database is a real client. Authentication, sessions, type mapping, catalog
queries and error mapping are all yours, and in both existing drivers that adds
up to a substantial crate. What core takes off your hands is the ODBC half: the
handle table, the UTF-16, the diagnostics format, the buffer copying and the
conversion tables. That half is identical for every database, and it is the
half where a mistake corrupts memory rather than returning a wrong answer.

[AGENTS.md](https://github.com/stackabletech/stackable-odbc-core/blob/main/AGENTS.md)
has the full walkthrough: how a call flows through the layers, what each
capability method means, the catalog and descriptor rules, and the Windows
Driver Manager checklist.
That one line expands to every exported `SQL*` entry point, plus `ConfigDSNW` on Windows.

`Backend` consists mostly of one-line capability declarations such as `supports_catalogs` and `identifier_case`.
None of them are defaulted, because a default would be a claim about your database that nobody ever checked, and a wrong one is invisible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
None of them are defaulted, because a default would be a claim about your database that nobody ever checked, and a wrong one is invisible.
None of them are defaulted, because a default would be a de facto and untested assumption about how your database behaves, and if it's wrong it is easy to overloook.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I applied all but this one where I like my original better. If you feel strongly about it I can change it though. Not a really strong opinion.

`StatementBackend` is the opposite and has no required methods, so you override only what your backend supports.

Don't look the list up.
Write the four associated types, run `cargo check`, and the compiler names what is still missing.

[AGENTS.md](https://github.com/stackabletech/stackable-odbc-core/blob/main/AGENTS.md) has the full walkthrough: how a call flows through the layers, what each capability method means, and the catalog, descriptor and Windows rules.

## Conformance

This implements ODBC 3.80 at the `SQL_OIC_CORE` level, the base of the
standard's three interface-conformance levels and the one an application may
assume of any driver. All four handle types can be allocated and freed, and all
five descriptor functions work. Descriptors are the standard's own way of
describing a bound column or parameter, and one can be shared between queries on
a connection.

This is a Unicode driver: every function that takes or returns a string is
exported only in its wide (`W`-suffixed) form e.g. `SQLConnectW`. The
Driver Manager translates for ANSI applications, so they keep working and the
driver never carries a second set of entry points. Functions with no strings
in their signature, such as `SQLFetch`, have one spelling and are exported
unsuffixed.

`CORE_EXPORTED_FUNCTIONS` in `src/function_id.rs` is the authoritative list of
what is exported, and a guard test pins every entry to a symbol that exists. The
deprecated ODBC 2.x functions are left out, because the Driver Manager already
emulates them on top of the modern ones and usually does it better than a driver
would, so exporting your own version switches that off rather than adding
anything. `SQLExtendedFetch` is the exception the Driver Manager does not map,
so core exports it.
ODBC 3.80 is implemented at the `SQL_OIC_CORE` level, which is the base of the standard's three interface-conformance levels and the one an application may assume of any driver.
All four handle types and all five descriptor functions work, and a descriptor can be shared between queries on a connection.

This is a Unicode driver, so anything taking or returning a string is exported only in its wide form (`SQLConnectW`) and the Driver Manager translates for ANSI applications.
Functions with no strings in their signature, such as `SQLFetch`, have one spelling and are exported unsuffixed.

`CORE_EXPORTED_FUNCTIONS` in `src/function_id.rs` is the authoritative list, pinned by a guard test.
The deprecated ODBC 2.x functions are absent on purpose: the Driver Manager already emulates them on top of the modern ones.
`SQLExtendedFetch` is the one the Driver Manager does not map, so core exports it.

## Limits

Each of these is reported to the application as unsupported rather than quietly
ignored, so a tool can react instead of trusting a wrong answer.

- **Results are read front to back only** (`SQL_SO_FORWARD_ONLY`), so there is
no jumping to a row and no going backwards. `SQLFetchScroll` accepts
`SQL_FETCH_NEXT` and rejects every other direction with `HY106`.
- **One row at a time.** There are no block cursors, so
`SQL_ATTR_ROW_ARRAY_SIZE` is fixed at 1. Asking for more returns 1 with an
`01S02` warning, and `SQL_GD_BLOCK` is never reported.
- **No bookmarks**, which are saved row positions an application can return to
later, and no automatic population of parameter metadata, so
`SQL_ATTR_AUTO_IPD` stays `SQL_FALSE`.
- **No async.** Every call runs to completion before returning:
`SQL_ASYNC_MODE` is reported as `SQL_AM_NONE`, and turning on
`SQL_ATTR_ASYNC_ENABLE` is refused rather than ignored. This is about the
calling thread, not the shape of the results. Rows still arrive one
`SQLFetch` at a time, the query timeout still bounds a slow query, and
`SQLCancel` still interrupts one. `Backend` is synchronous too, so a driver
built on an async client library bridges to it internally, for example with
a current-thread tokio runtime and `block_on`.
Each of these limits is actually reported back to an application that tries to use one of these features so it can react to it.

| Not supported | What the application sees |
|---|---|
| Scrollable cursors | `SQL_SO_FORWARD_ONLY`; `SQLFetchScroll` takes `SQL_FETCH_NEXT` and rejects the rest with `HY106` |
| Block cursors | `SQL_ATTR_ROW_ARRAY_SIZE` fixed at 1, returning 1 with an `01S02` warning; `SQL_GD_BLOCK` never reported |
| Bookmarks, automatic parameter metadata | `SQL_ATTR_AUTO_IPD` stays `SQL_FALSE` |
| Async | `SQL_AM_NONE`; `SQL_ATTR_ASYNC_ENABLE` is refused, not ignored |

Async here means the calling thread, not the shape of the results.
Rows still arrive one `SQLFetch` at a time, the query timeout still bounds a slow query, and `SQLCancel` still interrupts one.
`Backend` is synchronous too, so a driver built on an async client library bridges to it internally, for example with a current-thread tokio runtime and `block_on`.

## Drivers built on this crate

Each driver is a separate crate supplying only its `Backend` and
`StatementBackend` implementation.
Each is a separate crate supplying only its `Backend` and `StatementBackend` implementation.

- [stackable-odbc-trino](https://github.com/stackabletech/stackable-odbc-trino),
an ODBC driver for [Trino](https://trino.io/).
- [stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite),
a SQLite driver, used as a worked example and as the test driver for the
framework itself.
- [stackable-odbc-trino](https://github.com/stackabletech/stackable-odbc-trino), an ODBC driver for [Trino](https://trino.io/).
- [stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite), a SQLite driver, used as a worked example and as the test driver for the framework itself.

## Resources

- [ODBC API reference](https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/odbc-api-reference?view=sql-server-ver16),
the authoritative specification. It is the most detailed source and still not
an easy read.
- [Header files](https://github.com/microsoft/ODBC-Specification/blob/master/Windows/inc/sql.h)
for the unreleased ODBC 4 standard, mostly valid for the older ones too.
- [odbc-sys](https://github.com/pacman82/odbc-sys), the ODBC type definitions
this crate builds on.
- [ODBC API reference](https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/odbc-api-reference?view=sql-server-ver16), the authoritative specification.
It is the most detailed source and still not an easy read.
- [Header files](https://github.com/microsoft/ODBC-Specification/blob/master/Windows/inc/sql.h) for the unreleased ODBC 4 standard, mostly valid for the older ones too.
- [odbc-sys](https://github.com/pacman82/odbc-sys), the ODBC type definitions this crate builds on. Thank you!

## Getting help

- [GitHub Discussions](https://github.com/orgs/stackabletech/discussions) for
questions
- [GitHub Discussions](https://github.com/orgs/stackabletech/discussions) for questions
- [Discord](https://discord.gg/7kZ3BNnCAF) to talk to us
- [Issues](https://github.com/stackabletech/stackable-odbc-core/issues) for
bugs, and [SECURITY.md](SECURITY.md) for anything security-related
- [Issues](https://github.com/stackabletech/stackable-odbc-core/issues) for bugs, and [SECURITY.md](SECURITY.md) for anything security-related

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for building from source, running the
tests, and how the repository is laid out. [CHANGELOG.md](CHANGELOG.md) records
what changed in each release.
See [CONTRIBUTING.md](CONTRIBUTING.md) for building from source, running the tests, and how the repository is laid out.
[CHANGELOG.md](CHANGELOG.md) records what changed in each release.

## License

Apache-2.0. See [LICENSE](./LICENSE) and [NOTICE](./NOTICE).
Apache-2.0.
See [LICENSE](./LICENSE) and [NOTICE](./NOTICE).