Onboarding Intake(2/3): Onboard intakes resource - #1689
devanshcache wants to merge 7 commits into
Conversation
|
Hi @devanshcache , |
|
Hi @cgoetz-inovex, Thank you looking forward for your feedback :) |
|
This PR was marked as stale after 7 days of inactivity and will be closed after another 7 days of further inactivity. If this PR should be kept open, just add a comment, remove the stale label or push new commits to it. |
| maps.Copy(tempConfig, base) | ||
| tempConfig["intake_name"] = config.StringVariable(intakeNameMaxUpd) | ||
| tempConfig["description"] = config.StringVariable("Updated full intake description") | ||
| tempConfig["max_messages_per_hour"] = config.IntegerVariable(1100) |
There was a problem hiding this comment.
Would add also update values for the catalog stuff here
| } | ||
| } | ||
|
|
||
| func testIntakesConfigVarsMin() config.Variables { |
There was a problem hiding this comment.
In the API docs only uri and warhouse of catalog is marked as required. So the dremio part could be removed here (auth not marked as required). API doc bug or intentionally?
There was a problem hiding this comment.
It is intentional in the API:
- API allows
catalog.authto be omitted (defaulting totype: noneon the backend) to support unauthenticated Iceberg catalogs. In the provider schema, all Dremio fields are markedOptional: true. - However in live acceptance tests (
TestAccIntakesMin) terraform waits for the intake to reachactivestate. Without a reachable Iceberg catalogthe intake fails to connect and enters afailedstate, breaking the test waiter. - because Dremio is currently the only live Iceberg catalog available in our test environment, provisioning Dremio was necessary for the runner to connect successfully and reach
active.
| } | ||
|
|
||
| // getDremioPAT authenticates against Dremio UI API, enables PAT support key, resolves user UUID, and issues a PAT | ||
| func getDremioPAT(ctx context.Context, uiEndpoint, username, password string) (string, error) { |
There was a problem hiding this comment.
This function does not feel right.
Can you explain please why this is needed?
There was a problem hiding this comment.
I just had a deeper look into this and found that there is a documentation regarding the dremio PAT (https://docs.stackit.cloud/products/data-and-ai/intake/how-tos/configure-dremio-for-stackit-intake-integration/).
How is the customer supposed to use this in terraform? Such a getDremioPAT() function cannot be used by a customer. First a dremio instance needs to be created, then the customer needs to activate PAT for that instance via UI, get the dremio PAT via UI as well and is then able to create a intake instance?
There was a problem hiding this comment.
Please note that this is an acceptance test not a Terraform resource. To run it, we need to create an Intake instance, which requires obtaining a Dremio PAT.
There was a problem hiding this comment.
Yes, I am aware of that.
Those questions apply in general.
There was a problem hiding this comment.
can we resolve this comment?
There was a problem hiding this comment.
This doesn't actually resolve my original concern. My point was never about the test itself, but about what getDremioPAT() exposes:
There is currently no way for a customer to obtain a Dremio PAT through Terraform or any STACKIT API. The real flow today is:
- create the Dremio instance and user via Terraform
- log into the Dremio UI by hand
- manually flip on PAT support
- manually generate the token and paste it into dremio_personal_access_token_wo.
That's not a test artifact, that's the actual customer experience and honestly, it's a mess. Forcing a manual UI detour just to get a secret into the config defeats the point of using Terraform at all.
We already have the right pattern elsewhere in this provider: stackit_telemetryrouter_accesstoken, stackit_logs_accesstoken, stackit_modelexperiments_token, stackit_modelserving_token, and stackit_edgecloud_token all mint their secret server-side via the STACKIT API at resource create time, using the STACKIT project/service-account credentials the customer already has. No login against a separate system required. That's the correct shape for this too: a STACKIT API endpoint that issues a Dremio PAT for a given Dremio user, authenticated with STACKIT project credentials, not Dremio username/password.
getDremioPAT() in the acceptance test is effectively proving this gap by reverse-engineering undocumented Dremio UI endpoints (/apiv2/settings/..., /api/v3/user/...) just to get the test to run end-to-end. I don't want that pattern to exist even as a test helper. If the only way to make our own acceptance test work is to script around Dremio's internal UI API, that's a signal the product is missing a real API, not that the test needs a workaround.
There was a problem hiding this comment.
Thanks for the feedback. We agree that a native token-minting API/resource is the ideal long-term design.
Since token issuance is not yet available via the public API, the resource was designed to take the token as a credential input.
To address the concern with getDremioPAT() and avoid undocumented UI endpoints in the test suite, we could remove the helper entirely and load the token via an environment variable in the acceptance test (skipping the step if not set).
However, this requires maintaining a long-lived, pre-configured Dremio instance for CI and Dremio PATs have a maximum lifetime of 180 days, it would require periodic manual rotation to keep CI running.
Would you prefer moving to the environment variable approach despite the rotation overhead or is there another direction you would recommend?
|
This PR was marked as stale after 7 days of inactivity and will be closed after another 7 days of further inactivity. If this PR should be kept open, just add a comment, remove the stale label or push new commits to it. |
3 participants
Description
This PR onboards the new STACKIT Intake (ticket) service into the Terraform provider.
Intake is composed of three components:
Intake Runners: dedicated, isolated runtime data ingestion environment
Intakes: a specific data stream or topic within an Intake Runner
Intake Users: provides secure access credentials for your applications to connect to your Intake
This PR contains the Intakes part only to make a quicker and less overwhelming review.
Checklist
make fmtexamples/directory)make generate-docs(will be checked by CI)make test(will be checked by CI)make lint(will be checked by CI)