impl(bigquery): allow selecting jobs.insert for queries

[fornwall]

Make it possible to force the jobs.insert endpoint so dry runs can access statistics that jobs.query does not return.

[gemini-code-assist]

Code Review

This pull request introduces the ability to route BigQuery queries through the jobs.insert API instead of the default jobs.query by adding a with_jobs_insert builder method to the Query struct. This is particularly useful when the full Job response is needed, such as for dry-run parameter discovery. The query execution logic has been updated to check this new flag, and several unit tests have been added to verify the behavior under different configurations. There are no review comments, and I have no feedback to provide as the changes are well-implemented and documented.

[alvarowolfx]

Hey @fornwall, thanks sending PRs. Maybe next time would be nice to file an issue to we can discuss first.

I'm not on my computer right now to 100% confirm, but to get full job statistics, you just need to force Job creation, which can be controlled via the with_job_creation_mode(JobCreationMode::Required) (we default to Optional in our SDK). I'm not sure on the behavior with dry_run to be honest.

We need to discuss more internally if we want to have a method to force jobs.insert vs jobs.query.

[fornwall]

@alvarowolfx Thanks for the quick reply!

Maybe next time would be nice to file an issue to we can discuss first.

👍 Absolutely!

but to get full job statistics, you just need to force Job creation, which can be controlled via the with_job_creation_mode(JobCreationMode::Required) (we default to Optional in our SDK). I'm not sure on the behavior with dry_run to be honest

For dry runs no job is created (even with JOB_CREATION_REQUIRED) so one can't use jobs.query followed by a jobs.get call to get full statistics for dry runs.

Forcing jobs.insert is necessary for that, and without that possibility all use cases that needs full statistics information on dry runs (statementType, undeclaredQueryParameters, referencedTables, totalBytesProcessedAccuracy) are blocked.

That is, AFAIK - happy to be informed if I'm wrong, or if there are other alternatives here :).

We need to discuss more internally if we want to have a method to force jobs.insert vs jobs.query.

That's understandable!

Should I do anything (close this PR and/or open an issue) while you discuss this internally?

Btw, looked up what the java and python clients does here:

• java throws UnsupportedOperationException on dry runs using query() and in the exception message points at create() (which results in jobs.insert)
• python exposes the api method explicitly as an api_method parameter given to query()

[alvarowolfx]

Forcing jobs.insert is necessary for that, and without that possibility all use cases that needs full statistics information on dry runs (statementType, undeclaredQueryParameters, referencedTables, totalBytesProcessedAccuracy) are blocked.

That is, AFAIK - happy to be informed if I'm wrong, or if there are other alternatives here :).

Yeah, @fornwall you are correct, like I mentioned on the other PR, dry run queries was a totally blind spot on my implementation. I have been working on BQ SDK for the past years and never exercise that path that much (to be fair, BQ is huge). I worked a lot in Go and noticed that it's a problem on it too (we don't check the dry run flag to do routing, and queries are being routed to jobs.query there too https://github.com/googleapis/google-cloud-go/blob/f1e77ceda688f29544fe094f682a03515cb641a2/bigquery/query.go#L467).

Another problem that I found here is that Query::until_done is not going to work on dry run queries as it's never gonna be a CompleteQuery type since it doesn't have a job_id, for a similar reason on the other bug that you reported on #6514. So for dry run queries, customers have to just call Query::send() and getting a CompleteQuery that can be read is not gonna be possible (I need to think a bit if I should introduce a new error type of if the RPC Binding error is enough)

Should I do anything (close this PR and/or open an issue) while you discuss this internally?

If you don't mind, I'll open a PR superseding this one, because I think the best course of actions for now is to check for the dry_run parameter specifically and route to jobs.insert. Also changed the dry_run sample to be more explicit that a dry run query is not gonna become a CompleteQuery and users can get the full job stats from the Query handle. Still appreciate all your contributions, specially since we are getting close to launch this, so keep going.

[fornwall]

If you don't mind, I'll open a PR superseding this one

I don't mind at all! I'll close this one then

Still appreciate all your contributions, specially since we are getting close to launch this, so keep going.

👍 Thanks yourself for the quick feedback and the information sharing!

[fornwall]

to check for the dry_run parameter specifically and route to jobs.insert

👍 That works for my current use cases at least, and sounds logical.