chore(migration): Migrate code from googleapis/python-bigquery into packages/google-cloud-bigquery #14923
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat: support range in queries as dict * fix sys tests * lint * fix typo
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert pinned requirement version and add triple equal "===" prevents dependabot from attempting to upgrade it in the future --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
* Add ZSTD to compression types * feat: adds tests re Compression types * revise datatype from Enum to object * adds license text and docstring * change object back to enum datatype * updates compression object comparison * updates Compression class * jsonify and sort the input and output for testing * Update tests/unit/job/test_extract.py * moved json import statement * removed enums test and file --------- Co-authored-by: Ethan Steinberg <ethan.steinberg@gmail.com> Co-authored-by: Tim Sweña (Swast) <swast@google.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
* chore(python): bump idna from 3.4 to 3.7 in .kokoro Source-Link: googleapis/synthtool@d50980e Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:5a4c19d17e597b92d786e569be101e636c9c2817731f80a5adec56b2aa8fe070 * Apply changes from googleapis/synthtool#1950 --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
#1898) …PI_FIELD Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/python-bigquery/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Fixes #<issue_number_goes_here> 🦕
* fix: add types to DatasetReference constructor * fix: add types to DatasetReference constructor * fix: DatasetReference.from_string test coverage --------- Co-authored-by: Karel Serruys <karel.serruys@foodpairing.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: meredithslota <meredithslota@google.com> Co-authored-by: Tim Sweña (Swast) <swast@google.com>
…ady finished (#1900) fix: retry query job after ambiguous failures Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* feat: support range in queries as dict * fix sys tests * lint * add arrow support * fix python 3.7 test error * print dependencies in sys test * add unit test and docs * fix unit test * add func docs * add sys test for tabledata.list in arrow * add sys test for tabledata.list as iterator * lint * fix docs error * fix docstring * fix docstring * fix docstring * docs * docs * docs * move dtypes mapping code * address comment * address comment * fix pytest error * Revert "move dtypes mapping code" This reverts commit c46c65c822b3c8295d5d6650b1c9c97d35d2ba5b. * remove commented out assertions * typo and formats * add None-check for range_element_type and add unit tests * change test skip condition * fix test error * change test skip condition * change test skip condition * change decorator order * use a different way to construct test data * fix error message and add warning number check * add warning number check and comments
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* feat: support insertAll for range * revert INTERVAL regex * lint * add unit test * lint
* fix: add pyarrow version check for range support * add comment why we are making a separate constant --------- Co-authored-by: Tim Sweña (Swast) <swast@google.com>
* add new presubmit for test purposes * add additional sessions * Update .kokoro/presubmit/presubmit-2.cfg * Update .kokoro/presubmit/presubmit-2.cfg * added timer to nox sessions * Update .kokoro/presubmit/presubmit-2.cfg * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * removes references to most environment variables * testing the use of base names for the nox sessions * removes references to unneeded linting and typing env variables * change file name and update env_vars in presubmit-2 * remove timed decorators * revert several files * Update noxfile.py * remove test, remove unneeded vars, etc --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* feat: adds timer decorator to sessions * updates _calculate_duration function * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update samples/geography/requirements.txt --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * pin grpcio===1.62.2 for python 3.7 support of python 3.7 is dropped starting 1.63 --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
…1925) * perf: decrease the threshold in which we use the BQ Storage Read API * fix unit test * update comment
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* Update format_options.py to include the newly added map target type. The map target type creates a schema without the added key_value repeated field. * Added tests * add unit test * lint --------- Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com> Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
#1933) * chore(deps): bump requests from 2.31.0 to 2.32.2 in /samples/geography Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.31.0...v2.32.2) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * pin requests==2.31.0 for python 3.7 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
* chore: add warning if storage module not found * Update tests/unit/test_table.py Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> * Update tests/unit/test_table.py Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> --------- Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
* docs: add a TROUBLESHOOTING.md file with tips for logging * typo * finish my sentence --------- Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
…#2265) * chore: migrate test_base retry tests * migrate job_helpers test * migrate more tests * fix initiate resumable upload tests * fix failing tests * remove dead test code
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
This PR updates query handling to allow base config properties like job timeout, reservation, and a preview max slots field to leverage the faster path (e.g. using jobs.query rather than jobs.insert).
* bug: updates __eq__ comparison on TableConstraint * updates tests * moves test out of class to accommodate pytest params
* docs: clarify the api_method arg for client.query()
…y mean that the job has not completed or was unsuccessful (#2278) Internal issue b/440349994
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* feat: add additional query stats This PR adds support for incremental query stats.
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* docs: remove experimental annotations from GA features Corrects some documentation drift.
Context: internal issue b/445984807 comment 10.
* fix(job): honor custom retry in job.result() The `_AsyncJob.result()` method was not correctly passing the `retry` argument to the superclass's `result()` method when the `retry` object was the same as the default retry object. This caused the default retry settings to be ignored in some cases. This change modifies the `result()` method to always pass the `retry` argument to the superclass, ensuring that the provided retry settings are always honored. A new test case is added to verify that `job.result()` correctly handles both the default retry and a custom retry object. * Update tests/unit/test_job_retry.py * Update tests/unit/test_job_retry.py * blacken and lint * udpates retry handling and testing of retry handling * Update tests/unit/test_job_retry.py * Update tests/unit/test_job_retry.py * Update tests/unit/test_job_retry.py --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
* update pyproject.toml to follow PEP 639 * Update pyproject.toml PEP 639 Thanks for the feedback, I've removed the version number completely as requested. * Update pyproject.toml --------- Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
* chore(python): Add Python 3.14 to python post processor image Source-Link: googleapis/synthtool@16790a3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:543e209e7c1c1ffe720eb4db1a3f045a75099304fb19aa11a47dc717b8aae2a9 * Update samples/snippets/noxfile.py * Update samples/notebooks/noxfile.py * Update samples/magics/noxfile.py * Update samples/geography/noxfile.py * Update samples/desktopapp/noxfile.py * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>
* feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. * Update google/cloud/bigquery/routine/routine.py * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. - Added additional tests to improve code coverage based on feedback. * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. - Added additional tests to improve code coverage based on feedback. - Addressed PyType errors by using helper functions for type conversion. * Update tests/unit/routine/test_external_runtime_options.py * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. - Added additional tests to improve code coverage based on feedback. - Addressed PyType errors by using helper functions for type conversion. - Addressed formatting nits from code review. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
* feat: adds support for Python runtime 3.14 * adds step to install gdal * adds files required by pyarrow * adds repo required by pyarrow * corrects url to repo required by pyarrow * testing a theory with a conditional * testing a theory with a conditional version of ubuntu * testing a new approach to installing arrow * testing a new approach to dearmoring the key * back to the basics * trying a conditional again. * adds explanatory comment resets ubuntu version to latest * Apply suggestion from @chalmerlowe * Apply suggestion from @chalmerlowe * Apply suggestion from @chalmerlowe * Apply suggestion from @chalmerlowe
Towards googleapis/librarian#2456 Files removed which is no longer used - Owlbot config files, including owlbot.py - Sync repo settings config file - Release please config files
Temporarily pin `pytest < 9` to resolve the following issue
```
for invalid_view_value in invalid_view_values:
> with self.subTest(invalid_view_value=invalid_view_value):
tests/unit/test_client.py:810:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/contextlib.py:144: in __exit__
next(self.gen)
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/contextlib.py:144: in __exit__
next(self.gen)
.nox/unit-3-11/lib/python3.11/site-packages/_pytest/unittest.py:438: in addSubTest
self.ihook.pytest_runtest_logreport(report=sub_report)
.nox/unit-3-11/lib/python3.11/site-packages/pluggy/_hooks.py:512: in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.nox/unit-3-11/lib/python3.11/site-packages/pluggy/_manager.py:120: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.nox/unit-3-11/lib/python3.11/site-packages/xdist/remote.py:289: in pytest_runtest_logreport
self.sendevent("testreport", data=data)
.nox/unit-3-11/lib/python3.11/site-packages/xdist/remote.py:126: in sendevent
self.channel.send((name, kwargs))
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:912: in send
self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
^^^^^^^^^^^^^^^^^^^^
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1629: in dumps_internal
return _Serializer().save(obj) # type: ignore[return-value]
^^^^^^^^^^^^^^^^^^^^^^^
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1647: in save
self._save(obj)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1667: in _save
dispatch(self, obj)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1744: in save_tuple
self._save(item)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1667: in _save
dispatch(self, obj)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1740: in save_dict
self._write_setitem(key, value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1734: in _write_setitem
self._save(value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1667: in _save
dispatch(self, obj)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1740: in save_dict
self._write_setitem(key, value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1734: in _write_setitem
self._save(value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1667: in _save
dispatch(self, obj)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1740: in save_dict
self._write_setitem(key, value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1734: in _write_setitem
self._save(value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1667: in _save
dispatch(self, obj)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1740: in save_dict
self._write_setitem(key, value)
.nox/unit-3-11/lib/python3.11/site-packages/execnet/gateway_base.py:1734: in _write_setitem
self._save(value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <execnet.gateway_base._Serializer object at 0x7fc425a0c710>
obj = <object object at 0x7fc425bcb920>
def _save(self, obj: object) -> None:
tp = type(obj)
try:
dispatch = self._dispatch[tp]
except KeyError:
methodname = "save_" + tp.__name__
meth: Callable[[_Serializer, object], None] | None = getattr(
self.__class__, methodname, None
)
if meth is None:
> raise DumpError(f"can't serialize {tp}") from None
E execnet.gateway_base.DumpError: can't serialize <class 'object'>
```
The upstream issue is tracked in
pytest-dev/pytest-xdist#1273
This PR updates the librarian sha to support v1.0.0
… into migration.python-bigquery.migration.2025-11-24_21-55-24.migrate
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
30 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See #10980.
This PR should be merged with a merge-commit, not a squash-commit, in order to preserve the git history.