Skip to content

Commit 330783f

Browse files
hallvictoriagavin-aguiarAzureFunctionsPythonVictoria HallYunchuWang
authored
feat: SDK bindings changes (#1423)
* sdk working prototype * misc * new registry pseudo * prototype changes * weird dispatcher changes * almost works * working prototype!! * reset flag * support for mix & match in same func * added pytype to decode() * caching * added tests * revert later * lint, clean up, templates * reorder * reorder again * test pypi problems * correct dir * checking tests * 404s * investigating 500s * no cache * debugging * more helpful debugging * check raw bindings * runtime error * runtime error logging attempt * local logger * sys modules * typo * extensions version * testing changes -- revert later * added host logs * spacing? * debugging changes * Moved dateutil to install_requires (#1443) * Moved dateutil to install_requires * Flake8 fixes * Update Python Worker Version to 4.26.0 * net8 target framework * stdout is not none * tests pass locally * updating extension versions * temp table test fix * table and eventhub test fix * all blob tests * merge with dev * no importlib * specific tests only for deferred bindings * removed -e * syntax * ignore test, add back in cache * revert eh and table tests * blob extension resources * >=3.9 support + test fixes * lint * removed testing changes * lint again * unit tests fix * append to list * fixes + tests * reset flag, extra test * revert meta changes, tests passing locally * fixed tests * fix unit test, import by default * revert default import, meta refactoring * fixed meta refactor * added tests for helpers * type syntax * fixed tests * actually fixed tests * update base ext supported python version * fixing unit test timeouts? * setup.py too * installing from .[deferred-bindings] * update base ext supported python version again * update var name in setup.py * update var name in meta * refactor tests into sep folder * lint + install only .NET6 * remove script * edit workflow * import by default, misc fixes * revert, only changed var name * get_binding check * revert get_binding check, import by default * lint * rename, replace None checks, comments * fixed flag placement * attribute error + lint * fixed unit tests * yml bug/feature reports * moving flag set out of get_binding * lint + 100 * add registry none check * lint * move check * revert error raise * changed order * revert flag set * loader registry none check * adding back in checks * reports & workflow feedback * permissions + consumption * feedback * setup.py * db test csproj * remove 3.12 * combined db & e2e test workflow * fixed consumption workflow * fixed 3.7 tests * skip all tests * renamed registries as constants * add flag as part of FunctionInfo + tests * lint * added logs (fails until new base) * lint + base update * changing bind_name (fails until exts) * Revert "changing bind_name (fails until exts)" This reverts commit 7971498. * is_db property added to PTI * fixed test * quick log * better log * lint * refactor FI + pin pip * return cache val immediately * install .[test-db] only for >=3.9 * fixing installation * fixing installation pt2 * fixing installation pt3 * sep step for installations * sep step for installations pt2 * closed if + semicolon * updated refs to base * updated refs to blob (rerun later) * update setup.py * remove import try/catch * log bug fix * unit tests * pin pip * log if base not found * log delimiters * removed f strings * lint + feedback * pydocs + default val * get_raw_bindings method * var names + docs --------- Co-authored-by: gavin-aguiar <80794152+gavin-aguiar@users.noreply.github.com> Co-authored-by: AzureFunctionsPython <funcdisc@microsoft.com> Co-authored-by: Victoria Hall <victoria.hall@microsoft.com> Co-authored-by: wangbill <12449837+YunchuWang@users.noreply.github.com>
1 parent 0f8c67d commit 330783f

File tree

23 files changed

+1060
-63
lines changed

23 files changed

+1060
-63
lines changed

‎.github/ISSUE_TEMPLATE/deferred_bindings_bug_report.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: Python Worker Deferred Bindings Feature Request
2+
description: File a Deferred Bindings bug report
3+
title: "[Bug] Bug Title Here"
4+
labels: ["python", "bug", "deferred-bindings"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
This form will help you to fill in a bug report for the Azure Functions Python Worker Deferred Bindings feature.
10+
11+
- type: textarea
12+
id: expected-behavior
13+
attributes:
14+
label: Expected Behavior
15+
description: A clear and concise description of what you expected to happen.
16+
placeholder: What should have occurred?
17+
18+
- type: textarea
19+
id: actual-behavior
20+
attributes:
21+
label: Actual Behavior
22+
description: A clear and concise description of what actually happened.
23+
placeholder: What went wrong?
24+
25+
- type: textarea
26+
id: reproduction-steps
27+
attributes:
28+
label: Steps to Reproduce
29+
description: Please provide detailed step-by-step instructions on how to reproduce the bug.
30+
placeholder: |
31+
1. Go to the [specific page or section] in the application.
32+
2. Click on [specific button or link].
33+
3. Scroll down to [specific location].
34+
4. Observe [describe what you see, e.g., an error message or unexpected behavior].
35+
5. Include any additional steps or details that may be relevant.
36+
37+
- type: textarea
38+
id: code-snippet
39+
attributes:
40+
label: Relevant code being tried
41+
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
42+
render: shell
43+
44+
- type: textarea
45+
id: logs
46+
attributes:
47+
label: Relevant log output
48+
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
49+
render: shell
50+
51+
- type: textarea
52+
id: requirements
53+
attributes:
54+
label: requirements.txt file
55+
description: Please copy and paste your requirements.txt file. This will be automatically formatted into code, so no need for backticks.
56+
render: shell
57+
58+
- type: dropdown
59+
id: environment
60+
attributes:
61+
label: Where are you facing this problem?
62+
default: 0
63+
options:
64+
- Local - Core Tools
65+
- Production Environment (explain below)
66+
67+
- type: textarea
68+
id: additional-info
69+
attributes:
70+
label: Additional Information
71+
description: Add any other information about the problem here.

‎.github/ISSUE_TEMPLATE/deferred_bindings_feature_request.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
name: Python Worker Deferred Bindings Feature Request
2+
description: File a Deferred Bindings Feature report
3+
title: "Request a feature"
4+
labels: ["python", "feature", "deferred-bindings"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
This form will help you to fill in a feature request for the Azure Functions Python Worker Deferred Bindings feature.
10+
11+
- type: textarea
12+
id: binding-type
13+
attributes:
14+
label: Binding Type
15+
description: Add information about the binding type.
16+
placeholder: Is this on an existing binding or new binding?
17+
18+
- type: textarea
19+
id: expected-behavior
20+
attributes:
21+
label: Expected Behavior
22+
description: A clear and concise description of what you expected to happen.
23+
placeholder: What should have occurred?
24+
25+
- type: textarea
26+
id: code-snippet
27+
attributes:
28+
label: Relevant sample code snipped
29+
description: Please copy and paste any relevant code snippet of how you want the feature to be used. (This will be automatically formatted into code, so no need for backticks)
30+
render: shell
31+
32+
- type: textarea
33+
id: additional-info
34+
attributes:
35+
label: Additional Information
36+
description: Add any other information about the problem here.

‎.github/Scripts/e2e-tests.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
#!/usr/bin/env bash
22
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail --cov=./azure_functions_worker --cov-report xml --cov-branch --cov-append tests/endtoend/test_worker_process_count_functions.py tests/endtoend/test_threadpool_thread_count_functions.py
3-
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail --cov=./azure_functions_worker --cov-report xml --cov-branch --cov-append --ignore=tests/endtoend/test_worker_process_count_functions.py --ignore=tests/endtoend/test_threadpool_thread_count_functions.py tests/endtoend
3+
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail --cov=./azure_functions_worker --cov-report xml --cov-branch --cov-append --ignore=tests/endtoend/test_worker_process_count_functions.py --ignore=tests/endtoend/test_threadpool_thread_count_functions.py tests/endtoend
4+
python -m pytest -q -n auto --dist loadfile --reruns 4 --instafail --cov=./azure_functions_worker --cov-report xml --cov-branch --cov-append tests/extension_tests/deferred_bindings_tests

‎.github/workflows/ci_consumption_workflow.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ jobs:
3232
python-version: ${{ matrix.python-version }}
3333
- name: Install dependencies
3434
run: |
35+
python -m pip install --upgrade pip
3536
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple -U azure-functions --pre
3637
python -m pip install -U -e .[dev]
3738
python setup.py build

‎.github/workflows/ci_e2e_workflow.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ on:
1515
pull_request:
1616
branches: [dev, main, release/*]
1717
schedule:
18-
# Monday to Thursday 3 AM CST build
18+
# Monday to Friday 3 AM CST build
1919
# * is a special character in YAML so you have to quote this string
20-
- cron: "0 8 * * 1,2,3,4"
20+
- cron: "0 8 * * 1,2,3,4,5"
2121

2222
jobs:
2323
build:
@@ -61,6 +61,9 @@ jobs:
6161
python -m pip install --upgrade pip
6262
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple -U azure-functions --pre
6363
python -m pip install -U -e .[dev]
64+
if [[ "${{ matrix.python-version }}" != "3.7" && "${{ matrix.python-version }}" != "3.8" ]]; then
65+
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --pre -U -e .[test-deferred-bindings]
66+
fi
6467
6568
# Retry a couple times to avoid certificate issue
6669
retry 5 python setup.py build

‎.github/workflows/ci_ut_workflow.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ jobs:
5858
5959
python -m pip install --upgrade pip
6060
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple -U azure-functions --pre
61-
python -m pip install -U -e .[dev]
61+
python -m pip install -U -e .[dev]
6262
6363
# Retry a couple times to avoid certificate issue
6464
retry 5 python setup.py build

‎azure_functions_worker/bindings/__init__.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@
88
from .meta import has_implicit_output
99
from .meta import is_trigger_binding, load_binding_registry
1010
from .meta import from_incoming_proto, to_outgoing_proto, \
11-
to_outgoing_param_binding
11+
to_outgoing_param_binding, check_deferred_bindings_enabled, \
12+
get_deferred_raw_bindings
1213
from .out import Out
1314

1415

@@ -19,5 +20,6 @@
1920
'check_input_type_annotation', 'check_output_type_annotation',
2021
'has_implicit_output',
2122
'from_incoming_proto', 'to_outgoing_proto', 'TraceContext', 'RetryContext',
22-
'to_outgoing_param_binding'
23+
'to_outgoing_param_binding', 'check_deferred_bindings_enabled',
24+
'get_deferred_raw_bindings'
2325
)

‎azure_functions_worker/bindings/datumdef.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ def from_typed_data(cls, td: protos.TypedData):
9393
val = td.collection_string
9494
elif tt == 'collection_sint64':
9595
val = td.collection_sint64
96+
elif tt == 'model_binding_data':
97+
val = td.model_binding_data
9698
elif tt is None:
9799
return None
98100
else:

‎azure_functions_worker/bindings/meta.py‎

Lines changed: 123 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,83 @@
11
# Copyright (c) Microsoft Corporation. All rights reserved.
22
# Licensed under the MIT License.
3+
import os
34
import sys
45
import typing
56

67
from .. import protos
78

89
from . import datumdef
910
from . import generic
11+
1012
from .shared_memory_data_transfer import SharedMemoryManager
13+
from ..constants import CUSTOMER_PACKAGES_PATH
14+
from ..logging import logger
1115

1216
PB_TYPE = 'rpc_data'
1317
PB_TYPE_DATA = 'data'
1418
PB_TYPE_RPC_SHARED_MEMORY = 'rpc_shared_memory'
19+
# Base extension supported Python minor version
20+
BASE_EXT_SUPPORTED_PY_MINOR_VERSION = 8
21+
1522
BINDING_REGISTRY = None
23+
DEFERRED_BINDING_REGISTRY = None
24+
deferred_bindings_cache = {}
1625

1726

1827
def load_binding_registry() -> None:
28+
"""
29+
Tries to load azure-functions from the customer's BYO. If it's
30+
not found, it loads the builtin. If the BINDING_REGISTRY is None,
31+
azure-functions hasn't been loaded in properly.
32+
33+
Tries to load the base extension only for python 3.8+.
34+
"""
35+
1936
func = sys.modules.get('azure.functions')
2037

21-
# If fails to acquire customer's BYO azure-functions, load the builtin
2238
if func is None:
2339
import azure.functions as func
2440

2541
global BINDING_REGISTRY
2642
BINDING_REGISTRY = func.get_binding_registry()
2743

28-
29-
def get_binding(bind_name: str) -> object:
44+
if BINDING_REGISTRY is None:
45+
raise AttributeError('BINDING_REGISTRY is None. azure-functions '
46+
'library not found. Sys Path: %s. '
47+
'Sys Modules: %s. '
48+
'python-packages Path exists: %s.',
49+
sys.path, sys.modules,
50+
os.path.exists(CUSTOMER_PACKAGES_PATH))
51+
52+
if sys.version_info.minor >= BASE_EXT_SUPPORTED_PY_MINOR_VERSION:
53+
try:
54+
import azurefunctions.extensions.base as clients
55+
global DEFERRED_BINDING_REGISTRY
56+
DEFERRED_BINDING_REGISTRY = clients.get_binding_registry()
57+
except ImportError:
58+
logger.debug('Base extension not found. '
59+
'Python version: 3.%s, Sys path: %s, '
60+
'Sys Module: %s, python-packages Path exists: %s.',
61+
sys.version_info.minor, sys.path,
62+
sys.modules, os.path.exists(CUSTOMER_PACKAGES_PATH))
63+
64+
65+
def get_binding(bind_name: str,
66+
is_deferred_binding: typing.Optional[bool] = False)\
67+
-> object:
68+
"""
69+
First checks if the binding is a non-deferred binding. This is
70+
the most common case.
71+
Second checks if the binding is a deferred binding.
72+
If the binding is neither, it's a generic type.
73+
"""
3074
binding = None
31-
registry = BINDING_REGISTRY
32-
if registry is not None:
33-
binding = registry.get(bind_name)
75+
if binding is None and not is_deferred_binding:
76+
binding = BINDING_REGISTRY.get(bind_name)
77+
if binding is None and is_deferred_binding:
78+
binding = DEFERRED_BINDING_REGISTRY.get(bind_name)
3479
if binding is None:
3580
binding = generic.GenericBinding
36-
3781
return binding
3882

3983

@@ -42,8 +86,10 @@ def is_trigger_binding(bind_name: str) -> bool:
4286
return binding.has_trigger_support()
4387

4488

45-
def check_input_type_annotation(bind_name: str, pytype: type) -> bool:
46-
binding = get_binding(bind_name)
89+
def check_input_type_annotation(bind_name: str,
90+
pytype: type,
91+
is_deferred_binding: bool) -> bool:
92+
binding = get_binding(bind_name, is_deferred_binding)
4793
return binding.check_input_type_annotation(pytype)
4894

4995

@@ -71,8 +117,9 @@ def from_incoming_proto(
71117
pb: protos.ParameterBinding, *,
72118
pytype: typing.Optional[type],
73119
trigger_metadata: typing.Optional[typing.Dict[str, protos.TypedData]],
74-
shmem_mgr: SharedMemoryManager) -> typing.Any:
75-
binding = get_binding(binding)
120+
shmem_mgr: SharedMemoryManager,
121+
is_deferred_binding: typing.Optional[bool] = False) -> typing.Any:
122+
binding = get_binding(binding, is_deferred_binding)
76123
if trigger_metadata:
77124
metadata = {
78125
k: datumdef.Datum.from_typed_data(v)
@@ -93,6 +140,13 @@ def from_incoming_proto(
93140
raise TypeError(f'Unknown ParameterBindingType: {pb_type}')
94141

95142
try:
143+
# if the binding is an sdk type binding
144+
if is_deferred_binding:
145+
return deferred_bindings_decode(binding=binding,
146+
pb=pb,
147+
pytype=pytype,
148+
datum=datum,
149+
metadata=metadata)
96150
return binding.decode(datum, trigger_metadata=metadata)
97151
except NotImplementedError:
98152
# Binding does not support the data.
@@ -184,3 +238,61 @@ def to_outgoing_param_binding(binding: str, obj: typing.Any, *,
184238
return protos.ParameterBinding(
185239
name=out_name,
186240
data=rpc_val)
241+
242+
243+
def deferred_bindings_decode(binding: typing.Any,
244+
pb: protos.ParameterBinding, *,
245+
pytype: typing.Optional[type],
246+
datum: typing.Any,
247+
metadata: typing.Any):
248+
"""
249+
This cache holds deferred binding types (ie. BlobClient, ContainerClient)
250+
That have already been created, so that the worker can reuse the
251+
Previously created type without creating a new one.
252+
253+
If cache is empty or key doesn't exist, deferred_binding_type is None
254+
"""
255+
global deferred_bindings_cache
256+
257+
if deferred_bindings_cache.get((pb.name,
258+
pytype,
259+
datum.value.content), None) is not None:
260+
return deferred_bindings_cache.get((pb.name,
261+
pytype,
262+
datum.value.content))
263+
else:
264+
deferred_binding_type = binding.decode(datum,
265+
trigger_metadata=metadata,
266+
pytype=pytype)
267+
deferred_bindings_cache[(pb.name,
268+
pytype,
269+
datum.value.content)] = deferred_binding_type
270+
return deferred_binding_type
271+
272+
273+
def check_deferred_bindings_enabled(param_anno: type,
274+
deferred_bindings_enabled: bool) -> (bool,
275+
bool):
276+
"""
277+
Checks if deferred bindings is enabled at fx and single binding level
278+
279+
The first bool represents if deferred bindings is enabled at a fx level
280+
The second represents if the current binding is deferred binding
281+
"""
282+
if (DEFERRED_BINDING_REGISTRY is not None
283+
and DEFERRED_BINDING_REGISTRY.check_supported_type(param_anno)):
284+
return True, True
285+
else:
286+
return deferred_bindings_enabled, False
287+
288+
289+
def get_deferred_raw_bindings(indexed_function, input_types):
290+
"""
291+
Calls a method from the base extension that generates the raw bindings
292+
for a given function. It also returns logs for that function including
293+
the defined binding type and if deferred bindings is enabled for that
294+
binding.
295+
"""
296+
raw_bindings, bindings_logs = DEFERRED_BINDING_REGISTRY.get_raw_bindings(
297+
indexed_function, input_types)
298+
return raw_bindings, bindings_logs

0 commit comments

Comments
 (0)