Skip to content

Load namespace modules lazily to cut import time and memory - #627

Open
Mohith26 wants to merge 1 commit into
dropbox:mainfrom
Mohith26:perf/lazy-namespace-imports
Open

Mohith26 wants to merge 1 commit into
dropbox:mainfrom
Mohith26:perf/lazy-namespace-imports

Conversation

@Mohith26

@Mohith26 Mohith26 commented Nov 4, 2026

Copy link
Copy Markdown

Closes #269

import dropbox imports every namespace module, because the generated base.py and base_team.py start with from dropbox import <namespace> for all 21 namespaces. Most of that is team_log (about 122k lines), which a typical app never touches.

This changes the generator to emit lazy references instead:

from dropbox._lazy_namespace import LazyNamespace

team_log = LazyNamespace("dropbox.team_log")

LazyNamespace imports the real module through importlib.import_module the first time an attribute is read, so route methods (which look up team_log.get_events etc. inside the method body) work unchanged, and concurrent first use is covered by the import lock. from dropbox import team_log still returns the real module.

Changes:

  • generator/dropbox_python_client.stoneg.py: override _generate_imports
  • generate_base_client.py: generate base_team with the same generator as base (it used stone's built-in python_client), so it gets lazy imports too. The only other difference in base_team.py is that the abstract request signature now includes extra_headers=None, matching base.py.
  • dropbox/base.py, dropbox/base_team.py: regenerated with python generate_base_client.py, ruff format dropbox, ruff check --fix dropbox (same steps as spec_update.yml). Before my change, that pipeline reproduced the committed files exactly, so the diff is only the import block.
  • scripts/benchmark_import.py: measures import dropbox in fresh interpreters, with requests and stone imported first

Numbers from python scripts/benchmark_import.py 21 (Apple Silicon arm64, Python 3.12, median of 21 runs; traced memory median of 3):

before after
import dropbox time 60.9 ms 23.4 ms
traced memory (tracemalloc) 22.5 MB 6.5 MB
RSS increase 30.8 MB 7.8 MB
dropbox.* modules loaded 30 24

team, team_log, account, check, contacts, openid and secondary_emails are no longer loaded at import. files, sharing, paper, riviera and file_requests still load at import because some route methods use their types as default argument values. The cost of a namespace isn't removed, just moved to the first call that needs it.

Tests (test/unit/test_lazy_namespace.py): a fresh interpreter doesn't load team/team_log on import dropbox; team_log_get_events loads it and passes the real route object and arg type to request; 16 threads hitting a cold namespace at once get the same module; proxy dir/repr/missing attribute/missing module behavior; a user route still builds real files types.

Checklist

General Contributing

  • Have you read the Code of Conduct and signed the CLA?

Is This a Code Change?

  • Non-code related change (markdown/git settings etc)
  • SDK Code Change
  • Example/Test Code Change

Validation

  • Does tox pass? (I ran tox's commands directly: ruff check dropbox example test, ruff format --check dropbox example test, and the unit tests)
  • Do the tests pass? (pytest test/unit/: 125 passed)

import dropbox used to import every namespace module, including team_log,
because base.py and base_team.py import all of them at module level. The
generator now emits LazyNamespace proxies that import a namespace on first
use, and base_team is generated by the same generator so it gets them too.

Closes dropbox#269
@CLAassistant

CLAassistant commented Nov 4, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@greg-db

greg-db commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for submitting this! I'll ask the team to review it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce memory footprint of SDK by separating namespace imports

3 participants