Skip to content

feat(shield-swap): keep the profile account in the account keystore - #91

Merged
iamalwaysuncomfortable merged 9 commits into
masterfrom
feat/profile-account-keystore
Nov 17, 2026
Merged

iamalwaysuncomfortable merged 9 commits into
masterfrom
feat/profile-account-keystore

Conversation

@iamalwaysuncomfortable

@iamalwaysuncomfortable iamalwaysuncomfortable commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Profile.load_or_create() kept its private key in profile.json. It now finds and stores the key through aleo.keystore (added in #90), so one account file (~/.aleo/accounts/<name>.json) serves the Python SDK, Veil scripts and the Shield Swap command line.

The companion change for the shield-swap command line is ProvableHQ/veil#222.

What changes

A new home looks the account up before it creates anything:

  1. ALEO_PRIVATE_KEY, then ALEO_PRIVATE_KEY_FILE (SHIELD_SWAP_PRIVATE_KEY and SHIELD_SWAP_PRIVATE_KEY_FILE are aliases)
  2. the account file, selected by account, then ALEO_ACCOUNT, then default
  3. when neither exists and create is true, a new account, stored in the keystore

profile.json then records address, account, network and endpoint. It holds no private key.

New arguments on load_or_create(), all keyword-only with defaults that keep today's call working:

Argument Effect
create=True Pass False to raise ACCOUNT_NOT_FOUND instead of creating an account.
storage="plaintext" How a newly created key is held: plaintext, encrypted or keychain. none is rejected.
account=None Account name in the keystore.
accounts_dir=None Accounts directory. Not recorded; pass it again on later runs.
password=None Password of an encrypted account.
save_password=False Also save a new encrypted account's password in the macOS Keychain.

An existing profile is returned as recorded. Its key is looked up on the first read of private_key, and the result must derive the recorded address.

Changed behaviour worth reviewing

  • An existing profile.json with a private_key keeps working with no action. The first key read logs one WARNING on aleo_shield_swap.profile pointing at python -m aleo.keystore migrate. Nothing removes that key.
  • A key supplied through the environment is used for the process and is no longer copied into profile.json. A setup that relied on the copy needs the variable on every run, or python -m aleo.keystore import once.
  • Two new homes that select the same account name now share one account. Before, each home generated its own key. Pass account= to keep them apart. When several processes create the same missing account at once, one key is saved and every profile uses it.
  • Homes that share an account on one network issue swap counters through one new file, ~/.aleo/shield-swap/counters/<network>/<address>.jsonl (Profile.counters_path), so their swaps never derive the same blinded address. It holds counter numbers and timestamps; each home keeps its own journal and claims its own swaps. A journal that issued counters before the ledger existed publishes them when it is next opened. The async client and other machines are not coordinated.
  • Profile.load_or_create() raises ValueError when the selected account file would be profile.json or credentials.json in the profile home. The check uses the name that selects the account, including the name saved in an existing profile.
  • A profile written by this release has no key, so an earlier release cannot use it.
  • Keystore failures surface as aleo.keystore.KeystoreError with a stable code.

Versions

Following #80 and #82, every package moves together, to 0.9.0. aleo-shield-swap now requires aleo-sdk>=0.9.0, the first release with aleo.keystore, and its import guard names that upgrade.

Testing

  • Shield Swap SDK offline suite: 633 passed, 8 skipped, 95 deselected. tests/test_profile.py grows to 89 tests.
  • Bridge package tests (test_package.py, test_client_lifecycle.py): 18 passed. Keystore tests in sdk/: 299 passed, 1 skipped.
  • codegen/gen_context.py --check: clean. The agent guide grew, so its size cap in tests/test_gen_context.py moves from 36,000 to 38,000 characters.
  • pyright on aleo_shield_swap: 15 errors, all in client.py, async_client.py and _quotes.py on lines this change does not touch. None in profile.py.
  • A new autouse fixture in tests/conftest.py gives every test an empty home directory, clears the key, account and password variables, and replaces the Keychain tool with an in-memory one. The live and devnode tiers keep the real home for the proving cache, with the keystore still redirected.
  • Cross-checked against the Veil change: accounts created by shield-swap account create (plaintext and encrypted) opened in a Profile, and accounts created by a Profile opened in the command line. Neither side's profile.json or state.json held a key.
  • Not exercised: the live and devnode tiers, the real macOS Keychain through a profile, and Windows locally (CI covers Windows).

Docs

  • shield-swap-sdk/README.md, both AGENTS.md copies and the first-swap example now describe the account file.

Profile.load_or_create() stored its private key in profile.json. It now finds
and stores the key through aleo.keystore, so one account file serves the
Python SDK, Veil scripts and the Shield Swap command line.

- A new home looks the account up first (the key variables, then the account
  file) and creates and stores one only when none is found and create is true.
- load_or_create() gains create, storage, account, accounts_dir, password and
  save_password.
- A new profile.json records the address and the account name and holds no
  key. An existing profile.json with a key keeps working and logs one line
  pointing at python -m aleo.keystore migrate.
- A key from the environment is used for the process and is no longer copied
  to disk.

All packages move to 0.9.0, and aleo-shield-swap now requires
aleo-sdk>=0.9.0, the first release with aleo.keystore.
…e guide

Docstrings and comments only; no code changes.

- Describe the current contract instead of release history: a profile whose
  profile.json holds a private key, not one "written by an earlier release".
- Explain each storage choice by what it costs on later runs.
- Move the errors of ShieldSwap.from_profile() into a Raises section, and
  regenerate the agent guide from it.
- Lead helper docstrings with a verb.

@iamalwaysuncomfortable iamalwaysuncomfortable left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two issues reproduced locally at 07539dc9. The offline suite passes: 606 passed, 8 skipped, 95 deselected.

home.mkdir(parents=True, exist_ok=True)
data = {"address": address, "account": name,
"network": network, "endpoint": endpoint}
_write_private(path, data)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reject account paths that overlap profile files

On an empty home, Profile.load_or_create(home, accounts_dir=home, account='profile') writes the account key to home/profile.json, then this write overwrites it with profile metadata. Creation succeeds because the key remains cached, but its only stored copy is gone; reloading with the same arguments raises INVALID_ACCOUNT_FILE. I reproduced this locally. Reject account paths overlapping the profile or credentials files before writing anything.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f024460. Profile.load_or_create() now raises ValueError before it reads or writes anything when the selected account file is profile.json or credentials.json in the profile home. The check covers the account and accounts_dir arguments, ALEO_ACCOUNT and ALEO_ACCOUNTS_DIR, and a symlinked spelling of the home. It also covers an existing account file named credentials.json, which save_credentials() would otherwise replace. Other account names in the profile home still work. Six tests added in tests/test_profile.py.

from aleo import keystore
net = getattr(aleo, network)
try:
loaded = keystore.load(name=account, dir=accounts_dir, password=password)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Coordinate swap counters across homes sharing an account

Separate homes now automatically reuse the default account but retain separate journals and reservation locks. On the same network, both clients can reserve counter 0 and derive the identical blinded address before either swap finalizes. I reproduced this using the real identity derivation, with only the chain probe replaced by an unused-address response: the two reservations returned counters [0, 0] and identical blinded addresses. The chain check cannot detect pending reservations, so one swap will be rejected for reusing the identity. Shared accounts need counter reservations coordinated across profile homes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6189747. ShieldSwap.from_profile() now opens the journal with a counter ledger at ~/.aleo/shield-swap/counters/<network>/<address>.jsonl (Profile.counters_path). Every journal opened with the same ledger reserves and skips counters through it under one lock, taken before the journal's own, so homes that share an account on one network never issue the same counter. The ledger holds counter numbers and timestamps only; swaps and blinding factors stay in each home's journal. A test builds two clients from two homes on one account with the chain probe reporting every address unused and gets counters [0, 1, 2].

Not covered: the async client, which probes the chain without a journal, and the same account trading from another machine or another tool. Those keep relying on the chain check, as before this PR.

… own files

With the accounts directory set to the profile home, the account names
"profile" and "credentials" select profile.json and credentials.json.
Creating the profile then replaced the account file that had just been
written, and saving credentials would replace a stored one.

Profile.load_or_create() now raises ValueError for that selection before
it reads or writes anything. Other account names may still live in the
profile home.
…count

Profile homes that select one account now share its key, but each home
kept its own journal and counter cursor. On one network, two homes could
both reserve counter 0 before either swap finalized and derive the same
blinded address; the chain check sees only finalized swaps, so one swap
was rejected.

ShieldSwap.from_profile() now opens the journal with a counter ledger at
~/.aleo/shield-swap/counters/<network>/<address>.jsonl. Every journal
opened with the same ledger reserves and skips counters through it under
one lock, so homes that share an account never issue the same counter.
The ledger holds counter numbers and timestamps only, in a directory
readable by its owner. Swaps and blinding factors stay in each home's
journal, and Journal(path) without a ledger behaves as before.

@iamalwaysuncomfortable iamalwaysuncomfortable left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 6189747: both original reproductions now pass, but the two follow-up gaps below remain reproducible. Offline suite: 622 passed, 8 skipped, 95 deselected.

# directory named ~ in the cwd and writing the private key there.
# Treat ~ as the user's home, including paths passed directly from Python.
home = Path(home).expanduser() if home is not None else cls.default_home()
_reject_account_in_profile_files(home, account, accounts_dir)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Check the profile's saved account name for collisions

The guard runs before reading the existing profile, so it ignores the saved account name. I reproduced this by creating a profile for account credentials, moving its account file into the profile home, and reopening with accounts_dir=home without repeating account=. Loading the private key succeeds, but save_credentials(jwt=...) modifies the account file and subsequent unlocking raises INVALID_ACCOUNT_FILE. Apply the collision check to the actual selected name, including the profile's saved name.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 23995b7. For an existing profile, Profile.load_or_create() now applies the check to the name saved in profile.json when neither the account argument nor ALEO_ACCOUNT selects another one, which is the name the key is then read under. The reproduction (a profile saved for credentials, its account file moved into the home, reopened with accounts_dir=home) raises ValueError and leaves the account file unchanged. An account named credentials in an accounts directory outside the home still works, and ALEO_ACCOUNT still takes precedence over the saved name. Three tests added.

def __init__(self, path: "Path | str", *, counters: "Path | str | None" = None) -> None:
self.path = Path(path)
self._lock_path = self.path.with_suffix(".lock")
self._ledger = Journal(counters) if counters is not None else None

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Seed the shared ledger from existing journal reservations

Attaching an existing journal does not publish its previously reserved counters to the shared ledger. I reproduced an older journal containing pending counters [0, 1], reopened that profile through from_profile(), then opened a second home on the same account. The second home reserved counter 0 and derived the same blinded identity. The chain probe cannot catch the collision until the earlier swap finalizes. Synchronize existing reservations into the shared ledger under its lock when attaching the journal.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a7645a7. Opening a journal with a ledger now compares the journal's own cursor with the ledger's and, when the ledger is behind, appends one counters_skipped event to the ledger under its lock. The reproduction (a journal holding pending counters [0, 1], reopened through from_profile(), then a second home on the same account) now reserves counter 2. The same step runs after record_swap() and record_swap_failed(), because a swap on a caller-supplied identity records a counter the ledger did not issue. A journal the ledger already covers writes nothing, and a new journal does not create the ledger. Tests cover the journal and the two-client case.

One limit remains: the earlier home has to be opened once after upgrading. Until then its pending counters exist only in its own journal, which no other home can read.

…ledger

A journal that reserved counters before it was opened with a counter
ledger kept them to itself. Their swaps may not have finalized, so a
second home on the same account could reserve the same counter and
derive the same blinded address.

Opening a journal with a ledger now retires the journal's own counters
in the ledger when the ledger is behind, under the ledger's lock. The
same happens after a swap is recorded on an identity the caller chose,
which the ledger did not issue. A journal the ledger already covers
writes nothing.
…files

The check for an account file that is profile.json or credentials.json
ran before the existing profile was read, so it missed the name saved in
the profile. A profile saved for the account "credentials" and reopened
with the accounts directory set to its home loaded the key, and
save_credentials() then rewrote the account file.

Profile.load_or_create() now applies the check to the saved name when
neither the account argument nor ALEO_ACCOUNT selects another one.

@iamalwaysuncomfortable iamalwaysuncomfortable left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 23995b7. The saved-account-name collision and existing-journal counter reproductions from the previous review now pass. Two additional issues are reproduced below. Validation against an isolated copy of this commit: 630 passed, 8 skipped, 95 deselected; normal shared-ledger reservations also produced 80 distinct counters across four processes.

'after this process exits. Choose "plaintext", "keychain" or "encrypted".',
) from None
private_key = str(net.PrivateKey.random())
stored = keystore.save(private_key, name=account, dir=accounts_dir,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Adopt the account created by a concurrent startup

When two processes start different fresh profile homes with the same missing account name, both can finish keystore.load() with ACCOUNT_NOT_FOUND before either saves. Each then generates a different key, so one keystore.save() succeeds and the other raises ACCOUNT_EXISTS; that exception escapes and the second profile is never created. I reproduced this by synchronizing the two missing-account lookups: the outcomes were OK and ACCOUNT_EXISTS, while retrying afterward correctly adopted the winning account. This now affects independent homes because they share the default account. Serialize the create-or-load operation, or reload the winning account when creation loses the race.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the commit "adopt the account a concurrent startup stored". When keystore.save() fails with ACCOUNT_EXISTS after the lookup found no account, Profile.load_or_create() now loads the account the other process stored and builds the profile on it. The key generated by the losing process was never written. If the stored account cannot be opened, for example it is encrypted and no password is supplied, that error is raised and no profile is created. Two tests simulate the race by storing an account between the lookup and the save.

yield
else:
self._ledger.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
with self._ledger._locked(), self._locked():

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reject journal and ledger paths that share a lock file

The new counters argument accepts distinct data files whose lock paths collide. For example, Journal(home / "journal.jsonl", counters=home / "journal.counters").reserve_counters(1) hangs: both instances derive home/journal.lock with with_suffix(".lock"), and the second exclusive flock waits forever on the lock already held through the first file handle. I reproduced the hang in a child process and confirmed both acquisitions targeted the same inode; using counters.jsonl instead succeeds. Validate the resolved lock paths or acquire a shared lock only once so this configuration cannot deadlock.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the commit "take one lock when a journal and its ledger share a lock file". A counter change now opens both lock files, compares them with os.path.sameopenfile, and takes a single lock when they are one file, so the comparison also covers symlinks and hard links. The reproduction (journal.jsonl with counters=journal.counters) reserves and skips counters without waiting; the test runs it in a thread with a timeout.

… lock file

A journal and its counter ledger each lock the file their name gives
once the suffix is replaced with .lock, so journal.jsonl and
journal.counters in one directory lock the same file. Reserving a
counter took that lock twice through two handles and waited forever.

A counter change now compares the two open lock files and takes a single
lock when they are the same file.
Two processes that opened new profile homes for the same missing account
could both find no account and both generate a key. The second save then
failed with ACCOUNT_EXISTS and its profile was not created.

Profile.load_or_create() now loads the stored account when its own save
loses that race. The key it generated was never written.
@iamalwaysuncomfortable
iamalwaysuncomfortable merged commit 156df8e into master Nov 17, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant