Repository navigation
feat(shield-swap): keep the profile account in the account keystore - #91
Conversation
Profile.load_or_create() stored its private key in profile.json. It now finds and stores the key through aleo.keystore, so one account file serves the Python SDK, Veil scripts and the Shield Swap command line. - A new home looks the account up first (the key variables, then the account file) and creates and stores one only when none is found and create is true. - load_or_create() gains create, storage, account, accounts_dir, password and save_password. - A new profile.json records the address and the account name and holds no key. An existing profile.json with a key keeps working and logs one line pointing at python -m aleo.keystore migrate. - A key from the environment is used for the process and is no longer copied to disk. All packages move to 0.9.0, and aleo-shield-swap now requires aleo-sdk>=0.9.0, the first release with aleo.keystore.
…e guide Docstrings and comments only; no code changes. - Describe the current contract instead of release history: a profile whose profile.json holds a private key, not one "written by an earlier release". - Explain each storage choice by what it costs on later runs. - Move the errors of ShieldSwap.from_profile() into a Raises section, and regenerate the agent guide from it. - Lead helper docstrings with a verb.
iamalwaysuncomfortable
left a comment
There was a problem hiding this comment.
Two issues reproduced locally at 07539dc9. The offline suite passes: 606 passed, 8 skipped, 95 deselected.
| home.mkdir(parents=True, exist_ok=True) | ||
| data = {"address": address, "account": name, | ||
| "network": network, "endpoint": endpoint} | ||
| _write_private(path, data) |
There was a problem hiding this comment.
[P2] Reject account paths that overlap profile files
On an empty home, Profile.load_or_create(home, accounts_dir=home, account='profile') writes the account key to home/profile.json, then this write overwrites it with profile metadata. Creation succeeds because the key remains cached, but its only stored copy is gone; reloading with the same arguments raises INVALID_ACCOUNT_FILE. I reproduced this locally. Reject account paths overlapping the profile or credentials files before writing anything.
There was a problem hiding this comment.
Fixed in f024460. Profile.load_or_create() now raises ValueError before it reads or writes anything when the selected account file is profile.json or credentials.json in the profile home. The check covers the account and accounts_dir arguments, ALEO_ACCOUNT and ALEO_ACCOUNTS_DIR, and a symlinked spelling of the home. It also covers an existing account file named credentials.json, which save_credentials() would otherwise replace. Other account names in the profile home still work. Six tests added in tests/test_profile.py.
| from aleo import keystore | ||
| net = getattr(aleo, network) | ||
| try: | ||
| loaded = keystore.load(name=account, dir=accounts_dir, password=password) |
There was a problem hiding this comment.
[P2] Coordinate swap counters across homes sharing an account
Separate homes now automatically reuse the default account but retain separate journals and reservation locks. On the same network, both clients can reserve counter 0 and derive the identical blinded address before either swap finalizes. I reproduced this using the real identity derivation, with only the chain probe replaced by an unused-address response: the two reservations returned counters [0, 0] and identical blinded addresses. The chain check cannot detect pending reservations, so one swap will be rejected for reusing the identity. Shared accounts need counter reservations coordinated across profile homes.
There was a problem hiding this comment.
Fixed in 6189747. ShieldSwap.from_profile() now opens the journal with a counter ledger at ~/.aleo/shield-swap/counters/<network>/<address>.jsonl (Profile.counters_path). Every journal opened with the same ledger reserves and skips counters through it under one lock, taken before the journal's own, so homes that share an account on one network never issue the same counter. The ledger holds counter numbers and timestamps only; swaps and blinding factors stay in each home's journal. A test builds two clients from two homes on one account with the chain probe reporting every address unused and gets counters [0, 1, 2].
Not covered: the async client, which probes the chain without a journal, and the same account trading from another machine or another tool. Those keep relying on the chain check, as before this PR.
… own files With the accounts directory set to the profile home, the account names "profile" and "credentials" select profile.json and credentials.json. Creating the profile then replaced the account file that had just been written, and saving credentials would replace a stored one. Profile.load_or_create() now raises ValueError for that selection before it reads or writes anything. Other account names may still live in the profile home.
…count Profile homes that select one account now share its key, but each home kept its own journal and counter cursor. On one network, two homes could both reserve counter 0 before either swap finalized and derive the same blinded address; the chain check sees only finalized swaps, so one swap was rejected. ShieldSwap.from_profile() now opens the journal with a counter ledger at ~/.aleo/shield-swap/counters/<network>/<address>.jsonl. Every journal opened with the same ledger reserves and skips counters through it under one lock, so homes that share an account never issue the same counter. The ledger holds counter numbers and timestamps only, in a directory readable by its owner. Swaps and blinding factors stay in each home's journal, and Journal(path) without a ledger behaves as before.
iamalwaysuncomfortable
left a comment
There was a problem hiding this comment.
Re-reviewed 6189747: both original reproductions now pass, but the two follow-up gaps below remain reproducible. Offline suite: 622 passed, 8 skipped, 95 deselected.
| # directory named ~ in the cwd and writing the private key there. | ||
| # Treat ~ as the user's home, including paths passed directly from Python. | ||
| home = Path(home).expanduser() if home is not None else cls.default_home() | ||
| _reject_account_in_profile_files(home, account, accounts_dir) |
There was a problem hiding this comment.
[P2] Check the profile's saved account name for collisions
The guard runs before reading the existing profile, so it ignores the saved account name. I reproduced this by creating a profile for account credentials, moving its account file into the profile home, and reopening with accounts_dir=home without repeating account=. Loading the private key succeeds, but save_credentials(jwt=...) modifies the account file and subsequent unlocking raises INVALID_ACCOUNT_FILE. Apply the collision check to the actual selected name, including the profile's saved name.
There was a problem hiding this comment.
Fixed in 23995b7. For an existing profile, Profile.load_or_create() now applies the check to the name saved in profile.json when neither the account argument nor ALEO_ACCOUNT selects another one, which is the name the key is then read under. The reproduction (a profile saved for credentials, its account file moved into the home, reopened with accounts_dir=home) raises ValueError and leaves the account file unchanged. An account named credentials in an accounts directory outside the home still works, and ALEO_ACCOUNT still takes precedence over the saved name. Three tests added.
| def __init__(self, path: "Path | str", *, counters: "Path | str | None" = None) -> None: | ||
| self.path = Path(path) | ||
| self._lock_path = self.path.with_suffix(".lock") | ||
| self._ledger = Journal(counters) if counters is not None else None |
There was a problem hiding this comment.
[P2] Seed the shared ledger from existing journal reservations
Attaching an existing journal does not publish its previously reserved counters to the shared ledger. I reproduced an older journal containing pending counters [0, 1], reopened that profile through from_profile(), then opened a second home on the same account. The second home reserved counter 0 and derived the same blinded identity. The chain probe cannot catch the collision until the earlier swap finalizes. Synchronize existing reservations into the shared ledger under its lock when attaching the journal.
There was a problem hiding this comment.
Fixed in a7645a7. Opening a journal with a ledger now compares the journal's own cursor with the ledger's and, when the ledger is behind, appends one counters_skipped event to the ledger under its lock. The reproduction (a journal holding pending counters [0, 1], reopened through from_profile(), then a second home on the same account) now reserves counter 2. The same step runs after record_swap() and record_swap_failed(), because a swap on a caller-supplied identity records a counter the ledger did not issue. A journal the ledger already covers writes nothing, and a new journal does not create the ledger. Tests cover the journal and the two-client case.
One limit remains: the earlier home has to be opened once after upgrading. Until then its pending counters exist only in its own journal, which no other home can read.
…ledger A journal that reserved counters before it was opened with a counter ledger kept them to itself. Their swaps may not have finalized, so a second home on the same account could reserve the same counter and derive the same blinded address. Opening a journal with a ledger now retires the journal's own counters in the ledger when the ledger is behind, under the ledger's lock. The same happens after a swap is recorded on an identity the caller chose, which the ledger did not issue. A journal the ledger already covers writes nothing.
…files The check for an account file that is profile.json or credentials.json ran before the existing profile was read, so it missed the name saved in the profile. A profile saved for the account "credentials" and reopened with the accounts directory set to its home loaded the key, and save_credentials() then rewrote the account file. Profile.load_or_create() now applies the check to the saved name when neither the account argument nor ALEO_ACCOUNT selects another one.
iamalwaysuncomfortable
left a comment
There was a problem hiding this comment.
Re-reviewed 23995b7. The saved-account-name collision and existing-journal counter reproductions from the previous review now pass. Two additional issues are reproduced below. Validation against an isolated copy of this commit: 630 passed, 8 skipped, 95 deselected; normal shared-ledger reservations also produced 80 distinct counters across four processes.
| 'after this process exits. Choose "plaintext", "keychain" or "encrypted".', | ||
| ) from None | ||
| private_key = str(net.PrivateKey.random()) | ||
| stored = keystore.save(private_key, name=account, dir=accounts_dir, |
There was a problem hiding this comment.
[P2] Adopt the account created by a concurrent startup
When two processes start different fresh profile homes with the same missing account name, both can finish keystore.load() with ACCOUNT_NOT_FOUND before either saves. Each then generates a different key, so one keystore.save() succeeds and the other raises ACCOUNT_EXISTS; that exception escapes and the second profile is never created. I reproduced this by synchronizing the two missing-account lookups: the outcomes were OK and ACCOUNT_EXISTS, while retrying afterward correctly adopted the winning account. This now affects independent homes because they share the default account. Serialize the create-or-load operation, or reload the winning account when creation loses the race.
There was a problem hiding this comment.
Fixed in the commit "adopt the account a concurrent startup stored". When keystore.save() fails with ACCOUNT_EXISTS after the lookup found no account, Profile.load_or_create() now loads the account the other process stored and builds the profile on it. The key generated by the losing process was never written. If the stored account cannot be opened, for example it is encrypted and no password is supplied, that error is raised and no profile is created. Two tests simulate the race by storing an account between the lookup and the save.
| yield | ||
| else: | ||
| self._ledger.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) | ||
| with self._ledger._locked(), self._locked(): |
There was a problem hiding this comment.
[P2] Reject journal and ledger paths that share a lock file
The new counters argument accepts distinct data files whose lock paths collide. For example, Journal(home / "journal.jsonl", counters=home / "journal.counters").reserve_counters(1) hangs: both instances derive home/journal.lock with with_suffix(".lock"), and the second exclusive flock waits forever on the lock already held through the first file handle. I reproduced the hang in a child process and confirmed both acquisitions targeted the same inode; using counters.jsonl instead succeeds. Validate the resolved lock paths or acquire a shared lock only once so this configuration cannot deadlock.
There was a problem hiding this comment.
Fixed in the commit "take one lock when a journal and its ledger share a lock file". A counter change now opens both lock files, compares them with os.path.sameopenfile, and takes a single lock when they are one file, so the comparison also covers symlinks and hard links. The reproduction (journal.jsonl with counters=journal.counters) reserves and skips counters without waiting; the test runs it in a thread with a timeout.
… lock file A journal and its counter ledger each lock the file their name gives once the suffix is replaced with .lock, so journal.jsonl and journal.counters in one directory lock the same file. Reserving a counter took that lock twice through two handles and waited forever. A counter change now compares the two open lock files and takes a single lock when they are the same file.
Two processes that opened new profile homes for the same missing account could both find no account and both generate a key. The second save then failed with ACCOUNT_EXISTS and its profile was not created. Profile.load_or_create() now loads the stored account when its own save loses that race. The key it generated was never written.
Summary
Profile.load_or_create()kept its private key inprofile.json. It now finds and stores the key throughaleo.keystore(added in #90), so one account file (~/.aleo/accounts/<name>.json) serves the Python SDK, Veil scripts and the Shield Swap command line.The companion change for the
shield-swapcommand line is ProvableHQ/veil#222.What changes
A new home looks the account up before it creates anything:
ALEO_PRIVATE_KEY, thenALEO_PRIVATE_KEY_FILE(SHIELD_SWAP_PRIVATE_KEYandSHIELD_SWAP_PRIVATE_KEY_FILEare aliases)account, thenALEO_ACCOUNT, thendefaultcreateis true, a new account, stored in the keystoreprofile.jsonthen recordsaddress,account,networkandendpoint. It holds no private key.New arguments on
load_or_create(), all keyword-only with defaults that keep today's call working:create=TrueFalseto raiseACCOUNT_NOT_FOUNDinstead of creating an account.storage="plaintext"plaintext,encryptedorkeychain.noneis rejected.account=Noneaccounts_dir=Nonepassword=Nonesave_password=FalseAn existing profile is returned as recorded. Its key is looked up on the first read of
private_key, and the result must derive the recorded address.Changed behaviour worth reviewing
profile.jsonwith aprivate_keykeeps working with no action. The first key read logs one WARNING onaleo_shield_swap.profilepointing atpython -m aleo.keystore migrate. Nothing removes that key.profile.json. A setup that relied on the copy needs the variable on every run, orpython -m aleo.keystore importonce.account=to keep them apart. When several processes create the same missing account at once, one key is saved and every profile uses it.~/.aleo/shield-swap/counters/<network>/<address>.jsonl(Profile.counters_path), so their swaps never derive the same blinded address. It holds counter numbers and timestamps; each home keeps its own journal and claims its own swaps. A journal that issued counters before the ledger existed publishes them when it is next opened. The async client and other machines are not coordinated.Profile.load_or_create()raisesValueErrorwhen the selected account file would beprofile.jsonorcredentials.jsonin the profile home. The check uses the name that selects the account, including the name saved in an existing profile.aleo.keystore.KeystoreErrorwith a stablecode.Versions
Following #80 and #82, every package moves together, to 0.9.0.
aleo-shield-swapnow requiresaleo-sdk>=0.9.0, the first release withaleo.keystore, and its import guard names that upgrade.Testing
tests/test_profile.pygrows to 89 tests.test_package.py,test_client_lifecycle.py): 18 passed. Keystore tests insdk/: 299 passed, 1 skipped.codegen/gen_context.py --check: clean. The agent guide grew, so its size cap intests/test_gen_context.pymoves from 36,000 to 38,000 characters.pyrightonaleo_shield_swap: 15 errors, all inclient.py,async_client.pyand_quotes.pyon lines this change does not touch. None inprofile.py.tests/conftest.pygives every test an empty home directory, clears the key, account and password variables, and replaces the Keychain tool with an in-memory one. The live and devnode tiers keep the real home for the proving cache, with the keystore still redirected.shield-swap account create(plaintext and encrypted) opened in aProfile, and accounts created by aProfileopened in the command line. Neither side'sprofile.jsonorstate.jsonheld a key.Docs
shield-swap-sdk/README.md, bothAGENTS.mdcopies and the first-swap example now describe the account file.