at://bnewbold.net/com.whtwnd.blog.entry/3l5ii332pf32u

Back to Collection

Record JSON

{
  "$type": "com.whtwnd.blog.entry",
  "content": "\nAccount migration is an important feature of atproto, and has been possible in the live network since February of this year, but is still a bit scary and developer-oriented.\n\nAs part of documenting the migration process, I recently implemented basic account migration in the `goat` CLI tool for atproto. This blog post walks through both the \"automatic\" and \"manual\" variants of account migration, and then describes ways this process could be made even more safe and accessible.\n\nThe basic account migration steps are:\n\n1. Establish \"Deactivated\" Account on new PDS\n2. Migrate Content: Repository, Blobs, Preferences\n3. Update Identity\n4. Swap Active/Deactivated Accounts\n\nFor a longer description of atproto account migration, including a helpful diagram, see [the original write-up](https://github.com/bluesky-social/pds/blob/main/ACCOUNT_MIGRATION.md) by [@dholms.xyz](https://bsky.app/profile/dholms.xyz).\n\nYou can read more about `goat` in the [indigo repo](https://github.com/bluesky-social/indigo/tree/main/cmd/goat).\n\n## Independent PDS Hosting and Migration\n\nAs some background context, the Bluesky atproto services have been fully \"federated\" since early 2024. This feature is no longer \"beta\" and there are no significant restrictions on PDS instance size. There are abuse-prevention rate-limits on how fast accounts and content will be ingested from new PDS by the Bluesky Relay, but we are happy to raise them on request. There is no prior registraiton required, and there is no \"10 account limit\".\n\nOne restriction that is still in place is that accounts can not fully migrate inbound to the Bluesky PDS hosts. The identity aspect of migration might work for \"returning\" accounts, but the `importRepo` endpoint is not allowed. This is purely an operational issue where PDS worker processes currently \"hang\" while processing large imports, which impact other users. This isn't a big concern for smaller PDS instances, but for large instances with hundreds of thousands of accounts, and very large account imports (millions of records) it can cause significant service disruptions. Inbound account migration will be allowed once this operational issue is mitigated.\n\n## Automatic Account Migration\n\nIf the old PDS is online and the old account is active, and the DID is a `did:plc`, then migration is quite simple, with `goat` facilitating most steps with a single command.\n\nAhead of time, you should collect the following information:\n\n- your existing account credentials (`$ACCOUNTDID`, `$OLDHANDLE`, `$OLDPASSWORD`)\n- the new PDS host and service DID (`$NEWPDSHOST`, `$NEWPDSSERVICEDID`)\n- invite code for the new PDS (`$INVITECODE`)\n- desired new PDS handle and credentials (`$NEWHANDLE`, `$NEWEMAIL`, `$NEWPASSWORD`)\n\nThe current flow assumes that you get a new \"local\" handle on the new PDS. If you have a custom domain handle, you can update to that after the rest of the account migration is complete.\n\nLog in to your existing account using `goat`, using a full password (not app password):\n\n```bash\ngoat account login -u $OLDHANDLE -p $OLDPASSWORD\n```\n\nIdentity updates are a sensitive operation requiring a 2FA email token, so you need to request that token:\n\n```bash\ngoat account plc request-token\n```\n\nThe code (`$PLCTOKEN`) will arrive via email.\n\nThen you can run the all-in-one command:\n\n```bash\ngoat account migrate \\\n    --pds-host $NEWPDSHOST \\\n    --new-handle $NEWHANDLE \\\n    --new-password $NEWPASSWORD \\\n    --new-email $NEWEMAIL \\\n    --plc-token $NEWPLCTOKEN \\\n    --invite-code $INVITECODE\n```\n\nIf all goes well, the new account will be created, content will be migrated, and identity and account status will be synchronized. Log lines will indicate progress. You'll want to log out of all clients (including `goat`) and log back in.\n\nIf something goes wrong at the account creation step, it may be possible to re-run the entire command. If something fails after that, it may be necessary to complete the process \"manually\", using the steps below.\n\n## Manual Account Migration\n\nIn some situations the automatic process might not be possible or desirable. These more manual commands give more control over the process, work with different DID methods, give more direct control over the final DID document, result in local backups of all public data (and private preferences), and may allow recovery if the original PDS is down or uncooperative (assuming self-control of the DID).\n\nThese steps require flipping back and forth between `goat` logged in to the old PDS and the new PDS. Instead of showing those `goat account login` / `logout` commands, these directions will indicate which PDS should be authenticated at each step.\n\nYou can fetch metadata about a PDS, including the service DID and supported handle suffix:\n\n```bash\n# no auth required\ngoat pds describe $NEWPDSHOST\n\n# for example\ngoat pds describe https://bsky.social\n{\n  \"availableUserDomains\": [\n    \".bsky.social\"\n  ],\n  \"did\": \"did:web:bsky.social\",\n  \"inviteCodeRequired\": false,\n  \"links\": {\n    \"privacyPolicy\": \"https://blueskyweb.xyz/support/privacy-policy\",\n    \"termsOfService\": \"https://blueskyweb.xyz/support/tos\"\n  },\n  \"phoneVerificationRequired\": true\n}\n```\n\nTo create an account with an existing DID on the new PDS, we first need to generate a service auth token:\n\n```bash\n# old PDS\ngoat account service-auth --lxm com.atproto.server.createAccount --aud $NEWPDSSERVICEDID --duration-sec 3600\n```\n\nThis returns a large base64-encoded token (`$SERVICEAUTH`).\n\nNow an account can be created on the new PDS, using the existing DID:\n\n```bash\n# no auth\ngoat account create \\\n    --pds-host $NEWPDSHOST \\\n    --existing-did $ACCOUNTDID \\\n    --handle $NEWHANDLE \\\n    --password $NEWPASSWORD \\\n    --email $NEWEMAIL \\\n    --invite-code $INVITECODE \\\n    --service-auth $SERVICEAUTH\n```\n\nThe new account will be \"deactivated\", because the identity (DID) does not point to this PDS host yet. To log in to an account when the DID doesn't resolve yet, `goat` requires specifying the PDS host:\n\n```bash\ngoat account login --pds-host $NEWPDSHOST -u $ACCOUNTDID -p $NEWPASSWORD\n```\n\nYou can check the current account status like:\n\n```bash\n# new PDS\ngoat account status\n{\n    \"activated\": false,\n    \"expectedBlobs\": 0,\n    \"importedBlobs\": 0,\n    \"indexedRecords\": 0,\n    \"privateStateValues\": 0,\n    \"repoBlocks\": 2,\n    \"repoCommit\": \"bafyreie2o6idkbnpkhkwp6ocf7p5k7np2t7xnx3346zqc456f3avhsnhue\",\n    \"repoRev\": \"3l5ddasaitk23\",\n    \"validDid\": false\n}\n```\n\nNext to migrate content, starting with repo:\n\n```bash\n# old PDS\ngoat repo export $ACCOUNTDID\n\n# will write a CAR file like ./account.20240929112355.car\n\n# new PDS\ngoat repo import ./account.20240929112355.car\n```\n\nOnce all the old records are indexed, the new PDS will know how many blobs are expected (`expectedBlobs` in account status), and how many have been imported (`importedBlobs`). You can also check the specific \"missing\" blobs:\n\n```bash\n# new PDS\ngoat account missing-blobs\n\nbafkreibyu5mlurlwyjj2ewfjddmm7euiq47xisdyf4sil46s2zu4bultiu\tat://did:plc:c7ilkj3gs7mdo3d6vdbebgk2/app.bsky.actor.profile/self\nbafkreieymnbzgpcjdebyjewy3z7jmpqg6h3uf5fl4khuywz65tgmknvlgu\tat://did:plc:c7ilkj3gs7mdo3d6vdbebgk2/app.bsky.feed.post/3l5cs7sszcx2s\n[...]\n```\n\nTo export and import all blobs:\n\n```bash\n# old PDS\ngoat blob export $ACCOUNTDID\n\n# will create a directory like ./account_blobs/\n\n# new PDS\n# this requires the 'fd' (fd-find) and 'parallel' commands\nfd . ./account_blobs/ | parallel -j1 goat blob upload {}\n```\n\nYou can confirm that there are no missing blobs, and that the blob and record counts match the old PDS.\n\nNext, private Bluesky app preferences.\n\nAs a warning, the current Go code for serializing/deserializing preferences may be \"lossy\" if the preference schemas are out of sync or for non-Bluesky Lexicons, and it is possible this step will lose some preference metadata. This will hopefully be improved in a future version of `goat`, or when the preferences API is updated to be app-agnostic (\"personal data\" protocol support).\n\n```bash\n# old PDS\ngoat bsky prefs export \u003e prefs.json\n\n# new PDS\ngoat bsky prefs import prefs.json\n```\n\nWith all the content migrated to the new account, we can update the identity (DID) to point at the new PDS instance.\n\nFetch the \"recommended\" DID parameters from the new PDS:\n\n```shell\n# new PDS\ngoat account plc recommended \u003e plc_recommended.json\n```\n\nIf you are self-managing your identity (eg, `did:web` or self-controlled `did:plc`), you can merge these parameters in to your DID document.\n\nIf using a PDS-managed `did:plc`, you can edit the parameters to match any additional services or recovery keys. Save the results as `./plc_unsigned.json`. You will need to request a PLC signing token from the PDS:\n\n```shell\n# old PDS\ngoat account plc request-token\n```\n\nRetrieve the token (`$PLCTOKEN`) from email, then request a signed version of the PLC params:\n\n```shell\n# old PDS\ngoat account plc sign --token $PLCTOKEN  ./plc_unsigned.json \u003e plc_signed.json\n```\n\nIf that looks good, the PLC Op can be submitted from the new PDS:\n\n```shell\n# new PDS\ngoat account plc submit ./plc_signed.json\n```\n\nCheck the account status on the new PDS, and `validDid` should now be `true`.\n\nAs the final steps, the new PDS account can be activated:\n\n```shell\n# new PDS\ngoat account activate\n```\n\nand the old PDS account deactivated:\n\n```shell\n# old PDS\ngoat account deactivate\n```\n\nYou may chose to delete the old account once you are confident the new account is configured and running as expected.\n\n\n## Future Work\n\nThe repo import and blob migration steps could show progress bars, especially for accounts with a lot of content.\n\nThe auto-migrate tool could detect the current status better, and \"continue where it left off\" in some situations.\n\nGetting account migration integrated in to the Bluesky app itself is the ultimate goal, though having better support in external tooling is still a big step forward. An independent app (mobile, desktop, or server side) could help maintain backups of repo content, blobs, and identity, and facilitate account migration (or recovery) in cases where the original PDS is not available.\n\nIt would be good to have better tooling for PLC recovery keys. If `goat` was involved in that, it could help manage those extra keys during the migration process, or sign and submit PLC operations directly, instead of via PDS instances.",
  "createdAt": "2024-10-02T17:35:20.466Z",
  "theme": "github-light",
  "title": "Migrating PDS Account with `goat`",
  "visibility": "public"
}