ref PROD-244
Instead of generating one batch with all recipients, we now will generate
one batch per 1000 members and distribute the recipients across them.
ref PROD-233
Errors were not handled properly because of a missing rollback and try/catch.
Using a function is easier generally.
Also added ignored contraint checks to increase performance a tiny bit.
They ended up not mattering much, so we can consider to remove them again.
refs https://github.com/TryGhost/DevOps/issues/119
- members_newsletters needs members_subscribe_events, but it was also
then generating `subscriptions` records and the whole thing was really
slow
- for now, subscriptions is not a used table so we can remove use of it
- also adds support for generating more than one subscription record
with an 80% chance of being subscribed
refs https://github.com/TryGhost/DevOps/issues/119
- this function can simply call the `import` function, which performs
the same code as we had here
- this makes the code cleaner to read and understand
refs https://github.com/TryGhost/DevOps/issues/119
- this switches away from using a static list of names in favor of ones
generated by faker, so we don't run into duplicate names
- also minor code re-arranging
fixes https://github.com/TryGhost/Product/issues/3738https://www.notion.so/ghost/Member-Session-Invalidation-13254316f2244c34bcbc65c101eb5cc4
- Adds the transient_id column to the members table. This defaults to
email, to keep it backwards compatible (not logging out all existing
sessions)
- Instead of using the email in the cookies, we now use the transient_id
- Updating the transient_id means invalidating all sessions of a member
- Adds an endpoint to the admin api to log out a member from all devices
- Added the `all` body property to the DELETE session endpoint in the
members API. Setting it to true will sign a member out from all devices.
- Adds a UI button in Admin to sign a member out from all devices
- Portal 'sign out of all devices' will not be added for now
Related changes (added because these areas were affected by the code
changes):
- Adds a serializer to member events / activity feed endpoints - all
member fields were returned here, so the transient_id would also be
returned - which is not needed and bloats the API response size
(`transient_id` is not a secret because the cookies are signed)
- Removed `loadMemberSession` from public settings browse (not used
anymore + bad pattern)
Performance tests on site with 50.000 members (on Macbook M1 Pro):
- Migrate: 6s (adding column 4s, setting to email is 1s, dropping
nullable: 1s)
- Rollback: 2s
refs: https://github.com/TryGhost/DevOps/issues/11
This is a pretty huge commit, but the relevant points are:
* Each importer no longer needs to be passed a set of data, it just gets the data it needs
* Each importer specifies its dependencies, so that the order of import can be determined at runtime using a topological sort
* The main data generator function can just tell each importer to import the data it has
This makes working on the data generator much easier.
Some other benefits are:
* Batched importing, massively speeding up the whole process
* `--tables` to set the exact tables you want to import, and specify the quantity of each
As discussed with the product team we want to enforce kebab-case file names for
all files, with the exception of files which export a single class, in which
case they should be PascalCase and reflect the class which they export.
This will help find classes faster, and should push better naming for them too.
Some files and packages have been excluded from this linting, specifically when
a library or framework depends on the naming of a file for the functionality
e.g. Ember, knex-migrator, adapter-manager
no issue
Before, when base data included labels for members, the random generated labels would also be generated. This prevents that, and ensures that the base-data labels are applied correctly to members.
no issue
Previously the number of opened emails was being generated incorrectly as the number of delivered emails was being reported too high.
Also, the faker date function occasionally fails for dates which are
too close together so this switches to manually generating a date
between the two.
no issue
This should massively increase the speed of importing for the large dataset, which is important as the time to import it on Pro is >10 minutes at the moment
