Commit Graph

5 Commits

Author SHA1 Message Date
Simon Backx
285a684ef6
Updated data generator to support >2M members (#19484)
no issue

The data generator went out of memory when trying to generate fake data
for > 2M members. This adds some improvements to make sure it doesn't go
out of memory.

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2024-01-15 15:23:49 +00:00
Simon Backx
d2cb23c3fa
Wired up Docker setup script and increased data generation performance (#19420)
ref PROD-233

- Stored whether Docker is used in the config files
- When running `yarn setup`, any existing Docker container will be
reset. Run with `-y` to skip the confirmation step.
- `yarn setup` will always init the database and generate fake data
- Increased amount of default generated data to 100,000 members + 500
posts.
- Made lots of performance improvements in the data generator so we can
generate the default data in ±170s
2024-01-05 13:42:30 +00:00
Simon Backx
70b991cc1c Improved subscriptions in data generator
ref PROD-244

- Added support for canceled subscriptions and different subscription statusses
- Removed generation of susbcriptions table (not used)
- Added old canceled subscriptions for free members
- Added both positive and negative MRR events
2023-12-14 18:30:45 +01:00
Simon Backx
e1ef7c44d8 Fixed creating stripe subscriptions for complimentary members in data generator
ref PROD-244
2023-12-14 18:30:45 +01:00
Sam Lord
4ff467794f Entirely rewrote data generator to simplify codebase
refs: https://github.com/TryGhost/DevOps/issues/11

This is a pretty huge commit, but the relevant points are:
* Each importer no longer needs to be passed a set of data, it just gets the data it needs
* Each importer specifies its dependencies, so that the order of import can be determined at runtime using a topological sort
* The main data generator function can just tell each importer to import the data it has

This makes working on the data generator much easier.

Some other benefits are:
* Batched importing, massively speeding up the whole process
* `--tables` to set the exact tables you want to import, and specify the quantity of each
2023-08-04 13:36:09 +01:00