closes https://github.com/TryGhost/Team/issues/2572
The mentions browse output previously only showed resource info if the
mentioned resource type was a `post`.
Additionally, the `resource_type` column basically defaulted to `post`
regardless whether it was a page or in fact a post.
With this change we now have `resource_type` wired in to correctly
determine if the mentioned url was a page or a post.
fixes https://github.com/TryGhost/Team/issues/2562
New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)
UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen
Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).
Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
Refs TryGhost/Team#2459
-upgraded got from v9.6.0 to v11.8.6 to support following redirects (and
other fixes)
-got v12+ requires ESM, so we do not want to upgrade further at this
time
-required changes to a few libraries that use externalRequests
-mention discovery service tests updated to test for follow redirects
no issue
- Instead of running milestone service directly on boot, set a random
timeout of 0-4 days to run after boot
- Updated tests
- Service is still behind a beta flag
refs 0306c397d0
refs https://github.com/TryGhost/Toolbox/issues/522
- 'extraProperties' should have been cleaned up along with the referenced commit. This property does not perform any logic in current codebase (see ref) and makes it problematic to make "post" resource serialization more generic (for caching purposes).
refs https://github.com/TryGhost/Toolbox/issues/515
- We don't have a good way to test TTL caches without setting up Redis in the environment
- Adding in-memory cache adapter with TTL allows to run tests on CI without having to install Redis
- Also, TTL in memory cache can be a great substitution for Redis-based caches on instances that
have a lot of spare RAM and don't need to use Redis necessarily
- MemoryTTL cache accepts two parameters "TTL" and "max"
- TTL - is time in milliseconds to hold the value for in cache
- max - is the maximum amount of items to keep in the cache
- To use MemoryTTL cache specify following config in the cache section:
```
"adapters": {
"cache": {
"imageSizes": {
"adapter": "MemoryTTL",
"ttl": 3600
}
}
}
```
- Above config would apply MemoryTTL cache to imageSizes feature with TTL fo 3600 ms
Removed the wrapper class for the email service from coverage, because this only wires up a lot of dependencies, which is hard to test in a unit test because we also have to init all the dependencies in a unit test. It is already covered by E2E tests.
refs https://github.com/TryGhost/Team/issues/2550
By using cheerio to parse the HTML we can correctly look for elements
which use the target URL as the href attribute, rather than doing a
plaintext search. This closer to what the spec says.
refs https://github.com/TryGhost/Team/issues/2550
Whilst this isn't ideal making multiple requests for the same site
it's a first step towards verification properties, and can be
refactored to improve performance later. With the current volume of
incoming Webmentions we've seen so far this shouldn't be a problem.
[Added initial mentions-jobs
service](3656190114)
This is the result of running `cp -r jobs mentions-jobs` in the services
directory.
[Waited for mentions-jobs queue before
shutdown](2bb1a12a89)
This matches the functionality of the existing jobs service where we
will wait
for jobs to complete before closing the process.
[Used mentions-jobs service in the mentions
service](4e4f9fdd00)
This ensures that any delays in the mentions jobs queue does not effect
other
parts of the application.
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4
- Added a `slack-notifications` repository which handles sending Slack
messages to a URL as defined in our Ghost(Pro) config (also includes a
global switch to disable the feature if needed) and listens to
`MilestoneCreatedEvents`.
- Added a `slack-notification` service which listens to the events on
boot.
- In order to have access to further information such as the reason why
a Milestone email hasn't been sent, or the current ARR or Member value
as comparison to the achieved milestone, I added a `meta` object to the
`MilestoneCreatedEvent` which then gets accessible by the event
subscriber. This avoid doing further requests to the DB as we need to
have this information in relation to the event occurred.
---------
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
refs https://github.com/TryGhost/Team/issues/2561
- added simple socket-io implementation to Ghost server
- added alpha flag for websockets
- added route in admin to test websockets using a simple counter stored in server local memory (refreshes on reboot)
refs https://github.com/TryGhost/Toolbox/issues/522
- Browse endpoints for Posts and Pages are creating the most database traffic in the system. These are read-only endpoints that don't have to be fresh 100% of the time. Having optional cache allows to offload some of the database querying to more efficient storage.
- To enable cache for Posts/Pages browse endpoints there are two prerequisites:
- set 'hostSettings:postsPublicCache:enabled' to 'true' in the configuration file
- add 'postsPublic' cache adapter in cache configuration
- Example config for adapters with 60s TTL for a cached resource:
```
"adapters": {
"cache": {
"postsPublic": {
"adapter": "Redis",
"ttl": 60,
"keyPrefix": "site_id_here:posts-content-api:"
}
}
},
```
refs https://github.com/TryGhost/Toolbox/issues/522
- The caching rules for Content API resources like Tags / Posts / Pages / Authors are the same, so it makes sense to make a common repository which would take in a model as a parameter and perform the queries on that level instead of implementing repository-per-model.
- With this refactor getting Posts Content API caching would be much simpler change.
fixes https://github.com/TryGhost/Ghost/issues/16278
`urlToPath` is not a method that is defined on the StorageBase, and thus is not implemented on any custom storage adapter.
We need this method to prevent uploading a file in two different directories. Currently this is an edge case:
- If you upload a file at midnight between a month shift, it is possible that we upload one file at `/2023/02/file.png` and the original at `/2023/03/file_o.png`
- To make sure we upload the second at the same directory, we need the `urlToPath` implementation. The save method sadly only returns the URL, which could be different than the stored file path, so we cannot use that return value.
This fix only uses `urlToPath` if available, to prevent above issue.
fixes https://github.com/TryGhost/Team/issues/2433
- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
closes https://github.com/TryGhost/Team/issues/2551
Rather than blindly passing all data through the API we explicitly include each
new property. This allows us to make changes to the core entities without
affecting the API. The verified property is being added now to give design the
ability to display these mentions differently.
We also needed to include the verified property in the return value of toJSON,
this was missed as part of the original entity changes
no issue
- With the switch of using a `MilestoneCreatedEvent` we'll be decoupling the mailing functionality and not need `GhostMailer` as dependency in the package anymore
no issue
- In preparation of using event emitting for Milestone achievements, we needed to add a dedicated `MilestoneCreatedEvent` to the `Milestone` entity.
- The event will be emitted using `DomainEvents` when a new milesteone is saved, which will allow us to listen to these events.
no issue
- The way we're going to implement milestones diverged from the original idea of handling email sending within the milestone-emails package, as we'll be sending events instead and will utilise the StaffService to listen to them and send the emails
- This renames the package as well as the service in core itself and all relevant tests
no issue
- For better testability with in-memory repository, refactor the
milestones service to preserve the API instance
- Fetching the information about Stripe live mode from Stripe service
was causing difficulties when testing. As a workaround we switched to
reading the live mode keys and determine it that way.
---------
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
refs c489343831
- `lexicalLib.lexicalHtmlRenderer` can't be used directly because it's missing required config for nodes to render properly
- updated email serializers to use `lexicalLib.render()` instead
closes https://github.com/TryGhost/Team/issues/2558
- bumped `kg-lexical` packages so we're working with latest suite of default nodes and renderer
- added a `render()` method directly to our `lexicalLib` object
- allows us to pass through all of Ghost's config for image transforms etc in one place rather than every time we want to render something
no issue
- local development Ghost installs should not default to a locally served `koenig-lexical.umd.js` file but should use the default CDN version
- a locally served file should be set in `config.local.json` when you want to develop against an unreleased version
