fixes GRO-34
fixes GRO-33
This is a revision of a previous commit, that broke the browser tests
because changes in the data generator (requiring bookshelf had side
effects).
This adds a new way to run all tests with enforced numeric ObjectIDs.
These numeric ids cause issues if they are used withing NQL filters. So
they surface tiny bugs in our codebase.
You can run tests using this option via:
NUMERIC_IDS=1 yarn test:e2e
Removed some defensive logic that could be explained by this discovered
issue.
fixes GRO-34
fixes GRO-33
This also adds a new way to run all tests with enforced numeric ObjectIDs.
These numeric ids cause issues if they are used withing NQL filters. So they
surface tiny bugs in our codebase.
You can run tests using this option via:
NUMERIC_IDS=1 yarn test:e2e
Also removed some defensive logic that could be explained by unquoted ids.
refs https://ghost.slack.com/archives/C02G9E68C/p1695296293667689
- We block all outgoing networking by default in tests. When editing a
post/page, Ghost tries to send a webmention. Because of my earlier
changes
(1e3232cf82)
it tries 3 times - all network requests fail instantly when networking
is disabled - but it adds a delay in between those retries. So my change
disables retries during tests.
- Adds a warning when afterEach hook tooks longer than 2s due to
awaiting jobs/events
- We reduce the amount of webmentions that are send, by only sending
webmentions for added or removed urls, no longer when a post html is
changed (unless post is published/removed).
refs https://github.com/TryGhost/Product/issues/3850
- Added a recheck for recommendation related webmentions after boot (to
check missed webmentions during down time)
- Increased general timeouts to 15s for all webmention related HTTP
requests. Instead, increased retries to 3.
- Increased timeout for fetching webmention metadata from 2s to 15s
- Added more logging about verification and deletion status of
webmentions
refs: https://github.com/TryGhost/Toolbox/issues/595
We're rolling out new rules around the node assert library, the first of which is enforcing the use of assert/strict. This means we don't need to use the strict version of methods, as the standard version will work that way by default.
This caught some gotchas in our existing usage of assert where the lack of strict mode had unexpected results:
- Url matching needs to be done on `url.href` see aa58b354a4
- Null and undefined are not the same thing, there were a few cases of this being confused
- Particularly questionable changes in [PostExporter tests](c1a468744b) tracked [here](https://github.com/TryGhost/Team/issues/3505).
- A typo see eaac9c293a
Moving forward, using assert strict should help us to catch unexpected behaviour, particularly around nulls and undefineds during implementation.
refs TryGhost/Team#2844
- assertion was comparing a constant `now` with a `new Date()` that was
made on the next line — by passing now in as the end date to
`getMentionReport`, the end date should always equal the constant
- intent of the test is just to make sure a mentions report can
generate, so we can just use the variable now instead of creating new
Date() objects
refs TryGhost/Team#2833
- for mocha tests, we can add `this.retries(1)` to any flaky tests
- for playwright tests, we can add `test.describe.configure({ retries:
1})` to any `describe` block
- not a long-term solution, but it should help mitigate issues with flaky
tests in short term
closes https://github.com/TryGhost/Team/issues/2572
The mentions browse output previously only showed resource info if the
mentioned resource type was a `post`.
Additionally, the `resource_type` column basically defaulted to `post`
regardless whether it was a page or in fact a post.
With this change we now have `resource_type` wired in to correctly
determine if the mentioned url was a page or a post.
Refs TryGhost/Team#2459
-upgraded got from v9.6.0 to v11.8.6 to support following redirects (and
other fixes)
-got v12+ requires ESM, so we do not want to upgrade further at this
time
-required changes to a few libraries that use externalRequests
-mention discovery service tests updated to test for follow redirects
refs https://github.com/TryGhost/Team/issues/2550
By using cheerio to parse the HTML we can correctly look for elements
which use the target URL as the href attribute, rather than doing a
plaintext search. This closer to what the spec says.
refs https://github.com/TryGhost/Team/issues/2550
Whilst this isn't ideal making multiple requests for the same site
it's a first step towards verification properties, and can be
refactored to improve performance later. With the current volume of
incoming Webmentions we've seen so far this shouldn't be a problem.
closes https://github.com/TryGhost/Team/issues/2548
Rather than use a setter here we've used a verify method which takes the HTML
string and naively validates that the target URL is present. This is so that the
logic of verification is encapsulated in the Mention, and should mean that
erroneous verification doesn't happen.
We could consider down the line that the verify method fetches the content
itself, but if we're to do that we should pass in `got` as a param, so that it's
possible to stub in tests.
One thing to think about when it comes time to making this as performant as
possible is doing a single fetch of the source document and using that for
verification and metadata extraction. At that point we should probably
consolidate both of those operations, either moving the metadata extraction into
the Mention entity (passing in any necessary deps) OR we move the verification
out to the same layer as metadata extraction.
refs https://github.com/TryGhost/Team/issues/2534
As we're using soft deletes for mentions we need to store the `deleted` column
as well as enforce a `'deleted:false'` filter on the bookshelf model.
We've also implemented the handling for deleting mentions. Where we remove a
mention anytime we receive and update from or to a page which no longer exists.
Co-authored-by: Steve Larson <9larsons@gmail.com>
refs TryGhost/Team#2477
-removed post.edited as it was too inclusive
-changed to post.published, post.published.edited, post.unpublished
-blocked import and internal data from triggering mentions
This is a pretty simple way for us to track which webmentions are sent
by Ghost. Although it's easily spoofed, so are other approaches like
using a header (e.g. User-Agent). If we find that this data is being
spoofed we can look at different approach.
Becuase our receiving implementation stores the payload of the
Webmention, we'll be able to know inside Ghost which Mentions
originated from another Ghost installation, which is useful for stats
and gives us the possibility to display that information in the feed.
Longer term we might want to consider storing this data in a separate
column for Mentions, rather than the `payload` column - but that is
outside the scope of this change.
Based on a discussion in slack we want to make all metadata properties optional,
with the exception of the title, which will default to the host of the source
URL if it's missing.
This is so that we can accept as many webmentions as possible and convert them
into Mentions. If we were to have strictly validation, we'd end up having to
drop webmentions that didn't match our criteria, and lose important data.
Giving the title a default allows us to provide a consistent UI experience too.
refs https://github.com/TryGhost/Team/issues/2419
This is the initial stab at having everything wired up, we're not
using a queue but we are handling the processing of the Webmention
asyncrounsly so that the HTTP response can be end immediately.
We've also laid the groundwork for extending and implementing the
correct processing of Webmentions, for example checking if the target
URL exists in the system, pulling out the metadata from the Webmention
source and fetching any internal resources.
refs https://github.com/TryGhost/Team/issues/2435
We've made these fields optional, and we may need to extend this to other fields
too as we discover more about the data we're able to get access to.
refs https://github.com/TryGhost/Team/issues/2416
This extends the mock API to use a more formal pattern of moving our
entity code into a separate package, and use the service/repository
patterns we've been work toward.
The repository is currently in memory, this allows us to start using
the API without having to make commitments to the database structure.
We've also injected a single fake webmention for testing. I'd expect
the Mention object to change a lot from this initial definition as we
gain more information about the type of data we expect to see.
