Right now we limit the search results by applying it to the serp api level by using searchOptions.limit : n .

The problem is that some search results could be social media pages or website that we don't support, failing it on it. This ends up causing the /search endpoint to return less results than expected.

The idea here is that we should search for n + y over the limit, where n is the limit and y is a picked constant. That way if it fails, we can use the y links and try to call get documents on it until we hit the correct limit n.