Overcoming the 20,000 Objects Limit
As mentioned earlier, there is currently no way to get more than 20,000 objects back from a single API request, not even in a paginated fashion. In other words, the
total_count field in a request will never exceed 20,000, even though there might be two million objects in the RIPE Atlas data.
This limit exists for performance reasons. To overcome this limit, you can move the cursor in the request yourself, with the aid of the
id__gt query parameter.
Let's say you want to retrieve the first 80,000 measurements in RIPE Atlas, starting from the measurement with ID 2,000,000. You'd start out with a regular request, but with a query parameter
?offset=19900. This will get you the last page of the first 20,000 measurements in the RIPE Atlas database:
total_count field is 20,000. Now, scroll all the way down and look up the
msm_id of the last measurement in the list.
You can now use this
msm_id in a new request with the query parameter
id__gt=<YOUR LAST ID> to retrieve the next 20,000 measurements: