I don't really agree with the entire premise. At first I read about this trying to understand what the security implications are of obscured pagination tokens. Why would one want to obfuscate pagination tokens from hackers?
Turns out, it's to make things more opaque for one's own team ::scream emoji::
This is simply not a good way to work, sorry.
Either implement paging the way the frontend team wants and expects it to work, or take the time to explain in a clear and friendly why you cannot (i.e. the data is not structured as pages, specs were clear and rigid, whatever). With healthy interteam communication, there won't be a need to obfuscate pagination tokens in this manner.
It reads like OP built something as a backend engineer without reference to client or user needs and then threw it over the wall to the frontend team. Not good.
> or take the time to explain in a clear and friendly why you cannot
The next person to work on the frontend paging code will likely look at the existing code and infer the contract from there. Doesn't matter how healthy your inteream communication is, humans just straight up don't work like that.
The documentation already says you have to use the nextToken from the current request - adding more words to the effect of "no, really, you do *have* to use the nextToken from the current request" is sadly unlikely to help in practice.
It isn't about making things more opaque for one's own team, it's about stopping people *accidentally* doing something unsupported, which is if anything a kindness to your colleagues since they won't end up with an apparently working feature that will break unexpectedly later.
OP is aiming minimise unpredicted future pain for his team, his colleagues in the front-end team, and their users.
Your point of view would be absolutely correct in a world where "everybody would just" ... but humans, as a species, don't "just" - so code accordingly.
> The documentation already says you have to use the nextToken from the current request - adding more words to the effect of "no, really, you do have* to use the nextToken from the current request" is sadly unlikely to help in practice.*
Agreed. That is not a good fix, either.
Unfortunately, if I understood the situation, there was a disconnect between what the frontend team needed or expected (pagesize, offset), and what the backend engineer could or would deliver (next page). Rather than address this primary issue (lack of understanding), OP adds a technical fix to address what is fundamentally a human, "soft skills" problem.
It is almost certainly not OP's fault, but a lack of managerial guidance of team dynamics.
In my experience, separating teams into frontend and backend silos does not lead to good outcomes. Each team begins to see the other as a clueless adversary. But even if necessary, there must be strong efforts to unify their incentives and understanding.
So, I disagree that this fix is going to fix what's wrong.
I don't think you quite did understand - the frontend team didn't actually -need- that, they just happened to have a prebuilt UI component that -used- that, reached for that (entirely understandably, to my mind) without considering if it was the right thing to use there, then reverse engineered and abused the API to make it work.
What OP did was make a technical change to make that abuse impossible so that the conversation as to whether it was something the UI actually -did- need or if the UI should simply be written to expose the intended capabilities of the API happened naturally, as early as possible in the process.
You can't use technical measures to *fix* human problems, but in this case the technical measure exists to *surface* the human problem so it can then be resolved between the humans in question.
This is IMO a far better approach than the alternative, which would boil down to micromanaging the UI team's choices in a way that would slow everybody down and likely *would* produce the adversarial dynamic you describe.
> ...they just happened to have a prebuilt UI component
With respect, that's not what is described. OP rolled an API, handed documentation to frontend, and "after a week" frontend "came up" with a UI that held different expectations for how pagination works than the API actually worked. I'm being charitable in assuming that there was a good reason that this pagesize, offset pagination style could not work, but as written, there was no communication about that, neither to us the audience nor to OP's colleagues. In fact, frontend apparently expected this.
But why not discuss offset, pagination with colleagues? Why just present it as fait accompli and move on to create a library that further locks it in?
Maybe this may help. What if we are not talking internal development teams but something different, like a commercial/public API?
In those cases you cannot affort or expect to have meetings with folks to explian and communicate, and you also can appreciate more the abuse (unintended or not) that tokens can have.
I particularly liked that OP mentioned about expiration, key rotation and more advanced features you can achieve with his proposal, like switching schemes
Speculation here, but given the frontend team reverse engineered OP's API and attempted to work around its limitations, and then engineer countered by taking the time to craft a whole library, with very few words passed between, I suspect this is a long-standing problem.
However, I don't wholly blame OP. It smells to me like inexperienced or absent leadership.
Turns out, it's to make things more opaque for one's own team ::scream emoji::
This is simply not a good way to work, sorry.
Either implement paging the way the frontend team wants and expects it to work, or take the time to explain in a clear and friendly why you cannot (i.e. the data is not structured as pages, specs were clear and rigid, whatever). With healthy interteam communication, there won't be a need to obfuscate pagination tokens in this manner.
It reads like OP built something as a backend engineer without reference to client or user needs and then threw it over the wall to the frontend team. Not good.