I was unable to find a fix for this problem, thus, the only solution I see is disabling this functionality for the continuous readers.
Due to preloading leading pages, the layout got shifted once the pages got loaded.
This messed with the scroll position which caused the top of the scrolled into views pages to not be at the top of the viewport anymore.
Depending on the screen size and e.g. the set "reader width" pages might not have filled out the full img elements size due to "object-fit contain".
This was caused due to the image wrappers flex container "align-items center" which slightly reduced the images width to some ".xy" number (e.g. instead of wrappers width 258px, the images were something like 257.83px)
In case two pages were displayed with different dimensions, the bigger page might take up more than 50% of the screen.
In this case the bigger page got limited to 50% which caused in to unintentionally get reduced in size
