I would like to add an explanation to the solution. It wasn’t clear to me exactly what I needed to do. I took some time to figure it out though. Here is the explanation:
You need to edit the Detail view of the item that is being viewed, not the reference item that you want to show more/less of. So, for example, if I had a Customer table and a Payments table, and the Payments table has a reference to the Customer table, then I should also have a list of related Payments. If, when I click on a Customer record, I want to see more/less than the default 5 payments shown, I need to go to the detail view of the Customer, not the detail or inline view of the Payments. The Max. Nested Rows option in the detail view tells us how many rows of related items to the record do we want to see, not how many of these items we want listed when they are in a reference list.