Can you please show a photo/pic of your problem. I don't understand your difficulty. Is it inside or outside the beam?

But with regard to this riveting thing. Why don't you glue first with epoxyfiller the washer inside the beam over the rivet which is completely stick in. Use a auxiliary piece of wood to do so.(be sure you not glue the wooden piece also). Tighten/compress the rivet first after hardening of the epoxy. Be sure you use a rivet with a long enough shaft; it will make things easier and gives a larger plug of alu before the hole.
Wax the rivets before.

Ofcourse all assuming that the shape of the beam is flat under the traveller.