The riveting structures of 5052 aluminum alloy were prepared by self-piercing riveting (SPR) technique under convex bottom die and flat bottom die, and the mechanical response and failure behavior of SPR joints under tensile-shear and peeling load were analyzed by changing the lap form of sheets. The results show that the SPR joints with thinner sheet riveted as the upper sheet can bear higher shear load and increase failure displacement. When the sheet thickness is the same, the shear resistance and peeling resistance of the joints with flat bottom die are better than that of the joints with convex bottom die. From the macroscopic point of view, the main failure mode of joints with convex bottom die is that the button is pulled out from the lower sheet  adhering to the bottom of rivet. At the same time, there is failure mode of upper sheet fracture. The failure modes of joints with flat bottom die are pull-out failure and rivet embedded in the lower sheet caused by slight tearing of the upper sheet. From the microscopic point of view, the shear fracture of the upper sheet around the riveting position is the main failure cause of the peeling joint with flat bottom die. When the thinner sheet is riveted as the upper sheet, the fracture mechanism of the single-lap joint with convex bottom die is microporous aggregate ductile fracture of the upper sheet. The other joints that failured due to rivet pulling out show significant scratching appearance at the riveting position of the lower sheet.