What is Weber’s notion of rationality in terms of the law, specifically the evolution of societies from primitive, irrational types of law to a more “structured system of formal laws”? In other words, how is criminal justice handled in irrational societies versus rational societies?

The first thing to understand is that Weber (and other sociologists) does not necessarily believe rational societies are better; normally, we think of being "rational" as a good thing, but in sociology the term is assigned a more neutral connotation. An "irrational" society is not necessarily a bad one; rather, it is one that uses something other than formal logical arguments to make decisions.The key difference in terms of criminal justice involves the sort of decision procedures that would be employed to decide whether and how someone should face punishment. In an irrational society, there would be vague, informal customs, which might hold a lot of authority but were never very clearly defined (a "traditional society"); or there could be a strong, charismatic leader who would issue orders more or less as they pleased (a "charismatic society"). Whether the results were cruel or merciful would depend upon the personality and whims of the leader, and could vary capriciously from person to person and case to case.In a rational society, laws were written down, as clearly as possible, and judicial authorities were required to consult them and conform to them whenever they made their decisions. The terms for conviction and sentencing were written down long before, and applied to many cases in (ostensibly) the same way; logical arguments and evidence are used to determine who is guilty and how they should be punished. Almost all modern societies are of this form, so it seems obvious to us that this is how justice systems should work; but it is actually a quite radical departure from how decisions were made for most of human existence. The idea that a government attains its legitimacy from the rule of law and the system of bureaucratic institutions is very far removed from how most societies decided authority by custom, tradition, family ties, or outright force.Weber argued that it was more or less inevitable that societies would transition from irrational to rational because the rational system is more stable. When government is done by custom and the fiat of particular leaders, customs can change and leaders can die; but when government is done by the rule of law, explicit written codes with clearly-defined logical arguments, such an institution can preserve itself for a much longer period. Thus, by the random shocks of history, eventually every society should transition into a rational form and stay there. This is how he explained the fact that while almost all societies began in an irrational form thousands of years ago, almost all societies today are now in a rational form.