Territorial animals sometimes reduce the size of their defended area or even abandon it altogether. For example, during the winter, pied wagtails are often seen to switch between defending and sharing their feeding territories along riverbanks. Such flexible behaviour can be explained in terms of the shifting balance between the costs and benefits of fighting over space. In brief, animals will defend territories when the distribution of resources and the density of competitors make it economically advantageous for them to do so, but they will abandon territorial defense when this ceases to be the case. This can be seen most clearly in the context of feeding territories, where the benefits gained from ownership (energy taken in) are in broadly the same currency as the costs of defense (energy expended).

The simple graphs shown in the figure illustrate the costs and benefits of defending territories of different sizes. The model assumes that the energetic costs of fighting increase exponentially with the size of the territory because the defended area of a circular territory increases as the square of its radius. It also assumes that the benefits gained level off at larger territory sizes because there is a maximum rate of feeding beyond which animals cannot utilize more food. (Other models assume different shapes for these two curves, thus altering the predictions.) The net gain (or cost) for each territorial size is measured by the distance between the cost and benefit curves, as shown in the figure. The optimum territory size is the one corresponding to the maximum distance between the cost and benefit curves, indicating maximum net gain. Graph A shows that an increase in the density of available food (from B1 to B2) shifts the optimal territory size (i.e., the size that maximizes net gain) to the left, which means that owners should reduce the size of their territory. Thus, the model predicts that there should be a reduction in territory size in response to increased availability of food within it—a prediction shown to be true for species ranging from limpets to trout, hummingbirds, and squirrels. On the other hand, the cost of defending a territory of a given size can change; for example, it may increase as the number of individuals competing for a given patch increases.

Graph B in the figure shows that the size of a territory for which the benefits of ownership outweigh the costs of defense (i.e., there is net gain) becomes smaller as the cost of territorial defense increases from C1 to C2. Eventually a territory of any size ceases to be economically defensible (i.e., when C2 increases to C3). Therefore, the model predicts that territorial defense should be abandoned when a certain level of cost has been exceeded. Such an effect has been described for a variety of animal species, including migrating sunbirds defending patches of nectar-rich flowers and salmonid fishes defending feeding sites in streams.

Game theory: the Hawk-Dove model

The effect of increasing competitor density on territorial defense shows that the fitness consequences to an individual of behaving in a particular way depend on the presence and activities of other animals of the same species. These relationships are examined by models based on game theory, which have been particularly influential in explaining why many fights are resolved by display and threat rather than all-out attack. A game theory model, such as the famous Hawk-Dove model formulated in 1973 by English biologist John Maynard Smith and American biologist George Price, starts by defining a set of behavioral options or strategies chosen to highlight the question at issue. In the Hawk-Dove case, the objective is to understand the resolution of conflicts by conventional fighting. The model begins by identifying animals as always using displays (the doves) or always launching straight into escalated attacks (the hawks). The next step is to specify a set of assumptions about how fights are resolved. In the Hawk-Dove model, it is assumed that when a hawk fights another hawk, it sustains injury and loses 50 percent of the time but in the remaining 50 percent manages to injure the opponent and emerge victorious. A dove also wins 50 percent of fights against another dove, but only after a lengthy more-or-less mutual exchange of displays. Doves always lose fights against hawks, but they flee before they are injured. Hypotheses about the costs and benefits of using the two strategies are formalized in a payoff matrix such as that shown in the figure, which defines the net gain accruing to hawks and doves when paired against each kind of opponent. From these net gains and from the current relative frequencies of hawks and doves in a hypothetical population, the overall fitness of animals adopting each strategy can be calculated. From there the best fitness strategy for the entire population is identified. This “evolutionary stable strategy” (ESS) constitutes the most adaptive solution under a prescribed set of conditions; i.e., when adopted by most members of the population, it cannot be bettered by any other strategy. If the model depicts correctly the costs and benefits of displaying as opposed to fighting fiercely, the behaviour of real animals can be expected to resemble identified ESSs. The most important point of the model is that the best fighting strategy for one individual depends on what the competitors are doing.

A number of critical insights emerge from the Hawk-Dove model. First, it can be seen that, where the costs of escalated fighting are high, the hawk strategy is not evolutionarily stable, because, even though doves never win, they are also never injured. Therefore, the dove strategy can invade a population of hawks. This conclusion of Maynard Smith and Price’s famous model is very important in the history of behaviour science because it formalized the notion that conventional fighting can be explained by determining the costs to individuals and not by the benefits at the group level. The model also predicts that, unless the benefit of winning is greater than the risk of injury, the evolutionary stable position is actually a mixture of hawks and doves. Therefore, within the same population some individuals should be expected to use overtly aggressive tactics when competing for resources, while others should use nonaggressive means. This is a counterintuitive prediction, but it is supported by many well-documented examples. One is the ruff, a type of sandpiper native to Eurasia that is remarkable for its courtship plumage and behaviour. Most breeding male ruffs fight for small territories on which to display, but a significant minority do not fight but simply display to females from the territories of other males. Such behavioral diversity is sometimes associated with morphological variation. For example, in many species of arthropods, including fig wasps, dung beetles, and mites, some adult males have formidable weapons (such as enlarged jaws or horns) that they use when fighting over females, while others are relatively unarmed yet manage to gain mating opportunities by sneaking.

The Hawk-Dove model assumes that opponents, apart from how they behave, are identical. In reality this is unlikely to be the case, and later models relax the assumption, allowing individuals to differ in fighting ability. Yet even these models indicate that—again, because of the costs of fighting—in asymmetrical encounters both the stronger and the weaker opponent benefit from resolving fights early, on the basis of relative fighting ability, rather than from continuing to fight until one is beaten into submission. From this perspective, fights can be seen as a process whereby opponents gain reliable information about their relative fighting ability. For example, as described earlier, when one red deer stag challenges another over a group of females, the two males roar at each other. They gradually increase the rate at which they roar, and, unless the challenger can roar as fast as or faster than the defender, he will withdraw from the contest. Since roaring is energetically demanding, stags must be in good condition to roar at a rapid rate. Therefore, roaring is a good predictor of fighting ability, allowing stags to assess the probability of winning an escalated fight and to withdraw without injury if the odds are not favourable.

Unlike the hypothetical hawks and doves of Maynard Smith and Price’s model, potential opponents are also likely to differ in the value that they place on the disputed resource. A food item, for instance, is worth more to a starving animal than to a well-fed one. Game theory models that allow for such differences predict that fights will be longer and fiercer when the disputed resource is particularly valuable—hence the reason why the most spectacular and dangerous fights are over access to mates.