Given $F = (y^2 - (y')^2 - 2ycoshx$

$$\therefore \partial F{\partial y} = 2y - 2coshx, \frac{\partial F}{\partial y'} = -2y'$$

The necessary condition for this functional to be extremum is

$$\frac{\partial F}{\partial y} - \frac{d}{dx}\bigg(\frac{\partial F}{\partial y'}\bigg) = 0$$

Hence the equation becomes,

$$2y - 2coshx - \frac{d}{dx} \bigg(-2 \bigg(\frac{dy}{dx}\bigg)\bigg) + 0 = 0 \\ 2\bigg(\frac{d^2 y}{dx^2} \bigg) + 2y - 2coshx = 0 \\ (D^2 + 1)y - coshx = 0 \\ i.e. (D^2 + 1)y = coshx$$

The auxillary equation (A.E.) is $D^2 + 1 = 0 \\ \therefore D = \pm i$

$\therefore$ Complimentary function (C.F.) is $y = c_{1}cosx + c_{2}sinx$.................(1)

Now particular integral (P.I.) is

$$P.I. = \frac{1}{D^2 + 1} coshx = \frac{1}{D^2 + 1}\bigg(\frac{e^x + e^{-x}}{2}\bigg) \\ = \frac{1}{2} \bigg[\frac{1}{D^2 + 1}e^x + \frac{1}{D^2 + 1} e^{-x} \bigg] \\ = \frac{1}{2}\bigg[\frac{e^x}{2} + \frac{e^{-x}}{2}\bigg]$$ $$P.I. = \frac{coshx}{2} ...................(2)$$

$\therefore$ The complete solution is

$$y = c_{1}cosx + c_{2}sinx + \frac{coshx}{2}$$

Thus extremal of $\int\limits_{x_{1}}^{x_{2}} (y^2 - (y')^2 - 2ycoshx)dx$ is $$y = c_{1}cosx + c_{2}sinx + \frac{coshx}{2}$$