Solution:

We have $f(x,y)=x^3+3xy^2-3x^2-3y^2+4$

Step 1: We now find fx , fy and fxx , fyy , fxy

$f_x=3x^2+3y^2-6x~~~~; ~~~~~f_y=6xy-6y$

$f_{xx}=6x-6~~~~;~~~f_{xy}=6y~~~;~~~f_{yy}=6x-6$

Step 2: We now solve fx=0 and fy=0 simultaneously

Substituting fx=0, we get $x^2+y^2-2x=0$

Substituting fy=0, we get $xy-y=0$

$\therefore y(x-1)=0~~~~~~~~\therefore y=0 ~~or~~ x=1$

When y=0 , $x^2-2x=0 ~~gives~~ x=0 ~~or~~2$

Therfore, (0,0) and (2,0) are stationary points

When x=1 , $1+y^2-2=0 ~~gives~~ y=1 ~~or~~-1$

Therefore, (1,1) and (1,-1) are stationary points  

Step 3:

(a)At x=0 , y=0

$r=f_{xx}=-6~~,~~s=f_{xy}=0~~,~~t=f_{yy}=-6$

(\therefore rt-s^2=36-0=36>0)

(And ~~~r=-6<0)

Therefore , f(x,y) is maximum at (0,0)

Substituting x=0 and y=0 in f(x,y),we get,

 $(0)^3+3(0)(0)^2-3(0)^2-3(0)^2+4=4$ 

Therefore, Maximum Value = 4 .  

(b) At x=2 , y=0

$r=f_{xx}=12-6=6~~,~~s=f_{xy}=0~~,~~t=f_{yy}=6$

(\therefore rt-s^2=36-0=36>0)

(And ~~~r=6>0)

Therefore , f(x,y) is minimum at (2,0) 

Substituting x=2 and y=0 in f(x,y),we get,

$(2)^3+3(2)(0)^2-3(2)^2-3(0)^2+4=8-12+4=0$

Therefore, Minimum Value = 0 .  

(c) At x=1 , y=1

$r=f_{xx}=6-6=0~~,~~s=f_{xy}=6~~,~~t=f_{yy}=6-6=0$

(\therefore rt-s^2=0-36=-36<0)

Therefore, f(x,y) is neither maximum nor minimum. It is a saddle point.  

(d) At x=1 , y=1

$r=f_{xx}=6-6=0~~,~~s=f_{xy}=-6~~,~~t=f_{yy}=6-6=0$

(\therefore rt-s^2=0-36=-36<0)

Therefore, f(x,y) again is neither maximum nor minimum. It is also a saddle point.