The need for exploitation of sea resource, transportation on the sea and also the automatization of ship navigation makes the research on ship motion control practically significant. Nonlinear control methods can improve the performance and safety and economy of ship navigating systems. Also, as a nonlinear dynamical system, the maneuvering control of surface vessels has important theoretical values and is a worthy studying project. Based on the practical Project-Intelligent Autopilot and NSFC Project (60575047), creative work is done in this dissertation as follows. Firstly, the history and statue in quo of the control of surface vessels are summarized from practical approach. Theoretically, the control theory about underactuated surface vessels and also the related nonholonomic systems is introduced with emphasis. Secondly, the dynamical model of surface vessels with three degrees of motion is introduced. The property of the kinematics and dynamics is described respectively. A simplified model for heading control is given additionally. To be prepared for the research of the control problem of underactuated surface vessels, the analysis of the small-time locally controllability of the dynamical systems is made and shows that no continuous and time-invariant state feedback control exists for the stabilization of the underactuated surface vessels. Thirdly, a dynamical adaptive sliding mode control method is proposed for the heading control of the simplified Bech-Smitt model of surface vessels with nonlinear uncertain factors. Fourthly, based on the nonlinear model of fully actuated surface vessels with three degrees of freedom, a sliding mode control approach with backstepping design is proposed for the dynamical tracking control problem. Asymptotical stability is proven. Fifthly, based on the nonlinear model of underactuated surface vessels with three degrees of freedom, an adaptive control method is proposed for the dynamical tracking control problem under constant environmental disturbances. Sixthly, the stabilization problem of an underactuated surface vessel is addressed. A discontinuous control approach with two stage control laws switched on at given time is proposed based on the stability analysis of the global transformed system with the aid of terminal sliding mode (TSM) method. Finally, the main conclusions of the dissertation are summarized and the work for future research is put forward.