Wireless mesh networks (WMNs) have become the focus of research in recent years, owing to their great promise in realizing numerous next-generation wireless services. Driven by the demand for rich and high-speed content access, recent research has focused on developing high performance communication protocols, while the security of the proposed protocols has received relatively little attention. However, given the wireless and multi-hop nature of the communication, WMNs are subject to a wide range of security threats. In this dissertation, we study the security of two main design methodologies that emerged from recent research for achieving high performance data delivery in WMNs, namely, dynamic topology-aware adaptation and network coding. In addition, we also study the principles of designing efficient application layer security protocols for WMNs. Dynamic topology-aware adaption presents an important design principle that underlies many high performance network layer protocols proposed for WMNs. We study the unique security threats that exploit the cooperative nature of such protocols. The identified attacks can allow even only a few attacker nodes to distort the path selection process in the entire network and to gain control on a large portion of the traffic in the network. Our proposed defense mechanism relies on passive measurements for detecting attacks and cooperative accusation for identifying and isolating attacker nodes. Through both analysis and experimental evaluations, we show that our defense protocol is effective and incurs low overhead. Network coding is a major performance improvement technique for WMNs that has emerged in recent years. Numerous practical systems have demonstrated that network coding is able to achieve significantly improved performance over the traditional packet forwarding approach. We focus on studying the security aspects of applying network coding on WMNs. We first perform a systematic security analysis on existing network coding systems and uncover numerous security threats on various system components. We then focus on addressing a severe and generic attack against network coding systems, known as packet pollution attack. We propose the first practical defense mechanisms to pollution attacks for both of the two major wireless network coding approaches, intra-flow network coding and inter-flow network coding. Our defense uses efficiently computable random linear checksums and an efficient traceback mechanism to filter out polluted packets and identify attacker nodes. The experimental results show that the proposed mechanisms can effectively filter out polluted packets and quickly identify and isolate attacker nodes while incurring small computation and bandwidth overhead. On the application layer, we demonstrate the unique challenges and opportunities in designing efficient security protocols. We focus on the problem of providing data confidentiality for group communication on WMNs, and present a protocol framework designed specifically for WMNs. Our design employs decentralized group membership, promotes localized communication, and exploits the nature of wireless broadcast. Through both analytical and experimental evaluations, we demonstrate the importance of the design principles for the efficiency and performance of the application layer protocols on WMNs.