Electromagnetic fields can generate orientation-dependent, long range interactions between colloidal components that direct their assembly into highly ordered structures, such as small ordered clusters, chains, and large crystalline lattices. While much effort has been devoted to exploring the assembly of spherical colloids, few reports have investigated the directed assembly of non-spherical particles with Janus or patchy morphologies. Here, we use photolithographic techniques to fabricate a wide range of anisotropically shaped patchy particles and follow their assembly in liquid suspensions under the influence of electric and magnetic fields. We analyze the assembly of several types of patchy particles across a range of field parameters and fluid compositions, and report a number of distinct, well-ordered, assembly architectures including cylindrical, prismatic, and staggered chains. The structures assembled from anisotropic patchy components provide a glimpse into the range of architectures that can be created by combining field directed assembly with rationally designed particles. By using numerical simulations to model the electric and magnetic field interactions between these particles, we interpret the results of the assembly process and explain how they can be controlled by the position of the metal facet, the frequency (for AC fields), or magnetic susceptibility of the medium. The resulting structures, and similar ones produced through the field-directed assembly of patchy anisotropic particles, can possess unique electrical and optical properties and may have potential applications in a number of future technology applications such as microactuators, metamaterials and multiferroic materials.