Polygonal billiards are interesting examples whose classical dynamics is neither integrable nor chaotic. The motion in a typical polygon is conjectured to be ergodic on the three-dimensional constant-energy surfaces. But this is not rigorously proven so far. There is numerical evidence that motion on these energy surfaces may exhibit even stronger ergodic properties, e.g. mixing.
The motion in a rational polygon (all angles are rationally related to Pi) is restricted to two-dimensional invariant surfaces. That is similar to integrable systems, but the genus of these surfaces is larger than 1, so they do not have the topology of tori. Rational polygonal billiards are therefore also characterized as pseudointegrable. It is proven that the flow on such a surface is ergodic and not mixing. It is an open question whether this flow is typically weak mixing. Weak mixing as maximal ergodic property implies interesting (classical) spectral properties. I have studied the spectra of the barrier billiard, see Fig. 1, in [1]. Recently, I have discovered an interesting relation to Andreev billiards [3].
While the classical dynamics in rational polygons is close to integrability, it has been found that the energy eigenstates are similar to those in fully chaotic cavities. They look typically "irregular" as can be seen in Fig. 2. I have resolved the paradoxon by showing that appropriate superpositions of energy eigenstates share properties of eigenstates in integrable systems [2].
In collaboration with T. Gorin, G. Carlo and A. Bäcker, I study the structure of the energy eigenstates in more detail. Preliminary numerical results indicate that the eigenstates have multifractal properties in momentum space.
Polygonal billiards have interesting applications in mesoscopic optics. Presently, I investigate the emission properties of coupled dielectric resonators of hexagonal shape.