\section{Introduction}
\label{sec:intro}

Human-computer interaction is experiencing a significant increase in
popularity due to the high availability and low cost of various sensors.
Human Activity Recognition (HAR) is most often performed using
vision-based sensors, which often need to be pre-installed
in the environment, require favorable light conditions,
specific position and rotation of the user and significant computing power.
Inertial Measurement Units (IMUs) are a type of inertial sensor that provides a more pervasive experience.
IMUs have lower power consumption, tend to be smaller and can work in a
variety of environments.

Hand Gesture Recognition (HGR) -- a sub-field of HAR -- is of particular interest,
because hand gestures are a natural method of expression for humans.
Practical applications of HGR include
device control~\cite{12_he_jin_cellphones},
health monitoring, gaming, virtual and augmented reality
and biometrics~\cite{gait-biometry}.
The state-of-the-art approaches utilize a single (often handheld) IMU
sensor and a vocabulary of very simple gestures
\cite{27_gesture_keeper,30_online_rnn,06_costante_dataset,05_uwave}.
The use of multiple sensors is largely unexplored,
which can be attributed to their significant cost in the past.
At the same time,
various hand gestures used on a daily basis make distinctive use of each of the fingers,
making it hard to distinguish
these natural gestures using only one handheld sensor.

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{images/proto_wide.pdf}
  \caption{The \textsc{WaveGlove} prototype.}
  \label{fig:waveglove}
\end{figure}

To explore the multi-sensor scenario we built a custom hardware prototype
called \textsc{WaveGlove} and designed two gesture vocabularies:
\textsc{WaveGlove}-single with $8$ whole-hand movements and \textsc{WaveGlove}-multi with  $10$ carefully designed gestures, which exhibit different movements for each of the fingers.
Using the prototype we acquired $1000$ and $10000$ samples for datasets based on the
\textsc{WaveGlove}-single and \textsc{WaveGlove}-multi vocabularies, respectively.

In an effort to put our work in context, we use a set of $11$ datasets.
It includes $2$ datasets captured with \textsc{WaveGlove},
$3$ publicly available datasets we normalized as part of this work and
$6$ datasets normalized in \cite{41_standardization_sota}. These are used to evaluate a wide selection of Machine Learning approaches for hand gesture recognition. We reproduce multiple
previously published methods and propose an improved approach
based on the Transformer~\cite{attention-is-all},
which achieves state-of-the-art results.

To the best of our knowledge, our work is the first to specifically examine
the impact of using multiple sensors for HGR.
Through an ablation study we explore the accuracy implications of multiple sensors. We also compare the recognition performance of individual sensors
and demonstrate a dependency between sensor position and gesture type.

In summary, we built a custom multi-sensor hardware prototype and
used it to acquire datasets of over $11000$ samples.
We further present $11$ datasets in a uniform, normalized format,
use them to evaluate a number of recognition approaches.
Finally, we propose a Transformer-based model and
report state-of-the-art results in its evaluation on the presented datasets.
Using this model we conduct an ablation study which shows that
increasing the number of sensors leads to increased recognition accuracy, up to a point.

\section{Related work}
\label{sec:related}

Several recent comprehensive surveys summarize research and
define the state of the art in the field of HAR~
\cite{45_sota_challenges,35_overview_challenges}.
The most commonly used recognition methods include Classic Machine Learning,
although Deep Learning methods experience a significant
rise in popularity lately~\cite{42_comprehensive_survey}.

Catal et al.~\cite{method-catal} used classifier ensembles in combination
with hand-crafted features. Convolutional Neural Networks were
applied to the task of HAR in~\cite{method-chenxue} and
\cite{method-hachoi} demonstrated that 2D convolutions
achieve better results than one dimensional convolutions.
In \cite{30_online_rnn}, recurrent LSTM layers were used for online
recognition and \cite{13_deep_fisher} has achieved state-of-the-art results
using bidirectional LSTM and GRU layers.
DeepConvLSTM~\cite{deepconvlstm}
represents a combination of convolutional and recurrent layers.
Inspired by the Transformer~\cite{attention-is-all}
(which established a new state-of-the-art
for machine translation tasks),
\cite{32_self_attention} introduces the self-attention mechanism in the field of HAR.
Our work builds on these results and proposes an improved method
based on the self-attention mechanism.

Hand Gesture Recognition was previously explored in
\cite{12_he_jin_cellphones, 30_online_rnn, 06_costante_dataset, 05_uwave}.
In \cite{sign-language} a similar glove prototype was built to recognize
french sign language characters by estimating the rotation of each finger.
A single-sensor glove prototype called GestGlove~\cite{40_gestglove} was built
to recognize a simple set of hand gestures allowing phone control.
In \cite{icassp-last-year-forked}, the concept of forked recurrent neural networks
is applied to HGR.

One of the current issues in the field of HAR is the lack of a standardized benchmark. A base for a standardized benchmark with a predefined
methodology and publicly released code has been laid in \cite{41_standardization_sota}.
We build on top of this work and aim to extend the realm of reproducible HAR methods.
From the dataset perspective,
Opportunity~\cite{ds-opportunity1}, Skoda~\cite{ds-skoda1} and PAMAP2~\cite{ds-pamap2}
are three widely used HAR datasets, which use multiple sensors.
In \cite{41_standardization_sota}, $6$ more HAR datasets are identified and normalized.
The uWave~\cite{05_uwave} dataset presents one of the few available datasets which
focus on hand gestures.

\section{WaveGlove}
\label{sec:waveglove}

We built a custom hardware glove prototype called \textsc{WaveGlove}
(shown in Figure \ref{fig:waveglove}).
It is based on a left-hand silken glove to which 5 IMUs
are attached: one MPU6050~\cite{datasheet-mpu6050} module on each of the fingers. A multiplexer is present on the back of the palm, connected via
a flexible and long enough cable to allow comfortable use.

\begin{table*}[t]
    \centering
    \begin{tabular}{ccccccccccc}

     &
     \rot{\textsc{WaveGlove}-single} &

    \rot{\textsc{WaveGlove}-multi} &

    \rot{MHEALTH} &

    \rot{USC-HAD} &

    \rot{UTD-MHAD1} &

    \rot{UTD-MHAD2} &

    \rot{WHARF} &

    \rot{WISDM} &

    \rot{uWave} &

    \rot{Average}

    \\ \hline

    \multicolumn{1}{c|}{
        \begin{tabular}[c]{@{}c@{}}
            Baseline
            Decision Tree
        \end{tabular}
    } &
    \multicolumn{1}{c|}{99.10} &
    \multicolumn{1}{c|}{96.63} &
    \multicolumn{1}{c|}{\textbf{93.41}} &
    \multicolumn{1}{c|}{89.22} &
    \multicolumn{1}{c|}{67.51} &
    \multicolumn{1}{c|}{81.90} &
    \multicolumn{1}{c|}{66.26} &
    \multicolumn{1}{c|}{61.61} &
    \multicolumn{1}{c|}{70.72} &
    \multicolumn{1}{c|}{80.71}
    \\ \hline

    \multicolumn{1}{c|}{
        \begin{tabular}[c]{@{}c@{}}
            DeepConvLSTM
            (2016)\cite{deepconvlstm}
        \end{tabular}
    } &
    \multicolumn{1}{c|}{98.05} &
    \multicolumn{1}{c|}{99.30} &
    \multicolumn{1}{c|}{81.01} &
    \multicolumn{1}{c|}{83.97} &
    \multicolumn{1}{c|}{67.29} &
    \multicolumn{1}{c|}{86.50} &
    \multicolumn{1}{c|}{67.98} &
    \multicolumn{1}{c|}{\textbf{91.23}} &
    \multicolumn{1}{c|}{98.10} &
    \multicolumn{1}{c|}{85.94}
    \\ \hline

    \multicolumn{1}{c|}{
        \begin{tabular}[c]{@{}c@{}}
            DCNN Ensemble
            (2019)\cite{dcnn_ensemble}
        \end{tabular}
    } &
    \multicolumn{1}{c|}{-} &
    \multicolumn{1}{c|}{-} &
    \multicolumn{1}{c|}{93.09} &
    \multicolumn{1}{c|}{88.49} &
    \multicolumn{1}{c|}{62.03} &
    \multicolumn{1}{c|}{81.63} &
    \multicolumn{1}{c|}{75.50} &
    \multicolumn{1}{c|}{89.01} &
    \multicolumn{1}{c|}{-} &
    \multicolumn{1}{c|}{-}
    \\ \hline

    \multicolumn{1}{c|}{
    \begin{tabular}[c]{@{}c@{}}
        Transformer-based
        (proposed)
    \end{tabular}
    } &
    \multicolumn{1}{c|}{\textbf{99.40}} &
    \multicolumn{1}{c|}{\textbf{99.99}} &
    \multicolumn{1}{c|}{90.35} &
    \multicolumn{1}{c|}{\textbf{89.83}} &
    \multicolumn{1}{c|}{\textbf{76.32}} &
    \multicolumn{1}{c|}{\textbf{88.42}} &
    \multicolumn{1}{c|}{\textbf{78.63}} &
    \multicolumn{1}{c|}{84.53} &
    \multicolumn{1}{c|}{\textbf{98.80}} &
    \multicolumn{1}{c|}{\textbf{89.59}}
    \\ \hline
    \end{tabular}
\caption{
    Accuracy comparison of classification methods.
    The "-" symbol denotes that the given method was not applied on the dataset.
    Note that this table only showcases the most relevant methods and datasets. For the full set of results (over $12$ methods and $11$ datasets),
    exact model and hyperparameter description, as well as the code used to preprocess the input datasets and produce the results, please refer to
    \href{https://github.com/Zajozor/waveglove}{https://github.com/zajozor/waveglove}.
}
\label{tab:stats}
\end{table*}

\subsection{\textsc{WaveGlove} datasets}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/vocab-single-showcase.pdf}
    \caption{Eight gestures from the \textsc{WaveGlove}-single vocabulary.}
    \label{fig:single-vocab}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/vocab-multi-showcase.pdf}
    \caption{A selection of the gestures from the \textsc{WaveGlove}-multi vocabulary, which contains $10$ gestures in total.}
    \label{fig:multi-vocab}
\end{figure}

Two gesture vocabularies were used to acquire two separate datasets.

The \textsc{WaveGlove}-single vocabulary consists of $8$ gestures and a null class -- the state in which no gesture is performed. These gestures are simple movements, during which the whole hand performs a uniform motion (Figure \ref{fig:single-vocab}).
This set of gestures is motivated by \cite{gestures-identified},
which identified it to be a popular and user-preferred set for device control.
It is also one of the most often used gesture vocabularies
\cite{06_costante_dataset, 05_uwave, 18_egyptian}.
\textsc{WaveGlove}-single is primarily used as a reference point --
it uses the same set of gestures as previous work, with the only
difference being the use of five sensors instead of a single one.

On the other hand, \textsc{WaveGlove}-multi was designed as a set of custom hand gestures
which would showcase the potential of multiple sensors. These gestures consist of different movements
for individual fingers and some gestures differ only in specific finger movements.
This vocabulary is used to demonstrate that
the use of multiple sensors located on the hand significantly expands the space of
recognizable gestures and allows the hand gestures to become much more natural.
It contains $10$ gestures and a null class.

The key difference between the two gesture vocabularies lies in their level of complexity. While gestures from \textsc{WaveGlove}-single were successfully classified using a single sensor in previous work, the gestures in \textsc{WaveGlove}-multi are designed to fully utilize multiple sensors.
We take a closer look at this difference in an ablation study in Section \ref{sec:ablation}.

Several important factors affect the final quality of the dataset~\cite{55_datasets}.
In order to make our samples more realistic, ensure the recordings stay independent and improve the reproducibility of our experiments, we took several measures.
The gestures are recorded in short (approximately 2 minute) sessions.
Each session consists of recording a few samples for each of the gestures in one of the datasets.
In each session, the gestures are presented in a random order and the recording software requires a break (of a few seconds) between any two gestures.
This improves gesture independence and allows the data captured during the breaks to be used as the null class.

Before recording, the test subjects are first presented with uniform instructions.
The test subjects themselves mark the beginning and the end of a sample by pressing and
releasing a button on the keyboard. In case the subject made a mistake in the process
(eg. forgot to move the hand, or pressed the button too late), they are allowed to re-record a sample.

In total, the \textsc{WaveGlove}-single dataset contains over $1000$ samples
and the \textsc{WaveGlove}-multi dataset contains over $10000$ samples.
To the best of our knowledge \textsc{WaveGlove} is the first device utilizing multiple inertial sensors used to record publicly available datasets of this scale.

\section{Dataset normalization}
\label{sec:datasets}

In order to ensure fair comparison of the presentend HAR and HGR models, we prepare a set of $11$ datasets as a standardized benchmark.

We include the $6$ datasets preprocessed in \cite{41_standardization_sota}
using the Leave-One-Trial-Out (LOTO) folds, which ensure that all samples
from a single trial are either in the training or the test set.
The LOTO folds use semi-overlapping windows for sampling, while
the rest of the datasets are sampled using non-overlapping windows.
These $6$ datasets include MHEALTH and USC-HAD -- two commonly used HAR datasets.
The uWave~\cite{05_uwave} dataset is a single-sensor HGR dataset
with the same vocabulary as \textsc{WaveGlove}-single.
Finally, we include
the Opportunity~\cite{ds-opportunity1} and
Skoda~\cite{ds-skoda1} datasets,
along with the two \textsc{WaveGlove} datasets, resulting in $11$ datasets
in total.

\section{Methods}
\label{sec:results}

For classification we use methods ranging from Classical
Machine Learning to approaches based on Deep Learning.
To make our results comparable,
we reproduce several previously published methods.
We further propose a classification method which utilizes the self-attention mechanism
and is based on the Transformer.

A Transformer-based architecture was utilized for HAR in~\cite{32_self_attention},
upon which we further improve by simplification and by using a more commonly used dot-product attention.

Figure~\ref{fig:model-architecture} shows the proposed architecture.
The first linear layer transforms an input of size $B \times T \times S$
($B=$ batch size, $T=$ temporal length, $S=$ sensor channels)
to $B \times T \times 32$, creating a constant-size sensor embedding.
Next, positional encoding\cite{attention-is-all} and $4$
Transformer encoder layers are used to provide an input ($T \times B \times 32$) for a dot-product
attention layer (the final encoder output in the temporal dimension is used as the query).
The final linear layer yields a one-hot encoding for target classes
($B \times 32 \rightarrow B \times C$, $C=$ classes).

A hyperparameter sweep has been performed yielding the final, best-performing parameters.
Final model accuracy comparison can be found in Table~\ref{tab:stats}.

\begin{figure}[t]
  \centering

  \includegraphics[width=0.64\linewidth]{images/6_transformer_insp.pdf}
  \caption{
  Architecture of the proposed Transformed-based model.
  }
  \label{fig:model-architecture}
\end{figure}

\begin{figure*}[t]
  \centering

  \subfigure{
  \includegraphics[width=0.45\linewidth]{images/index-center.pdf}
  }\hfill
  \subfigure{ \includegraphics[width=0.45\linewidth]{images/pinky-center.pdf}
  }
  \caption{Confusion matrices of a baseline model trained solely on \textbf{index finger} (left) and \textbf{pinky} (right) sensor data}
   \label{fig:index-pinky}
\end{figure*}

\section{Ablation study}
\label{sec:ablation}

Most of the evaluated methods achieved near-perfect
accuracy on \textsc{WaveGlove} datasets.
Given the multi-sensor setup, this opens
up an opportunity to conduct an ablation study to examine the
effect of various factors on the final performance.

\subsection{Finger attribution}

In Figure \ref{fig:index-pinky} we compare the confusion matrices
obtained by training a model on data from only a single sensor.
It allows us to observe a few distinguishable patterns.

Using the index finger sensor, the
\textit{Metal} and \textit{Peace} gestures are classified as the \textit{Null} class since they do not include index finger movement.
The gesture pairs \textit{Ungrab}/\textit{Pinch closer} and
\textit{Grab}/\textit{Pinch away} include the same index finger movement,
resulting in higher inter-class confusion.
When using the pinky sensor,
the \textit{Peace} and \textit{Grab} gestures are often confused.
Several gestures are confused with the \textit{Null} class, because they do not involve
significant pinky movement.

This experiment helps us demonstrate that sensor placement on different fingers
can have significant effect on classification results, particularly when
various fingers are featured prominently in the gestures.

\subsection{Impact of multiple sensors}

Figure \ref{fig:attribution} depicts how the number of used sensors and the
training set size affect the final accuracy\footnote{When there are multiple combinations
for a certain amount of sensors, we plot the mean accuracy and the sleeves show the range.}.
With \textsc{WaveGlove}-single, the accuracy does not increase with the
addition of multiple sensors. This is what we would expect, as the
gestures in this vocabulary do not make use of multiple sensors.

Conversely, for the \textsc{WaveGlove}-multi dataset we observe a clear
positive impact of using more sensors on the accuracy. The improvements can be considered
significant when using up to three sensors. Beyond that the increase in performance is marginal.
This positive impact is even more significant when the training set size is limited.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{images/attribution-icassp.pdf}
  \caption{
  \hspace{0cm}
    Average accuracy based on the amount of sensors used and the training set size
    as measured on the two \textsc{WaveGlove} datasets.
  }
  \label{fig:attribution}
\end{figure}

\section{Conclusions}
\label{sec:conclusion}

In this work we present a custom multi-sensor hardware prototype,
which we used to acquire a dataset of over $11000$ gesture instances.
To the best of our knowledge this is the largest publicly available dataset of multi-sensor hand gestures.
For a fair comparison with prior work we use $11$ HAR datasets, implement several previously published methods and compare their performance.
We further propose a Transformer-based network architecture, which
shows promising classification results across multiple datasets.

In an accompanying ablation study, we identify that (only) relevantly designed gestures benefit from the use of multiple sensors.
By comparing classification performance for two different sensor locations we demonstrate that the recognition accuracy has strong dependence on the location of the sensor and the type of classified gestures.

\vfill\pagebreak

\bibliographystyle{IEEEbib}


\section{Introduction}\label{sec:intro}

When identical particles are brought in a perfectly periodic lattice potential, their degenerate energy states are reorganized into allowed energy bands separated by forbidden energy gaps due to the spatial orbital wavefunction overlap.~\cite{ashcroft1976solid}  An orbital overlap between neighboring particles is quantified by a hopping integral, which determines the bandwidth of the allowed energy bands, and the spectral energy gap appears proportional to the potential strength. These particles are beautifully represented by Bloch waves, solutions to a Hamiltonian with an effective single-particle periodic-potential. In order to resemble electrons in natural crystals, artificial periodic lattices are designed and created to engineer the strengths of the particles' orbital overlap and their interaction governed by lattice geometries. In particular, a honeycomb crystal structure appears ubiquitously in solids such as graphite,~\cite{wallace1947band} graphene,~\cite{castroneto2009electronic} carbon nanotubes,~\cite{saito1998physical} two-dimensional (2D) transition metal dichalcogenide,~\cite{geim2013van} and theoretical lattice models.~\cite{nussinov2015compass} Recently, tunable honeycomb lattices are synthesized to investigate massless Dirac energy dispersion and topological phases.~\cite{polini2013artificial} Exciton-polariton honeycomb lattices have been also created by etching methods, where bandstructures are measured and edge states are identified.~\cite{jacqmin2014direct, milicevic2017orbital,cerda-mendez2010polariton,milicevic2015edge,klembt2018ti} However, their work is limited to polaritons in a far-red-detuned regime, where an approximated photon description is sufficient. In this study, we realize the full bandstructures of Bloch exciton-polaritons in an artificial 2D honeycomb lattice at not only red-detuning ($\Delta < 0$) but also zero- and blue-detuning ($\Delta > 0$) values, resulting in different ratios of exciton and photon contributions. $\Delta$ is defined as the energy detuning between cavity photon and quantum-well exciton. This quantity controls the potential strength of Bloch exciton-polaritons, consequently, the band parameters such as bandgap energies, bandwidths, energy density of states and their hopping integrals.

Microcavity exciton-polaritons are dual-quasiparticles of photon-dressed excitons as a manifestation of strong light-matter coupling in a monolithic quantum well-microcavity structure.~\cite{weisbuch1992observation,carusotto2013quantum}  As composite bosons in a low-density limit, obeying Bose-Einstein statistics, exciton-polaritons exhibit macroscopic coherence above quantum degeneracy threshold via stimulated scattering process originating from exchange interactions.~\cite{deng2002condensation,kasprzak2006bose,balili2007boseeinstein,byrnes2014exciton} There are several methods to produce a lateral confinement for trapping exciton-polaritons by modulating spatially either photons or excitons.  A thin-metal film deposition,~\cite{lai2007coherent} an etching for a pillar,~\cite{bajoni2008polariton} and a partial-etching and overgrowth~\cite{eldaif2006polariton} are implemented to pattern photonic lattices. These methods extend to produce 2D exciton-polariton lattices~\cite{angelakis2017quantum} with limited tuning parameters to observe high-orbital condensation,~\cite{kim2011dynamical, kim2013exciton, kim2014band} spin-orbit coupling in the Lieb Lattice,~\cite{whittaker2018exciton} Dirac cones,~\cite{jacqmin2014direct} and condensation in an energy gap~\cite{winkler2015polariton} and to explore magnetic orders.~\cite{ohadi2017spin,berloff2017realizing}  Theoretically, the exciton-polariton Hamiltonian in the artificial lattices can be mapped to that of the Hubbard model, with which Mott transition can be explored in exciton-polariton lattices.~\cite{byrnes2010mott}  In order to construct such Hubbard Hamiltonian, a first step is to engineer its energy terms: an on-site interaction energy and a kinetic energy associated with a hopping integral.~\cite{mahan2000manyparticle} Quantifying these terms is directly linked to the bandstructures and their parameters. Here we alter the design parameters to vary the lattice potential strength, from which we quantify values of band parameters experimentally. We achieve the complete understanding of the engineered polaritonic bandstructures by developing our two-kind-boson mixture polariton band theory. The two-kind-boson band theory explicitly describes free quantum-well excitons and bound cavity photons in artificial honeycomb periodic potentials as well as the strong dipole interaction between excitons and photons. The calculated bandstructures from our two-kind-boson band theory are in an excellent agreement with the experimental ones at all detuning values. Especially, the two-kind-boson band theory is correct to explain the bandstructurues of the blue detuned devices.

The paper is organized as follows: Section~\ref{sec:exp} describes our device and experimental setup and presents measured bandstructures with varying experimental parameters. In Sec.~\ref{sec:thy}, we introduce a complete two-kind-mixture Hamiltonian to compute theoretical bandstructures by taking into account experimental parameters. We further discuss the comparison of theory and experiments and quantify band parameters in Sec.~\ref{sec:discussion}.

\section{Experiment}\label{sec:exp}

\begin{figure}[htbp]
\centering
\includegraphics[width=\columnwidth]{Fig1.pdf}
\caption{
(a) An illustration of prepared honeycomb lattices patterned by the etching-overgrowth method. Each site is addressed as a circular disk, whose lithographic diameter is 2 $\mu$m and etching height is approximately 5 nm.
(b) A real-space honeycomb lattice is sketched by two unit vectors, $\bm{a}_1$ and $\bm{a}_2$, where two sublattices are colored green and orange.  The site-to-site distance is denoted as $d$.
(c) The first four Brillouin zones (BZs) in an associated reciprocal space are built upon the reciprocal unit vectors  $\bm{b}_1 = (0, \frac{4\pi}{3d})$ and  $\bm{b}_2 = (\frac{2\pi}{\sqrt{3}d}, -\frac{2\pi}{3d})$ with high symmetry points $\it{\Gamma}$, $K$, $K^\prime$and $M$.
(d) A folded representation of multiple BZs taken from experimental data ($\Delta = -$18.2 meV) along the high-symmetry points  denoted in (c). The theoretical folded BZs are overlaid on top of the experimental data in white straight lines. Note that the momentums are projected to the $ k_y $ direction.
(e) A representative measured bandstructure of the $d$ = 3 $ \mu $m device at  $\Delta= -18.2$  meV along  higher BZ regions, the vertical $ K-K$ line labeled as line 2 in (c), where the Dirac cones are seen in the lowest energy bands.
(f) Another cross-sectional band structure is drawn along the line 3 in (c). Two distinct forbidden energy gaps are defined: $V_\text{23}$ and $V_\text{78}$, the energy gap between the second and third bands and the seventh and eighth bands, respectively.   The bandwidth of the lowest band $\Delta E_\text{S}$ is specified.  The computed energy bandstructures are shown in the straight white line in a repeated zone scheme, and yellow and blue in the color scale bar indicate high and low intensity data, respectively,  in (d), (e) and (f).
}
\label{fig:1}
\end{figure}

\begin{figure*}
 \centering
 \includegraphics[width=\textwidth]{Fig2.jpg}
 \caption{Direct comparison of  experimental bandstructures (left column, (a), (d), (g)) with calculated bandstructures by two different methods, the approximated $\tilde{\bm{H}}_1$ (middle column, (b), (e), (h)) and the complete Hamiltonian $\bm{H}$ (right column,(c), (f), (i)) at three different detuning values:  a red-detuned device with  $\Delta = -18.2$  meV (a),  a zero-detuned device with $\Delta = -0.3$ meV (d), and a blue-detuned device with  $\Delta = 7.1$  meV (g). These data are taken along the line 1 marked in Fig. 1(c). The vertical dotted lines in the experimental bandstructures indicate the zone boundaries.}
\label{fig:2}
\end{figure*}
In this study, we used a wafer which is composed of two stacks of four 7 nm-thick GaAs quantum-wells embedded in a $\lambda$/2-AlAs cavity structure sandwiched by 32-pair top and 37-pair bottom distributed Bragg reflectors, alternating AlAs and Al$_{0.2}$Ga$_{0.8}$As layers. The spatial cavity length variations over the wafer vary detuning values $\Delta$ from $ -18.2 $ meV to $ 7.1 $ meV in our sample. Detuning values are computed from $\Delta = E_c (k_\parallel = 0) - E_X (k_\parallel = 0) $, where $E_c$ and $E_X$ are independent photon and exciton energy.  An individual block has a pattern of a honeycomb array of circular disks by etching the top cavity layer, whose topology is consecutively translated to subsequent upper layers during overgrowth.~\cite{eldaif2006polariton, winkler2015polariton}.

Each circular disk of the honycomb lattice has a fixed diameter to be 2 $\mu$m, and its etching depth is set to be 5 nm. The cavity-layer thickness variations modulate the photonic energies in space to create a periodic potential whose amplitude ranges 1-5 meV. The resulting sample schematic is sketched in Fig.~\ref{fig:1}(a). The site-to-site distance $d$ of the neighboring sites has values of 3 and 4 $\mu$m, and each block for a specific $d$ has the size of a 150 $\mu$m-by-150 $\mu$m.  A unit-cell in the real-space honeycomb lattice potential is defined by primitive unit vectors $\bm{a}_1$  and $\bm{a}_2$, where the nearest-neighbor distance is $d$ denoted in Fig.~\ref{fig:1}(b). Their reciprocal lattice vectors $\bm{b}_1$  and  $\bm{b}_2$  construct 2D hexagonal Brillouin zones (BZ) with rotational symmetry points, $\it{\Gamma}, K, K^\prime$, and $M$ (Fig.~\ref{fig:1}(c)). We make 8 blocks with $d$ = 3 $\mu$m in the range of detuning $\Delta \in$ ($ -18.2 $ , 7.1) meV, whose corresponding photonic fraction values, $|C(k_\parallel = 0)|^2$ lie between 0.92 and 0.24, where $C$ is the Hopfield coefficient and $k_\parallel$ is the in-plane momentum (We will also use $ k $ to denote $ k_\parallel $ if no potential confusion will occur in the following text).

The sample containing many blocks is cooled down to 4 - 6 K, and is excited by a continuous-wave laser at the fixed wavelength 1.616 eV (767.205 nm) at the angle of 60-degrees. Our detection is not polarization-selective. We keep the laser power to be 0.1 - 1 mW, which is much lower than the threshold pump power values ~ 40 - 60 mW at various detuning positions. The laser spot is oval-shaped due to the finite-angle pumping scheme with a size of about 120 $\mu$m-by-60 $\mu$m.

The standard angle-resolved photoluminescence spectroscopy allows us to map the bandstructures of the honeycomb lattice in the extended zone scheme, and its folded-zone representation is presented in Fig.~\ref{fig:1}(d) along the three high-symmetry points ($\it{\Gamma}, K, M$) drawn in Fig.~\ref{fig:1}(c).
We plot representative experimental bandstructures taken at two distinct lines, line 2 (Fig.~\ref{fig:1}(e)) and line 3 (Fig.~\ref{fig:1}(f)) defined in Fig.~\ref{fig:1}(c). The full massless Dirac dispersions of the $s$-bands are captured in the second BZ cutting through $K$ or $K^\prime$ in Fig.~\ref{fig:1}(e), while only the lowest Dirac band is seen in the first BZ along $\it{\Gamma} - K$ (or $K^\prime$) (Fig.~\ref{fig:1}(f)).

Figure~\ref{fig:2} presents representative experimental polaritonic bandstructures ($d$ = 3 $\mu$m) taken along the line 1 in Fig.~ \ref{fig:1}(c) at three different detuning values $\Delta = -18.2 $ meV (a), $-0.3$ meV (d), and 7.1 meV (g), where the strong $\Delta$-dependence is observed. As $\Delta$ becomes positive, higher exciton fraction reduces the energy window from $\sim$ 17 meV (Fig.~ \ref{fig:2}(a)) to $\sim$ 5 meV  (Fig.~ \ref{fig:2}(g)). Consequently, the number of accessible bands within the energy window decreases as $\Delta$ is more positive. Noticeably, the flattened bands at the large momentum values are more visible within our detected angular regions. All of these behaviors are associated with the heavier effective mass of the resulting exciton-polaritons arising from the higher fraction of excitons as $\Delta$ increases. In addition, the spectral linewidths of individual bands become much broader in the blue-detuned devices (Fig.~ \ref{fig:2}(g)), resulting in the smaller forbidden energy band regions, which blur sharp separation of different bands.

\section{Theoretical model}\label{sec:thy}

In order to explain experimental exciton-polariton bandstructures, we develop theoretical models by solving a Hamiltonian within an augmented plane-wave method.  We first apply an approximated Hamiltonian with the effective mass of exciton-polaritons, which has been working well for photon-like exciton-polaritons in previous works.~\cite{kim2011dynamical, kim2013exciton, jacqmin2014direct}  The theoretically computed bandstructures obtained from the approximated Hamiltonian are displayed in Figs.~\ref{fig:2}(b)(e)(h). When we calculate the best fitted theoretical plots matching to experimental data, we have only three fitting parameters: the effective polariton potential strength  $V_{\text{eff}}$ as a function of $\Delta$, effective mass of exciton-polaritons $ m^* $, and site-to-site distance $d$. Our theoretical bandstructures plots also take into account experimental linewidths in energy and momentum. The intensity of each state at a particular momentum $\bm{k}$ is computed by projecting the Bloch wavefunction onto a free-moving photon and integrating over the first BZ. We also take into account the spectral and momentum linewidth information from experimental bandstructures.

Most features of allowed energy bands and gaps within the first BZ are reproduced very well in all three detuning regimes. In the red-detuned device, experimental data (Fig. \ref{fig:2}(a)) and theoretical plots by this method (Fig. \ref{fig:2}(b)) are in excellent agreement within the region of $|k_y| \leq 3 (\frac{2\pi}{d}\frac{2}{3})$. A slight difference between two is visible in a region of very large $k_y$ values ($> 3 (\frac{2\pi}{d}\frac{2}{3})$) within our optical access window determined by the numerical aperture (NA = 0.55) of our objective lens. However, this approximated Hamiltonian fails to explain the bandstructures in two other devices, where the  discrepancy between experiments and theory becomes dramatically noticeable in devices (Figs.~\ref{fig:2}(d) and (g), $\Delta = -0.3$ and $ 7.1 $ meV, respectively). The flattened nature of the lower-polariton (LP) higher bands at larger $k_y$ values originates from the bare exciton dispersion. The composition of exciton and photon fractions in the strongly-coupled LP dispersions depends on not only the $\Delta$ values, which is here defined at  $k_\parallel$ = 0 but also the non-zero $k_\parallel$-values. Near $k_\parallel$= 0, the photonic component is stronger, while the excitonic component is stronger near large $k_\parallel$-values. Therefore, the LP dispersion is photon-like near smaller  $k_\parallel$-values, while the LP dispersion is exciton-like at larger  $k_\parallel$ values. The cross-over  $k_\parallel$ value between two exciton-like and photon-like regimes  within the LP dispersion is determined by $\Delta$. For the blue detuned device, this cross-over $k_\parallel$-value is smaller so that it appears within our observation range determined by the objective numerical aperture, whereas this cross-over $k_\parallel$-value is near the edge of or beyond our observation access for the red detuned devices so that the dispersion looks parabolic.

For differently behaving two-kind entities, the confined cavity photons trapped by the engineered potential and the free excitons, we originally introduce the complete Hamiltonian, $\bm{H}$, which is divided into three terms: the exciton Hamiltonian $\bm{H}_X$, the cavity-photon Hamiltonian, $\bm{H}_c$, and the exciton-photon coupling Hamiltonian, $\bm{H}_{X-c}$, i.e.
\begin{equation}\label{key}
\bm{H} = \bm{H}_X + \bm{H}_c + \bm{H}_{X-c}.
\end{equation}
In momentum space, $\bm{H}_c$ is expressed as
\begin{widetext}
\begin{equation}
	\bm{H}_c =\left(\varepsilon_c + \frac{\hbar^2 (\bm{k}+\bm{G}_{h, h^\prime})^2}{2m_c}+V(\bm{G}_{h, h^\prime, h'', h'''})\right)  \bm{c}^\dagger_{\bm{k}+\bm{G}_{h, h^\prime}}\bm{c}_{\bm{k}+\bm{G}_{h'', h'''}},
\end{equation}
\end{widetext}
where $\varepsilon_c$ is the cavity-photon energy offset, the effective mass of cavity-photon is $m_c$,  and $\bm{c}^\dagger_{\bm{k}}$ and $\bm{c}_{\bm{k}}$ are the cavity-photon creation and annihilation operators at momentum $\bm{k}$.
$\bm{G}_{h, h^\prime}$ is the general reciprocal lattice vectors, expanded by $\bm{b}_1$  and  $\bm{b}_2$, such that $\bm{G}_{h, h^\prime} = h \bm{b}_1 + h^\prime \bm{b}_2$. The coefficients of $h,h'$ are the indices of 2D Fourier transformation. Note that  $V(\bm{G}_{h, h^\prime})$ is the Fourier transformation of the honeycomb lattice in real space, formed by the honeycomb arrays of $V(\bm{r})$ at each site,
\begin{equation}\label{key}
V(\bm{r})=-V_0 \cdot \theta (|\bm{R}-\bm{r}|),
\end{equation}
where $V_0$ is the maximum potential strength for the 100$\%$ cavity-photon, approximately 5 meV, and $\bm{R}$ is the radius of the circular potential well, $ \theta(x) $ is Heaviside step function.  On the other hand, $\bm{H}_X$ has only the kinetic energy term, reading
\begin{equation}\label{key}
\bm{H}_X = \Big( \varepsilon_X + \frac{\hbar^2 (\bm{k}+\bm{G}_{h, h^\prime})^2}{2m_X} \Big) \bm{a}^\dagger_{\bm{k}+\bm{G}_{h, h^\prime}}\bm{a}_{\bm{k}+\bm{G}_{h, h^\prime}},
\end{equation}
where  $\varepsilon_X$ is the exciton energy offset, $m_X$ is the effective mass of an exciton, and  $\bm{a}^\dagger_{\bm{k}}$ and  $\bm{a}_{\bm{k}}$ are the exciton creation and annihilation operators at momentum $\bm{k}$. $\bm{H}_{X-c}$ indicates the energy interchange of exciton and cavity photon,
\begin{equation}\label{key}
\bm{H}_{X-c} = g_0  (\bm{a}^\dagger_{\bm{k}+\bm{G}_{h, h^\prime}} \bm{c}_{\bm{k}+\bm{G}_{h, h^\prime}} +\bm{c}^\dagger_{\bm{k}+\bm{G}_{h, h^\prime}} \bm{a}_{\bm{k}+\bm{G}_{h, h^\prime}}),
\end{equation}
where $g_0$ = 5.75 meV is the strength of exciton-photon coupling.

The theoretical bandstructures by the complete $\bm{H}$ are presented in Figs.~\ref{fig:2}(c)(h)(i). In comparison to the experimental bandstructures as well as the approximated Hamiltonian calculation, we conclude that, only for extremely photon-like exciton-polaritons, the preliminary plane wave basis method can be still valid since the photon fractions are higher at almost all accessible momentum values. However, for more exciton-like exciton-polaritons, the complete Hamiltonian is required because the effect of free excitons are no longer negligible. We can also show the mathematical details of the relation between the approximated and the complete Hamiltonians in Appendix \ref{app:A}.

\begin{figure}[htbp]
	\centering  \includegraphics[width=\columnwidth]{Fig3.pdf}
	\caption{(a) The Lorentzian model fits our experimental data in the energy axis. The experimental data are taken at $\it{\Gamma}$. The fitting parameters of this particular plot read $(\epsilon_0, \gamma, b) \rightarrow (1581.23, 0.18, 35.44)$ in unit of (meV, meV, meV$^2$). (b) The Gaussian model fits our experimental data in the momentum axis. The experimental data are taken at where $ E = 1591.3$ meV. The fitting parameters of this particular plot read $(c, k_0, \gamma_k) \rightarrow (390.51, 4.02, 0.15)$, where $k_0$ and $\gamma_k$  have a unit of 1/$\mu$m.
	}
	\label{fig:3}
\end{figure}

Since the signals in photoluminescence experiments in Fig.~\ref{fig:2} are the intensity of LPs, we consider the population of LPs in theoretical plots. We assume that LPs trapped in the periodic lattice potential have been entirely converted to photons in the cavity. Therefore, we define the expected leakage LP emission intensity to be proportional to the projection from the Bloch wave function $ \psi_{n,\bm{k_\parallel}} $ to a free-moving photon $ \phi_{\bm{k_\parallel}}=e^{i\bm{k_\parallel} \cdot \bm{r}} $, namely, intensity \cite{lai2007coherent}
\begin{equation}\label{eq:rho}
\rho(n,\bm{k_\parallel})\propto\abs{\innerproduct{\phi_{\bm{k_\parallel}}}{\psi_{n,\bm{k_\parallel}}}}^2,
\end{equation}
where $ \bm{k_\parallel} $ is the in-plane lattice momentum.

The theoretical bandstructures presented in Fig.~\ref{fig:2} of the main text contain the spectral and momentum linewidth information from experimental bandstructures. Figure~\ref{fig:3} shows representative data of linewidth extractions in energy and momentum from a $d$ = 3 $\mu$m device at $\Delta = {-9.6}$ meV. The energy plot fits well with the Lorentzian equation (Fig.~\ref{fig:3}(a)), whereas the momentum cross-sectional plot works well with the Gaussian-shape fit (Fig.~\ref{fig:3}(b)). Therefore the overall emission intensity in Eq.\eqref{eq:rho} is modulated by another Lorentizian term along energy and Gaussian term along momentum. Namely, the overall intensity of the plot in the whole range at the momentum $\bm{k}$ and the energy $E$  is given by

\begin{equation}
I (E, \bm{k}) = \sum_i \sum_{\bm{h}} \sum_{\bm{k}_0} |a^i_{\bm{h}} (\bm{k})|^2 \frac{\exp{-\left(\frac{\bm{k}-\left(\bm{k}_0+\bm{h}\cdot\bm{b}\right)}{\gamma_k}\right)^2}}{|E - E_i(\bm{k})|^2+\gamma^2},
\end{equation}

with the energy and momentum relaxation rates $\gamma, \gamma_k$, respectively for the $i$-th energy state. $a^i_{\bm{h}}$ indicates component of eigenstate which corresponds $G_{\bm{h}}$ in the $i$-th energy state, where $ \bm{h}=\left(h_1,h_2\right) $ is the Fourier expansion order, $ h_1 $ and $ h_2 $ varying from $(-n, -n+1,..., n-1, n)$. $ \bm{b} $ is the reciprocal unit vector, given as $ \bm{b}=\left(b_1,b_2\right)=\left(\left(0,\frac{4\pi}{3d}\right),\left(\frac{2\pi}{\sqrt{3}d},-\frac{2\pi}{3d}\right)\right) $. $\bm{k}_0$ are the reference points taken from the experimental energy-dispersion relations, which are uniformly distributed in the first BZ. We typically take 25 points in each BZ in order to prevent the coarse granularities. Note that in Fig.~\ref{fig:2}, we plot the intensity actually in log scale, $\log(I(E,\bm{k}))$ in order to show the contrast between high and low intensity clearly.

\begin{figure}[htbp]
 \centering \includegraphics[width=9cm]{Fig4.pdf}
\caption{$\Delta$-dependent gap energies $V_{\text{23}}$ (a) and $V_{\text{78}}$ (b) between the second and the third states and between the seventh and the eighth states, which are defined in Fig.~\ref{fig:1}(f). The photonic fraction values, $|C|^2$, are drawn against different $\Delta$ values. The filled circles present theoretical values from the Hamiltonian calculations, whereas experimentally, the gap energy values are taken from two methods shown in open circles. The lower bounds are taken from $ \text{min}(E_{i+1})- \text{max}(E_{i}) $, whereas the upper bounds are from $ \text{min}(E_{i+1}-E_{i}) $, where $ \text{min}$ and $\text{max}$ are functions to take the minimum and maximum values. Note that we cannot extract the lower and upper bound of gap energies from experiments in blue-detuned devices due to the finite resolution and small energy windows. Normalized energy density of states are drawn from experimental (red) and theoretical (blue) band structures at $\Delta = -18.2 $ meV (c) and $\Delta =  7.1 $ meV (d).
 }
\label{fig:4}
\end{figure}

\section{Discussion}\label{sec:discussion}
Now we examine $\Delta$-dependent bandstructure parameters: the gap energy values and energy density of states (DOS) in Fig.~\ref{fig:4}.  $V_{\text{23}}$ and $V_{\text{78}}$ are explicitly indicated in Fig.~\ref{fig:1}(f), which separate between the highest $s$-band and the lowest $p$-band at zone boundaries and between the lowest $d$-band and the second lowest $d$-band, respectively.  The reason we look at these two energy gaps is that the gaps exist at all momentum values, while $V_{\text{67}}$ between the lowest $p$-band and the lowest $d$-band disappears at certain momentum values. Theoretical gap energy values are extracted from the Hamiltonian solutions denoted as filled circles in Fig.~\ref{fig:4}(a) and (b). Experimentally, owing to the spectral linewidths of bands, we take these values with two methods: One values are determined by projecting only peak values at a given wavenumber to the energy axis and finding the gapped regions, which are in the lower side, while other values are obtained by projecting all intensities to the energy axis and finding the distance between the peaks of the allowed bands, which often are higher values. These two values are drawn in open circles as a function of $\Delta$ in Fig.~\ref{fig:4}(a) and (b). Since the gap energy values are proportional to the effective potential strength, at different device locations, the trapping potential strength linearly increases with $|C|^2$. In other words, the more photon-like exciton-polaritons encounter the stronger trapping potential, leading to wider gap openings at the zone boundaries. The $V_{\text{23}}-\Delta$ and $V_{\text{78}}-\Delta$ trends follow very well with the $|C|^2-\Delta$ relation, where $|C|^2$ is the photonic fraction at the global minimum,  $k_\parallel$ = 0.

We also plot the energy DOS by integrating the intensities over momentum values at two extreme $\Delta$ values in Figs.~\ref{fig:4}(c) and (d) with both experimental and theoretical bandstructures. In solid-state systems, the DOS is a useful quantity to explain particle motions and compute various physical parameters such as particle numbers combining with appropriate particle distribution functions. The DOS of the red-detuned device exhibits a series of peaks and the bounded lowest band is isolated by the gapped regions clearly visible in Fig.~\ref{fig:4}(c). However, the DOS of the blue-detuned device consists of two broad regions with occupied energy states.

\begin{figure}
 \centering  \includegraphics[width=9cm]{Fig5.pdf}
\caption{ The tight-binding Hamiltonian fitting results of the nearest-neighbor hopping integral  (a) and the next-nearest-neighbor hopping integral  (b) from the $d$ = 3 and 4 $\mu$m devices directly. (c) The energy bandwidth of the lowest $s$-band $\Delta E_{\text{S}}$ defined in Fig.~\ref{fig:1}(f). The error bars are the standard deviation of the full-width at half-maximum (FWHM). (d) The $\Delta$ and $V_{\text{eff}}$ effect of delocalization. The contour plot of the spatial wavefunction FWHMs ($ \mu $m) is computed for the $d$ = 3 (blue straight line) and 4 (green dotted line) $\mu$m devices. The contour values start from 0.9 (0.95) in the upper right corner for  $d$ = 3 (4)  $\mu$m and end as 0.6 in the lower left corder for both  $d$ = 3 and 4 (green dotted line) $\mu$m.
 }
\label{fig:5}
\end{figure}

For the bounded isotropic $s$-bands, we apply the tight-binding approximation with two fitting parameters, the nearest-neighbor hopping integrals $t$ and the next-nearest-neighbor hopping integrals $t^\prime$ (see Appendix \ref{app:tb}). The energy dispersion is simply written in terms of the $f$-function  with the three neighbor sites, and the fitting results of  $t$ and $t^\prime$ from $d$ = 3 $\mu$m (blue circle) and $d$ = 4 $\mu$m (green square) are collected in Figs.~\ref{fig:5}(a) and (b). The $t$-values for the $d$ = 3 $\mu$m devices monotonically decrease in the positive $\Delta$ sides, but the  $t$-values for the $d$ = 4 $\mu$m devices are about 5-6 times smaller than those for the  $d$ = 3 $\mu$m devices, and the $\Delta$-dependence is weak. In addition, the amplitudes of $t^\prime$ are 10-20 times smaller than those of $t$, indicating the weak overlaps between the next-nearest-neighbor sites.

The monotonic decrease of the $t - \Delta$ relation in the $d$ = 3 $\mu$m devices is qualitatively similar to the $\Delta E_{\text{S}}-\Delta$ plot of the $d$ = 3 $\mu$m devices in Fig.~\ref{fig:5}(c). The bandwidth of the lowest $s$-band, $\Delta E_{\text{S}}$, reflects the strength of the overlap integral between neighboring sites. The greater the overlap is, the stronger curvature emerges, thus the wider band. In the positive $\Delta$ values, since exciton-polaritons are much heavier, they tend to be likely localized, reducing the overlap integral. In order to quantify the delocalization degree of exciton-polaritons, we compute how the $s$-band wavefunctions spatially spread from the center of the trap in Fig.~\ref{fig:5}(d) as a function of $\Delta$ and $V_{\text{eff}}$. These 2D plots teach that the photon-like exciton-polaritons tend to delocalize in a weaker trapping potential, whereas the exciton-like exciton-polaritons are localized in a trap. The difference between $d$ = 3 $\mu$m and $d$ = 4 $\mu$m is bigger in the negative $\Delta$ and shallower $V_{\text{eff}}$. This is partly because the more distant the traps are, the more isolated. Thus, we conclude that, when $d$ = 3 $\mu$m, the $s$-bands reveal the strong overlap integral between the wavefunctions, while for $d$ = 4 $\mu$m, $\Delta E_{\text{S}}$ is limited by exciton-polariton lifetimes.

\section{Conclusion}\label{sec:conclusion}
We successfully engineer artificial bandstructures of exciton-polaritons in a variability of exciton and photon fractions at different detuning values. The experimental bandstructures are completely understood by the two-kind-boson Hamiltonian, which requires to explicitly address both confined photons and free excitons. Our two-kind exciton-polariton lattice system is suitable to investigate a physical system, where both itinerant and localized particles coexist. Furthermore, the higher-orbital bands are accessible, which enables us to systematically examine various orbital physics in the photon-like and exciton-like regimes when they are placed at zone boundaries or near the Dirac cones in the honeycomb lattice. In addition, we may incorporate density and polarization engineering to prepare for Bloch exciton-polaritons that reveals spin order in the non-linear regime at exotic lattice geometries for studying the interplay of topology, spin and interaction.

\begin{acknowledgments}
N.Y.K acknowledges Y. Yamamoto, T. Oka, K. Sota, and A. Burkov for fruitful discussions. H.P., M.P. and N.Y.K are supported by Industry Canada and the Ontario Ministry of Research $\&$ Innovation through Early Researcher Awards. This research was undertaken thanks in part to funding from the Canada First Research Excellence Fund. H. P. thanks Z. Xie for the travel support to visit IQC. M.P. is a recipient of the Richard and Elizabeth Master Graduate Entrance Scholarship and NSERC Canada Graduate Scholarships-Master's Program. K. W., A. S., M. M., M. K., S. K., C. S., S. H. receive the support from the State of Bavaria.
\end{acknowledgments}