\section{Introduction}
In today's time where it's so easy to get lost in the world of the internet  \cite{c1}. To read about something online that is not even accessible to the person is what makes the internet so wonderful. The better thing is being able to experience those things without even leaving your house. This is where virtual reality comes (VR) comes in, reading about hot and humid rainforests in Africa while sitting in Canada is something that was not possible until recently. VR helps a person to experience what he read in real-time through VR devices such as VR headsets \cite{b31}. VR provides a simulated experience by tracking the person's movements and displaying a set of 3D videos to present it as a virtual world. Anything could be shown in this virtual world, ranging from true scenarios like rainforests or mountains to something that was created artificially like games. Another thing we will be discussing is Augmented reality(AR). AR is something that combines real world with the virtual world. It augments the interactive experience and represents the real world with some other computer generated content. The best example for this would be a famous game called Pokemon GO. The game asks people to find new pokemons by traveling in the real world.  \\
Although they both have their benefits, there also come some security threats that cannot be ignored. Both cases' biggest and most common threat would be a user's privacy. In case of VR a person's movement can be tracked whereas in AR a person's location could be tracked as well as permission for access to their could be compromised. It possesses a greater threat for AR as it is easily accessible in different devices such as mobiles and tablets. On the other hand, to access VR one would need a VR headset. Another threat that would be common for both is content displayed. As in both cases content is provided through a third party it could be hacked and changed. Thus providing different content than originally intended. Something that could be improved in both cases would be the authentication process. In both cases, a password only known to the user is used as authentication but it could be improved. VR could use the behavioral pattern such as walking and hand movement to authenticate the user as it can not be copied. AR authentication could be improved based on the devices that are being used such as mobile or tablet could use biometric authentication whereas a PC could use face recognition.\\
As per bailenson \cite{b19} VR can track movements of the user. This nonverbal action can be used for different means, such as identifying what kinds of advertisements are relevant to the user. This could be a huge risk as this could be also be used to impersonate someone. In the case of AR the input and output both could be compromised. As stated by Roesner ankohno \cite{b16}, raw videos are provided to third party which in turn creates an augmented output. The input needs to be verified as it could compromise others privacy also the user could show something that was not intended upon, resulting in privacy leaks. Similarly, the output provided by the third party should also be trusted and verified. A very good example on how AR and VR is potentially used to manipulate people was published in \cite{b11} explaining how a popular game like Pokemon GO which uses AR uses user data to manipulate the foot traffic and also provides the same data to VR industry in order for them to improve their Virtual World.
\\
This paper focuses on analyzing more of such threats and attacks that affect the security of AR and VR and how the data leaks could be used to manipulate a person's action. We will also discuss potential solutions that could be applied or whether no solution could be found.

\subsection{Prior Research}
To the best of our knowledge, there appear to be relatively few Systematic Literature Reviews on the application of AI/VI to the problem of cyber security (SLRs). Viswanathan Karthik \cite{b2} recently published a comprehensive study on security considerations in VI, and Allen, Paul G. Allen recently published a survey paper on security and privacy in AI\cite{b9}. The authors of both studies highlight the challenges and problems associated with the use of security services in centralised architecture in various application domains and provide a comprehensive review of current security methods in AI/VI for such security service applications in areas of authentication, confidentiality, privacy, access control, and data provenance. This study, in our opinion, provides a fundamental understanding of the cutting-edge technologies AR and VR. Furthermore, after learning about AR/VR, the research will concentrate on the security considerations in AR/VR. Furthermore, there are a few additional studies in which specific technologies, such as Google Glass in VR and how to make Glass more safe in terms of data and privacy, are examined.\\

Most of the studies are from 2022 and the oldest one referred here is from 2015, thus all the studies discussed are recent \cite{c2}. This also specifies that most of the studies regarding VR and AR were conducted in recent years. Although the attacks and threats identified are pretty generic showing that VR and AR are still in their early phases and very vulnerable. Some attacks are repeated in studies and more common, such keywords are identified after analyzing multiple papers. This keywords are “Unauthorized access”, “Human Joystick”, “Brute force”\cite{b5}. As this attacks are more common and can not be tracked easily, they could happen often until some steps are taken to prevent this. Another vulnerability are the devices that are used to access VR and AR. According to case study \cite{b4} the attackers could access the videos streamed in VR or the input given to AR. Thus a different output could be provided. In the case of AR the augmentation performed in live feed could give out different results than intended. Similarly, in the case of VR if the image fed in the console could be hacked and a different output could be shown. All such attacks will also be studied in this review.
\subsection{Research Goals}
The goal of this study is to examine existing research on the burgeoning topics of AR and VR. This document was written and summarized with security issues of AR and VR in mind. To concentrate the effort, we established three research questions, as indicated in the \hyperref[Table1]{table I}.

\begin{table}[htbp]
\caption{Research Questions}
\label{Table1}
\setlength{\tabcolsep}{10pt}
\renewcommand{\arraystretch}{2}
\begin{tabular}{| p{14em} | p{3.3cm} |}
\hline
\textbf{Research Question (RQ)} & \textbf{Discussion}\\
\hline
RQ1: What are the most recent AR and VR security applications? & A survey of the most recent practical applications will aid in comprehending the entire scope of AR and VR technology's influence on cyber security. \\
\hline
RQ2: How are augmented reality and virtual reality being utilized to enhance cyber security ? & Every firm needs a safe cyber security infrastructure today. Through visualization, AR and VR technologies may considerably assist IT and security personnel in developing a safe and resilient security architecture. Aside from that, AR/VR technology may assist companies in developing a proactive incident response strategy.\\
\hline
RQ3: What options/methodology are there for managing security solutions in augmented reality and virtual reality ? & The most important data given by AI/VI is the concealment of personal and secret data. Encrypting the data before releasing it can provide security to this operation. In this part, similarly we will look at comparable methodologies.\\
\hline
\end{tabular}
\end{table}

\subsection{Contribution and layout}
This SLR supplements previous research by providing the following contributions for people interested in AI/VI and cyber security to advance their work:\\
- To early 2018, we identified 80 primary studies relevant to AI and VR in cyber security. This list of studies can be used by other scholars to advance their work in this topic.\\
- We then choose 16 primary studies that match our quality evaluation criteria. These studies can serve as useful standards for comparison with related research. We do a thorough evaluation of the data included in the subset of 16 studies and provide the findings in order to represent the research, thoughts, and considerations in the disciplines of AI, VR in terms of cyber security.\\
- We make statements and provide recommendations to help with future work in this area.

The following is how this paper is organised: Section 2 discusses the strategies used to select primary studies for analysis in a systematic manner. Section 3 summarises the findings of all of the primary studies that were chosen. Section 4 addresses the findings in relation to the earlier mentioned study topics. Section 5 finishes the study and makes some recommendations for further research.

\section{Research methodology}

We used the SLR to answer study questions, following the guidelines presented by Paul J. Taylor and tooska dargahi \cite{b20}. We attempted to progress through the review of papers that had been shorted out. Moreover, we will conduct and report steps in iterations to allow for a full examination of the SLR.
\subsection{Selection of primary studies}
The initial research criteria was to find papers by using the basic keywords like “Virtual reality”, “Augmented reality”, “security”,”authentication” and “privacy”. To extract the necessary articles for our research, we attempted to search for papers in various search strings using boolean operators. The search terms were:\\

(“Virtual Reality” OR “Augmented reality” ) AND (“security” OR “authentication” OR “privacy”)
\\

This enabled us to present a variety of papers, some of which were relevant and others of which were not; the same keywords were utilized on distinct sources:\\

- IEEE Xplore\\
- Google scholar\\
- UOG library\\

	Depending on the search platform, the searches were conducted against the title, keywords, or abstract. The searches were carried out on October 5, 2022, and we processed all research published up to that point. The results of these searches were filtered using the inclusion/exclusion criteria described in Section 2.2 and will consider only those papers whose inclusion matches with the searched out paper.

\subsection{Inclusion and exclusion criteria}
Studies included in this SLR must provide empirical findings and may include case studies, AI and VI security problems, or reflections on existing security procedures using AI and VI. They must undergo peer review and be written in English. Any results from Google Scholar, IEEE Xplore, or ScienceDirect will be reviewed for conformity with these criteria, as the above mentioned platform has the chances to provide lower-quality publications or studies that are too old to be included in the review. This SLR will only feature the most recent version of a study. \hyperref[Table2]{Table II} shows the important inclusion and exclusion criteria.

\begin{table}[htbp]
\caption{Research Questions}
\label{Table2}
\setlength{\tabcolsep}{10pt}
\renewcommand{\arraystretch}{2.5}
\begin{tabular}{| p{14em} | p{3.3cm} |}
\hline
\textbf{Criteria for inclusion} & \textbf{Criteria for exclusion}\\
\hline
The report must provide true information concerning AI and VI, as well as the role of security. & Paper focusing on data tracking in AI and VI. Design and Strategies in AI and VI.  \\
\hline
Paper should be concerned more with terms related to confidentiality, security, authentication, etc. & Grey literature such as blogs and government documents.\\
\hline
The article must be a peer-reviewed publication from a conference proceeding or journal. & Non-English Papers.\\
\hline
\end{tabular}
\end{table}

\subsection{Selection results}
Initially, an estimated number of research publications discovered was roughly 1600 papers. Furthermore, following sophisticated screening, the number of relevant publications was decreased to 77. After a quick scan, it was discovered that around half of these articles were either vaguely linked to the issue or not at all relevant. Another significant finding was the lack of research articles focusing on both AR and VR. All of the investigations concentrated solely on one of them. After briefly skimming over the text and finding all of the significant keywords. The research where these keywords were most commonly used and briefly described were picked, and the total number of studies where the findings were relevant amounted to 16.

\subsection{Quality assessment}
The quality of primary research was evaluated using the guidelines established by Kitchenham and Charters [20]. This allowed for an evaluation of the articles' relevance to the research issues, taking into account any indicators of study bias and the quality of experimental results. The assessment procedure was based on that of Hosseini et al. [21]. To determine their efficiency, five randomly selected articles were submitted to the following quality evaluation process:

Stage 1: \textbf{Artificial Intelligence and Virtual Intelligence.} The article must be well-commented and focused on AI and VI or the security implications of AI/VI technology to a specific problem.
\\Stage 2: \textbf{Background.} The study aims and outcomes must be contextualized sufficiently. This will allow for a proper interpretation of the findings.
\\Stage 3: \textbf{AI/VI application}The study must include enough information to provide an accurate explanation of how the AI/VI was applied to a specific situation, which will aid in addressing research questions RQ1 and RQ2.
\\Stage 4: \textbf{Security and Privacy context} In order to aid in addressing RQ3, the paper must explain the security and privacy concerns.
\\Stage 5: \textbf{Data collection.} To assess accuracy, details on how the data was collected, measured, and reported must be provided.
\\

This quality rating criteria was then applied to all additional primary studies that were found \cite{c5,c6}.

\subsection{Data extraction}

All articles that passed the quality evaluation had their data extracted to determine the completeness of the data and the accuracy of the information included within the papers. The data extraction procedure was tested on a few studies initially before being expanded to encompass the whole collection of studies that passed the quality evaluation step. The following categories were assigned to the data: \\

\textbf{Context data:} Information on the study's purpose.\\
\textbf{Qualitative data:} The writers' findings and conclusions that are offered.\\
\textbf{Quantitative data:} Collecting and analyzing data from the study and generalizing the results. \\

\hyperref[methodology]{Figure1} depicts the process of papers selected in our research. It shows our database journal platform for finding papers using the boolean expressions to get the relevant papers. After applying advanced filter, around 80 papers are remaining that are relevant to our study. Reviewing further in details, final 16 papers were selected for this study.

\begin{figure}[htbp]
\centering
\includegraphics[width=9cm,height=9cm]{SLR_primary_studies.png}
\label{methodology}
\mbox{}
\caption{Methodology Process}
\end{figure}

\subsection{Data analysis}
We collected the data held under the qualitative and quantitative data categories to satisfy the goal of addressing the study questions. In addition, We go into details of the papers regarding authentication vulnerabilities that affect AI/VI.\\

\subsubsection{Publication over time}
The AI concept was first coined in year 1956 whereas VI was invented in 1976. Papers regarding AI/VI in terms of security, authentication or privacy were mostly found after the 20th century. This may highlight the newness of the ideas concerning cyber security applications for AI/VI .\\

Initially, an estimated number of research publications discovered was roughly 80 papers. After a quick scan, it was discovered that around half of these articles were either vaguely linked to the issue or not at all relevant. Another significant finding was the lack of research articles focusing on both AR and VR. We can see from the final 16 publications that there was little study done between 2015 and 2017. Furthermore, we can see that the graph of publishing has not significantly improved, indicating that researchers may be suffering, but this year has witnessed an explosion in research, with the largest number of papers published in this year, as seen in \hyperref[publication]{Figure2}.

\begin{figure}[htbp]
\centering
\includegraphics[width=9cm,height=7cm]{publicationOverTime.png}
\label{publication}
\mbox{}
\caption{The number of primary studies that have been published over time}
\end{figure}

\subsubsection{Significant keywords counts}
An analysis of keywords was done across all 16 studies in order to obtained generalize themes among the studies. As we can observe in the \hyperref[Table3]{Table III} quantitative and qualitative analysis allows us to analyse privacy problems, with authentication being the most commonly discussed topic \cite{c3,c4}.
\section{Finding}
All the primary research papers were read in full and relevant analysis is provided in \hyperref[Table3]{Table 3}. The table provides a view on what security threats are faced while using AR and VR. The major threats to these technologies were identified and then the studies were grouped in separate sections. Among these sections further subsections were classified to simplify the data. All the studies with authorization and authentication were grouped together in one section. The next section comprised of all the studies related to data storage, data leaks and data manipulation. The remaining categories are standalone and not sub-categorized which are object recognition and software vulnerabilities.
\\
It was found out that the major issue faced by this technologies is user authorization comprising of 43.8 \% of the total studies. The second group can be classified regarding data leaks and data manipulation resulting in total of 37.5\% of research papers.  The studies related to object recognition make a total of 12.5\% and the last section of software vulnerabilities makes a total of 6.3\% of the total studies. \hyperref[piechart]{Figure 3} depicts the graphical representation in form of pie-chart of above calculation.

\begin{table}[htbp]
\caption{The key research main results and topics}
\label{Table3}
\setlength{\tabcolsep}{10pt}
\renewcommand{\arraystretch}{2.5}
\begin{tabular}{|p{1cm}| p{14em} | p{1.5cm}|}
\hline
\textbf{Primary Study} & \textbf{Key Qualitative \& Quantitative Data Reported}& \textbf{Types of Security Applications}\\
\hline
\cite{p1} & Using the devices OS(most probably mobile phone) to recognize and authorize a user or an other object as desired. The device used for motion capture in the study is Microsoft Kinect and the application used for recognition of image is Privacy Googles. & Object recognition   \\
\hline
\cite{p2} & Identifying the threat in industrial AR such as unauthorized read access and building an AR edge computing architecture using edge servers to prevent such threats. & Unauthorized access\\
\hline
\cite{p3} & Object localization and object recognition in AR with the help of radio based object recognition. The device used here could be made by using localized equipment available in factories such as sensors of the mobile devices and a field technician. & Object recognition
\\
\hline
\cite{p4} & Preventing the tracking of non verbal data in VR by implementing government policies and self regulation from the companies. & Non verbal data leaks.
\\
\hline
\cite{p5} & Understanding the privacy law and regulations in place by government  and understanding the framework and limitations with those frameworks as location leaks and information leaks through AR. & Privacy issues
\\
\hline
\cite{p6} & Finding VR risks such as data interference and manipulation, analysing privacy policies and conducting public surveys to identify their knowledge about VR threats and privacy policies. & Privacy issues
\\
\hline
\cite{p7} & Proposing the convergence of Social Network and VR by making digital avatars and how they could lead to informational, physical and associational privacy threats which could only be prevented by legal policies and laws. & Data leaks
\\
\hline
\cite{p8} & Constructing an architecture for user authentication in VR such as 3D password, pattern lock and PIN. The hardware used is Leap Motion and Oculus Rift DK2 & user authentication
\\
\hline
\cite{p9} & A study conducted between 5 people regarding their perception about VR’s data collection, its’s data collection methods and what kind of data is collected. Moreover which VR system is more trusted by the users was also discussed. & Data storage
\\
\hline
\end{tabular}
\end{table}

\begin{table}[htbp]
\setlength{\tabcolsep}{10pt}
\renewcommand{\arraystretch}{2.5}
\begin{tabular}{|p{1cm}| p{14em} | p{1.5cm}|}
\hline
\cite{p10} & Checking the current user authentication protocols in AR and VR Head mounted displays, identifying their vulnerabilities and proposing a solution based on Zero - Trust algorithm(ZeTA). & User authentication
\\
\hline
\cite{p11} & Testing popular AR and VR headmounts such as oculus Quest, Hololens, Google Glass, Valve Index and others to find out all of them have common vulnerabilities and exposure(CVEs). They also have other drawbacks such as no multi factor authentication and flawed privacy policy. & Vulnerable software
\\
\hline
\cite{p12} & Understanding the AR output risk due to buggy and malicious third party software leading to obscuring of user’s view of the real world. Thus, proposing a solution by introducing their own AR platform Arya, which designs the output policy module.  & AR output
\\
\hline
\cite{p13} & Constructing an access control framework called Privacy Manager. The aim is to restrict the users access to AR application in certain environments where there is a risk of private data leaks. The framework creates minimum system overhead for the mobile device while providing location awareness. & Access authorization
\\
\hline
\cite{p14} & Analyzing different authentication methods in metaverse such as Information-Based authentication, Biometric authentication, Multi- Model authentication, Gaze- Based authentication and discussing there advantages and disadvantages. & User authentication
\\
\hline
\cite{p15} & A study on VR authentication through Task driven biometric authorization such as hand gestures and trajectory based authentication providing results based on 135 samples. & Authentication
\\
\hline
\cite{p16} & An experiment for eye based biometric authorization for VR application based on C++/OpenGl by implementing it on Valve’s open VR. The sample taken was in 60 Hz with non uniform time intervals. & User authorization
\\
\hline
\end{tabular}
\end{table}

\begin{figure}[htbp]
\centering
\includegraphics[width=9cm,height=7cm]{pie-chart-2.png}
\label{chart of theme of primary studies}
\mbox{}
\label{piechart}
\caption{The number of primary studies that have been published over time}
\end{figure}

\section{Discussion}
After the initial research, some of the keywords that were found relevant are “Privacy” \cite{b5}, “Authentication” \cite{b2},  “Data leaks”. All three of these keywords are found to be most vulnerable in AR and VR security. As we continued using these keywords, we also discovered several unrelated publications that used the same nomenclature but had distinct study motivations and conclusions\cite{b17} \cite{b18}  \cite{c7} \cite{c8}.
\\

\textbf{RQ1: What are the most recent AR and VR security applications?}
\\Considering the use AR and VR in security aspects for real world, there are very limited applications that could be found. One of the application for AR that’s recent but a very prime example for real life security is using AR devices for surveillance. The problems with regular surveillance is that the image available is in 2D and needs a good amount of camera for a large floor plan. The other major drawback is the competence of the operator. As there is always a risk of error with human supervision, the AR devices which could be used for face identification or object identification as well as could also identify the environment and with the use of proper hardware could also keep view of a wide surface area in 3D.\\
One such framework is discussed in \cite{b26}, where the operator could use AR technology to provide a 3D interface with the help of Virtual and Augmented reality. This is done by using a wide array of sensors for video and audio surveillance. The model works by initially creating a sample 3D model of the environment. This model could be made complex or could be kept simple based on the operator’s requirement.\\
Once the 3D model of the environment is created successfully the model than uses several service discovery protocols to find all the camera in the environment create a mesh network. For each camera found in the real world the model would then represent a virtual representation of the cameras in 3D format for operator where the operator could access the whole floor plan from a single point of view. The 3D world represented to the operator is connected to the real world image thus providing an easier access for operator to keep surveillance over a wide area easily.\\
One other implementation of VR for real world physical security would firearm training simulation\cite{b27}. It is not so much of a surprise that firearm training in real world could not be done daily due to high cost of firearms. Thus the training itself is short and spaced over time. Moreover, the occurrence of surprise attacker can not be simulated in real world gun ranges thus not providing proper training. Although this could be improved with the use of VR simulation.
This approach allows more effective training as multiple aspects could be introduced in a virtual world and can be controlled like pedestrians in a street, surprise occurrence of vigilante’s other aspects, providing a proper real life simulation for the trainee. This results in professional training with a higher emphasis on daily training while keeping the cost of training to a minimum. \\

\textbf{RQ2: How are augmented reality and virtual reality being utilized to enhance cyber security ?}\\
It is clear that with current scenario of world the importance IT department in any company has significantly increased. With the introduction of IT department in almost every field the risk of it being hacked also increases creating a major problem. For employees that are not well educated with cyberattacks it becomes an alarming problem as to simple cyber attacks like phising could lead to a huge data leak. To prevent such scenarios, it is necessary for any company to have employees that are well trained regarding cyber attacks.\\
\mbox{}
Training through AR/VR is being widely used and seems to be more efficient as it gives a virtual depiction as well as provides audio to support it’s curriculum. As discussed in \cite{b28}, AR is being used in a wide array of activities like Order breakdown, Warehouse operations, Quality control and much more. Another benefit is that it could be accessed through a large number of devices such as Tablets, smartphones, data goggles providing people with visualization of decision making simulation and also providing the effects of the decisions.
Thus, it is also possible to provide cybersecurity training through AR and VR devices. All we need is a proper curriculum. Considering multi generational businesses and a wide range of age in different businesses, the virtual depiction could help to provide a better understanding of the significance of cyber security.\\
\mbox{}
Another major application of AR is for the visualization of security data for cybernetic systems \cite{b29}. The study itself was conducted on students studying information security. The study focused on two factors which were accuracy and speed. As a result, the operator's contact with the data collected by information security systems should improve in terms of performance indicators (accuracy and speed). It was concluded that this approach should be applied for large volumes of data which have a greater significance towards accuracy and speed. \\

\textbf {RQ3: What options/methodology are there for managing security solutions in augmented reality and virtual reality?}\\
\mbox{}
The benefits of employing augmented reality tools are abundantly clear. Wearables enable workers to complete tasks more quickly, adaptably, and safely. They demonstrably make less mistakes and are happier employees. But as the name implies, smart glasses only function by gathering and analysing data. Therefore, a fundamental concern for any businesses employing AR technology is data security.\\
The best way to prevent data leaks is user authentication. Although, in the case of AR/VR the authentication could be done by hand gestures or body movements or by entering patterns and PIN which could easily be seen by a bystander or could also be recorded by someone else which could lead to someone else accessing the data. Thus, as discussed in \cite{p10}, the Zero – trust algorithm(ZeTA) could seem to prove a good fit for such cases. ZeTA is a knowledge-based authentication technique, which means that, like text passwords, the user must memorise a secret.\\
\mbox{}
The study itself used two people at a time in a single room to analyze whether one could analyze the second person could analyze the first person and vice versa.  In the experiment person 1 was authenticated three times while person 2 observed and then the positions were changed. Finally both were asked to answer questions in a survey.\\
\mbox{}
Another good methodology for authentication is using eye based biometric authorization in VR \cite{b16} as it is not possible for an outsider to track eye movements. The projet was modified in C++/OpenGl and Valve’s Open VR API was used with any VR HMD. The authentication process started with data collection then eye movement classification after which feature extraction is performed and then they try to match score calculations before taking a decision. The experiment itself was able to obtain 60 Hz signals with non- uniform time intervals but it was stated that the past experiments achieved 250 Hz signals and should be improved upon. Thus considering the current VR application, the eye based biometric authorization could be taken as a good example.

\section{Future research directions of AI/VI security}

As discussed above that AR and VR could be implemented in a wide range of fields like training to surveillance. This applications are a proof that although they are a fairly new technology but are widely applicable in the new digital world \cite{c9,c10}. One such application which takes this to another level completely changing the scenario of current world is Smart Cities.\\

A Smart \cite{b28} is a technologically advanced urban area that uses Information and communication technologies(ICT) with different kind of sensors for collecting specific type of data. This data is then used to increase the operational efficiency of the city and understand the requirements for public welfare. With the rapid advancement of technology in current era, it has become crucial to understand the dynamic infrastructure of a urban area and get certain data like air pollution, energy consumption, video surveillance and other aspects. For such analyses and data collection AR can be used to enhance the process \cite{b29}.\\

\textbf{Integration of AR and IoT for smart cities.} A distinctive immersion into Internet of Things (IoT) applications is provided by the integration of AR into smart cities. According to recent research, this can serve as a framework for an interactive demonstration of how public services like street control, video surveillance, solid waste collection, and parking management can be carried out and managed from a single platform, improving the safety, cleanliness, and livability of cities. The main purpose of a smart city is to connect everything together. This gives the users a easy and hassle free access to the all the data of city. As this provides an ease of access but also creates new risks and threats in the domain of cyber security. Users' and organisations' careless behaviour in smart cities can expose the entire city to cybercrime risk. This challenges and risk are duly noted and explained in \cite{b30}. It states that due to high reliance of smart city on ICT, several security threats like cyber attacks and leakage of private information could create a negative impact on the quality of living.\\

A review of such threats and risks was done in \cite{b27}, which states that the application of AR for smart cities can be categorized into five main classifications, including tourism, system monitoring, system management, education and instruction, and mobility. It shows the cross connection of this fields can cause major cyber attack threats. As every field and information would be interconnected then one vulnerability could lead to a major cyber attack. The study itself discusses how each area has its own pros and cons but also states that although there has been several studies regarding the attacks and proposes the solution but most of the previous work only attempts to prove concepts rather than providing with a analytical approach. Thus it could be stated that in the future we would need more research on  practical and analytical solution as the idea of smart cities keep becoming a reality.

\section{Conclusion}
This research has identified how AR and VR are widely used and are being applied to various sectors in real life. Although it seems that the technology helps in simplifying the daily needs and provides more entertainment with the help of immersive games and videos, they bring a huge drawback as the technology is still in it’s early phases. The research itself identifies some major security issues in the technology itself such as Authorization problems, data leaks, data manipulation, vulnerable software and unauthorized access. The best solution to such problems is to implement an adaptable framework that lets domain administrators and application developers implement access control policies to families of mobile applications. Implementing some policies and laws for preventing data leaks. One more solution is to create awareness among general public on how AR and VR work and what policies are implemented by the companies providing Hardware for such applications.\\
Apart from this several AR and VR security applications in real world as well as how AR and VR can be used to spread awareness regarding cyber security. The best way to apply this technologies in real world is to provide training in various sectors such as factories, firearms trainings, sales coaching and other areas. Considering the awareness regarding cyber security, a training module could be developed which provides virtual course on how the cyber attacks work and how they can be prevented. This could be most helpful in the current scenario as the society is going towards becoming a technologically advanced society.\\
Another big aspect is the application of AR in integration with IoT in smart cities. To simplify the concept this integration helps in connecting all the data collection methods and giving an easy access to all that data in a single place. Applying AR for data analysation provides a more immersive experience towards services like maintenance and installation of various infrastructures. Depending on the maintenance methods, a population of wireless nodes installed in a smart city setting can sense various factors thanks to the IoT infrastructure. Central data transport utilises any type of backbone city network. Additionally, a wireless infrastructure is provided to give city residents' and maintenance employees' smartphones network access via numerous access points (also known as WiFi APs or base stations) dispersed across the city. With the help of these components, it is possible to portray the sensors and actuators in a smart city as services that can be quickly accessible through graphical user interfaces based on actual images, without the need for a continuous connection to the backbone network.\\
Thus concluding that the AR and VR technologies have taken its roots in the industry providing a number of uses, it’s still vulnerable to many threats. To implement this technologies on a larger scale we still need to research this problems and provide a analytical solution to implement them bigger real life projects like smart cities.

\textbf{Declarations of interest}
\\
\mbox{}
There is no conflict of interest.
\\

\textbf{Acknowledgement}
\\
\mbox{}
None.


\section{Ablation Experiments}
To understand the significance of different elements and design choices in our framework, we perform a series of further ablation studies. We next elaborate on the settings, report the results and derive conclusion from each of these experiments.

\subsection{Significance of View-direction (4\textsuperscript{th}) Plane}
As we describe in Sec.~\ref{sec:planar_representation}, Chan \etal~\cite{chan2022eg3d} use a tri-plane model constituting three positional feature planes $\{P_{xy},P_{xz},P_{yz}\}$ for representing human face scenes.
We propose to add a 4\textsuperscript{th} feature plane $P_\text{dir}$ whose two axes span the 2D space of possible viewing directions for each point $p$. We do that in order to accommodate effects like non-uniform scene illumination, and subsequently to improve reconstruction performance, as we show in Fig.~\ref{fig:viewdir_effect}.

To evaluate the effectiveness of $P_\text{dir}$, we compare the proposed quadri-plane model with a tri-plane model comprising only the three positional planes. Note that we did not include SR model $F$ in this ablation experiment. We present a qualitative side-by-side comparison in the second part of the attached video file, and provide a quantitative performance evaluation in Tab.~\ref{tab:viewdir_plane}, where we measure the performance of the two models on the task of reconstructing $1400$ novel views corresponding to $7$ scenes, using the PSNR, SSIM and LPIPS metrics. The results of all metrics indicate a clear advantage for the proposed quadri-plane model.
\input{tables/viewdir_plane}

\subsection{Effect of Feature Planes Resolution $N$}
The spatial dimensions $N$ of the positional feature planes in the proposed quadri-plane representation model strongly affects representation performance, as we discuss in Sec.~\ref{sec:planar_representation} and demonstrate in Fig.~\ref{fig:plane_res}.
Too small planes cannot represent high visual frequencies. On the other hand, using $N$ values much larger than the resolution of the captured scene images results in visual artifacts when rendering novel views, as some plane regions remain uncovered by the information in ${\cal I}^s$.
As a rule of thumb, we set $N$ to be $1-2$ times the resolution of the captured images $I\in{\cal I}^s$. To validate this choice, we compare reconstruction performance over $1400$ novel views corresponding to $7$ scenes when setting $N$ to be a quarter, the same as or $4$ times the resolution of $I$. The representations in this experiment were learned using images $I\in{\cal I}^s$ of size $400\times400$, and we use each learned feature plane as is, without super-resolving it with $F$. The results in Tab.~\ref{tab:plane_res} suggest setting an appropriate resolution value $N$ has a significant effect, and support our rule of thumb.
\input{tables/plane_res}

\section{Quantifying Across View-direction Consistency}\label{sec:AVI}
The proposed method \emph{guarantees by design} to yield geometrically consistent results across different view-directions, as our SR model operates directly on the volumetric representation of the model, before this representation is being used for rendering 2D views of the scene. We demonstrate this in the video visualizations included in this supplementary material.

To further quantify this advantage, we introduce in Tab.~\ref{tab:reconst} the Across View-direction Inconsistency (AVI) metric, that measures the degree of geometrical inconsistency between every pair of test frames corresponding to adjacent viewing directions, and use it to verify the consistency benefit of our approach.
We compute this score using the formula in Eq.~\ref{eq:inconsistency} which we repeat here for convenience, followed by an elaborated description of each of its parts:
\begin{equation}\nonumber
    \text{AVI} = \frac{1}{n}\sum_{r\in{\hat i}_{\text{sr}}^k(r)}M_{k,k+1}\cdot \|\text{Warp}\bigl({\hat i}_{\text{sr}}^k\bigr)(r)-{\hat i}_{\text{sr}}^{k+1}(r)\|_2.
\end{equation}

Since we have access to the Blender models used to generate the synthetic scene datasets, we are able to compute ground truth optical flow motion vectors both for the forward flow $f_{k\rightarrow k+1}$ and for the backward flow $f_{k+1 \rightarrow k}$ between all adjacent view-direction pairs $k,k+1$. We utilize the \emph{backward flow} to warp each image to the next adjacent viewing direction via
\begin{equation}
    \text{Warp}(\hat{i}_{\text{sr}}^k)(r) = \hat{i}_{\text{sr}}^k(r + f_{k+1 \rightarrow k}(r)),
\end{equation}
where $r$ denotes the 2D pixel location.

Even though we have access to the ground truth motion vectors, the warping can still introduce ghosting artifacts due to occlusions, i.e., pixels in image $k$ that have no correspondence in image $k+1$. To prevent errors stemming from the ghosting artifacts from dominating the AVI metric, we introduce an error mask $M_{k, k+1}$, computed as follows: First, by using the \emph{forward flow}, we count how many pixels get mapped to the same pixel $r$:
\begin{equation}
    \text{Count}_{k, k+1}(r) = \#\{ q | \lfloor q + f_{k\rightarrow k+1}(q)\rfloor = r\}.
\end{equation}
Pixels from viewing direction $k+1$ for which this count is zero do not have a correspondence in viewing direction $k$. Thus, our mask is defined as
\begin{equation}
    \Tilde{M}_{k, k+1}(r) = \begin{cases}
     0, &\text{if Count}_{k, k+1}(r) = 0\\
     1, &\text{otherwise}
    \end{cases}.
\end{equation}
Since the floor operation in the count computation can still lead to some ghosting due to subpixel occlusions, we apply a morphological closing followed by erosion to the mask, using a "cross" structuring element of radius 1. The final mask in Eq.~\ref{eq:inconsistency} is thus given by
\begin{equation}
    M_{k, k+1} = \text{erosion}(\text{closing}(\Tilde{M}_{k, k+1})).
\end{equation}
When computing the AVI score, we only consider unmasked elements for the mean computation, i.e. $n = \#\{ r | M_{k, k+1}(r) = 1\}$ in Eq.~\ref{eq:inconsistency}.
The effects of the raw mask and the different morphological operators are visualized in Fig.~\ref{fig:masks}.
Note that we use the same motion vectors, and thus also the same masks, when computing the AVI score for each of the compared methods.
\begin{figure*}
\centering
\begin{tabular}{c c c c}
    \includegraphics[width=0.22\textwidth]{images/no_mask.png}&
    \includegraphics[width=0.22\textwidth]{images/with_raw_mask.png}&
    \includegraphics[width=0.22\textwidth]{images/with_closing.png}&
    \includegraphics[width=0.22\textwidth]{images/with_erosion.png}\\
    (a) Without masking &
    (b) With $\Tilde{M}_{k, k+1}$&
    (c) With closing &
    (d) With closing + dilation
\end{tabular}
\caption{\textbf{Effect of Mask $M_{k, k+1}$.} We use masking to  remove ghosting artifacts (\eg the ship's masts) in the warped image (a) prior to calculating the error. Here we illustrate the effect of using the mask with or without processing it with morphological image operators (b-d), and show that applying the fully processed mask yields significantly reduced ghosting artifacts (d).}
\label{fig:masks}
\end{figure*}
\section{Synthetic Dataset 3D Scene Models}
Our approach learns to super-resolve the volumetric representations by training on a set of training scenes ${\cal S}_t$. Being a deep-learning-based approach, it requires sufficient amount of training data to be able to generalize from the training scenes to unseen scenes. To meet this data requirement,
we augment the synthetic scenes from \cite{mildenhall2020nerf} with 8 additional sequences generated from the following freely available Blender scenes
\begin{itemize}
    \item \url{https://free3d.com/3d-model/bugatti-chiron-2017-model-31847.html}
    \item \url{https://free3d.com/3d-model/black-dragon-rigged-and-game-ready-92023.html}
    \item \url{https://free3d.com/3d-model/professional-scene-with-coca-cola-bottle-57999.html}
    \item \url{https://free3d.com/3d-model/gibson-es-335-816888.html}
    \item \url{https://free3d.com/3d-model/fuzzy-bear--10429.html}
    \item \url{https://free3d.com/3d-model/harley-davidson-low-rider-84874.html}
    \item \url{https://free3d.com/3d-model/holiday-beach-cartoon-scene-431138.html}
    \item \url{https://free3d.com/3d-model/donut-503129.html}
\end{itemize}

Training, validation and test sequences were generated using the Blender script provided by Mildenhall \etal~\cite{mildenhall2020nerf}.
\section{Additional Evaluations}

\subsection{Qualitative Assessment of View-direction Consistency}
In Tab.~\ref{tab:reconst} we quantitatively quantify the capability of each of the compared methods to produce super-resolved outputs that are geometrically consistent across viewing directions. While this consistency is less important when examining each output independently, it becomes crucial when merging the outputs into a video trajectory, where geometric inconsistencies across adjacent viewing direction manifest themselves as prominent fidgeting-like temporal artifacts. Therefore, to demonstrate the advantage of our volumetric SR approach, we
render a series of `fly-over' videos corresponding to the rendered scenes.
In the the first part of the attached video we compare our super-resolved outputs (on the left) with renderings corresponding to various baselines (on the right), demonstrating the significance of producing geometrically consistent reconstructions. Note how reconstructions by existing image SR methods on the left hand side exhibit jittering artifacts, in contrast to our method on the left hand side.

\subsection{Additional Visual Results}

\section{Conclusion}
In this work we introduce and investigate the novel task of neural volume super-resolution -- rendering high-resolution views corresponding to a scene captured at low resolution. We introduce a neural scene representation model that utilizes 2D feature planes, which we learn using LR image captures of the scene. Then to render novel scene views in high-resolution, we propose to super-resolve the 2D feature planes using a CNN model trained on a dataset of 3D scenes. Operating in the volumetric representation domain guarantees the geometrical consistency of the SR rendered images across different viewing directions. We validate our approach both quantitatively and qualitatively on a diverse set of unseen 3D scenes, demonstrating the significant performance improvement it yields over existing approaches.
Beyond the advantage in performing volumetric scene super-resolution, our work reveals the potential of this approach for reconstructing scenes captured under other types of realistic degradations (\eg noise, blur), which we hope to investigate in future work.
\section{Experiments}\label{sec:experiments}
\input{figures/results}
We next demonstrate the advantage of using our volumetric approach for super-resolving 3D scenes captured in LR.
To this end, we use the eight synthetic scenes available on the NeRF~\cite{mildenhall2020nerf} project page, as well as additional eight scenes rendered from 3D Blender models available online\footnote{Links to the these freely available Blender models are provided in the Supp. material. Our entire project code will become available online.}.
These scenes depict complex objects with fine details and textures, which are crucial for demonstrating and evaluating the SR quality.
For each scene $s$, we use 100 training images with dimensions $400\times400$ as the ground truth HR captures ${\cal I}_{\text{hr}}^s$, and then downsample each image by a factor of $4$ using the default bicubic kernel to simulate the set of LR captures corresponding to the scene, ${\cal I}_{\text{lr}}^s$.

\input{tables/reconstruction}
For our quantitative evaluation, we take a cross-validation approach, by training $4$ independent models (including both representation model $G$ and SR model $F$) on subsets of the $16$ scenes, while holding out either the `mic', `ship', 'chair' or `lego' scenes for validation.
We evaluate all methods on $200$ test images corresponding to novel views of each of these four unseen scenes, while comparing our approach with methods and baselines representing several different approaches, by using a set of evaluation metrics. We report the results in Tab.~\ref{tab:reconst}, and elaborate next on the metrics we employ, as well as on the baselines we compare to.
\myparagraph{Evaluation Metrics.}
We use PSNR and SSIM to measure how well methods minimize the error with respect to the corresponding GT HR images. To evaluate perceptual quality, we employ the LPIPS~\cite{zhang2018lpips} score, where lower means better.
We then introduce a new metric to demonstrate a key advantage of our approach, \ie that operating in the volumetric domain produces geometrically consistent renderings across different view directions. We therefore quantify the degree of geometrical inconsistency between every pair of test frames corresponding to adjacent viewing directions, by harnessing the GT optical flow that can be extracted for every adjacent frames pair in our synthetically rendered scenes;
we warp the first frame onto the second and then calculate the induced per-pixel root mean squared difference across color channels, resulting in per-pixel error map.
We then use the optical flow again to mask out map regions corresponding to object parts waning or waxing between the two frames, resulting in an error map like the one presented in Fig.~\ref{fig:teaser}. Finally, we compute the average to yield an Across View-direction Inconsistency score (AVI for short):

\begin{multline}\label{eq:inconsistency}
    \text{AVI} = \frac{1}{n}\sum_{r\in{\hat i}_{\text{sr}}^k(r)}M_{k,k+1}\cdot \|\text{Warp}\bigl({\hat i}_{\text{sr}}^k\bigr)(r)-{\hat i}_{\text{sr}}^{k+1}(r)\|_2.
\end{multline}
Here ${\hat i}_{\text{sr}}^k$ and ${\hat i}_{\text{sr}}^{k+1}$ are a pair of adjacent frames obtained through SR by a certain method, $\text{Warp}(\cdot)$ is the warping operator based on the GT optical flow, $M$ is the binary mask, filtering out justifiably inconsistent image regions and $n$ is the number of pixels. Please refer to the Supp. material for a detailed description of each operator.
\myparagraph{Baselines.}
We compare our approach with the leading single image SR methods, including the state-of-the-art SwinIR method~\cite{liang2021swinir} which uses visual transformers, the prominent SRGAN~\cite{ledig2017srgan} method\footnote{We use the distortion-minimizing training configuration (without an adversarial loss) from SRGAN to train their SRResNet architecture, to allow a fair comparison with our method, which is trained to minimize distortion with respect to the GT images.} and the EDSR method~\cite{lim2017edsr}, whose network architecture we employ for our model $F$.
Since the LR scene captures can be viewed as pertaining to an LR video sequence, we further compare with a state-of-the-art method for video spatio-temporal SR~\cite{geng2022video_sr}. We repeat the evaluation twice with all of these existing methods; once by evaluating using their models trained from scratch on our training set, and a second time using their pre-trained models available online, which were trained on much larger datasets (denoted by \emph{Data-Boosted} in Tab.~\ref{tab:reconst}).
We also report the performance of all methods relative to the simple bicubic image interpolation baseline in parenthesis.

Finally, we compare with two additional alternatives:
Either naively rendering high-resolution images using the learned LR feature planes (denoted \emph{naive}), or first using EDSR~\cite{lim2017edsr} to super-resolve each of the captured LR images in ${\cal I}_{\text{lr}}^s$ independently, and then using the resulting \emph{super-resolved} set to learn our quadri-plane representation model (denoted \emph{pre-SR}).

The PSNR and SSIM values we report in Tab.~\ref{tab:reconst} indicate that our approach yields lower error compared to all other baselines. The performance gap is especially prominent with respect to methods operating in the 2D domain (all but the `naive' and `pre-SR' baselines), with a PSNR increase of more than $1$db over all image SR methods trained on our training set.
This suggests an inherent advantage for operating in the volumetric, geometrically consistent domain. Nonetheless, using our method and super-resolving the feature planes further improves over \emph{all} baselines including the ones operating in 3D, as supported also by the visual comparison in Fig.~\ref{fig:results}.

In terms of perceptual quality, models of existing methods that were trained on large datasets yield better LPIPS scores, but our method scores on par with the leading baselines that share the same training data. Finally, the across view-direction inconsistency (AVI) scores reveal an inherent advantage for methods operating in the volumetric domain. Note that the AVI score cannot stand by itself, and one should examine it alongside the corresponding PSNR score. This is because one trivial way to achieve perfect consistency is to wipe out the image content in all frames. That explains why the (3D-based) `naive' baseline performs slightly better than us in terms of AVI, as our method yields sharper images that contain finer details.

In Fig.~\ref{fig:results} we compare our results with those of the leading evaluated baselines visually, by showing rendered LR novel views by our model (left column) followed (from left to right) by their versions super-resolved by two image SR methods, a video SR method, simple rendering using the LR planes (`naive') and our method (right column). Note that our method reconstructs finer details compared to all other baselines, and goes as far as reconstructing delicate structures such as the dragon's horns and tail, elaborated Lego parts, and even a subtle dent near the microphone's volume knob. Please refer to the supplementary material for additional comparisons and visualizations.

\myparagraph{Ablation Study.}
We further analyse the effect of using different feature plane sizes beyond the visual demonstration in Fig.~\ref{fig:plane_res}. We empirically find that the plane size should be set to $1$-$2$ times the resolution of the captured images set, while diverging from this range by a factor of $4$ (in either direction) yields a drop of $\sim1.7$db in PSNR. Similarly, we analyse the significance of the 4\textsuperscript{th} (view-direction) plane $P_\text{dir}$ introduced in this work, by averaging the performance gap over four scenes ($800$ images), and find that utilizing it leads to an increase of almost $3$db in PSNR, which aligns with the visual demonstration in Fig.~\ref{fig:viewdir_effect}. We provide the full results corresponding to these experiments and report additional ablation experiments in the supplementary material.

\section{Introduction}
\label{sec:intro}
Reconstructing latent high-quality images from images captured under non-ideal imaging conditions is a broadly studied research field. A large body of existing work explores methods for removing image noise \cite{buades2005non_local_denoising,kawar2021stochastic_denoising}, sharpening blurry images \cite{tao2018scale_recurrent_deblur,bahat2017deblurring_reblurring}, and increasing image resolution \cite{michaeli2013blind_recurrence,ledig2017srgan}. This direction includes recent works that are based on deep learning~\cite{ledig2017srgan,tao2018scale_recurrent_deblur}, mapping degraded images to their latent non-degraded counterparts, and have led to a significant leap in performance. At the heart of these methods typically lies the application of a series of 2D convolution operations to the degraded image \cite{lim2017edsr,ledig2017srgan}, which eventually yields the reconstructed image at a higher resolution.

Concurrently with research on image reconstruction, a rapidly growing body of work explores neural rendering, allowing not only to reconstruct a single image but unseen novel views from a set of observed images. In their work on Neural Radiance Fields (NeRF) \cite{mildenhall2020nerf} in 2020, Mildenhall \etal showed that state-of-the-art rendering quality can be achieved even with a lean Multilayer Perceptron (MLP) architecture as a coordinate-based representation for a 5D radiance field. Many works have since proposed improved and more efficient ways to represent 3D scenes based on a given finite set of 2D images, including methods that tackle modified lighting conditions \cite{srinivasan2021nerf_relight} or support dynamic scenes \cite{ost2021scene_graphs}. These existing methods directly supervise the scene representation using the observed images.

\input{figures/teaser}

In this work, we explore an alternative direction
at the intersection of image reconstruction and neural rendering, and propose the new task of neural volume super-resolution - learning to super-resolve neural volumes.
Specifically, given a finite set of low-resolution images corresponding to a 3D scene, we aim to render high-resolution images of the scene corresponding to \emph{infinitely-many} possible viewing directions, including novel unseen ones. Rather than operating on the rendered (2D) images and upsampling these images after generation, we propose a novel approach to tackle this task, by operating directly on the volumetric (3D) scene representation. An advantage of this approach, illustrated in Fig.~\ref{fig:teaser}, is that it guarantees by design a \emph{geometrically consistent} image reconstruction across all viewing directions, in contrast to operating in the 2D domain and separately reconstructing each rendered image.

As a practical and efficient approach for enabling upsampling neural volumes, we propose a new 2D plane-based scene representation that allows us to employ 2D convolutional neural networks. Specifically, we represent a 3D scene by implicitly associating each point in the 3D volume with a density value and a (view-direction dependent) RGB value following existing radiance fields works, while investigating an alternative way to infer these values. Chan \etal \cite{chan2022eg3d} accumulated features for each 3D point by projecting its location onto three \emph{2D} feature planes, for representing human faces. We instead propose a plane representation that allows representing general scenes, and introduce a fourth feature plane to enable view direction dependency. Given a desired viewing direction, we can reconstruct each image pixel by marching along its corresponding ray from the camera, taking into account the cumulative effect of the density and RGB values predicted by our model for each point along the ray.

Rendering a \emph{super-resolved} image from the proposed scene model, that was learned from a set of \emph{low-resolution} images, requires a single modification to our pipeline, while the rest remains unchanged: Before projecting each point in the volume onto the 2D planes to infer its density and RGB values, we super-resolve the three 2D planes using a learned super-resolution module that operates in feature space. This approach allows us to take advantage of existing methods explored for 2D super-resolution over the years.

We train our \emph{Neural Volume Super-Resolution} (NVSR) framework end-to-end using low-resolution and matching high-resolution image pairs corresponding to synthetic scenes from freely available online datasets.
We then apply the trained SR model to the 3D representations of unseen scenes for which only low-resolution images are available. We validate that unlike super-resolving images post-rendering, the resulting super-resolved rendered images are indeed consistent across different views. The proposed method outperforms such conventional post-rendering SR methods both qualitatively and quantitatively, exceeding state-of-the-art image SR methods by over $1$db in PSNR when trained on the same dataset.

Specifically, we make the following contributions:
\begin{itemize}
    \item We propose a method for super-resolving volumetric neural scene representations, which allows generating geometrically consistent, high-resolution novel views corresponding to a scene captured in low resolution.
    \item We introduce a super-resolution approach that operates directly in feature space. To this end, we develop a novel feature-plane representation model that allows for rendering viewing-direction dependent views of general 3D scenes.
    \item We assess our method for super-resolving unseen scenes, both qualitatively and quantitatively and validate that it produces multi-view consistent super-resolved images.
\end{itemize}

\section{Super-Resolving Neural Volumes}
\input{figures/overview}
We introduce a method for super-resolving the representation of a 3D scene  $s$ given ${\cal I}_{\text{lr}}^s$, a set of low-resolution (LR) captures of the scene and their corresponding relative camera poses. To this end, we devise two principle modules, namely a scene representation model $G$ and a super-resolution model $F$.
We first use the images in ${\cal I}_{\text{lr}}^s$ to learn the parameters $\theta_G^s$ of representation model $G$,
which comprises a decoder model $D$ and learned feature planes ${\cal P}_s$
corresponding to each scene $s$. Once trained, one can use $G$ to render novel low-resolution views of the scene for new (unseen) camera poses:
\begin{equation}\label{eq:rendering_lr}
    {\hat i}_{\text{lr}}(r)=G(\theta_G^s,r).
\end{equation}
We use $r$ here to denote a pixel in the rendered image and its corresponding camera ray interchangeably, and denote by ${\hat i}_{\text{lr}}$ the rendered images, as they too lack the high-resolution fine details which are missing from the images in ${\cal I}_{\text{lr}}^s$.
To render high-resolution (HR) views containing fine details and textures, we propose to apply model $F$ on the \emph{parameters} of model $G$ which constitute the volumetric representation of $s$, that is
\begin{equation}\label{eq:rendering_sr}
    {\hat i}_{\text{sr}}(r)=G(F(\theta_F,\theta_G^s),r).
\end{equation}
Here ${\hat i}_{\text{sr}}$ is the rendered super-resolved image and $\theta_F$ are the parameters of model $F$, which are learned independently of any one specific scene.

An overview of our approach is illustrated in Fig.~\ref{fig:overview}. We next elaborate on each of its two modules; we first describe our novel plane-based neural 3D representation model $G$, and then explain how we use this representation to perform volumetric SR using $F$.

\input{figures/viewdir_effect}
\subsection{Plane-Based Radiance Fields for Generic Scenes}\label{sec:planar_representation}
To take advantage of advances in (2D) image SR, and to allow computationally efficient volumetric SR, we propose to represent a scene $s$ with a novel \emph{quadri-plane} model comprised of four multi-channel 2D feature planes of size $N\times N\times C$, coupled with a small decoder neural network.

Similar to existing neural rendering methods \cite{mildenhall2020nerf}, we view the volume as a pair of 3D fields representing scene density $\sigma$ and corresponding emitted radiance (in RGB).
We then diverge from the typical scheme and propose a feature plane-based representation. Chan \etal~\cite{chan2022eg3d} recently proposed a scene representation using feature planes for generating human faces. Specifically, they align explicit feature vectors along three orthogonal axis-aligned feature planes $\{P_{xy},P_{xz},P_{yz}\}$.
Then each 3D position $p\in\mathbb{R}^3$ is queried by projecting it onto each of the three feature planes and retrieving the per-plane feature vector corresponding to the point via bilinear interpolation, then aggregating them into a point feature representation vector $v_p$. This vector is fed into a light-weight \emph{decoder} MLP $D$ to yield the density and RGB values corresponding to $p$, which are then used to render an image of the scene using neural volume rendering~\cite{mildenhall2020nerf}.
In this work, we propose several modifications to this approach which are critical for the proposed method to function. We describe these modifications in the following.
\myparagraph{View-direction Dependent Radiance.}
We introduce a 4\textsuperscript{th} feature plane $P_\text{dir}$ to accommodate effects like non-uniform scene illumination, as we show in Fig.~\ref{fig:viewdir_effect}. The two axes of this plane reflect the 2D space of possible viewing directions for each point $p$.
We then infer a complementary feature vector $v_p^\text{dir}$ using bilinear interpolation over $P_\text{dir}$ at coordinates $(\theta_p,\phi_p)$, corresponding to the viewing azimuth and elevation of point $p$, respectively.
Features from this plane only affect the radiance and not the density output of $D$, as unlike radiance, volume density does not vary with viewing direction in the real world~\cite{mildenhall2020nerf}.

\myparagraph{Representing \emph{General} Scenes.} To allow representing general scenes with arbitrary geometric structures beyond the human faces demonstrated in~\cite{chan2022eg3d}, we follow the practice from~\cite{mildenhall2020nerf} and use a pair of coarse and fine decoders
$D_c$ and $D_f$,
respectively,  that are \emph{shared} across all scenes;  we first apply the coarse decoder to a set of stratified sample points $p$ along rays corresponding to each pixel. We then use the resulting values to bias a subsequent sampling of points along each ray towards more relevant parts of the volume, and process those using the fine decoder.

Given a set of training scenes ${\cal S}_t$ and corresponding sets of captured images, we learn a set of four planes per scene ${\cal P}_s=\{P_{xy},P_{xz},P_{yz},P_\text{dir}\},~s\in {\cal S}_t$ jointly with the decoder pair parameters $\theta_D$ (shared across all scenes), which together constitute the representation model parameters $\theta_G^s=\theta_D\cup{\cal P}_s$.
To this end, we minimize the following image rendering penalty~\cite{mildenhall2020nerf}:
\begin{equation}\label{eq:lr_loss}
    {\cal L_\text{LR}}=\sum_{r\in {\cal I}_{\text{lr}}^s}\|G_c(\theta_G^s,r)-I_\text{lr}(r)\|^2_2+\|G_f(\theta_G^s,r)-I_\text{lr}(r)\|^2_2.
\end{equation}
Here $I_\text{lr}(r)$ is the ground-truth RGB pixel value corresponding to ray $r$ sampled from the training image $I_{\text{lr}}\in {\cal I}_{\text{lr}}^s$,
and
$G_c(r)$ and $G_f(r)$
are the corresponding rendered values obtained through volume rendering~\cite{mildenhall2020nerf} over the outputs of the coarse and fine decoders $D_c$ and $D_f$, respectively.

We tailor the resolution $N$ of the learned planes to the resolution of the images in ${\cal I}_\text{lr}^s$:
As we show in Fig.~\ref{fig:plane_res}, using planes that are too small prohibits the representation of high visual frequencies, similar to using NeRF~\cite{mildenhall2020nerf} without positional encoding. In contrast, using too large planes results in some plane regions having too little relevant information in the captured scene images ${\cal I}_{\text{lr}}^s$, which is manifested as visual artifacts when rendering novel scene views. We will revisit this trade-off in plane resolution later in the context of super-resolving the scene.

\input{figures/planes_res}
Note also that we choose a smaller resolution value $N_\text{dir}<N$ for the view-direction feature plane $P_\text{dir}$,
since the image set contains less information relevant to this plane:
Each image pixel carries information relevant to all points along its corresponding ray $r$ which, in turn, makes it relevant to many different locations on the positional feature planes $\{P_{xy},P_{xz},P_{yz}\}$. In contrast, all points along the ray correspond to the same viewing direction, meaning that the information in each pixel is relevant to a single point on $P_\text{dir}$.

\subsection{Super-resolving Implicit Representations}\label{sec:volumetric_sr}
To produce HR views of a scene given only a set of its LR captures, one could take a classic super-resolution approach and apply any one of many existing image SR methods on the rendered LR scene views. However, since such methods have no notion of the underlying three-dimensional scene, novel views contain inconsistent super-resolved details in the synthesized views (see Fig.~\ref{fig:teaser}).
Our approach addresses this issue and allows producing super-resolved, \emph{geometrically consistent} scene views.

As we discuss in Sec.~\ref{sec:planar_representation}, employing larger plane resolution $N$ allows us to represent and render higher visual spatial frequencies, corresponding to small details and fine textures. Therefore, substituting the set of planes ${\cal P}_s$ learned from the LR scene captures ${\cal I}_{\text{lr}}^s$ with a set of higher-resolution planes corresponding to the same scene $s$ can allow for rendering the desired HR views.

We propose to approximate sets of HR planes by learning plane super-resolution model $F$, which we apply on the learned LR planes (Eq.~\ref{eq:rendering_sr}) to obtain $\alpha$ times super-resolved versions of them, with $\alpha$ being the desired SR factor.
We use a single model to super-resolve all positional\footnote{We use the viewing direction plane as is without super-resolving it.
}
feature planes $P\in{\cal P}_s\setminus\{P_\text{dir}\}$, while taking advantage of our 2D planes-based representation model by employing an existing image SR network architecture for $F$. Specifically, we borrow the residual network architecture used for the successful 2D super-resolution method EDSR~\cite{lim2017edsr}. To learn the mapping from sets of LR planes to their HR counterparts, we train $F$ on a training set of scenes for which ground truth (GT) HR captures are available, by substituting $G_f(r)$ in Eq.~\ref{eq:lr_loss} with the output from Eq.~\ref{eq:rendering_sr} and using HR scene captures $I_\text{hr}\in {\cal I}_{\text{hr}}^s$, resulting in a second loss term:
\begin{equation}\label{eq:hr_loss}
    {\cal L}_\text{HR}=\sum_{r\in {\cal I}_{\text{hr}}^s}\|G_f(F(\theta_F,\theta_G^s),r)-I_\text{hr}(r)\|^2_2.
\end{equation}

We train our framework \emph{end-to-end} and learn the parameters of $G$ (including both decoder parameters $\theta_D$ and LR scene planes ${\cal P}_s$ corresponding to each training scene $s$) together with training our SR model $F$. To this end, we alternate between using the sets of GT HR captures ${\cal I}_{\text{hr}}^s$ corresponding to our training scenes for learning the parameters of $F$, and their synthetically downsampled versions ${\cal I}_{\text{lr}}^s$ for learning the LR plane sets ${\cal P}_s$. Both sets are exploited for learning the decoder parameters.

To promote the ability of $F$ to generalize to unseen scene planes, we add another penalty that encourages the per-feature-channel plane mean $\overline{P_c}$ to be close to zero: \begin{equation}\label{eq:zero_mean_loss}
    {\cal L}_P=\frac{1}{3C}\sum_{c=1}^C\sum_{P\in {\cal P}_s\setminus\{P_\text{dir}\}}\Bigl|\overline{P_c}\Bigr|
\end{equation}
This has a similar effect to pre-normalizing the input to the model, which is common practice for CNNs. Our training scheme therefore boils down to minimizing the following objective, with $\lambda_P$ being a weight hyper-parameter:
\begin{equation}\label{eq:total_loss}
    {\cal L}_\text{total}={\cal L_\text{LR}}+{\cal L_\text{HR}}+\lambda_P {\cal L}_P
\end{equation}

\subsection{Implementation Details}\label{sec:implementation}
Our decoder model $D$ (both coarse and fine models) computes the density value for each point $p$ by processing the concatenated feature vectors $v_p$ from the $3$ positional planes using $8$ consecutive fully-connected layers with $128$ channels and a skip connection every $3$ layers. We use the same architecture for computing the corresponding radiance value, with the exception that we feed it with the concatenation of all 4 feature vectors queried from all 4 planes $P\in{\cal P}_s$.
We employ feature planes with $C=48$ channels, and a resolution of $N=200$ or $N=800$ for positional planes representing scenes captured at $100$ or $400$ pixels image resolution, respectively. The resolution of our view-direction plane $P_\text{dir}$ is set to $N_\text{dir}=32$ in all cases. For our SR model $F$ we use the architecture from EDSR~\cite{lim2017edsr} with $32$ residual blocks and $256$ channels.

We train our entire model end-to-end using Adam optimizers with learning rates of $0.0005$ for the decoder and planes and $0.00005$ for $F$. At each step, we sample 4096 rays $r$ from an image $I$ corresponding to a scene $s\in{\cal S}_t$, all chosen randomly. We set $\lambda_P=1$ and train our framework to convergence using an NVIDIA A100 GPU.
\section{Related Work}\label{sec:related_work}
This work introduces a method for reconstructing images of a 3D scene by operating directly on its volumetric neural representation. Next, we provide background on the two most related fields, image super-resolution and neural 3D scene representation.

\myparagraph{Image Super Resolution}
Image super resolution (SR) belongs to a wide category of ill-posed image reconstruction tasks such as image denoising~\cite{buades2005non_local_denoising,kawar2021stochastic_denoising}, dehazing~\cite{li2018benchmarking_dehazing,bahat2016recurrence_dehazing}, deblurring~\cite{tao2018scale_recurrent_deblur,bahat2017deblurring_reblurring}, inpainting~\cite{bertalmio2000image_inpainting,jam2021inpainting_review} and more. Image SR aims to compensate for information lost during capturing as a result of the lossy image acquisition process, due to a resolution loss in the optical system and finite sampling by the sensor. Existing image SR methods produce a reconstructed image of the captured scene as if captured in ideal settings.
To recover the fine details lost due to the finite capturing resolution, some SR works propose to collect additional information from multiple captures of the same scene~\cite{farsiu2004fast_multiframe,wronski2019handheld_burst}, which typically allows increasing the resolution by only up to a factor of two. To allow higher SR factors, classical methods have proposed to employ various image priors that exploit the unique characteristics of natural images, \eg
their internal self-similarity~\cite{glasner2009recurrence,michaeli2013blind_recurrence}, heavy-tailed gradient magnitude distribution~\cite{sun2008gradient_prior} and distinctive image edge statistics~\cite{tai2010edge_prior}.

\myparagraph{Learning Image Super Resolution}
With the arrival of deep learning, researchers proposed to learn these image characteristics and train Convolutional Neural Networks (CNNs) on large image datasets~\cite{dong2014first_cnn_sr,lim2017edsr,liang2021swinir}, leading to unprecedented performance in terms of minimizing reconstruction error (\ie increasing PSNR). Following the introduction of Generative Adversarial Networks (GANs)~\cite{goodfellow2020gan}, some methods also began targeting alternative objectives, such as perceptual reconstruction quality~\cite{ledig2017srgan,wang2018esrgan} or exploring the diverse space of SR solutions~\cite{bahat2020explorable,lugmayr2020srflow}.
A further line of work on image super resolution takes relevant reference images as additional user input~\cite{zheng2018reference,dong2021rrsgan_reference} to better adapt to each specific scene. However, common to all of these methods is that they aim to super-resolve a \emph{single} image at a time. To the best of our knowledge, the proposed is the first work aiming to \emph{consistently} super-resolve images corresponding to the \emph{infinite} possible views of a given 3D scene, by operating directly on its neural representation. While methods for spatial Video SR~\cite{kappeler2016video_sr,liu2017robust_video_sr,geng2022video_sr} also attempt to
simultaneously handle sequences of inter-related frames, these methods are limited to positions along the original camera trajectory  and cannot render novel views. Moreover, they cannot inherently guarantee consistent reconstruction across frames, unlike our approach.

\myparagraph{Neural Scene Representations.}
An increasingly large body of work learns representations of 3D scenes, aiming to allow synthesizing novel scene views from arbitrary directions, as well as editing the scene \eg to allow modifying the perceived illumination settings or object properties.
Early works~\cite{agarwal2011building_rome,schonberger2016pixelwise} tried to learn an explicit 3D scene model by fitting it to a given set of images capturing the scene from different view directions.
The recent introduction of Neural Radiance Fields~\cite{mildenhall2020nerf} (NeRF) then brought upon a significant leap in reconstructed image quality, rendering photo-realistic views which include fine, high-frequency details. They rely on a lean coordinate-based Multilayer Perceptron (MLP) to implicitly represent the 5D radiance field, made possible mainly thanks to artificially introducing high frequencies into the model's input, in the form of sinusoidal positional encoding~\cite{tancik2020pos_encoding}.

Other methods proposed to use alternative implicit representation models based on, \eg, Voxels~\cite{liu2020sparse_voxels} or spherical harmonics~\cite{Fridovich-Keil_2022plenoxels}. Chan \etal \cite{chan2022eg3d} proposed to generate geometrically consistent human faces by representing the volume using three 2D feature planes coupled with a simple MLP decoder. We also adopt 2D feature planes to support general 3D scenes and view-direction dependency, and take advantage of the planar representation to perform our volumetric SR in 2D, while leveraging the existing rich knowledge base which already exists for 2D SR (see Sec.~\ref{sec:planar_representation}).

Many recent methods focused on enabling NeRF to learn from imperfect image sets, \eg using partly occluded~\cite{martin2021nerf_in_the_wild} or very few scene captures~\cite{jain2021few_shot_nerf}. In contrast, our work aims to endow the learned representation model with the ability to reconstruct finer details than the ones captured in the given image sets, by learning volumetric scene priors, \ie, a volumetric extension of (natural) image priors.

\myparagraph{Neural Radiance Fields for Image Reconstruction.}
In a separate line of work, some methods harness the implicit 3D representation of NeRF as a tool for image denoising~\cite{pearl2022nerf_denoising} or high dynamic range (HDR) reconstruction~\cite{mildenhall2022nerf_in_the_dark,huang2022hdr_nerf}. These methods take as input a burst of noisy or low dynamic range scene captures, respectively, and distill the multi-frame signal while relying on the geometrical consistency inherent to the 3D representation.
In contrast, our approach allows using a set of degraded (low-resolution) scene captures for synthesizing consistent novel scene views, containing fine details that are \emph{unavailable} in the input images, by relying on \emph{externally learned} volumetric priors
(see Sec.~\ref{sec:volumetric_sr}).