1. Introduction
Built heritage geometric modeling has been conducted using reality-based data [
1,
2,
3,
4]. The geometric model preserves the current shape of the heritage object, monitors eventual changes, and reconstructs them even if the historic buildings should have some critical evolutions or damages. It is no longer a problem to obtain an accurate 3D geometric model, yet it is still highly anticipated to perform semantic segmentation and labeling of elements [
5,
6,
7], and to improve the representation of the relationships between tangible and intangible heritage information [
8].
With the trend of information technology, building information modeling (BIM) has been widely developed to manage the geometry, semantic, attribute, energy, and relationship information. BIM is initially created to design and manage the life-cycle construction of new buildings. Further, BIM platforms are increasingly supportive of 3D point clouds representing the entity surface. Subsequently, a new concept of as-built BIM begins to recreate and manage existing buildings [
9], and the modeling from point clouds in the BIM environment has produced advancements in a scan-to-BIM process [
10]. By importing the point clouds into the BIM software, the building components can be created using the existing BIM Industry Foundation Classes (IFC) classes and self-defined structures. In this kind of information models, the representation of the building is not only a coherent geometrical modeling of the reality but also contains additional details typical of semantic, parametric, and relationship descriptions of its elements.
BIM draws attention from the field of heritage documentation and conservation [
11,
12,
13] and has generated a new issue of heritage/historic building information modeling (HBIM) [
14]. HBIM can either utilize the BIM concept to re-create “new” heritage according to the historical documentation or use the “as-built” BIM concept to map HBIM volume objects to the reality-based surface model [
15]. The HBIM model is no longer just a virtual representation and geometric reconstruction of heritage, but the sub-elements have become advanced objects with rich information, including the quantitative and qualitative description and strict relationship information.
Built heritage is characterized by heterogeneity in both aspects of geometry and semantics. Such heterogeneity of HBIM brings new challenges to the usual sense of BIM, considering the complex geometric structures and knowledge composing heritage. On the one hand, the irregular and unique structures forming the historic buildings make HBIM extraordinary and complicated compared to classical BIM modeling and models [
16]. Currently, most research groups aim to address these irregular shapes in the HBIM modeling process. Virtual Historic Dublin led by Murphy [
12,
14,
17] constructed the irregular structures of HBIM through geometric description language (GDL). GDL is used to build parametric elements regarding the historic building based on documentation. These parametric elements are then mapped to the reality-based data, and their parameters are optimized to obtain a current-state diagnosis. López et al. [
18] drew the reference lines and rules by analyzing bibliographic data to help the manual design of the HBIM model. Barazzetti et al. [
19,
20,
21] relied on non-uniform rational basis splines (NURBS) curves to preserves the details of complex structures in the BIM-based reconstruction from a point cloud. Garagnani [
22,
23] developed a plug-in named GreenSpider to produce the parametric HBIM elements, which imports vertexes of interest of the point cloud and then creates accurate components based on these discrete, selectable snap points.
On the other hand, some consider the difficulties of HBIM in representing and managing the vast and complicated knowledge related to non-geometrical aspects of the heritage. The integration of HBIM with ontologies becomes popular to extend the capabilities of HBIM in knowledge management and conservation. Pauwels et al. [
24,
25] developed an automatic EXPRESS-to-OWL tool that converts the HBIM IFC model to ifcOWL ontology. Simeone, Cursi et al. [
26,
27,
28] focused on the development of a semantic-enriched HBIM that integrates HBIM and a knowledge base through ontologies as a way to enhance knowledge representation and management.
The main contributions of this paper involve two aspects to deal with these two kinds of challenges as well. A new mesh-to-HBIM workflow is proposed in the modeling process, which transfers the holistic surface mesh to semantic HBIM model with reduced human involvement. Also, integration of obtained HBIM and ontology is explored with the capability of representation of a large amount of semantics, which combines the HBIM elements and historical knowledge in a unified BIM environment.
2. State of the Art
2.1. Scan-to-HBIM and Mesh-to-HBIM
Currently, HBIM modeling is generally as-built HBIM models for existing built heritage generated by the scan-to-HBIM workflows. The scan-to-HBIM mainly consists of two kinds of approaches, forward modeling and reverse modeling according to whether simulated primitives are predicted on the reference of historical documentation or not.
On the one hand, the forward design combines both the original HBIM concept from historical documentation and as-built HBIM concept from reality-based data [
12,
19,
29,
30,
31]. The parametric components composing heritage are first simulated based on the prior knowledge about the parameters and rules and try to predict what shape the building might be. Then the simulated primitives are mapped to the 3D point cloud space where to refine the parameters and obtain a current-state diagnosis. So the process depends on how to fit the standard shapes of HBIM components into the reality-based point clouds, which can be addressed via manual interpretation and possible shape-based recognition [
32].
On the other hand, reverse design directly creates the parametric primitives in the BIM platform according to the point clouds and other formats (such as surface mesh and solid) derived from the point clouds. This kind of approach is active and attractive in the process of scan-to-BIM, such as the automated/semi-automated algorithms proposed in [
7,
33,
34]. Those studies dealt well with planar walls and floors and some other regular structures (i.e., rectangular openings and cylinder columns). Current scan-to-HBIM remains mostly a manual process even though there are some commercial tools to help the scan-to-(H)BIM process, such as
ClearEdge3D Edgewise,
IMAGINit Scan to BIM,
Pointsense, and
Leica CloudWorx [
18,
35,
36,
37,
38]. Some also have tried to develop the plugins for the specific heritage structures; for example, some
Autodesk Revit API plugins were designed to create HBIM elements according to the sparse points which generally consider simple structures such as rectangular beam frame [
39] and columns [
22].
Moreover, most BIM software is not 3D-centric and does not exhibit the functions of freeform geometry modeler. They create the 3D model based on the rotation and extrusion of the predefined position in the 2D plane. The freeform design of users is limited when creating complex geometry in a 3D point cloud space directly. Thus, a key challenge today is to reduce the human involvement leading to as-built HBIM from point clouds [
33] and reserve the geometric accuracy for irregular details during its translation into BIM objects [
21,
37].
A surface mesh can accurately represent the irregular shapes of architectural heritage and their as-built conditions, but it is rarely managed in a BIM environment [
40]. Recently, HBIM modeling from mesh draws attention, and several works have mentioned the possibility to create HBIM models from surface mesh and solid geometry instead of point clouds. Some have converted the solid mesh into 3D solid primitives and then assembled these components to the HBIM model directly in the BIM software [
35,
41,
42]. In [
43], HBIM is constructed with three types of primitives and data sources, including point clouds, closed mesh, and CAD/.3dm components. Reference [
44] explored the potential to import surface mesh representing complex objects into BIM software to serve as the reference for creating the parametric elements.
In this paper, a mesh-to-HBIM workflow is developed, consisting of two steps: (i) Segmentation of surface mesh and generation of 3D volume primitives, and (ii) semantic HBIM modeling according to the solid primitives. The proposed approach is conducted in a semi-automated way without human drawing and deals with irregular shapes by NURBS. The obtained HBIM model is composed of BIM semantic elements, which are not defined with specific geometric parameters yet can be attached with the additional semantic, attribute, and relationship information.
2.2. HBIM and Ontology
Compared to the conventional geometric models, the HBIM model provides a unified environment for semantic/parametric elements, 2D/3D visualization and spatial/attribute database. But HBIM is still challenging in managing a large volume of semantics; for example, historical context, social information, environmental resources [
8]. Despite this, extremely heterogeneous knowledge is an essential asset for built heritage.
An ontology can serve as a data collector of all the semantics of entity to build a knowledge model, which is generally called a domain representing its primary entities (domain objects), the relations between them, the attributes (called properties) of these entities, and their values [
27,
45,
46,
47]. Previous studies show that ontology can overcome some limitations of the current BIM IFC data model. For example, Jung and Joo [
48] indicated using ontology in BIM framework to automate spatial and temporal interrelationship. Abanda et al. [
49] combined ontologies with a BIM model to facilitate information extraction for cost estimation application, in which ontology is employed to check the semantics and reason the descriptive logic. Ali and Mohamed [
50] encoded a BIM model using resource description framework (RDF) and then grouped the objects into clusters representing different trades. Zhang and Issa [
51] integrated BIM with ontology to generate partial BIM models based on queries.
Ontologies are increasingly used to overcome the barriers to heterogeneous semantic data sharing and integration about cultural heritage. Several advantages of ontology semantics in built heritage include: (i) Homogenous representation and management of all the knowledge related to heritage, (ii) sharing in a scientific community, and (iii) knowledge reasoning and analyzing. Ontologies play significant roles in connection with HBIM to extend the capability of HBIM in the aspect of knowledge representation and management [
8,
52,
53]. Currently, the combination of ontology semantics and HBIM geometric elements can be conducted in different environments: (i) BIM platform (typically as
Autodesk Revit) by direct BIM ad-hoc development and
DB Link to connect to external database [
45], (ii) ontology platform by migrating HBIM IFC files to ontology format [
25], and (iii) a newly developed unified platform [
27].
In this paper, we connect the HBIM geometric elements and ontology semantics in a unified BIM environment. The proposed approach consists of two steps: (i) Migrating HBIM IFC files to ontology platform to enrich the semantic information, and (ii) connecting the HBIM element with semantics in the BIM environment.
3. Methodology
The methodology consists of HBIM modeling and knowledge enrichment. A parametric HBIM model is obtained from the point clouds and conventional scan-to-HBIM process (the red part in
Figure 1), while a semantic HBIM model is created from the surface mesh and proposed mesh-to-HBIM workflow (the blue section in
Figure 1). For the scan-to-HBIM process, we manually draw the parametric HBIM elements (i.e., columns, roof, slabs, and walls) in the BIM platform (
Autodesk Revit in the study) on the reference of point clouds. For the mesh-to-HBIM process, we first extract the basic primitives from surface mesh (with the help of
Rhino 3D in the study) and then translate the solid geometry into semantic HBIM elements (
Autodesk Revit and
Dynamo in the research). Furthermore, the obtained HBIM models are extended with the capability of representation of heterogeneous knowledge by ontology semantic richness. The HBIM IFC model can be converted to ontology RDF model with consistent entities, and then semantics describing heritage can be enriched under ontology rules (the black part in
Figure 1). Finally, the HBIM geometric element and ontology knowledge is connected in a unified BIM environment via
Revit Dynamo tools (the green section in
Figure 1).
Autodesk Revit, a common BIM platform, provides friendly support of point clouds and becomes a hot as-built HBIM modeling environment [
18,
19,
22,
37,
54].
Autodesk Revit also includes a built-in
Dynamo visual programming tool to extend the BIM capabilities [
42,
55,
56,
57,
58].
Autodesk Dynamo is an open-source visual programming environment, which offers designers the ability to design and manipulate BIM elements by programming interactively. This project utilized
Dynamo to extend the capability of
Revit BIM environment by integration with conventional geometric models (such as from
Rhino 3D and
SketchUp) and ontology/database knowledge description.
3.1. Study Area
The St-Pierre-le-Jeune Church, Strasbourg, France, was built between 1889 and 1893 and designed by German architects Skjold Neckelmann and August Hartel. The church is in the neo-Romanesque style, with its imposing 50 m dome inside and a diameter close to 20 m at the base. Our previous project [
59,
60] took in total 2755 UAV images and around 200 terrestrial images for the exterior and generated the point clouds based on dense matching. The meshing was then performed on the merged point cloud using the Poisson method (level of Octree 13). Finally, the textures from all images were projected on the unified 3D mesh. The final 3D model is a holistic geometric model in mesh surface format, which can be viewed through the following link:
https://skfb.ly/PtQF [
60]. The study area in this paper is the front part of the church, including the main façade and two square belfries (
Figure 2).
3.2. HBIM Environment for Semantic Modeling and Knowledge Management
Both HBIM and ontology utilize the object-oriented approach consisting of a primitive system to describe the characteristics of built heritage elements as well as their relationships. They represent heritage through a synthesis of three main concepts: “Classes”, “properties”, and “rules/relationships”. A “class” is a group of elements with a standard set of parameters and similar representation, which is the base for the HBIM modeling, ontology semantics richness, and information fusion. That is, HBIM and ontology rely on corresponding “classes” and entities with different types of properties and relationships.
In the study, the “classes” and entities are first created as geometric HBIM elements, then enriched with semantic knowledge in ontology platform, and merged in a unified BIM environment. Thus, the “classes” and entities are described by two different kinds of details:
(i) HBIM elements: The library of parametric “classes” needs to be built to define the structural components of the historic building. Then the instances can be created by tuning parameters (such as shape, size, and other properties) of the general “class”, and they can be linked together with strict spatial relationships. The spatial relationships are fixed, even if the sizes or shapes of the elements change.
(ii) Ontological knowledge: The “classes” and entities are converted from the HBIM elements with unique identifiers. The knowledge related to the historic building and its sub-elements can be enriched by ontology properties and relationship. The ontological knowledge is finally connected with HBIM elements in the BIM environment.
3.3. Conventional Scan-to-HBIM
Given that many of default BIM primitives regard modern and contemporary buildings, the structures composing the heritage need to be self-defined. In the study, reverse modelling is used to create the primitives constituting the St-Pierre-le-Jeune Church directly. The clean point cloud (
Figure 3a) is loaded into the BIM platform to provide the reference of geometric shapes. Instead of point cloud segmentation, manual design of the basic shapes composing heritage is conducted by visual interpretation.
Figure 3c shows some typical BIM primitives whose parameters and shapes are determined by the point cloud. The final stage is the map** of BIM objects onto the 3D surface model. The semantic HBIM model (
Figure 3b) is created by assembling these primitives whose position is defined by point cloud as well.
However, BIM platform is initially developed for new buildings, which builds the 3D models according to rules and parameters. Also, BIM software (including Autodesk Revit adopted in the study) is not a 3D-centric and a freeform geometry modeler. They mostly create the 3D model based on the rotation and extrusion of the predefined 2D shape in the plane. Thus, the manual drawing of scan-to-HBIM is time-consuming because the direct modeling in 3D point cloud space is not accordant with the design concept of BIM.
3.4. Mesh-to-HBIM
The above self-defined primitives from point clouds are parametric “classes” composing a historic building, whose shape can be altered and refined later. BIM platform also supports another kind of non-changeable primitives, which can also attach properties; except their geometric shape cannot be further modified. These primitives are generally derived from the existing solid geometry. The point clouds representing the regular shape can be translated into simple solid meshes through low-poly meshes [
9,
19]. But it is worth to transfer the point clouds into triangulated mesh before converting into a solid element when dealing with surfaces that are irregular and strongly deformed [
41].
The proposed mesh-to-HBIM process aims to transfer the holistic surface mesh to volume semantic HBIM. The original triangulated mesh is the accurate fitting of the point clouds and reconstructs heritage using 3D surface, whereas HBIM is dedicated to a 3D solid element and cannot support the unclosed surface mesh. The process, therefore, mainly consists of three steps: Primitive extraction based on multi-layered segmentation in Rhino 3D, a transformation from surface element to a volume component using extrusion and NURBS functions in Rhino 3D, and generation of BIM component using Revit Dynamo packages.
(1) Segmentation of Holistic Mesh
The holistic mesh is fundamentally segmented into parts representing the church (i.e., walls, roofs, and windows). There have been lots of platforms to segment and edit 3D triangulated meshes, and
Rhino 3D, a solid modeling software with powerful mesh processing functions, is adopted in the project. The “Explode” command in
Rhino 3D is used to break the holistic mesh down into small patches. The “Explode” command can be conducted iteratively until the scale of the patches is small enough. Then, the users can call the “Join” command to connect patches to form a single object. Taking the roof structure as an example (
Figure 4), the roof consists of several patches (yellow parts), and the patch needs further explosion if it involves non-roof structures.
(2) Faces to Solid
These 3D elements are still in 3D surface mesh format (
Figure 4e) and need to be transferred to solid primitives considering that HBIM is a volume model. Meshes represent 3D surfaces as a series of discreet facets and yield a large file size with small enough facets to smoothly describe the irregular shapes. The “MeshToNurb” command in
Rhino 3D is used to refine these triangulated meshes to NURBS surfaces, which are the mathematical representations of the surfaces with reduced file size. Then, the surfaces are extruded to solid primitives (
Figure 4f) using “offset surface” command with a thickness parameter.
(3) HBIM Components
The solid elements (
Figure 5) are finally imported into the BIM environment where additional characteristics are attached to each element.
Spring Node, an open source
Dynamo package to interact between a solid element and
Revit BIM “class” and entity, is used to translate the volume element to the BIM environment (
Figure 4g). This kind of HBIM components can attach attribute and material information, but their geometric shape cannot be modified in the BIM environment.
3.5. HBIM with an Extended Capability of Knowledge Representation
The similar object-oriented fashion to the modeling of the architectural object makes it possible to connect the databases between an ontology-based system and HBIM. Consequently, this connection was established by assigning the same identifier to both the representations of the corresponding entity in the two modeling environments (
Figure 6). The research aims to extend the HBIM capability of attaching heterogeneous knowledge. Instead of the separate creation of HBIM and ontology model, an interactive way consisting of the transformation, richness, and combination is introduced based on the unique IDs of elements.
(1) Transformation
Currently, several BIM IFC to ontology RDF/OWL conversion procedures has been proposed. Among them, an IFC-to-RDF conversion tool developed by Pauwels and Terkaj [
61] can automatically map IFC extensible properties to RDF data according to the IfcOWL ontology. Once the HBIM model has been built in a BIM environment, the geometric element based model is straightforwardly exported to IFC format and transferred to the ontology RDF format by using IFC-to-RDF conversion tool. After the transformation, the general “classes” and individual entities are reserved with the unique identifier (IDs).
(2) Knowledge Richness
Protégé, an open resource platform to construct domain models and knowledge-based applications with ontologies [
62], is adopted to enrich the properties and relationships for the HBIM entities. In the cultural heritage context, the International Committee for Documentation Conceptual Reference Model (CIDOC CRM) provides ISO standardized definitions as well as a leading ontological reference for describing the implicit and explicit concepts and relationships about historic buildings [
46]. To align our work with a bounder research community effort, we build the ontology model in the context of CIDOC CRM (
Figure 7). Data properties in ontology represent attributes and object properties represent the relationship between different classes (
Figure 8).
(3) HBIM Expansion
The drawbacks of the ontology analysis come to the requiring of long SPARQL queries to extract relevant information [
63] and the accessing of the HBIM element. Thus, a correspondence between the HBIM element and ontology semantics is built in the BIM environment.
Revit Dynamo provides the interactive interface in the study, where users can browse the semantic information reserved in the ontology database and the 3D model in BIM platform at the same time (
Figure 9).
4. Results and Discussion
4.1. Parametric/Semantic HBIM Models
HBIM is a detailed semantic model composed of the defined “classes” and entities. In this project, three geometric models are obtained, including the holistic surface mesh model, parametric HBIM model from point clouds, and semantic HBIM model from the mesh. Compared to the holistic surface mesh, the HBIM are semantic volume models with attached properties (
Table 1). Two types of BIM “classes” and HBIM models are obtained: (i) The parametric “classes” and entities consisting of a set of geometric rules and properties generate the parametric HBIM from the point clouds, while (ii) the non-changeable “classes” and entities with fixed shape generate the semantic HBIM from the surface mesh.
The obtained HBIM model (
Figure 10 left) using point clouds is a geometric model with parameters and semantics. Users can alter the settings in the
Revit interface or
Dynamo nodes. That is, we can change the shape of the HBIM elements, and the geometric accuracy of the model extremely depends on the manual drawing works. The level of detail (LoD) of the HBIM model relies on the design of “classes” and entities, in which the more parameters and higher accurate geometry are manually created, the higher LoD is obtained. The parametric HBIM model corresponds to the most-used LoD3 in CityGML or LoD300 in IFC, which contains the full exterior of an architectural model with complete wall and roof structures, doors and windows [
64].
The obtained HBIM model (
Figure 10 right) using surface mesh is a geometric model with semantics and non-editable geometric instances. The model has potential to attach attribute, material, and temporal information in the BIM platform, yet its geometric aspect information is limited to non-editable and no changeable parameters available. The geometric information of the model derives from the surface mesh without human modification. The accuracy is affected by computer performance, considering that the huge size of meshes and large scale of the building may go beyond the capabilities of the software. The detail of the model depends on the surface mesh and the transformation from the surface to solid. Here, we can notice that the central part of the façade seems to become rough. One reason comes from our reduction of the resolution. The current model is about 100 MB, which is relatively huge for
Dynamo and
Revit platforms.
4.2. HBIM and Ontology: Geometry and Semantics
The IFC data model of parametric HBIM (via scan-to-HBIM workflow) is transferred to ontology RDF format, in which the classes and entities are reserved. An ontology model of the church is built with its historical knowledge, such as building style, architect, maintenance date. The combined entity thus includes both object-oriented information, geometric information, typically created in BIM platform, and knowledge information, usually enriched in ontology modeling systems. They are connected via the same labeling of the entity in the two modeling environments.
Revit Dynamo provides the integration platform (shown in
Figure 11), where users can browse the semantic information reserved in the ontology database and the 3D model in BIM at the same time. When choosing the geometric element in
Revit BIM platform, the corresponding knowledge can be displayed in the
Dynamo “watch” window. The browsing result can be exported as a text file for further application.
The connection of the knowledge base and HBIM is based on the unique IDs. That is, two separate ontologies are needed to connect with scan-to-HBIM and mesh-to-HBIM, respectively. In the paper, we created a knowledge model based on scan-to-HBIM to explore the feasibility of the proposed approach. The current ontology model consists of limited historical context (such as architect, renovation time, and construction style), but the detail of the knowledge base can be further enriched in ontology environment according to the user’s needs. The drawback of the Dynamo environment lies in the lack of property names, which are replaced with numbering list in the “watch” window of the knowledge.
5. Conclusions
In summary, with the support of point clouds in the BIM software, the reality-based modeling of built heritage can be directly conducted in a BIM environment. The obtained object-oriented HBIM model not only achieves the parametric and semantic modeling of the geometry but also manages the attribute, material, and relationship of the elements. But the scan-to-HBIM still faces some challenges, typically including the time-consuming manual drawing of the complex shape composing heritage and the support of a large amount of knowledge related to heritage.
To deal with the heterogeneity of geometry and knowledge composing heritage, we proposed a mesh-to-HBIM and an HBIM-ontology-integration by using Revit Dynamo visual programming tools to extend BIM off-the-shelf capabilities (Autodesk Revit).
On the one hand, the mesh-to-HBIM modeling generated from holistic surface mesh reduced lots of individual efforts compared to scan-to-HBIM by time-consuming manual drawing. The obtained HBIM, however, possesses no specific geometric parameters because the automatic BIM primitive generation from surface mesh returns non-changeable solid elements. So, it depends on the users’ need. If the user prefers parametric geometry, a complete and personified “class” and entity has to be drawn manually; but if the user prefers the original mesh geometry and semantics, the mesh-to-HBIM provides a work-saving way.
On the other hand, HBIM is extended with the integration of ontology via Revit Dynamo, where users can browse the semantic information reserved in an ontology database and the 3D model in HBIM at the same time. The extended HBIM model fully represents and comprehends a historical building regarding its materials, construction components, and old memories in addition to the geometric shape.
The future work will go deeper into the mesh segmentation. The current mesh-to-HBIM process relies on the surface mesh segmentation in the Rhino 3D platform. Our ongoing work focuses on how to detect the semantic elements based on the point cloud segmentation algorithms.