Gutenbrain: An Architecture for Equipment Technical Attributes

Extraction from Piping & Instrumentation Diagrams

Marco Vicente

1 a

, Jo

˜

ao Guarda

2 b

and Fernando Batista

3 c

piping, instrumentation and other related equipment and their physical process flow. They are critical in en-

gineering projects to convey the physical sequence of systems, allowing engineers to understand the process

flow, safety and regulatory requirements, and operational details. P&IDs may be provided in several formats,

including scanned paper, CAD files, PDF, images, but these documents are frequently searched manually to

identify all the equipment and their inter-connectivity. Furthermore, engineers must search the related tech-

nical specifications in separate technical documents, as P&ID usually don’t include technical specifications.

This paper presents Gutenbrain, an architecture to extract equipment technical attributes from piping & in-

strumentation diagrams and technical documentation, which relies in textual information only. It first extracts

equipment from P&IDs, using meta-data to understand the equipment type, and text coordinates to detect the

equipment even when it is represented in multiple lines of text. After detecting the equipment and storing it in

allowing engineers to save thousands of hours in maintenance engineering tasks.

1 INTRODUCTION

In the Oil & Gas Upstream industry, the life cycle of

any new asset (offshore or onshore) starts with the En-

gineering, Procurement & Construction phase (EPC).

The first activity of a Maintenance & Inspection En-

gineering Contract (also called a MIEC) is to create

the asset register, which is the hierarchy of all the

equipment. This is the very first mandatory mile-

stone of a MIEC to be able to move on to other activ-

b

https://orcid.org/0000-0002-5733-9553

c

https://orcid.org/0000-0002-1075-0177

dures, and manuals, and link them to the asset regis-

ter. The asset register is created based in the infor-

mation provided in Piping and Instrumentation Dia-

grams (P&ID). P&IDs are detailed representations of

engineering schematics with piping, instrumentation

and other related equipment and their physical pro-

cess flow. P&IDs are critical in engineering projects

ment and their inter connectivity. Furthermore, en-

gineers must search technical specifications in sepa-

rate technical documents, as P&ID usually don’t in-

clude technical specifications. The process consists in

gathering all documentation coming from EPC con-

to identify all the tags of the equipment, group them

by system and sort them as per their parent/child re-

lationship. When a new revision of such a document

appears, it must be processed again to make sure there

is no impact on the asset register. So, this documenta-

tion searching activity is heavy, time consuming and

with low value for the project. The information is hard

to find, and the manual processing and the redundancy

of the activity sometimes leads to human error.

This paper presents Gutenbrain, an architecture to

in multiple lines of text. After detecting the equip-

ment and storing it in a database, it allows retriev-

ing and inferring technical attributes from the related

technical documentation using two question answer-

ing models based on BERT-like contextual embed-

dings, depending on the equipment type meta-data.

One question answering model works with free ques-

tions of continuous text, while the other uses tabu-

lar data. This ensemble approach allows us to extract

technical attributes from documents where informa-

3 describes the proposed architecture. Section 4 re-

ports the achieved evaluation results. Finally, Section

5 presents the main conclusions, and pinpoints future

working directions.

2 RELATED WORK

This section presents an overview of the existing liter-

ature. It starts by focusing on the literature concern-

ing information extraction from diagrams, and then

focuses on the retrieval and inference of technical at-

tributes from additional data.

(Moreno-Garc

ıa et al., 2019) presents a comprehen-

sive study on the techniques used in the piping & in-

strumentation diagrams domain, referring that most

work that has been done in this field focus on using

computer vision models and algorithms to extract at-

tributes based on the shapes present in the documents.

The early work reported by (Yu et al., 1997) uses

a set of rules applied to the lines of a symbol to clas-

sify it on generic engineering drawings, and not only

P&IDs, and each symbol has a set of rules. (Wenyin

the use of a multi-scale sliding window and Con-

nected Component Analysis to locate the symbols and

a Convolutional Neural Network (CNN) to classify

them. This model requires labelled data to learn, and

therefore a large sample of data has to be labelled in

order to apply this method.

In the last few years, researchers applied several

Computer Vision techniques in the P&ID domain.

(Elyan et al., 2018) proposed a heuristic to locate

symbols and random forest combined with clustering

more robust and modern methodology was proposed

by (Rahul et al., 2019). They use a Fully Convolu-

tional Network (FCN) and can do all the segmenta-

tion (detection and classification) with a single model.

It also uses rules to detect connections and relation-

ships. Finally, (Gao et al., 2020) uses the ResNet-

50 (He et al., 2015), a Faster Regional Convolutional

Neural Network (Faster RCNN), backbone with data

augmentation techniques to detect and classify sym-

bols. Once again, rules are applied to infer connec-

tions and relationships.

smaller and lighter. To further improve the perfor-

dataset. It has more than 100,000 question-answers

pairs from Wikipedia.

The models described before don’t work well with

tabular data. Therefore, (Herzig et al., 2020) intro-

duces TAPAS, an extension to BERT to encode tables

as input, to do question-answering over tables without

using logical forms as previous literature suggested.

Recently, (Chen et al., 2020) proposed a method to

apply Q&A to both textual and tabular data. “Early

fusion´´ is used to fuse tabular and textual units into

pre-trained Q&A models. (Nandy et al., 2021) also

proposed a pipeline, for text only, using a model built

on RoBERTa (Liu et al., 2019).

3 PROPOSED ARCHITECTURE

This section describes the proposed two-steps

methodology for extracting technical attributes from

P&ID and technical documentation in detail.

To be able to detect equipment in P&IDs, we first in-

gest all documentation into a database. Then, we use

this information and meta-data to detect equipment.

extracting all the text and text coordinates into a

type. Second, we search in the technical documen-

tation for documents classified as having information

regarding that type of equipment. Third, we divide

the information of these documents in two separate

contexts, namely: continuous text information, and

tabular information. Forth, we generate questions for

each technical attribute. For the question construc-

tion, we send a full text question. e.g.: what is the

power of the air conditioner. Finally, we send the

questions and the context information to two sepa-

the question answering model TAPAS (Herzig et al.,

2020). TAPAS is a BERT-like transformers model,

pretrained on raw tables and associated texts, allow-

ing to query tabular information. Finally, we compare

the best result of each model and save in the database

the one with higher score. Figure 5 shows the extrac-

tion of technical attributes.

REST APIs developed with fast-API and deployed in

Uvicorn, an ASGI server. Information extracted is

stored in MongoDB. MongoDB allows queries with

regular expressions, critical to equipment detection

in the P&IDs. To allow users to query semantically

in the information of all equipment, we use Elastic-

search, a search engine based on the Lucene library.

Documents are stored in a persistent storage account,

enabling the usage of containers for all other build-

ing blocks of the architecture. Finally, Google OCR

This section describes the results obtained for the pro-

posed Gutenbrain architecture, also reporting on the

faced challenges. It starts by focusing on the stage

of extracting information from P&ID, and then on the

stage of retrieving technical attributes from the stored

data.

4.1.1 OCR

Most of the P&IDs are scanned, have images, or are

exported from CAD files into vectors. The use of

of the text is extracted, ignoring the characters

overlapped with symbols, or 2) overlapped shape

and characters are merged and the retrieved text is

incorrect. In both cases, the equipment tag is unre-

coverable, as we have partial or incorrect information.

Second, multiple lines are hard to detect when

there is little space between, as we see in Figure 8.

This means that relevant information is lost, either by

removing full equipment or part of multi-line equip-

the text as separate words or separate text blocs. An

example can be seen in Figure 9.

Figure 9: Nonexistent spaces added by OCR capture.

To tackle these challenges, we performed an OCR

benchmark between Tesseract, Azure and Google

OCR. Tesseract is convenient, as it works on-

premises, but its performance is sub-par when com-

paring with both Azure and Google. When compar-

ing Azure and Google OCR regarding how they tackle

the three above mentioned challenges, Google outper-

4.1.2 Equipment Sanitisation

When extracting equipment, their codes can have dif-

ferent separation of components, as they are usually

not normalised. Sometimes they have hyphen sepa-

rating attributes (equipment type, system, subsystem),

other times they have space, and others they are in

multiple lines, having only the break of line to sepa-

rate them, as shown in Figure 11. To ensure the same

equipment is considered as such if it appears in sev-

eral documents, we sanitise the equipment. We add

containing 607 equipment records. These documents

are very big, but unlike the computer vision ap-

proaches where documents are split and shrinked, we

enlarged them in order to improve the OCR results.

The dataset is composed of a mix of documents with

native text, images, vectors and some text without

Unicode mapping.

Precision =

T P

T P + FP

(1)

false positives, and FN represents false negatives. We

also present the F1-measure, a metric that combines

precision and recall. Table 2 shows the achieved re-

sults, revealing that our system is able to achieve an

impressive performance, specially in terms of Preci-

sion. We can see that the recall is highly affected

when the document must be processed with OCR in

order to retrieve the text, since it introduces errors.

Table 2: Performance results for the equipment extraction

stage.

Results Precision Recall F1-measure

Text 0,984 0,824 0,897

Average 0,972 0,712 0,822

Current studies of P&ID information extraction

use image-based techniques, recognising symbols

through template matching. When compared with the

approach of previous works, our architecture achieves

4.2.1 Preprocessing of Technical Documentation

To be able to use technical documents’ information

in the question answering models, data was extracted

to the database in two different stacks: 1) continu-

ous text, and 2) tabular data. This will be important

when extracting technical attributes, as it is different

to search in continuous text or in tabular data. Be-

sides extracting this information, each document was

know its model, use, flow rate, output, pressure, in-

take temperature, viscosity and input connection.

The first step is to find the documents that have

create a set of questions to be used in both ques-

tion answering models. We use questions in full sen-

tences: ”what is the <attribute> of the <equipment

type>”. e.g.: what is the flow rate of the centrifu-

gal pump? The next step is to send the question and

all continuous text as context to our DistilBert model.

The model gives us the best answer and their score.

Afterwards, we send the same question to our TAPAS

base model, but using as context all tabular informa-

tion found in the selected documentation. Again, it

models. As opposed to directional models, which

read the text input sequentially (left-to-right or right-

to-left), the Masked Language Model (MLM) objec-

tive enables the representation to use both the left and

the right context, which allows to pre-train a deep

bidirectional Transformer. Since maintenance engi-

neering has special jargon, BERT’s sub-word repre-

sentations and word-piece tokenization are useful for

the out-of-vocabulary words that often appear in the

corpus. This only mitigates the special jargon issue to

can show the documents in the right pages for engi-

neers to validate the extracted attributes.

WORK

We have proposed an approach to extract equipment

and technical attributes from P&IDs and retrieve tech-

nical documentation from technical sheets, and de-

scribed an architecture to support this approach. We

have performed experiments on a manually labelled

dataset of P&IDs, containing 607 equipment, and

the performance results for the equipment extraction

tion of the spare parts to be procured and put in stock

for later use, and the definition of maintenance plans,

procedures, and manuals. The proposed architecture,

Gutenbrain, allows the saving of 16k per project, ei-

ther by extraction automatically the equipment infor-

mation or by allowing to search in the technical in-

formation using semantic search in content otherwise

unsearchable. The experimental validation presents

an average reduction of approximately the 60% of

engineers’ effort in cumbersome tasks of extracting

bols classification in engineering drawings. In 2018