Rosary Labs / Blog
Can Vision-Language Models Reliably Extract Equipment Tags from P&IDs?
We experimented with five state-of-the-art Vision Language Models to extract equipment tags from Piping and Instrumentation Diagrams (P&IDs). The findings revealed varying levels of accuracy, hallucination tendencies, and practical limitations.
3 min read, 4 January 2026
How did it all start?
We were curious about the generic capability of Vision Language Models (VLMs) to extract equipment tags from P&IDs, specifically their accuracy. We tested ChatGPT 5.2, Claude Opus 4.5, Gemini 3 Pro Preview, GLM 4.6v, and Qwen3 235B.
Experiment setup
To ensure a reliable comparison, we created a ground-truth dataset from manual extractions. This dataset acted as the accuracy benchmark for VLM-generated results.
Ground-truth dataset
The dataset was manually extracted by an instrumentation engineer:
- Sensors (57): TAH10X, XA10X, XR10X, HS10X, ZI10X, LIR100, LAL100, LAH100, PIR100, LAL101, LAH102, PIR101, FIR100, PAL101, PAH101, ZA104, FI100, TI10X, LI100, PIT100, LI101, LI102, PIT101, FIT100, ZI104, YLAR, YLG, LCP100, OIT10X, LIT100, LE100, LS101, LS102, ZS104, FIR101, FIA101, FIT101, PAH200A, PAL200B, PIR200C, PI200, PIT200, PDIT201, PDI201, PDIR201B, PDAH201A.
- Valves (32): V-101 through V-114, V-117, and V-201 through V-222.
- Pumps: P-101, P-102, P-103.
- Motors: M-101, M-102, M-103.
- Filters: F-201, F-202, F-203.
Time to experiment with Vision Language Models
We tested models representing the most advanced capabilities from OpenAI, Anthropic, Google, Alibaba, and Z.ai:
And the prompt?
Extract all the equipment tags from this attached piping and instrumentation diagram,
be as thorough as you can as if your life depends on it. Extract it as an
instrumentation engineer, for the purpose of asset registry. Extract unique
engineering tag identifiers only. Format your output as a block of newline
separated text. Go!
Our initial prompt revealed unexpected VLM behaviours and made the differences between models immediately visible.
Vision Language Model output
Claude Opus 4.5
Claude returned a detailed equipment-tag list, including pumps, motors, filters, valves, pressure and flow instrumentation, level instruments, and control equipment. It also added useful notes explaining the tag families and their engineering roles.
Gemini 3 Pro Preview
Gemini grouped its output into booster pumps and motors, bag-filter vessels, valves, and instrumentation. The structured grouping was useful, but several tags differed from the ground truth and required verification.
GPT-5.2
GPT-5.2 refused to produce an unverified asset register from a low-resolution raster preview. It correctly identified that guessing tag IDs could create harmful false records and requested a higher-quality source PDF or cropped regions.
Qwen3 VL 235B and GLM-4.6V
The remaining models produced partially useful extraction lists, but both showed the same central limitation: a plausible-looking list is not necessarily an accurate list. Hallucinated and misread identifiers still need to be measured against ground truth.
What we learned
VLMs can reason about diagram context, but they should not be evaluated from a single impressive example. A production workflow needs:
- High-resolution source documents
- A ground-truth dataset
- Structured output
- Duplicate and hallucination checks
- Quantitative metrics for precision, recall, and F1 score
The next step is to formalise the evaluation process. Read Part 1: quantitative evaluation to see how we measure these results.