Education
This page introduces the concepts and classification of smart microscopy, based on Rates, Passmore, Norlin & Kapitein (2025) npj Imaging. It is intended as a starting point for anyone new to the field. Anything missing or unclear? Let us know in the image.sc forum!
What is Smart Microscopy?
A conventional microscope is set up once and then runs with fixed parameters throughout the experiment. A smart microscope, by contrast, continuously analyses what it is seeing and responds — adjusting resolution, speed, field of view, or illumination in real time, depending on what is happening in the sample.
Smart microscopy is relevant across many biological applications, from capturing rare fast events to gently imaging fragile live samples over long periods. It is also not limited to light microscopy — feedback and automation appear in electron microscopy, atomic force microscopy, and other imaging modalities.
Why Do We Need Smart Microscopes?
Modern microscopes face fundamental physical constraints. Fluorescent dyes bleach, live samples can be damaged by too much light, and the diffraction limit caps spatial resolution. These constraints lead to unavoidable trade-offs, often illustrated by the pyramid of frustration:
Improving one parameter typically compromises others. Smart microscopy helps navigate these trade-offs dynamically, rather than accepting a fixed compromise.
For example, high spatial resolution usually requires slower imaging — but slow imaging may miss fast cellular events. Smart microscopy addresses this by letting the microscope adapt: imaging slowly most of the time, and only switching to high-speed mode when something worth capturing is detected.
A Brief History of Smart Microscopy
Automation has been part of microscopy for decades, but the transition to truly smart systems — where the microscope reacts to what it sees — is more recent.
Early automated systems (e.g. Zeiss Axiomat) could move the sample mechanically, but in parallel to imaging — the microscope did not yet react to what it saw.
Integrated digital cameras appear (Zeiss Axiovert 135, Leica M40) and Nikon introduces autofocus for z-positioning. Still automation without feedback on image content.
GFP and its variants revolutionise biology. Live-cell imaging becomes standard — and so does the challenge of balancing image quality with sample health.
This open-source, microscope-agnostic software standardised microscope control and made automation broadly accessible. A turning point for community-driven development.
Deep learning transforms image analysis — segmentation, detection, tracking — making real-time image interpretation computationally feasible for the first time.
Platforms like ImSwitch, Navigate, Arkitekt and others; LLMs being explored as user interfaces; community-driven interoperability standards. The field is rapidly expanding.
Key milestone
Smart microscopy became truly "smart" when real-time image analysis was coupled to automated hardware control — the microscope could observe the sample, interpret what it saw, and change its own settings, all within a single experiment.
Smart Microscopy Goals
Before choosing a smart microscopy approach, the most important question is: what is driving this experiment? We classify smart microscopy goals into five categories. These are not mutually exclusive — many experiments combine more than one.
🎯 Quality-Driven
Maintain a defined image quality metric throughout the experiment — SNR, sharpness, aberration level, or illumination uniformity.
Maintain quality⚡ Event-Driven
Detect a rare biological event (e.g. cell division, protein aggregation) and trigger a change in acquisition only when it matters.
Detect rare events📍 Target-Driven
Keep a specific region or specimen in the field of view throughout the experiment — tracking moving cells or maintaining an optogenetic illumination spot.
Follow the target🧠 Information-Driven
Use prior or population-level knowledge to direct imaging resources where they are informative, avoiding redundant acquisitions.
Use prior knowledge🔄 Outcome-Driven
Actively perturb the sample (e.g. via optogenetics) and respond to the biological reaction — creating a closed loop between the hardware and the biology.
Control the biologyQuality-Driven
Quality here refers to an experiment-specific metric — it could be signal-to-noise ratio, level of optical aberrations, focus sharpness, or even minimising photobleaching. A classic example is adaptive optics: deformable mirrors continuously correct wavefront aberrations so the point spread function stays optimal, even deep in tissue.
Event-Driven
Many biological events of interest are infrequent and unpredictable. Event-driven microscopy monitors at a low (less damaging) acquisition rate, detects the signature of an event, and only then switches to demanding high-resolution imaging.
Target-Driven
Living specimens move — cells migrate, worms crawl, embryos rotate. Target-driven microscopy tracks the specimen and adjusts stage position, field of view, or illumination pattern to keep the region of interest centred.
Information-Driven
When prior knowledge about the sample already exists — for instance, what a relevant phenotype looks like in a well-characterised model organism — that knowledge can guide where and how the microscope images, avoiding time and light spent on uninformative regions.
Outcome-Driven
The most sophisticated category: the microscope introduces a controlled perturbation and responds to how the biology reacts. This connects smart microscopy to cybergenetics — using control theory to programme living systems. Optogenetics is currently the most common perturbation tool in this context.
Smart Microscopy Strategies
After defining the goal, the next question is how the automation is implemented. All smart microscopy systems share the same basic loop:
Camera or detector captures the current frame
Software extracts features from the image in real time
A control algorithm determines what, if anything, needs to change
Hardware responds: stage moves, laser dims, filter changes…
Real-Time Analysis
The microscope must interpret its own images fast enough to act on them during the experiment. Common analysis tasks include:
Defining the outline of cells or structures in 2D or 3D. Common tools: CellPose, StarDist, Segment Anything Model (SAM).
Identifying that a specific event has occurred — an intensity peak, a morphological change, or a shift in dynamics.
Following a moving object through time, from simple centroid tracking to full shape-based approaches. Tools: WormTracker, Deep-worm-tracker, Cell Tracking Challenge entries.
Anticipating an event before it happens, using regression models or neural networks — letting the microscope prepare in advance.
Assigning cells or regions to categories (e.g. infected vs. healthy), using clustering algorithms (K-Means, hierarchical) or deep learning classifiers.
Feedback Control
Once the image has been analysed, a control algorithm decides what to do. The right strategy depends on how complex and variable the biology is:
Open-Loop Control
Applied sporadically, with no measurement of the outcome. Suitable for event-driven experiments where a well-understood action (e.g. phototag a cell) is triggered once.
Example: phototagging a detected cell with a predefined pulse of light.
Closed-Loop Control
Continuously measures the outcome and corrects based on it. The PID controller is the standard engineering solution.
Example: adaptive optics that constantly re-measures aberrations and updates the deformable mirror correction.
Adaptive Control
The controller's own parameters are updated based on the behaviour of the biology. AI and reinforcement learning are increasingly used here, especially for stochastic or heterogeneous biological systems.
Example: optogenetic cell steering where each cell responds differently to light.
A priori vs. on-the-fly automation
Not all automation needs to happen during the experiment. A priori automation — optimising settings before imaging begins — is well established and useful when conditions are stable. On-the-fly automation is what defines smart microscopy: settings change in response to the sample, in real time.
Actuation: What Can the Microscope Change?
Actuators are the hardware components the control algorithm can operate. They fall into four categories:
⚙️ Optomechanics
Stage position, z-depth, objective selection, filter wheels, scan region, illumination wavelength and intensity, wavefront correction. The physical "knobs" of the microscope.
🧪 Perturbations
Controlled inputs to the sample: light pulses for optogenetics or FRAP, drug flow via perfusion chambers, temperature changes, electric fields. Voluntary changes that make the sample react.
💾 Storage & Data
Smart data management — compressing or deleting irrelevant frames on the fly, cropping to regions of interest, real-time deskewing. Crucial for long acquisitions that would otherwise overflow storage.
💬 Communication
Connecting the microscope to the user or other systems: graphical region-selection interfaces, push notifications when events are detected, and increasingly, plain-language interfaces powered by large language models.
Current Challenges
Smart microscopy has made enormous progress, but important open problems remain. Being aware of these helps both in using existing systems critically and in contributing to the field.
- Bias — human and algorithmic Choices made during system design (training data selection, event definitions) encode biases. Explainable AI (xAI) methods that reveal why an algorithm made a decision are a growing priority.
- Interoperability Most smart microscopy systems are built for a specific microscope or software stack. Making systems work across different hardware and imaging modalities requires open APIs and universal protocols.
- Data volume and formatting High-resolution time-lapse experiments generate massive datasets. Standardised file formats and metadata (e.g. via the Open Microscopy Environment, OME) are essential for storage, reproducibility, and cross-lab comparison.
- Reproducibility The "black box" nature of many AI models makes experiments hard to reproduce. Standardised calibration protocols, annotated training datasets, and transparent reporting are needed.
- No one-size-fits-all solution Every biological question needs a different configuration. Modular, system-agnostic designs — where components can be combined or reused across experiments — are the direction the field is moving.
- Accessibility Smart microscopy tools often require significant technical expertise. Simpler interfaces, better training resources, and plain-language interaction will be critical for adoption across biology as a whole.