Embedded System Development Process
for AI Systems

Lecture Agenda

7.1Development Process Models for AI Systems1 Hour
7.2Requirements Engineering for AI Embedded Systems1 Hour
7.3Design & Implementation of AI Systems1 Hour
7.4Integration and Testing of Embedded AI Systems1 Hour
7.5Case Studies and Failures in AI Embedded Systems1 Hour

Learning Objectives

Compare traditional embedded SDLC models against AI/ML-augmented development lifecycles
Elicit and specify requirements that account for data, model behavior, and resource constraints
Apply hardware-software co-design principles to deploy AI models on constrained devices
Design and execute a testing strategy that addresses the non-determinism of ML components
Analyze real-world embedded AI failures to extract actionable engineering lessons

7.1 — Development Process Models
for AI Systems

Duration: 1 Hour

Why Traditional Models Fall Short

Classic embedded lifecycles (Waterfall, V-Model) assume deterministic, specification-complete behavior
AI components are data-dependent, probabilistic, and drift over time — behavior isn't fully known until trained and tested
Two artifacts to manage instead of one: code and model/data pipeline, each with its own versioning and change cadence
Model performance is empirically determined, not provably correct from specification
Data quality and representativeness affect correctness — bugs can live in the data, not the code

Traditional embedded lifecycles like Waterfall and the V-Model are built on a fundamental assumption: you can fully specify what the system should do before you build it, and then verify that the implementation matches the spec. AI breaks this assumption in multiple ways. First, an ML model's behavior is emergent from training data — you can't write a complete specification for what a neural network will do. Second, model behavior changes over time as real-world data drifts away from the training distribution. Third, you now have two separate artifact streams — code and the data/model pipeline — each evolving at different rates with different versioning needs. A bug might live in the data preprocessing, not the code.

The V-Model, Adapted for AI

The V-Model naturally maps to embedded AI development if we add a parallel branch. The traditional left side shows decomposition from system requirements down to module design and implementation. The right side mirrors each level with corresponding verification. For AI, we insert a data and model V alongside the software V. Data requirements and model requirements on the left connect horizontally to data quality validation and model validation on the right. The key adaptation is synchronizing this AI V with the software V at integration milestones. When the software integration test happens, you simultaneously verify that the model meets its latency and accuracy targets on the actual hardware. This parallel-V structure preserves the traceability that safety-critical domains require while accommodating the empirical nature of ML development.

Agile for Embedded AI

Sprints split into code sprints (firmware/software) and model sprints (experiment-driven, metric-tracked)
Hardware-in-the-loop testing doesn't fit neatly into 2-week sprints
Model training cycles can be long and expensive (hours to days of GPU time)

Agile: Parallel Track Pattern

Software/Firmware Track (Scrum)

2-week sprints
Sprint planning, daily standup, review
Embedded HW testing in sprint
Continuous integration for code

Model Track (Kanban + Experiment Tracking)

WIP-limited flow
Experiment logging (MLflow, W&B)
Longer cycle: data → train → evaluate
Promotion gates at integration points

▼ Integration Checkpoint: Model + Software merged and validated on target ▼

This slide shows the parallel-track pattern in more detail. On the left, the software and firmware team runs standard two-week Scrum sprints with daily standups, sprint reviews, and embedded hardware testing baked into each sprint. On the right, the ML team runs a Kanban flow with work-in-progress limits. They log experiments using tools like MLflow or Weights & Biases. Their cycle is longer because it encompasses data collection, model training, and evaluation. At scheduled integration checkpoints — perhaps every four to six weeks — the model track delivers a validated model bundle to the software track, and the combined system is tested on target hardware. This pattern gives each discipline the cadence it needs while maintaining alignment.

MLOps and Its Embedded Variant (TinyMLOps)

This diagram shows the standard MLOps loop with embedded-specific additions highlighted in orange. The standard loop flows from data collection through feature engineering, training, validation, deployment, monitoring, and back to retraining. For embedded AI, we add two critical stages. First, model compression — quantization, pruning, and knowledge distillation — between validation and deployment. This is where we reduce model size and computation to fit hardware constraints. Second, cross-compilation and target-specific deployment — converting the compressed model to run on TFLite Micro, ONNX Runtime, or a vendor SDK, then flashing it to the device. On the monitoring side, we face a constraint: limited telemetry bandwidth means we can only send statistical summaries, not raw data, back from the field. The retraining loop is consequently slower and more expensive than cloud MLOps.

TinyMLOps: Embedded-Specific Constraints

Model compression stage (quantization, pruning, distillation) required before packaging
Cross-compilation / target-specific build stage: converting to TFLite Micro, ONNX Runtime Mobile, vendor SDKs
On-device monitoring is constrained — limited telemetry bandwidth, statistical summaries only
Retraining loop is slower and costlier: OTA update risk, certification re-runs, limited connectivity
Model rollback capability is essential — every OTA model update carries risk

Let's elaborate on the embedded-specific constraints that differentiate TinyMLOps from standard cloud MLOps. The model compression stage is not optional — you must quantize or prune the model to fit within the target's memory and compute budget. This means the validation stage must test both the uncompressed model accuracy and the post-compression accuracy, documenting any degradation. Cross-compilation introduces another source of potential numerical differences — the model may behave slightly differently when compiled for an ARM Cortex-M versus running in a Python training environment. On-device monitoring must be lightweight — you can't stream raw sensor data to the cloud. Instead, you send statistical summaries like accuracy trends or confidence distributions. And the retraining loop is inherently slower because each OTA update requires risk assessment, potentially re-running safety certifications, and handling cases where the device may have limited or no connectivity.

Hybrid Process Model (Recommended)

Concept & Feasibility — define the AI task, target hardware envelope, power/latency budget
Data Strategy — sourcing, labeling, augmentation plan
Parallel Tracks — Software/Firmware (V-model or Agile) + Model (iterative MLOps)
Co-Design Integration Points — scheduled sync where model size/latency checked against hardware budget
System Integration & Test
Deployment & Field Monitoring
Continuous Improvement Loop — retraining, OTA updates

The hybrid process model is what I recommend as a reference model for embedded AI projects. It starts with concept and feasibility — before writing any code or collecting data, you define the AI task, the target hardware, and the power and latency budgets. This prevents the common mistake of training a model that works great on a GPU but can't run on the target. Next comes a data strategy — where will data come from, how will it be labeled, what augmentation is needed? Then you run parallel tracks: the software team follows V-Model or Agile, while the ML team runs iterative experiments. Integration checkpoints are critical — these are scheduled milestones where you verify that the model still fits within the hardware budget after each experiment cycle. After system integration and testing, you deploy with field monitoring. The loop closes with continuous improvement through retraining and OTA updates.

Process Model Comparison

Aspect	Waterfall	V-Model	Agile	MLOps-Hybrid
Handles changing requirements	Poor	Poor	Good	Good
Handles data-driven uncertainty	Poor	Moderate	Moderate	Strong
Traceability / certification fit	Strong	Strong	Weak	Moderate
Suited to safety-critical embedded AI	Rare use	Common (adapted)	Rare alone	Common (with V overlay)
Manages data/model lifecycle	No	No	Partial	Yes

This comparison table summarizes why no single traditional process model works well for embedded AI. Waterfall and the V-Model provide strong traceability needed for certification but handle changing requirements and data-driven uncertainty poorly. Agile is responsive to change but lacks the traceability required by safety standards and doesn't naturally manage the ML lifecycle. The MLOps-Hybrid model — overlaying MLOps onto a V-Model backbone — is the strongest overall fit. It handles changing requirements and data uncertainty through iterative experimentation while maintaining enough structure for certification through the V-Model overlay. For safety-critical applications like automotive or medical devices, the V-Model backbone is essentially mandatory, but it must be augmented with parallel ML experiment tracks and MLOps practices.

Section 7.1 — Recap

No single process model fully fits embedded AI — most teams overlay MLOps onto a V-Model or Agile backbone
The core shift: manage data and model lifecycle as first-class artifacts, not side effects of "software"
Embedded constraints (compression, cross-compilation, limited telemetry) add stages that cloud MLOps doesn't need
Safety-critical domains require the traceability of V-Model with the responsiveness of iterative ML experimentation

Section 7.1 recap: The key takeaway is that no single traditional process model works for embedded AI development. Instead, the most effective approach is to overlay MLOps practices onto either a V-Model or Agile backbone, depending on your safety and certification requirements. The fundamental shift is treating data and models as first-class artifacts with their own versioning, validation, and lifecycle management — not as afterthoughts to software development. Embedded AI also adds unique constraints that cloud MLOps doesn't face: mandatory model compression, cross-compilation for target architectures, and limited telemetry for field monitoring. For safety-critical applications, the V-Model backbone provides the traceability that standards require, while the MLOps overlay provides the iterative experimentation that ML development demands.

7.2 — Requirements Engineering
for AI Embedded Systems

Duration: 1 Hour

What's Different About AI Requirements

Traditional: "the system shall do X given input Y" — fully specifiable
AI requirements often stated statistically: "classify with ≥95% precision on the target distribution"
Requirements must cover four layers: System, Functional/Behavioral, Data, and Model Performance
"Correctness" is probabilistic, not binary — a fundamentally different contract

Traditional embedded requirements follow a pattern: the system shall do X given input Y, and you can verify this with a test case that checks exact expected output. AI requirements break this pattern because model behavior is emergent from training data, not derived from a formal specification. Instead, you state requirements statistically — the model shall classify with at least 95 percent precision on the target distribution. This means requirements must cover four distinct layers: system-level requirements about overall behavior, functional requirements about the AI task itself, data requirements about training data quality and representativeness, and model performance requirements about accuracy, robustness, and fairness. The correctness contract is probabilistic rather than binary, which has profound implications for verification and validation.

Functional & Non-Functional Requirements

Functional

Task definition, input/output modality, operating conditions

Non-Functional (Embedded-Specific)

Latency — real-time inference deadlines
Power/energy budget — battery-operated, energy harvesting
Memory footprint — SRAM/Flash limits (often KB, not MB)
Thermal constraints — sustained inference may cause throttling
Reliability & availability under intermittent connectivity

Functional requirements for AI systems define the task itself — what classification or regression problem are we solving, what are the input modalities like camera, microphone, or accelerometer, and what operating conditions must be supported. But the embedded context adds significant non-functional requirements. Latency is critical for real-time systems — a self-driving car can't wait 200 milliseconds for an object detection inference. Power and energy budgets matter enormously for battery-operated or energy-harvesting devices. Memory footprint is often measured in kilobytes rather than megabytes, meaning model architecture decisions are directly constrained. Thermal constraints mean sustained inference at high frequency may cause the processor to throttle. And reliability requirements must account for intermittent connectivity — the system must function gracefully when it can't reach the cloud.

Data Requirements (Unique to AI Systems)

Data sourcing — representativeness of target deployment environment
Labeling requirements — accuracy, inter-annotator agreement thresholds
Data volume & class balance — minimum samples per class, augmentation strategy
Data governance — privacy, consent, regulatory compliance (biometric, medical data)
Edge case & out-of-distribution (OOD) requirements — what must the system do when it doesn't know?

Data requirements are entirely new to embedded AI compared to traditional software requirements. Data sourcing requirements must specify that training data is representative of the actual deployment environment — training on sunny daytime images and deploying to a night-time application will fail. Labeling requirements define the accuracy needed from human annotators and set inter-annotator agreement thresholds — if two labelers disagree more than 10 percent of the time, the labels may be too noisy. Volume and class balance requirements ensure sufficient examples of each scenario, especially rare but critical cases. Data governance is increasingly important — regulations like GDPR and HIPAA constrain what data can be collected and how it must be handled. Perhaps most critically, you need out-of-distribution requirements — what should the system do when it encounters something it's never seen before? Failing silently is unacceptable for safety-critical systems.

Model Performance Requirements

Beyond accuracy: precision/recall tradeoffs relevant to the application cost matrix
Robustness under sensor noise, adversarial perturbation, lighting/temperature variation
Explainability/interpretability requirements (often mandated in regulated domains)
Fairness/bias requirements across relevant subpopulations
Calibration — confidence scores should reflect actual likelihood of correctness

Model performance requirements go well beyond simple accuracy. You need to specify precision and recall tradeoffs based on the application's cost matrix. For a medical screening AI, missing a positive case — a false negative — is far more costly than a false alarm. Robustness requirements specify how the model must perform under real-world conditions like sensor noise, partial occlusion, varying lighting, or temperature-induced sensor drift. In regulated domains, explainability requirements mandate that the model's decisions can be interpreted — a doctor needs to know why the AI recommended a treatment, not just what it recommended. Fairness and bias requirements ensure the model performs equitably across different demographic subgroups. Finally, calibration requirements ensure that when the model says it's 90 percent confident, it's actually correct about 90 percent of the time — poorly calibrated confidence scores undermine trust and decision-making.

Safety & Regulatory Requirements

Standard	Domain	Key AI-Relevant Aspect
ISO 26262	Automotive	ASIL levels, hardware metrics, safety lifecycle
IEC 61508	General industrial	SIL levels, systematic capability
IEC 62304	Medical software	Software safety classes, risk management
DO-178C	Avionics	DAL levels, with ED-324 AI supplement

Safety requirements must define fallback/degraded-mode behavior when the model is uncertain
Traceability: every model behavior must trace back to a validated requirement

Safety and regulatory requirements are critical for embedded AI in regulated domains. ISO 26262 for automotive defines Automotive Safety Integrity Levels that dictate the rigor of development processes. IEC 61508 covers general industrial applications with Safety Integrity Levels. IEC 62304 specifically addresses medical device software. DO-178C for avionics has been supplemented with ED-324 and AMC 20-115 specifically addressing AI and ML in airborne systems. A critical requirement across all these standards is specifying fallback and degraded-mode behavior — what happens when the AI model is uncertain, produces anomalous outputs, or the hardware is degraded? You must define these safe states explicitly. Traceability is another challenge — every model behavior should trace back to a validated requirement, which is difficult for black-box neural networks and drives interest in more interpretable architectures.

Elicitation Techniques Adapted for AI

Stakeholder interviews plus data audits — what data actually exists vs. what's assumed
Scenario-based elicitation including failure scenarios — what happens when the model is wrong?
Prototyping with a baseline model early to surface unstated requirements
Requirements co-evolve with data exploration — expect iteration, not a frozen spec

Requirements elicitation for AI systems requires adapted techniques. Standard stakeholder interviews must be supplemented with data audits — understanding what data actually exists in the production environment versus what stakeholders assume exists. A stakeholder might say "we have years of sensor data" but upon auditing, discover the data was collected at a different sampling rate, with different calibration, or with significant gaps. Scenario-based elicitation must include failure scenarios — stakeholders must explicitly decide what the system should do when the model is uncertain or wrong. Prototyping with a baseline model early in the process is invaluable because it surfaces unstated requirements that stakeholders didn't think to mention until they see the system making real predictions. And critically, expect requirements to co-evolve with data exploration — you won't freeze the spec on day one. The data reveals capabilities and limitations that reshape the requirements.

Example: Smart Doorbell Person Detector

ID	Requirement	Type
FR-01	System shall detect human presence in frame within 200ms of trigger	Functional
NFR-01	Inference shall consume ≤15mA average current on MCU X	Non-Functional
NFR-02	Model binary shall fit within 256KB flash	Non-Functional
DR-01	Training data shall include ≥30% low-light/night images	Data
MR-01	False negative rate for adult humans shall be ≤2% at 95% CI	Model Perf.
SR-01	On sensor fault, system shall default to "assume presence" (fail-safe)	Safety

This example shows how the four requirement layers work together in practice for a smart doorbell person detector. FR-01 is a traditional functional requirement — detect humans within 200 milliseconds. NFR-01 and NFR-02 are embedded-specific non-functional requirements — the inference must consume no more than 15 milliamps and fit within 256 kilobytes of flash memory. DR-01 is a data requirement unique to AI — at least 30 percent of training images must be low-light or night scenes, ensuring the model works in the conditions where it will actually be used. MR-01 is a model performance requirement — the false negative rate for adult humans must be no more than 2 percent at 95 percent confidence interval. SR-01 is a safety requirement — if the sensor fails, the system should default to assuming someone is present, which is the safer option. Together, these six requirements span all four layers and would be impossible to achieve with traditional requirements alone.

Section 7.2 — Recap

AI requirements engineering adds data and statistical performance contracts on top of traditional specs
Safety-critical systems require explicit degraded-mode and traceability requirements
Requirements co-evolve with data exploration — early prototyping surfaces hidden assumptions
Requirements must balance precision vs. recall based on real-world cost asymmetries

Section 7.2 recap: Requirements engineering for AI embedded systems adds two new dimensions — data requirements and statistical performance contracts — on top of traditional functional and non-functional specifications. Safety-critical systems require explicit degraded-mode behavior when the model is uncertain and traceability from model behavior back to validated requirements. Requirements should co-evolve with data exploration rather than being frozen upfront, which means early prototyping with baseline models is essential. And you must carefully balance precision versus recall based on the real-world cost asymmetries of your application — missing a pedestrian is far worse than a false alarm, but an overly sensitive system that cries wolf loses user trust.

7.3 — Design & Implementation
of AI Systems

Duration: 1 Hour

Layered Embedded-AI Architecture

Communication / Monitoring Layer OTA updates, telemetry, drift monitoring

▲

Decision / Actuation Layer Post-processing, control logic, fallback controllers

▲

Inference Layer ML model execution, accelerator/NPU/MCU, runtime (TFLite, ONNX)

▲

Preprocessing Layer Filtering, normalization, feature extraction, windowing

▲

Sensing Layer Sensors, ADC, signal conditioning, calibration

The layered reference architecture for embedded AI systems consists of five layers. At the bottom, the sensing layer encompasses physical sensors, analog-to-digital conversion, and signal conditioning — the quality of sensor data directly limits model performance. Above that, the preprocessing layer handles filtering noise, normalizing values, extracting features, and windowing temporal data — this layer is often implemented in firmware and can significantly affect both latency and accuracy. The inference layer is where the model actually executes, whether on an MCU, NPU, or DSP, using a runtime like TFLite Micro or ONNX Runtime. The decision and actuation layer implements post-processing of model outputs, control logic, and crucially, fallback controllers that take over if the AI pipeline fails. Finally, the communication and monitoring layer handles OTA updates, telemetry streaming, and drift monitoring — the feedback loop that enables continuous improvement.

Hardware-Software Co-Design

Compute Target	Example	Model Constraints	Typical Use
MCU-class	Cortex-M4/M7	Models in KBs, integer-only ops	TinyML, keyword spotting
MCU + NPU/DSP	Ethos-U, Hexagon	Larger models, quantized INT8	Visual wake word, anomaly detection
SoC with GPU/NPU	Jetson-class	Larger CNNs/transformers feasible	Object detection, multi-modal

Co-design loop: architecture search constrained by target's memory, compute (MACs/sec), and power
Roofline analysis determines if a model is compute-bound or memory-bound on the target

Hardware-software co-design means the hardware target drives the model design, not the reverse. At the MCU-class level with Cortex-M processors, you're limited to models measured in kilobytes using integer-only operations — this is the domain of TinyML applications like keyword spotting and simple anomaly detection. Adding a neural processing unit or DSP accelerator like ARM Ethos-U or Qualcomm Hexagon allows larger models with INT8 quantization, suitable for visual wake word detection or more complex anomaly detection. At the SoC level with GPU or NPU like NVIDIA Jetson, you can run larger CNNs and even transformer models for object detection and multi-modal processing. Roofline analysis is a valuable tool here — it helps you determine whether your model is compute-bound (limited by operations per second) or memory-bound (limited by data transfer bandwidth), which guides optimization decisions.

Model Optimization Techniques

Quantization: FP32 → INT8/INT4; QAT (quantization-aware training) retains better accuracy than post-training
Pruning: structured (remove channels/filters, hardware-friendly) vs. unstructured (sparse weights, needs special HW)
Knowledge distillation: train a small "student" model to mimic a larger "teacher"
NAS: Neural Architecture Search constrained by hardware cost models (latency-aware)
Operator fusion & graph optimization during compilation (TFLite converter, TVM, vendor SDKs)

Model optimization is where the rubber meets the road for embedded AI deployment. Quantization reduces numerical precision from 32-bit floating point to 8-bit or even 4-bit integers. Post-training quantization is simpler but may lose more accuracy than quantization-aware training, which incorporates quantization simulation into the training process. Pruning removes unnecessary weights or entire channels — structured pruning that removes complete filters is hardware-friendly because it reduces actual computation, while unstructured pruning creates sparse matrices that need specialized hardware support. Knowledge distillation lets you train a compact student model that mimics a larger teacher's soft output probabilities, often achieving better accuracy than training the small model from scratch. Neural Architecture Search automates the search for optimal model architectures, but for embedded targets, the search must be constrained by hardware cost models like latency and memory, not just accuracy. Finally, compiler-level optimizations like operator fusion — merging conv + bias + activation into a single kernel pass — can dramatically improve inference speed.

Optimization Impact Example

Technique	Model Size	Accuracy (mAP)	Inference Latency
Baseline (FP32)	4.2 MB	78.3%	85 ms
+ INT8 PTQ	1.1 MB	77.1%	32 ms
+ INT8 QAT	1.1 MB	77.9%	31 ms
+ Structured Pruning + QAT	0.6 MB	76.5%	18 ms
+ Knowledge Distillation	0.5 MB	77.2%	15 ms

Example values for object detection on ARM Cortex-M7 @ 400MHz. Always validate on actual target hardware.

This table illustrates the cumulative impact of optimization techniques on a real object detection model deployed on an ARM Cortex-M7 at 400 megahertz. The baseline floating-point model is 4.2 megabytes with 78.3 percent mAP and 85 millisecond latency. Post-training quantization to INT8 reduces the model to 1.1 megabytes with only a 1.2 percent accuracy drop but achieves a 2.6x speedup. Quantization-aware training recovers about 0.8 percent of that accuracy loss. Adding structured pruning cuts the model in half again with a modest additional accuracy trade-off. Knowledge distillation from a larger teacher further improves the size-accuracy-latency trade-off. The key point is that these techniques stack — and you must validate each step on actual target hardware, not just a simulator, because numerical behavior can differ. The goal is finding the minimum acceptable accuracy within your size, latency, and power budgets.

Toolchains for Embedded AI

Model Runtimes

TensorFlow Lite Micro
ONNX Runtime Mobile
Apache TVM
CMSIS-NN

Development Platforms

Edge Impulse
STM32Cube.AI
MCUXpresso SDK
NVIDIA JetPack

Version Control Triad

Git (code)
DVC (data/models)
MLflow (experiments)
Git-LFS (large files)

The toolchain for embedded AI spans three categories. Model runtimes handle model loading and inference on the target — TensorFlow Lite Micro is the most widely used for microcontrollers, ONNX Runtime Mobile offers cross-platform support, Apache TVM provides ahead-of-time compilation for diverse hardware targets, and CMSIS-NN is ARM's optimized neural network library for Cortex-M processors. Development platforms like Edge Impulse and STM32Cube.AI provide end-to-end workflows from data collection through deployment, abstracting much of the complexity. The version control triad is crucial — you need to version three distinct artifact types: code (managed by Git), model weights and datasets (managed by DVC or Git-LFS), and experiment configurations (managed by MLflow). Without this triad, you can't reproduce a model deployment, which is essential for debugging and certification.

Design Patterns for Reliability

Confidence thresholding — reject/flag low-confidence inferences rather than acting on them
Ensemble-lite / redundancy — cheap secondary heuristic check on top of model output for safety-critical actuation
Watchdog + fallback controller — classical control logic that takes over if the AI pipeline stalls or produces anomalous output
Graceful degradation — reduce model complexity/frequency under thermal or power stress

Several design patterns help make embedded AI systems more reliable. Confidence thresholding is the simplest — if the model's output confidence is below a threshold, the system flags the result rather than acting on it, triggering a fallback behavior. Ensemble-lite adds a cheap secondary check — for example, a simple rule-based detector that verifies the neural network's output makes physical sense, providing a safety net for safety-critical actuation. The watchdog plus fallback controller pattern is essential for safety-critical systems — a classical control algorithm monitors the AI pipeline and takes over if inference stalls, produces outlier outputs, or exceeds timing deadlines. Graceful degradation reduces the model's complexity or inference frequency when the system is under thermal or power stress, maintaining some capability rather than failing completely.

Implementation Checklist

Model fits memory budget with margin (typically ≥20% headroom)
Inference latency measured on actual target hardware, not simulator only
Numerical behavior validated after quantization (accuracy delta documented)
Fallback / safe-state logic implemented and testable independently of the model
OTA update mechanism supports model rollback
Confidence thresholding implemented with threshold derived from validation data

Before declaring implementation complete, run through this checklist. First, verify that the model fits within the memory budget with at least 20 percent headroom — you need margin for runtime memory allocation, stack space, and potential future updates. Second, always measure inference latency on the actual target hardware, not just a simulator — cycle-accurate simulators exist but don't capture all real-world timing behavior. Third, document the accuracy delta after quantization — if quantization caused a 3 percent accuracy drop, that needs to be accepted and tracked. Fourth, the fallback and safe-state logic must be implemented and independently testable — you should be able to verify the fallback works even if the model is completely broken. Fifth, ensure your OTA mechanism supports model rollback — if a new model performs worse in the field, you need to revert. And finally, confidence thresholds should be derived from validation data analysis, not arbitrary values.

Section 7.3 — Recap

Embedded AI design is fundamentally a co-design problem — hardware limits shape model choice, not the reverse
Optimization techniques (quantization, pruning, distillation) trade accuracy for footprint/latency
Must validate post-transformation on actual hardware, not just in the training environment
Reliability patterns (confidence thresholding, watchdog, fallback) are requirements, not nice-to-haves

Section 7.3 recap: The fundamental insight is that embedded AI design is a co-design problem — the hardware target determines what models are feasible, not the other way around. Optimization techniques like quantization, pruning, and knowledge distillation trade accuracy for reduced footprint and latency, and these trade-offs must be validated after transformation on the actual target hardware, not assumed from training-environment results. The toolchain must version code, data, and models as a triad. And reliability design patterns — confidence thresholding, ensemble checks, watchdog controllers, and graceful degradation — are not optional extras but essential requirements for any deployed AI system, especially in safety-critical domains.

7.4 — Integration and Testing
of Embedded AI Systems

Duration: 1 Hour

Why AI Testing Differs

Non-determinism — same input can behave differently across FP vs. quantized builds, or across HW revisions
No complete oracle — "correct" output is statistical, not exact
Distributional risk — system may pass all test cases yet fail on real-world data drift
Emergent behavior — model behavior at scale may differ from controlled test environments

AI testing differs from traditional software testing in four fundamental ways. First, non-determinism: a model running in floating-point may produce slightly different outputs than the same model quantized to INT8, and hardware-level differences like FPU vs. integer-only execution paths add another source of variation. Second, there's no complete oracle — in traditional testing you define expected output exactly, but for a neural network, the best you can do is statistical expectations. Third, distributional risk means the model could pass every test case in your test suite but still fail in the real world because the deployment distribution has drifted from the training distribution. Fourth, emergent behavior means that model behavior observed at scale in production may differ from behavior in controlled testing environments due to rare input combinations or environmental conditions.

The Adapted Testing Pyramid

The adapted testing pyramid for embedded AI has five layers. At the base, unit tests verify individual functions — preprocessing correctness, tensor shape handling, quantization arithmetic. These are fast, cheap, and numerous. Above that, model validation tests check accuracy, precision, recall, and confusion matrix analysis on held-out and adversarial test sets. Integration tests verify the full pipeline from sensor through preprocessing through inference to actuation, running on target hardware. Hardware-in-the-loop tests use real sensors and actuators with simulated or replayed environment stimuli — for example, playing back recorded sensor data through the full system. At the apex, field and operational tests involve pilot deployment with monitoring before full rollout. Notice the pyramid narrows upward — you run many fast unit tests and fewer expensive field tests. But each layer adds coverage that lower layers cannot provide.

Model-Specific Test Categories

Test Type	Purpose	Key Metric
Regression	New model ≥ old model on full benchmark	Accuracy delta ≥ 0
Robustness	Noise, occlusion, lighting shifts, adversarial	Degradation bounds
Edge case / OOD	Inputs outside training distribution	Fallback trigger rate
Latency & Resource	WCET for inference, not just average	P99 latency, peak RAM
Bias / Fairness	Performance across subgroups	Max performance gap

Beyond the structural pyramid, ML introduces specific test categories. Regression testing verifies that a retrained model performs at least as well as the previous version on the full benchmark suite — not just on the new data it was retrained with. Robustness testing injects sensor noise, occlusion, lighting shifts, and adversarial perturbations to verify the model degrades gracefully. Edge case and out-of-distribution testing deliberately feeds inputs outside the training distribution to verify that safe fallback mechanisms trigger correctly. Latency and resource testing focuses on worst-case execution time, not just average — a model that averages 30 milliseconds but occasionally spikes to 500 milliseconds may miss a real-time deadline. And bias or fairness testing evaluates performance stratified across relevant subgroups — for example, verifying that a face detection model performs equally well across different skin tones.

Test Data Management

Held-out test set must be strictly isolated from training/validation to avoid leakage
Maintain a golden dataset version-locked to each model release for reproducible regression
Synthetic / augmented data can supplement rare edge cases (simulated rain, sensor dropout)
Data lineage tracking: where each test sample came from, how it was labeled, and who approved it

Test data management is often overlooked but critical for reliable testing. The held-out test set must be strictly isolated from training and validation data — any leakage inflates measured accuracy and creates a false sense of security. A golden dataset should be version-locked to each model release — when you retrain and deploy a new model, you run the same golden dataset to ensure comparable results. This golden set should include representative samples from all deployment conditions and edge cases. Synthetic and augmented data can supplement rare scenarios that are expensive or dangerous to collect in the real world — simulated rain, sensor dropout events, or unusual lighting conditions. Finally, data lineage tracking documents where each test sample originated, who labeled it, what the inter-annotator agreement was, and who approved it for inclusion in the test set. This lineage is essential for regulatory compliance and for debugging when test results are unexpected.

Verification & Validation (V&V) for Certification

Verification

"Was the model built right?"

Traceability from requirement to test case
Training process reproducibility
Code and model review

Validation

"Was the right model built?"

Operational need met in real conditions
Explainability tools (saliency maps, SHAP)
Independent review of performance claims

Documentation trail: dataset lineage, training config, evaluation report, known limitations

Verification and validation serve different but complementary purposes in certification. Verification asks "was the model built right?" — it focuses on whether the development process followed the plan, whether requirements trace to test cases, and whether the training process is reproducible. If someone reruns the training pipeline with the same data and configuration, they should get the same model. Validation asks "was the right model built?" — does the model actually meet the operational need in real-world conditions? Explainability tools like saliency maps and SHAP values help during validation by showing whether the model attends to the right features rather than spurious correlations. For example, if a medical AI is supposed to detect tumors but is actually reading the hospital watermark on X-ray images, explainability tools reveal this. The documentation trail for regulated domains must include dataset lineage, training configuration, evaluation results, and known limitations.

Continuous Testing / CI for ML

On every model or code commit → unit tests → accuracy regression → resource/latency check on target
Gate merge on thresholds — model doesn't merge if accuracy drops below baseline
Canary deployment — roll out to 5% of devices, monitor, then expand
Shadow mode — run new model alongside old, compare outputs without acting on new model

Continuous integration for ML extends the traditional CI pipeline with ML-specific stages. On every commit — whether code or model — the pipeline runs unit tests, then accuracy regression against the golden dataset, then resource and latency checks on target hardware or an accurate emulator. The merge is gated on predefined thresholds — if accuracy drops below the baseline, the model doesn't merge. Deployment strategies also differ. Canary deployment rolls the new model out to a small percentage of devices first, monitoring performance metrics before expanding. Shadow mode is even more cautious — the new model runs in parallel with the production model, receiving the same inputs and producing outputs that are compared, but the new model's outputs are never acted upon. Only after shadow mode confirms the new model behaves correctly is it promoted to production.

Section 7.4 — Recap

Testing embedded AI requires layering classical practices (HIL, WCET) with ML-specific ones (regression, robustness, OOD, fairness)
Passing tests is not sufficient proof of safety — continuous field monitoring closes the gap
Test data management (isolation, golden sets, lineage) is as important as the tests themselves
Canary and shadow deployment strategies reduce risk of new model rollout

Section 7.4 recap: Testing embedded AI requires a layered approach that combines classical embedded testing practices like hardware-in-the-loop testing and worst-case execution time analysis with ML-specific practices like regression testing, robustness testing, out-of-distribution testing, and fairness testing. Passing test cases is necessary but not sufficient — continuous field monitoring with lightweight drift detection closes the gap that statistical testing alone can't cover. Test data management — strict isolation, version-locked golden datasets, and data lineage tracking — is as critical as the tests themselves. And deployment strategies like canary releases and shadow mode reduce the risk of rolling out new models to production devices.

7.5 — Case Studies and Failures
in AI Embedded Systems

Duration: 1 Hour

Each case: Context → What Went Wrong → Root Cause → Lesson

Case 7.5.1 — Autonomous Driving: Fatal Pedestrian Collision (2018)

Context

A self-driving test vehicle operating in autonomous mode struck and killed a pedestrian crossing outside a crosswalk at night.

What Went Wrong

The perception system detected the pedestrian but repeatedly reclassified the object type, resetting its predicted trajectory each time. Emergency braking was disabled in autonomous mode, and the safety driver was not attentive.

Root Cause

Classification system not designed to handle jaywalking pedestrians reliably. Disabled automatic emergency braking during testing. Inadequate human-oversight design.

Lesson

Safety-critical embedded AI needs an independent, simpler safety layer (e.g., classical collision-avoidance braking) that operates regardless of the primary model's classification confidence or consistency.

This 2018 incident in Tempe, Arizona involved a self-driving test vehicle that struck and killed a pedestrian crossing the road outside a crosswalk at night. The perception system actually detected the pedestrian but repeatedly reclassified the object — cycling between vehicle, bicycle, and other categories — and each reclassification reset the predicted trajectory, preventing the system from building a consistent threat assessment. Compounding this, automatic emergency braking had been intentionally disabled during autonomous mode testing, and the human safety driver was not paying attention. The root cause traces to multiple process failures: the perception model lacked robustness for out-of-distribution scenarios like jaywalking, the system had no independent safety fallback, and the human oversight design was inadequate. The critical lesson: always maintain an independent, simple safety layer that operates on raw sensor data without relying on the AI model's classification. Classical collision-avoidance braking using radar or lidar should have been active regardless of what the camera-based classification system was doing.

Case 7.5.2 — Flight Control: Repeated Nose-Down Commands (2018–2019)

Context

A flight control augmentation system used a single angle-of-attack sensor to trigger automatic nose-down trim commands.

What Went Wrong

A faulty sensor reading caused repeated, aggressive automatic trim inputs that pilots struggled to counteract, contributing to two fatal accidents.

Root Cause

Reliance on a single sensor input without redundancy or plausibility cross-checks. Insufficient disclosure and training on the system's authority.

Lesson

Embedded systems with autonomous actuation of critical controls require sensor redundancy, cross-validation, and clear override/fallback authority for human operators.

The Boeing 737 MAX MCAS incidents in 2018 and 2019 killed 346 people across two crashes. The Maneuvering Characteristics Augmentation System was designed to automatically apply nose-down trim based on a single angle-of-attack sensor. When that sensor failed — giving erroneous high angle-of-attack readings — MCAS repeatedly pushed the nose down, and pilots struggled to counteract the repeated commands. The root cause was a single point of failure: the system relied on one sensor without cross-validation against the second AoA sensor, without airspeed consistency checks, and without limits on cumulative trim authority. Additionally, pilots were not adequately trained on MCAS behavior or given clear override procedures. The lesson for embedded AI is stark: any system that can autonomously actuate critical controls must have sensor redundancy with cross-validation, clear authority limits, and explicit human override mechanisms. These are not afterthoughts — they must be fundamental design requirements from the start.

Case 7.5.3 — Clinical Decision-Support AI Underperformance

Context

A commercial AI system was deployed to recommend cancer treatment plans, trained substantially on synthetic and limited real patient data.

What Went Wrong

The system recommended treatments misaligned with clinical guidelines in various real-world cases and failed to generalize beyond its narrow training distribution.

Root Cause

Insufficient representative training/validation data. Performance claims not adequately validated against independent, real-world outcomes before deployment.

Lesson

Model validation must specify representativeness of the deployment population. Performance claims should be independently verified before high-stakes deployment — not accepted from vendor benchmarks alone.

This case involves a commercial AI system deployed for clinical decision support in cancer treatment planning. The system was trained substantially on synthetic patient data and a limited set of real patient records. In clinical practice, it reportedly recommended treatments that deviated from established clinical guidelines for various patient profiles that weren't well-represented in training. The root cause was fundamentally a data requirements failure — the training data was not representative of the diverse patient population it would encounter in deployment. Performance claims made by the vendor were based on benchmarks that didn't reflect real clinical diversity. The lesson here reinforces section 7.2 on requirements: model validation requirements must explicitly specify the representativeness of the deployment population. And performance claims from vendors must be independently verified using real-world data from the actual deployment context before high-stakes deployment, not accepted at face value from controlled benchmarks.

Case 7.5.4 — Voice Assistant: False-Trigger Privacy Incidents

Context

Always-listening embedded voice assistants use a low-power wake-word detector to trigger cloud-based full processing.

What Went Wrong

Wake-word detectors occasionally misfired on unrelated speech, causing unintended audio capture and transmission to the cloud.

Root Cause

Models tuned primarily for recall (avoid missing activations) with insufficient attention to false-positive rate under diverse acoustic conditions.

Lesson

Requirements must balance false-positive vs. false-negative costs based on privacy and trust impact. Sensitive data should be filtered/minimized as close to the sensor as possible.

Multiple incidents have occurred where always-listening voice assistants like Amazon Echo and Google Home falsely triggered on speech that sounded like the wake word, causing unintended recording and cloud transmission of private conversations. In one widely reported case, a family's private conversation was forwarded to a random contact through the assistant's messaging feature. The root cause was a requirements trade-off that prioritized recall — avoiding missed activations — over precision — avoiding false activations. The model was tuned to be sensitive to the wake word, which inevitably increased false-positive rates, especially in noisy environments or with speech that phonetically resembled the trigger phrase. The lesson is about requirements balance: in privacy-sensitive applications, the cost of a false positive (unintended data capture) may be far worse than the cost of a false negative (user repeating the wake word). Requirements must explicitly weight this trade-off based on the trust and privacy impact, not just headline accuracy metrics.

Case 7.5.5 — Industrial Predictive Maintenance: Silent Model Drift

Context

An embedded vibration-analysis model deployed on factory equipment to predict failures.

What Went Wrong

Accuracy degraded silently over months as machinery wear patterns and ambient conditions shifted from the training distribution.

Root Cause

No on-device or fleet-level drift monitoring. Retraining cadence not tied to any measured performance signal.

Lesson

Embedded AI needs lightweight on-device drift/confidence monitoring and defined retraining triggers — silence is not evidence of continued correctness.

This case involves an embedded vibration-analysis model deployed on factory equipment to predict bearing failures and other mechanical issues. After several months of deployment, the model's prediction accuracy silently degraded as the machinery's actual vibration patterns shifted due to wear, environmental changes, and maintenance interventions that weren't present in the training data. The root cause was the absence of any drift monitoring — neither on-device nor at the fleet level. The model's accuracy was never re-measured against real-world data, and retraining was on a fixed calendar schedule rather than triggered by any performance signal. This is a textbook example of distributional risk from section 7.4. The lesson: every deployed embedded AI system needs lightweight on-device monitoring that tracks confidence distributions and prediction statistics. When these statistics shift significantly from training-time baselines, it should trigger a retraining evaluation. Don't assume silence means the model is still working correctly.

Cross-Case Themes

Theme	Cases Illustrating It
Single point of sensor/model failure without redundancy	Flight control, autonomous vehicle
Insufficient representative training/validation data	Clinical AI, predictive maintenance
Missing independent safety fallback layer	Autonomous vehicle, flight control
Wrong optimization target (recall vs. precision, benchmark vs. real-world)	Voice assistant, clinical AI
No field monitoring for drift/degradation	Predictive maintenance

Looking across all five cases, five recurring themes emerge. Single points of failure — whether in sensors or model classification — appeared in both the flight control and autonomous vehicle incidents. Insufficient training data representativeness was the root cause in both the clinical AI and predictive maintenance cases. Missing independent safety fallback layers contributed to severity in the autonomous vehicle and flight control cases. Wrong optimization targets — tuning for recall when precision matters, or benchmarking against non-representative data — affected the voice assistant and clinical AI systems. And absent field monitoring for drift allowed the predictive maintenance model to degrade silently. These themes directly map to the process areas covered in sections 7.1 through 7.4, reinforcing that failures typically stem from process gaps, not just model limitations.

Section 7.5 — Recap

Nearly every failure traces back to process gaps covered in 7.1–7.4: weak requirements, missing co-design safeguards, insufficient testing/monitoring
Robust process — not just a better model — prevents failures from recurring
Independent safety layers and redundancy are non-negotiable in safety-critical applications
Field monitoring with drift detection is essential — deployed models are never "done"

Section 7.5 recap: The most important takeaway from these case studies is that nearly every failure traces back to process gaps covered earlier in this lecture — weak or incomplete requirements from section 7.2, missing co-design safeguards from section 7.3, or insufficient testing and monitoring from section 7.4. The lesson is that robust process, not just a better model, is what prevents these failures from recurring. Independent safety layers and redundancy are non-negotiable in safety-critical applications — never let a single AI model be the only thing between the system and a catastrophic outcome. And field monitoring with drift detection is essential because deployed models are never truly done — the world changes, data distributions shift, and without monitoring you won't know until something goes wrong.

End-of-Lecture Summary

Process

Overlay MLOps onto V-Model/Agile; manage code, data, and model as parallel first-class tracks

Requirements

Specify data, statistical performance, and safety fallback behavior explicitly

Design

Co-design model and hardware together; optimize and validate post-transformation

Testing

Layer classical (HIL, WCET) with ML-specific (regression, robustness, OOD, fairness) and continuous field monitoring

Failures

Consistently trace to skipped redundancy, unrepresentative data, missing fallback logic, or absent drift monitoring

To summarize the entire five-hour lecture: First, process — overlay MLOps onto your V-Model or Agile backbone and manage code, data, and model as parallel first-class tracks with independent versioning and lifecycle management. Second, requirements — go beyond traditional functional specs to explicitly specify data requirements, statistical performance contracts, and safety fallback behavior. Third, design — co-design the model and hardware together, recognizing that hardware constraints drive model choice, and always validate optimizations on actual target hardware. Fourth, testing — layer classical embedded testing practices like HIL and WCET analysis with ML-specific practices like regression testing, robustness testing, out-of-distribution testing, and fairness testing, and close the loop with continuous field monitoring. Fifth, failures — real-world cases consistently show that incidents trace to process gaps: skipped redundancy, unrepresentative training data, missing fallback logic, or absent drift monitoring. A robust process is ultimately what prevents these failures from recurring.

Thank You

Questions & Discussion

Section 7 — Embedded System Development Process for AI Systems

Embedded System Development Processfor AI Systems

Lecture Agenda

Learning Objectives

7.1 — Development Process Modelsfor AI Systems

Why Traditional Models Fall Short

The V-Model, Adapted for AI

Agile for Embedded AI

Agile: Parallel Track Pattern

Software/Firmware Track (Scrum)

Model Track (Kanban + Experiment Tracking)

MLOps and Its Embedded Variant (TinyMLOps)

TinyMLOps: Embedded-Specific Constraints

Hybrid Process Model (Recommended)

Process Model Comparison

Section 7.1 — Recap

7.2 — Requirements Engineeringfor AI Embedded Systems

What's Different About AI Requirements

Functional & Non-Functional Requirements

Functional

Non-Functional (Embedded-Specific)

Data Requirements (Unique to AI Systems)

Model Performance Requirements

Safety & Regulatory Requirements

Elicitation Techniques Adapted for AI

Example: Smart Doorbell Person Detector

Section 7.2 — Recap

7.3 — Design & Implementationof AI Systems

Layered Embedded-AI Architecture

Hardware-Software Co-Design

Model Optimization Techniques

Optimization Impact Example

Toolchains for Embedded AI

Model Runtimes

Development Platforms

Version Control Triad

Design Patterns for Reliability

Implementation Checklist

Section 7.3 — Recap

7.4 — Integration and Testingof Embedded AI Systems

Why AI Testing Differs

The Adapted Testing Pyramid

Model-Specific Test Categories

Test Data Management

Verification & Validation (V&V) for Certification

Verification

Validation

Continuous Testing / CI for ML

Section 7.4 — Recap

7.5 — Case Studies and Failuresin AI Embedded Systems

Case 7.5.1 — Autonomous Driving: Fatal Pedestrian Collision (2018)

Context

What Went Wrong

Root Cause

Lesson

Case 7.5.2 — Flight Control: Repeated Nose-Down Commands (2018–2019)

Context

What Went Wrong

Root Cause

Lesson

Case 7.5.3 — Clinical Decision-Support AI Underperformance

Context

What Went Wrong

Root Cause

Lesson

Case 7.5.4 — Voice Assistant: False-Trigger Privacy Incidents

Context

What Went Wrong

Root Cause

Lesson

Case 7.5.5 — Industrial Predictive Maintenance: Silent Model Drift

Context

What Went Wrong

Root Cause

Lesson

Cross-Case Themes

Section 7.5 — Recap

End-of-Lecture Summary

Process

Requirements

Design

Embedded System Development Process
for AI Systems

7.1 — Development Process Models
for AI Systems

7.2 — Requirements Engineering
for AI Embedded Systems

7.3 — Design & Implementation
of AI Systems

7.4 — Integration and Testing
of Embedded AI Systems

7.5 — Case Studies and Failures
in AI Embedded Systems