Microsoft is expanding the capabilities of its Windows Copilot Runtime by introducing AI-driven imaging APIs, designed to enhance image processing functionality within Windows applications. This update also integrates Phi 3.5 Silica, a custom generative AI model tailored for Copilot+ PCs. These advancements were unveiled at the Microsoft Ignite conference, signaling a significant step forward in developer tools for AI-enhanced applications.
The new imaging APIs will leverage on-device AI models, ensuring secure and efficient integration for developers and independent software vendors (ISVs). These APIs will be accessible starting January as part of the experimental release of Windows App SDK 1.7. Developers can expect a suite of powerful features, such as image description capabilities that generate text descriptions of images, offering improved accessibility and automation opportunities for various use cases.
Additional features include image super resolution, which enhances image quality and upscales resolution, and image segmentation, allowing for precise separation of foreground and background elements. This functionality is especially useful for applications focused on image and video editing, enabling seamless background removal powered by the Segment Anything Model (SAM).
The APIs also include object erase, which removes unwanted objects from an image while blending the surrounding background for a polished result. Moreover, an optical character recognition (OCR) API will enable developers to recognize and extract text from images, opening up possibilities for document digitization and data extraction solutions. These capabilities highlight Microsoft’s commitment to providing robust tools for developers looking to integrate AI into their Windows applications.