As one who started my career writing software, when I hear “operating system” I naturally think of computer operating systems (OS). A computer OS provides lots of benefits to both software developers and the people who use the computer. In general, the OS defines the standards for how other software works with the computer and provides a standardized interface between the different components of the entire system (both hardware and software).
I think the analogy of a computer OS to a business OS is valid. The business OS should simplify introducing innovations into the business in a way that protects the integrity of the entire business. The business OS should provide mechanisms for gracefully dealing with conflicts within the business. The business OS should also simplify everyone’s interactions with the business (employees, customers, investors) by ensuring consistency and hiding complexity.
In reality, your business already has an operating system. It just may not be well documented, efficient, effective, or consistent. And if it’s not any of those things, then it probably isn’t helping you much in terms of supporting innovation while protecting the core. A business operating system must take into account the nature of the business and its mission/culture. The reason for implementing the business operating system will also strongly influence how it is developed. I encourage you to be very intentional about your operating system. Let me know if you need any help.